It was with great interest I attended a workshop on multimodal analysis in HUMlab today, led by Sigrid Norris, Associate Professor of Communication Studies and Director of the Multimodal Research Centre at Auckland University of Technology in New Zealand. I would like to make a summary of some of the major points of the workshop here as a service to those who could not be there in person. While I paid careful attention throughout the workshop, it is possible I misread something, if this is the case, please comment and let me know.
My experience with multimodal analysis first began with the work of HUMlab doctoral candidate (and friend) Therese Örnberg Berglund. Since then I have explored it somewhat with the work of Gunther Kress and Theo van Leeuwen on the representational configurations that are possible with multimodal discourse. I have enjoyed the insights the work of Kress and van Leeuwen provided, but I always had some difficulties aligning them with the study of cultural communication in the context of English Literature.
Today Sigrid led a motivated group in the exploration of first what is multimodal discourse analysis, and then how it can be done.
The discussion began with the question, What are modes? While we fumbled around with various conceptions of what is a mode, Sigrid provided us with the concise definition; “a system of mediated actions”. In a material sense these mediated actions can be divided into levels. The example of ‘sound’ as a mode was (rightly) considered too general. Sound can be broken down to ‘noise’, ‘spoken language’ and ‘music’ (to name but a few). Other modes that can be used in analysis include; facial expressions, written language, bodily postures, proxemics and gaze. In responding to a question, Sigrid explained that the material, mode or object of study should direct the inquiry, and do not try to fit the question or the theory to the object of inquiry.
Once the mode has been identified, a theoretical level has to be introduced into the inquiry and analysis. Sigrid uses Activity Theory, or “social actors acting with/through a mediated means/cultural tools”. I immediately thought of Actor Network Theory and the work of Bruno Latour. Sigrid was quick to point out that Activity Theory discounted the agency of non-humans. For myself this provoked much thought and it will continue for some time yet. Multimodal Discourse Analysis differs from Activity Theory in respect to the action-levels (which they do not use). In Activity Theory actions are divided into lower-level; with beginning and ending points, composed of a single mode, they have goals and are what compose higher-level actions; these provide context to actions, and can be understood as topics and themes. Both lower and higher level actions are heuristic, in the sense they do not exhibit a reality of or on their own, but are rather parts of analysis for “experience-based techniques for problem solving, learning, and discovery” (Wikipedia).
With this basic introduction to the theoretical and modal concerns of Multimodal Discourse Analysis (time was short and it seems to fly by), we moved on to the ‘How’ component of the workshop. The method begins with transcription. This is time consuming (machine transcription is discouraged by Sigrid and I think she provided some good reasons for this, but I think it is up for discussion as well and she seems to follow the ‘it depends on the materials line’). Transcription, along with the actual utterances, actions, gestures and so on, includes:
* Transcribing the modes
* Transcribing the similarities within modes
* Explanation of contexts for the materials selected
When transcribing objects are seen as “frozen actions”, an obvious radical departure from Actor Network Theory. Texts are also ‘frozen actions’ on a higher-level. The compositional nature of the transcribed objects can be explained in terms of modal density; where rule modes are switched between foreground, midground and background. Attention to multiple groundings is at the heart of multimodal analysis. I noted that the mediation of this type of multimodal analysis is not mediated in the sense of the representational (something Kress and van Leeuwen seem to adhere to), but it is about the mediation of relationships. Once again this difference was a source of some thought for me.
I should add at this stage that the methodological focus for the workshop was the analysis of online videos. The examples provided by Sigrid were videos from her own field research, but ones that dealt with the same modes that many online videos do.
Once the analysis begins it is important to pay attention to the features of the object of study (in this case video); music and image, rhythms (actions and sound, movements) and the foreground-background continuum. This final point, the continuum between foreground and background and where the social actor can be positioned with it, represents the consciousness of the social actor. The “social actors also co-construct interactions through other modes than language” along with “other levels of attention and awareness”. Sigrid rightly (I believe) pointed out that “restricting analysis to the verbal runs the risk of missing so much information”.
Finally an important point was made, that the presentation of the data from multimodal analysis is multimodal itself.