UNIT 6.2 – Multimodality

Halliday and grammar

Language goes beyond formal rules of correctness. It is a means of representing patterns of experience… It enables human beings to build a mental picture of reality, to make sense of their experience of what goes on around them and inside them (Halliday 1985: 101).

E.g., US plates not only provide information, they talk to/about the owner’s experience. The meaning that you get does not come just from words.

Multimodality

Multimodality refers to a field of application rather than a theory. A variety of disciplines and theoretical approaches can be used to explore different aspects of the multimodal landscape (Bezemer and Jewitt 2010: 180)

The analysis of multimodality is intrinsically eclectic, as researchers tend to draw on various theories to carry out the study of multimodal products.

Definitions

• “texts which combine and integrate the meaning-making resources of more than one semiotic modality – for example, language, gesture, movement, visual images, sound and so on – in order to produce a text-specific meaning” (Thibault 2000, 311)
• “the use of several semiotic modes in the design of a semiotic product or event” (Kress and Van Leeuwen 2001, 20)
• “in the modern world, archetypical multimodal texts such as films, television programmes and websites, have greatly broadened the scope of such studies” (Taylor 2013:98)
  

Example: Museums

The Three-Gorges Museum, in Chonqing, that sells the visitors a very specific narrative of how millions of people had to leave their homes to build the dam: they were happy to help, and they were on their way to get rich.

Multimodality (Kress 2012)

• Verbal language is only part of the product: all modes in a multimodal ensemble are treated as contributing to the meaning of that ensemble.
• Central to meaning-making are rhetoric, selection, and design.
• In a multimodal approach, all meanings, in any mode in a culture are explicit meanings.
• Signs have meaning, signs to be explored include layout, color, writing, image, font, … (VISUAL COMMUNICATION)
  
  • The meanings of a triangle and a circle are different in the viewer’s mind.
    

The Grammar of Visual Design

Although there is no universal grammar of visual communication, Western iconographical elements exert a great influence upon other cultures.

In addition, “a spoken text is never just verbal, but also visual, combining with modes such as facial expression, gesture, posture and other forms of self-presentation” (van Leeuven 2006: 41). E.g.: Jesus Christ.

On many occasions, we get meaning through something else (not words) that is multimodal. When we speak, we provide meaning not just with the words we use but with our gestures, facial expressions, etc.

Example: Advertising

Verbal elements complemented by the photograph, who projects an image of a handsome Asian man: Chinese, English, the red color, the prestige of the name (a German gymnast of Vietnamese and German origin) and, above all, the accentuation of the Asian features.

Multimodality: images and theme

Verbal language plays a central role in the dissemination of a message, but it is often complemented by graphs, pictures, drawings, etc.

In this article, the pictures speak to the readers before they actually look at the headline. The pictures do not only complement a story, they convey a topic. The topic of the headline and the lead confirm it.

Legitimation

Images also contribute to the legitimation of official discourse.

In this section of the English language news web funded by the Chinese government, the smiling picture of the President of China is thematized and accompanies the verbal themes: Xi’s moments, Xi, Book, Experts, Xi, Xi’s global approach.

Social media

The appearance of social media has added another layer to the production and consumption of multimodal product, including (fake) news.

Compared to the mass media era, the current age displays a kaleidoscopic mediascape of television networks, newspapers, and magazines (both online and print), YouTube, WikiLeak, and LiveLeak content, Astroturf think tanks, radical websites spreading misinformation using journalistic formats, Twitter and Facebook among other social media, troll factories, bots, and 4chan discussion threads, among others. (Bennett and Livingston, 2018)

Tik Tok, Twicht, Forocoches, etc.

Disinformation occurs both in dictatorial regimes and democratic ones

Disinformation is commonly associated with authoritarian regimes such as China, among others. In China, over 2 million persons are paid a modest amount to post comments on social media. Collectively, approximately 448 million phony comments are posed by the wumao dang each year. (Bennet and Livingston 2018)

During the 2016 US presidential campaign, Donald Trump wrote: “I truly LOVE all of the millions of people who are sticking with me despite so many media lies. There is a great SILENT MAJORITY looming!”

Multimodal discourse analysis

Multimodal discourse analysis develops the Hallidayan notion of mode (and the textual function of language), referring to ‘modes of communication’ (rather than Halliday’s ‘mode of meaning’) and focusing on the fact that discourse involves multiple modes which often work together.

Three levels of multimodal DA: focus on a specific text (e.g., and advert), on the representation of an even, or the discourse of a party or brand

E.g.: Cosmopolitan is “not just about selling magazines”. It is also about selling values of independence, fun and power. Women appear as playful fantasies. The cafes they go to, and the clothes they wear are discussed. Central to the branding of magazines, TV programmes, fashion and Cosmo cafés are the strategies employed through the setting, color and lighting on the images, and the agency it gives to women in their pursuit of success and fulfilment. Machin and Thornborrow (2003) argue the power women have in the magazine, however, ‘is always connected to they body, rather than their professional competence’.

Paltridge (2006: 190)

The 3 functions of language

Halliday (1994) connected the 3 aspects of context of situation (field, tenor, mode) to functions of language:

• The ideational, for representing our experience of the world;
• The interpersonal, for creating, ratifying or negotiating our relationship with the people with whom we are communicating;
• The textual, for joining sentences and ideas together in particular ways to form cohesive and coherent texts in ways that people can make sense of.

R. H. Jones (2012) applies this to multi-modal discourse analysis.

Multimodal DA activity

Find an advertisement form a magazine, website, billboard, or some other medium which features on or more images and analyze it considering how the visual elements (as well as the text) create ideational, interpersonal, and textual meaning. Also consider how these three kinds of meaning work together to promote a particular ‘version of reality’ or to create and reinforce a certain set of social relationships. Use the following sets of questions to guide your analysis:

Ideational function (field)

• Who/What are the main participants in the image?
• Is the image narrative (representing figures involved in action or events), classificatory (representing figures in ways in which they are related to one another in terms of similarities or differences or as representatives of types), or analytical (representing figures in ways in which parts are related to wholes, e.g., objects standing for human activists)?
• What are the processes portrayed in the image and how are they portrayed?
• What are the primary vectors formed by actions, gestures, gaze, and the positioning of the figures?
• If there are multiple vectors, how do they interact with one another?
  

Interpersonal function (tenor)

• From what perspective are the figures in the image shown? How does this create a position for the viewer?
• Are the figures depicted close up or far away from the viewer?
• Are the figures looking at the viewer or away?
• What kind of relationship do they establish with the viewer through things such as gaze and gesture?
• Does the image seem realistic, and how does this affect how the viewer related to the figures of the image?
  

Textual function (mode)

• What are the most prominent and least prominent elements in the image?
• What is the center of the image?
• What is the relationship between the background and the foreground?
• What is on the top (‘ideal’/’result’) section of the image and what is in the bottom (‘real’/’cause’) section?
• What is on the left (‘given’) and what is on the right (‘new’)?
• How does the placement of elements in the image affect how the viewers’ eyes are likely to move across it?
  

Ideology (interpretation)

• How do the choices about what has been included in the image and what has been excluded portray a certain version of reality?
• Are the figures in the image portrayed is stereotypical or unexpected ways?
• Are some active or others passive? What is the significance of this?
• What do you think the image is trying to get you to think or do?