Reading Images: Multimodality, Representation and New Media © 2004 Gunther Kress

Image: Roman Duszek © 2003

Conference presentation Video

Keywords


Multimodality, literacy, representation, design, media affordances

Abstract


In this paper I wish to point to what I see as the central issues in the linked shifts in representation and dissemination: that is, from the constellation of mode of writing and medium of book / page, to the constellation of mode of image and medium of screen. In particular I will draw attention to consequent shifts in authority, in changes in forms of reading, shifts in shapes of knowledge and in forms of human engagement with the social and natural world.
Paper


Readers of this journal are experts in design. What I can offer is a particular take on certain issues in design from the perspective of (Social) Semiotics, and more specifically, from the perspective of multimodality, which deals with all the means we have for making meanings – the modes of representation - and considers their specific way of configuring the world. To make this concrete, here is a small example. Say I am designing a biology text-book. The subject matter is ‘plant-cells’. If I use words, I will have to say “Every cell has a nucleus”. If I use an image, I will need to place a large dot somewhere in the circle which indicates the cell to represent the nucleus (Figure 1).

Figure 1. Cell and nucleus





To reflect on this: in writing or speaking I have to use a sentence in which two entities – cell and nucleus – are related by a verb, have, indicating a possessive relation: the cell has something (much like: “I have a car, a house, two kids, etc”). I could use a different verb: “In every cell there is a nucleus”. The meaning is now quite different: about existence, there is and location, in. If I draw, I have to place a large dot representing the nucleus somewhere in a circle which represents the cell. Wherever I place it, someone looking at the image is entitled to assume that the nucleus actually is where I have placed it in the circle/cell – whether I intended to or not, or whether it actually belongs there or not. Each mode forces me into making certain kinds of commitments about meaning, intended or not. The choice of mode has profound effects on meaning, and textbook designers, for instance, need to be aware of such meaning effects of different modes.

Meanings are always disseminated through particular media: the medium of the book; or the medium of the CD-ROM, involving still and moving images, speech, writing, cartoon-like characters in comic strips, music, and so on. It might be the medium of the teacher’s body, involving speech, movement and gesture. All media offer specific possibilities to the designer, and to the reader/user in their reading and / or use.

The approach from Social Semiotics not only draws attention to the many kinds of meanings which are at issue in design, but the “social” in “Social Semiotics” draws attention to the fact that meanings always relate to specific societies and their cultures, and to the meanings of the members of those cultures. Semiotics takes the sign - a fusion of a form and a meaning – as its basic unit. In making signs we –embedded in our cultures - select forms in such a way that they expresses the meanings that we ‘have’ always ‘aptly’; hence signs always express, through their form, the meanings that the makers of signs have wished to make.

Take a simple example. I am in an American airport, looking for something to eat. I see a sign Bar and Grille, outlined in lurid red neon lights. Being hungry, I am attracted by “Grille”; I am aware that I am particularly drawn by the “e” on “Grille”. As a semiotician – even a hungry one – I wonder about this ”e”, in part because just the night before I have had a discussion with a colleague about how signs work. I order a brisket sandwich and think about this sign. What the “e” tells me is something about tradition and ‘Englishness’; it relates to many other signs I have seen where the “e” has had similar meanings, as in “Ye olde gifte shoppe”. And, even though I know it is a marketing gimmick, I want to be seduced by its meanings. Of course all the other parts of the sign also mean: the ‘Grill’ – with or without the “e” - speaks of barbeques, of the outdoors, of freshly cooked food. For the sauce I had the choice of mild, medium and make my day (- which I chose; and it did). “Bar” has its specific meanings for Americans reading the sign; and the lurid red neon sign of course ‘means’ to attract my attention, and maybe offer whatever promises ‘lurid red’, in the context of “Bar”, might hold.

All these are social meanings, specific to a particular culture. At the same time they are chosen, put together for their potential to mean, by the deliberate action of the designer. The sign - a complex message of words, of letters, of colour and font-types with all their cultural resonances - reflects the interests of its designer as much as the designer’s imagined sense of those who will see and read the sign. The sign is based on a specific rhetorical purpose, and intent to persuade with all means possible those who pass by and notice it.
Modes and their affordances: the materiality of modes
The kinds of meanings made by the letter “e”, by the word “Grille”, and by the colour “red” are different kinds of meanings. Not only do they mean different things, they mean differently. You can’t look up the meaning of “e”, nor the meaning of “lurid red”. What “e” does is not so much refer to some object, such as a Grill, or a Bar, but rather to evoke by cultural associations. It has a history of use in particular places (in ‘marketing speak’ for instance), and it is knowing its provenance that gives it its meaning. “e” puts me in the world of ‘Olde England’ with all its mythic associations. In one sense, colours work similarly: I have encountered the colour ‘red’ in many instances, as in “red light district”, as a colour of lipsticks: so in this context it is eroticized. Words have their histories, but they also refer; they name things (as nouns) or actions (as verbs) or attributes (as adjectives) or as relations of location (as prepositions), and so on.

One of the present tasks of a social semiotic approach to multimodality is to describe the potentials and limitations for meaning which inhere in different modes. For that, it is essential to consider the materiality of modes. Speech uses the material of (human) sound; writing uses the material of graphic substance. There are things you can do with sound that you cannot do with graphic substance, either easily or at all; not even imitate all that successfully graphically. The up and down of the voice, which produces the melody of (English) speech, makes many meanings, from straightforward questions to highly modulated ones: imagine saying, in a tone of incredulity, ”you did what?”; to many varying forms of emotion and affect. Even highly experienced writers find it impossible to reproduce these meanings in writing and need to take recourse to devices such as “… she said incredulously”. Maybe the major shift in the new landscape of communication in this respect lies in the increasing use of image, even in situations where previously writing would have been used. Consequently an urgent task is understanding the different affordances of writing and image.

In alphabetic cultures writing tends to start, in words, grammar and syntax, as the transcription of speech. It quickly develops its own structures and forms (syntax, punctuation, layout, for instance), so that written English is now very different to spoken English; yet writing does ‘lean on’ speech. Speech happens in time: one sound, one word, one sentence follows another. The ‘logic’ of temporal sequence is the major principle of ordering of languages such as English. Speech and writing are organized by the logic and the ordering principle of sequence in time. This underlies the syntax of English, which is enormously more complex than mere sequence, but is there nonetheless. If I have two simple sentences, such as:

The mists dissolved and the sun rose. It matters in what sequence I place them.
The sun rose and the mists dissolved is very different in meaning from
The mists dissolved and the sun rose.

The one tells us how weather works; the other puts us in the magical, mysterious world of Lord of the Rings maybe. Sequence implies causality: the sentence which comes first seems to be causally prior to that which comes after. But notice that that is so whether I want that meaning or not: I cannot but order them in some way. If I have two friends, Amanda and Josh, and they have jus got married, I might want to say either Amanda married Josh or Josh married Amanda; the two are different in causal terms – who was responsible for what. They are also different in terms of affective ‘proximity’: I may be closer to Amanda than to Josh, and so I place that person’s name first.

In speech as in writing we use words. Yet only that for which there is a word can be brought into communication: no word, no communication about it. In image, if there is something that we wish to depict, we can depict whatever we want. We don’t ask: ”Is there an appropriate image we can use?” Contrary to common sense assumptions about language, words are vague. You have no doubt fully understood the sentences about Josh and Amanda, yet you know very little about either of them: how tall Josh is, what age Amanda, what colour hair they have, and so on. If you saw a photo of them, or even a drawing, much of this would be clear. Words are (relatively) vague, often nearly empty of meanings; by contrast images are full, ‘plain’ with meaning. With image the placement of the depicted entities relative to one another in the image-space is the principle used for making meaning. Take the two images below, drawn by the then four year-old Georgia.

Figure 2a, 2b. Georgia at the side of her mother, and Georgia between her parents







The difference in meaning depends on the relation of the depicted entities to each other in the frame of the picture-space: the resultant difference in Georgia’s sense of herself and her family is an effect of these spatial relations. In drawing the materiality of sound is not available for making, to indicate just how ‘being’ Georgia’s parents seem to her, instead the affordance of space is used – making things taller or shorter, broader or thinner. In fact, Georgia was quite a bit taller than she drew herself here; and her father was quite a bit shorter than her mother. Size here shows the metaphoric use of vertical extension: Georgia sees her parents as affectively /psychically much taller than they actually were; and she makes her father seem as tall as her mother by ‘lifting him off the ground’ somewhat. That leaves aside the meanings of colour.

One further point needs to be mentioned here; it follows from the distinct ordering principles of the two modes. The written text – as indeed the spoken – forces the reader (and the listener) to stick to its order: the elements have to be read in the sequence in which they occur. That is not the case, or far less so, with the image text. Yes, the elements are there in certain spatial relations, but how the reader reconstitutes them is largely up to the reader. The order of the written text is fixed; the order of the image text is (relatively) open.

Media and their interrelation with modes
Modes and media exist in culturally and historically shaped ‘constellations’. The one that has dominated the alphabetic cultures of the ‘West’ over the last 300 years or so is that of mode of writing with medium of book and page. Writing as mode and book as medium have shaped western imagination, forms of knowledge, practices of reading; the technology of writing has shaped the book, and the technology of the book has shaped how writing has developed. The traditional book represented the work of the author, who had laboured to produce a text, which in its ordering represented a ‘body of knowledge’ or the shape of the world – whether fictional or actual. Chapters in the book were coherent and complete in themselves; paragraphs had their logic; and sentences derived their form and purpose from the organization of the paragraph and the larger text.

In that world the reader’s task was to attempt to follow the pre-given ordering of the written text, embodying the authority of the author, working assiduously to reproduce the meaning which the author had intended for the reader. In that world, authors could confidently speak and act on behalf of the reader, as did the author of the example in Figure 3, The Boy Electrician: “The prime instinct of almost any boy is to make and to create… At seven he will wire the whole house with his telephone system made from empty tins connected with varying lengths of string. His older brother will improve on that by purchasing a crystal, a telephone receiver, and a few pieces of insulated copper wire…” (p 5)
Figure 3. Spread from “The Boy Electrician”



Certain texts – novels for instance – encourage the reader to engage in the semiotic work of imagination, following the given order of words on the line but filling the relatively ‘empty’ words with the reader’s meaning. Contemporary texts - whether information books of all kinds, web-pages, the screens of CD ROMs, and so on - in their increasingly often image-like textual organization, ask the reader to perform different semiotic work, namely to design the order of the text for themselves. Consequently two phenomena are now becoming noticeable, as in Figure 4, which had been present but never noticed before: the entry point of the ‘page’ and its reading path.
Figure 4. Home page of the University of London Institute of Education (www.ioe.ac.uk)

The page of The Boy Electrician has one entry point, at the top left of the page; it had long become naturalized and therefore was no longer visible. Nor was the reading path: it asked the reader to follow the lines, in the order in which the culture had determined. The page/screen in Figure 4 has, by contrast, about 13 entry points. The reader interest determines where he or she wishes to enter the page. The same applies to the ‘reading path’ which the reader (now usually called a ‘visitor’) wishes to construct: it too is determined by the reader’s interest.

For design this is a crucial factor, and a profound change. The designer of such ‘pages’ / sites is no longer the ‘author’ of an authoritative text, but is a provider of material arranged in relation to the assumed characteristics of the imagined audience. The power of the designer is to assemble materials which can become ‘information’ for the visitor, in arrangements which might correspond to the interests of the visitor. For the visitor however “Information is material which is selected by individuals to be transformed by them into knowledge to solve a problem in their life world” (Boeck, 2002)

Making texts and reading texts
In the conception outlined here, the processes of making texts and reading texts are both are processes of design; and both are in important sense inversions of the social and semiotic arrangements of the era of the dominance of the constellation of writing and book. It has now been overtaken by the new constellation of image and screen. The (at least mythically) dominant media are now those of the screen - whether of the Gameboy, the mobile telephone, the PC, or still the TV and video. The book and its page had been the site of writing and the logic of writing had shaped the order of the page and the book; the screen is the site of the image and the logic of the image is shaping the order and the arrangements of the screen.

Writing can appear on the screen; but when it does it is subordinated to the logic of the image; just as image could appear on the page, though subordinated to the logic of writing. The logic image will more and more shape the appearance and the uses of writing, a process which is already apparent in many instances of public communication. In the former arrangement, the figure of the author and the mode of writing dominated; in the new arrangements the designer and the mode of image dominate; the story-board is an apt metaphor for this change - image led, and very often the product of a design-team.
Design as choice in context
In the multimodal landscape of communication, choice and therefore design become central issues. If I have a number of ways of expressing and shaping my message, then the questions that confront me are: which mode is best, most apt, for the content / meaning I wish to communicate? Which mode most appeals to the audience whom I intend to address? Which mode most corresponds to my own interest at this point in shaping the message for communication? Which medium is preferred by my audience? Or by me? How am I positioning myself if I choose this medium or this mode rather than those others? All of these call for choices to be made, resting on my assessment of the environment in which communication takes place, in all its complexity, in its widest sense, in which a commodity – the smell of my shampoo, the packaging of the bag of flour, the shape of the bottle of soft-drink – are all ‘messages’ to interpret. The question of choice is illustrated by the contrast of say, Figure 3 with Figure 4, or of Figure 3 with Figure 5.
Figure 5. Visual geography-tectonics

The page in Figure 3 is the realization of choices – of stylistic choices in relation to writing, choices of font (though for any one publishing house there might not have been choice), the framings of the text through syntax (marked by punctuation) and in text (marked by paragraphing, for instance), and by layout in spacings, as well as the frame around the ‘densely printed page’ (Reading Images, 1996). However, these choices had nearly faded into invisibility through the two aspects of habituation and convention. By contrast, the page in Figure 5 shows a plethora of choices made and realized through the modes of writing, layout, colour, and image.

Design is a prospective enterprise. The question it asks is: “what, in this environment, with this kind of audience, with these resources that are available for implementing my design, given these social, economic, ‘political’ constraints, and with my interests now at this moment, is the best way of shaping that which I wish to make, whether as ‘message’ or as any object (of design)?” Here, briefly, are two examples, showing choices made and interests expressed. Figure 6 is the result of the request of the teacher of a class of six year-olds to “make me a drawing and write me a story of our trip to the British museum.”
Figure 6. Notes from a trip to the British Museum

The different ‘take’ on the representation of the day in writing and drawing is startling (all the images and stories showed this contrast): salient object-entities in spatial relation in the visually represented world, contrasted with salient events/actions in temporal relations. We might dismiss this as childish representation. Or we might say that these six-year olds are using the two modes of writing and image in line with their inherent affordances – the (transformed) recollection of the visually encountered world through the spatially organized mode; and the (transformed) recollection of the actionally experienced world through the temporally organized mode. If we take that approach we see that the children have made apt use of the affordances of each mode. The facts of the representational world are certainly moving in the same direction.

In the next example two modes co-exist in one integrated textual object, the question is the same one: what are the principles for the use the modes (the question of “principled use” can and needs to be asked of all my examples). At the end of four lessons on ‘plant cells’, the teacher had asked the 14 year old students, working in groups of four, to prepare a slide of the epidermis of an onion, look at it through a microscope, and then “write what you did” and “draw what you saw”.
Figure 7a, 7b. Eye-piece of the microscope, and Cells as a “brick wall”

The teacher had given two additional instructions: ”put your writing at the top of the page, and the drawing at the bottom”, and “use only black pencils in your drawing”. Apart from the different responses to this instruction (7 b used colour pencils) there is the startling difference in what each “saw” and what each wrote. One written text is a recount, the other a procedure. The recount, generically speaking, says: “this is what happened”; the procedure, generically speaking, says: “this is what ought to happen”. The drawings differ equally profoundly. One declares “this is what theory tells us is the case” (on a worksheet there had been a comment “what you should see is something like a brick wall; each cell is a brick”); the other declares “this is what I actually observed and recorded”. The first is the "truth" of theory; the second is the "truth" of the empirical, reliably recorded.

The question of design is in the center here. The matter at issue is of course ‘plant cells’; but maybe even more than that it is: ”what is it to be scientific?” In each case the answer is broadly the same (though differently realized modally): “to be scientific is to adhere to the "truth" of theory”. In Figure 7 a, the student lodges the "truth" of the facts of the empirical world in the drawing, and the "truth" of theory through the replicability of scientific practice in the written text. In Figure 7 b, the student lodges the "truth" of scientific theory in the drawing, and the "truth" of actual practice in the written. In each case event-like representation uses the mode of writing; and the representation of object-entities is lodged in drawing.
Design as a part of rhetoric of communication
The contemporary social world is marked by increasing fragmentation and individuation (Beck, 1986); in stark contrast to the world of the 19th and early 20th century, the world of stable structures and of individual integration and definition in those structures. Strong frames, and integration into strong frames had their analogues in communication through stable genres, and through stable modal ‘choices’. In periods of stability the question of effective communication is answered by the idea of convention and of competent action in relation to those conventions. In periods of fragmentation and individuation communication is fraught: each environment of communication asks that social and ‘political’ relations, tastes, needs and desires be newly assessed. The question of rhetoric – how to make my communication most effective in relation to this audience, here and now - has moved newly, urgently into the center. Rhetoric has become a major issue for design.
Bibliography Kress, G.R. (2003). Literacy in the new media age. London: RoutledgeFalmer
Kress, G.R. and van Leeuwen, T. (1996). Reading Images: the grammar of graphic design. London: Routledge
Kress, G.R. and Van Leeuwen, T. (2002). Multimodal Discourse: the modes and media of contemporary communication. London: Edward Arnold
Jewitt, C. and G.R. Kress (eds) (2003). Multimodal Literacie. New York: Peter Lang
Burn, A. and Parker, D. (2003). Analysing media texts. London: Continuum
Gunther Kress

Gunther Kress is professor of Education/English at the Institute of Education, University of London. He has a specific interest in the interrelations in contemporary texts of different modes of communication - writing, image, speech, music - and their effects on forms of learning and knowing. He is interested in the changes - and their effects and consequences - brought by the shift in the major media of communication from the page to the screen. Some of his recent publications are: Reading Images: the grammar of graphic design; Before Writing: rethinking the paths to literacy; (both published by Routledge); Multimodal teaching and learning: the rhetorics of the Science Classroom (Continuum); Multimodal Discourse: the modes and media of contemporary communication (Edward Arnold); and Literacy in the New Media Age (Routledge).
Contact Links