Action and Meaning

Throughout the brief history of cognitive science, debates over the nature of knowledge representation have raged. In the 1970s, the debate was between those who thought that knowledge was represented as images -- modal, or sensory representations -- and those who thought that knowledge was represented propositionally. That particular debate ended in a stalemate, upon the realization that you could account for pretty much any data set from either perspective. If you can't distinguish between perspectives, you can't really debate them. Despite the stalemate, most cognitive scientists who've studied knowledge representation over the last two decades have adopted the propositional perspective.

However, with the publication of a paper titled "Perceptual Symbol Systems" by Larry Barsalou in 1999, the imagists began to make a comeback under the guise of perceptual symbol systems (PSS) theory. The basic idea behind PSS is that all knowledge representations receive their content from perceptual information. For example, when we activate our represenation of a zebra, we are activating an image of zebras. Reasoning then occurs through perceptual simulations, which I usually describe as little movies in the head (though keep in mind that perceptual representations need not be visual -- they could be auditory, gustatory, olfactory, haptic, or whatever).

The motivation for PSS is as old as thinking about thinking: the symbol grounding problem. PSS theorists argue that it impossible to connect amodal (i.e., non-perceptional, arbitrary symbols, like words) to the perceptual content that we get from the outside world. The problem is often expressed in versions of Searle's famous Chinese Room thought experiment. For example, Steve Harnad gave us " The Chinese/Chinese Dictionary-Go-Round," which goes like this:

My own example of the symbol grounding problem has two versions, one difficult, and one, I think, impossible. The difficult version is: Suppose you had to learn Chinese as a second language and the only source of information you had was a Chinese/Chinese dictionary. The trip through the dictionary would amount to a merry-go-round, passing endlessly from one meaningless symbol or symbol-string (the definientes) to another (the definienda), never coming to a halt on what anything meant.

The only reason cryptologists of ancient languages and secret codes seem to be able to successfully accomplish something very like this is that their efforts are grounded in a first language and in real world experience and knowledge. The second variant of the Dictionary-Go-Round, however, goes far beyond the conceivable resources of cryptology: Suppose you had to learn Chinese as a first language and the only source of information you had was a Chinese/Chinese dictionary! This is more like the actual task faced by a purely symbolic model of the mind: How can you ever get off the symbol/symbol merry-go-round? How is symbol meaning to be grounded in something other than just more meaningless symbols? This is the symbol grounding problem.

The problem, as expressed by Harnad and many PSS theorists, is that in order to be able to interpret and use a symbol, you'll need another symbol that expresses its meaning, and so on, in a circle or an infinite pile of symbols.

The incredibly sophisticated solution to the symbol grounding problem in PSS theory: lines.1

The symbol grounding problem is an important one, but as soon as Barsalou published his paper on PSS, it became clear to most researchers that it wouldn't solve it. The biggest fault of PSS is that to make it work, you ultimately need some amodal representations (as Barsalou and others later admitted2). If you need amodal representations, then you either have to admit that amodal representations can be connected to perceptual representations, or that PSS suffers from the exact problem that strictly symbolic (propositional) models do. Furthermore, since PSS is essentially just another version of the mental image perspective from the 1970s, it suffers from the same problem that that perspective did: how do you distinguish it, empirically, from propositional accounts? It turns out that you really can't. So, PSS doesn't really get us anywhere theoretically, and for the most part, most cognitive scientists still think of representations in propositional terms. But PSS has done one thing for the discipline: produced a bunch of interesting experiments. In the rest of this post, I'm going to describe one set of experiments, to give you an example.

One version of PSS is Art Glenberg's3 indexical hypothesis of language comprehension. I'll let Glenberg explain the basics of the indexical hypothesis to you4.

One such account is provided by the indexical hypothesis (IH), which proposes that meaning is based on action. For example, consider how a situation (e.g., a room with a chair) could be meaningful to an animal. By hypothesis, the meaning of the situation consists of the set of actions available to the animal in the situation. The set of actions results from meshing (i.e., smoothly integrating) affordances to accomplish action-based goals. Affordances are potential interactions between bodies and objects. Thus, a chair affords sitting for adult humans, but not for mice or elephants, who have the wrong sorts of bodies to sit in an ordinary chair. A chair also affords standing-on for the human. If the human has the goal of changing a light bulb in a ceiling fixture, the meaning of the situation arises from meshing the affordances of a light bulb (it can be held in the hand) with the affordances of the chair (it can be stood on to raise the body) to accomplish the goal of changing the bulb. (pp. 558-559)

The indexical hypothesis involves three processes: mapping words, phrases, or sentences onto perceptual representations (you know, mapping amodal symbols onto perception -- the very problem PSS is supposed to avoid), then deriving affordances from the perceptual symbols, and finally transforming the words, phrases, or sentences into "action-based meanings." For example5:

[O]ne can judge that the sentence, "Hang the coat on the upright vacuum cleaner" is sensible, because one can derive from the perceptual symbol of the vacuum cleaner the affordances that allow it to be used as a coat rack. Similarly, one can judge that the sentence "Hang the coat on the upright cup" is not sensible in most contexts, because cups do not usually have the proper affordances to serve as coat racks. (p. 569)

As a test/demonstration of the indexical hypothesis, Glenberg and Kaschak conducted a study in which participants read sentences one at a time, some of which made sense and some of which did not (e.g., "Boil the
air."), and were told to indicate as quickly as they could whether each sentence made sense. Of the sentences that did make sense, half involved movement away from the body (e.g., "Close the drawer.") and half motion towards the body (e.g., "Open the drawer."). In order to respond to each sentence, participants had to press one of three buttons, one close to the body, one about arms length from the body, and one in between the two. In the away-from-the-body condition, responding that the sentence made sense involved moving the hand from the middle button to the furthest button (and thus away body), and in the towards-the-body condition, responding that the sentence made sense involved moving the hand from the middle button to the closest button (and thus towards the body). The indexical hypothesis predicts that participants will be able to respond faster when their movement is consistent with the movement in the sentence. When participants read the sentence "Open the drawer," for example, they will respond that it makes sense faster when they answering involves moving from the middle to the closest button.

Guess what? It worked. Participants' reaction times improved between 25 and 100 ms, depending on the sentence type, when the movement was consistent with the meaning. Glenberg and Kaschak argue that this supports the idea that our representations of the sentences are coupled to representations of action. Presumably, reading the sentence primes the toward or away movement, which allows us to perform it faster. And you know, I buy that explanation. It makes a lot of sense that thinking about a movement would prime that movement. Does that imply that our representation of the sentence is composed of perceptual and motor information? Well, no, but who cares? As long as a few cognitive scientists do believe that, we'll get cool experiments like this one. Sure, the experiment doesn't really tell us anything about sentence comprehension (other than that, you know, sentences have meanings that are connected to the world -- wasn't that the problem we were trying to solve, not to demonstrate? I forget sometimes), but it was fun to read about it.

¹From Zwaan, R.A., & Madden, C.J. (2005). Embodied Sentence Comprehension. In: D. Pecher & R.A. Zwaan (Eds.), The Grounding of Cognition: The Role of Perception and Action in Memory, Language, and Thinking. Cambridge, UK: Cambridge University Press.
²E.g., Prinz, J. (2002). Furnishing the Mind. Cambridge, MA: Bradford Books, MIT Press.
³Glenberg, A. M. (1997). What memory is for. Behavioral & Brain Sciences, 20, 1-55.
⁴Glenberg, A. M., & Kaschak, M. P. (2002). Grounding language in action. Psychonomic Bulletin and Review, 9, 558-565.
⁵Glenberg & Kaschak (2002).