Clive Thompson has a wonderful article in the NY Times Magazine on Watson, the supercomputer programmed to excel at Jeopardy. Thompson delves into the clever heuristics used to generate singular answers to ambiguous questions. (Watson relies on massive amounts of parallel processing, so that "he" is running thousands of Google searches simultaneously.) While Watson's performance is certainly impressive, I thought the most interesting part of the story involved the failings of the machine. It's easy to rhapsodize about the ever escalating speed of microchips, but it turns out that Watson is often too slow at ringing the buzzer:
In more than 20 games I witnessed between Watson and former "Jeopardy!" players, humans frequently beat Watson to the buzzer. Their advantage lay in the way the game is set up. On "Jeopardy!" when a new clue is given, it pops up on screen visible to all. (Watson gets the text electronically at the same moment.) But contestants are not allowed to hit the buzzer until the host is finished reading the question aloud; on average, it takes the host about six or seven seconds to read the clue.
Players use this precious interval to figure out whether or not they have enough confidence in their answers to hazard hitting the buzzer. After all, buzzing carries a risk: someone who wins the buzz on a $1,000 question but answers it incorrectly loses $1,000.
Often those six or seven seconds weren't enough time for Watson. The humans reacted more quickly. For example, in one game an $800 clue was "In Poland, pick up some kalafjor if you crave this broccoli relative." A human contestant jumped on the buzzer as soon as he could. Watson, meanwhile, was still processing. Its top five answers hadn't appeared on the screen yet. When these finally came up, I could see why it took so long. Something about the question had confused the computer, and its answers came with mere slivers of confidence. The top two were "vegetable" and "cabbage"; the correct answer -- "cauliflower" -- was the third guess.
To avoid losing money -- Watson doesn't care about the money, obviously; winnings are simply a way for I.B.M. to see how fast and accurately its system is performing -- Ferrucci's team has programmed Watson generally not to buzz until it arrives at an answer with a high confidence level. In this regard, Watson is actually at a disadvantage, because the best "Jeopardy!" players regularly hit the buzzer as soon as it's possible to do so, even if it's before they've figured out the clue. "Jeopardy!" rules give them five seconds to answer after winning the buzz. So long as they have a good feeling in their gut, they'll pounce on the buzzer, trusting that in those few extra seconds the answer will pop into their heads. Ferrucci told me that the best human contestants he had brought in to play against Watson were amazingly fast. "They can buzz in 10 milliseconds," he said, sounding astonished. "Zero milliseconds!"
This anecdote highlights one of the most impressive talents of the human mind. We don't just know things - we know we know them, which leads to feelings of knowing. I've written about this before, but one of my favorite examples of such feelings is when a word is on the tip of the tongue. Perhaps it occurs when you run into an old acquaintance whose name you can't remember, although you know that it begins with the letter "J." Or perhaps you struggle to recall the title of a recent movie, even though you can describe the plot in perfect detail.
What's interesting about this mental hiccup is that, even though the mind can't remember the information, it's convinced that it knows it. We have a vague feeling that, if we continue to search for the missing word, we'll be able to find it. (This is a universal experience: The vast majority of languages, from Afrikaans to Hindi to Arabic, even rely on tongue metaphors to describe the tip-of-the-tongue moment.) But here's the mystery: If we've forgotten a person's name, then why are we so convinced that we remember it? What does it mean to know something without being able to access it?
This is where feelings of knowing prove essential. The feeling is a signal that we can find the answer, if only we keep on thinking about the question. And these feelings aren't just relevant when we can't remember someone's name. Think, for instance, about the last time you raised your hand to speak in a group setting: Did you know exactly what you were going to say when you decided to open your mouth? Probably not. Instead, you had a funny hunch that you had something worthwhile to say, and so you began talking without knowing how the sentence would end. Likewise, those players on Jeopardy are able to
ring the buzzer before they can actually articulate the answer. All they have is a feeling, and that feeling is enough.
These feelings of knowing illustrate the power of our emotions. The first thing to note is that these feelings are often extremely accurate. The Columbia University psychologist Janet Metcalfe, for instance, has demonstrated that when it comes to trivia questions, our feelings of knowing predict our actual knowledge. Think, for a moment, about how impressive this is: the metacognitive brain is able to almost instantly make an assessment about all the facts, errata and detritus stuffed into the cortex. The end result is an epistemic intuition, which tells us whether or not we should press the buzzer.
The second important feature of these feelings of knowing is their speed. As Thompson makes clear, it's the speed of these inexplicable hunches that allow the human contestants to defeat Watson. Although our meaty computer only requires 12 watts of electricity - we are a damn efficient information processing device - we're still able to react before the supercomputer, which requires a massive air-conditioner to cool itself down. In the human brain, these primal emotions have been bootstrapped to self-awareness, so that many of our feelings are short, speedy summaries of our own vast hard drive. They are what urge us to raise our hand, or keep on trying to remember a name, or press the buzzer.
The larger point is that we won't get a genuinely "human" version of artificial intelligence (not to mention more energy efficient computers) until our computers start to run emotion-like algorithms. What Watson needs isn't a bigger hard drive or some more microchips - he needs to develop feelings of knowing, which will tell him that he probably knows the answer even if he's still drawing a blank.
For decades, we've assumed that our emotions interfere with cognition, and that our computers will outpace us precisely because they aren't vulnerable to these impulsive, distracting drives. But it turns out that we were wrong. Our fleeting feelings are an essential aspect of human thought, even when it comes to answering the trivia questions on Jeopardy.
Update: Vaughan Bell has more.
- Log in to post comments
Feelings have always included a judgement of valency, which is, if you like, a kind of thinking; I guess we shouldn't be surprised that they're closer to "cognition" than we've thought.
You might be interested in the work of the psychologist Eugene Gendlin, who has created a methodology for deriving both therapeutic steps and new theories from the "feeling of knowing".
The emotional brain is the center of our cognitive apparatus. The fleeting feelings are signals to the rational brain to get ready to translate the answers for communicative purposes. We haven't been "wrong" about this for some time now.
I'm not so sure if these metacognitive premonitions should be viewed as emotions. I'm also not so sure it should so readily be applied to Watson.
Depending on how you view the Google searches Watson does, you could argue that he has either all of the knowledge on the (searchable) web, or none of it (since he has to "go out" and look for the answer). So if the human premonition comes from knowing what we know, Watson should either have it always or never.
If I understand correctly, Watson's main challenge is not retrieving the answer from a database of knowledge, but more or less understanding multiple texts (both the question and the search results). That seems like a fundamentally different task. It might be more similar to giving people a math problem. In that case, I don't think they'll have the same kind of hunches (although they will probably still be able to give an estimate of whether they will arrive at an answer in the next couple of seconds based on how far they are along already and their overall assessment of their math skills). Making such an assessment could of course also benefit machines.
Anyway, I think having these hunches in humans is very interesting. I wonder what is causing it. Could it perhaps be that memory retrieval is (in these cases) fairly trivial, but coupling it to the actual words is not? Or in some cases a failed connection between episodic and semantic memory? Interesting stuff...
Recommendation for further reading on this subject: Blink! by Malcolm Gladwell. It's an extremely interesting book on the power and nature of this sort of sub-thought thinking (hunches, etc.).
Interesting post: You reminded me of David Foster Wallace's excellent piece on the show Jeopardy, so I reread "Little Expressionless Animals", by DFW. For those interested, the story is contained in DFW's collection of stories entitled "Girl With Curious Hair".
You (parenthetically) mention right away that Watson "is running thousands of Google searches simultaneously".
Actually, Thompson (also parenthetically) notes that the computer is offline: "Watson is not connected to the Internet; like all âJeopardy!â competitors, it knows only what is already in its âbrain.â"
I was on Jeopardy some twenty years ago and won two games. I had practiced by taping the show and then using the VCR pause button as if it was the buzzer. What that taught me was exactly what you mention: By the time Alex finished reading the question, I knew that I knew the answer, even if I hadn't yet thought of the answer.
On the actual show, the most nerve wracking seconds were the few times that it took more than a second to retrieve that answer that I knew that I knew.
Implementing "feeling" in an algorithm?
Yeah, right after we solve the hard problem of conciousness (and then we will come to the conclussion that the wall behind me is currently implementing the algorithm for "feeling a craving for cherry icecream".
I've won eight Jeopardy games, and my usual routine was something like this: clue revealed, scan the text for the relevant question, grasp the answer (realizing whether I know it or not is more often than not seemingly instantaneous with verbalizing the answer -- if there's a "tip of my tongue" feeling, that was often a bad sign -- I would never buzz in without having the answer ready), double-check the answer, and prime my eyes and thumb to buzz in. Under optimal conditions, all of that would happen before Alex finished reading the clue.
Many years ago it dawned on me that, as I get older and my memory weakens, more and more often I find myself unable to retrieve a memory, but paradoxically I often know, with a feeling of certainty, that I USED to remember it. I often reflect on how bizarre it is that we can forget something we made a concerted effort to remember, yet not forget that we once knew it--a fact which we made no conscious effort at all to commit to memory.
I agree that this experience doesn't feel like an emotion. It's more like some kind of propriosense or reflexive cognition, maybe tied into the Theory of Mind (applied to ourselves?).
Yes, I agree with Mr Michael Akerman, Blink of Gladwell offers very interesting insight into this.
#8 : Perhaps not implementing "feeling" in the supercomputer, but if we were able to instead quantify the "derivative" of knowledge---able to model how rapidly the system was approaching a correct answer---we could have it predict whether it would be able to get the correct answer in the five seconds it gets after buzzing in. That way, the computer could buzz in as early (or earlier) than the contestants.
Depending on how the system builds its results (perhaps some version of Bayes' Theorem?) this could actually be possible. With Bayes' Theorem, you could measure how fast your confidence in a belief is growing---thus the "derivative" of "knowledge".
so informative, thanks to tell us.
okey oyna okeyburda okeyoyna