Watson on Jeopardy!: the five question marks that mattered most

The category was U.S. Cities. It was Final Jeopardy!, so every player had to write an answer, even a machine that wasn't sure. IBM's Watson wrote "What is Toronto?????" with five question marks. The right response was Chicago. Toronto isn't a U.S. city at all.

That miss gets remembered as a joke. I think it was the most honest moment of the whole match.

Three nights in February

The contest aired February 14 through 16, 2011, two games spread across three broadcasts. Watson played Ken Jennings and Brad Rutter, the show's two biggest champions. IBM's own history page gives the final totals as $77,147 for Watson, $24,000 for Jennings and $21,600 for Rutter. Wikipedia lists the first-place prize as $1 million, which IBM gave to charity.

Jennings added a remark to his Final Jeopardy! response, a line borrowed from The Simpsons: "I for one welcome our new computer overlords." It's the part everyone quotes. The part worth studying is how Watson decided when to talk.

Confidence, not chat

According to IBM, Watson ran hundreds of algorithms at once on each clue, generated candidate answers, gathered evidence for them, and scored the results. Confidence went up when many independent methods agreed. It only buzzed in when that confidence was high enough, and each answer took about three seconds. It played without an internet connection, working from material loaded in ahead of time, such as Wikipedia, dictionaries and books. The hardware filled a room: 10 racks, 90 servers and 2,880 processor cores, built on IBM Power7. A team of more than two dozen people, led by principal investigator David Ferrucci, spent five years on it.

Notice what Watson was not. You couldn't talk to it. You couldn't tell it to explain its answer simply, or change its tone, or try again with more context. It took a clue in a fixed format and returned a short answer. Calling it a chatbot would be wrong, and calling it the ancestor of ChatGPT is a stretch. The lineage of today's chat models runs through other research.

What Watson did better than many chatbots

But Watson had one habit that a lot of modern chat tools still don't show by default. It knew how sure it was, and it acted on that. The buzzer only went off above a threshold. When it was forced to answer anyway, the question marks were its way of saying "low confidence."