GPT-3.5 sat a practice bar exam, and its second guess was the clever part

On December 29, 2022, Michael Bommarito II and Daniel Martin Katz, posted a paper to arXiv with one of the better titles of the year: "GPT Takes the Bar Exam."

They didn't mean the whole exam. They tested OpenAI's text-davinci-003 model (one of the GPT-3.5 family) on the Multistate Bar Examination, the multiple-choice section, using a complete NCBE practice exam.

What happened

The headline number is 50.3%. That's the share of questions the model got right with the best prompt and settings the authors found. It sounds like failing, and on most of the test it was. But the paper puts the random-guessing baseline at 25%. The model doubled that, and on two subjects, Evidence and Torts, it reached a passing rate.

A couple of other details in the abstract are interesting for anyone who writes prompts. The authors tried fine-tuning the model on practice questions, and at their scale it didn't beat the plain, zero-shot version. What did help was tuning the settings and the prompt itself. So, in a paper about one of the hardest professional exams around, the wording of the question mattered more than extra training. I find that oddly encouraging.

They also said that their ability to interpret the results was limited because GPT is proprietary. They couldn't look inside it. Their conclusion was that a model would likely pass this part of the exam in the near future.

The number that matters more

Buried in the abstract is the stat I'd put on the front page. When the model ranked its top answers, the correct one was in its top two 71% of the time and in its top three 88% of the time.

Read that again. Its first pick was right about half the time. Its top three nearly always contained the right answer.

That tells you something practical about how to use these tools. The model's first answer is not its only good idea. Very often the better answer is sitting at number two, and you never see it, because you asked for one answer and it gave you one.

Ask for ranked options

For anything with judgment in it (a subject line, a name, a way to phrase a hard email, a plan), ask for a few options, ranked, with a reason for each.