What was the Med-PaLM paper?

It was "Large Language Models Encode Clinical Knowledge," posted to arXiv on December 26, 2022 by Karan Singhal and 29 co-authors. It tested large language models on medical question sets and introduced Med-PaLM.

How well did the models do?

Flan-PaLM reached 67.6% accuracy on MedQA, more than 17% above the previous best. But human reviewers found real gaps in its answers, and Med-PaLM still fell short of clinicians.

Should I use a chatbot for health questions?

Use it to prepare, not to decide. It can help you understand terms and write down questions for your doctor. The decision belongs with a professional who can examine you.

Dec 26, 2022 · Med-PaLM paper goes up on arXiv

A medical AI scored high on exams, and the paper still said it wasn't a doctor

By R.Solace · Written November 4, 2026

🤖 AI-assisted, human-curated. How we use AI →

The day after Christmas 2022, a paper went up on arXiv with a plain, confident title: "Large Language Models Encode Clinical Knowledge." Thirty authors, led by Karan Singhal. It's the paper that introduced Med-PaLM.

The headline number was strong. The more interesting part is the sentence the authors wrote right after it.

What the paper found

The team put together a benchmark called MultiMedQA, which combines six existing medical question-answering datasets, and added a new one, HealthSearchQA, made from medical questions people search for online. Then they tested PaLM, a 540-billion-parameter model, and Flan-PaLM, a version of it tuned to follow instructions.

Flan-PaLM set a new best score on every multiple-choice set in the benchmark. On MedQA, one of those sets, it scored 67.6%, beating the previous best by more than 17%.

If the story stopped there, it would be a story about a machine passing a test. It doesn't stop there. The abstract says human evaluation showed key gaps in Flan-PaLM's answers. So the authors tried a technique they call instruction prompt tuning and produced Med-PaLM. It did better. And in their words, it remained inferior to clinicians.

I respect that sentence. It would have been easy to lead with the exam score and bury the rest.

The gap between a test and a patient

Multiple-choice questions have one right answer sitting in the list. Real health questions don't. They come with a body, a history, medications, and details the person asking doesn't know are important. A model can be very good at the first kind of question and still miss things in the second.

That gap is why my rule for health, money, and legal questions is the same: use the chatbot to plan, not to decide. Let it help you understand what a word means, organize what you know, and figure out what to ask. Then take that to a person who's trained and accountable.

Here's the difference in a prompt.

Before:
I've had a headache for four days and my left eye is blurry.
What's wrong with me?

R.Solace

R.Solace writes about prompts the way a shop owner talks about orders: what you ask for is what you get.

Try free prompts →The full timeline →

A medical AI scored high on exams, and the paper still said it wasn't a doctor

What the paper found

The gap between a test and a patient

Try it

Quick answers

Sources