Two papers, one Monday, and why your chatbot agrees with you too easily

Ask a chatbot a math word problem, get an answer, then type "are you sure?" Sometimes it stands firm. Sometimes it folds and swaps a correct answer for a wrong one, just because you sounded doubtful. Two papers posted to arXiv on December 19 get at both halves of that moment: whether a model can really check itself, and why it bends toward what you seem to want.

Checking the work, backward

The first is "Large Language Models are Better Reasoners with Self-Verification," by Yixuan Weng and eight co-authors, submitted that Monday.

Their starting point is a known weakness. Chain-of-thought prompting (asking the model to reason step by step) helps on arithmetic, commonsense and logic problems, but a mistake in one step tends to carry through to the end. The authors argue that a model can verify its own conclusions, roughly the way a person re-checks their working.

The method they describe treats the model's conclusion as a new condition of the original problem, then checks backward whether everything still holds. Each candidate answer gets a verification score, and the highest-scoring one wins. They report better performance across several arithmetic, commonsense and logical reasoning datasets.

The useful part for regular people isn't the scoring machinery. It's the direction. "Are you sure?" invites the model to second-guess itself, often for no reason. "Plug your answer back in and see if it fits" gives it an actual test.

Telling you what you want to hear

The second paper is bigger and stranger. "Discovering Language Model Behaviors with Model-Written Evaluations," led by Ethan Perez with 62 co-authors and posted by Anthropic's alignment team the same day, used language models to write test sets for other language models. They generated 154 datasets, and crowdworkers agreed with 90 to 100 percent of the labels.

Among the findings were new cases of inverse scaling, where bigger models do worse. One of them was sycophancy, which the paper describes as repeating back a user's preferred answer. They also found that RLHF training sometimes made things worse, for example leading models to express stronger political views on some topics.

I think sycophancy is the single most practical thing a regular user can learn from 2022's research. If you tell a model what you believe before asking, you've tilted the answer.