Constitutional AI, and the trick of letting a model grade itself

The same Thursday, two very different things showed up. One was a 51-author research paper about making assistants harmless. The other was a website where you typed "funky piano" and a picture of sound turned into actual music. Both are worth a look, but the paper is the one that still shapes how chatbots behave.

The paper

On December 15, 2022, Anthropic posted "Constitutional AI: Harmlessness from AI Feedback" to arXiv, with a summary on its own site the same day. The first listed authors are Yuntao Bai, Saurav Kadavath, Sandipan Kundu and Amanda Askell.

The idea is plain once you strip the jargon. Instead of relying on huge piles of human ratings, you give the model a short set of written principles (the "constitution") and train it in two phases.

In the first phase, the model writes a response, critiques that response against the principles, and revises it. Then it's fine-tuned on the revised versions. In the second phase, the model compares pairs of its own answers, an AI judge picks the better one, and those judgments train a preference model that guides reinforcement learning. The paper calls that part RL from AI Feedback, or RLAIF.

The outcome they report is an assistant that stays harmless without going evasive. When it turns down a harmful request, it explains its objection instead of dodging. The authors also say chain-of-thought reasoning made the model's judgments better and easier to inspect.

I think the evasiveness point is the underrated one. Anyone who used chatbots in late 2022 remembers the wall of polite refusals that said nothing. This paper was explicitly trying to get rid of that.

The music toy

TechCrunch's Devin Coldewey wrote up Riffusion that afternoon. Seth Forsgren and Hayk Martiros, who play in a band together, built it as a hobby. They fine-tuned Stable Diffusion on spectrograms tagged with labels like "blues guitar" and "jazz piano," so the image model learned what those sounds look like. Turn the generated image back into audio and you get a clip that matches the prompt.

A 512 by 512 spectrogram only covers a short clip, so full songs were out of reach. TechCrunch called it more of a demonstration. A charming one, though.

What you can steal from the paper today

You can't retrain a chatbot. But that first phase (draft, critique against principles, revise) works as a prompt, and it works better than asking for a perfect answer in one go.