· Self-Instruct and HyDE papers appear
Self-Instruct taught a model with its own homework. Keep yours too.
🤖 AI-assisted, human-curated. How we use AI →
Five days before Christmas, seven researchers put up a paper with a cheeky pitch. What if a model wrote its own training instructions?
Self-Instruct
"Self-Instruct: Aligning Language Models with Self-Generated Instructions" went up on arXiv on December 20, 2022. The authors are Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi and Hannaneh Hajishirzi.
The framework works in a loop. A pretrained language model generates instructions, inputs and outputs. The pipeline throws away the invalid ones and the ones too similar to what it already has. Then the original model is fine-tuned on what's left.
They tried it on vanilla GPT-3 and reported a 33 percent absolute improvement on a benchmark called Super-NaturalInstructions, which they say put it on par with OpenAI's InstructGPT-001. In a human evaluation on new tasks written by experts, the Self-Instruct model beat tuning on existing public instruction datasets by a large margin, while still trailing InstructGPT-001 by about 5 percent. They released their synthetic dataset and code.
The filtering step is the part I'd underline. Generating thousands of examples is easy. Deciding which ones are worth keeping is the actual work.
HyDE, the same afternoon
A second paper landed the same day, from Luyu Gao, Xueguang Ma, Jimmy Lin and Jamie Callan. "Precise Zero-Shot Dense Retrieval without Relevance Labels" introduced HyDE, short for Hypothetical Document Embeddings.
The trick is fun. When you search, an instruction-following model first writes a hypothetical answer document, which may be wrong in its details. That fake document is turned into an embedding and used to find real documents that sit close to it. The authors report it clearly beat the unsupervised retriever it started from and performed comparably to fine-tuned ones, across web search, question answering and fact verification, in several languages.
A made-up answer as a search key. It sounds backward until you realize a guessed answer looks a lot more like a real answer than the question does.
The lesson: keep your own good examples
Part of Self-Instruct's recipe was keeping what was good and dropping what wasn't. It's easy to do the opposite with prompts. You write a great one, get a great answer, close the tab, and start from nothing next week.
R.Solace writes about prompts the way a shop owner talks about orders: what you ask for is what you get.