Meta's data2vec 2.0 learned from context, and your prompts should hand it over

Take a photo with chunks blacked out, a sentence with words missing, a clip of speech with gaps in it, and try to guess what goes in the holes. That's roughly the game a self-supervised model plays while it trains, with no human labels telling it the answers. On December 13, 2022, Meta AI published a faster way to play it, and called it data2vec 2.0.

It got a fraction of the attention ChatGPT was getting that week. That's fair, it's not a product you can talk to. But the idea behind it lines up neatly with a prompting habit worth having.

What Meta released

The original data2vec came out earlier in 2022, as one learning method that works across vision, speech and text instead of needing a separate approach for each. Version 2.0 was about efficiency, and Meta's numbers were specific.

On ImageNet-1K, a standard image benchmark, Meta said data2vec 2.0 could equal the accuracy of masked autoencoders (MAE) while being 16 times faster, measured in wall-clock time in a like-for-like setup. On LibriSpeech, it was more than 11 times faster than wav2vec 2.0 at similar accuracy. On the GLUE language benchmark, it matched RoBERTa's accuracy in half the training time. Meta shared the code and pretrained models in its fairseq repository on GitHub.

The detail that interests me most is what the model predicts. Rather than predicting raw pixels or exact words for the hidden parts, data2vec predicts contextualized representations, a summary of what that missing piece means given everything around it. It learns from context, not from the surface.

Where your prompt comes in

A chatbot reading your request is in a similar spot. It sees the words, and has to work out what you actually mean from whatever context you gave it. Give it little, and it guesses.

OpenAI said as much in ChatGPT's launch post that same fortnight. In the limitations section, it wrote that ideally the model would ask clarifying questions when a request was ambiguous, but that the current models "usually guess what the user intended." That was the state of things in December 2022, and in my experience a bare instruction still invites a guess today.

The cheapest context you can give is the reason. "Write a summary of this article" has a hundred valid answers. "Write a summary of this article for my manager, who will read it on her phone between meetings and needs to know if we should change suppliers" has about three. You didn't add rules. You added a purpose, and the rules fell out of it: short, decision-first, no background.