Text Data Is Running Out. Meta Is Betting on Video.

The scaling laws that drove the LLM era assumed an expanding supply of high-quality text. That assumption is running out of room. Meta FAIR researchers, alongside collaborators at NYU, have published findings suggesting that the next training frontier is not better text — it’s unlabeled video, and it comes with a surprising property: raw video without text annotations doesn’t hurt language model performance at all.

The Text Data Ceiling

The evidence for text data exhaustion is no longer theoretical. Prior research has established that language models have worked through the substantial majority of publicly available, high-quality internet text. The compounding returns from more data depend on the data being new and informative — once the same concepts, phrasings, and relationships have been seen repeatedly, additional tokens of the same type contribute diminishing marginal improvement.

The practical consequence is that labs approaching the frontier of available text data face a choice: synthesize training data (which introduces its own quality and distribution concerns), find new modalities, or accept that the current scaling trajectory has a ceiling. Meta’s research represents a serious attempt at the second option.

The Video Finding

Meta’s “Beyond Language Modeling” paper tests a direct question: does training on unlabeled video — footage with no accompanying text, no captions, no annotations — dilute language capability? The answer is that it doesn’t. Models trained on a combined corpus of text and raw video outperformed text-only baselines on language validation datasets.

The scale at which this holds is significant. Training on 20 billion visual tokens alongside 80 billion tokens from video, image-text pairs, or plain text outperformed models trained on 100 billion tokens of a single modality. More total data, from a richer distribution of modalities, produces better models than optimizing the single modality you started with.

The Scaling Asymmetry Problem

The paper surfaces a structural challenge that the finding also begins to address. Vision and language have fundamentally different scaling requirements: at 1 trillion parameters, the relative need for vision data compared to language data grows 51-fold compared to smaller models. This asymmetry means that as models scale, the vision side demands disproportionately more training data to keep pace.

Meta’s architectural response — a Mixture-of-Experts approach that allows specialized processing for each modality without proportional compute increases — reduces this asymmetry by approximately 50%. That’s not a complete solution, but it’s a meaningful compression of the structural imbalance that would otherwise make multi-modal scaling prohibitively expensive.

Yann LeCun participated in this work before his recent departure from Meta, which adds context to his long-standing position that video and embodied experience are necessary for genuine world model development — a view that this paper provides empirical support for.

Content created with AI assistance and reviewed for accuracy.