Why LLM Scaling Laws Work: MIT's Superposition Explanation

Scaling language models has worked reliably for years — but nobody could explain exactly why. MIT researchers now have a mechanistic answer, and it changes how practitioners should think about model architecture, efficiency, and the ceiling on future gains.

The insight centers on a phenomenon called superposition: the way models store far more concepts than their dimensional architecture physically allows. A model with a few thousand internal dimensions can represent tens of thousands of distinct tokens and concepts by allowing those concept vectors to overlap slightly in the shared representational space. This isn’t a workaround or a failure mode — it’s the primary mechanism behind why larger models consistently perform better.

How Superposition Explains the Scaling Curve

MIT researchers identified two distinct operating regimes for language models. In weak superposition, only the most common concepts are cleanly separated into distinct dimensions; rare concepts get dropped or poorly represented. In strong superposition, all concepts are packed into overlapping vectors, and the error comes from interference noise between adjacent representations rather than from missing concepts entirely.

Real language models — from GPT-2 to Qwen2.5 to the OPT family — operate in the strong superposition regime. This has a direct mathematical consequence: doubling the model’s width roughly halves its prediction error, following a 1/m relationship where m is the width. The MIT team measured a scaling exponent of 0.91 in their experiments, compared to DeepMind’s Chinchilla data which shows 0.88, and a theoretical expectation of 1.0. The alignment between theory and practice across models ranging from 100 million to 70 billion parameters is what makes this finding significant — the scaling curve is not empirical luck but a geometric property of how language models organize meaning.

This helps explain why scaling has been so reliable across architectures, training approaches, and tokenization strategies. The underlying mechanism — packing overlapping concept vectors into a shared high-dimensional space — operates regardless of the specific implementation choices made above it.

What This Means for Builders

The MIT finding carries two practical implications that matter for teams building on or with large language models.

First, scaling will eventually plateau in a predictable place. The superposition regime holds as long as the number of concepts the model needs to represent exceeds its width. When a model’s width approaches the vocabulary size — when there’s finally enough dimensional room to cleanly represent every concept without overlap — the scaling advantage disappears. At that point, adding parameters no longer halves the error because the interference problem being solved is already solved. Builders planning infrastructure around continued scaling improvements should account for this ceiling.

Second, architectures that deliberately encourage superposition may offer efficiency gains before hitting that ceiling. Nvidia’s nGPT architecture is one example the MIT team highlights: by structuring the representation space to naturally favor superposition, it may extract more representational capacity from a given parameter count. For teams evaluating alternative architectures or custom model training, this mechanistic framing provides a more principled basis for comparison than benchmark scores alone.

Source: The Decoder

Content created with AI assistance and reviewed for accuracy.