The Unreasonable Effectiveness of Compression

Start by stripping the mystique off compression, because compression by itself is worthless.

There are roughly three hundred forty million people in the United States. I can compress all of them into a single word: people. That is a reduction of enormous magnitude — hundreds of millions of distinct entities, each with a full biography, collapsed into six letters. By any measure of raw size reduction, it is a spectacular compression.

It is also nearly useless. The word people tells you almost nothing about next year’s unemployment rate. It does not predict fertility, diet, voting behavior, median income, migration, or the failure rate of small businesses in Ohio. It discards essentially all of the structure that would let you say what happens next. It is a very small representation of a very large thing, and it buys you nothing.

Now consider a different compression of the same population: an actuarial life table. It is also small. It also throws away almost everything about every individual. But it will tell you, with unnerving accuracy, how many of those three hundred forty million people will be alive in twelve months. It survives contact with the future.

Both are compressions. Only one of them is intelligent.

Compression alone is cheap. Predictive compression is intelligence.

This is the distinction the rest of this series is built on, so it is worth stating in its sharpest form:

A compression earns the name intelligence when it preserves enough structure from the past to predict previously unseen Actual.

Not summarize. Not store. Not retrieve. Predict — specifically, predict things that were not in the record when the compression was made.

Take four observations, in Cartesian coordinates:

(1, 7)
(2, 9)
(3, 11)
(4, 13)

There are two fundamentally different things an entity can do with these.

The first is to store them. Four ordered pairs, faithfully recorded. Ask the store for the value at x = 3 and it returns 11, instantly and correctly. Add nine hundred ninety-six more observations and it will hold all thousand with perfect fidelity. Nothing is lost. Nothing is approximated. This is a database, and databases are extraordinarily good at what they do.

The second is to notice that

y = 2x + 5

The equation is smaller than the data. Six characters against a thousand ordered pairs. Measured as storage reduction alone it is an impressive result, and it is also the least interesting thing about it.

The interesting thing is that the database can answer questions about points 1 through 1,000, and the equation can answer questions about point 1,001. And point 3,000. And point one trillion. The database’s competence ends precisely where its records end. The equation’s competence does not end anywhere, so long as the relationship that generated the data continues to hold.

The database stores observations. The model stores the relationship that produced the observations.

A database remembers the past. A model can contain the future.

And so the criterion for a good compression is not what most people assume. The goal is not to find the smallest description of what happened.

The ultimate compression is not the smallest representation of what happened. It is the smallest representation that can tell us what happens next.

This idea has a serious lineage, and it is worth knowing that the ground under it is solid.

Claude Shannon established in 1948 that information has a measure.² For a source emitting symbols with probability p(x), the optimal code length for symbol x is −log₂ p(x) bits, and no uniquely decodable code can beat the entropy of the source on average. This is the hinge on which everything else turns, because it means a probability and a code length are the same object viewed from two directions. If you can predict well, you can compress well. If you can compress well, you were predicting well whether you meant to or not.

Shannon proved this on himself three years later. In “Prediction and Entropy of Printed English,” he sat human subjects down and had them guess the next letter of a text, letter after letter, and estimated the entropy of English from the statistics of their guesses: somewhere between 0.6 and 1.3 bits per letter.³ It is, seventy-five years early, a next-token prediction benchmark.

In the 1960s, three people independently arrived at the same deeper idea from three directions. Ray Solomonoff, Andrey Kolmogorov, and Gregory Chaitin each defined the complexity of an object as the length of the shortest program that produces it on a universal machine.⁴ The definition is machine-independent up to an additive constant, which is what makes it a real quantity rather than an artifact of notation. It is also uncomputable — you cannot in general find the shortest program, because doing so would require solving the halting problem. This matters, and I will return to it: the ideal compression is provably out of reach, which means every actual intelligence is an approximation to something it can never attain.

Solomonoff went further and proved the theorem that this entire series is a commentary on. Define a universal prior over hypotheses that weights each program by 2 raised to the negative of its length — short programs are more probable, long ones less. Predict using that prior. Then, for any computable process generating the data, the total expected squared error of your predictions, summed over the entire infinite future, is bounded by a constant:

Σ Sₙ ≤ K(μ) · ln 2 / 2

where K(μ) is the Kolmogorov complexity of the true generating process.⁵ Read that again. Not the error per step. The error summed over all steps, forever, is finite and bounded by the compressibility of reality itself. A predictor built on the principle “prefer the shortest explanation” converges on the truth, and the total price it pays for getting there is set by how compressible the truth was.

Jorma Rissanen made the practical version of this in 1978 with Minimum Description Length: choose the model M minimizing L(M) + L(D|M), the cost of describing the model plus the cost of describing the data given the model.⁶ Chris Wallace and David Boulton had arrived at essentially the same principle a decade earlier under the name Minimum Message Length.⁷ Both are Occam’s razor rendered as arithmetic. Overfitting, in this language, is not a mysterious statistical pathology. It is a failure of compression — a model that costs more to describe than the data it was supposed to explain.

And in 1984 A. P. Dawid put the epistemological point plainly. The purpose of statistical inference, he argued, is “to make sequential probability forecasts for future observations, rather than to express information about parameters.”⁸ Not fit. Forecast. The model is judged as the future arrives.

Shane Legg and Marcus Hutter later wrote the whole thing as a single expression. An agent’s universal intelligence is

Υ(π) = Σ_μ 2^(−K(μ)) · V_μ^π

— its expected performance across all computable environments, each weighted by its algorithmic simplicity.⁹ Performance, weighted by compressibility. That is a formal definition of intelligence with Solomonoff’s prior sitting inside it.

One caveat belongs here, because it is the strongest objection to everything above. Wolpert’s No Free Lunch theorems establish that averaged uniformly over all target functions, any two learning algorithms have identical expected performance off the training set.¹⁰ No compression scheme is universally superior. This is true, and it does not damage the thesis — it explains it. The load is carried entirely by the word uniformly. Averaged over all logically possible universes with equal weight, prediction is impossible. We do not live in all logically possible universes with equal weight. We live in one that is unreasonably compressible, and Wigner’s puzzle is precisely the observation that we do.

Newton did not maintain a database of falling objects.

The Principia of 1687 contains no table of every apple, cannonball, and planet ever observed. It contains a relationship, stated — and this detail matters — as a proportionality rather than an equation. Newton wrote that gravity is “proportional to the several quantities of matter which they contain,” and elsewhere that it holds in the “reciprocal duplicate proportion” of distance.¹¹ There is no G anywhere in Newton. The constant arrived later; Cavendish, in 1798, was not measuring G at all but the density of the Earth, and the modern notation F = Gm₁m₂/r² is a convention of the 1870s and 1890s.¹² Newton compressed the heavens into a ratio.

It is worth being precise about what that compression buys, because it is routinely stated wrong. Newton’s second law alone — F = ma — does not imply that a feather and a bowling ball fall together. You need the gravitational relation as well. Combine F = GMm/r² with F = ma and the falling body’s own mass appears on both sides and cancels:

a = GM/r²

The acceleration is independent of what is falling. And that cancellation is not a logical necessity; it happens only because inertial mass and gravitational mass are empirically equal — a fact no theory has derived from anything more basic, which general relativity elevates to a postulate rather than explains, and which everybody has repeatedly measured. On August 2, 1971, David Scott dropped a 1.32 kg aluminum hammer and a 0.03 kg falcon feather from about 1.6 metres on the surface of the Moon, on television. NASA’s own mission report notes the result was “predicted by well-established theory, but a result nonetheless reassuring considering both the number of viewers that witnessed the experiment and the fact that the homeward journey was based critically on the validity of the particular theory being tested.”¹³

That last clause is the whole argument in one sentence. Three men were riding home on a compression.

The compression’s reach is easier to see in the sky. In 1705, Edmond Halley looked at the comets of 1531, 1607, and 1682, concluded they were one object, and wrote: “Hence I dare venture to foretell, That it will return again in the Year 1758.”¹⁴ He died in 1742. On Christmas Day 1758, a German farmer named Johann Georg Palitzsch found it, and it reached perihelion on March 13, 1759 — within about a month of a refined prediction that Alexis Clairaut, Jérôme Lalande, and Nicole-Reine Lepaute had computed by hand, accounting for the perturbing pull of Jupiter and Saturn.

A dead man predicted a return nobody had witnessed, fifty-three years in advance, because he had a compression rather than a record.

And then Mercury.

In 1859, Urbain Le Verrier — who had found Neptune by arithmetic — noticed that Mercury’s orbit was wrong. Its perihelion advanced faster than Newton allowed. The number is small and it is almost always quoted carelessly, so here it is stated properly. Mercury’s perihelion appears to advance about 5,600 arcseconds per century relative to the equinox. Roughly 5,025 of those arcseconds are not Mercury moving at all; they are the Earth’s own reference frame precessing under our feet. About another 532 come from the real gravitational tug of Venus, Jupiter, Earth and the rest. Subtract both and about 43 arcseconds per century remain, unexplained.¹⁵

Forty-three arcseconds per century is about one forty-third of the apparent width of the full Moon — a human hair held at arm’s length, roughly a third of a metre away — accumulated over a hundred years. It is 0.77 percent of the total. Le Verrier proposed an unseen planet — Vulcan — and generations of astronomers went looking. It was never there.

Einstein published the resolution on November 18, 1915.¹⁶ General relativity produces the 43 arcseconds without adjustment, without a new planet, without a free parameter. Newton’s compression had been discarding something for two hundred and twenty-eight years, and Mercury was the first place the discarded thing became large enough to see.

Notice what did not happen. Newtonian mechanics was not thrown away. It is still what flies spacecraft; navigation software is built on a perturbed Newtonian framework with relativistic corrections layered on. GPS is the sharpest illustration: an orbiting clock runs about 45.7 microseconds per day fast from the weaker gravitational field and about 7.1 microseconds per day slow from its orbital velocity, for a net gain near 38.6 microseconds per day, and the correction is engineered into the satellites’ clock frequency before launch.¹⁷ Newton plus a correction, still, in 2026.

George Box gave us the standard phrasing. In his 1976 Fisher Memorial Lecture he wrote: “Since all models are wrong the scientist must be alert to what is importantly wrong. It is inappropriate to be concerned about mice when there are tigers abroad.”¹⁸ The famous compact version — “Essentially, all models are wrong, but some are useful” — appears as a complete sentence only later, in Box and Draper’s 1987 book.¹⁹

Box was making a statistician’s point. Sharpen it into an epistemological one:

All compressions discard something. Intelligence is revealed by how long the discarded information remains irrelevant to prediction.

Newton discarded the curvature of spacetime. That omission stayed irrelevant for two and a quarter centuries and remains irrelevant today for almost every purpose a human being has. That is not a defect of the compression. That is a measurement of its quality.

Which suggests that intelligence is not one quantity but several, and I will hold these loosely for now rather than pretending they form an equation:

• Compression — how little structure is required to do the predicting.
• Predictive accuracy — how closely the retained structure matches Actual.
• Predictive horizon — how far forward it holds before the discarded material starts to matter.
• Predictive scope — how wide a range of phenomena it covers.

Newton scores extraordinarily on all four, and eventually — only eventually — gives way on the third. Vulcan scored well on none.

Now the reason this is not merely a philosophy of science.

A large language model is trained by next-token prediction: adjust parameters to maximize the log-probability the model assigns to the token that actually came next. That objective has a name in statistics — log loss, or cross-entropy — and by Shannon’s correspondence it is identical to the code length the model would achieve as a lossless compressor. This is not an analogy. A DeepMind team stated it flatly in 2023: “minimizing the log-loss is equivalent to minimizing the compression rate of that model used as a lossless compressor with arithmetic coding, i.e., current language model training protocols use a maximum-compression objective.”²⁰

Training a language model is running a compression contest. Marcus Hutter has been saying so since 2006, when he put up prize money for compressing Wikipedia: “Being able to compress well is closely related to intelligence… If you can compress the first 1GB of Wikipedia better than your predecessors, your (de)compressor likely has to be smart(er).”²¹ The claim has since been measured. Across 31 public language models and 12 benchmarks, downstream capability tracks compression efficiency with a Pearson correlation around −0.95 — as close to a straight line as this field produces.²² The same DeepMind paper found Chinchilla 70B, trained on text, compressing ImageNet patches to 43.4% and audio to 16.4% of raw size, beating PNG at 58.5% and FLAC at 30.3% at their own specialties.²³

Those are raw rates, and the paper is explicit that they disregard the size of the compressor. Count Chinchilla’s seventy billion parameters as part of the code and the two-part description is worse than the raw file by two orders of magnitude. That is not a footnote to be buried — it is the honest boundary of the claim. A language model is a superb compressor given the model, and an absurd one if you have to ship the model with every file. Which is exactly why the interesting quantity was never file size. It is how much of reality one fixed model can go on predicting.

So: are these models just very large databases?

No, and the arithmetic settles it before the philosophy starts. They do memorize — this is established, not speculative. Researchers have extracted verbatim training data from GPT-2, shown that memorization grows with model scale and data duplication, and pulled over ten thousand unique memorized examples out of a production system for two hundred dollars in queries.²⁴ Anyone who claims a language model contains no memorized text is wrong.

But recent work measuring capacity finds models in the GPT family hold roughly 3.6 bits per parameter.²⁵ Extrapolate that rate — and it is an extrapolation, since the measurements ran from half a million to 1.5 billion parameters — and a 175-billion-parameter model has a total memorization budget on the order of 79 gigabytes, against a training corpus far larger and a space of possible outputs incomparably larger than either. Most of what such a model does cannot be stored. It has to be reconstructed.

And the same research found something that should be pinned to the wall of every argument about this: models memorize until their capacity fills, and then generalization begins. Compression is not a side effect of scale. It is what a system is forced into when it runs out of room to remember.

Ask a model for a children’s book about a lighthouse keeper who is afraid of the dark, in anapestic tetrameter, ending on an unresolved chord. No such object exists. The model does not retrieve it. It predicts it — from a learned structure that encodes what “book” means, what “children’s” implies about vocabulary and stanza length, what anapestic tetrameter does to a line, how narrative tension resolves or refuses to. That structure was compressed out of an enormous number of examples, none of which was this one.

Which brings us to the failure that everyone reaches for, and which I want to reframe rather than excuse. The same model that produces that book may state the wrong year for a treaty. It predicts bookness superbly and a date badly.

That is not a contradiction. It is a resolution boundary. Predictive competence is not one thing — it exists at levels, and a compression can be excellent at the level of structure while having lost the level of specific fact. Genre survives compression. An isolated date, mentioned once in a corpus, often does not.

That distinction is where this series is going. But first there is a simpler question to settle, and it is the one the thousand points were really asking.

Where, exactly, does compression stop being storage and start being intelligence?

The answer is not in the model. It is at a boundary the model has to cross.