Closed Form and Distributed Form

You will hear that neural networks “don’t use math,” or “aren’t really doing mathematics,” or represent some non-mathematical alternative to the symbolic tradition.

This is false and it is not a small error.

A neural network is a composition of affine transformations and elementwise nonlinearities. It is a function. It has a domain and a codomain. It is differentiable almost everywhere, which is the entire reason it can be trained. Its training objective is a well-defined statistical quantity — cross-entropy — with an exact information-theoretic meaning, as Part I established. Every operation it performs is arithmetic.

There is no less mathematics in a transformer than in Newton’s Principia. There is vastly more.

The real distinction is not mathematics versus not-mathematics. It is about where the structure sits and who can read it.

Newton’s law is an explicit compression: the discovered structure is written out in symbols, in a form a human can inspect, manipulate, and check.

The properties that follow from this are worth listing, because they are the reason the symbolic tradition dominated science for three centuries.

It can be read. The relationship is not merely present in the representation; it is stated by the representation. You can see that the force is proportional to each mass and inversely proportional to the square of the distance, and you can see it without running an experiment.

It extrapolates according to a known structure. Push the inverse square to a distance nobody has measured and you know exactly what it will say and why. There is no question of whether the model has “seen enough examples” out there — the functional form determines the answer, and you can see the functional form.

It can be verified analytically. Not merely tested — proved consistent or inconsistent with other results. You can show that Newton’s law implies Kepler’s ellipses. You can show that general relativity reduces to Newtonian gravity in the weak-field limit. That kind of check is only available on expressions you can manipulate symbolically.

It composes. Explicit laws combine with other explicit laws. This is how physics accumulates rather than merely accretes.

It transports. The law can be written on a board in Kyoto and used in São Paulo. It is substrate-independent to a degree nothing else in this essay is.

One honest qualification, since precision is the point of this series. “Closed form” applies to the law, not always to the consequences of the law. Newton’s law of gravitation is a closed-form algebraic expression. The general three-body problem it produces admits no solution in closed form: Bruns in 1887 and Poincaré in 1889 proved the system has no further algebraic or uniform analytic first integrals, and the convergent series Sundman later constructed converges far too slowly to be of any practical use. Explicit compression buys you a readable generator; it does not guarantee that everything the generator implies is readable. Compact laws routinely have intractable consequences.

A learned neural function is an implicit or distributed compression: the discovered structure is genuinely present, but it is spread across parameters in a form that corresponds to no single readable statement.

The technical term is a distributed representation, and the point of it is that concepts are not stored in locations. There is no gravity parameter. What there is, at any given layer, is a high-dimensional vector space in which directions carry meaning and features are superposed on top of each other.

The superposition part deserves a moment, because it is the crux. Anthropic’s work on toy models found that networks represent more features than they have dimensions, with features “represented as almost-orthogonal directions in the vector space of neuron outputs,” exploiting the fact that “it’s possible to have exp(n) many ‘almost orthogonal’ … vectors in high-dimensional spaces.”¹ High-dimensional geometry has an unintuitive property: you can pack exponentially many nearly-perpendicular directions into a space of dimension n, so long as you tolerate slight interference between them.

That is a compression scheme. It is a very good one. It is also the reason nothing in there is readable, because every parameter participates in many features and every feature is spread over many parameters.

Progress on reading it has been real and is worth stating precisely rather than dismissing. Sparse autoencoders trained on a production model’s middle-layer activations extracted up to 34 million interpretable features, and clamping a single one — the Golden Gate Bridge feature — to ten times its maximum activation caused the model to describe itself as the bridge.² That is a genuine handle on the interior. It is also not a napkin. Thirty-four million features described in natural language is a different kind of object from F = Gm₁m₂/r², and pretending otherwise helps nobody.

Two cautions about the geometry, because both are commonly overstated.

The famous word-vector analogy is weaker than advertised. “King minus man plus woman equals queen” appeared in a 2013 paper and became the standard demonstration that embeddings encode linear semantic structure.³ Subsequent evaluation showed the standard test excludes the three input words from the candidate answers — and without that exclusion, the offset returns one of the input words 98% of the time (93% for one of them specifically, 5% for another).⁴ The regularity works well for narrow relation types, especially morphological ones, and is substantially confounded with generic neighborhood-density effects.⁵ Something real is there. The popular story about it is not what is there.

Dimensionality is not one number. People say “these models work in twelve thousand dimensions” as though it were a property of language models generally. It is not. GPT-3’s residual stream is 12,288 wide across 96 layers, and that is a fact about GPT-3.⁶ GPT-2’s variants ranged from 768 to 1,600. Frontier model dimensions are simply not published — OpenAI’s GPT-4 report states outright that it “contains no further details about the architecture (including model size) …”⁷ And residual width is not the same quantity as parameter count, which is not the same as the intrinsic dimensionality of the learned representations, which is measured separately and comes out far lower, varying by layer.⁸ Three different numbers, three different meanings, routinely conflated into one.

The comparison people want is a ranking. The comparison that is useful is a division of labor.

Explicit compression is superior where you can get it. Readable, verifiable, composable, exact within its domain, transportable across every substrate. If a phenomenon admits a compact law, find the compact law. Nothing beats it.

Explicit compression is unavailable for most of reality. There is no equation for what makes a sentence idiomatic, what a face looks like, when a joke lands, or what a given piece of music will feel like to a listener. Not “we haven’t found it yet” — there is good reason to think the structure in these domains is genuinely high-dimensional and irregular, with no low-complexity symbolic generator waiting to be discovered. Three hundred years of trying to write down grammar as rules is the relevant historical evidence. (Protein folding is a useful contrast rather than an example: it does have a physical generator in quantum mechanics. What defeated it for fifty years was tractability, not the absence of a law — which is why a learned model could crack it.)

Distributed compression works where explicit compression is unavailable. That is the entire reason the field turned to it. The domains that resisted symbolic treatment for decades fell to learned representations within a few years of sufficient scale.

So the honest statement is:

Newton gives us the compression in closed form. A learned model gives us the compression in distributed form. The first is available only where reality happens to be simple. The second is available almost everywhere, at the cost of legibility.

And the cost is real. A distributed compression is harder to trust precisely because you cannot see the functional form and therefore cannot see where it will break. Recall the two findings from Part II: a transformer trained on Othello moves builds a manipulable internal board state, and a transformer trained on taxi routes builds an internally incoherent map whose routes stop being valid once detours are introduced at even one percent of steps.⁹ Both are distributed compressions. From the outside, before testing, they looked the same.

That is the whole liability, in one sentence. An explicit compression tells you its failure modes. A distributed compression makes you discover them.

Which is why the most interesting work in this subject is the work that converts one into the other.

Symbolic regression searches the space of expressions for a compact formula fitting the data. Schmidt and Lipson’s 2009 system, given raw motion-tracking data from a double pendulum and no physics whatsoever, recovered Hamiltonians, Lagrangians, and conservation laws.¹⁰ Udrescu and Tegmark’s AI Feynman used physics-motivated structure — dimensional analysis, symmetry detection, separability testing — to solve all 100 of 100 equations drawn from the Feynman Lectures, against 71 for the best previous public software.¹¹ Cranmer and colleagues took the further step of training a graph neural network on a physical system, then running symbolic regression on the network’s learned internal messages rather than on the raw data — extracting known force laws and producing a new analytic formula for a dark matter halo’s excess overdensity as a function of its neighbours.¹²

Read that last one again, because it is the shape of the thing. Distributed compression first, on data too raw and noisy for symbolic search. Then symbolic compression of the distributed compression. The network functions as an instrument that finds structure, and the symbolic stage converts the found structure into something a human can read.

Mathematics itself is now producing results this way. DeepMind’s collaboration with mathematicians used networks to detect patterns that suggested — and then humans proved — new theorems in knot theory and representation theory.¹³ FunSearch used a language model in a search loop to find new constructions for the cap set problem, improving on the best known results in extremal combinatorics.¹⁴ AlphaTensor found a 4×4 matrix multiplication algorithm using 47 multiplications in a finite field, improving on Strassen’s two-level algorithm there for the first time in fifty years.¹⁵ AlphaGeometry solved 25 of 30 recent olympiad geometry problems against 10 for the previous best method.¹⁶ At the 2024 International Mathematical Olympiad, AlphaProof and AlphaGeometry 2 together solved four of six problems for 28 points out of 42 — the top of the silver-medal range, one point below gold — graded independently by a Fields Medalist and the chair of the problem selection committee.¹⁷

Every one of these outputs is an explicit object. A theorem. A construction. An algorithm. A proof. Checkable by a human or by a proof assistant, portable, composable, permanent.

The direction of travel is unmistakable: distributed compression is being used to produce explicit compression.

Which is, if you look at it squarely, what human science has always been. A physicist spends decades building a distributed, largely unreadable intuition inside a biological network — an intuition they cannot fully articulate, that shows up as a sense that an equation is wrong before they can say why. And occasionally that intuition condenses into something short enough to write on a board.

Newton’s law is not the compression. Newton’s law is the readable output of a compression that lived in Newton and died with him.

The term for a distributed compression that has captured the generative structure of a domain, rather than merely its surface statistics, is a world model.

The idea is not new — Ha and Schmidhuber trained agents entirely inside their own learned simulations of an environment in 2018, and Yann LeCun has argued since 2022 that predictive world models are the missing piece of machine intelligence.¹⁸ What is new is the evidence that world models emerge from prediction alone, without anyone designing them in. The Othello and space-and-time findings from Part II are exactly that: structure nobody put there, built because it was the cheapest route to predicting the next token.

This is the thesis of the whole series arriving at the machines from the other direction. If intelligence is predictive compression, then a system trained only to predict, at sufficient scale, on sufficiently structured data, should be forced into building models of whatever generates the data. Not because anyone wanted world models. Because a world model is the shortest program that predicts a world.

And the counter-evidence is equally on-thesis. The taxi model built a world model that was incoherent — good enough to predict its training distribution, not good enough to survive perturbation. That is a compression with a short predictive horizon. It is Part I’s four dimensions again: it scored on accuracy and failed on horizon and scope, and the only way anyone found out was by asking it something it had not seen.

So there are two forms of the same thing.

One is written in symbols, sits on a page, can be checked by a stranger in another century, and covers the thin slice of reality that happens to be simple enough to admit a law.

The other is written in parameters, sits in a machine, cannot be read by anyone including its makers, and covers nearly everything else.

The first has been humanity’s most reliable instrument for three centuries. The second is four decades old in principle and roughly one decade old in practice, and it has already gone places the first could not reach.

Neither is more mathematical. Neither is more intelligent. They are the same operation — find structure in observation, discard what does not predict, keep what does — differing in whether the result is legible.

The interesting question is no longer which one wins. It is what happens as the second one gets better at producing the first.

And behind that sits a harder question, which the whole series has been walking toward and which the concluding piece will take up directly: if intelligence is predictive compression, and if predictive compression can run on either substrate, then what exactly is a human in this picture — and what would it mean for something to be very much better at it than we are?