Where Do We Measure Intelligence?

The framework I use puts a coordinate on that location.

R = A / E

A is Actual — common reality as it arrives, indexed to the entity experiencing it. Not reality in the abstract; the specific slice of it that shows up at this entity’s boundary.

E is Expected — the magnitude of what the entity had committed to before Actual arrived.

R is their ratio. When the world delivers exactly what was expected, R = 1. When Actual overshoots expectation, R > 1. When it undershoots, R < 1.

Expected has internal structure. In the broader framework it is written as the magnitude of a complex quantity:

E = |P + iI|

where P is the prediction machine — the real component, the part of the entity that extrapolates forward from compressed experience — and I is idea orientation, the imaginary component, the part that is not extrapolating from what has been but is oriented toward what could be. The magnitude is what Actual gets compared against:

E = √(P² + I²)

The two components matter separately, and the complex form is doing real work rather than decoration. Pure prediction with no idea orientation is a machine that only ever expects more of the same. Pure idea orientation with no prediction is a machine that expects things with no reason. Expectation, as any actual entity generates it, is a composition of both, and its magnitude is what reality falsifies.

Then define surprise as the natural log of the ratio:

S = ln(R) = ln(A/E)

Now the geometry of the thing becomes clear. If A = E, then R = 1, and

S = ln(1) = 0

Zero surprise. The world did what was expected.

The first instinct — I have watched it happen in every room where this is presented — is to locate intelligence in P. P is literally named the prediction machine. If intelligence is predictive compression, and P is the predictive organ, then surely intelligence is a property of P.

Reject this. It is the same error as measuring neuron count, dressed in better notation.

P is where prediction is generated. It is not where prediction quality is evaluated. You can inspect P in unlimited detail — trace every parameter, map every pathway, watch the whole extrapolation form — and you will still not know whether the prediction it just produced is any good. That information does not exist yet. It comes into existence when A arrives.

This is not a limitation of our instruments. It is a fact about the structure of prediction. A forecast has no truth value at the moment of its making. It acquires one at the moment of its testing, and the testing happens somewhere else — at the boundary where E, already committed, meets A, newly arrived.

Intelligence lives in the ratio, not in either term.

Which means intelligence is relational. It is not a substance an entity possesses. It is a relationship between what an entity expected and what reality delivered, sustained across many arrivals. An entity is not intelligent in a vacuum any more than a key is functional in a room with no locks.

Here is where I want to be exact, because the framework can be misused at this point and the misuse would be fatal.

A single instance of S = 0 proves nothing.

A stopped clock reads correctly twice a day. When it does, A = E and S = 0, and the clock is not intelligent. A coin that guesses “heads” every time is right about half the time. A forecaster who always predicts tomorrow’s weather will resemble today’s is right most days in most climates, having compressed nothing beyond persistence.

Single-instance agreement between expectation and Actual is cheap and it is available to anything, including nothing.

The claim has to be about persistence under novelty:

The more intelligent the entity, the more persistently its surprise remains near zero as previously unseen Actual continues to arrive.

Three conditions are doing work in that sentence, and all three are load-bearing.

Persistently — across many arrivals, not one. A single low-surprise event is noise. A long run of them is structure.

Near zero — not exactly zero. Exact zero would mean the compression discarded nothing that mattered, which no compression achieves. We are measuring how small the residue stays.

Previously unseen — the arrivals must be outside the record. Low surprise on things already stored is a measurement of memory, and we settled in Part II that memory is evaluated against the past.

Remove any one of these and the criterion collapses into something a lookup table could satisfy.

S is signed, and this creates an immediate and serious technical problem that the framework has to face directly rather than paper over.

Because S = ln(A/E), overshooting expectation gives S > 0 and undershooting gives S < 0. The two are symmetric in log space, which is one of the virtues of using a ratio rather than a difference. But it means that you cannot score an entity by averaging its surprise.

Consider a forecaster who is wildly wrong in both directions. On Monday, Actual is triple Expected: S = ln 3 ≈ +1.10. On Tuesday, Actual is one-third of Expected: S = ln(1/3) ≈ −1.10. Average signed surprise: exactly zero.

By that measure, a forecaster who missed by a factor of three twice in two days is indistinguishable from one who was perfect twice. The errors annihilated each other.

This is not a flaw peculiar to this framework. It is the standard distinction in forecast verification between bias and accuracy, and conflating them is one of the most common errors in applied statistics. Mean signed error measures bias — systematic tendency to over- or under-predict. It says nothing whatsoever about magnitude of error. To measure accuracy you need the sign removed:

|S|     — mean absolute log error
S²      — mean squared log error

Both are honest. They differ in temperament: squaring punishes large misses disproportionately and is the right choice when a catastrophic error is worse than several small ones; absolute error treats all misses in proportion and is more robust to outliers.

Reporting both signed and unsigned versions is better than either alone, because they answer different questions. A forecaster with near-zero mean |S| is accurate. A forecaster with near-zero mean S but large mean |S| is unbiased and wrong — flailing symmetrically around the truth. That entity has no compression. It has a well-centered random number generator.

It is worth knowing that ln(A/E) is not an invention of convenience. It has an established life in the forecast-verification literature under the name log accuracy ratio, and two independent bodies of work arrived at it for exactly the reasons above.

Chris Tofallis showed in 2015 that the mean absolute percentage error — probably the single most widely used accuracy measure in business — is biased: when used to select among competing models, it systematically favors models whose predictions are too low. His proposed replacement is the log of the accuracy ratio, log(prediction/actual). He also proved something elegant: minimizing squared log-ratio error by least squares yields a model that predicts the geometric mean, exactly as minimizing squared error yields the arithmetic mean.¹

Steven Morley and colleagues independently formalized the same quantity for space-weather forecasting in 2018, and derived from it two metrics that map precisely onto the distinction above: median symmetric accuracy, built from the median of |ln ratio|, and symmetric signed percentage bias, built from the signed median.² Accuracy from the unsigned form. Bias from the signed form. Two numbers, two questions, never confused.

That is the correct treatment of S, and I am adopting it: report |S| or S² for accuracy, and signed S for bias, and never let one stand in for the other.

Now the correction that matters most, because the temptation is enormous and the error would be disqualifying.

S = ln(A/E) is not Shannon entropy. Entropy is −Σ p(x) log p(x): a functional of an entire probability distribution, summing over every possible outcome weighted by its own probability. It is a property of a distribution, not of a pair of magnitudes. S involves no distribution at all.

S = ln(A/E) is not surprisal. Surprisal is −log p(x): the negative log of the probability assigned to the outcome that occurred. It requires a probability. S is the log of a ratio of two magnitudes — how big Actual was against how big Expected was. Those are different objects. A/E is not a probability and does not behave like one; it is unbounded above, and log of it is unbounded in both directions, which no surprisal is.

S = ln(A/E) is not variational free energy. In the predictive-processing literature, free energy is an upper bound on surprisal computed through an approximate posterior over hidden causes. It is a bound on a quantity S is not.

The structural resemblance is real and worth naming precisely: S is a log ratio, in the same family as a log-likelihood ratio, and it inherits the properties that make log ratios useful — symmetry between overshoot and undershoot, additivity across multiplicative processes, dimensionlessness. That is a genuine mathematical kinship. It is not an identity, and this series will not claim one.

I am stating this plainly because the alternative — quietly borrowing the authority of information theory by using its vocabulary — is exactly the move that makes frameworks like this one dismissible. The framework does not need the loan. Log ratios are respectable on their own terms, as Tofallis and Morley demonstrate.

There is a mature discipline devoted to this exact problem — evaluating predictions against arrivals — and it is worth taking what it knows.

A scoring rule assigns a number to a probabilistic forecast once the outcome is known. The critical property is propriety. As Gneiting and Raftery define it: “A scoring rule is proper if the forecaster maximizes the expected score for an observation drawn from the distribution F if he or she issues the probabilistic forecast F, rather than G ≠ F… In prediction problems, proper scoring rules encourage the forecaster to make careful assessments and to be honest.”³

That is a remarkable thing to be able to engineer. A proper scoring rule makes honesty optimal. Under an improper rule, a forecaster who knows the rule can score better by lying about their beliefs. Under a proper rule, the best available strategy is to report exactly what you think.

The logarithmic score, S(p, x) = log p(x) — or equivalently the log loss −log p(x), which is the quantity you minimize rather than maximize — is strictly proper, and it has a further property that ought to be better known: it is essentially the unique local strictly proper scoring rule, the only one that depends solely on the probability assigned to the outcome that actually happened and ignores the shape of the rest of the distribution.⁴ If you want a scoring rule that is honest and cares only about what occurred, log score is not one option among many. It is the option.

And notice what the log score is. It is the code length. Assigning probability p to what happened costs you −log p bits to encode it. A forecaster’s log score is the number of bits they need to write down reality. Scoring a prediction and compressing an observation are the same operation.

Two further pieces of machinery are worth carrying:

Calibration versus resolution. Allan Murphy showed in 1973 that a forecast’s score decomposes into reliability (do your stated probabilities match observed frequencies?), resolution (do your forecasts vary from the base rate in ways that track reality?), and irreducible uncertainty.⁵ These come apart. A forecaster who says “30% chance of rain” on days it rains 30% of the time is perfectly calibrated and, if they say it every day, completely useless — zero resolution. Calibration without resolution is honest ignorance. Both are needed.

Confidence is not accuracy. Modern deep networks are systematically overconfident: more accurate than their predecessors and worse at knowing it, with stated confidence exceeding empirical accuracy across the board.⁶ This is directly on point. An entity’s internal certainty is a fact about P. It carries no information about the E/A relationship until A arrives. The entity that feels most sure is not thereby the entity that is most right.

Where the mapping is imperfect, I will say so. Proper scoring rules operate on probabilistic forecasts — full distributions over outcomes. The Reality Equation as stated compares magnitudes. These are not the same formalism and I am not going to force one into the other. What transfers is the discipline: separate accuracy from bias, separate calibration from resolution, separate confidence from correctness, and never score a predictor on data it has already seen.

With the measurement location established, a whole category of claims can be dismissed cleanly. None of the following is evidence of intelligence:

• many neurons
• many parameters
• a large memory
• a large database
• a complicated architecture
• prestigious credentials
• encyclopedic recall
• high confidence
• fluency
• speed
• correct answers to questions with known answers

Every one is a fact about the interior. Not one of them touches the surface where E meets A. Each can be maximized to arbitrary levels by an entity with no predictive structure at all — and each is, routinely, in both machines and people.

The procedure that does constitute evidence is unglamorous and it has four steps:

1. The entity commits to an expectation. 2. Actual arrives. 3. The two are compared. 4. Repeat, on arrivals the entity has not seen.

Steps 1 and 2 must not be reordered, and this is the condition everyone cheats. A prediction registered after the fact is not a prediction. A model evaluated on its own training data is not being evaluated. A benchmark whose answers are in the training corpus measures the database. The entire integrity of the measurement rests on E being fixed before A is known — which is why prediction markets, sealed forecasts, and held-out test sets all exist, and why all three are so easy to corrupt.

Picture the entity as bounded. Inside: the compression, P and I, whatever machinery produces expectation. Outside: Actual, arriving continuously, indifferent to what was expected.

Between them, a surface.

Intelligence is not inside the boundary. It is not outside it either — Actual has no opinion about anyone’s model. Intelligence is the quality of the fit at the surface, sustained over time, as new Actual keeps arriving.

This is why you cannot answer “is this thing intelligent” by opening it up. Interpretability research can tell you a great deal about how a model represents the Golden Gate Bridge; it cannot tell you whether the model is intelligent, because that question was never located inside the model.

You do not measure intelligence by looking inside the entity. You measure it where expectation meets Actual.

Intelligence is not measured where the prediction is made. Intelligence is measured where the prediction meets reality.

Which raises a problem nobody anticipates.

If intelligence is the sustained approach of S toward zero — if the mark of a well-compressed domain is that reality keeps doing what was expected — then intelligence, when it is working, produces nothing to notice. Zero surprise is, definitionally, the absence of an event.

The better an entity gets at predicting a region of reality, the less that region announces itself.

Which means intelligence, at its most successful, disappears.