The Vanishing Act

Attention is scarce. This is not a metaphor about busy lifestyles; it is a hard constraint. Sensory systems deliver vastly more information per second than can be processed at the level where conscious deliberation happens, and the excess is not queued. It is discarded.

Given scarcity, the allocation question is unavoidable: what gets processed?

The answer, empirically, is that attention is recruited by prediction failure. Not by importance. Not by magnitude. By mismatch.

The simplest demonstration is the oldest form of learning we know. Present an organism with a repeated, inconsequential stimulus and its response declines — habituation, characterized systematically by Thompson and Spencer in 1966 and observable in essentially everything with a nervous system.¹ Nothing about the stimulus changed. What changed is that it became predictable, and predictable stimuli lose their claim on the organism.

The neural version is repetition suppression: repeat a stimulus and the neural response to it shrinks. The interesting question is whether that shrinkage is fatigue or expectation, and the evidence points at least partly toward expectation — manipulate what a subject expects to be repeated, holding the actual repetition constant, and the suppression tracks the expectation.² The brain is not tired of the stimulus. The brain already had it.

The electrophysiology gives it a signature. Play a sequence of identical tones and then a deviant one, and roughly 100–250 milliseconds later there is a negative deflection in the difference wave — the mismatch negativity. It appears without attention, without task relevance, without the subject noticing. The brain has registered a violated regularity before anyone decided to care.³

And the cost of this allocation policy is measurable. In the best-known demonstration in cognitive psychology, subjects counting basketball passes failed — 46 percent of them, across all conditions — to notice a person in a gorilla suit walking through the middle of the scene, stopping, and beating their chest.⁴ The gorilla was large, central, and visible. It was not attended to, because attention was elsewhere and the gorilla was not a violation of what the visual system had been asked to model.

There is a research program built entirely on this, and it deserves to be stated carefully — both because it is the most developed formal account of the phenomenon and because it is more contested than popular treatments admit.

Rao and Ballard’s 1999 model of visual cortex proposed that feedback connections from higher to lower cortical areas carry predictions of lower-level activity, while feedforward connections carry the residual errors between those predictions and what actually arrived.⁵ Trained on natural images, such a network develops receptive fields resembling those of real simple cells, and the error-carrying units reproduce extra-classical effects like end-stopping. The architecture is inverted from the textbook picture: what flows up is not the signal but the part of the signal that was not anticipated.

Karl Friston generalized this into the free-energy principle. In his own summary: “Adaptive agents must occupy a limited repertoire of states and therefore minimize the long-term average of surprise associated with sensory exchanges with the world.” And: “Surprise rests on predictions about sensations, which depend on an internal generative model of the world. Although surprise cannot be measured directly, a free-energy bound on surprise can be, suggesting that agents minimize free energy by changing their predictions (perception) or by changing the predicted sensory inputs (action).”⁶

That second clause is the elegant part. Two ways to reduce mismatch: update the model, or change the world until it matches the model. Perception and action as a single operation viewed from two sides.

Within this framework, attention has a formal definition. It is the precision assigned to prediction error — the estimated reliability of an error signal, implemented as gain on the units carrying it.⁷ Attending to something is not shining a light on it. It is deciding that mismatch from that source is trustworthy enough to act on. Errors you judge unreliable get down-weighted into irrelevance; errors you judge precise get amplified into consciousness.

Andy Clark’s framing, which pushed this into philosophy of mind, is that brains are essentially prediction machines, continuously attempting to match incoming sensory input against top-down expectation.⁸ Jakob Hohwy extended the same principle across perception, attention, action, delusion, and the self.⁹

Now the part that most treatments skip.

Predictive processing is a serious, productive, and genuinely contested framework. It is not established fact, and this series will not lean on it as though it were.

The most careful recent assessment of the neurophysiological evidence puts it plainly: these models “have become increasingly influential in cognitive neuroscience,” but “are also criticized for lacking the empirical support to justify their status.”¹⁰ The same review notes that many of the phenomena routinely cited as evidence for predictive processing “can also be accommodated within traditional models by invoking additional mechanisms” — repetition suppression, for instance, is explicable by plain neural adaptation without any predictive machinery. Studies testing the framework’s most distinctive claims have produced directly conflicting results.

The honest verdict, as of the most recent thorough review: real support for the general claim that top-down expectation shapes early sensory processing; weak and contested support for the specific architectural claims — segregated prediction and error populations, particular laminar and oscillatory signatures, hierarchical message-passing.

There are structural objections too. The “dark room problem” asks why, if organisms minimize surprise, they do not seek out the most predictable possible environment and remain there. The standard reply is that organisms carry evolutionarily-shaped priors entailing exploration and homeostasis, so a dark room is in fact highly surprising for a creature like us.¹¹ That reply is coherent, and it also illustrates the deeper worry: a framework that can absorb any counterexample by positing whatever priors are needed to fit the observed behavior is difficult to falsify. Philosophers have documented this explicitly — the free-energy principle “has been called a postulate, an unfalsifiable principle, a natural law, and an imperative,” and “its epistemic status is unclear.”¹² A sharper critique argues that the framework illegitimately slides from Markov blankets as a statistical modeling device to Markov blankets as the literal physical boundary of an organism.¹³ Even the attention-as-precision reduction has drawn direct dispute.

I am reporting all of this because the argument of this series does not depend on predictive processing being correct. The argument depends on something much weaker and much better established: attention is recruited disproportionately by prediction failure. That is supported by habituation, by mismatch negativity, by inattentional blindness, and by the plain phenomenology of the floor. Predictive processing is the most developed theory of why. If it is wrong in its architectural specifics, the observation survives.

One more distinction, because it is constantly blurred. Midbrain dopamine neurons encode a reward prediction error — the gap between received and expected value.¹⁴ That is a different signal, in a different system, in a different currency, from the perceptual prediction error of Rao and Ballard. Both are “actual minus expected,” and the shared mathematical form is genuinely interesting, but they are not the same thing and treating them as one is a popularization error. Actual arrives in more than one register.

Now put the pieces together, because the consequence is strange and I think underappreciated.

Attention goes where prediction is failing. Prediction fails where compression is poor. Therefore conscious experience is systematically biased toward the regions where an entity’s intelligence is worst.

Not where it is best. Where it is worst.

Every domain you have genuinely mastered has become invisible to you. The native speaker does not experience grammar; the grammar of a language you are learning is nothing but experience. The experienced driver does not experience the clutch; the learner experiences almost nothing else. The physicist does not experience the algebra; they experience only the part that will not come out. Mastery is precisely the process of a domain going quiet.

Which produces the observational effect at the center of this piece:

We experience the failures of intelligence far more vividly than its successes. The successes disappear into normality.

This is not a psychological quirk. It is forced by the structure. A success is S near zero, and S near zero is nothing happening. There is no way to build a system that both predicts well and continuously experiences its own predicting, because the experience is made of the residue and good prediction leaves none.

The implications compound in unpleasant directions.

Skill is systematically under-credited. The competent person, doing the thing they are best at, has the least to report about it. Ask an expert how they did it and you will usually get a bad answer — not because they are withholding, but because the process that produced the answer generated no prediction error and therefore left no experiential trace. Expertise is largely unavailable to introspection by construction.

Infrastructure is invisible until it fails. Water systems, power grids, supply chains, immune responses, institutional norms. Every one is a compression of enormous accumulated learning about what reality does, and every one is experienced as nothing right up until the moment it stops working — at which point it is experienced as a catastrophe and, frequently, as evidence that it was never any good.

Progress feels like decline. As predictive systems improve, the set of things that surprise us shrinks toward the residual — the hard cases, the anomalies, the failures. Attention, being drawn to mismatch, concentrates entirely there. The result is a population that experiences a steadily improving system as a steadily worsening one, because the improving part goes silent and the failing part gets all the light. This is not irrationality. It is attention working exactly as designed, applied to a moving baseline.

And this applies with unusual force to artificial intelligence. A model that produces ten thousand competent responses and one fabricated citation will be discussed almost exclusively in terms of the fabricated citation. The ten thousand generated no surprise, so they generated no experience, so they generated no discourse. The one failure generated all three.

I want to be careful here, because this is where the argument could be misused. The failures are real and they matter. A fabricated citation in a legal brief is a serious professional harm regardless of how it was produced, and the fact that attention is drawn to failures does not mean attention is wrong to be drawn to them. It is the correct allocation policy: mismatch is where information is, and where the work remains.

The claim is narrower and it is about calibration. When you form a judgment about the intelligence of any entity — an institution, a person, a machine, yourself — you are working from a sample that is structurally biased toward that entity’s worst-compressed domains. The successes are not merely underweighted in your assessment. They were never in your assessment, because they were never in your experience.

An entity that predicted its entire environment perfectly would have S = 0 everywhere, always. Nothing would ever mismatch. Nothing would ever recruit attention. There would be no prediction error anywhere for consciousness to be made of.

Whatever such an entity is, its inner life would not resemble ours — and by every account of consciousness that ties experience to prediction error, it might not have one at all.

That is the shape of the thing:

Intelligence, at the limit of its success, erases the evidence of itself.

We only ever see it working badly. When it works well, there is nothing to see.

Which sets up the last confusion this series has to clear away — the most common one, and the one that has done the most damage to how people evaluate both machines and each other.

If we only notice the failures, we will draw our conclusions from them. And the failures we notice most are not failures of intelligence at all.

They are failures of memory.