The Numerator Never Moved

01The divergence is methodological, not national

There are now five published government evaluations of Microsoft 365 Copilot — three British, two Australian — and they are the best evidence anyone has, because they were conducted by the buyer rather than the seller, describe their methods, and are sitting on public websites in full. Read them side by side and a pattern falls out that is much more interesting than any individual finding.

The evaluations that measured perception found substantial savings. The evaluations that tried to measure output found nothing. And the same institution produced both answers, in the same quarter, about the same cohort of people.

Figure 01

What each evaluation measured, and what it found

Evaluation	How it measured	Headline	What it said about output
UK GDS Jun 2025	Self-reported survey, banded answers, midpoint-imputed, capped at 60 min. No control group. 7,115 responses	26 min/day	Not measured. 82% said they would not want to return to working without it; 17% reported no saving at all
UK DWP 2025	Self-reported against a matched comparison group. No baseline, no objective metric	19 min/day	Not measured. The figure is 11–12 minutes before controlling for prior attitude to AI
UK DBT Aug 2025	Diary study plus observed tasks completed by users and non-users, marked blind	No effect found	“The evaluation did not find evidence that time savings have led to improved productivity.” On two of four blind-marked tasks the Copilot users scored lower
Australian DTA Oct 2024	Independent evaluator (Nous Group). Pre/post surveys, focus groups. 5,765 licences, 2,000+ participants, 50+ agencies	~1 hr/day claimed	Not directly measured. Report states the time figures “are approximations and likely quote the upper bound”
Australian Treasury Feb 2025	14-week trial, 218 staff, mixed methods, voluntary, no counterfactual	No clear evidence	“The evaluation did not find clear evidence that Copilot helped improve work outcomes.” Improved workflows rated “No evidence of progress”

The Australian Digital Transformation Agency’s trial is the largest and the friendliest to Microsoft: 69% of completers agreed Copilot improved the speed of their work, around 65% of managers observed a positive impact on their team, fewer than 3% saw a negative one, and 86% wanted to keep it. Those are real results and I am not going to wave them away.

But the same report contains the finding that undoes the reading. Asked about verification, participants described unpredictability and missing context that “required time spent on output verification and editing which negated some of the efficiency savings.” Quality gains were “more subdued relative to efficiency gains.” Participants said they would not use it for important documents. One department’s internal mid-trial survey found 60% had to make moderate-to-significant edits. And up to 7% of participants reported that for some tasks — slides, preliminary data analysis, code — Copilot added time.

02Why perception and output come apart

Here is the part that is not a story about Microsoft at all.

Reality is a ratio — what arrives over what was expected. A / E. A self-report instrument cannot see that ratio. It can only see the difference between how the day felt and how the respondent believes the day would otherwise have felt, which is a comparison between two expectations. A blind-marked task measures the thing that actually arrived. These are not two estimates of the same quantity. They are measurements of different terms.

This is the first principle, and it does not require anyone to be lying or mistaken. A copilot removes one obligation — produce a draft — and leaves five. You still have to know what you want. You still have to ask. You still have to read what comes back, judge whether it is right, correct it, and own the result. Several of those got harder, because evaluating work you did not produce is slower than evaluating your own. You know where your errors live. You do not know where its errors live, so you have to check everywhere.

Time saved that does not show up in the output was not saved. It was moved — out of doing and into supervising — and supervising is the part nobody wanted.

Which makes the ratio’s verdict arithmetic rather than opinion. The numerator does not move: the thing still has to be finished, judged and owned by you, and a draft is not an Actual. The denominator rises twice — once from the pitch, which promises this will be easy, and once from the invoice, because a line item creates an expectation that something changed. Same A, larger E, so R falls. That is the OO, and you feel it every time the draft comes back nearly right.

Note what is absent from that argument: any claim about quality. The model can be excellent and R still falls, because R is not a measure of how good the output is. It is the distance between what arrived and what you were already expecting. A better assistant raises E at least as fast as it raises A. On this reading, improving the model cannot fix the attach rate, and three years of improving the model has not.

03What a paid seat actually does in a day

The second thing the evaluations give you, which no vendor number does, is telemetry. The DBT evaluation reports usage for a thousand paid licences over a 63-working-day pilot, taken from Microsoft’s own Copilot dashboard.

Figure 02

Share of licensed users active, UK DBT pilot

• Weekly active64%
• Daily, working days30%
• Daily, all days21%
• PowerPoint, daily7%
• Loop / Edge, daily<1%

• 1,000 licences · 63 working days · Oct–Dec 2024 · source: Microsoft’s own dashboard
• Mean of 72 Copilot actions per user across the whole pilot — 1.14 per user per working day

Just over one interaction, per user, per working day. On a licence costing thirty dollars a month.

Which brings up the definitional point, and this is where the marketing and the measurement come apart most sharply. Gartner’s Dan Wilson, presenting that firm’s 2025 Microsoft 365 and Copilot survey, put it plainly: “The default setting within the Microsoft solution for adoption is one use for 28 days.” His own proposed bar was one use per week, “conservatively,” and probably one use per day. The GDS evaluation defines an active user identically — at least one interaction in the previous thirty days — which is how the same programme that shows 21% daily active in DBT’s telemetry reports 83% adoption in the GDS write-up. Both numbers are honest. Only one of them describes a habit.

And Microsoft’s own public bar, set out by Jared Spataro in July 2026, is “high usage, meaning monthly active usage above 80% across a customer’s user base.” Monthly. Against thirty million paid seats, the company has never published a daily or weekly active figure at all. The most specific thing on the FY26 Q4 call was that average weekly engagement is “on par with Outlook and Teams,” with no number attached to either side of the comparison.

From the same Gartner survey: of the organisations that had finished a Copilot pilot, 5% said they were moving to larger deployment. Sample of 187 IT and support leaders, 127 with pilot or rollout experience, presented May 2025. Nearly half rated Copilot as showing “some value, shows promise” — which is what a pilot says when it cannot justify the rollout and cannot bring itself to say no.

04The demand signal, when people get to choose

All of the above is about whether Copilot works. There is a separate question, which is what people pick when nothing is stopping them, and there is one good dataset on it.

Recon Analytics has been surveying US workers with paid AI subscriptions — more than 150,000 respondents between July 2025 and January 2026. Among workers whose employer offers only Copilot, 68% adopt it, which tells you the tool is usable. Among workers whose employer offers Copilot, ChatGPT and Gemini together: Copilot 8%, ChatGPT 70%, Gemini 18%. Over the same period Copilot’s share of US paid AI subscribers fell from 18.8% to 11.5%, and 44% of lapsed Copilot users gave distrust of the answers as the reason.

Eight percent, given a free choice, from inside the ecosystem where people already live all day, with the integration advantage that was supposed to be decisive. That is the number I would put next to the seat count. Microsoft disclosed over 30 million paid Copilot seats in July against more than 450 million commercial seats — under seven percent attach after nearly three years, with the distribution already paid for, the default position in every app, and a sales organisation pointed at it. Seats are growing fast; ten million in a quarter is the fastest the product has moved. Both facts are true. The pivot happened anyway.

05What Microsoft’s own trial declined to look at

The largest controlled study of Microsoft 365 Copilot was run by Microsoft. Six thousand workers across 56 firms, licences randomly allocated among eligible staff, held six months or more, outcomes taken from product telemetry. It is a serious piece of work and its results are modest and believable: twelve minutes a week less time reading email, Word documents completed about 6% faster, no detectable change in meeting load. Weekly usage among licensed workers averaged 38%.

Then there is this, in the authors’ own words: “we do not observe anything about the content produced by these workers, so our focus is on how workers allocate their time and interact with each other rather than on the quality of their work.”

The numerator was not in the study. Not hidden, not fudged — declared, on page one, as out of scope. The same is true of Microsoft’s 2023 research synthesis, which states that it “primarily explored tasks on the AI-friendly side of the jagged technological frontier” and did not attempt ecologically valid scenarios. Every rigorous measurement of this product, by its maker and by its buyers, has measured time. One evaluation measured output and reported a null.

So when Microsoft named a new category in June — Autopilots, “always-on agents that work autonomously, with their own identity,” acting “without needing to be prompted each time” so that work “continues even when your attention is elsewhere” — read it as a product description and it is unremarkable. Read it against the five evaluations and it is precise. Every clause names one of the five remaining obligations and deletes it. No prompting. No reading. No attention. That is not a better assistant. It is the abandonment of assistance as a category, by the company holding the most expensive evidence in the world that assistance does not sell.

And it is, on this reading, the correct move. A tool that leaves the verb with you can only ever raise your expectation, and the ratio will punish it every time. A tool that removes the verb entirely produces no comparison at all, because you were not expecting anything yet. R sits at one. Nothing felt.

Which is the sentence to sit with, because it is also the definition of having no error signal. The preceding essay in this series borrowed aviation’s answer to exactly that condition: not a permission prompt, which asks you to approve an action while the machine stays where it is, but a disengagement — reachable without letting go, announcing itself louder than any other alarm in the cockpit, and practised on ordinary days when nothing has gone wrong. Nobody wants a copilot. The evidence on that is now five governments deep. The open question is whether anyone intends to sell you the way back down.

06Where this stops being true

Already true

The five evaluations are published in full by the UK Government Digital Service (June 2025), the UK Department for Work and Pensions (2025), the UK Department for Business and Trade (August 2025), the Australian Digital Transformation Agency (October 2024, evaluator Nous Group) and the Australian Treasury / Australian Centre for Evaluation (February 2025). The quoted findings are verbatim. The DBT telemetry figures — 21% daily, 30% working-daily, 64% weekly active, 72 actions per user over the pilot — are from its Microsoft dashboard data over 1,000 licences. Gartner’s 5%-of-finished-pilots figure is from its 2025 Microsoft 365 and Copilot survey, n=187, presented by Dan Wilson in May 2025, as is the one-use-in-28-days remark. Recon Analytics’ figures come from its US AI survey, 150,000+ respondents, July 2025 to January 2026. Microsoft’s own randomised study is Dillon, Jaffe, Peng and Cambon, arXiv:2504.11443, and the quoted scope limitation is theirs.

What has to happen for the argument to hold

Someone has to run the DBT design — blind-marked output against a non-using comparison group — on the 2026 product, at scale, and still find nothing. If a well-powered 2026 evaluation measures output and finds a real gain, the measurement gap closes and this essay becomes a description of an early product rather than of a category. Separately, autonomous agents have to convert materially better than assistants did on comparable distribution. If Autopilot-class products also land near seven percent, the ceiling was never about assistance versus autonomy.

Where the shorthand breaks

All five trials ran in 2024, and that is the strongest fair objection to everything above. The DBT blind-marked task counts are single digits. The GDS 26-minute figure depends on midpoint imputation of banded answers with a 60-minute cap and no control group; the DWP 19 minutes is 11–12 before controlling for prior attitude to AI. The DTA trial was voluntary and self-selected. Recon’s data is self-reported and US-only. The 450-million commercial base includes frontline and education seats that were never realistic Copilot candidates, so the true addressable attach rate is better than 6.7% by an amount nobody outside Redmond can compute. And the same evaluations that produced the nulls also produced a real, replicated accessibility finding I have not given space to: neurodiverse staff and non-native English speakers reported significant, statistically detectable benefit in both the DBT and GDS studies and in the Australian Treasury trial. That result deserves its own essay and does not fit in this one.

Where I am probably wrong

The strongest objection is that measuring output at six months is measuring the wrong thing on the wrong clock. Spreadsheets did not show up in productivity statistics for a decade either, and an evaluation that blind-marks four tasks after eight weeks would have found nothing in 1983 and been wrong. If that is what is happening, then the nulls are a story about evaluation windows and I have built an argument on the limits of the instrument rather than on the limits of the tool. There is also a much duller explanation available for all of it: seven percent is simply what a thirty-dollar upsell does in year three, and the philosophy is decoration on a sales cycle. I do not think so — the 8%-when-given-a-choice number is hard to explain that way. But that is the bet, and it is the part to check me on.