A pilot exists to produce a decision, and most of them produce a testimonial instead. McKinsey’s 2026 survey of 1,719 respondents in 97 nations found 80% saying AI had improved their own productivity while 6% credit it with 5% or more of company EBIT. Your pilot measures the first number. The rollout you are about to fund will be judged on the second.
Four of the ten rules below belong in place before the first session, where most AI enablement pilots lose the reading. If you are reading this with a finished pilot and a happy group, some of the error is already baked in, and the honest move is to name it rather than round up.
The decision a pilot exists to produce
Most advice here was written for a different reader. Search the term and you get scorecards built on tool-call failure rates, straight-through completion and P95 latency: right for shipping an agent into production, wrong for a 140-person company that gave its finance team assistants and ran two sessions.
Your pilot is not a system under test. It is a group of people under test, which moves the error somewhere else. The gain you measured belongs partly to who was in the room, partly to the novelty of a new tool, and partly to one person who spent evenings keeping the group going. Multiply the headcount by twelve and two of those three vanish.
So the decision is narrow: does the result hold when the people change, the attention thins and the novelty fades.
Four rules to set before the first session
Rule 1. Write the three thresholds before you pick the people
Name the number that triggers a rollout, the one that buys another iteration, and the one that ends it. Write all three down, date them, send them to whoever signs the next invoice.
In the wrong order, the thresholds get written to fit the result. Almost nobody does it first, because a pre-committed stop threshold feels like forecasting your own failure to a CFO. It reads as the opposite: a program that arrives with a stopping rule gets believed when it reports a win, and a pilot that cannot fail has told you nothing.
Rule 2. Staff the cohort to look like the company, not like the volunteers
Who you train decides your headline number more than what you teach. In the study of 5,179 customer support agents by Erik Brynjolfsson, Danielle Li and Lindsey Raymond, access to an assistant raised issues resolved per hour by about 14% on average, roughly 34% for novice and lower-skilled workers, and close to nothing for the most experienced and highest-skilled.
Read that as a warning about pilot design. Eager juniors post a number your senior staff cannot reproduce; three sharp operators post a flat one that condemns a program the rest of the company needed. Take a mixed group: a span of tenure, one skeptic, two who did not volunteer.
Rule 3. Baseline one repeating task with a timestamp, after the licenses are live
Pick one deliverable the group produces weekly that someone already timestamps: a reconciliation, a proposal pass, a client summary. Time it for two weeks before the first session and after the licenses are switched on, so the software’s own effect sits inside the baseline and what follows belongs to the program.
Surveys do not substitute. One measured baseline carries more weight in a finance review than a company-wide perception survey, and it costs an afternoon. The fuller instrument set sits in our measurement playbook; one timestamped task clears the bar for a pilot.
Rule 4. Decide how many observations you need, not how many people
Fifteen people measured once give you fifteen numbers, and week-to-week variance on real work will usually swamp any gain under about a fifth. The same fifteen measured weekly for six weeks give you ninety, enough to tell a 10% change from an ordinary slow week.
This is the cheapest upgrade available to an AI pilot evaluation, and it is missing from every scorecard in this field. Repeated measures on one task beat a broad survey of six. Choose the task that repeats most often, not the one with the biggest theoretical upside.
Three rules for the weeks the pilot runs
Rule 5. Train a second group a month later and get your comparison for free
Train group A in month one and group B in month two, measuring both in both months. Group B is your untrained comparison at no cost, and month two tells you whether group A’s gain survived four weeks with no facilitator present.
Staggering protects you politically too. A single cohort invites the question nobody can answer: was the quarter simply busier. Two cohorts answer it with your own data instead of a vendor’s benchmark.
Rule 6. Count the usage you cannot see
Admin consoles report the sanctioned tool and nothing else. Gartner’s Global Labor Market Survey for the first quarter of 2026, across 12,004 employees and managers in 40 countries, found 88% of employees with enterprise AI access also using personal AI tools for business tasks.
So a pilot can read dead in the dashboard and alive in practice. Ask the group in writing at weeks three and six how many days in the last five they used an assistant for work, and on which account, stating that the personal-account answer carries no consequence. Without that line you get the answer people think is safe. Measuring AI adoption from the console alone gives you a floor, not a reading.
Rule 7. Log every unpaid hour the pilot consumes
Keep a running list: the champion’s evenings, IT hours on provisioning, manager time, the prompt library somebody maintained on a Saturday, the two people answering questions in a side channel. Record them as they happen, because nobody reconstructs them later.
These costs do not scale in a straight line, and they are why phase two lands over budget. At the Bureau of Labor Statistics’ $46.89 per hour worked in June 2026 for private industry, thirty unlogged hours is real money sitting outside your ai training roi figure and outside the quote you will compare it against.
Three rules for reading the result
Rule 8. Discount the novelty window and decide on the later number
Week one measures interest. Week six measures behavior. A pilot that reports a day-seven spike as proof is reading a curve at its highest point, and the drop between weeks two and five is the most informative part of the exercise.
Set the decision reading at week six at the earliest and keep week one as a contrast, not a result. Two readings far apart tell you the program needs reinforcement built in, cheaper to learn now than after 180 people have been through it.
Rule 9. Keep capability, output and money as three separate answers
Did people get better at the work, did a deliverable change, did the saved hours land anywhere. Under deadline pressure these collapse into one slide, and that collapse is where credibility goes.
Perception makes the first question treacherous. In METR’s randomized trial, 16 experienced open-source developers working on 246 real tasks came out 19% slower with AI access while estimating afterward that they had been 20% faster. METR now treats that magnitude as historical, tied to early-2025 tools; the perception gap is the part that travels, because self-reports can invert the sign of a real effect. Where hours are claimed, decide at manager level which backed-up work receives them, then measure that work. Our guide to building a model finance will not dismiss sets that conversion rate and defends it line by line.
Rule 10. Name what will not survive multiplication, then fund it or discount it
List every input the pilot ran on and divide each by the rollout factor. Three recur in every program we have delivered:
- A champion’s attention. One person giving 15 people two hours a week cannot give 180 the same, and that attention is often the active ingredient.
- Small-group practice. Hands-on work on the team’s own documents degrades above roughly 20 people per session. A rollout that quietly becomes a webinar is not what you tested.
- Self-selection. Volunteers arrive having already decided it works. The next 165 have not.
Fund the equivalent at scale, or cut the forecast. The failure mode here is a rollout priced on the pilot’s number and staffed on none of the pilot’s conditions.
Four readings instead of a 24-point scorecard
Fifteen people support four honest readings, the AI enablement metrics worth carrying into an approval meeting. More is precision you did not buy.
| Reading | When | Scale | Iterate once | Stop |
|---|---|---|---|---|
| Method change on the named task | Week 6 | Most of the group works differently and can show you how | A few can, the rest reverted | Nobody changed method |
| Time on the task, repeated measures | Weeks 1 and 6 | Gain holds past the first two weeks | Gain visible but inside the noise | No gain, or slower |
| Weekly use for real work | Weeks 3 and 6 | 40% to 60%, personal accounts counted | 25% to 40% | Under 25% |
| Where the saved hours went | Week 6 | A named backlog, deadline or deferred hire | A manager intends to decide | Nowhere, nobody owns it |
Two cautions. Strong use with weak method change is enthusiasm, not capability, and scaling it buys more enthusiasm. Strong method with no reallocation is a management problem in a program’s clothes: a different purchase, sometimes none. The diagnostic for telling those apart sits in our guide to diagnosing why training did not work, and the bought-in version in what a capability audit covers and what it costs.
What the 95% number does and does not say
Somebody in your approval chain will quote the claim that 95% of enterprise AI pilots fail. It comes from MIT’s Project NANDA report of July 2025. Knowing what it measures beats arguing with it: a base of roughly 300 public deployments, 52 executive interviews and 153 survey responses, and a finding that those programs showed no measurable profit impact. That is not the same as nothing working. Gartner’s 2024 prediction that 30% or more of generative AI projects would be abandoned after proof of concept named the causes: weak data, thin risk controls, rising cost, unclear business value.
Treat both as priors about scaling discipline, not forecasts for your cohort. McKinsey’s 2026 numbers give the useful reading: personal gains are easy to produce, firm-level gains are rare, so a pilot that measures only the first has said nothing about the second. That gap is the argument for buying capability rather than seat count, which we make in the case against equating licenses with capability.
For your peer set, the Census Bureau’s Business Trends and Outlook Survey put 32% of US firms with 100 to 249 employees using AI in a business function as of May 2026. Your comparison is no longer companies doing nothing.
When the honest answer is stop
A gate that cannot return stop is a budget request with extra steps.
Stop when nobody changed method by week six, both use readings sit under 25%, and a follow-up session moved neither. Stop also, the harder call, when capability clearly landed and no manager will name where the hours go. That case needs a decision from the person who owns the deliverable, not more enablement, and buying a rollout is the most expensive way to avoid making it.
Write the result up either way, unflattering version and all. In our experience, programs that report an early failure get a budget correction quickly; the ones that report only wins lose the room at renewal.
Frequently asked questions
How long should an AI pilot run before you evaluate it?
Six weeks, with readings at weeks one, three and six. The 30-day convention in most published scorecards lands inside the novelty window, where use is still high and reverted method has not surfaced. Two extra weeks removes the main source of false positives.
How many people should be in a pilot cohort?
Twelve to twenty, mixed by tenure. Past twenty, hands-on work on the group’s own documents turns into a lecture and you stop testing what you intend to buy. Below twelve, one person’s odd month moves the whole result.
What adoption rate should a pilot cohort reach?
Between 40% and 60% using an assistant weekly for real work by week six, counting personal accounts. Under 25% usually means the content ran on sample data rather than the team’s own work. Above 60% in week one means nothing yet.
Our pilot group loved the sessions. Is that enough to scale?
Satisfaction predicts attendance at the next session and little else. METR’s trial is the sharpest illustration available: developers who were measurably slower with AI estimated afterward that they had been 20% faster. Ask the group what they now do differently and have them show you, which enthusiasm cannot answer on its own.
How do I work out the return on a pilot?
Count only hours that landed on backed-up work, value them at loaded cost rather than wage, and subtract the unpaid hours the pilot consumed. BLS puts private-industry compensation at $46.89 per hour worked in June 2026, 30.0% of it benefits, so wage-only models understate both sides by a third.
Why did our result shrink when we rolled it out?
Three causes, usually together: the cohort was less experienced or more willing than average, the champion’s per-person attention fell, and small-group practice became a large-group briefing. The NBER support-agent study found gains near 34% among novices and close to zero among the most experienced, enough on its own to halve a rollout number.
What should a pilot cost?
A first cohort trained hands-on sits at the low end of the $2,000 to $15,000 range the market charges, and the diagnosis before it should cost nothing. If the pilot points at process rather than skill, you are looking at a custom automation build in the $25,000 to $150,000 range, worth separating before anyone quotes you.
Who should own the scale decision?
The manager who owns the deliverable being measured, with L&D or operations running the instrument. When the decision sits with whoever bought the program, the reading drifts toward the purchase, which is the mechanism behind most pilots that scaled and should not have.
What do I report when the pilot is inconclusive?
The reading, the reason and a date. Name which of the four readings came back clean, which did not, what would resolve it, and when you report again. Finance teams accept an honest in-progress result with a decision date far more readily than a hopeful number they cannot audit.
Key takeaways
- A pilot’s output is a decision, not a testimonial, and the three thresholds get written before the people are chosen.
- Cohort choice sets the headline number: near 34% for novices and close to zero for the most experienced, in the largest field study available.
- Ninety observations from fifteen people beat fifteen, so pick the weekly task and measure it weekly.
- Two cohorts a month apart give you a comparison group, a decay reading and an answer to the busy-quarter objection, at no extra cost.
- Decide on the week-six reading and keep week one as a contrast; the drop between them is the most useful line in the report.
- A credible ai pilot evaluation lists what will not multiply, namely a champion’s attention, small-group practice and self-selection, then funds it or discounts the forecast.
To pressure-test a pilot you have run, or to design the readings before the next cohort starts, book a free AI readiness call. If your pilot worked and the constraint sits with how the work is organized, we will say so rather than sell you a rollout.
Arthur Wandzel
