Two numbers from McKinsey’s November 2025 State of AI survey explain why measuring AI training ROI has turned into a board-level headache. Eighty-eight percent of organizations report regular AI use in at least one business function, up from 78% a year earlier. Only about 39% can point to any enterprise-level EBIT impact, and most of those put the figure below 5%.
The space between those two numbers is not a licensing gap. It is a capability gap that nobody measured, which in practice is a capability gap nobody fixed. This playbook is the measurement system for AI enablement work: what to baseline before the first session, how to score whether capability moved, how to attribute the change when you have no control group, and how to tell at month six whether the gain survived.
Why measuring AI training ROI goes wrong
Most programs are measured with the two cheapest numbers to hand: how many people finished the training, and how many people logged into the tool afterward. Both are easy to pull. Neither answers the question being asked.
Zapier’s own guide to AI adoption metrics puts training completion rates on its list of the four numbers to track. That is a fair description of what companies do measure, and a poor description of what they should. A completion rate tells you attendance. A login count tells you curiosity. Neither tells you that the contracts analyst can now turn a 40-page vendor agreement into a risk summary that her boss signs off without rework.
This is a solved problem in evaluation practice and an unsolved problem in AI programs. Donald Kirkpatrick’s four levels, carried forward by Jim and Wendy Kirkpatrick in their 2016 ATD Press volume, separate reaction (did they like it), learning (did they acquire the skill), behavior (are they doing it at work), and results (did the business outcome move). Jack and Patti Phillips added a fifth level, ROI, with the steps most people skip: isolate the program’s effect from all the rest of that quarter, convert the impact to money, and count fully loaded costs including the hours participants spent away from their desks.
Almost every AI enablement program stops at Level 1 and Level 2, then reports a Level 4 claim. Reaction scores were high, attendance was high, and the deck says the program drove efficiency. Any finance partner who has read one of these decks knows the move and discounts the whole thing.
There is a second failure that is harder to see. Teams measure in the wrong order. They start at the money layer because that is what the CFO asked about, work backwards, and find that none of the lower-level data exists to support the number. The sequence runs the other way, and each layer is only worth measuring if the layer beneath it is already true.
The four layers of AI enablement measurement
Four layers, in dependency order. Usage, then capability, then work, then money. A claim at any layer is worthless if the layer below it was never established.
| Layer | Question it answers | Primary measure | What it cannot prove |
|---|---|---|---|
| Usage | Are people touching the tools at all? | Weekly active share of the trained group; sessions per person per week | That anyone got better |
| Capability | Can they now produce work they could not produce before? | Scored output on a fixed task set, judged against a rubric | That the work is reaching the business |
| Work | Has the workflow changed, and is the capacity redirected? | Cycle time, rework rate, throughput on one named process | A dollar figure |
| Money | Is there a return a finance team will sign? | Benefit-cost ratio, payback period, ROI percentage | Anything, if the layers below are empty |
The dependency rule does the real work here. If weekly active use among the trained group is 30%, there is no point scoring capability, because two-thirds of your sample did not practice. If capability moved but cycle time did not, the problem sits in the process, not the people, and no amount of further training will shift the number. If the first three layers look good and the money layer is flat, you have either measured the wrong business metric or you are looking too early.
Working down the stack also tells you when to stop. For a $6,000 workshop, a defensible Layer 3 result is enough and the full Phillips exercise costs more than it informs. For a program running across nine roles and two quarters, the money layer earns its keep. The decision of how deep to instrument belongs with the person authorizing the spend, which is covered from the budget side in the AI enablement playbook.
Layer 1: usage, where measuring AI adoption stops being useful
Usage is a floor, not a finding. Its only job is to tell you whether the rest of the measurement is even interpretable.
Three numbers cover it. Weekly active share of the trained cohort, measured in the admin console of whichever assistant the company standardized on. Sessions per active person per week. And thirty-day repeat rate, which catches the pattern where everyone logs in the week after the session and nobody returns.
Set the bar before you look. Among people who attended hands-on training, weekly active use below roughly 60% by day 30 means the program did not land, and the fix is almost never more training. Gallup’s workforce tracking found frequent AI use at work, defined as a few times a week or more, sitting at 26% of US employees in Q4 2025 and near 30% by May 2026. A trained cohort that cannot clear double the national rate did not get much from the session.
The same Gallup series points at the usual cause. Only about a quarter of US employees say their employer has communicated a clear plan for integrating AI into how work gets done. People who have no idea whether using the tool is encouraged, tolerated, or quietly logged will choose not to risk it. That is a policy and communication failure, not a skills failure, and it is the subject of the safe AI use playbook.
What usage data cannot do, at any volume, is tell you that capability moved. A person can open an assistant eleven times a day and produce worse work than they did by hand, because they accept the first draft and never check the numbers in it. Telemetry counts keystrokes. It does not grade output.
Layer 2: capability, the measurement almost nobody takes
Here is the measurement that separates a serious program from a line item: a scored task, run on the team’s own documents, before the training and again after.
Not a quiz. Multiple-choice questions about prompting theory correlate with nothing. The instrument is a real deliverable the person already owes someone, produced under observation, timed, and scored against a rubric by the person who normally reviews that deliverable.
Four rubric dimensions carry almost all the signal:
- Completeness. Does the output hold all the finished version needs, or does it stop at the parts the assistant volunteered?
- Accuracy. Every figure, date, name, and citation checked against the source document. This dimension catches the single most common failure in untrained AI use, which is a fluent draft with three wrong numbers in it.
- Iteration efficiency. How many turns did it take to reach a usable draft? A trained user gets there in one or two. An untrained user either accepts turn one or thrashes through nine.
- Judgment. Did the person notice what the tool should not have been asked to do, and route it elsewhere?
Score each dimension 0 to 3 and you have a 12-point capability score per task, per person, comparable across the before and after measurement. That is a number with a method behind it, which is more than a completion rate has ever had.
Two design notes matter more than they look. First, the scorer should be the person who owns the quality of that output in normal operation, not a trainer and not the participant. Self-assessed confidence rises after every workshop ever run, the bad ones too. Second, the task set has to be role-specific, because a sales lead and a controller fail at other points on the same assistant. Role segmentation of curriculum, and so of the task set, is the subject of the role-based enablement playbook.
We build the capability check into the sessions we run for a second reason as well: it is usually the first time the team sees what good looks like on their own documents rather than on a demo prompt.
Layer 3: work, where capability lands or evaporates
A trained person inside an unchanged process produces the same output on the same schedule, slightly less miserably. Capability that does not change the shape of the work shows up nowhere a CFO can see it.
McKinsey’s 2025 survey makes this concrete. The small group of firms reporting real AI-driven profit impact, around 6% of respondents, are far more likely to be redesigning workflows rather than bolting AI onto existing steps: roughly 55% of them report fundamental workflow redesign against about 20% of everyone else. Training is an input to that redesign, not a substitute for it.
So pick one process, not nine, and instrument it. The candidates worth measuring share three traits: the process runs weekly or more, it has a definable start and end, and somebody already complains about how long it takes. Monthly close prep. Proposal turnaround. Candidate screening. First-response time on inbound support. Board-pack assembly.
Then take three measures on it:
- Cycle time. Start to finish, in hours or days, measured the same way before and after. Not the time spent at a keyboard, the elapsed time, because elapsed time is what the business feels.
- Rework rate. Share of outputs sent back for revision. This number is the honest counterweight to speed, and it is the one that exposes a program that taught people to produce fast garbage.
- Throughput. Units per week per person on the named process. Proposals out, files reviewed, tickets closed.
Hold the definition fixed. The most common way a Layer 3 measurement dies is that somebody improves the definition halfway through, and the before and after numbers no longer line up. Write the definition down in the baseline week and refuse to change it until the program is over.
One more caution on benchmarks. The research on AI-assisted productivity is real and specific: Brynjolfsson, Li and Raymond’s study of 5,179 customer support agents, published in the Quarterly Journal of Economics in 2025, found issues resolved per hour rose 14% on average, 34% among novices, and hardly at all among the most experienced agents. That distribution is the useful finding, not the headline percentage. Gains concentrate where the skill floor was lowest, which means your own result depends on who you trained, and a vendor’s benchmark is not your number.
Layer 4: money, and what a CFO will accept
Three figures, and they are all derived from the same two inputs:
- Benefit-cost ratio: total benefits divided by fully loaded program cost.
- ROI percentage: net benefits divided by cost, times 100. A BCR of 3.0 is an ROI of 200%.
- Payback period: cost divided by monthly benefit.
The inputs are where the argument happens. On the cost side, fully loaded means the facilitation fee plus participant time at a blended loaded rate plus the license spend attributable to the trained group plus the hours spent on measurement itself. Programs that count only the invoice understate cost by a factor of two or more, and a finance team spots the omission on sight.
On the benefit side, there is one conversion rule that keeps the number honest: hours freed are not money until the capacity is redeployed to work the company wanted. Forty hours a week of reclaimed drafting time is a real gain if those hours go into more proposals or fewer contractor invoices. If the work simply got more pleasant, the benefit is retention and morale, which are worth claiming as intangibles and not worth converting to dollars.
Phillips’ own method says to record intangible benefits separately rather than forcing a conversion. Take that advice. A table with three dollar figures and four named intangibles reads as competent. A table that monetizes employee satisfaction reads as a sales deck.
The one-week baseline protocol
The baseline is the only step in this entire system that cannot be recovered later. Once the training has happened, the pre-period is gone. Every other mistake here is fixable; this one is not.
It takes about a week of part-time effort. Run it in the week before the first session.
- Pick five to eight recurring tasks, weighted toward the roles being trained. Each task must produce an artifact somebody reviews, and must recur at least weekly. Write a one-line definition of “done” for each.
- Collect three samples per task, from three separate people where you can. Three is not statistical power; it is enough to catch the case where one person’s number is an outlier and you would otherwise have built the whole program’s story on it.
- Time them honestly. Elapsed time from start to the version the reviewer accepted, waiting and revision included. Self-timed is acceptable if the instruction is written down and identical for everyone.
- Score them cold on the four rubric dimensions, by the normal reviewer, before anyone knows what the training will contain.
- Keep the artifacts. Store the actual documents, not just the scores, in a folder with the date. At month six you will want to re-read the pre-training output, and a spreadsheet of numbers will not be enough to settle an argument.
- Record the Layer 3 baseline for your one named process: cycle time, rework rate, throughput, with the definitions written out.
- Pull a usage snapshot from the assistant’s admin console for the week before training, so the day-30 comparison has a floor to stand on.
- Name the owner. One person, by name, who owns the re-measurement dates. Measurement that belongs to a committee does not happen.
Total cost: roughly 10 to 20 hours of staff time for a 40-person program, most of it spent timing work that was happening regardless. Skipping it saves those hours and costs you the ability to make any claim at all.
The design of the session itself should be built against this baseline, so that the drills are the tasks you measured. That coupling is the argument of the workshop design playbook, and it is why the baseline comes first in the sequence.
Attribution without a control group
“Isolate the effects of the program” appears in every summary of the Phillips method and almost never gets explained for a company of 200 people running a single cohort. Here are the real options, ranked by what they cost and what they can honestly support.
| Method | What it costs | What it supports |
|---|---|---|
| Staggered cohorts | Two to four weeks of scheduling patience | “Group A moved after training in March, Group B moved after training in April, and neither moved before.” The strongest claim available in a normal company. |
| Held-back comparison group | One team’s goodwill and a delayed start | A clean before and after against a matched untrained group in the same period. |
| Trend break on an existing series | Nothing, if the metric was already tracked | “The line bends at the training date, and nothing else changed that week.” Weak alone, strong in combination. |
| Participant estimation with a discount | One carefully worded survey question | A directional figure, labeled as an estimate, discounted for optimism. |
| Pre and post only | Nothing | “Capability moved.” Not “training caused the revenue.” |
Staggered cohorts are the practical winner for most companies, because the scheduling constraint is one you already have. You were going to train sales in March and finance in April anyway. Measuring both groups in both months turns an operational necessity into an attribution argument at no extra cost.
The discount on participant estimation deserves a number. When people are asked how much of a gain they attribute to the training, apply the confidence they themselves give you: if the average respondent says 60% attributable with 70% confidence, carry 42% of the claimed benefit. State the arithmetic in the footnote. A discounted number that shows its work beats an undiscounted one every time it reaches a finance review.
And when none of this is available, say so. “We can show capability moved and cycle time fell; we cannot isolate the training from the new template rollout in the same quarter” is a sentence that builds credibility. The alternative, a confident causal claim that collapses under one question, costs more than the admission.
A worked ROI example, and the discount it deserves
Illustrative numbers, with a company that does not exist. The shape is what matters.
A 180-person services firm trains 40 people across operations and finance. Fully loaded cost:
- Facilitation and materials at the market rate for a hands-on team workshop: $9,000
- Participant time: 40 people x 4 hours x $60 blended loaded rate: $9,600
- Measurement: baseline plus two re-measurements, about 30 hours of internal time: $1,800
- Total: $20,400
Benefit, from the audited task set. Each trained person runs the measured tasks about four times a week. Average elapsed time per run fell 12 minutes, verified against the baseline artifacts, with rework rate flat. That is 48 minutes per person per week, or roughly 32 hours a week across the cohort, which at the same $60 rate is $1,920 of capacity per week. Over 40 working weeks: $76,800.
Raw BCR: 3.8. Raw ROI: 276%. Now discount it properly.
- Redeployment: only the share of freed hours that went into named work counts. Say two-thirds did, and one-third went into slack that the company was happy to give back. Benefit falls to about $51,000.
- Attribution: staggered cohorts support most of it, not all. Carry 75%. About $38,000.
- Decay: the gain does not hold flat for a year (next section). Apply the retention curve rather than the day-90 rate. About $30,000.
Adjusted BCR: 1.5. Adjusted ROI: 47%. Payback: under nine months.
That is a worse number and a far better answer. A 47% return with a visible discount ladder survives a finance review. A 276% return with no discounts gets read as a vendor’s arithmetic, and the next program request gets scrutiny the program may not deserve.
Retention: capability decays on a known curve
The whole field measures a step change at day 90 and stops. Then the number gets repeated at the end of the year as though it were still true.
Training transfer decays, and the decay has been measured for twenty years. Saks and Belcourt’s study of 150 organizations, published in Human Resource Management in 2006, found that about 62% of training content was being applied right after training, around 44% at six months, and roughly 34% at one year. There is no reason to assume AI fluency behaves better than other applied skills; several reasons to assume worse, since the tools themselves change under the user.
So the measurement calendar has four dates, not two:
| Date | What to re-measure | What you are watching for |
|---|---|---|
| Day 0 | Full baseline | The only irreversible step |
| Day 30 | Usage only | Did it land at all |
| Day 90 | Capability score plus Layer 3 process metrics | The headline result |
| Day 180 | Same capability task set, abbreviated | The decay slope |
| Day 365 | Layer 3 process metrics, usage | Whether it became how work is done |
Day 180 is the date nobody books and the one that changes decisions. A capability score that held within a point or two of the day-90 result means the behavior is embedded. A score that fell by a third means you bought a quarter of productivity and labeled it a transformation.
Two interventions hold the curve up, and neither is a refresher course. The first is workflow redesign, the thing McKinsey’s high performers do: if the new process cannot be executed the old way, the capability stays because it is load-bearing. The second is a named owner with one recurring artifact, usually a shared prompt library plus a 20-minute weekly clinic where people bring one stuck task. Cheap, unglamorous, and the difference between 34% and most of it.
Whether that owner is an internal hire or a retained partner is a budget question with a real answer either way, worked through in the buy-versus-build playbook.
When the numbers do not move
Sooner or later a program posts a flat result. The useful response is a diagnosis, because the four causes need four fixes and three of them are not about the training.
Measurement failure. There was no baseline, or the task set missed what people spend their week on, or the definition of cycle time changed mid-flight. Symptom: the capability scores moved but nothing at Layer 3 did, and nobody can agree on what the before number was. Fix: re-baseline and re-measure on a tighter task set. Cost: a quarter.
Training failure. The session taught features rather than the team’s own work. Symptom: high reaction scores, low day-30 usage, no change in iteration efficiency. Fix: redesign the session around the measured tasks. This is the one case where more training is the answer, and only in a different form.
Work failure. Capability moved, the process did not. Symptom: good capability scores, good usage, flat cycle time, and someone can name the step that still requires the old method. Fix: change the process. No training spend will reach this.
Horizon failure. You measured a lagging business metric too early, or picked one too far from the work. Symptom: Layers 1 through 3 all moved and the Layer 4 metric is noise. Fix: wait, and swap the business metric for one within two steps of the process you instrumented.
Four causes, four different conversations. A program that can say which one it hit has learned more than the original ROI figure was worth, and it is the reason the day-90 review should include the people who do the work rather than only the people who signed for it.
What measurement costs, and the claims that fail an audit
Measurement is not free, and a program that spends more proving itself than improving anything has lost the plot. Budget 5% to 10% of program cost for the whole measurement effort. For a 40-person program that is roughly 10 to 30 hours of internal time: the baseline week, two re-measurement sittings, and an afternoon of arithmetic.
Most of those hours belong to reviewers who were going to read the work anyway. The genuinely new cost is the scoring, which runs about 10 minutes per artifact once the rubric exists. Nobody needs a platform for this. A spreadsheet and a dated folder of documents outperform every analytics purchase at this scale, because the bottleneck is attention rather than tooling.
Four claims come up in almost every program review and none of them holds up when someone pushes:
“The training drove the revenue increase.” Revenue moves for a dozen reasons per quarter. Unless you ran staggered cohorts against a revenue-adjacent metric and nothing else changed, this claim invites a question you cannot answer. Claim the process metric and let the reader connect it.
“We freed 1,400 hours, which is 0.7 of an FTE.” Freed hours convert to headcount only if someone removes the headcount or declines a hire they had budgeted. Short of that, the honest form is capacity redirected to named work, with the work named.
“Comparable firms see a 27% productivity gain.” Benchmark figures in this market circulate without traceable sources, and the ones that are real describe another population. The support-agent study found gains of 14% on average but 34% for novices and close to nothing for the most experienced staff, which means the number depends entirely on who you trained. Your cohort is your benchmark.
“Post-session confidence scores rose 40%.” Confidence is a reaction measure wearing a capability costume. It rises after weak sessions too. Report it as color, never as evidence.
What does survive an audit: a dated baseline, a rubric with a named scorer, a fixed process definition, a discount ladder with the arithmetic shown, and a flat admission of what the design cannot isolate. That package holds up for one reason: it is the only version of the story that volunteers its own weaknesses before a reviewer has to find them.
The AI enablement metrics that fit on one page
If the whole system has to compress into something a COO can hold in their head, this is it:
- Weekly active share of the trained cohort. One number, monthly. Below 60% at day 30 means stop and fix buy-in.
- Capability score on a fixed task set. Out of 12, scored by the real reviewer, taken at day 0, 90, and 180.
- Cycle time and rework rate on one named process. Same definition every time, no exceptions.
- One business metric within two steps of that process. Chosen before the program starts, not after the results arrive.
- Fully loaded cost, including participant hours. Known from week one.
Five lines. Everything else on the typical AI dashboard is instrumentation looking for a question. The reason this list is short is that a measurement program competing for attention with the work it measures loses, and a measurement program that nobody runs produces exactly the same evidence as no measurement at all.
The deeper argument, that buying licenses and buying capability are two purchases with two proof burdens, runs through the case for treating enablement as construction rather than procurement.
Frequently asked questions
My CFO wants one number. What do I give them?
Give them the benefit-cost ratio with the discount ladder visible underneath it. One line for raw benefit, one each for redeployment, attribution, and decay, then the adjusted figure. A BCR of 1.5 that shows its arithmetic gets approved more often than a BCR of 3.8 that does not, because the first one answers the follow-up questions in advance and the second one invites them.
Do we need a control group to prove AI training worked?
You do not, and you almost certainly will not get one. Staggered cohorts do most of the same work: train sales in March and finance in April, measure both groups in both months, and the untrained group becomes the comparison at no extra cost. Where even that is impossible, a pre-and-post result plus a trend break on an already-tracked metric supports a capability claim, as long as you stop short of claiming the training caused a revenue change.
Is usage data from our admin console enough?
It is enough to tell you whether to keep measuring. Weekly active users, sessions per person, and thirty-day repeat rate establish that people are practicing. They say nothing about quality, which means they cannot distinguish a team producing better work faster from a team producing worse work faster. Pair every usage number with a scored output or do not report it.
How long before we see an ROI figure we can defend?
Ninety days for a capability and process result, six months for anything financial, twelve months before you know whether it stuck. Any vendor promising a defensible financial return at day 30 is either measuring reaction scores or measuring nothing. The day-180 re-measurement is the one that determines whether the day-90 number was real, and it is the date most programs never book.
What should an AI enablement program cost, and how much of that is measurement?
Hands-on team workshops run roughly $2,000 to $15,000 in the current market depending on depth, cohort size, and how much of the content is built on your own documents. Custom automation work that follows sits in another band, roughly $25,000 to $150,000. Budget 5% to 10% of program cost for measurement, which in practice means 10 to 30 hours of internal time rather than a line-item purchase.
Can we just survey people on how much time they save?
You can, and you should discount what they tell you. Self-reported time savings run optimistic in every training context measured, and confidence rises after bad sessions as reliably as after good ones. If you use estimation, ask for the attribution percentage and the respondent’s confidence in it separately, then multiply the two and carry that fraction of the claimed benefit.
Our capability scores went up but nothing downstream changed. What happened?
The process did not change. Someone in the chain still requires the old artifact, the old handoff, or the old approval, so the faster draft waits in the same queue it always waited in. Find the step that still demands the previous method and fix that; further training will not touch this problem, and the firms getting profit impact from AI are the ones redesigning the workflow rather than speeding up its parts.
How do we measure safe use, not just speed?
Add two measures to the Layer 3 set: verification rate, the share of outputs where the reviewer can see that figures were checked against a source, and incident count, meaning confidential material pasted where it should not have been. Both are countable, both move with training, and both are what a board asks about after the first bad week.
Does any of this work for a 60-person company with no L&D function?
It is designed for that case. The baseline protocol takes one week of part-time effort, the capability rubric needs no software, and the whole system runs on a spreadsheet plus a named owner. The constraint at that size is not analytics capacity, it is that measurement competes with revenue work, which is exactly why the one-page metric list is five lines instead of twenty.
Where to start
Do the baseline. Not the dashboard, not the vendor selection, not the deck about measurement philosophy. Pick the five tasks, collect three samples each, score them cold, and store the artifacts with a date on the folder.
That week of work is what turns every later claim from an assertion into evidence, and it is the one step that expires. Everything else in this playbook can be added in month three.
If you want a second opinion on which tasks to baseline and which process to instrument before you commit a budget, book a free AI-readiness call. We will walk your actual workflows and tell you what is measurable, the cases where the honest answer is that a workshop is not what you need yet.
Arthur Wandzel
