The single highest-leverage finance-discipline activity in an AI project happens at kickoff and takes 4 to 8 hours: naming the counterfactual. The counterfactual is the comparison alternative; what the world would have looked like without the AI project; and it is the foundation of most defensible ROI claim. Without an explicit counterfactual the benefit cannot be attributed to the project, which is the threshold internal audit applies. Four candidates dominate: do-nothing, off-the-shelf alternative, in-house build, and prior process. Each is right in some contexts and wrong in others. This piece is the framework for choosing among them, with the math for each and the test of when not to pick it. Project teams that select the counterfactual at kickoff with finance sign-off produce ROI claims that survive audit; teams that select after launch find themselves restating, defending, and rebuilding their cost basis under adversarial conditions.
This is a spoke under the AI project economics manifesto. The manifesto argues that AI economics requires evaluation-cost framing rather than feature-cost framing. The counterfactual is the foundation of any cost-versus-benefit argument under that framing; the answer to the question “compared to what?”
Why counterfactual selection matters
ROI claims that fail audit nearly usually fail on the counterfactual line. The sponsor presents a benefit ($1.2M saved, 27 percent productivity gain) without specifying the alternative against which the benefit is measured. The auditor traces the claim, finds no counterfactual, and the claim cannot be attributed to the project.
The counterfactual is also the line where ROI claims are most often inflated unintentionally. A team that picked an aggressive counterfactual (“we would have hired 12 engineers”) at kickoff produces an ROI number 3x what a conservative counterfactual (“we would have bought the off-the-shelf product”) would have produced. Both can be defensible; the choice between them shapes the defensibility of the claim. We name the four most common counterfactuals, when each is right, when each is wrong, and the math.
The why-most-AI-ROI-claims-fail-audit piece covers the seven failure modes that include counterfactual omission as the most common.
Counterfactual 1: do-nothing
The math. Counterfactual cost is zero. Counterfactual benefit is zero. The AI project’s full cost is compared against its full benefit, and the ratio is the ROI.
When it’s right. When the project is genuinely net-new capability that no alternative would have produced and the do-nothing path was a credible decision option at kickoff. Examples: an internal tool that no vendor sells, a research-side capability that no team would have built absent the AI program, a strategic experiment whose outcome was uncertain enough that “not doing it” was a real choice.
When it’s wrong. When the alternative to building was hiring contractors, buying off-the-shelf, or expanding headcount. In those cases the do-nothing baseline understates the true alternative cost and inflates AI ROI. A telltale sign: when the project was being scoped, finance asked “what would we do otherwise?” and the answer was specific and credible. If finance had a specific alternative in mind, do-nothing is not the right counterfactual.
Math caveat. Even when do-nothing is the right counterfactual, the ROI claim must include the opportunity cost of the capital or engineering time invested. The opportunity cost framework covers how to model this.
Counterfactual 2: off-the-shelf alternative
The math. Counterfactual cost is the off-the-shelf product’s annual contract value plus integration cost. ROI is (off-the-shelf cost) minus (AI build cost) over a defensible time horizon, typically 3 years.
When it’s right. When a credible commercial alternative existed at the time of decision and the buyer formally evaluated it. The strongest evidence: vendor quotes, RFP responses, or pricing pages from the kickoff period. Off-the-shelf is the cleanest counterfactual to defend at FP&A because the cost basis is auditable from vendor pricing.
When it’s wrong. When the off-the-shelf product does not do what the AI build does. Many AI projects produce capability that no vendor offers (yet, or at the level of integration the team needs). Forcing an off-the-shelf counterfactual on a non-substitutable build inflates the alternative cost (because the vendor cost includes capability the team did not need) or understates it (because the vendor cost does not include capability the team did need). The substitutability test must hold.
Math worked example. A team builds an internal AI triage agent for $400K. The counterfactual is a vendor product at $180K ACV plus $80K integration plus $35K annual maintenance; $295K year one and $215K year two. Three-year counterfactual cost: $725K. Three-year build cost: $400K + $200K (year 2 retainer) + $200K (year 3 retainer) = $800K. The off-the-shelf counterfactual makes the AI build look more expensive than the alternative. ROI defense requires capability arguments (the agency build was demonstrably better at the task) or strategic arguments (data residency, customization).
Counterfactual 3: in-house build
The math. Counterfactual cost is fully loaded engineering time at burdened FTE rates, plus AI infrastructure, plus the senior architectural judgment that an external team brought.
When it’s right. When the buyer was actively considering an in-house build at kickoff and had the team capacity to execute. The counterfactual is what the in-house team would have spent.
When it’s wrong. When the in-house team did not have the senior AI engineering capacity to execute, or when in-house staffing would have required hiring that the company was not prepared to do. In those cases the in-house counterfactual is hypothetical at a level audit will reject. A telltale sign: the project sponsor cannot name the in-house engineers who would have led the project.
Math worked example. A $500K agency build’s in-house counterfactual is typically $700K to $1.1M. The math: 1.5 to 2x duration (in-house teams take longer because they do not have a reusable platform, eval suite, or senior architectural pattern), at 1.2x to 1.4x burdened cost per engineer-hour (after fully loaded benefits, equity, and ramp time), plus the same AI infrastructure cost. The TCO comparison piece walks through the worked example.
Counterfactual 4: prior process
The math. Counterfactual cost is the prior process’s annual operating cost; labor, tools, error correction, opportunity cost. ROI is (prior process annual cost minus AI project annual cost) over the relevant horizon.
When it’s right. When the AI replaces or augments an existing workflow whose cost is observable in the previous year’s cost-line ledger. Prior-process counterfactual is the most rigorous because the cost basis is observable rather than hypothetical.
When it’s wrong. When the prior process did not produce the same output. If the AI does the prior process plus new capability, comparing the AI’s full cost against the prior process’s narrower cost overstates ROI. The fix: decompose the AI’s cost into the substitutable fraction (compared against prior process) and the new-capability fraction (compared against do-nothing or off-the-shelf).
Math worked example. A support team’s prior process costs $1.4M/year in labor (12 FTEs at fully loaded $115K). The AI triage agent costs $400K build plus $200K/year operating, and reduces required FTEs from 12 to 4.5; saving $862K/year in labor. Year-one savings: $862K minus $400K minus $200K = $262K. Years 2-3 savings: $862K minus $200K = $662K each. Three-year ROI: $1.586M against $800K invested = 198 percent return.
Picking the right one
A simple selection algorithm.
| Question | If yes | If no |
|---|---|---|
| Does a credible commercial alternative exist that you formally evaluated? | Off-the-shelf | Continue |
| Does the AI replace an existing workflow with observable cost? | Prior process | Continue |
| Was the in-house team capacity available and would they have executed? | In-house build | Continue |
| Is the project genuinely net-new with no realistic alternative? | Do-nothing | Reconsider scope |
The algorithm runs top to bottom. Off-the-shelf and prior process are preferred when available because their cost bases are observable. In-house build is acceptable when the in-house team was a real option. Do-nothing is the lowest-rigor counterfactual and should be reserved for genuinely net-new projects.
The selection happens at kickoff with finance sign-off. The artifact is a one-page document naming the counterfactual, citing the cost basis (vendor quotes, prior-year cost ledger, in-house FTE plan), and documenting finance acceptance. Post-launch, the audit references this document; the counterfactual cannot be re-litigated if finance signed off at kickoff.
The triple-counterfactual presentation
For high-stakes projects, present three counterfactuals in parallel.
| Counterfactual | Cost basis | Year-3 ROI |
|---|---|---|
| Off-the-shelf | Vendor pricing | 45 percent |
| Prior process | Prior-year ledger | 198 percent |
| In-house build | Burdened FTE | 80 percent |
The CFO sees the project’s value across plausible alternatives. The lowest-ROI counterfactual (45 percent against off-the-shelf) is the most defensible to the audit committee; if the project beats the lowest-ROI counterfactual, it survives audit regardless of the alternative chosen. Triple-counterfactual presentation is the strongest possible ROI defense.
The investment thesis template accommodates the triple-counterfactual format. The ROI staircase piece covers how the counterfactual interacts with the four stages of ROI realization.
Frequently asked questions
What is a counterfactual in the AI ROI context?
A counterfactual is the comparison alternative; what the world would have looked like without the AI project. Without an explicit counterfactual, the benefit cannot be attributed to the project. Audit-defensible AI ROI claims name the counterfactual on the same page as the benefit number.
Why is the do-nothing counterfactual sometimes wrong?
Because the alternative to building was rarely doing nothing. If the team would have hired contractors, bought an off-the-shelf tool, or expanded headcount, the do-nothing baseline understates the true alternative cost and overstates AI ROI. Do-nothing is right only when the project is genuinely net-new capability that no alternative would have addressed.
When is the off-the-shelf counterfactual the right one to pick?
When a credible commercial alternative exists at the time of decision and the buyer evaluated it. The counterfactual cost is the off-the-shelf product’s annual contract value plus integration cost. Off-the-shelf is the strongest counterfactual when defending AI build investment to FP&A because it is auditable from vendor pricing pages.
What’s the math when the in-house build is the counterfactual?
In-house counterfactual cost is fully loaded engineering time at burdened FTE rates, plus AI infrastructure, plus the senior architectural judgment that an external team brought. A typical in-house counterfactual for a $500K agency build is $700K to $1.1M because in-house teams take 1.5 to 2x longer at higher loaded cost per person-hour.
When is the prior process the right counterfactual?
When the AI replaces or augments an existing workflow with measurable cost. The counterfactual is the prior process’s annual operating cost; labor, tools, error correction, opportunity cost. Prior-process counterfactual is the most rigorous because the cost basis is observable in the previous year’s cost-line ledger.
Can multiple counterfactuals be presented in the same ROI claim?
Yes, and we recommend it. Present three counterfactuals: do-nothing, off-the-shelf, and prior process. Show ROI against each. The CFO sees the project’s value across the range of plausible alternatives. The lowest-ROI counterfactual is the most defensible to the audit committee.
How do you defend a counterfactual that the auditor disagrees with?
By documenting the counterfactual selection at kickoff with finance involvement. If finance signs off on the counterfactual at kickoff, post-launch audit cannot reasonably re-litigate it. Counterfactual selection is a kickoff-phase decision; making it after launch invites the disagreement that audit then resolves against the sponsor.
What if the counterfactual cost is genuinely uncertain?
Use a range. A defensible counterfactual is “off-the-shelf annual contract value of $200K to $280K based on three vendor quotes.” The range is auditable; the midpoint or lower bound is the defensible benefit basis. Single-point counterfactual estimates that turn out wrong are harder to defend than a range whose lower bound holds.
How does counterfactual selection differ for cost-out versus revenue-in AI projects?
Cost-out projects (replacing process or vendor): counterfactual is the displaced cost. Revenue-in projects (new capability): counterfactual is the do-nothing or partial-substitute revenue trajectory. Revenue-in counterfactuals are harder to make rigorous because the alternative-world revenue is hypothetical; cost-out counterfactuals are easier because the displaced line is observable.
When is the counterfactual not worth picking carefully?
Almost rarely. The counterfactual is the foundation of the entire ROI claim. Spending 4 to 8 hours at kickoff naming the counterfactual, getting finance sign-off, and documenting the cost basis is the highest-leverage finance-discipline activity in an AI project. Skipping it is the most common reason ROI claims fail audit later.
Key takeaways
- Counterfactual selection at kickoff with finance sign-off is the highest-leverage finance-discipline activity in an AI project.
- Four counterfactuals dominate: do-nothing, off-the-shelf, in-house build, prior process. Each is right in specific contexts.
- Off-the-shelf and prior process are preferred when available because their cost bases are observable in vendor pricing or prior-year ledger; in-house and do-nothing are weaker because their cost bases are hypothetical.
- Triple-counterfactual presentation (three counterfactuals in parallel) is the strongest possible ROI defense for high-stakes projects.
- The counterfactual is documented in a one-page artifact at kickoff; post-launch audit references this document and cannot reasonably re-litigate the counterfactual choice.
The counterfactual is not a math exercise. It is a decision about which alternative the project’s value should be measured against, made at kickoff, signed off by finance, and held through audit. Project teams that treat counterfactual selection as the first move of finance discipline produce ROI claims that hold; teams that postpone it find themselves choosing under adversarial conditions when audit forces the question.
Arthur Wandzel
