Most AI MVP budgets in 2026 put zero dollars against eval engineering — not because founders or vendors disbelieve in evals, but because the SOW template they inherited from 2018 SaaS work had no field for it. The line is missing for the same reason a 1950s blueprint has no ethernet jack: the form predates the requirement. Three founder behaviors keep the omission in place — rolling eval into QA, splitting eval across PM and engineering, deferring eval to v2 — and the cost is the same in each: a silent regression on the first model swap, surfaced as customer complaints in week three of production. Across the proposals I have reviewed this year as CTO of SFAI Labs, more than seven in ten arrived with a line for QA, design, and PM — and no separate line for evaluation engineering.
This article extends the AI MVP economics playbook and the idea-to-product manifesto. It pairs with the eval budget rule and the five-line MVP cost decomposition. The frame here is procurement-side: why the field is empty on the form, and what to do about it before counter-signing.
The 2018 SOW template is why the field is empty
The SaaS SOW template the AI agency market inherited was forged between 2014 and 2018, when “AI” in a procurement document meant a recommender or a sentiment classifier — deterministic, versioned monthly, tested with ordinary unit tests. The template has five header lines: discovery, design, engineering, QA, PM. A few add infrastructure as a sixth. None have a separate line for evaluation engineering, because in 2018 there was nothing to evaluate beyond what unit tests covered. The field was not omitted; it was never drawn.
Procurement reviews proposals against a comparison spreadsheet whose columns are those five header lines plus a total. A 2026 proposal with six lines reads as a vendor that does not understand the template. Agencies that add a sixth line lose deals to agencies that fit the form. The line stays missing not because anyone in the room disbelieves in evals — most senior engineers on both sides know it should exist — but because the procurement comparison is structurally incompatible with a sixth row.
MIT’s NANDA study and McKinsey’s State of AI both put the 2026 pilot-to-production failure rate above 70% — MIT’s GenAI scale-failure number near 95%, McKinsey’s ROI failure number at 73% (McKinsey State of AI). The gap between adoption and value is, mechanically, an eval-discipline gap. The founder counter-signing a five-line SOW is betting against the data — because the form did not give them a sixth row to push back on.
What eval engineering actually buys
Eval engineering produces the artifact your AI feature is contractually held to. It buys five things — all senior AI engineer work, not QA work:
| Sub-line | What it produces | What it is not |
|---|---|---|
| Test-set authoring | 80–200 labeled cases sampled from the production workload distribution | Not a backlog of bug repro steps |
| Rubric design | Written scoring criteria, often anchored binary or 3-point; LLM-as-judge prompt or human rubric | Not a 1–5 Likert form |
| Harness setup | The eval runner in CI; one-shot, scheduled, and on-PR execution | Not the unit test runner |
| Regression suite | Pass-fail gates that block merges and deploys on quality drop | Not a smoke test |
| Observability hookup | Production traces joined to eval baselines; per-call grading | Not log aggregation |
The five sub-lines compound. The test set is the corpus the rubric scores. The rubric is the function the harness runs. The harness emits the signal the regression suite gates on. Skip one and the chain breaks. They are not a buffet.
Stack Overflow’s 2025 Developer Survey puts senior AI engineers at $180–$220 fully-loaded. CIO.com puts a basic eval-platform integration at 80–200 hours, or $12K–$50K at ML engineering rates (CIO.com) — and that is plumbing before any test-set or rubric work. A founder reading a proposal that prices QA at $80/hr and assumes QA does evals is paying $80/hr labor against $200/hr work.
Where the line should land in the SOW
The eval engineering line lands as a peer of engineering, not as a child of QA. In the SOWs I draft and review at SFAI Labs, the field structure is:
- Discovery (workload analysis, eval-set scoping)
- Design (UX, surface area definition)
- Engineering (feature build)
- Evaluation engineering (test-set, rubric, harness, regression, observability hookup)
- QA (deterministic surface — auth, navigation, deterministic backend)
- Project management
- Infrastructure
Evaluation engineering precedes QA because the eval suite is what QA runs against on the non-deterministic surface — QA on its own cannot grade an LLM response. The owner is one named senior AI engineer accountable for the feature’s behavioral contract. The dollar share lands in the 20–30% band, per the eval budget rule. Below 20% means one of the five sub-lines is silently absent.
Counter-sign a SOW whose line 4 is named, dollarized, and owned. Refuse a SOW whose evaluation work is described inside line 3 or line 5 without a separate dollar value. Description without a dollar is not a deliverable; it is a hope. Anthropic’s Building Effective Agents makes the same point: eval scope is a function of agent surface area, and the suite is the deliverable that compounds over the life of the system. A deliverable that compounds must have a line.
Founder mistake one: rolling eval into QA
The most common version of the missing line is “included in QA”. A QA lead at $80–$120/hr is assigned a job that requires a senior AI engineer at $180–$220/hr. The hourly delta is the smallest of the problems.
QA grades against deterministic specifications: string match, HTTP status, field value. LLM output does none of that. A correct response can be phrased ten ways, all correct, zero matching a deterministic string. The QA team either approves outputs that are off-thesis (passes the string check, fails the customer’s need) or rejects outputs that are on-thesis. Either way the release ships with a quality signal decorrelated from actual quality.
Folding eval into QA also absorbs the human ground-truth cost silently. Domain expert grading hours — the partner attorney for a legal feature, the senior radiologist for a clinical one — is the most-skipped sub-line in 2026 AI proposals, per CIO.com. The grading is treated as the founder’s free labor: hours appear in week four, surprise them in week five, slip the launch in week eight.
Founder mistake two: splitting eval across roles
The second pattern is the SOW where eval appears in three descriptions — engineering (“our engineers write evals”), PM (“our PM owns acceptance criteria”), QA (“QA runs the eval suite”) — and is dollarized in none. The proposal looks comprehensive; the work is unowned.
With no named owner, the test set never reaches 200 cases, the rubric never gets calibrated, the harness never runs in CI, the regression suite never gates a merge, the observability hookup never lands. Every sub-line is partially done by the adjacent role; no sub-line is fully done. Distributed accountability is, in practice, no accountability — the result the Ringelmann experiments documented in 1913.
The fix is one name on the line. One senior AI engineer is accountable for all five sub-lines. PM and QA support; they do not own a fraction. If the agency cannot put a single name on line 4, the line does not exist as a deliverable.
Founder mistake three: deferring eval to v2
The third pattern is the SOW where eval is described as “post-launch work” or “v2 scope”. The founder hears “ship the MVP, then add evals once we know what to evaluate”. The sequencing is wrong on two grounds.
Engineering: the eval suite is not bolted onto a shipped system; it defines what the system is supposed to do. Without it, v1 engineers have no shared definition of correct, and v1 ships against their implicit assumptions. When the suite is finally written in v2, it grades v1, finds the system does not meet the suite, and the team has no principled way to decide whether the system is wrong or the suite is wrong. Building evals after the system is the cheapest way to ship something nobody can defend.
Economics: retrofitting evals after production is 2x–4x the cost of building them pre-production. A pre-production suite is built against the workload the team is actively designing. A post-production suite is reverse-engineered from logs, calibrated against examples nobody remembers writing, and reconciled with whatever behavior v1 already has. The first two mistakes ship v1 with a quality signal decorrelated from quality. The third ships v1 with no signal at all and then pays a premium to add it after the system has calcified its assumptions.
The cost of skipping the line: silent regression on model swap
The cost of shipping without eval engineering becomes visible the first time the underlying model is updated. Frontier providers ship silent updates on a several-month cadence — Claude Opus 4.8 to 4.7, GPT-5 to 5.5, Gemini 2.51 to 3.2. The semantic version may be a point release; the behavior change underneath is not. A frontier point release on a feature with no regression suite typically surfaces three to seven silent regressions in the first week of post-swap traffic.
Without an eval suite the founder finds those regressions through customer complaints, not metrics. Lag from swap to first complaint is roughly two weeks; lag from first complaint to root-cause is another one to two weeks of release-note reading and log bisection. The customer-visible defect window opens for three to four weeks — paid as churn, refunds, and reputation damage on the customer success ledger.
An eval suite gating the model swap reduces that window to zero. The suite runs against the new model before deployment; thresholds fail; the swap rolls back to the prior alias; the team investigates with bounded scope. Skipping the line is buying the product without the insurance — and in 2026, the premium gets called roughly once per quarter.
How to read a proposal for the missing field
Three checks the founder can run on any AI MVP proposal in under five minutes:
Check 1: count the lines. A 2026 AI MVP SOW should have a separate, dollarized evaluation engineering line. Five lines means the field is empty. Six means somebody drew the field in. No dollar value = no deliverable.
Check 2: name the owner. Line 4 should have one accountable senior AI engineer named in the staffing plan. “The team” or “engineering” means no one. A line owned by no one is a description, not a deliverable.
Check 3: verify the percent. The line should land in the 20–30% band of build cost, per the eval budget rule. Below 20% means one of the five sub-lines is silently absent. Above 35% means platform-grade work is being pulled forward from the post-MVP retainer.
If any check fails, the proposal is the 2018 template in 2026 clothing. Do not reject the agency — it is fitting the template procurement expects. Add line 4 to the comparison spreadsheet, send the revised template back, and ask for a proposal that fills it. The fix is upstream of the agency. The fix is in the form.
The eval line is missing because the field was never drawn — and the founder is the only party who can draw it. The redraw is a single edit. It separates the 30% of 2026 AI MVPs that ship a defensible quality bar from the 70% that ship against a form whose fields predate the work.
For the procurement discipline of pricing evals as a contractual artifact rather than a documentation deliverable, see stop paying AI agencies for documentation — pay them for evals. The eval suite is the artifact the engagement is held to and the artifact the engagement should be paid against. The line exists when the founder draws the field.
FAQ
Why is the eval line missing from most AI MVP SOWs in 2026?
The SaaS SOW template the agency market inherited was forged in 2014–2018 with five header lines — discovery, design, engineering, QA, PM. AI work of that era was deterministic; the template never had a sixth field. Procurement spreadsheets enforce the five-line template, so agencies that add a sixth row lose deals to agencies that fit the form.
What does eval engineering buy that QA does not?
Five sub-lines: test-set authoring (80–200 cases), rubric design (anchored binary or 3-point), harness setup (CI runner), regression suite (pass-fail gates blocking merges), observability hookup (production traces joined to eval baselines). QA grades deterministic outputs against fixed strings; eval grades non-deterministic LLM outputs against rubric thresholds.
Where in the SOW should the eval line land?
As line 4 of a seven-line structure: discovery, design, engineering, evaluation engineering, QA, PM, infrastructure. Evaluation engineering precedes QA because the eval suite is what QA runs against on the non-deterministic surface. Owner: one named senior AI engineer.
How big should the eval line be?
20–30% of build cost. Below 20% means one sub-line is silently absent; above 35% means post-MVP platform work is being pulled forward. See the eval budget rule.
What is the cost of skipping the line in v1?
A frontier point release on a feature with no regression suite surfaces three to seven silent regressions in the first week of post-swap traffic. The founder finds them through customer complaints two to four weeks later — paid as churn and refunds, not on the engineering ledger.
Is “the agency does evals as part of engineering” the same as a dedicated line?
No. Eval named in the engineering description and not dollarized as a separate line is unowned work. One name on one dollarized line is the structural fix.
Why is splitting eval across PM, engineering, and QA worse than one named line?
Distributed accountability is no accountability. When eval is named in three role descriptions, each role assumes one of the others is doing it. One name on the line means one performance review captures the suite’s quality.
What about building evals in v2 after the MVP ships?
Two problems. Engineering: the suite defines what the system is supposed to do; without it, v1 ships against implicit assumptions. Economic: retrofitting evals after production is 2x–4x the cost of building them pre-production.
How do I tell in five minutes if a proposal is missing the line?
Three checks: count the lines (six, not five), name the owner (one senior AI engineer), verify the percent (20–30% of build). Failing any check means the proposal is the 2018 template in 2026 clothing — redraw the form, do not reject the agency.
Does the eval tool (Promptfoo, Inspect, Langfuse, DeepEval) affect line size?
Tooling is 5–10% of the eval line — most are open source or run under $1,000/month at MVP scale (Anthropic’s eval guidance). The other 90–95% is senior AI engineer labor.
Next step
If you are scoping a 2026 AI MVP, the AI MVP scoping worksheet names the seven SOW lines, the five eval sub-lines under line 4, and the percent band the proposal should land in before counter-signing. It is the redraw of the 2018 template, with the field for evaluation engineering already drawn.
Dirk Jan van Veen, PhD
