An eval-bound SOW ties contractual acceptance to a frozen eval set and a written pass-rate threshold — not to “delivered” alone. It is the single change that turns an AI Statement of Work from a wish into a remedy. With it, every dispute becomes a technical question — the suite passed or it didn’t — instead of a political one. The four clauses below are what to look for when red-lining your next AI contract.
This is the procurement companion to the eval-first build playbook inside the broader idea-to-product manifesto. Where the playbook scopes the feature in evals, this article writes the scoping into a contract the vendor cannot quietly back away from.
Table of Contents
Why “delivered” is not an acceptance criterion
Most software SOWs treat “deliverable” as a binary the buyer ticks off: the feature exists, the milestone closes, the invoice is approvable. That works for a 2018 SaaS feature where the code either persists the row or it doesn’t. It does not work for a 2026 AI feature.
An AI feature is probabilistic. It does the right thing on most inputs and the wrong thing on some — and the distribution of those failures is what the buyer is paying the vendor to engineer down. If the contract says nothing about quality on the failure distribution, the vendor has shipped “delivered” the moment the demo works on three hand-picked cases.
Three structural reasons make this the central commercial problem of an AI build. Model behavior moves under the product — Anthropic shipped Claude Opus 4 through Opus 4.7 and Sonnet 4.6 over the past year, OpenAI shipped GPT-5 and successor updates, and each transition silently shifts behavior on any system pinned to a model alias (Artificial Analysis). The long tail is where failures live: Gartner has consistently reported that roughly 85 percent of AI pilots stall before reaching production at scale, a pattern McKinsey corroborates in its State of AI editions. And the legal frame for “complete” is undefined without an eval contract — most software-services SOWs define completion as “the deliverable was produced” plus a 5-to-15-day acceptance window for material defects. For AI, “material defect” is undefined unless the contract names what good looks like, on what inputs, at what threshold.
The fix is four clauses any competent counsel can adapt.
The four clauses that make a SOW eval-bound
A SOW becomes eval-bound when these four contractual moves are present and reference each other. None alone is sufficient. A frozen eval set with no threshold is a quiz with no passing grade. A threshold with no frozen set lets the vendor choose the inputs that pass. A regression suite the buyer never receives at handoff lives inside the vendor’s git history. A right to test that is not gated to payment is a courtesy.
Clause 1. The frozen eval set
The first clause names a specific, dated, immutable eval set as the artifact against which delivery is graded. “Frozen” means three things: named (filename or hash), dated, and unmodifiable without a written change order.
Illustrative framing — to be adapted by counsel, not used verbatim:
Acceptance Eval Set. The deliverable shall be evaluated against the eval set named
acceptance_v1.jsonland committed to the project repository under/evals/acceptance/no later than the close of Milestone 2. The set will contain no fewer than [N] representative inputs per capability listed in Schedule A. Once committed, the set is frozen for the remainder of the engagement. Any modification requires a written change order.
The moves that matter: the set lives in the buyer’s repo, it is sized per capability rather than aggregate, and any change requires a written change order. The change-order requirement is the single mechanism that prevents the most common vendor pattern — quietly removing failing inputs as the project ages.
Clause 2. The pass-rate threshold
The second clause writes the pass threshold into the contract as a numeric value per capability. A threshold without a number is the vendor’s invitation to argue “the model is broadly working” at every milestone.
Threshold values vary by task type — classification might be graded at 92 percent accuracy on priority labels, extraction at 95 percent exact-match on structured fields, generation at 80 percent rubric-pass on a five-criterion rubric, an agentic capability at 75 percent trajectory-correct against the gold trace. Numbers are negotiated during scoping and locked when the eval set is frozen.
Illustrative framing:
Acceptance Threshold. The deliverable shall be deemed accepted only when, on the frozen Acceptance Eval Set, each capability in Schedule A achieves at least the pass-rate threshold in Schedule B when executed against the production model and prompt configuration. Aggregate pass rates do not satisfy this clause; each capability must independently clear its own threshold.
Three moves matter: thresholds are per capability (a high-volume capability cannot drown out a structurally broken one), thresholds live in a Schedule (so they can be updated by change order), and thresholds are met against the production configuration (not a development model that won’t ship).
A common founder mistake is to set the threshold too high in early scoping. Set it where the feature is commercially defensible and let engineering work move the measured number above it. A threshold the team can clear with margin is one the team will defend honestly.
Clause 3. The regression-suite handoff
The third clause names the regression suite as a deliverable, owned by the buyer at handoff, runnable by the buyer’s team or a successor vendor.
For an AI build, code is not the most load-bearing deliverable — the eval suite, harness configuration, prompt artifacts, rubric files, and the CI workflow that runs them on every commit are. They are what lets the buyer detect regression after the engagement ends. A delivered AI feature without its eval suite is a one-shot demo.
Illustrative framing:
Eval Suite Handoff. The Acceptance Eval Set, the rubric files, the eval harness configuration, the production prompt artifacts, and the CI workflow that executes the suite shall be delivered to the buyer’s repository on or before the final milestone. All such artifacts are work made for hire, owned by the buyer. The suite shall execute end-to-end on the buyer’s CI environment without dependencies on vendor-private services.
Promptfoo, Inspect, Langfuse, and DeepEval are the four eval harness tools that dominate 2026 stacks. A vendor who insists on a proprietary harness is asking the buyer to accept lock-in on the quality system.
Clause 4. The right to test before payment
The fourth clause ties payment to verified pass on the frozen eval set, executed by or on behalf of the buyer, before the milestone invoice is approvable. Without this clause, the first three are advisory.
Illustrative framing:
Acceptance Procedure. Within five business days of vendor notification that a milestone deliverable is ready, the buyer shall execute the Acceptance Eval Set against the production configuration. The deliverable shall be deemed accepted, and the corresponding milestone invoice shall become approvable, only upon written confirmation that each capability in Schedule A has cleared its threshold in Schedule B. If any capability has not cleared, the buyer shall provide a written defect report, and the vendor shall remediate within the cure period in Section [X].
Three moves matter: payment is tied to verified pass (not vendor self-report), the buyer or its agent runs the suite, and a cure period (typically 5 to 15 business days) is named so the contract specifies remediation rather than dissolving into dispute.
Together, the four clauses turn the eval contract into a procurement instrument with teeth — and make the engagement smoother in practice, because the vendor knows from day one what acceptance looks like and can engineer toward it.
Vendor objections and how to answer them
The first time you propose eval-bound clauses, the vendor will push back. The objections are predictable. Each has a counter-position you can hold without burning the relationship.
“We can’t commit to a threshold before scoping.” Correct — the threshold is written into Schedule B during the scoping milestones, not in the master SOW. If the vendor cannot commit by the end of scoping, they cannot commit to building the feature.
“Evals are a deliverable, not a gate.” Self-serving. If evals are not a gate, the buyer is paying for output the buyer cannot grade. Treating evals as a gate is what makes the work measurable — which is what makes the engagement payable on a fixed price.
“The eval set will keep growing as we learn.” It will grow — but the acceptance set in Clause 1 is frozen, and additions require change orders. The vendor’s exploratory evals can grow freely; the acceptance set is locked.
“Thresholds force us to cheat the rubric.” The fix is rubric quality during scoping. A rubric that can be cheated by re-prompting was not decomposed enough — push the decomposition until each sub-criterion has an anchored example. The eval rubric template walks the moves that prevent this.
“Right to test before payment makes our cash flow a hostage.” The counter is the cure period. The buyer is not refusing to pay — the buyer is paying on verified pass. The vendor’s cash flow is hostage to engineering quality, which is the entire point of an outcome-based engagement.
“Other clients sign on outputs, not on evals.” Possibly — and those clients are paying for demos they will not be able to maintain.
A vendor who cannot work under eval-bound clauses after a real conversation is not a vendor a non-engineer founder should hire. The vendor eval-discipline guide is the screening pass that should run before the SOW round.
Red flags in a non-eval-bound SOW
A 15-minute scan of an inbound SOW can tell you whether the draft is eval-bound.
| Red flag | What to look for | Why it matters |
|---|---|---|
| Acceptance defined as “deliverable was produced” | Acceptance section with no threshold or eval set reference | The default and the most expensive single defect |
| Quality language is qualitative only | “High-quality” or “reasonable accuracy” without numeric thresholds | The vendor cannot be held to a number that does not exist |
| Eval set referenced but not frozen | “We will agree on test cases” without a frozen, dated artifact | Allows the vendor to grow or shrink the set during the build |
| Threshold aggregate, not per capability | “The system shall achieve 90 percent accuracy” without decomposition | A high-volume capability masks a broken one |
| Eval suite not in deliverables | Deliverables list code, docs, and demo — not rubric files or CI workflow | The buyer inherits a feature without regression detection |
| Payment released on demo | Milestone payment tied to demo sign-off, not eval pass | Turns the contract into a payment for a demo |
| No model-migration clause | Contract silent on mid-build model upgrades | The buyer inherits a system whose model can be silently swapped |
A SOW with three or more red flags is structurally not eval-bound. Red-line each one against the four clauses above. Most law firms with an AI practice will draft the language once and reuse the template on subsequent engagements.
Where the model-migration clause fits
Frontier model updates are the silent risk an eval-bound SOW must name. A SOW that does not specify what happens at a model migration pre-commits the buyer to either accepting silent behavior changes or paying for unscheduled remediation. The clause is short and lives in the Acceptance section, immediately after the threshold language:
Model Migration. The acceptance thresholds in Schedule B are met against the model alias in Schedule C as of Milestone 2. Should either party elect to migrate to a successor model alias during the engagement, the Acceptance Eval Set shall be executed against the successor model, and any capability that fails to clear its threshold shall be remediated by the vendor before the migration is complete. The vendor shall not unilaterally migrate the production model alias without written notice and a successful eval re-run.
Two moves matter: the buyer is named in the migration decision, and the eval suite is the migration test. Founders who skip this clause discover their AI feature has changed behavior in production six weeks after the engagement closes — because the vendor moved the production alias forward as a routine update.
What to do next
If you are red-lining an AI SOW this week, three actions will change the outcome.
First, walk every clause in the vendor’s draft against the four-clause anatomy above. Mark the gaps — a 15-minute scan is usually sufficient. Second, replace generic acceptance language with eval-bound language. The illustrative clauses are starting points, not legal advice — counsel should adapt them to your jurisdiction. Third, schedule a call with the vendor to walk the four clauses and the six common objections. A vendor who can hold the conversation is one you can hire; a vendor who deflects is one who will deflect on quality in week 10.
SFAI Labs runs a 30-minute idea-to-product review for founders red-lining an inbound AI SOW. We walk the eval-bound clauses against your specific contract, flag the structural defects, and leave you with a marked-up draft you can take to counsel. Book a discovery call.
For the contract-anatomy companion, see what a defensible idea-to-product SOW looks like with examples.
Frequently asked questions
What is an eval-bound SOW in plain English?
An eval-bound SOW defines “delivered” as “the deliverable cleared a written quality threshold on a frozen set of test inputs”. Instead of paying on demo sign-off, you pay on verified pass against a test suite both parties agreed to upfront. The four moves are a frozen eval set, a pass-rate threshold per capability, a regression-suite handoff, and a right to test before payment.
Are the clause samples here legally meaningful, or do I need a lawyer?
You need a lawyer. The samples are starting points to anchor the conversation with counsel — not vetted for any particular jurisdiction or contract framework. The structural moves matter; the specific language must be drafted by counsel familiar with your existing master services agreement.
Does this work for fixed-price contracts or only time-and-materials?
Both, and it is more important on fixed-price. On T&M, the buyer can stop paying at any milestone if quality drifts. On fixed-price, the buyer has pre-paid and needs a contractual instrument to enforce quality on the back end. Eval-bound clauses are what protects fixed-price buyers from paying for a deliverable that does not clear quality.
How is the eval set sized?
A working range is 80 to 150 inputs per capability for classification and extraction, 30 to 60 inputs per capability for agentic or generation tasks that are expensive to grade, plus 10 to 30 refusal inputs per capability. A typical acceptance set is 150 to 500 inputs total. The sizing logic is covered in the eval-first build playbook.
What pass-rate threshold should I write?
It depends on task type. A common starting framing: 90 to 95 percent for classification with well-defined labels, 85 to 95 percent exact-match for extraction, 75 to 85 percent rubric-pass for generation with anchored rubrics, 70 to 80 percent trajectory-correct for agentic tasks. Set the threshold where the feature is commercially defensible, not at the ceiling.
Can I withhold payment if the suite fails?
Yes — that is the purpose of Clause 4. The contract specifies a defect-report procedure and a cure period (typically 5 to 15 business days) within which the vendor remediates. If failure persists past the cure period, the contract should specify the buyer’s remedy — payment hold, kill-fee provisions, or termination for cause depending on severity.
What if the vendor proposes their own eval harness?
Structural red flag. The eval suite is one of the most load-bearing assets the buyer takes from the engagement; if it lives in vendor-proprietary tooling, the buyer is locked into that vendor for post-engagement quality maintenance. Require an open-source harness (Promptfoo, Inspect, Langfuse, DeepEval are the four most common in 2026) executable on the buyer’s CI.
How does this interact with model migrations during the build?
The eval suite is the migration test. Before any model alias upgrade, the suite runs against the new model and the report is diffed against the prior model; any capability that regresses by more than the agreed delta blocks the migration. Teams without this clause discover regressions in production after the engagement closes.
What if the vendor says “no one else asks for this”?
That information is useful — it tells you which vendors have built eval discipline as a core competence and which have not. The set that can comfortably work under eval-bound clauses is growing on every quarterly survey. The set that cannot is a set you are filtering out, which is the entire purpose of a procurement screen.
How do I introduce these clauses without souring the relationship?
Frame them as risk-management requirement, not distrust. Most vendors prefer eval-bound clauses once they understand them, because the clauses make the work measurable and the engagement defensible against future scope creep. “Our risk team requires acceptance to be tied to measurable quality on a frozen test set. Here’s the language we’d like to add — can we walk through it together?”
Arthur Wandzel
