The user story is a 20-year-old SaaS convention, and it understates the spec for any AI feature shipping in 2026. It was designed for deterministic software, where the acceptance test is binary: the user logs in, or they do not; the report renders, or it does not. AI features violate that presumption. A summarizer that returns a non-empty string can summarize — and can also score 0.43 on faithfulness, hallucinate citations in 14 percent of outputs, and pass the user-story acceptance test while quietly destroying trust by week eight. The mismatch is not a tooling problem. It is a vocabulary problem.
The argument here: the WHAT-layer of a user story — the load-bearing middle of As a [user], I want [capability], So that [outcome] — is the single defect SaaS-trained product teams carry into AI work without noticing. The fix is to keep the WHO and the SO-THAT, kill the WHAT-as-acceptance, and replace it with a 4-section eval block. This article is a companion to the eval-first build playbook and inherits the master argument from the idea-to-product manifesto. If your team is sprint-planning an AI feature this quarter and the backlog still reads As a / I want / So that, this is the read.
Decision scope
This article is an editorial decision framework, not legal, financial, security, or accounting advice. Treat the worked example and numeric thresholds as illustrative planning heuristics. Calibrate against your own domain, traffic distribution, harness, and procurement model before signing.
What a user story actually says — and what it doesn’t say for AI
Mike Cohn’s 2004 User Stories Applied formalized the three-clause form the industry has used for two decades: As a [persona] / I want [capability] / So that [outcome]. The INVEST framework attached six properties (Independent, Negotiable, Valuable, Estimable, Small, Testable). For deterministic software, it works. For AI features, the format is partially broken — and the broken part is the one most teams treat as load-bearing.
| Clause | What the team uses it for | Survives AI? |
|---|---|---|
As a [persona] (WHO) | Identifies the user and prioritization context | Yes |
I want [capability] (WHAT) | Acceptance criteria — the contract closes when implemented | No |
So that [outcome] (SO-THAT) | Confirms the story is worth doing | Yes, with a caveat |
The WHO clause survives unchanged. Whether the user is a regulated-industry attorney, a sales-ops analyst, or a free-tier consumer still drives prioritization, traffic-distribution sampling, and risk tolerance.
The SO-THAT clause survives with a caveat. The outcome — “review the contract in 5 minutes instead of 30” — is the right business framing. The caveat: you can no longer verify the outcome by inspecting whether the capability was implemented. You verify it by measuring whether the capability performs at the level the outcome requires. That measurement is what evals are for.
The WHAT clause is the one that breaks. “I want a system that summarizes the contract” gets ticked off the moment a string of summary-like text comes back. The format has no slot for faithfulness, completeness, input distribution, failure modes, per-call cost, latency, refusal behavior, or the regression rule that applies when a future model upgrade moves the score.
INVEST’s Testable property is where it shows up most acutely. For deterministic software, Testable means “we can write a unit test.” For AI, Testable means “we can write an eval suite, define a passing threshold per axis, and gate the deploy on the suite running green in CI.” These are different artifacts. Pretending the user-story format covers both is the mistake. The fix preserves what works and replaces what does not.
The 4-section eval-scoped feature block
A working eval-scoped feature spec has four sections. Each is a contract artifact — versioned, signed, living in the repository alongside the code, named in the SOW. The eval-first PRD covers how this nests inside the PRD; what follows is the feature-level block that replaces the WHAT-clause.
Section 1 — Input distribution. The realistic input space the feature will see, sampled or synthesized. Names suite size (typical 80 to 250 per capability), source (production logs, domain-expert-labeled synthesis, public benchmark, or a weighted mix), stratification (which sub-populations are over-sampled), and versioning (git tag, hash, new inputs added by PR). A suite of 30 cherry-picked inputs is worse than no eval — it produces false confidence at a point production traffic does not match.
Section 2 — Rubric. What “good” means, decomposed by axis. Canonical axes for an LLM-output feature: groundedness, refusal-correctness, format adherence, latency, cost, and a safety subset (jailbreaks, PII leakage, prompt injection). Each axis is graded binary or 3-point, anchored against language drafted with a senior engineer and signed by the founder. The MT-Bench paper (Zheng et al., NeurIPS 2023) documents that LLM-as-judge on a 1–10 Likert scale shows verbosity, position, and self-preference biases that make a “score ≥ 4” threshold unreliable. The 2026 default is anchored binary or 3-point grading, with a human spot-check on a 20-output sample per cycle.
Section 3 — Threshold. The numeric pass bar per axis, with a written rationale tying the number to a commercial constraint. “0.92 because under 0.90 user-reported hallucinations cluster and the regulated-industry buyer will not approve below 0.92” is defensible. “0.92 because it sounds high” is not. Per-axis floors matter as much as the aggregate: a 0.85 average can hide a catastrophic 0.40 on the safety axis. The suite fails if any axis falls below its floor — even if the average passes.
Section 4 — Regression rule. The CI gate runs the suite on every PR touching a prompt, retrieval index, model alias, system prompt, or tool schema; deploys are gated on green. The suite reruns on every provider model-alias upgrade, and the diff is filed as part of the migration decision. If the suite drops below threshold, the milestone is not paid and the build holds until it passes. New axes are added by amendment; thresholds raised by quarterly review; the suite is not edited in flight to lower the bar to meet the score.
Section 4 is what makes the spec a continuous contract over the life of the feature, not a one-time acceptance event — a property no user story has ever had.
When user stories still help (the WHY, not the WHAT)
User stories should not disappear from your backlog. They should stop being the load-bearing artifact for AI feature acceptance. The WHO clause informs input-distribution stratification in section 1 of the eval block; the SO-THAT remains the prioritization argument; the format remains the right cover sheet for non-engineering stakeholders, who will not read an eval suite directly.
The cleanest mental model: keep the user story as the prioritization unit; replace the acceptance criteria with the eval block. The team reads the story to remember why the work matters and the eval block to know when the work is done. The WHAT-clause stopped doing both jobs around the time LLM-graded outputs became the dominant interaction model.
Worked example: the same feature in both forms
Take a contract-clause summarizer for a legal-tech MVP — a typical mid-market AI product in 2026.
Form A — Agile user story with Gherkin
Story: As an in-house attorney, I want the system to summarize each clause of an uploaded contract, So that I can review a 40-page agreement in under 10 minutes.
Acceptance criteria (Gherkin):
- Given an uploaded PDF, When the user clicks “Summarize”, Then the system displays a summary of each clause.
- Given a clause longer than 500 words, When summarized, Then the summary should be no more than 100 words.
- Given a multi-page contract, When processed, Then all clauses should be summarized.
- Given the summary panel is open, Then the user should be able to expand to view the original clause.
Three of the four criteria are UX requirements. The fourth — “summarize each clause” — is binary in form, meaningless in substance. Team ships a system returning a string per clause; criteria met; product not deployable. The story does not say how faithful the summary must be, what to do with ambiguous or non-English clauses, how to handle indemnity / IP clauses where a wrong summary is a commercial liability, or what happens when the provider silently upgrades the model alias and faithfulness drops 8 points overnight.
Form B — Eval-scoped feature block
WHO: In-house attorney at a mid-market regulated-industry company. SO-THAT: Review a 40-page agreement in under 10 minutes (down from a 45-minute baseline).
Input distribution (Section 1): 200 contracts, stratified — 60 standard commercial, 40 employment, 40 SaaS service, 30 IP-assignment / NDA, 20 cross-border (non-English clauses), 10 adversarial. Clauses labeled by an in-house attorney with gold-standard summaries. Suite versioned
v1.4.2.Rubric (Section 2): Groundedness (3-point, LLM-judge + attorney spot-check on 20 per cycle), Completeness (binary), Refusal-correctness (binary, on ambiguous / redacted clauses), Safety subset (binary, IP-assignment / NDA), Format (binary, 30 to 100 words, schema match), Latency (P95 per clause), Cost (per contract).
Threshold (Section 3): Groundedness 0.88 weighted, floor 0.92 on safety subset; Completeness 0.85; Refusal-correctness 0.95 (false-positive refusals capped 0.05); Safety subset zero misstated obligations; Format 0.98; Latency P95 under 8s; Cost under $0.06 per contract, alarm at $40K / month.
Regression rule (Section 4): Suite runs on every PR touching prompt / retrieval / model alias / system prompt; deploy gated on suite green. Suite reruns on every provider model-alias upgrade. If any axis drops below floor, milestone is not paid until the suite passes. New axes added by signed amendment.
Form B is four times the length on the page. It is also the only one that gives the team and founder a shared definition of “done.” Form A closes the contract in week nine. Form B closes when the suite passes. The extra weeks, if they happen, are the real cost of the feature — Form A hides it in a future quarter, paid in user trust; Form B pays it on the way in.
What this changes downstream
Two downstream artifacts shift. Sprint planning: engineers stop estimating story points for “implement summarizer” and start estimating cycles to lift a specific axis from a current score to threshold. “Lift groundedness from 0.79 to 0.88 on the safety subset” becomes the sprint goal. After three or four sprints, the team has a defensible cycle-time-to-score-delta curve and the retro becomes a numeric conversation — not unauditable arguments about scope creep. The AI feature sizing framework covers Small / Medium / Large in eval-cost units; stop budgeting in story points covers the budgeting end.
The founder / PM role: the founder was, in practice, the author of the WHAT-clause (“I want a contract summarizer”). Under the eval-scoped block they author two things engineers cannot — the rubric anchors (the eval rubric template gives the six-section format) and the threshold rationale (why 0.88, not 0.80 or 0.95). Every threshold the founder signs is enforced by CI for the life of the feature. A founder who signs safety-subset-floor: zero misstated obligations has bought continuous protection. A founder who signs So that I can review the contract in 5 minutes has bought a feeling. For procurement, stop scoping AI projects in features covers the contract-language end of the same shift.
Get the scoping worksheet
If you are writing the next sprint backlog or vendor SOW, the highest-impact move is to swap the WHAT-clause for an eval block before the contract closes. The AI MVP scoping worksheet bundles the 4-section block, rubric anchors, threshold rationale template, and regression-rule language into one PDF you can take into the next planning meeting.
FAQ
Should we delete all user stories from our backlog?
No. The WHO and SO-THAT clauses remain useful for prioritization and stakeholder alignment. What changes is the acceptance criteria — the WHAT-clause stops being load-bearing. Keep the story as the prioritization unit; replace the acceptance criteria with the eval block.
Does this apply to every feature, or only AI features?
Only AI features — those where the output is generated by a probabilistic model (LLM, vision, recommender, ranker). Deterministic features still scope cleanly as user stories with Gherkin. A typical AI MVP has roughly 60 percent AI features (eval-scoped) and 40 percent deterministic (story-scoped).
What if our team has never written an eval before?
The 2026 open-source baseline is Promptfoo, Inspect AI, or DeepEval; Langfuse for hosted. First suite takes 1 to 2 weeks for a senior engineer working with a domain expert. Subsequent suites take 3 to 5 days. The eval-first PRD describes the kickoff.
Doesn’t Gherkin (Given / When / Then) solve this?
Partly, for the UX-shape of acceptance criteria. Not at all for the AI-quality-shape. Gherkin handles state transitions and binary outcomes; it does not handle continuous quality distributions, per-axis floors, or regression rules. Adding Then the summary should be faithful pretends faithfulness is binary, and it is not.
How do we estimate eval-scoped work?
Cycle-to-score-delta estimates. After three or four sprints calibrating against the suite, the team has a defensible curve — “moving groundedness from 0.79 to 0.83 took 4 days last cycle.” The AI feature sizing framework covers Small / Medium / Large in eval-cost units.
What happens to Jira and Linear?
They still work. The user story sits at the top of the ticket — WHO and SO-THAT only. The body links to the version-controlled spec file in the repo, where the eval block lives as a code artifact.
Who writes the eval block — the founder or the engineer?
Both, in defined parts. The senior engineer drafts sections 1 (input distribution) and 4 (regression rule). The founder drafts section 3 (threshold and rationale). Section 2 (rubric) is co-authored: engineer drafts anchor language, founder signs that it is correct for the domain. Single-author eval blocks fail.
Doesn’t this make sprint planning slower?
The first two cycles, yes. After that, faster. Eval-scoped sprints almost never have ambiguous acceptance — the suite is green or it is not. The savings are in avoided arguments, avoided rework, and avoided post-launch hallucination triage.
Does this work for non-LLM AI, like recommenders or classifiers?
Yes, with different axes. A recommender’s rubric has precision@k, recall@k, diversity, freshness. A classifier’s has precision, recall, F1, per-class floors. The 4-section structure is invariant; the axes inside the rubric are task-specific.
What’s the single biggest mistake teams make when first switching?
Writing the eval block as a document instead of a code artifact. A markdown file describing the suite is not the suite. The suite is harness config, input JSONL, rubric prompts, and CI gate — all version-controlled, runnable, reproducible. Shipping the cover sheet without the artifact gives the appearance of eval scope and none of the protection.
Dirk Jan van Veen, PhD is the CTO of SFAI Labs. He writes on the technical scoping and evaluation discipline of shipping AI features. See also the eval-first build playbook, the eval-first PRD, and the AI PRD explained primer for non-engineers.
Dirk Jan van Veen, PhD
