The spec is wrong in week 2 because it was written in 2018-shape in week 0. Two weeks in, the lead engineer Slacks: “the model is producing X, your spec doesn’t say what to do.” The founder edits the doc. Three days later: another message. By week 3, the spec is a moving target and the eval bar — if there was one — has slid to whatever the current prompt produces.
This is the rewrite cycle: the dominant failure mode of AI requirements written by non-engineer founders, caused by five authoring habits learned from SaaS. The rules below name those habits and replace them, each paired with the week-2 symptom it prevents.
It builds on the idea validation playbook, part of the idea-to-product manifesto.
Table of Contents
- The rewrite cycle and why it happens
- Rule 1 — Specify by outcome, not feature
- Rule 2 — Anchor every task to an eval
- Rule 3 — Write the failure budget on day 1
- Rule 4 — Name the no-AI fallback per task
- Rule 5 — Lock the model tier, leave the alias free
- The one-page audit
- Frequently Asked Questions
- Closing
The rewrite cycle and why it happens
A SaaS PRD assumed the implementation was deterministic. “When the user clicks Save, the row is persisted” is one assertion the team either ships or does not. The spec could pin UI because the engine underneath — a database, an API — was predictable.
An AI implementation is not deterministic. The same input produces different outputs across runs, and the model layer changes underneath the spec inside a quarter. Anthropic’s Building Effective Agents treats measurable, falsifiable success criteria as a structural requirement of any agent spec — not a QA afterthought. McKinsey’s State of AI reports roughly 78% organizational AI adoption with a small fraction capturing material value; the throughline is teams ship at a quality bar that was never written down.
Five patterns produce most of the rewrites we see. Five rules fix them.
Rule 1 — Specify by outcome, not feature
The rule. Every section starts with the user’s job-to-be-done, not the UI element. Feature lists are engineering outputs, not founder inputs.
Applied. “The user processes the morning inbox in under 15 minutes by acting on a prioritized queue of 5–10 actionable items, where actionable means one-click reply, archive, or escalate.”
Violated. “The app shows a left-hand panel with priority buckets P1–P5 and color-coded badges, with a reply button under each email and a snooze dropdown.”
Week-2 symptom. The team discovers the model is better at writing the reply than at sorting into 5 buckets, so the natural UI is a draft-reply card with a “send / edit / archive” tray — no panel, no buckets. The UI spec is torn up. The outcome the founder cared about was a 15-minute inbox; the feature list was a guess from a 2018 mental model.
Authoring prompt. “For each capability, answer: what is the user trying to accomplish, measured in time, money, or risk avoided? Do not name a UI element until the team has built a feasibility probe.”
Founder owns the outcome, engineer owns the form. Pin both and the spec gets rewritten when the model shapes the form differently than imagined. See the AI PRD explained for the document-level treatment.
Rule 2 — Anchor every task to an eval
The rule. Every task has an eval set, a rubric, and a pass-rate threshold. No eval, no requirement. A line of spec without an eval anchor is a wish.
Applied. “Task: classify inbound email into a 1–5 priority bucket. Eval: 20 hand-labeled emails covering edge cases (vendor invoices that look like pitches, short messages from large clients, bilingual threads, reply-all noise). Rubric: pass if the bucket is within one level of the human label AND the rationale references at least one signal from the body. Threshold: 85% pass on Claude Opus 4.8, temperature 0.”
Violated. “Task: classify inbound email into priority buckets and explain the priority. Acceptance: the priority feels correct during the pilot.”
Week-2 symptom. The team ships at “founder says looks right.” Three weeks later, a P4 email gets labeled P1, the user loses 20 minutes, and the founder asks the team to “make the priority better.” With no falsifiable target, the team prompt-tunes until the most recent example looks better — silently regressing three earlier ones. The spec gets amended retroactively, but the team has been shaping the prompt for three weeks and the eval is now grading its own homework.
Authoring prompt. “For each task, write ten hard inputs the founder is already worried about. If the founder cannot name ten, the task is not yet understood well enough to spec.”
The eval-first PRD makes the structural argument; this is its operational compression. The project-scope parallel lives in stop scoping AI projects in features, scope them in evaluations.
Rule 3 — Write the failure budget on day 1
The rule. Per task, name the acceptable rate of each kind of failure. Not “no errors.” A specific number per failure mode, written before code.
Applied. “Priority-classification failure budget: ≤8% recoverable errors (P3 labeled P4 — user finds the email via search); ≤1% dangerous errors (P4 labeled P1 — user interrupted during deep work); 0% data-handling errors (priority returned with no rationale).”
Violated. “The system should be accurate. Errors should be rare. We will tune as we go.”
Week-2 symptom. The eval returns 84% pass — one point under threshold. The team asks whether it matters. The founder asks what kind of errors live in the missing 16%. Nobody can answer because the budget was never decomposed. Two days of categorization later, most failures turn out to be over-priority errors (dangerous; budget should have been 1%). Two days lost to a 30-minute week-0 conversation.
Authoring prompt. “For each task, list the three kinds of error the implementation can produce. For each, write the maximum acceptable rate and the user-visible consequence of one such error.”
“85% pass” is a number. “85% pass with ≤1% over-priority errors” is a contract. The first is renegotiable in week 2; the second is not.
Rule 4 — Name the no-AI fallback per task
The rule. Every task has two rows: an AI row and a no-AI fallback row. The fallback runs when the model is unavailable, the input is out of distribution, or confidence drops below a named threshold.
Applied. “AI row: Claude Opus 4.8 prompt returns bucket plus rationale; confidence threshold = rationale references at least one body-text signal. Fallback row: route to manual inbox with an ‘uncategorized’ badge; never assign a bucket without a referenced signal; never block inbox loading on model latency above 3 seconds.”
Violated. “If the AI fails, the system should gracefully handle the error.”
Week-2 symptom. The model API has a 90-second outage on a Tuesday morning. The spec has nothing to fall back to, so the inbox view goes blank. The user files a support ticket. The founder amends the spec, but the team ships a hotfix that doesn’t match the rest of the spec because there is no rest-of-spec fallback pattern. Every subsequent fallback is bespoke.
Authoring prompt. “For each task, write what the user sees if the model is unreachable for 60 seconds, if the model returns a low-confidence answer, and if the input is in a language or format the eval never covered.”
The fallback row is co-equal with the AI row. Production-grade systems route a meaningful share of steady-state traffic through fallback paths because confidence thresholds catch hard inputs the eval under-represented. The fallback answers a question SaaS PRDs never had to ask: what is the floor on user experience when the model layer is unavailable?
Rule 5 — Lock the model tier, leave the alias free
The rule. Name a model tier — frontier, mid, small — not a specific alias. Run the eval and threshold against a named alias at authoring time for reproducibility, but do not contractually bind the implementation to that alias.
Applied. “Priority classification runs on a frontier-tier model (Claude Opus class, GPT-5 class, or Gemini 2.5 Pro class — see the Artificial Analysis leaderboards). Eval-time alias: Claude Opus 4.8, temperature 0. Substitution: any model in the frontier tier, provided the eval threshold is re-met.”
Violated. “The system uses Claude Opus 4.8 for priority classification.”
Week-2 symptom. Six weeks in, Anthropic releases a new Claude Opus with better quality and lower latency. The team wants to migrate; the spec says “Claude Opus 4.8.” Founder, legal, and implementation spend two days renegotiating to permit the swap. Meanwhile, cost-per-call could have dropped 30% the day of release. The model layer turns over inside a quarter, sometimes inside a month. A spec aliased to a single version is a procurement liability disguised as precision.
Authoring prompt. “For each task, name the tier required and the lowest tier on which the eval still passes. The spec authorizes any model in the named tier, subject to a re-run of the eval.”
Tier-not-alias also forces a cost discipline the violation hides: a task that needs frontier is not the same investment as one that runs on mid. For the cluster-level case that the PRD needs a structural rewrite, see rethinking the PRD for AI-native products.
The one-page audit
Open the current AI requirements doc and apply the five rules in order. The audit is binary per rule per task — pass or fail.
| Rule | Audit question | Pass criterion |
|---|---|---|
| 1 | Does each task lead with the user’s job-to-be-done, measured in time/money/risk? | Yes, no UI element named before the outcome. |
| 2 | Does each task have an eval set, a rubric, and a pass-rate threshold? | Eval ≥ 10 cases, 3-line rubric, single-number threshold. |
| 3 | Does each task have a failure budget broken out by error kind? | At least 3 error kinds named with maximum rates and user-visible consequences. |
| 4 | Does each task have a no-AI fallback row co-equal with the AI row? | Three fallback states defined: model unreachable, low-confidence, out-of-distribution. |
| 5 | Does each task name a model tier with the alias free? | Tier named, eval-time alias named separately, substitution policy stated. |
A task that fails any rule is a task the build team will revisit in week 2.
A spec passing all five rules satisfies the AI-native requirements definition this cluster has been developing across the AI PRD explained, the eval-first PRD, and rethinking the PRD for AI-native products. It is the spec a non-engineer founder can hand to a vendor or in-house team and hold them accountable to weeks or months later.
For a copy-paste template — per-task eval grid, failure budget table, tier-substitution policy — request the AI MVP scoping worksheet.
Frequently Asked Questions
How long should an AI requirements doc be?
Shorter than the SaaS-era PRD. A three-task MVP fits in three to five pages — one per task, half for the failure budget, half for the tier policy. A 30-page doc usually means the founder is hedging because the eval set is missing.
Who writes the eval set — the founder or the team?
The founder. The team can shape the rubric and threshold, but eval cases come from domain knowledge only the founder has. A team writing its own evals is grading its own homework, and the threshold becomes unfalsifiable inside two weeks.
What if the founder cannot list ten hard inputs for a task?
That is the signal the task is not ready for build. Spend a week in the workflow with a notebook. Ideas that fail in production usually fail because the spec was written from memory, not from the workflow itself.
How does this interact with the SOW the founder signs with a vendor?
The SOW points at the requirements doc. The doc defines “delivered” via the eval threshold (Rule 2) and the failure budget (Rule 3). A SOW with no attached five-rule-compliant doc is one the vendor can interpret loosely. Milestone billing should tie payment to the eval threshold passed, not to documents shipped.
Is “frontier-tier” too vague to put in a contract?
No, when paired with the eval-time alias and substitution policy: “frontier-tier model, evaluated against Claude Opus 4.8 at temperature 0, substitutions permitted in tier provided the threshold is re-met.” That is enforceable. “Claude Opus 4.8” alone is a procurement liability dressed as precision.
What is the most common authoring mistake against these rules?
Skipping Rule 3 — the failure budget. Founders write the eval and feel done, but without decomposed failure rates per error kind, the threshold has no asymmetry. A reply-drafting task that “passes 85%” might still hallucinate factual claims 1% of the time — the failure mode that ends the product.
Do these rules apply to non-LLM AI systems?
Yes. The rules generalize to any probabilistic system — classification, retrieval, recommendation, vision. Every element is model-architecture-agnostic.
How often should the requirements doc change during the build?
The eval set grows. The rubric stays fixed unless a new failure mode appears. The threshold moves only if the failure budget changes. Fallback rows and tier policy are stable. If the spec is amended weekly, Rule 1 or Rule 2 was not met at authoring time.
What is the smallest version of the spec that still passes all five rules?
For a single-task MVP, two pages: one paragraph of outcome, a 10-case eval with rubric and threshold, three error-kind rates, three fallback states, one tier-and-alias entry. Two pages is the floor; anything tighter is a hedge.
How does this relate to the broader cluster on idea validation?
The validation playbook ends with a defensible PRD. This article keeps that PRD durable through the build.
Closing
The week-2 rewrite cycle is not a sign the founder is bad at requirements — it is SaaS-era habits applied to a non-SaaS-era implementation. Run the audit against the current doc tonight: tasks that fail Rule 1 get renegotiated first; tasks that fail Rule 2 lose their quality bar; tasks that fail Rules 3, 4, or 5 ship and quietly underperform.
The non-engineer founder shipping AI products in 2026 does not need to learn engineering. They need requirements writing that respects the probabilistic substrate underneath. Five rules. One page. Two weeks of build, unrewritten.
Arthur Wandzel
