Your AI MVP cannot launch without a frozen test set the team can replay in under ten minutes. Most teams ship without one because the eval conversation defaults to tooling — Promptfoo or Inspect, Braintrust or Langfuse — and tooling is the wrong frame. The runner is a commodity. The test set is the artifact. A launch test set has five operational properties, comes from three named sources, takes eight steps to build, and has three failure modes that look fine until production. Walk this piece and you can ask a vendor for the test set, name what good looks like, and refuse delivery if it is not on the table.
This article builds on the eval-first build playbook, part of the idea-to-product manifesto. It pairs with the eval rubric template every non-technical founder should ask for — that piece is the how of one rubric; this is the what and how many of the inputs the rubric grades. For the count-discipline read, see why your MVP should ship with 3 evals, not 30. For the cost consequence of skipping the work, see anatomy of a runaway AI project — 5 cost-side root causes.
The five properties of a launch test set
A launch test set is not a spreadsheet of prompts. It is a deliverable with five operational properties, each with a binary check the founder can run against the vendor or the internal team. If any property fails, the set is a draft and the MVP is not ready to ship.
1. Representative
The set mirrors the input distribution the feature will see in production. If production traffic is sixty percent NDAs, twenty-five percent vendor SOWs, and fifteen percent customer order forms, a representative 30-row set has roughly 18 NDAs, 7 SOWs, and 5 order forms. The mix is not the founder’s intuition; it is the actual distribution from real or pilot logs. Binary check: ask for the input-distribution table next to the test set and confirm the proportions match a documented source.
2. Hard
The set includes edge cases, adversarial inputs, and ambiguous prompts the team would rather not think about. Easy inputs do not discriminate between a good build and a bad one — every model passes them. The signal lives in the hard subset: the malformed PDF, the clause that looks like indemnification but is functionally a limitation of liability, the prompt that asks for something out of scope. Twenty to thirty percent of inputs should be hard. Binary check: ask the engineer to walk through one input and explain why it is hard.
3. Scored
Every input has a rubric-graded expected behavior. No input in the set has a grade of “I’ll know it when I see it.” A rubric anchors the grade with named criteria — clause coverage, risk-flag accuracy, redline format, refusal correctness — and language for each level. The output is a number on each criterion plus a pass / fail roll-up. Binary check: pick three rows at random and ask what score the system produced last run on each dimension.
4. Frozen
The set is locked before the baseline run and does not change after. Frozen does not mean static forever — the launch-eval edition is versioned and tagged in the repo, and any change opens a new version. Teams that edit the set after baseline are tuning the bar to the system instead of the system to the bar, and the resulting scores are not interpretable. Binary check: ask which git commit hash represents the launch version. “We update the spreadsheet” means the set is not frozen.
5. Replayable
The full set runs end-to-end in under ten minutes on standard developer hardware. A test set that takes an hour is one the team will not run on every prompt change, and the team that does not replay on every change ships regressions. The under-ten-minute target forces a small set, a fast harness, and a fast judge. Binary check: ask the engineer to run the set in front of you and time it.
Three sourcing approaches, used together
A launch test set is built from three sources, layered in a fixed order.
Founder-authored seed (10 inputs, week one). The founder writes ten canonical inputs by hand — the happy-path requests they would demo to an investor. This source encodes the founder’s mental model of the feature, which is the asset the vendor is being paid to build to.
Customer-sourced (15 inputs, week two). Pull anonymized inputs from a closed pilot, an existing product surface, or customer-development transcripts. These inputs encode the production distribution the founder’s head and synthetic data cannot reach. They almost always surface a behavior the founder did not anticipate, and that surprise is the source’s value.
Synthetic-augmented edge cases (5 inputs, week three). Use a frontier model — Claude Opus 4.8, GPT-5, or Gemini 2.5 Pro — to generate adversarial variations of the existing 25 inputs. Every synthetic input is founder-reviewed before it enters the set. Synthetic-only sets are a known failure mode; small-dose, reviewed augmentation is the workable compromise.
The mix is not negotiable. Founder-only is unrepresentative. Customer-only is under-adversarial. Synthetic-only is uncalibrated. Layered, the three cover the distribution, the long tail, and the edges.
The eight-step build sequence
The build is a fixed sequence. Skipping or reordering steps produces a set that fails one of the five properties.
Step 1 — Define the unit of work. Write one sentence that names what the AI feature does on a single input. “Given a contract PDF, extract the named clauses and flag risk levels.” Not “review contracts” — too broad. Not “extract section 4.3” — too narrow.
Step 2 — Draft the rubric. Write the rubric before the first input. Three to five named dimensions, each with anchored language for high, medium, and low scores. Drafting the rubric first prevents the set from drifting toward the inputs the team happens to have.
Step 3 — Seed 10 founder-authored inputs. Write the founder’s ten canonical inputs by hand. Number them. Note the expected behavior on each in one line.
Step 4 — Customer-source 15 inputs. Pull or recreate fifteen inputs from real data. Anonymize. Number sequentially after the founder seed.
Step 5 — Synthetic-augment to approximately 30 inputs. Use a frontier model to generate five adversarial variations. Founder reviews and accepts or rejects each. The set lands at twenty-eight to thirty-two inputs.
Step 6 — Calibrate the rubric across three graders. Three independent graders — typically founder, vendor lead, and a domain advisor — score the same five inputs. Where scores diverge, the rubric language is rewritten until the next round agrees within one point on every dimension. Without this step, the rubric is one person’s judgment dressed up as a measurement.
Step 7 — Freeze the set. Tag the set in version control. Commit the inputs, the rubric, the calibration notes, and the baseline scores. The frozen set is now the SOW deliverable.
Step 8 — Wire to CI replay. Add the set to CI so it runs on every prompt change, model swap, or retrieval-index update. The replay reports a diff against the last green run; regressions block the merge.
Eight steps, roughly fifteen to twenty engineering hours plus four to six founder hours.
Three anti-patterns that look fine until launch
The patterns below pass internal review and fail in production.
Anti-pattern 1 — Cherry-picked easy inputs. The set is thirty inputs the model already gets right. The score is high. Production traffic includes the long tail the set did not cover, and the regression rate climbs in week one of launch. The fix is the hard property — twenty to thirty percent of inputs should be inputs the team would rather not include.
Anti-pattern 2 — Unfrozen test set. The team keeps editing as the build progresses. An input that fails gets reworded “for clarity.” A rubric anchor that flags too many failures gets softened “to be fair.” The score trends up because the bar is moving, not because the system is improving. The fix is the frozen property — once baseline is committed, the set does not change until a tagged new edition is opened.
Anti-pattern 3 — Rubric without calibration. One person — usually the vendor lead — writes the rubric and scores every input. Two months in, a different engineer scores the set and gets numbers two to three points different on every dimension. The fix is step 6 — three graders calibrate against five inputs until they converge, and the rubric language is rewritten until calibration holds.
Worked example: a 30-input contract-review test set
A 30-input launch test set for a contract-review feature whose unit of work is “given a contract PDF, extract named clauses and flag risk levels.”
Input categories (sized by production distribution)
| Bucket | Count | Notes |
|---|---|---|
| NDA (mutual and one-way) | 9 | Most common; usually short |
| MSA (master service agreement) | 7 | Long, complex; high stakes |
| Employment offer | 5 | Boilerplate plus negotiated terms |
| Vendor SOW | 5 | Scope, deliverables, payment |
| Customer order form | 4 | Pricing, term, renewal |
Within each bucket, roughly twenty-five percent of inputs are hard — malformed PDFs, ambiguous clauses, mis-scanned pages, or out-of-scope requests the system should refuse.
Rubric dimensions
| Dimension | What it measures | Scale |
|---|---|---|
| Clause coverage | Did the system surface every named clause in the contract? | 0 — 5 |
| Risk-flag accuracy | Are the high / medium / low labels correct against a domain-expert key? | 0 — 5 |
| Redline format | Are the redlines in a paste-back format the user can apply? | 0 — 3 |
| Refusal correctness | Did the system refuse on out-of-scope inputs (non-contract docs, prompt-injection)? | 0 — 2 |
Maximum per input is 15. Launch threshold across the 30 inputs is 12.5 average with no input below 8.
Replay budget
Thirty inputs, each one a model call plus a model-as-judge grading call. With Claude Sonnet 4.5 as the system and Claude Opus 4.8 as the judge, parallelism at ten concurrent inputs, the suite replays in roughly six minutes — well under the ten-minute target. GPT-5 with Gemini 2.5 Pro as judge runs in the same range.
Hand this template to a vendor and say: build me the equivalent for my feature. The vendor either delivers or cannot, and the conversation stays concrete.
Frequently asked questions
What is an AI MVP eval test set? A small, fixed set of representative and adversarial inputs with rubric-graded expected behavior, frozen before launch and replayed on every prompt or model change. It is the artifact the team is graded against, not the runner the grades happen inside.
Why under ten minutes for the replay? Because anything longer is a set the team will not run on every change. Replay frequency requires speed; speed requires a small set, a fast harness, and a fast judge.
How is the launch test set different from the production eval suite? The launch set is small, frozen, and run on every change to gate the merge. The production suite is larger, evolving, and run weekly against sampled production traffic to surface drift. You ship the launch set first.
Can the test set be entirely synthetic? No. Synthetic inputs do not encode the production distribution and they do not surprise the founder. They are a useful augmentation — five out of thirty — but a synthetic-only set is a known failure mode.
Who writes the rubric — the founder or the vendor? The founder writes it in prose; the vendor formats it for the harness. The rubric is a domain artifact. A vendor who writes it alone is writing a rubric the model can pass, not one that protects the user. See the eval rubric template every non-technical founder should ask for.
How many inputs is the right number for an MVP? Roughly thirty for a single-capability feature. Smaller feels under-tested; larger blows the ten-minute replay budget. For multi-capability features, build one thirty-input set per capability and run them in parallel.
What if my feature is generative and the rubric feels subjective? Decompose harder. Even generative features have measurable sub-criteria — groundedness on the brief, locale correctness, length correctness, tone match against examples. Anchored language plus three-grader calibration gets the rubric to a defensible place.
How long does the eight-step build take? Three weeks elapsed. Roughly fifteen to twenty engineering hours plus four to six founder hours. Founder hours cluster in steps 1 to 3 and step 6; engineering hours in steps 4, 5, 7, and 8.
What goes in version control? The inputs, the rubric, the calibration notes, the baseline scores, and the runner configuration. The set is a code artifact. Spreadsheets do not survive version discipline; CSV plus YAML rubric plus runner config does.
How does this relate to the eval-first build playbook? The eval-first build playbook is the full nine-stage scoping workflow; this article unfolds stage 4 (“test-set construction”) into the operational deep dive.
Get the template, book the discovery call
If you want the worked example as a downloadable template — a CSV scaffold for the inputs, a YAML scaffold for the rubric, and a runner config for Promptfoo or Inspect — book a discovery call. The first call is scoped to one feature and walks the eight-step build against your inputs. You walk away with the four artifacts that turn an MVP launch from a wish into a deliverable: the frozen test set, the rubric, the calibration record, and the replay wiring.
The eval test set is a deliverable. Treat it like one and the MVP ships at quality. Skip it and you are paying for output you cannot grade.
Dirk Jan van Veen, PhD
