How do I run this remotely when the vendor will not screen-share the artifact live?

Make the artifact a written precondition for the second meeting. After the first call, email a one-paragraph request for a redacted frozen eval set the vendor can walk through together — capability breakdown, case count, rubric structure, harness configuration. A vendor who has the artifact will produce it; a vendor who does not will either delay or quietly drop off. Both outcomes are diagnostic.

How to evaluate an AI vendor's eval discipline before signing

Q: Should the eval workstream be priced separately from the build?

Yes, on any build over 75,000 USD. A separate line item with fixed price, fixed case count, and named deliverables is the only way to ensure eval work survives schedule pressure. When evals are bundled into the build price, they are the first thing cut when the build runs hot.

The most expensive mistake a non-technical founder makes signing an AI vendor in 2026 is taking “we have evals” at face value. Six vendors will say the phrase. Three mean a serious workstream — frozen test sets, calibrated judges, regression suites gating deploys. Two mean a one-time spreadsheet reviewed by an intern. One means nothing at all. From the founder’s seat, in a 60-minute call, all six sound articulate. This article is the script that separates them.

This checklist extends the eval-first build playbook within the idea-to-product manifesto. It is built to be run, not read — six questions for a 60-minute first vendor call, what good and weak answers sound like, a three-archetype taxonomy, and one walk-away rule that overrides everything else.

Decision scope

This article is an editorial diligence framework, not legal, financial, security, or procurement advice. Treat the questions, calibrations, archetypes, and the walk-away rule as planning heuristics for a first vendor call. Calibrate against your own contracting model, regulatory context, and risk tolerance before signing any agreement.

Why eval discipline is the right diligence axis

A non-technical founder cannot evaluate the substantive quality of an AI vendor’s work — they cannot read the code, audit the prompts, or grade the LLM-as-judge harness. What they can do is interrogate the system the vendor uses to verify their own work — and that system is the eval suite.

Two 2026 shifts make this load-bearing. Frontier models update on the cadence of weeks — Anthropic shipped Claude Opus 4 through Opus 4.8 and the Sonnet 4.6 family; OpenAI shipped GPT-5 and successor updates; Google shipped the Gemini 2.5 line. Each transition silently changes the behavior of any system pinned to a model alias. The long tail is where the product is graded — McKinsey’s State of AI and BCG enterprise-AI surveys report roughly 85 percent of AI pilots stall before production. The diagnosis is rarely a model problem; it is a measurement problem.

The vendor without a frozen eval set rebuilds their understanding of “is this working” on every release. The vendor with an eval set but no judge calibration reports scores that may not correlate with real quality. The vendor without CI-gated regressions ships the next deploy on hope. A 60-minute script that surfaces which is true puts the buyer ahead of 90 percent of the market. The sibling field guide to evaluating an AI agency in under 90 minutes covers broader agency evaluation; this is the focused, eval-axis-only script.

The 6-question diligence script

Six questions, in this order, on a 60-minute call — roughly 8 minutes per question with 12 minutes buffer. The order is load-bearing: Q1 establishes whether they have a methodology; Q2 grounds it in a real engagement; Q3 stress-tests the threshold; Q4 surfaces measurement integrity; Q5 disarms survivorship bias; Q6 calibrates the judge.

Question 1 — “Show me your last eval set.”

Ask in the first 10 minutes. The vendor either has a redactable artifact they can walk you through under NDA, or they do not.

Strong answer: “Yes — under NDA, we can walk you through three prior engagements. The set has roughly 200 cases broken out by capability — entity extraction, summarization, refusal handling — with a per-case rubric and a structured input schema. Here is the harness we ran it in. Here is the directory structure on disk.”

Weak answer: “We do evals in every engagement” with no specifics. Or “we use Promptfoo / Inspect / Braintrust” — a tool name is not a methodology. Or “we can describe one in concept” without a concrete artifact, which is the answer of a vendor who has never built one.

What it tells you: whether the vendor has done this before, or whether your engagement is the one where they learn.

Question 2 — “What was the threshold you committed to in writing?”

The eval set is only diligence if there is a contractual quality bar tied to it. A set without a threshold is reporting infrastructure; a set with a threshold is a contract.

Strong answer: “On the most recent engagement, the contractual bar was 92 percent overall pass rate weighted by capability, with hard floors — refusals above 99, entity extraction above 90, summarization above 85. The threshold was negotiated against a first-week baseline. Milestone payment was tied to the suite passing.”

Weak answer: “We aim for high quality” or “we don’t usually put a number in the contract.” The vendor either commits to a number against a fixed set, or they have not agreed to a quality bar.

What it tells you: whether the vendor sells quality theater or contractual quality. The number itself matters less than the fact that one exists.

Question 3 — “How did you measure regression between releases?”

Once the threshold is set, the question becomes whether new releases stay above it. Regression measurement is the engineering practice that turns a one-time eval into a continuous quality bar.

Strong answer: “Every PR runs the frozen set on the changed code via a GitHub Action that posts pass-rate as a status check. Below threshold, the deploy is blocked. We also re-run on every model alias update from Anthropic, OpenAI, or Google — and treat that as a regression review even if no application code changed.”

Weak answer: “We re-run evals at the end of each sprint” or “we’ll set up CI later.” Manual gating fails the first time someone is under deadline pressure. Sprint-end gating misses every regression introduced mid-sprint.

What it tells you: whether the eval suite is load-bearing in deployment or decorative.

Question 4 — “Who wrote the rubric, and how do you know it is right?”

The rubric is where the suite reflects the product or drifts from it. A rubric written by an engineer alone, with no domain review, can grade outputs that are technically correct and commercially worthless.

Strong answer: “Co-authored. Our eval-engineering lead drafts a v1 from the spec; the client’s domain expert reviews and revises; we calibrate against 30 to 50 hand-labeled cases before freezing. Rubric edits invalidate the prior pass-rate, so changes are tracked and re-baselined.”

Weak answer: “Our team writes the rubric based on the requirements” with no client involvement. Or “the rubric is straightforward — output looks right or it does not.” A rubric written without domain input is the vendor grading themselves.

What it tells you: whether the eval reflects the product or reflects the vendor’s preferences.

Question 5 — “Show me an eval where you failed.”

This question disarms survivorship bias. Every vendor presentation is a highlight reel. The vendor who has done real eval work has failed real evals — caught regressions that hurt, missed regressions they wish they had caught, miscalibrated judges they had to throw out.

Strong answer: “On the engagement before last, our LLM-as-judge gave above-90 scores on a refusal capability where humans graded the same outputs in the 60s. We discovered it when the client flagged a production issue our suite had not caught. We re-calibrated against 100 hand-labeled cases, found the judge prompt was over-rewarding apparent compliance, and rewrote it. Pass rate dropped 18 points on re-baseline. The lesson reshaped how we calibrate every judge since.”

Weak answer: “We have not really had a failure like that” or a story about a client who did not listen. The vendor either has war stories that produce learning, or they have not been in the war.

What it tells you: whether the eval practice iterates from real failure or theorizes from a deck.

Question 6 — “How do you calibrate LLM-as-judge?”

The judge is the grader of the grader. Any vendor reporting eval scores generated by an LLM-as-judge is reporting numbers that may correlate with nothing real unless the judge has been calibrated against humans.

Strong answer: “For each judge, we hand-label 50 to 100 cases and run the judge on the same cases. We compute inter-rater agreement — Cohen’s kappa or percent agreement — and only trust the judge above a threshold (kappa above 0.6 or percent agreement above 85). If short, we revise the judge prompt and re-calibrate. We re-calibrate when the model behind the judge updates.”

Weak answer: “We use a strong model as the judge — GPT-5 or Claude Opus 4.8” with no calibration step. A judge not calibrated against humans is reporting numbers of unknown meaning.

What it tells you: whether the eval scores you will be shown weekly are real measurements or noise.

The 3 vendor archetypes

After running the six questions, vendors fall into one of three archetypes. Picking the right archetype matters as much as picking a strong individual vendor inside one.

Archetype 1 — eval-first

Methodology is published or documented in a way that survives the team. Frozen eval sets exist on prior engagements and can be walked through under NDA. Contractual quality thresholds are routine. Regression suites are CI-gated. Judges are calibrated against humans on every engagement. The team includes at least one engineer whose primary identity is eval engineering. Signal mix: strong on all six questions, concrete artifacts on at least four. Best for: builds where the quality bar is contractual — regulated industries (healthcare, finance, legal), consumer products where wrong outputs are user-visible, B2B SaaS where the AI feature is the moat. Trade-off: 20 to 25 percent of build budget on the eval line item. The argument is that the alternative is a quality cliff after launch.

Archetype 2 — eval-adjacent

Methodology exists in tribal form — senior engineers know how to build evals but it is not written down. Frozen sets exist on some prior engagements, not others. Contractual thresholds are negotiable but not standard. Regression is mostly manual; CI is sometimes wired. Judges are used but calibration is intermittent. Signal mix: strong on Q1 and Q3, uneven on Q4 and Q6, notable on Q5. Best for: lower-risk builds — internal tools, prototypes, B2B features where the user is forgiving. Trade-off: the founder must specify the eval workstream explicitly in the contract — case count, capabilities, threshold, calibration step, CI integration. Without that, the workstream defaults to tribal practice, which is uneven by definition.

Archetype 3 — eval-absent

“Evals” appears in marketing copy. No published methodology. No frozen sets to walk through. No contractual threshold language. Manual gating only. LLM-as-judge used without calibration, or not at all. Signal mix: weak on Q1, Q2, and Q6. Sometimes plausible on Q3 (“of course we re-run tests”) but unable to produce a CI configuration. Best for: nothing where the founder cannot independently verify quality. Acceptable only when the founder has internal engineering staff to run the eval workstream and the vendor is a pure body shop. Trade-off: hiring this archetype without internal staff to compensate is the procurement decision that lands at the 85-percent-of-pilots-stall statistic.

The walk-away rule

One rule overrides all six questions and all three archetypes.

If, within two weeks of asking, the vendor cannot show you — under NDA, in a screen-share — a frozen eval set from a prior engagement, walk away.

The artifact cannot be retroactively produced. A vendor who has built one before has the artifact, the harness, the rubric, and the war stories. A vendor who has not cannot fabricate a credible artifact in two weeks. Weak signals on individual questions are sometimes recoverable — methodology can get written down, calibration added, thresholds negotiated in. A frozen suite that does not exist cannot be conjured. Ask in the first call; if no screen-share happens within two weeks, you have your answer.

Two refinements. Redaction is fine — client names, prompts, real data can all be redacted. What you need to see is structure (capability breakdown, case count, rubric per case, harness configuration). The vendor who refuses to share even redacted artifacts is signaling absence, not confidentiality. Single-engagement is a yellow flag, not a red — one prior engagement they can walk through is more credible than three they can only describe. Three concrete artifacts is meaningfully stronger than one; the second and third tell you the practice repeats.

Sibling content on the broader buyer-diligence axis: evaluating LLM development companies and how to pick an AI development partner when you have never built software.

Scoring the call

A scoring grid produces a defensible rank across three candidate vendors:

Question	Strong (3)	Mixed (2)	Weak (1)	Absent (0)
Q1 — show me your last eval set	Concrete artifact, screen-share offered	Concrete description, artifact promised	Tool names only	Generic “we do evals”
Q2 — what was the threshold	Specific percentage, contract-tied	Specific percentage, informal	Range with no commitment	“We aim for high quality”
Q3 — how measured regression	CI-gated, blocks deploys	CI in some repos	Manual sprint-end	“We will set that up later”
Q4 — who wrote the rubric	Co-authored, drift-tracked	Vendor-drafted, client-reviewed	Vendor-only	“Rubric is straightforward”
Q5 — show me a failure	Specific story with quantified lesson	General story, vague lesson	Hypothetical	“Has not really happened”
Q6 — how calibrate judges	Cohen’s kappa, threshold-gated	Sampled human comparison	“We use a strong model”	Not addressed

Interpretation:

15 to 18: eval-first archetype. Default to hire pending references and pricing.
10 to 14: eval-adjacent. Hire only with explicit eval workstream in contract.
5 to 9: eval-absent with some performance. Hire only with internal engineering to compensate.
0 to 4: eval-absent. Walk.

The walk-away rule overrides the grid. A vendor scoring 17 who cannot produce a redacted artifact within two weeks scores zero on the override and is out regardless.

A complementary reference-call script for verifying answers with prior clients lives in the AI agency reference call — 11 questions that surface real client outcomes. Run the diligence script with the vendor first; run the reference-call script with two of their prior clients before signing.

Closing

A 60-minute first vendor call is not enough time to verify the substance of an AI build vendor’s work. It is exactly enough time to test the system the vendor uses to verify their own. The six questions are the script; the three archetypes are the verdicts; the walk-away rule is the override.

Running this script next week: schedule three candidate calls, run the same six questions on each, score the grid, ask each for the redacted artifact, then run the reference-call script with the two strongest candidates’ prior clients before any contract conversation. The whole cycle costs about 8 hours of founder time over two weeks. The downside protection against signing a 150K USD build with an eval-absent vendor is worth roughly 100x that.

For a second pair of eyes on a vendor proposal before signing, book an idea review with SFAI Labs.

Frequently asked questions

What exactly is “eval discipline” and why is it different from QA?

Eval discipline is the engineering practice of measuring whether an AI system produces correct outputs against a frozen, structured test set — with capability-level rubrics, calibrated LLM-as-judge graders, and CI-gated regression suites tied to a contractual quality threshold. Traditional QA tests deterministic software against expected outputs; eval discipline tests probabilistic AI outputs against a graded rubric. The skills overlap roughly 30 percent. A team strong at QA is not automatically strong at evals.

How long should a vendor need to produce a redacted frozen eval set under NDA?

Two weeks is the cap. Most eval-first vendors produce one in three to five business days because the artifacts exist on disk and only need screening. A vendor who needs more than two weeks is either rebuilding the artifact or asking client permission for the first time. Both are weak signals.

Is a published eval methodology required, or is internal documentation enough?

Internal documentation is the floor; published methodology is the ceiling. Tribal-knowledge methodology cannot survive team turnover or be inspected by a buyer. Eval-first vendors typically have a public document — blog post, conference talk, GitHub README — that another engineer could reproduce.

What if the vendor says “we use Promptfoo / Inspect / Braintrust — that’s our methodology”?

The tool is the harness, not the methodology. A harness without a frozen case set, a per-capability rubric, a calibrated judge, and a CI gate is just a runner. Ask “what cases are in the set, who wrote the rubric, how is the judge calibrated” — the answer tells you whether the tool is a discipline or a checkbox.

Should the eval workstream be priced separately from the build?

Yes, on any build over 75K USD. A separate line item with fixed price, fixed case count, and named deliverables is the only way to ensure eval work survives schedule pressure. When evals are bundled into the build price, they are the first thing cut when the build runs hot. Sibling guide how much does AI eval engineering cost on a fixed-price MVP covers line-item ranges.

What is Cohen’s kappa and why does it come up in Q6?

Cohen’s kappa is a statistical measure of inter-rater agreement that corrects for chance agreement. When calibrating an LLM-as-judge against human labels, kappa tells you whether the judge agrees with humans at a rate meaningfully above random matching. Above 0.6 is substantial agreement; above 0.8, near-perfect. Vendors who name kappa (or equivalent like F1 or percent agreement against a threshold) signal familiarity with calibration as a discipline.

Can I substitute one of the six questions if I am short on time?

Q5 (show me an eval where you failed) surfaces the most signal per minute, so it is the one to keep. Q1 produces the artifact request, so it is also non-negotiable. With only 30 minutes, run Q1, Q3, Q5, and Q6.

Make the artifact a written precondition for the second meeting. After the first call, email: “To move to the second meeting, please prepare a redacted frozen eval set from a prior engagement we can walk through together.” A vendor who has the artifact will produce it; a vendor who does not will either delay or quietly drop off. Both outcomes are diagnostic.

Does this script apply to AI consultancies and individual contractors, or only agencies?

It applies to any vendor whose deliverable is an AI system you cannot verify yourself. The question is whether the vendor has a system for verifying their own work you can interrogate. For consultancies offering an eval audit (rather than a build), the script inverts slightly — you are buying their methodology directly, so Q1 and Q4 carry more weight than Q3.

What if my candidate vendor is brand new and has never run a paid engagement before?

The walk-away rule applies more strictly, not less. A new vendor without prior artifacts cannot be assessed on this axis. Acceptable only if (a) the founders have eval reputations from prior employers — ex-Anthropic, ex-OpenAI, ex-Scale eval team — confirmed by references, or (b) you have internal engineering staff to own the eval workstream and the vendor is acting as a body shop.