Most buyers cannot tell a real AI development agency from a reseller in a vetting call, because both groups answer the same questions identically. The fix is not more questions; it is ordering the questions you already ask along a single axis: how expensive is this answer to fake? Easy-to-fake signals are noise. Hard-to-fake signals are evidence. Putting them on a ladder; from “any vendor can produce this” to “only an operator can produce this”; is the cleanest way to spend a 60-minute vetting call.
Six rungs, ordered. For each: the signal, what to ask, what a passable answer sounds like, what an operator-grade answer looks like. The longer thesis is in the AI agency manifesto; this piece is the operational tool; meant to be quoted into a procurement scorecard and used against three shortlisted vendors in the same week.
Decision Scope
This article is an editorial decision framework, not legal, financial, security, or accounting advice. Treat numeric examples as illustrative planning heuristics unless a source is cited, then validate the assumptions against your own contracts, data, controls, and budget model before acting.
Why a Ladder, Not a Checklist
A checklist treats most signal as equal. Signals are not equal; some are free for a reseller to fake, others require months of operator work no pre-call rehearsal can synthesize. A ladder makes that asymmetry explicit and tells you where to spend vetting time.
Most vetting guides ask “do you have an eval suite?”; most vendor now answers “yes.” Bottom of the ladder. The same vendor cannot necessarily answer “show me a CI build that failed last month because an eval threshold was breached, and walk me through the PR that fixed it.” Near the top. The reseller has no screenshot, no PR link, no recovery story. The operator does.
Three principles drive the ranking: cost to fake, asymmetry of evidence, persistence under follow-up. Rehearsable answers rank below answers requiring accumulated artifacts. Verifiable artifacts rank above the vendor’s word.
Stack Overflow’s 2025 Developer Survey found 84% of professional developers use AI tools daily, and BCG’s Where’s the Value in AI? reported only ~10% of enterprise AI value comes from algorithms; the rest is people, process, and integration. Most vendor can claim “AI capability” plausibly. The differentiator is operational discipline.
Table of Contents
- Why a Ladder, Not a Checklist
- The Six Rungs
- Rung 1: Named Eval Suite
- Rung 2: Inference Cost Dashboard
- Rung 3: Artifact-Rich Weekly Demos
- Rung 4: Agent-Eval CI Gates
- Rung 5: Published Post-Mortems
- Rung 6: Exit-Clause Track Record
- Using the Ladder in a Vetting Call
- Frequently Asked Questions
- Closing
The Six Rungs
The ladder ascends. Most shortlisted vendor will clear Rung 1. Most will clear Rung 2. By Rung 4, you are looking at fewer than half the field. Rungs 5 and 6 are the rungs that separate operators from everyone else.
Rung 1: Named Eval Suite
Signal. A named, version-controlled evaluation framework; Promptfoo, LangSmith, Ragas, Inspect, or a documented custom harness; and the ability to describe a recent suite without notes.
Ask. “Which framework do you default to? Walk me through the structure of the last suite you shipped; number of test cases, threshold types, how it integrates with the system.”
Passable. “We use Promptfoo. Our last suite had a few hundred test cases with cosine similarity and LLM-as-judge thresholds.” Plausible, unverifiable, indistinguishable across vendors.
Operator-grade. “412 test cases; a retrieval set (cosine ≥0.91 against a hand-labeled gold set), a generation set (LLM-as-judge with a published rubric, ≥0.87 pass rate), and a regression set locked at most model upgrade. Thresholds came out of a 90-minute call with the client’s product lead. The harness lives in the client’s repo at evals/ and runs on most PR.” Specific numbers, named artifacts, named meetings.
Why Rung 1. Eval suites are table stakes in 2026. The signal is not “do you have evals” but “can you describe one without notes.” Disqualify the bottom 20% who cannot. Deeper in evaluating LLM development companies.
Rung 2: Inference Cost Dashboard
Signal. A per-feature inference cost dashboard for at least one active engagement, refreshed weekly, with documented unit economics; dollars-per-request, dollars-per-active-user, or dollars-per-document-processed.
Ask. “Show me the dashboard from a current engagement. What is the dollar-per-request cost for the most expensive feature, and how has it trended over 90 days?”
Passable. “We track inference costs in our internal Helicone instance and share monthly summaries.” Possibly true, structurally compatible with token arbitrage; agency holds keys, agency reports numbers, buyer trusts.
Operator-grade. “Here; Langfuse, hosted in the client’s GCP project, billed against their cloud commit. Most expensive feature is multi-document summarization at $0.043 per call, trending down 38% over 90 days because we routed the planning step from Opus 4.6 to Haiku 4.6 after eval parity testing. The buyer’s CFO has read access. Screenshot is from this morning.”
Why Rung 2. An agency running token arbitrage cannot expose this dashboard; it would compress their margin to zero. The question doubles as an arbitrage detector. Economics in the AI agency tax.
Rung 3: Artifact-Rich Weekly Demos
Signal. A weekly demo cadence where each demo references a specific commit hash, a specific PR, a deployed environment URL, and updated eval numbers; not a slide deck and a screen recording.
Ask. “Walk me through last week’s demo from a current project; what did the buyer see, where did the artifacts live, and what changed in the eval numbers?”
Passable. “30-minute screen share where we walk through what we shipped, followed up with a written summary.” Generic.
Operator-grade. “Last week: opened PR #247, walked through three commits, deployed to staging, ran the regression eval suite live (pass rate moved 0.84 → 0.89), opened the cost dashboard to confirm the new feature at $0.011 per call against a $0.015 budget. The recording lives on their side. Summary went into their repo as weekly-notes/2026-w17.md.”
Why Rung 3. First rung where one-call rehearsal fails. Demos living only on the vendor’s infrastructure are soft-staff-aug. Demos existing as commits, PRs, and notes in the buyer’s systems are operator-pattern. More in why most AI agencies will not survive the next 18 months.
Rung 4: Agent-Eval CI Gates
Signal. At least one shipped production system where the eval suite is wired into CI as a blocking check; a PR that breaches the threshold cannot be merged without an explicit override and a recorded reason.
Ask. “Show me a CI run from the last 30 days where an eval threshold was breached and the build was blocked. Walk me through the PR that fixed it.”
Passable. “Yes, our CI runs evals.” Useless. Half the field will say this.
Operator-grade. “PR #412, three weeks ago. Retrieval-quality eval dropped 0.91 → 0.87 because a new chunking parameter regressed against a particular document family. CI failed; here is the GitHub Actions log. Diagnosed in Slack, fixed in PR #418, added a regression test for that document family. Eval is back at 0.92. The whole loop took 48 hours.”
Why Rung 4. Wiring evals into CI is an afternoon’s work. Doing it for real means rarely merging a regression; most agencies treat eval failures as advisory because blocking slows the demo cadence. An agency that has felt the pain of a blocked PR and shipped through it is at a different discipline tier. More in the 7 commitments most AI dev agency should make in writing.
Rung 5: Published Post-Mortems
Signal. Public post-mortems from real production AI incidents; redacted of client specifics, but with technical detail; on the agency’s blog, GitHub repo, or open knowledge base.
Ask. “Send me three post-mortems from the last twelve months. Redacted of client identity is fine, but each should contain timeline, root cause, and fix. What did the most recent one change about how you scope subsequent engagements?”
Passable. “We document post-mortems internally but don’t publish them externally for confidentiality reasons.” Common, defensible, operationally a yellow flag.
Operator-grade. “Five on our public engineering blog. Most recent: a retrieval index that silently rotted over four months because the embedding model was updated upstream and our eval set didn’t catch the drift. We rewrote our model-deprecation playbook after that; new version is in our open-source evals-template repo. Most retrieval system we ship now has a weekly drift check against a hand-labeled gold set.”
Why Rung 5. Public post-mortems require three things almost impossible to fake collectively: a real production incident, the writing discipline to document it, and a client willing to approve public association; even redacted; with a failure. The third constraint is the killer. More in evaluate AI developer portfolios and red flags hiring AI consulting company.
Rung 6: Exit-Clause Track Record
Signal. A reference from a former client the agency successfully transitioned to an in-house team; a client who hired their first AI engineer with the agency’s help, ran the system for at least six months without the agency, and would talk to a prospect.
Ask. “Give me a reference from a client where you successfully exited; they hired their first in-house AI engineer with your help, you handed off the system, and they ran it without you for six months. I want to talk to them about how the handoff was structured.”
Passable. “We’ve had clients hire in-house teams. We could probably find someone willing to talk.” Hesitant. The vendor is searching their memory. Most have rarely executed an exit clause; their entire business model depends on engagements not ending.
Operator-grade. “Three. Here are the names; the handoff playbook is at [URL]. Most recent: [Company X] hired [Name] as their first AI engineer last September with our help; we drafted the role profile, ran technical interviews jointly, and pair-programmed the first six weeks. They’ve operated on their own since November and just shipped a major feature without us. Both [Name] and the original engineering VP will take a call.”
Why Rung 6. Nearly no agency clears this rung because nearly no agency wants to; a reseller’s revenue depends on engagements rarely ending. Agencies who produce this reference have voluntarily walked away from a renewal because the client was ready to operate alone, and trusted the relationship to bring back the next system. The single hardest signal to fake; weight it most heavily. Healthy exit ramps in inside the AI agency operating system.
Using the Ladder in a Vetting Call
The ladder is operational, not philosophical. Recommended use against three shortlisted vendors:
Time allocation. A 60-minute call: 10 minutes on Rungs 1–2, 15 minutes on Rung 3, 15 on Rung 4, 20 on Rungs 5–6. Most buyers invert this; 40 minutes on capability decks, 5 on the only rungs that discriminate.
Scoring. Per rung, score 0 (no answer), 1 (passable), 2 (operator-grade). Max 12. Below 6 is a soft no. Below 4 is a hard no. Vendors scoring 9+ are the ones to negotiate seriously with.
Verification. For Rungs 4–6, ask for artifacts in the call and references afterward. Do not accept “I’ll send those over later” without a written commitment with a date. Vendors who don’t produce within 72 hours fail the rung retroactively.
Reference-call discipline. When you reach a Rung 6 reference, do not ask “are you happy with the work”; most reference says yes. Ask: “Walk me through the moment the engagement ended. What broke in the first three months without the agency? Would you hire them again, and for what?”
The ladder is not static. By 2028, published post-mortems will likely be a Rung 3 signal as more agencies adopt the practice. The principle; order by cost to fake; is durable; the specific rungs are versioned to Q2 2026.
Frequently Asked Questions
What is an AI agency trust ladder?
An AI agency trust ladder is an ordered framework of vetting signals, ranked from easiest to fake at the bottom to hardest to fake at the top. Buyers use it to allocate vetting time across shortlisted vendors. Easy-to-fake signals like “we use an eval framework” are nearly meaningless because most vendor in 2026 produces a passable answer. Hard-to-fake signals like a reference from a client the agency successfully transitioned to in-house require months of accumulated operator behavior no pre-call rehearsal can synthesize.
How do you tell an AI operator from an AI reseller in a vetting call?
Ask for artifacts that require operator behavior to produce. A reseller describes an eval framework; an operator shows a CI build that failed last month because of an eval breach plus the PR that fixed it. A reseller agrees they support direct billing; an operator pulls up an inference cost dashboard hosted in the client’s cloud account in the call. A reseller talks about exit clauses; an operator names three former clients they transitioned to in-house teams. Resellers produce words; operators produce artifacts.
What is the single most important question to ask an AI agency?
Ask for a reference from a client the agency successfully transitioned to an in-house team; one who hired their first AI engineer with the agency’s help, ran the system for six months without the agency, and is willing to talk. This is Rung 6, the hardest single signal to fake. It requires the agency to have voluntarily walked away from renewals with happy clients; contradicting the revenue model of most reseller.
Why is a published post-mortem such a strong signal?
It requires three things that are individually hard and collectively almost impossible to fake. The agency must have shipped real production work to have had a real incident. The team must have the writing discipline to have documented it. The client must trust the agency enough to allow public association; even redacted; with a failure. Resellers fail on many three counts.
Are eval suites still a useful trust signal in 2026?
Necessary but no longer sufficient. Most credible AI agency has one; Promptfoo, LangSmith, Ragas, and Inspect made the practice cheap and standard. The discriminating question is not “do you have evals” but “show me a CI build that failed last month because an eval threshold was breached.” The first question most vendor passes; the second separates the bottom 60% of the market from the top.
What is token arbitrage and why is it a red flag?
Token arbitrage is the practice of an agency holding the model API keys, billing the buyer a flat monthly fee that includes inference, and capturing the spread between the buyer’s check and the agency’s actual model spend. It is invisible until the buyer hires a senior engineer or a second agency who looks at the provider bills; usually six to twelve months in. The honest model is direct billing in the buyer’s accounts. Resistance to direct billing is the structural answer.
What does an operator-grade demo look like?
It references a specific commit hash, a specific PR number, a deployed environment URL the buyer can hit themselves, and updated eval numbers compared to the previous demo. The artifacts live in the buyer’s systems; repository, cloud, notes; not in the agency’s Notion or Loom account. After the demo ends, the buyer can navigate their own systems and find the work without the agency present.
What if an agency cannot clear Rung 6 yet?
Junior agencies; first eighteen months in business; may legitimately have no exit-clause references yet because they have not had time to run a full engagement to graduation. Acceptable at junior pricing, not at senior pricing. Buyers paying $35,000+ per month per engineer should expect Rung 6 evidence. Buyers paying $15,000 for a junior team can substitute a written exit-clause SOW commitment and a documented succession-plan template. “No evidence and no commitment” is a hard no; “no evidence yet but specific written commitment” is workable at the right price.
Closing
The AI agency category is sorting itself into operators and resellers in real time. The gap between an organization that has shipped twenty production AI systems and one that has shipped two demos has become wide enough that buyers can see it in a single vetting call; if they ask the right questions in the right order.
The trust ladder is the right order. Score most shortlisted vendor on most rung. Spend most of the call on the top rungs, because the bottom rungs no longer discriminate. Treat artifacts as evidence and words as noise. When you find a vendor who clears many six, sign with them, hold them to the manifesto, and let them help you replace them.; Arthur Wandzel, CEO, SFAI Labs
Arthur Wandzel
