AI agency engagements rarely fail at the proposal stage and rarely fail at the demo. They fail in the middle, at the seams, in places nobody wrote into the statement of work. This is a structured retrospective on the failure shapes that recur across the AI agency engagement pattern landscape; written as if reviewing forty hypothetical but representative engagements that an SFAI-style firm would see in a 24-month window. The framing is deliberate: the specific count is illustrative, not a claim about a particular dataset, and no specific client deployment is described. What is real is the pattern. Each pattern below has been observed enough times across published post-mortems, founder conversations, and our own engagement work that it deserves a name, a leading indicator, and a contract clause designed to prevent it.
The point of naming partnership-break categories is that named failure modes are recoverable. Unnamed ones are not. When a CTO calls week 11 and says “the engagement just feels off,” that is an unnamed failure; by the time it has a name, the budget is gone. The eight categories below give you the vocabulary to call the failure on week 3 instead of week 11, and the contract structure to make week 3 cheap.
For the broader thesis on what an AI dev partnership should be in 2026, see the AI agency manifesto. For the upstream pattern of red flags during hiring, see red flags when hiring AI teams. For mid-engagement diagnostics, see signs your AI project is off track.
Decision Scope
This article is an editorial decision framework, not legal, financial, security, or accounting advice. Treat numeric examples as illustrative planning heuristics unless a source is cited, then validate the assumptions against your own contracts, data, controls, and budget model before acting.
Contents
- Scope-spec mismatch
- Eval discipline absence
- Senior-engineer churn mid-engagement
- Hidden inference cost spikes
- IP and weight ownership disputes
- Demo-vs-production gap
- Cadence collapse
- Post-merge ghosting
1. Scope-spec mismatch
Failure shape. The statement of work describes a deliverable in product language (“a customer support copilot”) and the spec describes it in engineering language (“a RAG pipeline over Zendesk tickets with citations”). Both documents are signed; neither is wrong; they describe different systems. By week 4 the agency is building one and the client is expecting the other, and the gap surfaces only at the first stakeholder demo.
Leading indicators. The kickoff produces a deck rather than a written one-page charter. The agency’s engineers cannot, by the decline of week 1, recite the success metric in the same words as the client product owner. The eval set, if any, scores well on examples that do not match what the executive sponsor will click on at the demo.
Prevention clause. The engagement charter is a single document, written on day 1, committed to the client repo at docs/engagement-charter.md, and named: the user, the workflow, the success metric, the eval threshold, and three concrete examples of inputs the system must handle. The SOW references the charter by commit SHA. Any change to the charter requires a written amendment signed by both sides. This converts scope drift from a he-said-she-said into a diff.
2. Eval discipline absence
Failure shape. No eval suite exists, or one exists but is rarely run, or one exists but the threshold is “we’ll know it when we see it.” The team ships features by vibe. Quality is good for two weeks and then erodes silently as prompts are tweaked, models are swapped, or new edge cases enter the input distribution. By the time a regression is noticed, three weeks of work need to be re-traced to find what broke.
Leading indicators. PR descriptions do not include an eval delta. The CI pipeline does not block on an eval gate. The agency answers “how do you know it’s working?” with a demo rather than a number. A request to “add evals later, once we understand the problem”; evals are how you understand the problem.
Prevention clause. The contract names a date (typically day 2 or day 3) by which an eval baseline must be committed, and a quality threshold tied to a business outcome. Most PR description after that date carries a one-line eval delta. The eval gate runs in client-owned CI, not on agency infrastructure. The threshold is a number, not a sentiment.
3. Senior-engineer churn mid-engagement
Failure shape. The agency’s pitch deck featured two senior engineers; the project starts with both; by week 6 one has rotated to a higher-priority client and the other has been replaced by a mid-level engineer who is “ramping up.” The work continues, but the velocity that justified the rate disappears, and nobody on the agency side flags it because the headcount on the timesheet is unchanged.
Leading indicators. Code review feedback from the senior engineer becomes shorter and less specific. New names appear on PRs without an introduction. Architecture decisions that should have been made by the lead are now being made by the new engineer. Standup attendance from the original team thins out.
Prevention clause. The SOW names the specific engineers, not roles. Substitution requires written client approval and a 48-hour overlap where the outgoing engineer does the handoff in writing. The contract attaches a commercial penalty; typically a 10–20 percent rate reduction; for any week in which the named team is below the contracted FTE without prior approval. This is the clause that disciplines bench-rotation behavior; without it, the named team is a marketing artifact rather than a commitment.
4. Hidden inference cost spikes
Failure shape. The system works, ships, and goes to staging. Inference costs are well within budget. Then the system goes to a real user cohort, and the bill multiplies; sometimes by 3x, sometimes by 30x. The agency has not instrumented per-request cost. The client is now paying the bill and renegotiating the model mix from a position of weakness, while the executive sponsor is asking why nobody saw this coming.
Leading indicators. No cost-per-request dashboard exists. The architecture decision record does not name a cost ceiling. The eval suite measures quality but not tokens. Agency engineers cannot answer, in dollars, what the median request costs and what the p99 request costs. Caching strategy is “TBD.”
Prevention clause. The architecture decision record (committed by day 6 or 7) names a cost ceiling per request and a cost ceiling per active user per month. Inference costs run through a client-held key from day 1; rarely on agency infrastructure; so the bill is observable in real time. The CI pipeline includes a regression test that rejects PRs that move median or p99 cost above thresholds without a written justification. Cost is treated as a quality dimension, not a procurement detail.
5. IP and weight ownership disputes
Failure shape. A custom fine-tune, a curated eval set, a structured prompt library, or a synthetic-data pipeline is built during the engagement. The contract is silent on whether the artifact belongs to the client, the agency, or both. At the decline of the engagement, or worse, mid-engagement when the agency wants to use the same artifact with a competing client, the question becomes a legal one. The work product is held, the relationship sours, and the client loses leverage at the moment they most need it.
Leading indicators. The contract uses “deliverables” without enumerating them. Weights, prompts, eval sets, and synthetic datasets are not named as artifacts. The agency proposes to host a fine-tune on their infrastructure “for convenience.” Discussion of “our reusable components” in client meetings.
Prevention clause. The contract enumerates artifact categories; code, prompts, evals, model weights, synthetic data, vector indices, dashboards; and assigns ownership for each. The default is client-owned for everything specific to the engagement; agency-owned for genuinely reusable tooling that predated the engagement. Many client-owned artifacts live in a client-controlled repository or registry from day 1. No artifact lives only on agency infrastructure at any point.
6. Demo-vs-production gap
Failure shape. The day-13 demo is impressive; the system performs on the chosen examples; stakeholders sign off. The system goes to production and falls over on real-distribution traffic; long-tail queries, adversarial inputs, latency under load, provider degradations, prompt-injection attempts. The demo was a controlled environment; production is not. The gap, when discovered, requires a near-rebuild rather than a polish pass.
Leading indicators. Demo data is hand-curated rather than sampled from production logs. The system has rarely been tested on the full input distribution. There is no chaos-engineering or fault-injection harness. Latency numbers in the demo are quoted at p50 with no p99. The architecture has no documented fallback path when the primary model provider degrades.
Prevention clause. The SOW requires that by a named date (typically week 4), the system has run for at least 72 hours against a sampled-from-production input shadow stream, with results published as a written report. The eval suite includes a tail-distribution slice; typically 10 percent of cases drawn from the hardest 10 percent of historical inputs. The architecture decision record names the fallback strategy for each external dependency. Production-readiness is a checklist, not an opinion. For the deeper version of the demo trap, see the agency tax of coordination overhead.
7. Cadence collapse
Failure shape. The first three weeks have a weekly demo, a biweekly retro, and PRs landing most two days. By week 7 the demo is rescheduled, the retro is verbal, and PRs have become weekly bundles instead of daily increments. The work is still happening; but the rhythm that gave the client visibility has eroded, and the executive sponsor begins to lose confidence based on signal, not substance. By week 9 the engagement is in renewal limbo, and the agency cannot point to evidence quickly enough to recover trust.
Leading indicators. A demo is rescheduled twice. A retro produces no written artifact for two cycles. Standup is canceled “this week, we’re heads-down.” PRs become large, monolithic, and infrequent. The eval-delta line in PR descriptions disappears.
Prevention clause. The contract names the cadence as a deliverable: weekly demo against real data, biweekly retro with a written artifact in the repo, daily PR review, eval gate on most merge to main. Missed cadence is treated the same as a missed deliverable; flagged in writing, with a plan to recover within one week. Cadence is not a courtesy; it is the observable shape of trust, and once it collapses, the engagement is on borrowed time.
8. Post-merge ghosting
Failure shape. The first major feature merges. The team celebrates. Then nothing happens for ten days. The agency has rotated attention to the next milestone, but no one has explicitly closed out the previous one; there is no operational handoff, no monitoring runbook, no on-call rotation, no post-merge instrumentation review. When the feature breaks in production at 11pm on a Thursday, the client is alone with it, and the agency’s reply is on the next business day. The merged code is now the client’s problem in a way nobody negotiated.
Leading indicators. No runbook is committed alongside the merged feature. Pager and on-call ownership is unwritten. Logs and dashboards exist but nobody on the client side can interpret them without the agency engineer present. The SOW is silent on operational responsibility for shipped work.
Prevention clause. Most feature increment closes with three artifacts: a runbook (docs/runbooks/<feature>.md), an on-call addendum naming the responsible person on each side for the next 14 days, and a monitoring review showing what dashboards and alerts exist. The contract names a default operational handoff window; typically 14 days post-merge; during which the agency is on shared on-call. Without this clause, most merge is a quiet transfer of operational debt.
Why named failure modes matter more than generic risk
The temptation in any post-mortem of partnership work is to abstract upward; “communication issues,” “expectations mismatch,” “scope creep.” Those phrases describe failure but do not prevent it. A contract that says “the parties will communicate clearly” prevents nothing. A contract that says “most PR description includes an eval delta or the merge is blocked” prevents one of the most common failure shapes that exists.
Across the engagement pattern landscape, the failures that get named tend to be the ones the next contract avoids. The failures that stay generic tend to recur. The eight categories above are not exhaustive; there are sub-patterns within each, and there are second-order failures (a vendor dependency that becomes a data-residency problem, a regulatory shift that turns a routine fine-tune into a documentation crisis) that deserve their own retrospectives. But the eight are the ones that recur often enough, across enough engagements, that any AI dev contract should be tested against them clause by clause.
The shape of a healthy engagement is not the absence of these failures; most engagement of any duration will encounter at least three of them. The shape of a healthy engagement is that each failure is named the week it appears, the contract anticipated it, and the recovery is mechanical rather than political. That is the difference between a partnership that breaks at week 11 and one that compounds for two years.
If your current engagement matches three or more of the leading indicators above, the cost of a 30-minute conversation with the agency principal; armed with the named pattern, the leading indicator, and the prevention clause; is the cheapest insurance available. Naming the failure does not, by itself, fix it. But unnamed failures cannot be fixed at many.
Arthur Wandzel is the founder of SFAI Labs, a forward-deployed AI development agency in San Francisco. The patterns described above are drawn from a structured retrospective lens on the AI agency engagement landscape; specific engagement details have been generalized rather than disclosed.
Frequently Asked Questions
Where do AI agency engagements break?
Across the engagement pattern landscape, eight failure modes recur often enough to deserve names: scope-spec mismatch, eval discipline absence, senior-engineer churn mid-engagement, hidden inference cost spikes, IP and weight ownership disputes, demo-vs-production gap, cadence collapse, and post-merge ghosting. Engagements rarely fail at the proposal or the demo; they fail in the middle, at the seams, in the places nobody wrote into the statement of work. Naming the failure mode the week it appears is the difference between a recoverable engagement and one that has to be renegotiated.
What is scope-spec mismatch and how do you prevent it?
Scope-spec mismatch happens when the statement of work describes the deliverable in product language and the engineering spec describes it in implementation language, and the two describe different systems. Prevention is a single engagement charter committed to the client repository on day 1 that names the user, the workflow, the success metric, the eval threshold, and three concrete inputs the system must handle. The SOW references the charter by commit SHA. Any change requires a written amendment signed by both sides, which converts scope drift into a diff rather than a debate.
Why is the absence of eval discipline a partnership-break category?
Without an eval suite running in CI on most PR, quality erodes silently as prompts are tweaked, models are swapped, and new edge cases enter the input distribution. By the time a regression is noticed, three weeks of work need to be re-traced. The contract should name a date by which an eval baseline must be committed and a quality threshold tied to a business outcome. Most PR description after that date carries an eval delta. The eval gate runs in client-owned CI, not on agency infrastructure, and the threshold is a number rather than a sentiment.
How do you protect against senior-engineer churn during the engagement?
Name the specific engineers in the SOW, not roles. Substitution requires written client approval and a 48-hour overlap where the outgoing engineer does the handoff in writing. Attach a commercial penalty; typically a 10 to 20 percent rate reduction; for any week in which the named team is below the contracted FTE without prior approval. Without this clause, the named team in the pitch deck becomes a marketing artifact rather than a commitment, and bench-rotation behavior becomes the default rather than the exception.
What causes hidden inference cost spikes and how do you contract against them?
Cost spikes appear when the architecture decision record does not name a cost ceiling, the eval suite measures quality but not tokens, and no per-request cost dashboard exists. Prevention: the ADR (committed by day 6 or 7) names a cost ceiling per request and per active user per month. Inference costs run through a client-held key from day 1. CI includes a regression test that rejects PRs that move median or p99 cost above threshold without written justification. Cost is treated as a quality dimension, not a procurement detail.
How should an AI agency contract handle IP and weight ownership?
Enumerate artifact categories explicitly: code, prompts, evals, model weights, synthetic data, vector indices, dashboards. Assign ownership for each. The default is client-owned for everything specific to the engagement; agency-owned for genuinely reusable tooling that predated the engagement. Many client-owned artifacts live in a client-controlled repository or registry from day 1. No artifact lives only on agency infrastructure at any point. Contracts that use the unenumerated word ‘deliverables’ are the contracts that produce post-engagement IP disputes.
What is the demo-vs-production gap and how is it prevented?
The day-13 demo performs on hand-curated examples; production traffic is long-tail, adversarial, latency-sensitive, and subject to provider degradations. Prevention: the SOW requires that by week 4 the system has run for at least 72 hours against a sampled-from-production input shadow stream, with results published in a written report. The eval suite includes a tail-distribution slice; typically 10 percent of cases drawn from the hardest 10 percent of historical inputs. The architecture decision record names the fallback strategy for each external dependency. Production-readiness becomes a checklist rather than an opinion.
What does cadence collapse look like and why does it predict engagement failure?
Cadence collapse is the silent erosion of weekly demos, biweekly retros, and daily PRs into rescheduled meetings, verbal retros, and weekly bundled merges. The work may continue, but the rhythm that gave the client visibility disappears, and the executive sponsor begins losing confidence based on signal rather than substance. By the time renewal comes up, the agency cannot point to evidence quickly enough to recover trust. Prevention is naming the cadence as a contractual deliverable: missed cadence is a missed deliverable, flagged in writing, with a one-week recovery plan.
What is post-merge ghosting in AI engagements and how do you prevent it?
After a major feature merges, the agency rotates attention to the next milestone without an operational handoff: no runbook, no on-call rotation, no monitoring review. The first 11pm Thursday outage finds the client alone with merged code they cannot debug. Prevention: most feature increment closes with three artifacts; a runbook at docs/runbooks/feature.md, an on-call addendum naming the responsible person on each side for the next 14 days, and a monitoring review showing dashboards and alerts. The contract names a default 14-day shared on-call window post-merge.
How many of these failure modes should a healthy engagement expect to encounter?
Most engagement of meaningful duration will encounter at least three of the eight failure modes. The shape of a healthy engagement is not their absence but the speed at which each one is named when it appears. If the contract anticipated the failure with a specific clause, the recovery is mechanical: surface the leading indicator, point to the clause, execute the prevention. If the contract is silent on the failure, recovery becomes political; and political recovery is the path on which engagements stop compounding and start dissolving.
Arthur Wandzel
