DIY AI builds do not fail because the tool wrote bad code. They fail in six named ways, and the tool was complicit in five of them. Cursor, Claude Code, Lovable, Replit Agent, and v0.dev let a non-engineering founder ship a working prototype in a weekend. None of them prompt the founder to install an eval suite, write fallback paths, pin model checkpoints, wire observability, or assign a second pair of eyes. The prototype demos. The product breaks. The real mechanism, in every one of the six modes below, is a production discipline the founder treated as optional because the toolchain hid it.
It builds on the DIY-with-AI manifesto and the broader idea-to-product manifesto. Companion pieces the DIY AI MVP: 5 founder profiles and can I build an AI app with Claude Code cover who DIY works for.
Table of Contents
The 6 failure modes at a glance
Every DIY AI MVP failure we have post-mortemed maps to one of six modes. Skim the table. Find your symptom. Read the matching section.
| # | Failure mode | Symptom | One-line corrective |
|---|---|---|---|
| 1 | No eval set | “Worked on demo inputs; broke on a customer input I never tested.” | Scaffold an 80–150-row eval dataset in week 1 and gate every change behind it. |
| 2 | Hallucinated production | “I shipped the happy path. No error handling, no retry, no rate-limit guard.” | Run a 10-item production-readiness checklist before any customer touches it. |
| 3 | No fallback path | “LLM returns a 500, the whole product returns a 500.” | Document a degraded-mode response for every LLM-dependent path. |
| 4 | Model alias drift | “Prompts worked in March, silently degraded in May. No code changes.” | Pin to dated checkpoints, version prompts, run weekly regression evals. |
| 5 | No observability | “Customer complained about a response. I cannot find the conversation.” | Install Langfuse, Helicone, or PostHog LLM observability at week 1. |
| 6 | Over-trusted self-review | “Reviewed my own code with the agent that wrote it. A subtle bug shipped.” | Buy four hours per month of senior-reviewer time. |
The pattern is identical across all six: a production discipline senior engineers install instinctively, that the DIY toolchain does not prompt for, and that the founder treated as optional. Stack Overflow’s 2025 survey reports 76% of developers using AI coding tools daily (Stack Overflow 2025); GitHub Octoverse 2025 documents 20M-plus AI-assisted developers (GitHub Octoverse 2025). DIY shipping is at an all-time high. So is shipping into one of these six modes.
Mode 1: no eval set
Symptom. A customer reports a broken output. The founder cannot reproduce it. The retry works fine — or fails differently. The founder has no way to know whether the next fix lands.
Mechanism. The prototype was built against three to five hand-picked prompts. The product needs to work on the sixth — the one the customer typed at 4 a.m. with three typos and a domain noun. The “tests” the AI wrote are unit tests on wrappers around the LLM call, not eval tests on the LLM output. The framing is in the eval-first build playbook; the argument for buyers is in stop scoping AI projects in features.
What to do. Build an eval dataset of 80–150 representative inputs in week 1. Sources: customer interviews, beta-user logs, your own edge cases. Tag each row with expected behavior (target output, must-include, must-not-include). Run on every model change, prompt change, and refactor. Gate releases on no-regression. Half a day in Cursor or Claude Code. Highest-ROI hour in the whole project.
Mode 2: hallucinated production
Symptom. The product works on the demo and falls apart in front of a customer. Errors are not handled, edge cases throw 500s, the input form accepts whatever the customer types — including the string that crashes the LLM call. The founder thought they shipped a product; they shipped a demo with a domain name pointed at it.
Mechanism. The AI tool wrote the optimistic version of every code path. Asked to “build a chat interface that calls the OpenAI API,” it assumes the API succeeds, the user typed something reasonable, the network is up, and the model returned the format the parser expects. None of those hold in production. Senior engineers ask “what happens when this fails?” at every layer. The founder did not. This is the AI build trap in a different form: not over-built capability, but under-built reliability.
What to do. Before customer one, run a 10-item production-readiness pass. Each answers yes or the founder fixes it now:
- Every LLM call wrapped in try/except with a logged failure mode?
- Every external API call has an explicit timeout (10–15 s, not the SDK default)?
- Every user input validated before going into the prompt?
- Every prompt has a token-budget assertion (malicious input cannot blow the bill)?
- Every paid endpoint behind authentication?
- Every secret in an env var loaded at runtime, not in code?
- Every cost-bearing API call rate-limited per user and per day?
- Every database write inside a transaction?
- Every customer-visible error message human-readable (no stack traces)?
- Every deploy reversible (a
revertworks without a locked-in migration)?
Run Cursor or Claude Code against the checklist as a review pass. A day. Catches roughly 80% of mode-2 surface before customer one.
Mode 3: no fallback path
Symptom. Anthropic or OpenAI has a minor outage. Half a percent of calls return 500s for 45 minutes. Your product is unusable for 45 minutes. A customer complains on social and you discover your status page is your X timeline.
Mechanism. The LLM call is the load-bearing leg. No degraded mode. Senior engineers design at least one fallback — a cached response, a “high demand” message, a hand-off to a different model, a queue-and-retry pattern. The DIY founder skipped this because the tool did not prompt for it.
What to do. Two patterns cover most cases.
Pattern A — Cached fallback. If the call fails, return a stored “we couldn’t generate a custom response, here is the best stock answer for your question type.” Log every firing.
Pattern B — Two-vendor failover. If the primary (Claude Opus 4.8) fails three times in fifteen seconds, retry against a secondary (GPT-5 or Gemini 2.5 Pro). Both anthropic and openai SDKs are a one-line swap; both publish stable identifiers (Anthropic models, OpenAI models).
For most DIY projects, Pattern A is enough. Half a day. Before customer one.
Mode 4: model alias drift
Symptom. Prompts that worked in March silently produce worse results in May. No code changes. No prompt changes. The eval suite — if there is one — quietly drifts down 4% on a Tuesday. If there is no eval suite, customers find the regression first.
Mechanism. Model aliases like gpt-5 and claude-opus-4 are not stable pointers. They route to whichever underlying variant the vendor wants you on this week. When Anthropic ships an updated Sonnet, the claude-sonnet-4 alias may route to it without a code change. Vendors publish deprecation timelines for named checkpoints (Anthropic deprecation policy), but the alias moves silently inside that policy. Senior engineers pin to dated checkpoints (claude-opus-4-2026-03-15, gpt-5-2026-04-01). The DIY founder did not because the documentation example used the alias.
What to do. Four steps. (1) Pin to dated checkpoints in production; treat the alias as development-time only. (2) Version your prompts (prompts/summarize-v1.4.md). (3) Once mode 1 is fixed, run the eval suite on a weekly cron — a 3% drop is the canary, days before customers find it. (4) Every 4 to 8 weeks, re-pin to the latest stable checkpoint and re-run the suite. Promote if it passes; stay on the previous pin if it does not.
Cost: one hour per week. Benefit: knowing your quality is steady instead of discovering it has been degrading for a month.
Mode 5: no observability
Symptom. A customer escalates: “the bot told me to do something dangerous yesterday at 4 p.m.” The founder pulls up the codebase, the database, the API console. Nothing answers what the LLM actually returned to that user at 4 p.m. The founder apologizes, refunds, and cannot fix the underlying problem.
Mechanism. IDE logs (Cursor’s chat panel, Claude Code’s transcript file, the dev console) do not survive to production. The production system is stateless: prompt in, response out, no trace. Senior engineers wire LLM observability on day one. The DIY founder did not because no AI coding tool prompts for it. By the time the founder realizes they need observability, they need it retroactively, and that does not exist.
What to do. Install LLM observability at week 1, before customer one.
- Langfuse (langfuse.com) — open source, generous free tier, prompt management built in. Default DIY pick.
- PostHog LLM observability — if you already use PostHog for product analytics, both co-exist.
- Helicone (helicone.ai) — proxy-based, drop-in for the OpenAI / Anthropic SDK.
Wire four things: every prompt and response logged with session ID, user ID, timestamp, model identifier; latency and token cost per call; a web UI to find any user’s session in thirty seconds; alerts on cost spikes (a runaway loop can drain $400 in a weekend).
Mode 6: over-trusted self-review
Symptom. A subtle correctness bug ships. A customer finds it. The founder traces it to a code path written with Claude Code or Cursor, reviewed once, approved. The tests pass. The bug is in a corner case the founder did not test, the AI did not test, and the tests the AI wrote pass for the wrong reason — they test against the agent’s understanding of the spec, which is the founder’s incorrect understanding.
Mechanism. The human-loop failure. The founder is designer, engineer, QA, and reviewer — all at once. Both halves of the loop share the same priors. When the founder is wrong, the AI reinforces the mistake. Senior teams solve this with review by a second human. AI tools imply they replace the second reviewer. They replace the junior engineer, not the senior reviewer.
What to do. Buy four hours per month of senior-reviewer time. The reviewer reads your codebase, runs your eval suite, pulls production traces, and writes a report. Cost: $400 to $2,000/month. The cost-side view is in anatomy of a runaway AI project.
Sources: a staff-engineer friend (trade review for product feedback); a part-time contractor via Upwork or Toptal ($200/hour, 4 hours, $800/month); an agency on retainer for senior review only.
The corrective sequence — week by week
| Week | Mode | Deliverable | Cost |
|---|---|---|---|
| 1 | 1 (eval set) | 80–150-row eval dataset + a pnpm eval or make eval command. |
1 day + $0. |
| 1 | 5 (observability) | Langfuse, PostHog LLM, or Helicone installed; every call logged. | 0.5 day + $0–$50/mo. |
| 2 | 4 (alias drift) | Production code pinned to dated checkpoints. Prompts versioned. Weekly eval cron. | 0.5 day + $0. |
| 2 | 2 (hallucinated production) | 10-item production-readiness checklist run; all items resolved. | 1 day + $0. |
| 3 | 3 (fallback) | Pattern A (cached) fallback for every LLM-dependent path. | 0.5–1 day + $0. |
| Ongoing | 6 (self-review) | Senior reviewer engaged for 4 hours/month before customer 10. | $400–$2,000/mo. |
Total: 4.5 founder-days plus $400–$2,000/month for a senior reviewer. Below one mid-tier consultant’s day rate. Skip and the modes find the project anyway — later, in front of paying customers, at 20 to 50 times the prevention price.
What to do next
- Download the AI MVP Scoping Worksheet. Turns the six modes into pre-build constraints. Catches failures at scope time, not customer time.
- Audit your current project. Walk the six modes. Pick the top unfixed one. Fix it this week.
- Read the wider frame. The DIY-with-AI manifesto covers when DIY is the right path.
- Match the right founder profile. The DIY AI MVP: 5 founder profiles describes which shapes converge on DIY success.
The wrong move is to read about exotic failure modes (prompt injection, data leakage) while the six basics are unfixed. The right move is to fix mode 1 this week.
Frequently Asked Questions
Which of the 6 failure modes hits DIY AI MVPs first? Mode 1 (no eval set), inside the first two weeks of customer use. The prototype demos cleanly on hand-picked inputs, then breaks on the first input outside that distribution. Mode 5 (no observability) hits second — the founder cannot diagnose mode 1 without it. Modes 2, 3, 4, and 6 emerge over the following one to three months.
How much time should I budget for fixing all 6 modes? 4.5 founder-days for modes 1 through 5, plus 4 hours per month of senior reviewer. One work week of investment plus $400–$2,000 per month. Below one mid-market consultant’s day rate. Skipping is 20 to 50 times more expensive when the failures surface in production.
Can I skip mode 6 if I am a careful, technical founder? Even technical founders ship mode-6 bugs — solo review against AI-generated code shares the same priors as the code. Skip-bar: staff-level fluency in the stack, a passing eval suite, observability already in place. For most DIY founders, mode 6 is non-optional.
Are these failure modes specific to Claude Code, or do they apply to Cursor and Lovable too? Every DIY AI build tool — Cursor, Claude Code, Lovable, Replit Agent, v0.dev, Bubble + AI. The tool is not the variable. The variable is the production discipline the founder skipped.
What is the cheapest single improvement I can make this week? Install Langfuse or PostHog LLM observability (mode 5). Half-day install, free tier covers pre-revenue. Pair with a 30-row eval dataset (mode 1) and the two cover the most acute failure surface for under a day of work.
Do these failure modes go away with better models — Claude Opus 4.8, GPT-5, Gemini 2.5 Pro? No. Better models reduce mode 1 at the margin but do not change the structural problem (no eval suite). Modes 2 through 6 are model-independent — they are about the system around the model.
When should I stop DIY-ing and hire an agency or contractor? Three signals: one of the six modes has bitten you twice and you cannot install the fix; 100-plus paying customers or 10-plus paying teams; more than 30% of your week on engineering instead of selling or building product. Any one is the hand-off prompt.
Is there a worksheet that turns these 6 modes into pre-build constraints? Yes — the AI MVP Scoping Worksheet. Each mode maps to a section that asks “have you scoped this in?” Catches failures at scope time. Free.
Arthur Wandzel
