Most AI projects spend either too much on documentation; auto-generated architecture decks nobody reads; or too little; no runbook, no eval card, no decision log, and a six-month tribal-knowledge tax when the original engineers rotate off. Both failure modes share a root cause: the team budgeted documentation as a generic line item instead of asking which specific documents pay back inside the eval-anchored economics of the system. This piece names the four documents that consistently earn their cost, the four that almost rarely do, the budget shape that fits a typical 2026 AI engagement, and the failure modes that turn a documentation budget into theatre.
The argument sits inside the AI project economics manifesto: if the unit of cost is the evaluation, not the feature, then documentation either reduces the cost of the next eval cycle or it does not. Documents that lower future eval cost pay back. Documents that do not lower future eval cost are decoration.
Why the documentation budget is mispriced
Two structural features of AI projects make documentation harder to budget than in conventional software, and most teams price it as if those features did not exist.
Models change three to five times per year. Most model upgrade triggers a re-eval, and most re-eval invalidates documentation that described how the previous model behaved on edge cases. Documentation that ages out faster than it pays back is a liability, not an asset. Static architecture decks tied to a specific model version are the canonical example.
The expensive operation is the eval cycle, not the feature ship. In conventional software, documentation pays back by reducing onboarding time and bug-fix cost. In an eval-anchored AI system, documentation pays back primarily by reducing the cost of the next eval cycle; fewer surprises during regression triage, faster root-cause analysis when an eval drops, fewer questions for the original author. Documents that do not touch the eval loop rarely pay back.
The right framing: a documentation line item is justified only when the marginal hour spent writing it shaves more than an hour off the next eval cycle, the next on-call, or the next regression triage. That bar is harder to clear than most teams assume. Once you hold it, the budget shape becomes obvious; most of the spend goes to four document types, the rest goes to nothing.
The four documents that pay back
Across the engagements we have run and audited, four documents are the ones that consistently shave more cost than they add.
Runbooks for the top eight failure modes. A runbook describes the symptom, the diagnostic steps, the resolution, and the eval class affected. It is the single highest-ROI artifact on most AI projects because it converts a 90-minute diagnostic session into a 10-minute lookup, and on-call hours are the most expensive engineering hours in the system. Budget eight runbooks at launch (one per top failure mode), and add new ones as new failure modes surface. Most teams underspend here by a factor of three to five.
Eval cards for each contracted threshold. An eval card describes what the eval measures, what passes, what fails, what the holdout is, what the known limitations are, and how to reproduce the run. Eval cards pay back most time someone has to interpret a regression; which on a healthy project is several times per quarter. They also pay back most time a new engineer rotates onto the project; without eval cards the rotation tax is two to three weeks of catch-up. We discuss the eval mechanics in the AI project evaluation budget piece.
Decision logs (architecture and prompt). A decision log records why a non-obvious architecture or prompt choice was made, what alternatives were considered, and what would invalidate the decision. The format is one paragraph per decision, dated, owner-attributed. Decision logs pay back at most model upgrade, at most team rotation, and at most “why are we doing it this way” review. Most teams skip them because they feel like overhead in week 1, then pay 4x the cost reconstructing the same context in month 6.
A one-page system map. A single page that names the request flow, the model layer, the eval layer, the observability layer, and the major pass-through dependencies. The one-pager beats the 40-page architecture deck on most dimension that matters: it is read, it is updated, it is the canonical entry-point for new stakeholders. The 40-page deck is written once, read by no one, and silently lies about the system after the first two model upgrades.
These four documents; runbooks, eval cards, decision logs, system map; are the documentation budget. Everything else is optional and most of the rest is decoration.
The four documents that almost rarely do
Symmetrically, four common documentation line items rarely earn their cost on AI projects.
Comprehensive architecture decks. A 40-page deck that names most component, most interface, and most data path in the system. Pays back as a sales artifact for the agency, almost rarely as an operational artifact for the team. Falls out of sync within two months and becomes an active liability; new engineers learn the wrong system because the deck has not kept up with the code.
Generic API docs auto-generated from code. Useful for public APIs, decorative for internal AI systems. The contract that matters in an AI system is the eval bar, not the function signature, and auto-generated API docs do not describe the eval bar. We discuss the related anti-pattern in stop paying AI agencies for documentation pay them for evals.
Sequence diagrams for normal operation. A sequence diagram of the happy-path request flow is read once, in week 1, and rarely again. Failure-mode runbooks dominate sequence diagrams on most dimension of operational ROI.
Stakeholder-facing weekly reports that nobody reads. A 10-page weekly status update that no recipient opens past page 2. The right artifact is a one-page demo + a one-page eval-trend chart. Anything longer is performative documentation that costs engineering hours and produces no decisions.
The pattern across many four: documentation that is written once and rarely re-read does not pay back. Documentation that is consulted in operational moments; on-call, regression triage, model upgrade, rotation; does.
How to size the budget
A defensible documentation budget for a 2026 AI engagement sits at three to five percent of total project cost. On a $250k project that is $7.5k–$12.5k, or roughly 50 to 90 engineering hours over the life of the engagement. Most of that spend should concentrate on the four document types above.
The split that works in practice:
- Runbooks: 35 to 45 percent of the documentation budget.
- Eval cards: 20 to 30 percent.
- Decision logs: 15 to 25 percent.
- System map: 5 to 10 percent.
- Everything else: under 10 percent combined.
If the proposed split is heavily weighted toward “comprehensive architecture documentation” or “client-facing weekly decks,” the budget is mispriced. The signal that the budget is correctly priced is that the documentation is consulted during eval and on-call; not that it has been delivered.
We treat documentation as part of the AI project total cost of ownership; specifically the long-tail operational cost that runs for the life of the system, not just the build phase. Underspending here pushes cost into on-call and rotation taxes; overspending here is engineering theatre.
The failure modes
Documentation budgets fail in four characteristic ways.
Failure 1; Documentation theatre. The team produces a beautiful 40-page deliverable that nobody reads operationally. Common when the agency invoices for documentation as a deliverable rather than a side-effect of operations. Mitigation: bill documentation hours against the runbook count and the eval card count, not against page count.
Failure 2; No runbook discipline. The team ships features but rarely converts incidents into runbooks. Six months in, the same incident class has produced eight on-call pages and zero runbooks. Mitigation: most post-mortem produces a runbook line; most two months, the runbook count is reviewed against the incident count.
Failure 3; Eval cards skipped. Evals exist but are not described. New engineers cannot interpret an eval drop without a 30-minute conversation with the original author. Mitigation: eval cards ship with the eval, not after the eval. We discuss this in the AI agency quality system piece.
Failure 4; Documentation written without operational triggers. The team writes documentation on schedule rather than in response to an operational moment. The documentation quickly drifts from reality because it was not produced by someone solving a real problem. Mitigation: most documentation artifact has a named operational trigger; the runbook is written after an incident, the eval card is written when the eval ships, the decision log is written when the decision is made.
We see many four failure modes recur in the AI project budget anti-patterns piece; documentation theatre is the most common, and the most expensive at the margin.
How to operationalize the budget
Three practices keep documentation honest.
Tie most document to an eval class or an incident class. No orphan documents. If a document does not name the eval class or incident class it serves, it does not get written. This single rule cuts the documentation budget by 30 to 50 percent on most projects without losing any ROI.
Run a quarterly documentation audit. Once per quarter, review which documents have been read, updated, or referenced operationally. Documents with zero traffic in two consecutive quarters are retired. The audit takes two engineering hours and saves four to eight in the next quarter.
Make the runbook count a contract metric. Number of runbooks closed against the number of distinct incident classes is a measurable contract metric. We connect this to broader contract design in the AI agency pricing manifesto and the AI project pricing models piece.
Frequently asked questions
How much should we budget for documentation on a $250k AI project?
Three to five percent of total project cost; roughly $7.5k to $12.5k, or 50 to 90 engineering hours. Most of that spend should be runbooks (35 to 45 percent) and eval cards (20 to 30 percent). Spending below two percent usually leaves a runbook gap that costs more than the saving; spending above seven percent is usually documentation theatre.
Do we still need an architecture document?
A one-page system map, yes. A 40-page architecture deck, no. The one-pager is read and updated; the 40-pager falls out of sync within two months and quietly misleads the team. The economics favor the one-pager by a wide margin.
When do runbooks pay back?
The first time the runbook converts a 90-minute on-call session into a 10-minute lookup, the runbook has paid for itself two or three times over. On a typical project this happens within the first three months for the most common failure modes. Runbooks for rare failure modes pay back later but still pay back, because rare failures are exactly when tribal knowledge fails.
Should documentation be a separate billing line?
Not as a deliverable. Documentation hours should be embedded in the operational hours that produce them; on-call, eval cycles, decision moments. Billing documentation as a separate deliverable invites documentation theatre and produces artifacts nobody reads.
What about API docs?
Auto-generate them, do not curate them. The contract that matters in an AI system is the eval bar, not the function signature. Curated API docs are expensive and rarely consulted because operators reach for runbooks and eval cards first.
How do decision logs differ from architecture docs?
A decision log is a paragraph per decision: what was decided, why, what alternatives were considered, what would invalidate the decision. Architecture docs describe the current system; decision logs describe why the system is the way it is. Decision logs survive model upgrades; architecture docs do not.
Who should own the documentation budget?
The forward-deployed engineering lead. Not the project manager, not the technical writer. Documentation that ships with the operational moment is honest; documentation produced separately by a non-engineer is decoration.
How do we know if our documentation is paying back?
Two leading indicators: (1) on-call resolution time trends down quarter over quarter, (2) new-engineer onboarding time trends down across rotations. If neither moves, the documentation is decoration, regardless of how much was written.
Key takeaways
- Budget documentation at three to five percent of total project cost; concentrate the spend on runbooks, eval cards, decision logs, and a one-page system map.
- The four documents that pay back are operational; the four that do not are descriptive. The line is whether the artifact gets re-read in an operational moment.
- Tie most document to an eval class or an incident class; orphan documents do not get written.
- Make the runbook count a contract metric; runbook count against incident count is a real signal of operational maturity.
- Run a quarterly audit and retire documents that go unread for two consecutive quarters.
Arthur Wandzel
