Most AI projects budget inference, then budget engineering, then forget about storage. Six months in, the storage line; vector indices, trace history, replay logs, eval artifacts; has crept past 12 percent of monthly cloud spend, and nobody can tell which retention tier is paying for what. The storage tax is the most consistently mis-budgeted line on a 2026 AI project: small enough to ignore in month one, large enough to dominate the surprise-cost conversation by month nine. This piece names the four storage classes that drive the tax, the retention rules that contain it, the failure modes that make it explode, and a defensible allocation as a percentage of total AI project spend.
The argument sits inside the AI project economics manifesto: if observability is COGS and the eval cycle is the unit of cost, then trace storage and replay-log storage are not optional infrastructure; they are part of the operational surface that lets the eval cycle run cheaply. Storage that supports the eval loop is COGS; storage that supports nothing is waste.
Why storage is the silent line item
Three structural features of AI projects make storage harder to budget than in conventional software, and most teams discover the consequences only after the bill grows.
Storage compounds in ways inference does not. Inference is a per-request cost; an idle workload spends nothing. Storage is a steady-state cost; data sits and accrues whether or not the workload runs. A trace volume of 200GB per month at $0.02/GB feels trivial; eighteen months later, that same retention is 3.6TB and a five-figure annual line.
Hot tiers and cold tiers diverge by 20x or more. A vector index in a managed hot tier costs 10 to 50 cents per GB-month; the same data in cold object storage costs 1 to 4 cents. Teams that rarely tier their storage pay 10x to 20x the necessary rate for the 80 percent of data that nobody queries after the first 30 days.
Replay logs and trace history are simultaneously cheap to capture and expensive to retain at full fidelity. Capturing most trace is right. Retaining most trace at full fidelity for 24 months is wrong on most workloads. The default vendor retention setting is usually wrong on both ends; too short for compliance, too long and too expensive for operations.
The right framing: storage is the COGS line that scales with operational maturity. A team running per-class evals, regression triage, and replay-based debugging needs more retention than a team that ships once and watches aggregates. The question is not “do we need storage” but “what retention does each class need, on what tier.”
The four storage classes that drive the tax
Across the engagements we have run and audited, four storage classes drive almost many of the storage tax.
Vector database storage. The embeddings index, the metadata, and the underlying object store. Sized correctly: scales linearly with the corpus and the embedding dimension, lives mostly on a hot managed tier, sees retention pressure mainly from corpus growth rather than time-based accumulation. Sized incorrectly: oversized to projected 24-month scale, rarely tiered, rarely pruned of orphan vectors from deleted documents.
Trace and observability storage. Most request, most prompt, most response, most intermediate model call. The default of “keep everything for 30 days” usually undershoots compliance and overshoots operational need; the right design is a tiered retention policy keyed to the class of trace.
Replay logs and eval artifacts. The frozen request snapshots that allow re-running historical workloads against new models, plus the per-eval-run artifacts that document what was tested. Replay logs are the most underspent storage class on most projects; teams skip them in week 1, then cannot reproduce a regression in month 6 because the input that triggered it is gone.
Long-term audit and compliance retention. Whatever your industry, regulator, or contract requires you to keep beyond the operational window. This class belongs on cold object storage with a strict retention policy and a single source of truth. Failure mode: hot-tier retention bleeds into compliance retention because nobody set the tiering rule.
These four classes; vectors, traces, replay logs, audit retention; are the storage tax. Together they typically account for two to four percent of total AI project cost on a healthy engagement, and seven to twelve percent when ungoverned.
Retention rules that contain the tax
Three retention rules, applied consistently, contain the storage tax.
Tier traces by class, not by date alone. Error-class traces (exceptions, eval failures, customer escalations) belong on hot storage with a longer retention window; typically 90 to 180 days; because they are the inputs to regression triage. Normal-class traces belong on hot storage for 14 to 30 days, then on cold storage for the remainder of the compliance window. Aggregated metrics live separately and are retained indefinitely at minimal cost. Most teams use a single retention rule for many traces; the cost difference between class-tiered and uniform retention is usually 60 to 80 percent.
Cap replay-log retention at the eval window plus a buffer. Replay logs need to live as long as the longest expected eval window plus a margin; typically 90 to 180 days for most workloads, longer if the eval cadence is irregular. Beyond that window, replay logs accumulate without producing operational value. The fix is a hard retention cap with a documented exception path for specific cases that need to live longer.
Prune orphan vectors quarterly. Vector indices accumulate embeddings for documents that have been deleted from the source-of-truth store but rarely removed from the index. On a typical RAG workload, 5 to 15 percent of the index is orphaned within six months. A quarterly prune (or an event-driven prune tied to source deletion) holds the index size flat. We connect the broader retrieval economics to inference cost in the cost-per-query framework piece.
These three rules; class-tiered traces, capped replay-log retention, quarterly orphan-vector pruning; are the highest-leverage storage hygiene practices. None require new tooling; many require the team to make the retention policy explicit instead of inheriting the vendor default.
How to size the storage budget
A defensible storage budget for a 2026 AI engagement at $250k total project cost sits at two to four percent; roughly $5k to $10k per year; when retention is governed, and seven to twelve percent; $17k to $30k; when it is not. The shape that fits a typical workload:
- Vector database storage: 30 to 45 percent of the storage budget.
- Trace and observability storage: 25 to 40 percent.
- Replay logs and eval artifacts: 15 to 25 percent.
- Long-term audit retention: 5 to 15 percent.
If the storage line creeps above five percent of total project cost, the cause is almost usually one of the four failure modes below. Storage cost should be reviewed monthly during the build phase and quarterly afterward. We treat storage as part of the AI project total cost of ownership; specifically the recurring infrastructure cost that runs alongside inference and licenses.
The four failure modes
Storage budgets fail in four characteristic ways.
Failure 1; Single-tier retention. Many traces live on hot storage for 90 days because that was the vendor default. Cost is 5x to 10x what it would be on a class-tiered policy. Mitigation: write a retention policy that distinguishes error-class traces from normal-class traces and aggregated metrics; apply it before the volume becomes painful.
Failure 2; Vector index sized for hypothetical scale. The team commits to a tier sized for projected 24-month scale at month 1; the projection does not materialize, and the team pays for unused capacity for the full term. Mitigation: size for current scale plus a six-month runway; resize on schedule rather than upfront.
Failure 3; No replay logs. The team skips replay-log capture in week 1 to save cost, then cannot reproduce a regression in month 6 because the input that triggered it is gone. Mitigation: capture replay logs from day one with a hard retention cap; the storage cost of replay logs is almost usually much smaller than the engineering cost of a regression that cannot be reproduced.
Failure 4; Audit-retention spillover. Hot-tier retention silently extends to the full compliance window because nobody set the tiering rule. The team pays hot-tier rates for cold-tier data for years. Mitigation: a single document that names the retention window and the tier for each storage class, reviewed at most compliance audit.
We see many four failure modes recur in the AI project FinOps playbook and the 6 hidden taxes on most AI project; the storage tax is the most quietly cumulative of the hidden taxes.
How to operationalize storage hygiene
Three practices keep storage cost honest.
Write a one-page storage policy. A single page that names each storage class, its retention window, its tier, and its expected size at the next review cycle. The policy is the artifact that converts vendor-default retention into deliberate retention. We discuss the broader operational discipline in the AI agency quality system piece.
Review storage cost monthly during build, quarterly afterward. Storage cost grows in steady-state; a quarterly review catches the growth before it becomes a surprise. The review takes 30 minutes and saves multiples of that in retention cleanup work avoided.
Tie storage cost to operational outcomes. Storage that supports the eval loop, the replay flow, or the regression triage is COGS and earns its cost. Storage that supports none of those is waste and gets pruned. The “what does this storage line enable” question is the highest-leverage hygiene question on the storage budget.
Frequently asked questions
How much should we budget for storage on a $250k AI project?
Two to four percent of total project cost when retention is governed; roughly $5k to $10k per year; and seven to twelve percent when retention is ungoverned. The 2x to 3x gap between governed and ungoverned storage is the single biggest lever in the storage budget.
Should we keep most trace at full fidelity?
No. Tier traces by class; error-class traces on hot storage with a 90- to 180-day window, normal-class traces on hot storage for 14 to 30 days then cold, aggregated metrics retained indefinitely at minimal cost. Uniform retention is the most common and most expensive mistake.
How long should replay logs live?
The eval window plus a buffer; typically 90 to 180 days for most workloads. Beyond that, replay logs accumulate without producing operational value. Set a hard cap with an exception path for specific cases that need to live longer.
What is an orphan vector?
An embedding in the vector index that points to a document that has been deleted from the source-of-truth store but rarely removed from the index. On a typical RAG workload, 5 to 15 percent of the index is orphaned within six months. Prune quarterly, or event-drive the prune to source deletion.
Should we keep replay logs even if the storage cost feels high?
Almost usually yes. The storage cost of replay logs is usually much smaller than the engineering cost of a regression that cannot be reproduced. Skipping replay logs to save 1 percent of storage cost commonly costs 10x more in regression triage time over a 12-month window.
How does storage cost interact with model upgrades?
A model upgrade triggers a re-eval, which consumes replay logs. If the replay logs were pruned aggressively, the re-eval cost goes up because the team has to reconstruct the workload. Retention windows should align with the model-upgrade cadence; typically three to five times per year.
Should we use a managed vector database or self-host?
Managed for most workloads in 2026. The pricing of managed services (Pinecone, Weaviate Cloud, Turbopuffer, pgvector on managed Postgres) compressed during 2025; the engineering cost of self-hosting now exceeds the license fee on most projects. The exception is unusual scale or compliance constraints.
Where does storage cost show up on the P&L?
Inside the inference-and-infrastructure COGS line, alongside model API spend and license cost. Storage that is not classified as COGS (for example, audit retention treated as G&A) usually escapes operational scrutiny and grows unchecked. We discuss the broader margin model in the AI project gross margin reset piece.
Key takeaways
- The storage tax is two to four percent of total AI project cost when governed, seven to twelve percent when not; the gap is the single biggest lever.
- Four storage classes drive the tax: vectors, traces, replay logs, audit retention. Each needs a class-specific retention rule.
- Class-tiered trace retention, capped replay-log retention, and quarterly orphan-vector pruning are the highest-leverage hygiene practices.
- Replay logs are almost usually worth their storage cost; skipping them to save 1 percent of storage cost commonly costs 10x more in triage time.
- Write a one-page storage policy; review storage cost monthly during build, quarterly afterward.
Arthur Wandzel
