How AI inference cost works (and how to budget for it)

AI inference cost is one formula plus three tier choices. The formula is cost = input_tokens × input_rate + output_tokens × output_rate + cached_tokens × cached_rate — and the tier choice (frontier vs workhorse vs fast-small) is a 10x lever, not a 10% one. Founders overpay because they pick the wrong tier, not because they wrote too-long prompts. This guide walks the formula from first principles, gives 2026 rate ballparks across three tiers from public vendor pricing pages, names three budgeting techniques a non-technical founder can use in a vendor conversation, and closes with the same 10,000-query workload costed three ways so the tier spread is visible at a glance.

This is the founder explainer that pairs with the AI MVP economics playbook, which decomposes a 6–12 week build line by line. It sits inside the broader idea-to-product manifesto, alongside two companion guides — AI infrastructure cost for an MVP, which covers all five infra lines, and the hidden cost of AI, which names the token streams the proposal forgets.

The unit is tokens, not requests

Every frontier-model vendor — Anthropic, OpenAI, Google — prices the same way: per token, not per request. A token is roughly four characters of English text, which means a 200-word prompt is around 270 tokens and a 500-word response is around 675 tokens. Vendor docs publish exact tokenizers for the curious; the four-characters approximation is good enough for budgeting.

There are three token classes, each with a separate rate:

• Input tokens — the prompt the team sends to the model. Includes the system prompt, the conversation history, any retrieved documents, and the user’s query.
• Output tokens — the response the model returns. Typically priced 4x to 5x higher than input tokens because generation is the expensive part.
• Cached tokens — input tokens that the vendor has stored from a prior request and can re-serve cheaply. Priced at a 50% to 90% discount versus standard input tokens.

The reason “request count” is the wrong unit: two requests can differ in cost by 100x. A short classification call (50 input tokens, 20 output tokens) and a long retrieval call (4,000 input tokens, 800 output tokens) are both “one request” but the second is roughly 80x more expensive at identical model tier. Founders who budget on requests-per-month rather than tokens-per-month get blindsided by long-context features they did not realize had landed in the product.

The cost-per-task formula

The honest cost-per-task formula is three terms summed:

cost_per_task = (input_tokens × input_rate)
              + (output_tokens × output_rate)
              + (cached_tokens × cached_rate)

Rates are quoted per million tokens. The arithmetic to convert a per-million rate to a per-task figure is straightforward — divide by one million, multiply by the token count.

A worked single-task example at workhorse tier ($3 per million input tokens, $15 per million output tokens as a representative 2026 mid-tier rate; re-verify current vendor pricing before contracting):

• 1,200 input tokens × ($3 / 1,000,000) = $0.0036
• 600 output tokens × ($15 / 1,000,000) = $0.0090
• Cost per task = $0.0126, or roughly 1.3 cents.

That cost-per-task is the foundation of every downstream budget. Monthly cost is cost-per-task × tasks-per-month. Cost-per-1,000-users is cost-per-task × tasks-per-user × 1,000. Margin on a paid feature is feature-price minus cost-per-task minus everything-else.

Three honest extensions to the basic formula:

1. Multi-call workflows — an agentic or retrieval-augmented workflow may run 3 to 8 model calls per user-facing task. Multiply by the call count, not by the user-facing-request count.
2. Cached prefix — if the system prompt is stable (most products), the cached rate applies to that portion of every call after the first. A 1,500-token cached system prompt at 80% off compresses the input bill by 50% to 80% on production traffic.
3. Batch discount — non-real-time work (overnight evals, bulk document processing) qualifies for batch APIs at roughly 50% off on Anthropic and OpenAI. Real-time user-facing traffic does not.

Decoding cost-per-query — a defensible unit-economics framework walks the metric-level argument: when cost-per-query is the right anchor versus when cost-per-action is. This guide stays one layer down, on the arithmetic.

2026 rates across three model tiers

Public 2026 vendor pricing pages cluster into three tiers. The exact dollar figures change — Anthropic, OpenAI, and Google all repriced multiple times in 2025 and 2026 — so re-check the vendor page before signing a contract. The tier structure, however, has been stable through frontier-model cycles and is what to budget against.

Tier	Example models (June 2026)	Input rate per 1M tokens	Output rate per 1M tokens	Best for
Frontier	Claude Opus 4.8, GPT-5 (reasoning), Gemini 2.5 Pro	$10–$20	$50–$100	Complex reasoning, agent planning, LLM-as-judge for evals
Workhorse	Claude Sonnet 4.6, GPT-5 standard, Gemini 2.5 Flash	$2–$5	$10–$25	Production traffic for most user-facing features
Fast-small	Claude Haiku 4.5, GPT-5 Mini, Gemini 2.5 Flash-Lite, Llama 3.3 hosted	$0.10–$1.00	$0.40–$5.00	High-volume classification, extraction, routing

The spread between tiers is the central budgeting fact: a frontier-tier task costs roughly 10x to 50x what the same-shape task costs on fast-small tier. For a workload where fast-small tier produces acceptable quality, choosing frontier tier is a 10x to 50x overspend that no amount of prompt engineering will recover.

Sources for tier-level rates: Anthropic pricing, OpenAI pricing, Google Gemini API pricing. Current SOTA leaderboard for tier-to-quality mapping: Artificial Analysis LLM leaderboard.

Two caveats. First, rates change. Anthropic, OpenAI, and Google reprice on cycles measured in months — pull the live page before quoting a customer or signing a vendor. Second, tier and model name are not the same thing. A proposal that says “we will use Claude” without naming the tier is hiding a 50x cost decision behind one word.

Three budgeting techniques that survive contact with a vendor

Three budgeting techniques cover the realistic founder conversations. Pick one as the primary anchor; the others stay in the back pocket as sanity checks.

Technique 1 — Per-task target

Set a target cost-per-task and design the workflow to clear it. A target of one cent per task on workhorse tier, for example, caps input-plus-output token spend at roughly 2,000 input tokens plus 400 output tokens — which forces tight prompts, smaller retrieval windows, and shorter responses.

Use it when: per-user margins are thin or unit economics must be defensible before scale. Avoid it when: complex agentic workflows have high value-per-task and a 3x cost-per-task spread is acceptable.

Technique 2 — Monthly cap

Set a total monthly inference budget — $500 or $5,000 — and wire alerts at 50%, 80%, and 100% of the cap. The team accepts variable cost-per-task in exchange for a hard ceiling on the bill.

Use it when: early-stage MVPs have unpredictable traffic and the founder cares more about a worst-case bill than a tight per-task number. Avoid it when: traffic is steady and a mid-month cap-fire would cause user-facing outages worse than a higher bill.

Technique 3 — Cost per 1,000 users

Translate cost-per-task into cost-per-1,000-users by multiplying by average tasks-per-user. A product where each user runs five tasks per month at one cent per task costs five dollars per 1,000 users — a clean input to a pricing or LTV model.

Use it when: a B2C product needs a published unit-economics narrative, particularly when raising capital. Avoid it when: in B2B, one customer is a sales cycle and tasks-per-user varies by two orders of magnitude across the customer base.

The three are not mutually exclusive. The cleanest founder budget uses a per-task target as the design constraint, a monthly cap as the financial guardrail, and cost-per-1,000-users as the investor-facing summary.

Worked example — 10,000 queries per month, costed three ways

The same 10,000-query-per-month product, designed identically (1,200 input tokens, 600 output tokens per query, no cache, no batching), costs roughly:

Tier	Input cost	Output cost	Total / month	Cost per query
Frontier (e.g., Claude Opus 4.8 at $15 in / $75 out per 1M)	$180	$450	$630	6.3¢
Workhorse (e.g., Claude Sonnet 4.6 or GPT-5 standard at $3 in / $15 out per 1M)	$36	$90	$126	1.3¢
Fast-small (e.g., Claude Haiku 4.5 or GPT-5 Mini at $0.80 in / $4 out per 1M)	$9.60	$24	$33.60	0.34¢

(Rates are illustrative June 2026 ballparks from public vendor pricing pages; re-verify before contracting.)

Three things worth naming about that spread:

1. Tier choice is the dominant lever. Frontier-to-fast-small is a roughly 19x range on identical workload. Prompt-engineering optimizations typically compress the bill by 20% to 40% — meaningful, but not in the same league as tier choice.
2. Cache compresses the bill by 30% to 60% on top of any tier, mostly on the input side. A 1,500-token system prompt cached at 80% off across 10,000 queries claws back roughly $30 per month on frontier tier and roughly $7 per month on workhorse tier. Worth wiring in, not worth obsessing over before product-market fit.
3. The bill above is production traffic only. Add eval traffic (200-sample suite × 30 runs/month is 6,000 paid calls, often LLM-as-judge against a higher tier) and developer traffic (manual testing, ad-hoc debugging). Realistic total token bill is 1.3x to 2x the production line, as decomposed in the hidden cost of AI: token bills, observability, on-call.

For the full operating-cost picture across all five infra lines (not just inference), see AI infrastructure cost for an MVP.

Reading the inference line in a vendor proposal

An honest 2026 inference line has six elements:

1. Which model and tier the proposal assumes.
2. A cost-per-task figure with the input-tokens and output-tokens math shown.
3. A monthly cost projection at the proposal’s traffic level.
4. A separate line for eval traffic at the same model or one tier higher (LLM-as-judge pattern).
5. A re-pricing clause naming what happens if vendor rates change or the team migrates models.
6. Whether cache and batch discounts are wired in, and the assumed compression.

Three red flags that signal a proposal not worth signing without rework:

• “Infrastructure included” with no monthly figure or tier named. Read it as “we have not modeled this and you will see it on a separate invoice in month three.”
• A single inference number with no input-output token assumption. The number is uncheckable and re-negotiable once traffic shows up.
• Frontier tier defaulted when the workflow is classification, extraction, or routing. Those run cleanly on fast-small tier; a frontier default is a 10x to 50x overspend the founder is being asked to fund.

Frequently asked questions

What is the difference between input tokens and output tokens?

Input tokens are what the team sends to the model — prompt, system instructions, retrieved context, and conversation history. Output tokens are what the model returns. Vendors price output tokens at 4x to 5x the input rate because generation is the more expensive operation. For typical workflows, output dominates the bill even though input tokens are higher in count.

How do I estimate token counts without writing code?

Four characters per token is a usable approximation for English text. A 100-word paragraph is roughly 130 tokens. Anthropic, OpenAI, and Google all publish browser-based tokenizers — paste the prompt in, see the count. The rough estimate is enough until the bill is non-trivial.

What is prompt caching, and how much does it really save?

Prompt caching lets the vendor store the static portion of a prompt — typically the system prompt and stable context — and re-serve it at a 50% to 90% discount on the input rate. Realistic compression on production token bills is 30% to 60%, mostly on input. Eval traffic does not benefit (prompts under test change), so cache is a production optimization, not a total-bill optimization.

Is the batch API really 50% off?

Yes, for non-real-time work. Anthropic and OpenAI both publish batch APIs at roughly 50% off standard rates with a 24-hour-or-less turnaround. Use batch for overnight evals, bulk classification, and backfill work. Do not use it for user-facing requests — the SLA does not fit the product experience.

How often do model rates change?

Often enough to budget against ranges rather than point estimates. Anthropic, OpenAI, and Google each repriced multiple times across 2025 and 2026 — generally downward at fixed quality, upward at the new frontier. Any planning number more than three months old is suspect.

Can I just use the cheapest model and save the difference?

Sometimes, yes — for classification, extraction, routing, and short-form generation, fast-small tier produces acceptable quality at 10x to 50x less cost. For complex reasoning, multi-step planning, or LLM-as-judge, dropping down quietly degrades quality. A/B test the workflow at workhorse and fast-small against a fixed eval set, and pick the cheapest tier that clears the quality bar.

What is LLM-as-judge, and why does it show up in the cost line?

LLM-as-judge is an evaluation pattern where one model grades the output of another. Teams run production on workhorse tier and judge on frontier tier, which means eval calls cost roughly 5x to 20x what production calls cost. The pattern is correct — judges need to be smarter than the model under test — but it is a real line item that thin vendor proposals omit.

How do I budget inference cost when traffic is unpredictable?

Use a monthly cap with alerts at 50%, 80%, and 100%. Pair it with a per-task target so engineering knows the design constraint. For a pre-launch MVP, $200 to $500 per month is a defensible production-inference budget on workhorse tier at 5,000 to 20,000 queries per month — re-budget once real traffic shows up.

Should the founder pick the model, or the vendor?

The founder picks the tier; the vendor picks the model. Tier is a budget conversation — frontier vs workhorse vs fast-small. Model is an engineering conversation — which workhorse-tier vendor’s strengths match the workload. A founder who insists on a specific model is usually optimizing the wrong layer.

What is a defensible 2026 monthly inference budget for an MVP at 10,000 queries per month?

Production traffic on workhorse tier with caching wired in lands at $80 to $150 per month. Add eval and developer traffic (25% to 60% on top), and the realistic total inference line is $100 to $250. Frontier-tier defaults push it to $500 to $1,000 at the same volume; fast-small tier compresses it to $30 to $60.

Where to go next

• The AI MVP economics playbook — line-by-line build cost for a 6–12 week MVP.
• AI infrastructure cost for an MVP — all five running-cost lines, not just inference.
• The hidden cost of AI — the three token streams (production + eval + developer) and the observability and on-call lines that hide in “infra”.
• Decoding cost-per-query — when cost-per-query is the right unit-economics anchor versus when cost-per-action is.
• The idea-to-product manifesto — the overview for the whole program.