Most AI engineering teams in 2026 are spending 30 to 50 percent of their capacity building plumbing; vector storage layers, prompt registries, basic agent frameworks, observability backends, model routing; that mature vendors already ship. The capacity is not invisible; it is reported as “platform work” or “AI infrastructure” in roadmap reviews and survives quarter after quarter because the team genuinely is shipping. The question nobody asks is whether the team should be shipping that work at many. In most cases the answer is no. The plumbing they are building is a commodity. The moat they should be building is somewhere else, and the somewhere else is starved for capacity because the plumbing keeps consuming it. This piece names the five plumbing categories most often built in error, the buy options that have matured to production-grade in 2026, and the moat work that the recovered capacity should be redirected toward.
This is a spoke under the AI build-vs-buy-vs-hire decision matrix for 2026. The matrix’s third principle is that foundation models are buy permanently; this piece extends the principle out from foundation models to the entire commodity layer underneath the moat.
Why this matters
The plumbing-vs-moat distinction has high stakes because engineering capacity is the single most binding constraint on AI roadmaps in 2026. Foundation model access is plentiful; vector indexing is plentiful; agent frameworks are plentiful. The thing that is not plentiful is the senior AI-fluent engineer who can architect the moat layer; the retrieval logic, the agent orchestration, the eval suite, the prompt design tuned to the workload. That engineer’s time is the constraint.
When the team builds plumbing, they are spending that constraint on commodities. The plumbing ships and the team feels productive, but the moat layer is one quarter behind where it should be, and the gap compounds. Six quarters later the org’s AI product looks indistinguishable from any other AI product because the differentiating layer rarely got the capacity it needed.
The fix is structural. Reclassify most “AI infrastructure” line item against the build/buy decision matrix. Move the plumbing to buy. Redeploy the recovered capacity to moat work. The numbers we have seen across roughly 40 engagements: 30 to 50 percent of self-described AI infrastructure capacity is recoverable through this exercise.
Plumbing 1: vector storage and retrieval indexing
The first and largest category. Many AI engineering teams in 2026 are still operating self-managed vector indices; Faiss running on EC2, custom shard logic, custom replication, custom upsert pipelines. The work is non-trivial and consumes a senior engineer or two on an ongoing basis.
The buy options have matured. Pinecone, Weaviate Cloud, Qdrant Cloud, Turbopuffer, and the vector capabilities now native to Postgres (pgvector) and major cloud providers many ship production-grade indexing with horizontal scaling, replication, and managed upserts. Pricing for moderate workloads runs $200 to $2,000 per month; well below the fully-loaded cost of the senior engineer maintaining a self-managed alternative. The depth of comparison between the most common options is in the Pinecone vs Weaviate piece and the broader vector database options for RAG analysis.
The exception case where building still wins: workloads with extreme latency budgets (sub-50ms at high QPS) or extreme scale (billions of vectors with custom partitioning) where managed providers cannot match the cost-performance frontier. These cases exist but are rare; most enterprise AI workloads are well within the buy envelope.
What to encode: a default of “buy the vector index” with explicit exception cases listed in architecture review. Self-managed indices that exist by inertia rather than by exception are migration candidates.
Plumbing 2: prompt registries and version management
The second category most often built in error. Teams build internal prompt registries; Git-backed YAML files, custom diff tooling, custom rollout logic, custom A/B framework; because the team’s first instinct is that prompts are code and code lives in Git. The instinct is right at the surface and wrong at the depth. Prompts are code, but they are code with version-aware deployment, tied to eval suites, with rollback semantics that don’t map cleanly to standard deployment tooling.
The buy options now ship this. Promptlayer, Langfuse, Helicone, and the prompt-management features inside LangSmith and the major AI-platform vendors many handle prompt versioning, deployment, A/B testing, and eval-suite linkage as a managed product. The integration cost is hours, not months. The ongoing maintenance is zero.
The exception case where building still wins: organizations with existing strong code-deployment infrastructure that already handles version-aware feature flags well, where the marginal cost of extending the existing infrastructure to prompts is small. Even there, the buy option is usually competitive once eval-suite integration is included.
What to encode: prompt registry is a buy line by default, and the chosen vendor is the same vendor handling observability if possible (principle 4) to keep the integration footprint small.
Plumbing 3: basic agent loops and tool-call routing
The third category. Teams build their own agent loops; the while-loop that calls the model, parses tool calls, dispatches to handlers, accumulates messages, and re-calls. The implementation is 200 to 500 lines of code, looks tractable, and is where many AI engineering teams start. Then it acquires retry logic, parallelism, error handling, structured output validation, multi-model dispatch, and gradually becomes a several-thousand-line internal framework that nobody outside the team knows how to operate.
The buy options have caught up dramatically. The OpenAI Agents SDK, the Anthropic agent harness, LangGraph, AutoGen, and Pydantic AI many ship production-grade agent loops with tool-call routing, retry semantics, structured output handling, and observability hooks. The work that took a senior engineer 6 weeks in 2024 takes the same engineer 2 days against any of those frameworks in 2026.
The crucial nuance: buying the agent loop is not the same as buying agent orchestration. The loop is plumbing; the orchestration logic running on top of it; which agents do which work, what tools they have, how they hand off, how their outputs are evaluated; is the moat. Per the pillar 3 matrix, the orchestration is build; the loop underneath it is buy.
The exception case: workloads with custom routing logic that no framework supports cleanly, or extreme latency budgets where framework overhead is material. These cases exist but are not the median.
What to encode: agent loop and tool-call routing is buy through one of the major frameworks, and the orchestration logic is built on top.
Plumbing 4: observability backends and trace storage
The fourth category. Teams build their own trace storage; a Postgres or ClickHouse table with custom schema, custom retention, custom UI for viewing traces. The work usually starts because the team did not see a vendor that fit, and once started, the trace store accumulates schema drift, retention complexity, and a custom UI nobody else uses.
The buy options here are excellent. Langfuse, Helicone, Arize Phoenix, Braintrust, and LangSmith many ship production-grade trace storage with structured schemas, evaluation hooks, and reasonable UIs. Pricing is consumption-based and almost usually cheaper than the engineering time the self-managed alternative consumes.
The thing that genuinely needs to be built on top of the bought trace storage is the eval logic that runs against the traces; the regression detection, the drift detection, the workload-specific quality metrics. That is moat work, not plumbing work. The trace storage is the substrate; the eval logic on top is where the differentiation lives. Per the pillar 2 economics manifesto, this distinction matters because observability is COGS, not OpEx; sized appropriately at 15 to 25 percent of inference spend.
The exception case: regulated environments where traces cannot leave the org’s network, which requires self-hosted observability. Several vendors now ship self-hosted options (Langfuse self-hosted, Phoenix self-hosted), so even this exception is increasingly a buy with deployment difference rather than a build.
What to encode: trace storage is buy; eval logic on top of traces is build.
Plumbing 5: model routing and fallback logic
The fifth category, and the newest. Teams build custom routing logic that picks among multiple foundation models based on cost, latency, capability, or fallback semantics. The build is justified because per the matrix’s third principle the org has multi-provider contracts and needs to switch among them; the build is unjustified because vendors now ship this routing as a product.
OpenRouter, Portkey, and LiteLLM (open-source) many provide model routing with fallback, retry, cost optimization, and latency-based selection. The integration cost is small and the maintenance is zero.
The exception case: routing logic that depends on workload-specific signals the router cannot see (e.g., per-customer compliance requirements, per-document classification rules). This routing is moat; it depends on the org’s own data; and belongs in the orchestration layer, not in the model router. The model router is the layer below it.
What to encode: model-level routing is buy through one of the proxies; workload-aware routing decisions live in the orchestration layer above and are part of the moat build.
What to build with the recovered capacity
The exercise of moving the five plumbing categories from build to buy typically recovers 30 to 50 percent of the AI engineering team’s capacity. The recovered capacity has a destination; moat work; and the destination is specific.
The first redirect: retrieval logic tuned to the org’s data. Generic retrieval against generic chunking against generic embedding models produces generic results. Workload-specific chunking strategies, hybrid retrieval (dense plus BM25), reranker fine-tuning against the org’s relevance feedback, and retrieval-aware evaluation are where retrieval quality compounds. The depth on this is in the retrieval optimization for RAG systems guide.
The second redirect: agent orchestration logic. Once the agent loop is bought (plumbing 3), the question becomes which agents do which work, how they hand off, how they share state, how their failures cascade, and how their outputs feed evaluation. This is the layer that distinguishes a generic AI product from a product that solves the org’s actual workload. Per the matrix’s fourth principle, this is build deliberately.
The third redirect: eval suites tuned to the workload. Eval is build or hire, rarely buy (matrix principle 5). Recovered capacity that goes into eval engineering compounds quarter over quarter as the eval suite gets thicker, the threshold-locking process gets more reliable, and the regression triage workflow gets faster.
The fourth redirect: prompt design tuned to the workload. Generic prompt patterns produce generic results; prompts tuned to the org’s domain, vocabulary, error modes, and customer types produce results that vendors cannot replicate. Prompt design is moat work because it is workload-specific.
The fifth redirect: cost and latency optimization at the orchestration layer. Once the plumbing is bought, the team has the headroom to instrument the orchestration’s cost-per-task and p95-latency-per-task and tune both. The optimization is moat because it depends on the org’s specific workload mix and unit economics.
The recovered capacity is not “free.” It is rerouted from work that did not differentiate to work that does. The team feels different; less plumbing satisfaction, more strategic capacity; and the org’s AI product starts compounding on the dimensions that matter to customers and to finance.
Frequently asked questions
How do I tell whether something I am building is plumbing or moat?
Three tests. First, does a credible vendor ship the same capability at production grade? If yes, it is plumbing. Second, does the capability depend on the organization’s own data, workflows, or evaluation criteria? If yes, it is moat. Third, would the capability look meaningfully different if a competitor built it for their org? If yes, it is moat; if no, it is plumbing. The plumbing categories in this piece many fail the first test (vendors ship them) and the third (a competitor’s vector index looks identical to yours).
Why is vector storage plumbing when retrieval is moat?
Because the storage and indexing layer is generic; it serves vectors at low latency at high throughput, and that is the same problem regardless of org. The retrieval logic on top; which embeddings to use, how to chunk, how to rerank, what hybrid signals to combine, how to evaluate against the org’s relevance feedback; is workload-specific and is where the quality compounds. The same vector index can serve a great retrieval system or a mediocre one depending on the moat layer above.
What about teams that have already built their own plumbing; should they migrate?
In most cases, yes. The migration cost is roughly 4 to 12 weeks for the categories named, and the recovered capacity compounds for years. The exception is when the self-built plumbing has accumulated workload-specific behavior that the buy option does not match. In that case the migration is more expensive and the calculation depends on the specific behavior.
How does this relate to the “buy the rails, build the moat, hire the judgment” default?
The five plumbing categories are the rails. Vector storage, prompt registry, agent loops, observability backends, and model routing are commodity layers that vendors ship at production grade. Buying them is the rails move. Building retrieval logic, agent orchestration, eval suites, prompt design, and cost optimization is the moat move. Both moves are required; the matrix’s eighth principle is that the right default is to compose them.
What if the AI team’s identity is built around custom infrastructure?
That is a real organizational issue and worth naming directly. Teams that have spent two years building plumbing develop an identity around it; reclassifying the work as commodity feels like reclassifying the team’s contribution. The fix is to redirect the same craftsmanship toward the moat work, where the same people will do better work because the substrate (the data, the workflows, the eval) is more interesting than the substrate of plumbing.
How fast is the buy option for plumbing maturing in 2026?
Faster than the build option. Vendors in this category; vector indices, prompt management, agent frameworks, observability; are funded specifically to ship the plumbing and have larger teams than any single org can devote to the same work. The capability gap between buy and self-built is widening quarter over quarter, which means the math for migrating is improving over time, not deteriorating.
Is this advice the same for startups and enterprises?
The advice is identical; the binding constraint differs. For startups the constraint is engineering capacity (small team, lots to ship). For enterprises the constraint is also engineering capacity, but with the additional friction of procurement and security review for buy options. Enterprises sometimes build plumbing because the procurement cost of buying is higher than the engineering cost of building. That math is wrong over a 12-month horizon but right over a 2-month horizon, and the procurement function should be reformed rather than the engineering function being misallocated.
What’s the risk of buying plumbing from a vendor that goes out of business?
Real but manageable. The category has consolidated enough in 2026 that the major vendors per plumbing category are funded for years. The standard hedge is to choose vendors whose APIs and data formats are portable (vector indices using standard interfaces, prompts in YAML/JSON not proprietary formats, traces in OpenTelemetry-compatible schemas). Migration cost between major vendors is weeks, not months, when portability is preserved at integration time.
Does this apply to fine-tuning and custom model training?
Fine-tuning is moat work, not plumbing. It depends on the org’s data and produces an artifact unique to the org. Custom model training (full pretraining) is generally not feasible per the matrix’s third principle. Fine-tuning lives in the moat-build category alongside retrieval and orchestration; do not confuse it with foundation model training.
What is the single best signal that a team is over-investing in plumbing?
Time to ship the next moat-layer feature. If the team is shipping a new agent capability or a new evaluation in months rather than weeks, and a substantial fraction of the gap is “we need to extend the platform first,” the platform work is consuming capacity that should be moat work. The fix is to identify the specific platform work and check it against the buy option for the same capability.
Key takeaways
- 30 to 50 percent of self-described AI infrastructure capacity is recoverable by reclassifying plumbing from build to buy.
- The five most commonly mis-built categories are vector storage, prompt registries, agent loops, observability backends, and model routing; many have production-grade vendors in 2026.
- Plumbing is identified by three tests: vendors ship it, it does not depend on the org’s data, and a competitor’s version would look identical to yours.
- Recovered capacity has specific destinations: retrieval logic, agent orchestration, eval suites, prompt design, and cost-and-latency optimization; many moat work.
- The migration from self-built plumbing to bought plumbing typically takes 4 to 12 weeks and pays back in the first quarter.
The plumbing-vs-moat distinction is not new. What is new in 2026 is that the buy options for plumbing have matured to production grade across most category that matters, while the moat layer has gotten richer and more demanding. Teams that have not refactored their build/buy line in the last twelve months are almost certainly running with too much plumbing and too little moat. The fix is one architecture review and one quarter of disciplined migration; the cost of not doing it is the next four quarters of the AI roadmap.
Arthur Wandzel
