Promptfoo vs Inspect vs Langfuse: which eval tool your MVP partner should use

The three names in any 2026 AI MVP proposal are Promptfoo, Inspect, and Langfuse — and they are not direct substitutes. Each occupies a structurally different position in the eval pipeline. Promptfoo is the pre-deploy regression suite that runs in CI. Inspect AI is the research-grade framework for evaluating agents, tool use, and sandboxed execution. Langfuse is the production observability platform that traces a deployed app and lets you score live outputs. A mature stack often uses two of the three. A vendor who picks the wrong tool for the wrong position is not wrong about the tool — they are wrong about the problem. This piece is the founder’s decoding key: what each tool is, where each fits, and how to challenge a proposal that names one when another would fit better.

It builds on the eval-first build playbook, part of the idea-to-product manifesto. It pairs with the eval rubric template every founder should ask for — that gives the rubric; this piece gives the harness that grades against it — and with the eval test set every MVP needs before launch. The commercial frame lives in stop paying AI agencies for documentation, pay them for evals.

Table of Contents

• Why founders read tool proposals, not pick tools
• Three tools, three positions in the pipeline
  • Promptfoo — the pre-deploy regression suite
  • Inspect AI — the research-grade eval framework
  • Langfuse — the production observability platform
• The side-by-side decision matrix on five axes
• When each one fits (and when it is wrong)
• What to challenge in a vendor proposal
• The mature 2026 stack — usually two of the three
• Frequently Asked Questions
• Closing

Why founders read tool proposals, not pick tools

The non-technical founder is not the right person to pick the eval tool. The vendor’s senior engineer is. But the founder is the right person to verify the choice — to read the vendor’s selection out of the proposal and ask whether the tool fits the problem shape. Same pattern as the rubric template: engineer drafts, founder verifies structurally.

The structural frame: an LLM eval pipeline has three positions.

1. Pre-deploy regression suite — a fixed test set, run in CI on every change, gates the deploy.
2. Research-grade eval — a richer framework for agents, multi-turn dialogues, tool use, and sandboxed execution.
3. Production observability — trace ingestion from a deployed app, on-trace scoring, drift monitoring.

Promptfoo, Inspect AI, and Langfuse dominate one position each. One tool per position is the typical pattern. A vendor using one tool for all three is forcing it out of its sweet spot.

Three tools, three positions in the pipeline

Promptfoo — the pre-deploy regression suite

Promptfoo is an open-source, MIT-licensed eval framework maintained by Promptfoo Inc. The contract is a declarative YAML test suite (promptfooconfig.yaml) naming prompts, models, cases, and assertions. The CLI runs the suite and produces side-by-side HTML reports plus CI-friendly JSON.

What it does best:

• Static regression suites in CI. promptfoo eval runs on every PR; pass/fail gates the merge; reports diff against the previous run.
• Prompt and model A/B matrices. Two prompts, three models, one report — the side-by-side founders can read.
• LLM-as-judge assertions. Built-in graders including model-graded-closedqa, factuality, g-eval, with the judge model named in config.
• Red-team mode. promptfoo redteam ships adversarial test generation for safety regression suites.

What it is structurally not: a production observability tool (no continuous trace ingestion from a deployed app), and not an agent-eval framework with sandboxed code execution.

Founder-facing read: a promptfooconfig.yaml checked into the repo plus a CI workflow that runs it on every PR — that is the regression-suite position done right.

Inspect AI — the research-grade eval framework

Inspect AI is an open-source, MIT-licensed Python framework maintained by the UK AI Security Institute (UK government body, formerly the AI Safety Institute). It is the framework UK AISI uses for frontier-model government safety evaluations — institutional provenance that signals the problem shape: rigorous, reproducible, agent-grade.

The contract is Python. An eval is a Task composed of a Solver (model plus tools plus scaffolding) and one or more Scorers. Inspect supports tool use natively, sandboxed code execution (docker and local modes), multi-turn agent loops, and a provider registry covering OpenAI, Anthropic, Google, AWS Bedrock, and open-weights models.

What it does best:

• Agent evaluations. Multi-turn agents that call tools and iterate need a scaffolded loop. Inspect’s Solver is built for it.
• Sandboxed code execution. Coding-agent and security evals depend on isolation.
• Programmatic composition. Python beats YAML when eval logic needs branching, dynamic inputs, or composition with existing test infra.
• Reproducibility. The eval log is a structured artifact; inspect view is the read interface.

What it is structurally not: a YAML-driven regression suite a founder can scan, and not a production observability tool — Inspect grades runs, not live traffic.

Founder-facing read: if the feature is a single-turn classifier, Inspect is overkill. If it is a multi-turn agent that calls tools or executes code, a Promptfoo-only proposal is under-equipped.

Langfuse — the production observability platform

Langfuse is an open-source observability and eval platform maintained by Langfuse GmbH (Berlin). OSS core is MIT-licensed; a hosted tier exists and self-hosting is supported. The contract is SDK instrumentation: the deployed app emits traces — every LLM call, retrieval step, tool invocation — into Langfuse, which renders the trace tree, cost-per-call, latency, and lets you score outputs.

What it does best:

• Production trace ingestion. Every LLM call captured with full context — prompts, completions, tools, metadata, user IDs, costs.
• On-trace LLM-as-judge evaluators. Scheduled judges run on incoming traces; drift surfaces in dashboards.
• Human annotation queues. PMs and founders open the UI and grade live outputs; labels feed the next rubric and regression suite.
• Dataset-based offline evals. Static datasets also run through Langfuse, though this overlaps a Promptfoo suite.

What it is structurally not: a CI-first regression suite with pass/fail merge gates (Promptfoo is more native), and not an agent-eval framework (it traces; it does not scaffold).

Founder-facing read: a proposal without Langfuse (or Phoenix, Helicone, LangSmith) for the post-launch phase has no visibility on production quality. That is a gap, not a tool preference.

The side-by-side decision matrix on five axes

Axis	Promptfoo	Inspect AI	Langfuse
CI integration (pre-deploy gate)	Native — promptfoo eval in GitHub Actions / GitLab CI / any runner	Strong — Python-native, runs in any CI; less out-of-the-box scaffolding than Promptfoo	Possible via dataset runs; not the native flow
Regression suite shape	Declarative YAML, fixed test set, diff reports	Programmatic Python, dynamic suites, agent-grade composition	Dataset-driven; UI-readable, less CI-native
LLM-as-judge native	Yes — model-graded assertions library built in	Yes — Scorers including LLM judges, custom Python graders	Yes — scheduled judge evaluators on live traces
Production observability	Limited — import traces as datasets, not continuous	Limited — eval-time only, not production tracing	Native — SDK ingests live traces, dashboards, drift
Cost / hosting model	OSS free; self-host; no required hosted tier	OSS free; self-host; no hosted tier from AISI	OSS free self-host or paid hosted tier; pricing per ingested observation

Read across the row, not down. No tool wins every axis. A founder reading a proposal that names one tool for all five jobs should ask which jobs the other two are filling.

When each one fits (and when it is wrong)

Three problem shapes show up in MVP proposals.

Problem A: single-turn classifier or generator. Email priority classifier; copy rewriter; structured extraction over a document. One LLM call scored against a rubric.

• Right: Promptfoo (regression suite) + Langfuse (post-deploy observability). Inspect is overkill.
• Wrong: Inspect-only — research-grade framework for a CI-suite job. The build is slower than it needs to be.

Problem B: multi-turn agent with tool use. A research agent that browses and synthesises; a coding agent that runs commands; a sales-discovery agent calling CRM and outbound. The output is a trajectory, not a single call.

• Right: Inspect AI (framework) + Langfuse (observability). Promptfoo can hold simple multi-turn fixtures; the agent-scaffold layer belongs in Inspect.
• Wrong: Promptfoo-only — the YAML contract distorts the agent’s control flow.

Problem C: production-shipped product, post-launch drift focus. System is live; the founder wants to monitor quality, score live outputs, and grade drift across model upgrades.

• Right: Langfuse as the primary surface; Promptfoo or Inspect as the pre-deploy gate on each release.
• Wrong: Promptfoo-only — no observability on live traffic. Flying blind after launch.

Pick the tool whose sweet spot matches your problem shape.

What to challenge in a vendor proposal

Five questions a founder asks when the vendor names a tool.

1. Which position in the pipeline is this tool filling? If the answer is all three, push back. Each tool’s sweet spot is one position; two of three is the mature pattern.

2. Where does the regression suite live and how does CI gate it? A suite that is not version-controlled and not gated by CI is theatre. Ask for the file path of the eval config and the CI workflow that runs it. (See the eval test set every MVP needs before launch.)

3. If this is an agent build, why not Inspect? Tool use, sandboxed execution, or multi-turn trajectories without Inspect needs justification. Team Python depth or prior tooling is acceptable; we always use Promptfoo is not.

4. What is the production observability plan? If the answer is we will add it later, that is a yellow flag. Drift starts on day one; the observability surface ships with launch.

5. What harness does the rubric grade against? The eval rubric template is harness-agnostic. Name the harness explicitly with its config under version control. A rubric without a named harness is a rubric without a grader.

If the vendor cannot answer these cleanly, the problem is the engineering judgment, not the tool.

The mature 2026 stack — usually two of the three

The most common pattern in well-engineered MVPs is two of the three.

• Promptfoo + Langfuse — single-turn or short multi-turn features. Promptfoo gates the deploy; Langfuse watches the live traffic. The most common pairing across SFAI builds.
• Inspect AI + Langfuse — agent builds. Inspect handles the eval-time framework with sandboxes and tool scaffolding; Langfuse handles the production trace.
• All three — rare. Used when the product mixes single-turn and agent features under one platform.

Single-tool stacks happen in two cases: the build is genuinely small enough (one feature, internal-only), or the engineer is over-fitting to a familiar tool. Either way, the founder hears an explicit rationale, not a default.

Cost is not load-bearing. OSS self-hosting of all three is modest. Hosted Langfuse scales with traffic; Promptfoo and Inspect have no required hosted tier. All MIT-licensed OSS cores — no dependency risk for an MVP.

Adjacent landscape: DeepEval, Braintrust, LangSmith, Phoenix (Arize), Helicone, Patronus compete in the same three positions. If a vendor names one of the alternatives, the five challenge questions still apply — the structural axes are tool-agnostic.

Frequently Asked Questions

Should the founder pick the tool, or only verify the choice?

Verify. The vendor’s senior engineer picks based on problem shape and team depth. The founder verifies that the tool’s structural position matches the problem and that production observability is included from launch — same pattern as the rubric template.

Is one of these three the best overall?

No. They occupy structurally different positions. Promptfoo is the strongest pre-deploy regression suite for YAML evals in CI. Inspect AI is the strongest framework for agent evals with tool use and sandboxes. Langfuse is the strongest open-source production observability surface. Ranking them is a category error.

Can a small MVP get away with just one?

Sometimes. A single-turn classifier with tiny traffic can launch with Promptfoo alone if observability is added before traffic scales. No production observability after launch is the most common regret in retrospectives — the quality system playbook treats observability as table stakes.

What about LangSmith, DeepEval, Braintrust, Phoenix, or Helicone?

Each competes in one of the three positions. LangSmith and Braintrust are commercial platforms spanning regression suites and observability — strong but vendor-locked. DeepEval is closest to Promptfoo with a richer metric catalog. Phoenix (Arize) and Helicone are Langfuse alternatives. Same structural decision logic.

Does the choice depend on the model provider?

No. All three tools support OpenAI, Anthropic, Google, and most open-weights providers. When the model alias upgrades (per the Artificial Analysis leaderboard), the suite reruns and produces a diff report.

Is the open-source license enough, or do we need the hosted tier?

OSS-self-hosted is enough for most MVPs. Hosted tiers (Langfuse Cloud, Promptfoo Cloud) buy convenience — SSO, team management, retention guarantees — not capability. Revisit when traffic grows or compliance pushes you to a managed surface.

How does LLM-as-judge bias affect tool choice?

All three inherit the known biases — verbosity, position, self-preference — documented in the MT-Bench paper. The mitigation is rubric design (binary or 3-point anchors, asymmetric per-axis floors), not the tool. Bias work happens at the rubric layer.

Should the harness ship before or after the first model code?

Before. The eval-first build playbook argument is that the eval suite is the spec, written before the prompt. The harness sets up in the first two weeks of a build, alongside the eval-first PRD.

Who owns the eval config artifacts after the engagement?

The founder. The promptfooconfig.yaml, the Inspect Task files, the Langfuse project and saved evaluators all live in the founder’s repo and account. Vendors who withhold these as IP are running the anti-pattern named in stop paying AI agencies for documentation, pay them for evals.

Closing

The right way to read an AI MVP eval-tool proposal is structurally. There are three positions in the pipeline — pre-deploy regression, research-grade eval, production observability — and three tools that dominate those positions in 2026: Promptfoo, Inspect AI, and Langfuse. Each tool is the best at one position and acceptable at none of the others. A vendor who proposes one tool for all three is mismatching the tool to the problem. A vendor who proposes two of the three, with each one named against its position, is doing the work.

The founder’s verification is the same five questions on every proposal: which position is each tool filling, where does CI gate the suite, why this tool and not its alternative, what is the production observability plan, and where in the repo do the harness artifacts live. If the vendor answers cleanly, the choice is sound. If they cannot, the tool is not the problem.

Have a vendor proposal that names one of these tools and want a second opinion on the fit? Book a 30-minute idea review. Bring the proposal, the feature description, and the user persona. We read the tool choice against the problem shape and tell you within thirty minutes whether the structural call is right and what to challenge if it is not.