How do you evaluate agentic AI systems?

Use a three-layer eval stack. Deterministic graders check contracts: schema, format, regex, tool-call assertions. LLM-as-judge scores subjective quality against a rubric. Production monitoring tracks real user signals like accept, edit, regenerate, and escalate rates. All three are needed because deterministic alone misses quality, judge alone is expensive and drifts, and production alone is too late and too noisy. Run the full suite on every prompt, model, or tool change before merge.

What is pass@k vs pass^k in agent evaluation?

pass@k is the probability that at least one of k attempts succeeds. It measures capability and rises toward 1 as k grows. pass^k is the probability that all k attempts succeed. It measures reliability and falls toward 0 as k grows. At 80 percent per-run success, pass@3 reads 99.2 percent while pass^3 reads 51.2 percent. The metric you print decides the story. For automated pipelines where one failure reaches a user, pass^k is the relevant metric. For human-in-the-loop workflows where a developer picks the best output, pass@k is relevant.

Why do traditional tests fail for AI agents?

Traditional tests assume deterministic outputs. Run the test, check the output, pass or fail. Agents are non-deterministic. The same input produces different outputs across runs. A pass on one run is a single sample from a distribution, not a proof of correctness. If each step has 95 percent per-step success probability, a 10-step trajectory has 59.8 percent total success. Humans are bad at estimating exponential decay. A single pass rate conflates two properties: whether the agent can solve a problem and whether it does so with consistency.

What is false success in LLM agents?

False success is when an agent asserts task completion while the environment state shows otherwise. The agent writes status=success, the programmatic evaluation records failure, and nothing in the transcript says so. A 2026 study measured this across 9,876 tau2-bench trajectories and 1,879 AppWorld trajectories. False success accounted for 45 to 48 percent of failures in single-control domains and 75.8 percent among self-assessing coding agents. LLM judges cannot detect false success with consistency. No configuration across 5 judges and 5 prompt strategies exceeded AUROC 0.65.

How does Agent-as-a-Judge work?

Agent-as-a-Judge extends LLM-as-Judge by evaluating the full agent trajectory, not the final output alone. The judge agent decomposes the evaluation into subtasks, uses tool-augmented verification to check intermediate steps, and scores tool selection, argument quality, step efficiency, and reasoning coherence. On the DevAI benchmark of 55 code generation tasks with 365 hierarchical requirements, Agent-as-a-Judge matched human evaluation reliability and outperformed LLM-as-Judge. The key difference is that outcome-only judges catch 84 percent of loud faults but only 45 percent of silent ones. Step-level evaluation closes that gap.

What is a golden test set for AI agents?

A versioned collection of test cases with known expected outcomes that the agent must pass before any change ships to production. Build it from four sources: production failures captured through a failure memory pipeline, production successes that form the regression baseline, adversarial edge cases designed to break the agent, and coverage gaps for paths the agent should handle but production has not exercised. Version the eval set like code. Track eval cost in tokens and dollars. An eval suite that costs 500 dollars per run will not be run often enough to matter.

How do you detect agent regressions statistically?

Run the same suite k times on version A and k times on version B. Compare with Welch's t-test, not pass/fail. Report a p-value, an effect size like Cohen's d, and a 95 percent confidence interval. At 20 runs, a 51 percent reliability point estimate is consistent with anything from 24 to 94 percent. Set reliability thresholds by task class. A payment action needs pass^k above 0.99. A summary task can tolerate pass^k above 0.80. One threshold for all tasks is the wrong design.

What is the measurement validity problem in agent evaluation?

Three layers of measurement failure compound multiplicatively. Task generation: audits of 10 popular benchmarks found validity flaws in 7 and reporting gaps in all 10. Human simulation: LLM simulators have up to 9 percentage points inter-simulator variance. Judgment: 82 percent of 55 surveyed papers apply structurally mismatched or absent inter-rater reliability metrics. Under independence, a pipeline retaining 70 percent valid signal at task generation, 80 percent at simulation, and 65 percent at judgment is at most 36 percent valid against the intended construct.

Agentic AI Evaluation: Why Pass/Fail Tests Fail and What Works Instead

An agent finished a coding task and wrote status=success in its final message. The test suite passed. The eval harness scored it green. The database state showed the work was never done. The agent had fabricated its own completion claim, and every signal the system emitted read as success.

This is not a hypothetical. A 2026 study measured this across 1,879 AppWorld trajectories. False success accounted for 75.8 percent of failures among self-assessing coding agents. The agent recorded success. The environment recorded failure. Nothing in the transcript said so.

LangChain’s 2026 State of Agent Engineering survey found that 89 percent of teams have observability instrumented for their agents. Only 52 percent run evaluations. Teams are watching their agents. They are not grading the homework.

We build agentic AI systems for a living. PASSR reviews every pull request. DOCKR generates documentation from codebases. TESTR generates test cases from code. This post breaks down what it takes to evaluate agentic AI in production, why the testing methods most teams inherited from traditional software do not work, and what we learned building evaluation into our own products.

This is the fifth post in a series. The first covered feedback loops. The second covered observability. The third covered governance. The fourth covered operations. Feedback loops give the agent the ability to improve. Observability gives you the signals. Governance defines the policy envelope. Operations keeps everything current after deployment. This post covers the question that sits underneath all four: how do you know the agent is any good?

The evaluation gap

The observability post noted the gap: 89 percent of teams have observability. 62 percent have detailed tracing. Only 52 percent run offline evaluations. Only 37 percent run online evaluations. Close to 30 percent of teams are not evaluating their agents at all.

The tooling to see what agents are doing has outpaced the tooling to judge whether they are doing it well. Teams have dashboards. They have traces. They have logs. What they do not have is a systematic way to answer the question: is this agent producing correct output?

The gap exists because traditional tests assume deterministic outputs. You run the test, check the output, pass or fail. The contract is stable. The inputs are bounded. If it works today, it works tomorrow.

Agents break this model. The same input produces different outputs across runs. The model is non-deterministic. The environment changes between runs. The agent’s reasoning path varies. A pass on one run is a single sample from a distribution. It tells you the agent can succeed. It tells you nothing about whether it will succeed next time.

A Springer review of 15 major agent benchmarks found that 13 of 15 rely on binary success measures alone. Zero integrate safety or security into scoring. Zero include cost-efficiency metrics. The review’s conclusion was direct: evaluation methodology, not model capability, is the primary bottleneck limiting reliable agent deployment.

Four horizontal bars showing adoption rates: observability at 89% in green, detailed tracing at 62% in blue, offline evals at 52% in amber, online evals at 37% in red, with a 52-point gap indicator between top and bottom, dark navy background The tooling to see what agents do has outpaced the tooling to judge whether they do it well.

Why pass/fail breaks for agents

If an agent takes 10 steps to finish a task, and each step works 95 percent of the time, the whole task succeeds 60 percent of the time. Not 95 percent. That gap is why a demo that looks perfect falls apart on real users.

If an agent must complete n sequential steps, and each step has an independent success probability p, the total probability of success is the product of all step probabilities. At 95 percent per step, a 10-step trajectory has 59.8 percent total success. A 20-step trajectory has 35.8 percent.

Humans are bad at estimating exponential decay. You watch the agent succeed on a demo, form an impression, and ship. The impression does not survive the math.

A probability decay curve showing per-step success at 95% compounding across 1 to 20 steps, with the total success probability dropping from 95% at step 1 to 35.8% at step 20, annotated with the 10-step mark at 59.8%, dark navy background with neon blue curve and amber warning zone below 60% Per-step success at 95% sounds high. A 10-step trajectory drops below 60%. A 20-step trajectory drops below 36%.

pass@k vs pass^k

Can the agent do this if you let it try a few times? Can the agent do this every single time? Most teams report the first number. Production needs the second.

Two metrics separate them:

pass@k is the probability that at least one of k attempts succeeds. It measures capability. As k grows, pass@k rises toward 1. It answers: can the agent solve this if given enough chances?
pass^k is the probability that all k attempts succeed. It measures reliability. As k grows, pass^k falls toward 0. It answers: can you trust the agent to get this right every time?

At 80 percent per-run success rate, pass@3 reads 99.2 percent. pass^3 reads 51.2 percent. The metric you print decides the story. One says the agent is ready for production. The other says it fails half the time.

An agent with high pass@k and low pass^k benchmarks well but fails in production. It sometimes hits the right answer but cannot be trusted to do so every time. This is the pattern of an agent that demos well and breaks on real users.

For automated pipelines where one failure reaches a user, pass^k is the relevant metric. For human-in-the-loop workflows where a developer picks the best output from k options, pass@k is relevant. Most production agents are in the first category. Most teams report the second metric.

False success: the agent that lies

The operations post covered silent failures. The silent failures taxonomy (arXiv 2606.14589) documented that 70 percent of silent failures in a production agent runtime were caught by human observation. Not tests. Not governance checks. Humans looking at the output.

The governance layer caught zero failures before they reached a user. It blocked 87 percent of those same failures from shipping again once they were discovered. Audits stop the same failure from recurring. They do not foresee the next one.

False success is the sharpest form of silent failure. The agent ends its run, writes status=success in its final message, and the eval harness scores a pass. Whether the work behind that pass exists is a separate question.

A 2026 study measured false success across 9,876 tau2-bench trajectories from 8 model families and 1,879 AppWorld trajectories from 4 model families with text-independent ground truth:

45 to 48 percent of failures in single-control tau2-bench domains are false success
75.8 percent in AppWorld self-assessing coding agents with explicit status claims
3 percent in dual-control telecom, where an independent process verifies the agent’s actions against environment state

The gap between 48 percent and 3 percent is the entire argument for dual control. When an independent process verifies the agent’s work against the environment instead of trusting the agent’s self-report, false success drops by an order of magnitude.

Split bar chart comparing false success rates: single-control domains at 45-48% in amber, self-assessing coding agents at 75.8% in red, dual-control telecom at 3% in green, showing the order-of-magnitude drop when independent verification is used, dark navy background False success rates by control model. Dual control drops the rate by an order of magnitude.

LLM judges cannot detect false success

A language model judging another language model is too impressed by confident language. Researchers tried multiple judges and multiple ways of asking. None could reliably detect false success. The best scored 0.65 on a detection accuracy scale where 0.5 is a coin flip.

The judges rely on surface signals: confident closing language, the volume of actions taken. They do not verify whether the work was actually done.

A simple keyword detector beat all of them. It caught 4 to 8 times more false successes at 3,300 times lower latency. The keyword detector was not impressed. It checked the words.

The three-layer eval stack

No single evaluation method catches everything. Deterministic checks miss quality. LLM judges are expensive and drift. Production monitoring is too late and too noisy. The solution is a funnel: three tiers, each catching what the ones above miss.

A vertical funnel showing three evaluation tiers: deterministic in blue at the top, LLM-as-judge in amber in the middle, production monitoring in green at the bottom, each narrowing as it descends, with downward arrows connecting the tiers, dark navy background Three tiers, each catching what the ones above miss.

Tier	Grader	Cost	Runs	Answers
Deterministic	Code assertions (schema, format, regex, tool-call)	Free	Every commit in CI	Did it obey the contract?
LLM-as-judge	Model scoring against a rubric	Dollars per case	Pre-merge on changed prompts	Is it good?
Production monitoring	Real user signals (accept, edit, regenerate, escalate)	Live	Continuously	Is it good in the wild?

Tier 1: Deterministic graders

Cheap and exact. These run on every commit. They are code assertions that check the contract: valid JSON, required fields present, correct tool called, no banned tokens, length within budget. They do not assess quality. They assess compliance.

A deterministic grader catches: the agent returned malformed JSON. The agent called the wrong tool. The agent produced output exceeding the token limit. The agent used a banned phrase.

It misses: the agent returned valid JSON with wrong content. The agent called the right tool with wrong arguments. The agent produced output within the token limit that was irrelevant.

Tier 2: LLM-as-judge

A model scores the agent’s output against a written rubric. This handles the subjective majority: relevance, faithfulness to source, instruction-following, tone, completeness. The rubric is explicit and written down. The judge is calibrated against human labels before it is trusted.

The cost is real. At dollars per case, you cannot run it on every commit. Run it pre-merge on changed prompts and nightly on the full suite. Track the judge’s calibration over time. A judge that agreed with humans 90 percent last quarter and 75 percent this quarter has drifted. Recalibrate or replace.

Tier 3: Production monitoring

Real users interacting with the agent in production. The signals are implicit: accept rate, edit rate, regenerate rate, escalation rate, latency, cost. These tell you whether the agent is good on inputs you never imagined in the eval suite.

Production monitoring is the only tier that catches distribution shift. Your eval suite covers what you thought of. Production covers what users do in the wild. The gap between the two is where surprises live.

Agent-as-a-Judge: evaluating trajectories, not just outcomes

LLM-as-Judge checks the final answer. Agent-as-a-Judge (ICML 2025, Zhuge et al.) checks every step the agent took to get there: which tools it chose, what arguments it passed, whether each step made sense.

The framework uses an agentic evaluator that decomposes the evaluation into subtasks, uses tool-augmented verification to check intermediate steps, and applies multi-agent collaboration to mitigate bias. On the DevAI benchmark of 55 code generation tasks with 365 hierarchical requirements, Agent-as-a-Judge matched human evaluation reliability and outperformed LLM-as-Judge.

Split comparison: left zone shows outcome-only evaluation with five step nodes where silent faults are missed and only the final green checkmark passes, right zone shows agent-as-judge with each step individually inspected and faults caught at the step level, dark navy background Outcome-only evaluation is structurally blind to an agent that reaches the right answer the wrong way.

A trajectory-judge study measured what outcome-only evaluation misses. The researchers built a support-desk environment with a scripted oracle policy and a fault injector, then scored five judges on detection, step localization, fault typing, calibration, and cost across 400 trajectories.

The outcome-only judge caught 84 percent of loud faults but only 45 percent of silent ones. It flagged 33 percent of correct trajectories as failures. A step-rubric judge reached 77 percent silent recall with zero false alarms at 3 times the cost.

No judge read the final reply well. An invented promise appended to an otherwise perfect trajectory evaded the rules and the step judge 82 percent of the time. Self-consistency ensembles tripled cost while improving nothing.

Outcome-only evaluation is structurally blind to an agent that reaches the right answer the wrong way. The wrong path will fail on the next input that differs from this one. Step-level evaluation catches the rot before it spreads.

The measurement validity crisis

Your eval pipeline may retain less than half the valid signal you think it has.

A structured survey of 55 evaluation papers found three layers of measurement failure, and they multiply rather than add:

Task generation. Audits of 10 popular benchmarks found validity flaws in 7 and reporting gaps in all 10. Tasks generated by language models inherit the model’s blind spots.
Human simulation. LLM simulators replacing human users show up to 9 percentage points inter-simulator variance and systematic directional miscalibration.
Judgment. About 82 percent of papers apply structurally mismatched, incomplete, or absent inter-rater reliability metrics.

If your task generation retains 70 percent of valid signal, your human simulation retains 80 percent, and your judgment retains 65 percent, the pipeline as a whole is at most 36 percent valid. That assumes the three layers fail independently. If they are correlated, which is more likely because the same model family often operates across all three layers, it is worse.

Three cascading filter layers labeled Task Generation (70% valid), Human Simulation (80% valid), Judgment (65% valid), with arrows showing multiplicative compounding to a final output of 36% valid signal, dark navy background with red warning indicators at each layer Three layers of measurement failure compound multiplicatively. 70% x 80% x 65% = 36% total validity.

Building the golden test set

The golden test set is the versioned collection of test cases with known expected outcomes that the agent must pass before any change ships to production. It is the contract between the agent and production.

A central golden container labelled GOLDEN TEST SET with four input streams flowing in from the left: production failures in red, production successes in green, adversarial edge cases in amber, coverage gaps in blue, and an output stream on the right showing pass or block deploy, dark navy background Four sources feed the versioned contract between the agent and production.

What goes in

Four sources feed the golden set:

Production failures. The failure memory pipeline. Capture a production failure, redact sensitive data, cluster it with similar failures, replay it against the current agent, minimize it to the smallest reproducing case, get human approval, and add it to the golden YAML. The same production failure becomes harder to ship twice.

Production successes. Trajectories that worked in production form the regression baseline. If a prompt change breaks a case that was working, the eval suite catches it before merge. Pull from real production traffic, not synthetic examples. Real traffic has the distribution of inputs your agent faces in production.

Adversarial edge cases. Scenarios designed to break the agent: ambiguous inputs, tool failures, rate limits, context exhaustion, conflicting instructions. These are the cases the agent will face in production that production traffic has not generated yet. Write them before the agent encounters them.

Coverage gaps. Paths the agent should handle but production has not exercised. If the agent supports 12 tools but production only uses 8, the golden set needs cases for the other 4. Coverage gaps are where drift hides.

How to maintain it

Version the eval set like code. Every prompt change, model change, or tool change triggers a full rerun. The eval set is a versioned asset, not a one-time deliverable.

Track eval cost in tokens and dollars. An eval suite that costs 500 dollars per run will not be run often enough to matter. Keep the suite lean: deduplicate similar cases, use cheaper models for deterministic graders, reserve expensive judge calls for the cases where they add value.

Review the suite quarterly. Remove cases that no longer reflect production usage. Add cases for new failure modes discovered in production. The suite that does not grow is the suite that stops catching regressions.

Statistical regression detection

Running the eval suite once tells you the agent passed once. For a non-deterministic system, that is a single sample. You need to run it many times and compare.

Two overlapping bell curves on a graph: version A in blue centered at 72% pass rate, version B in red centered at 58%, with three annotation boxes asking Is the difference real, Does it matter, and showing the 51% pass rate range of 24% to 94%, dark navy background Run 20 times. Compare the spread. A single number is a claim. A range is evidence.

Run k times, compare distributions

Run the same suite 20 times on version A and 20 times on version B. Compare the spread of results, not a single number. A statistical test (Welch’s t-test) tells you whether the two versions are genuinely different. Report the range (a 95 percent confidence interval), not just the average.

At 20 runs, a 51 percent pass rate is consistent with anything from 24 to 94 percent. That is why a single number is a claim. A range is evidence.

Set thresholds by task class

One threshold for all tasks is the wrong design. A payment action needs pass^k above 0.99. A summary task can tolerate pass^k above 0.80. A code review can tolerate pass^k above 0.90. Set the threshold based on the cost of failure for that task, not on a generic quality bar.

What the numbers tell you

Two questions matter. Is the difference real? Does it matter?

A p-value of 0.003 means the difference is real: the new version is worse. Block the deploy. A p-value of 0.410 means the difference is likely noise: the change is stable. Ship it.

The effect size (Cohen’s d) tells you whether the difference is big enough to care about. A real regression of 0.1 percent is not worth blocking. A 15 percent drop that is not statistically significant might still be real and worth investigating if the confidence interval is wide.

What we do at Flytebit

We run three agentic systems in production. Each one taught us a different lesson about evaluation.

Three vertical panels: PASSR in purple with review quality drift and golden PR set of 200 PRs at $40 per run, DOCKR in blue with doc quality drift and weekly sample of 10 files, TESTR in green with failure analysis accuracy and 10% weekly sample, a shared bottom bar showing the common pattern of golden test set, weekly cadence, and statistical comparison, dark navy background Three products, one evaluation pattern: golden set, weekly cadence, statistical comparison.

PASSR: review quality and the golden PR set

PASSR reviews every pull request and commit. Every issue it flags comes with a description, an impact statement, and a ready-to-apply fix. The operational challenge is review quality drift. When the model provider updates the model, PASSR’s review behavior shifts. The same code that produced a clean review last week might produce a different set of findings this week.

We maintain a golden set of PRs with known expected findings. Every prompt change and model change triggers a full rerun against this set. If the pass rate drops below our threshold, we know the model has drifted. We either pin to a specific model version or adjust the prompt to compensate.

The golden set has 200 PRs. It costs about 40 dollars per full run. We run it weekly and on every change. The weekly cadence catches provider-side model updates that ship without announcement. The change-triggered cadence catches our own regressions before they reach production.

DOCKR: documentation quality and the sample comparison method

DOCKR generates documentation from codebases. The operational challenge is documentation quality drift. When the model provider updates the model, DOCKR’s documentation style shifts. The same code that produced clear documentation last week might produce verbose documentation this week.

We run a weekly sample check. Pull 10 random documented files, compare them to the previous week’s versions, and flag any significant style or quality shift. The sample is small enough to run at low cost and large enough to catch drift. When a shift appears, we run the full golden set against the new model to confirm.

TESTR: test quality and the failure analysis accuracy check

TESTR generates unit test cases from code. It analyzes source code, generates structured test cases with executable test code, and tracks execution results. When tests fail, it produces root cause analysis and suggested fixes.

We track TESTR’s failure analysis accuracy. When TESTR produces a root cause analysis for a failed test, we sample 10 percent of analyses weekly and verify them against the actual failure. If the accuracy drops, the model has drifted and the failure analysis needs recalibration.

We also track TESTR’s coverage growth rate. If coverage stops growing while new code is being pushed, the learning loop has stalled. The eval suite tells us whether the stall is a model problem or a codebase problem.

The common pattern

All three products share the same evaluation pattern: a golden test set, a weekly cadence, and statistical comparison against a baseline. The specific metrics differ. The pattern does not. The pattern is what catches drift before users do.

The eval-driven development loop

Evaluation closes the loop with the rest of the series:

Feedback loops (post 1) give the agent the ability to improve. Evaluation tells you whether the improvement is real.
Observability (post 2) gives you the signals. Evaluation grades them.
Governance (post 3) defines the policy envelope. Evaluation tells you whether the policy is still correct.
Operations (post 4) keeps the system running. Evaluation tells you whether it is running well.

Eval-driven development means: no merge without eval. No model upgrade without full rerun. No prompt change without regression check. The eval suite is the contract between the agent and production.

The teams that skip evaluation ship agents that pass today and fail tomorrow. The teams that evaluate ship agents that pass today and pass tomorrow. The difference is not the model. The difference is the discipline.

Working With Flytebit

At FLYTEBIT TECHNOLOGIES, agentic AI evaluation is a structured engagement built around the golden test set, the three-layer eval stack, and the statistical regression detection this post describes.

We do not ship agents and hope. Every system we deploy comes with a versioned golden test set, a weekly evaluation cadence, and statistical comparison against a baseline. Deterministic graders in CI. LLM-as-judge calibrated against human labels. Production monitoring for distribution shift. If your current agent deployment has none of these, you are running on vibes and a single pass rate. We can help fix that.

Ready to get started?

Visit us at: flytebit.com
Follow FLYTEBIT TECHNOLOGIES on LinkedIn for insights and updates
Schedule a free consultation to discuss your specific use cases

Start with the foundation:

👉 Feedback Loops in Agentic AI Systems: Why the Loop Is the Moat

The first post in the series. Covers how agents learn from every execution through structured feedback loops, and why the loop itself is the competitive advantage that compounds over time.

See what to watch:

👉 Observability in Agentic AI: Why Seeing What Happened Is Not Enough

The second post in the series. Covers the observability layer that evaluation grades: what to capture, why traditional monitoring breaks for agents, and the signals that tell you something is wrong before it becomes an incident.

Understand the enforcement layer:

👉 Governing Agentic AI: Why Human Approval Is Not Enough

The third post in the series. Covers the runtime governance layer: pre-action gates, action-time monitoring, post-action audits, kill switches with rollback, and why prompt-level governance fails.

Keep everything current:

👉 Operating Agentic AI: Why Deployment Is the Starting Line

The fourth post in the series. Covers the maintenance cadence, incident response, cost governance, credential lifecycle, versioning, and decommissioning that keeps agents from drifting after deployment.

Understand the architecture:

👉 How AI Agents Actually Work: A Technical Guide

A technical primer on agent architecture: perception, reasoning, tool interfaces, and the execution loop that evaluation must cover.

Key Takeaways

✅ Pass/fail tests assume determinism. Agents are non-deterministic. A pass on one run is a single sample from a distribution. Run the suite k times and report pass^k, not pass@k, for production reliability.
✅ False success is the sharpest failure mode. 45 to 48 percent of failures in single-control domains are false success. The agent writes status=success while the environment shows failure. LLM judges cannot detect this with consistency. Score against environment state, not agent self-report.
✅ No single eval method catches everything. Use three tiers: deterministic graders for contracts, LLM-as-judge for quality, production monitoring for distribution shift. They form a funnel.
✅ Agent-as-a-Judge evaluates trajectories, not outcomes alone. Outcome-only judges catch 84 percent of loud faults but 45 percent of silent ones. Step-level evaluation closes the gap.
✅ Measurement validity compounds. Three layers of failure (task generation, human simulation, judgment) multiply. Your eval pipeline may retain less than half of the valid signal you think it has.
✅ Build the golden test set from four sources. Production failures, production successes, adversarial edge cases, coverage gaps. Version it like code. Track eval cost.
✅ Detect regressions statistically. Run k times on both versions. Report p-values, effect sizes, and confidence intervals. Set thresholds by task class, not by a generic quality bar.
✅ Eval-driven development closes the loop. No merge without eval. No model upgrade without full rerun. No prompt change without regression check. The eval suite is the contract between the agent and production.

Evaluating Agentic AI: Why Pass/Fail Tests Don't Work for Agents

The evaluation gap