What is Eval Drift?
EvaluationThe eval set aging away from what users actually ask. The suite keeps passing while production degrades underneath it, which is why the eval set itself is a scheduled review item, not a build-once artifact.
Why It Matters
The eval suite is the yardstick, and yardsticks drift too. An eval set written at launch slowly stops matching what users ask, while the suite keeps reporting green. Drift in the eval is worse than drift in the agent, because the instrument itself is now lying.
Two things move independently: the questions users ask, and the judge doing the scoring.
The Two Drift Channels
Stale cases. The eval set was built for what users asked at launch. Their questions moved; the suite did not. Every case that no longer reflects production is a false green, and enough of them make the pass rate a ritual instead of a measurement.
Judge drift. An LLM judge is a model, and providers update models. A judge that agreed with human labels 90 percent last quarter and 75 percent this quarter is scoring against a different rubric than the one it was calibrated on.
Where It Breaks
Provider-side updates arrive unannounced. The model that passed the suite in June is not the model being called in September, so a static suite ends up certifying a system that no longer exists. That is the eval measuring the wrong thing.
The quieter failure is a suite nobody reviews. Cases accumulate, none get removed, coverage gaps for new tools never get added, and the pass rate stays comforting while its meaning erodes.
How Flytebit Handles It
The eval set is a scheduled item on the LLMOps cadence: quarterly review, stale cases removed, new production failures added through the failure memory pipeline. PASSR’s weekly suite exists for exactly this reason: provider model updates shift review behavior with no code change on our side, and the weekly run against the golden dataset is what catches it. The maintenance rules are in Evaluating Agentic AI, and the cadence they sit inside is in Operating Agentic AI Systems.