What is Regression Gate?
EvaluationAn eval wired into the release path with a threshold: score below the line and the change does not ship. The gate is what turns an eval from a report into a control.
Why It Matters
An eval that produces a report nobody acts on is a dashboard. The gate is what makes it a control: wire the suite into the release path, set a threshold, and a score below the line stops the deploy. Without it, eval results compete with deadlines, and deadlines win every time.
For a non-deterministic system, a single passing run is a single sample. The gate has to compare distributions, not point scores, or it will block good changes and wave through bad ones.
How It Works
Run the suite 20 times on the candidate version and 20 times on the baseline, then compare the spreads. A statistical test (Welch’s t-test) answers whether the two versions genuinely differ. The effect size (Cohen’s d) answers whether the difference is big enough to care about. Report a 95 percent confidence interval, because at 20 runs a 51 percent pass rate is consistent with anything from 24 to 94 percent. A single number is a claim. A range is evidence.
Setting the Threshold
One threshold for every task is the wrong design. A payment action needs pass^k above 0.99; a summary task tolerates 0.80. Set the bar by the cost of failure for that task class, and a real regression of 0.1 percent stops being worth a blocked deploy while a 15 percent drop demands a second look even when the test says noise.
Where It Breaks
Reading noise as signal is the common mistake: a p-value of 0.410 says the difference is likely noise, and blocking anyway teaches the team to route around the gate. The opposite failure is a gate tuned so tight that flaky tests block every merge. Engineers stop trusting it, and the control dies quietly.
A gate that only watches the model version misses most of what changes. Prompts, tool definitions, and retrieval config all shift behavior without touching the model.
How Flytebit Handles It
Our gates run against the versioned manifest, not the model alone: a prompt edit, model bump, tool change, or config update each triggers a full rerun against the golden dataset, and a drop below the task-class threshold blocks the merge. The statistical method is documented in Evaluating Agentic AI, and the operating cadence the gate sits inside is in Operating Agentic AI Systems.