What is LLM-as-Judge?
EvaluationUsing a model to score another model's output against a rubric. It catches quality failures deterministic checks miss, at a price: judge models cost money per run and drift as providers update them.
Why It Matters
Most of what makes an answer good cannot be asserted in code. Relevance, faithfulness to the source, instruction-following, tone: grading those needs something that can read. LLM-as-judge puts a model on that job, scoring output against a written rubric. It sits in the middle of the eval stack, more discerning than a regex and cheaper than a human.
How It Works
The judge receives the agent’s output plus an explicit rubric and returns a score per criterion. Two disciplines make its scores mean anything: the rubric is written down rather than implied, and the judge is calibrated against human labels before anyone trusts it. Because each call costs real money, the cadence is deliberate: pre-merge on changed prompts, nightly on the full suite.
Where It Breaks
Judges drift. A judge that agreed with human labels 90 percent last quarter and 75 percent this quarter changed underneath you, because the provider updated it. Recalibrate or replace, but do not keep scoring against a judge you no longer recognize.
Judges are also blind to false success. A 2026 study ran 5 judges with 5 prompt strategies against agents claiming success while the environment disagreed; no configuration beat AUROC 0.65, where 0.5 is a coin flip. Confident closing language and a long action list impress a language model. A plain keyword detector caught 4 to 8 times more false successes at 3,300 times lower latency, because it checked the words instead of admiring them.
How Flytebit Handles It
Judge calls occupy the middle tier of our eval stack: deterministic graders on every commit, judge scoring pre-merge and nightly, production monitoring continuously. Judge calibration is a tracked metric on the LLMOps cadence in its own right, because a drifted judge is a broken instrument reporting green. The tier design and the calibration data are in Evaluating Agentic AI.