What is Golden Dataset?

Why It Matters

An eval suite is only as good as the questions inside it. A golden dataset is the versioned collection of cases with known expected outcomes that every change must pass before it ships. Without a fixed set, “did this change make things worse” has no answer, because nothing stayed constant to compare against.

Golden because it does not move. The agent changes, the model changes, the prompts change; the yardstick stays put, so the system’s movement becomes measurable.

The Four Sources

Production failures. Captured through a failure memory pipeline: redact the sensitive data, cluster with similar failures, replay against the current agent, minimize to the smallest reproducing case, get human sign-off. The same production failure becomes harder to ship twice.

Production successes. Trajectories that worked form the regression baseline. A prompt change that breaks a previously working case gets caught before merge.

Adversarial edge cases. Inputs designed to break the agent: ambiguous requests, conflicting instructions, boundary values nobody asked for yet.

Coverage gaps. Paths the agent should handle but production has not exercised. An agent that supports 12 tools while production uses 8 needs cases for the other 4, because coverage gaps are where drift hides.

Where It Breaks

Cost kills suites quietly. An eval set that costs $500 per run will not get run often enough to matter. Keep it lean: deduplicate similar cases, use cheap deterministic graders where they suffice, reserve expensive judge calls for the cases where subjectivity is the point.

Staleness is the slower failure. A set built at launch and never revisited drifts away from what users actually ask, which is eval drift: the suite keeps passing while production moves on.

How Flytebit Handles It

PASSR’s golden set holds 200 PRs with known expected findings and costs about $40 per full run. It runs weekly and on every prompt or model change: the weekly cadence catches provider-side model updates that ship unannounced, and the change-triggered runs catch our own regressions. The construction method and the maintenance rules are in Evaluating Agentic AI.