What is Runbook?
Operations (LLMOps & AgentOps)The documented procedure for a known failure: what to check, what to revert, who to call. For agents the runbook reads trajectories and manifests, not stack traces: the evidence is behavioral, not infrastructural.
Why It Matters
Traditional incident response assumes deterministic systems: the service is either working or broken, and the runbook isolates the failure and rolls back. An agent can be running, producing plausible output, and causing damage at the same time. Microsoft’s incident-response guidance for AI makes the break explicit: the remediation stages stay the same (contain, investigate, recover), but the harm categories and the evidence are different. The agent runbook reads trajectories and manifests rather than stack traces, because the failure is behavioral.
The Severity Tiers
P0: active damage. Runaway spend, data exposure, unauthorized actions in progress. Kill switch immediately, rollback in-flight actions, page the incident commander. Target: kill under 60 seconds, rollback under 5 minutes.
P1: bounded violation. Tool misuse or a governance breach with contained damage. Revoke tool permissions, terminate sessions, page the agent’s owner. Target: contain under 5 minutes.
P2: quality drift. Degradation with no harm. PagerDuty’s guidance is blunt about this tier: a judge-score drop is not fixable at 2 a.m. by an on-call responder, and paging for it burns the rotation without fixing anything. Route to the feature team for next business day.
P3: informational. Drift indicators, cost trends, scheduled maintenance. Log and move on.
Where It Breaks
The first break is severity miscalibration: treating every quality blip as P0 burns the on-call rotation, while treating real P0s as drift lets damage run. The second is evidence hygiene: an incident record that mixes what the team knows with what it guesses becomes unreliable exactly when regulators or customers ask for it, so assertions, hypotheses, and decisions belong in separate fields. The third is containment mistaken for isolation: network isolation stops new calls but does nothing about the credentials held, the sessions open, and the outputs already shipped.
How Flytebit Handles It
Our agent runbooks start from the first five minutes: stop new damage, open the incident record with separated assertion and hypothesis fields, identify when behavior first deviated from baseline, map the blast perimeter of outputs already delivered, and partition actions by reversibility. The full incident-response model is in Operating Agentic AI Systems, and the operating model is our LLMOps work.