An agent spent $47,000 on a single task. Every trace on the dashboard stayed green the entire time. The agent was not failing. It was doing exactly what it was designed to do: try to complete the task. It just never stopped trying. This is not hypothetical. It happened in a documented 2026 post-mortem of a LangChain multi-agent pipeline that ran for 11 days before anyone noticed.
Observability showed what happened. Nobody stopped it.
That gap, between seeing and governing, is where most agentic AI deployments are stuck right now. LangChain’s 2026 State of Agent Engineering survey found that 89% of teams have observability instrumented for their agents. Only 52% run evaluations. Teams are watching their agents. They are not grading the homework.
We build agentic AI systems for a living. PASSR reviews code. DOCKR generates documentation. TESTR generates tests. All three run as background agents in engineering pipelines. Observability is the baseline that makes all three governable. This post breaks down why observability matters for governing agentic AI, what it must capture, where it stops being enough, and what we learned building it into production.
The Observability Gap
Observability and governance answer different questions.
Observability asks what the agent did, how long it took, what tools it called, and what it cost.
Governance asks whether the agent should have done it, whether it followed the required lifecycle, whether the right tools were called in the right order, and whether the output was verified against acceptance criteria. Was the work authorized at this scope?
These are different questions that require different systems to answer. Observability systems are built for the first set. Governance systems are built for the second. Adding governance features to an observability platform does not solve the structural problem, any more than adding security features to a CI/CD platform solves security.
The LangChain survey makes the gap concrete. 89% of teams have observability. 62% have detailed tracing that lets them inspect individual agent steps and tool calls. For teams already in production, those numbers climb to 94% and 71.5%. But only 52% run offline evaluations on test sets. Only 37% run online evaluations monitoring real-world performance. Nearly 30% of teams are not evaluating their agents at all.
The tooling to see what agents are doing has outpaced the tooling to judge whether they are doing it well.
Agents are not traditional software. A traditional API call has predictable cost and deterministic output. An agent’s cost is unbounded, depending on steps, tokens, retries, and duration. A normal run might cost $3. The same agent hitting an edge case can cost $47,000, and every trace on the dashboard stays green the entire time. The agent is not failing. It is doing exactly what it was designed to do, which is try to complete the task. It just never stops trying.
Observability tells you the $47,000 run happened. It does not tell you it should not have. And it does not stop it while it is happening.
The gap between observability and governance: two different questions, two different systems
Why Traditional Monitoring Breaks for Agents
Traditional monitoring assumes deterministic execution. Same input, same output. You watch for errors and latency spikes. When something breaks, you get an alert. You fix it.
Agents break that model in four specific ways.
Four ways agents break the assumptions traditional monitoring was built on
Non-deterministic execution
Same prompt, different outputs. Same agent, different tool choices, different orderings. Failures cannot be reproduced by replaying input. A traditional monitoring system that relies on reproducing incidents is useless when the incident is a one-off path the agent took because it read an ambiguous instruction in a new way this time.
Compounding errors
Traditional errors are isolated. An API call fails, the caller handles it. Agent errors propagate silently. An agent that retrieves wrong context does not know it is wrong. Every subsequent action is contaminated. A misinterpretation at step one becomes wrong retrieval at step two becomes hallucinated policy at step three. By step ten the agent is operating in a reality unrelated to user intent. Traditional monitoring catches the error at step ten. It does not catch the contamination at step one.
Cost as a runtime variable
A traditional API call has predictable cost. An agent’s cost depends on how many steps it takes, how many tokens it consumes, how many retries it attempts, and how long it runs. There is no natural ceiling. A cost alert that fires at $100 tells you the agent already spent $100. A cost bound that stops execution at $100 prevents the agent from spending $101. The difference between an alert and a bound is $46,900 in the $47K scenario.
Behavioral drift
Agents drift over time. Each individual deviation is small and produces an acceptable outcome. The drift is visible only in the pattern across sessions: the agent took a small shortcut in session one, a slightly larger shortcut in session three, and by session seven the shortcuts have accumulated into a governance violation. Observability traces for each session look normal in isolation. The deviation in each session is within noise levels. Only a layer that tracks behavioral patterns over time can detect the drift trajectory before it crosses a safety boundary.
| Traditional monitoring | Agent observability |
|---|---|
| Deterministic Same input, same output | Non-deterministic Same input, different paths |
| Isolated errors One call fails, caller handles it | Compounding errors Wrong step contaminates all subsequent steps |
| Predictable cost Fixed cost per call | Unbounded cost $3 normal, $47K edge case |
| Static behavior Code does not change between runs | Behavioral drift Agent shortcuts accumulate over sessions |
| Reproducible Replay input to debug | One-off Cannot replay the exact failure path |
| Alert after failure Notification when something breaks | Need bounds before damage Circuit breaker stops execution at threshold |
What Observability Must Capture for Governance
If observability is the foundation of governance, the record it produces has to be good enough to answer governance questions. Most teams cannot answer basic questions about their agents today. They know the model name and the latency. They do not know the tool arguments, the guardrail verdicts, the per-request cost, or the retry behavior.
Three things observability must capture to answer governance questions: the linked trajectory, the tool call record, the decision record
The trajectory, not the API call
In an agentic system, a single user request expands into a chain of model calls and tool invocations spread across services. The unit of visibility is the trajectory, covering one user intent and all the model calls and tool invocations that follow from it. Application-level instrumentation produces partial visibility because each service is instrumented independently. Provider SDK calls often go untraced. Tool executions performed by an agent framework land outside the application’s span tree. The result is a set of disconnected fragments rather than one trajectory you can inspect.
Without a trace that links them, a policy violation cannot be attributed to an identity. A cost spike cannot be explained, and an incident cannot be reconstructed afterward.
Tool calls as first-class records
Tool calls are where agentic behavior becomes consequential. They are also the layer application logs miss most often. A model call that returns text is low-risk. A tool call that writes a file, sends a request, modifies a database, or spends money is where the agent touches the real world. Observability that captures model calls but not tool calls gives you the agent’s thinking without its actions.
The OWASP Top 10 for LLM Applications 2025 lists excessive agency as a distinct risk category. The practical control for it is tool-call logging. If you cannot see which tools the agent called, with what arguments, and in what order, you cannot detect excessive agency. You cannot govern it either.
Decision records
For compliance under frameworks like the EU AI Act, a decision record needs to answer a specific question. For this one output, on this one date, given this one input, why did the system produce this result?
A sparse log that records a timestamp and a top-level output label does not satisfy this. The log needs to support forensic review. At minimum, a decision record should capture:
- A stable decision ID
- A timestamp
- The exact model version used at inference time
- A hash of the prompt template in effect at that moment
- The IDs of any retrieved context passed to the model
- The input data
- The output produced
- Any human oversight action taken (approved, rejected, modified, expired)
Article 12 of the EU AI Act requires high-risk AI systems to allow automatic recording of events over the lifetime of the system. The logs must allow reconstruction of system behavior, not just record that events occurred. Retention is at least six months. This is a design requirement placed on providers. You build it in before deployment. You do not bolt it on after.
Three architectural gaps cause most teams to fail this requirement without realizing it. Prompt version drift: prompts get edited without a version tied to individual decisions. Retrieval-set capture: the index keeps changing, and unless the exact set of retrieved documents is captured per decision, no one can explain why the model produced a specific output. Model version pinning: providers roll versions behind the same API alias, sometimes without a changelog. A decision made through a given alias in March and another in July may have run on different weights.
None of these are compliance checkboxes. They are schema decisions that have to be made before a system ships, because they cannot be retrofitted onto data that was never captured.
Where Observability Stops Being Enough
Observability is necessary. It is not sufficient on its own.
The gap between observability and governance is structural. Observability tells you what happened. What the agent is allowed to do is a separate question, and governance answers it. These are different systems built for different questions.
Consider the $47K loop again. Observability captured every step. Every trace was green, the cost was recorded, the tool calls were logged. Everything worked as designed. What was missing was a bound. A circuit breaker that stops execution at $100. An alert at $100 says the agent already spent $100. A bound at $100 stops the agent from spending $101.
Four capabilities sit in the gap between observability and governance. Observability tools do not provide them.
Execution bounds. Circuit breakers, not alerts. A bound stops execution when a threshold is hit. 100 steps. $10 per execution. 3 iterations per sub-task. When a bound triggers, execution stops. Damage is capped.
Policy enforcement. A dashboard shows you the action after it happened. A policy gate stops it before it moves. A refund that shows up in a trace already went through. Blocking it at the gate means the money never moves.
Behavioral pattern tracking. Observability provides point-in-time snapshots. Each trace is an independent record. The system does not learn from past failures to prevent future ones. If an agent enters a loop on Monday and you fix the specific trigger, an identical loop with a different trigger will succeed on Tuesday. Governance tracks patterns across sessions.
Evidence trails. When a regulator, a customer, or your own board asks for the record, best-effort telemetry that sampled away the interesting spans does not answer the question. An evidence trail is tamper-evident, complete, and reconstructable.
OpenAI and Anthropic both built runtime governance on top of observability for their own agents. OpenAI’s Preparedness Framework describes a monitor that supervises agent actions for harm and forces human approval before logging all actions in an uneditable database with asynchronous review. That setup goes beyond a latency dashboard. It includes full-history capture, a supervising monitor, forced human approval, and an immutable action log.
The two most safety-conscious AI companies on earth looked at their own agents. They decided observability was not enough and built governance on top of it.
We cover the governance layer in depth in a separate post. Observability is the foundation. The foundation has to be solid before you build on top of it.
Four capabilities that observability tools do not provide, each sitting in the gap between seeing and governing
What We Learned Building Observability Into Production Agents
We run three agentic systems in production. PASSR reviews every pull request. DOCKR generates documentation from codebases. TESTR generates test cases from code.
Structured signals beat raw logs
Early on, PASSR logged every finding as text. A developer would dismiss a finding, and the log said “finding dismissed.” That told us something happened. It did not tell us why. We switched to structured signals: “developer applied the suggested fix,” “developer replied with a challenge and the finding was withdrawn,” “developer dismissed without reading.” The granularity of the signal determines the quality of the loop. A vague signal produces a vague system. With structured signals, you can detect patterns.
Memory is the simplest form of closed-loop observability
DOCKR does not regenerate documentation for the entire codebase on every push. It analyses the diff, identifies what changed, and updates only the affected files. This requires persistent state. Without it, the system starts from scratch every time. With it, processing time stays bounded even as the codebase grows.
An agent that cannot remember what it did last time cannot improve on it. Memory is what turns point-in-time observability into pattern tracking. The trajectory of past executions is the raw material for detecting drift and proving compliance.
The loop needs time to compound
On day one, TESTR had no execution history. Its suggestions were based on code analysis alone. After 90 days of execution data, its suggestions are informed by patterns of failure and coverage gaps. The system gets better, but only if you give it enough cycles to close the loop.
Observability is what makes the loop visible. Without it, you cannot tell whether the system is improving or degrading. You cannot tell whether a change you made helped or hurt. You are flying blind.
Human feedback is the primary signal
Human judgment is the highest-quality feedback signal available. It captures business logic and team conventions that no environment signal can encode. In PASSR, the human decides whether to apply a fix. DOCKR sends documentation to a human for review. TESTR requires human approval before any test case runs.
This is also the most expensive signal. It does not scale the way external feedback does. But it is the signal that keeps the other signals honest. An agent that auto-applies feedback without human review can fit to noise. It can game the metric. The human review step is what keeps the feedback signal aligned with what you actually care about.
Cost bounds, not cost alerts
We learned this the hard way. An agent without a cost bound will spend until it succeeds or until it runs out of budget. A cost alert tells you the money is gone. A cost bound stops the spending before it reaches that point. The difference matters when the edge case is $47,000.
Every agent we ship now has execution bounds: step limits, cost limits, iteration limits, and timeout thresholds per sub-task. When a bound triggers, execution stops. The agent reports back. A human decides what to do next.
Three products, three hard-won lessons in observability
A Practical Observability Checklist
If you are shipping agents to production, here is what to verify.
- Can you reconstruct the full trajectory? From user intent through every model call and tool invocation to the final output. Not fragments. One linked trace.
- Are tool calls captured as first-class records? Not buried in application logs. Tool name, arguments, results, timestamp, and the model call that triggered them.
- Can you detect behavioral drift across sessions? Not just within a single run. Patterns tracked over time, correlating incidents and deviations across executions.
- Do you have execution bounds? Circuit breakers, not alerts. Step limits, cost limits, iteration limits. When a bound triggers, execution stops.
- Can you produce an evidence trail? Tamper-evident records that a regulator or auditor can read. Decision ID, model version, prompt hash, retrieved context, input, output, human oversight action.
- Is human oversight measurable? Intervention rate. Override rate. Approval latency. How many approval requests expired with no human decision. If you cannot measure oversight, it is decorative.
Key Takeaways
- Observability is the foundation of governance. Without traces and tool-call records, you cannot audit decisions, detect drift, debug failures, or prove compliance. You cannot govern what you cannot see.
- Observability and governance answer different questions. Observability asks what happened. Governance asks whether it should have happened. You need both. One without the other is either a dashboard or a policy with no verification.
- Traditional monitoring breaks for agents. Non-deterministic execution, compounding errors, unbounded cost, and behavioral drift are failure modes that traditional monitoring was not built to handle.
- The trajectory is the unit of visibility. One user intent, many model calls, many tool invocations. Disconnected fragments are not enough. The trace has to link them.
- Observability stops where governance begins. Execution bounds, policy enforcement, behavioral pattern tracking, and evidence trails are capabilities that observability tools do not provide. They sit in the gap.
- Cost bounds, not cost alerts. An alert tells you the agent already spent $100. A bound stops it from spending $101. The difference is $46,900.
Related Reading
👉 Feedback Loops in Agentic AI Systems
The loop drives improvement. Observability makes the loop visible. Without observability, you cannot tell whether the loop is working.
The five core components of agentic systems and what makes them different from the AI tools you are already using.
👉 AI Code Review: From Nice-to-Have to Mission-Critical
How PASSR uses structured signals, resolution tracking, and per-issue conversation to make agent behavior auditable.
Working with FLYTEBIT
At FLYTEBIT TECHNOLOGIES, we build agentic AI systems that run in the background so your team can focus on what matters. See how agentic AI systems work across the full development pipeline.
PASSR is our AI code review tool. DOCKR is our AI documentation generator. TESTR is our AI test generation platform. All three are built with observability and human-in-the-loop governance from day one.
If you need AI software development services that cover the full pipeline, from strategy to deployed agents, start with a conversation.
Ready to get started?
- Visit us at: flytebit.com
- Follow FLYTEBIT TECHNOLOGIES on LinkedIn for insights and updates
- Schedule a free consultation to discuss your specific use cases
Ready to Transform Your Business with AI?
Let's discuss how Agentic AI and intelligent automation can help you achieve your goals.