Your agents pass their evals and still break your processes.

The eval dashboard is green. Response quality is up, hallucination rate is down, the golden test set passes at 97 percent. By every measure your AI team reports, the customer-service agent is performing well.

It is also, this quarter, the system that issued a refund above the approval threshold without routing to a manager, closed a compliance-flagged ticket without the mandatory review step, and updated a customer record without writing the change note your auditors expect on every account modification.

None of those events touched an eval. All of them are incidents.

Evals answer the wrong question

An eval asks: given this input, is the output good? It's the right question for model selection and regression testing, and teams have gotten genuinely rigorous about it. But it evaluates the agent as a text-producing function.

In production, your agent is not a text-producing function. It's a participant in a business process — and processes have requirements that are orthogonal to answer quality:

• Check entitlement before issuing the refund, not after.
• Obtain approval when the amount exceeds the threshold — every time, not usually.
• Write the audit entry for every record modification, even when nothing interesting happened.
• Escalate to a human when the confidence conditions aren't met, rather than proceeding gracefully.

A model can satisfy every quality bar you set and violate all four. The refund email it writes without approval is fluent, accurate, and empathetic. That's precisely what makes the failure invisible to output-based evaluation: the output is fine. The path to it wasn't.

The failure is silent by construction

When a human employee skips a mandatory approval, there's often friction that surfaces it — the form has a required field, a colleague notices, the weekly review catches it. When an agent skips a step, there is no friction at all. The agent doesn't throw an exception on a missing approval; skipping it is a well-formed execution. The workflow completes, the customer is satisfied, latency and uptime look great.

Your existing monitoring stack is built to catch systems that stop working. An agent that breaks process is working — smoothly, politely, at scale. Uptime green, cost per ticket down, and a growing pile of unauthorized actions that nobody will see until an auditor, a regulator, or an angry customer goes looking.

This is worth stating plainly: for agentic systems, the dangerous failure mode is not degraded output. It's conformant-looking execution that quietly departs from the defined process. Evals cannot see it because the deviation isn't in the text. Infrastructure monitoring cannot see it because nothing is down.

Borrow the discipline that already exists

Enterprises solved this problem once, for humans and ERP systems, under names like process controls, segregation of duties, and conformance checking. The discipline translates directly; only the actor is new.

1. Define the workflow explicitly. Not in a policy paragraph — as a checkable sequence: steps, ordering constraints, required evidence per step. If the process only exists as prose, deviation is a matter of opinion.
2. Instrument steps as events. Each meaningful step the agent takes should emit a record: what, when, on whose authority, with what inputs. The systems the agent touches keep their own logs; the process-level trail — this run, these steps, this order — is the thing nobody captures by default.
3. Check conformance per run, not per quarter. Compare each execution against the defined workflow while the run is fresh. A skipped step should page someone the way a failed health check does — because for an agent, it is one.
4. Keep the record. When someone eventually asks "did the agent follow the process on the Hendricks account," the answer must be a lookup, not an archaeology project across six systems' logs.

Teams sometimes object that this is heavyweight. It's the opposite of heavyweight: it's a step list and a diff. What's heavyweight is reconstructing agent behavior after the fact from raw model transcripts — which is what every team without process instrumentation ends up doing, at the worst possible time, under the least favorable scrutiny.

Keep the evals; they're necessary. But when your auditor — or your board — asks how you know the agents follow the processes they're embedded in, "our eval scores are strong" answers a question nobody asked. The question they asked has a one-word answer today in most organizations, and the word is we don't. The teams that fix that before their first process incident get to fix it calmly.