Evaluate the state an agent leaves behind

Start where the transcript stops

Consider a hypothetical support agent asked to refund an eligible order. It calls a payment tool, receives a timeout, retries and tells the user the refund is complete. A transcript evaluator might reward the answer for being clear and helpful. The business needs a different answer: how many refunds committed, against which payment, under whose authority?

The timeout separates observation from reality. It might mean the request never arrived, the service rejected it, or the refund committed and the response disappeared. Treating these possibilities as equivalent hides the failure that matters. Evaluation must inspect authoritative records, not infer a successful transition from confident language or a successful HTTP response.

Begin with a bounded workflow, such as refunding one order through a named tool contract. Identify its entry conditions, allowed actions and terminal outcomes. This is an evaluation design exercise, not a claim that any particular integration reproduces every production dependency.

Turn policy into observable invariants

Before generating conversations, write the conditions that must remain true. Separate safety properties, which prohibit a bad transition, from eventual completion requirements. An agent that refuses every request can satisfy a refund limit while failing its purpose. Conversely, successful completion does not excuse an unauthorized intermediate action.

Specify currency handling, partial refunds, pending entries and observation timing alongside the rule. A check over cached application state is not equivalent to a check over the committing ledger. If authoritative evidence is unavailable, record an inconclusive result rather than quietly passing the run.

Construct the failure as a sequence

Now place disturbances around the transition you want to understand. In the hypothetical refund workflow, lose the response after commit, deliver an old status read, then introduce a second worker handling the same request. The sequence tests whether uncertainty triggers reconciliation or another irreversible action. A generic instruction to “simulate an API error” leaves these distinctions unspecified.

Vary one causal factor first. Compare a lost response before commit with one after commit while holding the initial records and user request constant. Then combine faults deliberately, documenting which interaction the combined scenario probes. Otherwise a failure may be interesting but impossible to attribute.

Include legitimate alternatives: two separate refund requests should not automatically collapse into one, and a denied action may need escalation rather than silence. These counterexamples prevent a guard from looking correct merely because the scenario collection rewards inaction. Keep scenario selection separate from the release threshold so the suite cannot become a collection of convenient passes.

Distinguish replay from re-execution

A trace records a path; a snapshot records state at a boundary. Neither alone recreates an execution. OpenTelemetry describes traces as the path a request takes and spans as units of work, with identifiers and context connecting them [1]. That structure helps correlate agent decisions, tool requests and downstream activity. It does not automatically capture database contents or make an agent deterministic.

For an inspectable run, retain a starting snapshot identifier, schema and policy versions, agent configuration, tool inputs and outputs, scheduled faults and durable state transitions. Control clocks, random seeds and event ordering where feasible. Model sampling, concurrent workers, changing remote services and unrecorded reads can still introduce nondeterminism.

Label two different operations honestly. A recorded replay displays the captured events and checks their recorded consequences. A fresh re-execution restores an agreed snapshot and runs the agent again. It may take a different path. Feeding recorded tool responses to a new agent can isolate decision behavior, but no longer tests live tool semantics or the consequences of different calls.

Make the outcome checker independently inspectable

Evaluate transitions as well as final state. An unauthorized refund followed by a compensating adjustment can leave the balance looking correct while still violating policy. Join the action history to the final records, using stable operation identifiers rather than matching text descriptions.

Keep the checker simpler than the behavior under test. Deterministic rules can verify ledger limits and approval relationships; human review may be needed for ambiguous policy interpretation. If a model assists that review, preserve its rubric and version, and do not let its judgment override a concrete financial violation.

Test the checker itself with deliberately corrupted fixtures: a missing refund entry, a mismatched payment identifier, an approval issued after the action. Confirm that each produces the expected failure or unknown state. This costs maintenance effort, especially as schemas evolve, but prevents a broken oracle from turning missing evidence into apparent reliability. Capture the query or rule version with every result.

Use the result to narrow the next decision

Report which configurations and scenario families were exercised, how many executions completed, and which checks were inconclusive. Repeated runs from one snapshot help expose nondeterminism, but do not establish coverage of different customers, policies or workload conditions. A clean result is bounded evidence, not a universal safety claim.

The practical decision may be to block automatic refunds while allowing read-only investigation, add a reconciliation step, or expand tests around concurrent requests. Name the residual uncertainty and the owner responsible for resolving it. More realistic state and scheduling improve diagnostic power, but also increase setup costs and the surface that must be maintained.

NIST positions its AI Risk Management Framework as voluntary guidance for incorporating trustworthiness considerations into design, development, use and evaluation [2]. A workflow evaluation can contribute to that broader work. It cannot replace security review, operational controls or an accountable release decision. The useful outcome is a precise explanation of what was tested and what remains unknown.