A voice agent's transcript is a claim, not evidence
Enterprise voice agents answer phones for regulated businesses, and they do real work during the call: they verify identity, record payments, look up accounts, and transfer callers to human specialists. A mistake costs money or a customer, and the transcript can hide it. The agent can say "your payment is recorded" when the tool failed, returned nothing, or never ran.
I built the parts of the evaluation and debugging path described below: the console engineers use to compare an agent change against an approved baseline, the APIs behind it, tool-result capture during simulated calls, and the case catalog that decides what counts as "the same test." I also fixed call-state defects in the live runtime. The rule throughout: score what the agent did, keep what was never judged out of the score, and never let new evidence overwrite old evidence.
Store the tool's return value, not only its arguments
In an evaluation run, a model plays the caller, the agent runs its normal logic, and a grader judges the result against written expectations. The record kept which tools the agent called, with which arguments, but not what came back.
I added completed tool results to each turn. The runner pairs every tool invocation with a runtime execution from the same turn: by tool name, and by arguments when the name repeats. If the match is still ambiguous, it attaches nothing instead of guessing. Only completed executions qualify, and each output is redacted before it is stored. The console shows arguments and return value next to the agent's words. New runs get this evidence; old verdicts are not regraded.
An execution error is not a failed behavior
If a backend times out, the agent's behavior was never tested. Counting that as a failure makes a change look worse and sends the engineer to the prompt instead of the infrastructure.
The comparison APIs I wrote compute the pass rate from scored attempts only. Conversations that did not finish appear as their own count, not as passes or failures. When a candidate is compared with an approved baseline, a case is excluded if either run has unscored attempts or if the two runs executed it a different number of times, and the comparison states how many cases it excluded for each reason.
An experiment must not move the record
Originally, a quick rerun of one case was recorded like a full nightly run: it added an almost empty column to the results matrix and moved the suite's pass rate. One person's experiment changed everyone's view.
I separated scratch runs from the tracked record. Scratch runs stay out of the matrix, suite metrics, and case verdicts, so a quick pass does not turn a failing case green. A finished, fully scored batch can be promoted on purpose. A rerun reads its batch's own case IDs, so a three-case retry cannot silently widen into the whole suite. Selected reruns resolve to the suite's current cases, so a rerun tests today's definition. It adds evidence; it does not reproduce the original attempt.
A display name is not an identity
The case catalog derived each case's identity from its suite, category, and goal. Two cases with the same goal but different assertions received the same identity, and the surviving assertion depended on row order in the source file. One test silently replaced another.
I rebuilt the loader. Exact duplicates collapse. Distinct assertions under one name get content-derived variant IDs, and authors can pin an explicit ID that survives any edit. Duplicate identities fail validation, and a failed sync blocks the service from becoming ready. A regression test confirms that reordering the source rows produces identical cases. Edited cases get new versions; historical definitions and their runs stay intact.
A late event from one call must not change the next
The runtime holds a long-lived connection to the contact-center system that outlives each call. Events arrive late: a disconnect after cleanup has finished, a consult-leg success after a transfer was cleared, an event from the previous customer's transfer. When state belongs to the connection instead of the call, those events corrupt the current call.
Transfers make this concrete: a classifier hearing a live person is not that person accepting the caller. I fixed the transfer and disconnect paths so that a late disconnect keeps its telemetry but cannot restart cleanup, a new call drops the previous call's transfer-cleanup state, a late consult success cannot revive a cleared call, and a stale consult event cannot replace the current one. Tracing in the persistent worker is also isolated per call. Each fix has a regression test.
A release is four facts, not one
A source fix, a built artifact, a deployed environment, and a retest are separate claims. A passing unit test proves only the first, so I report each one on its own evidence.
Limits
I did not build the whole evaluation platform. The changes above are implemented with regression tests; I make no support-cost or production-quality claims. The demo is a generic reconstruction with fictional scenarios, responses, and scores.