An AI investigator is only useful if it cannot invent evidence
Scout is an investigation system for our voice-agent calls. It finds specific call behaviors inside populations of tens of thousands of calls. I designed and built all of it myself: ingestion, embeddings, indexing, the agent tools, the service, and its deployment. A teammate asks a question, such as why payment calls stall after a customer offers a future date. An AI agent then works through masked call transcripts and runtime traces with Scout's tools: semantic search, cohort building, event-sequence search and full-call reads. It records hypotheses, findings and decisions as it goes. Every finding must cite specific turns in specific calls, and Scout checks those citations against the source.
That check is the core idea. In the research harness that became Scout, the code rejected a submitted finding if it cited a call the model had not opened, a turn it had not seen, a quote that did not match the source exactly, the wrong speaker, or a customer turn that did not come before the agent's response. Invented evidence fails a structural check before it reaches a reviewer.
The workspace above reconstructs that loop. Pick a cohort, compare its routes, draft a finding, and see it stay a candidate until the sources support it.
I built every layer, research harness first
The first version was Call Behavior Lab. It had a bounded, read-only exporter, frozen call sets with whole-day splits, cached Titan embeddings over dialogue windows, deterministic action features, cohorts and route graphs at several resolutions, and a budgeted agent loop. The model chooses what to search and read. Code enforces the scope, budgets, source access and evidence rules.
Scout turns that into a service. An ingestion worker indexes calls with privacy checks and resumable checkpoints. OpenSearch holds separate, versioned vectors for passages, runtime events and hybrid cohorts. An authenticated MCP API lets coding agents investigate one tenant at a time, and it revalidates source hashes when it reads evidence. A CLI journals each investigation and renders a report and a local viewer. The services run on AWS ECS.
Scout grades findings by their evidence. A finding is a supported problem only when it has prior-state, trigger, response and consequence turns with verified source hashes, plus evidence that the behavior was avoidable and harmful. Anything less is saved as an observation. It is not discarded, and it is not promoted.
More model proposals do not mean more findings
In a matched benchmark of 18 investigations, models made 27 proposals. Of 74 calls that passed a first review, only 34 survived the independent check. Finding candidates was easy. Verifying them was the bottleneck.
So an empty result is a valid result. An agent can submit no findings with a stated reason. A run that exhausts its tool or time budget is recorded as incomplete, which is different from a clean "no finding" and different from a transport failure.
Similarity, order and source review are three different signals
Semantic search finds calls that sound alike. In one 20,000-call pilot, the neighbors of an unusual call mostly shared its payment topic, and a source screen found no other exact copy of its behavior. Event-sequence search shows what happened before and after an action, but two phrases in the same call do not prove their order or meaning. Only a full-call read confirms the behavior. Scout keeps these signals separate in its tools and in its reports.
The same rule decided representation choices. I tested detailed model-generated turn labels against cheap deterministic action features. The labels cost more and did not produce more independently supported discoveries, so deterministic actions stayed the default.
A count is not a rate without a denominator
A literal search found 54,090 calls that contained one repeated prompt. A stratified review of 200 of those calls found 12 strong examples of the actual failure: the agent asked again about paying today after the customer had already given a future date. The weighted estimate was about 720 affected calls (interval 244 to 1,585), roughly 75 times fewer than the search count.
Scout's reports therefore keep stored population, calls actually read, candidates returned and confirmed support as separate numbers. When source coverage is unknown, a zero is reported as inconclusive, not as zero calls.
A real match is a reason to look, not proof of cause
One discovered behavior was the agent losing an amount the customer had already given and restarting payment selection. It matched a bug that a customer had already reported, which made it worth engineering review. It did not establish a shared cause: the ticket did not identify the same calls, and a later holdout review did not reproduce the defined behavior. Scout carries that uncertainty into the handoff instead of turning it into a fix.
Limits
Model agreement is not accuracy. Quote checks prove that cited evidence exists, not that the judgment is correct, and verifier precision has not been measured against human-adjudicated labels. Scout supports investigation and engineering review; it does not change agent behavior or prove business impact by itself. The workspace above uses a fully synthetic dataset, and its counts describe that fixture only.