What it is
Each turn of a call is a relay: the caller stops talking, speech-to-text sends a transcript, the model replies, and text-to-speech streams it back. If any step stalls, the caller hears silence, and on a phone call silence sounds like a dropped line.
These are bugs from that path on our bank deployment. I chose what to chase, read the evidence and decided the fixes. Coding agents wrote most of the code.
Where the 2.7 seconds went
One turn spent 2.726 s in the model step. A rebuild of the call log showed that 1.413 s of it was our own setup, before the provider saw the request. That setup included four uncached lookups of the model policy.
Now the policy is resolved once when the call arrives and pinned for the call. We also added a warmup to prime the provider's prompt cache. It turned out to be failing: it asked for 1 token, and the provider's minimum is 16. That's fixed too.
Setup is off the reply path. A clear end-to-end speedup isn't proven yet, so I'm not claiming one.
The silence that was a 400
An agent went quiet on a staging call. When a reply chunk was only whitespace, we still sent the speech provider an empty "end of text". It answered 400, and our retry reused a closed tokenizer.
The fix skips empty chunks and retries only before the first audio plays, never mid-sentence. 59 tests cover it. On the next call, first speech audio came in 0.237 s.
It then went silent after the introduction, for a second reason: an old model client had streaming switched off, so speech got no text. After that fix, a scripted test caller and my own phone call both heard the agent answer.
The second call that heard nothing
Agents there keep one phone line open across calls. Call one worked, and call two got near-silent audio. A harness that runs two calls through the real audio component reproduced an earlier form of the failure. Narrowing down merges then pointed to a likely cause: a cleanup change from main that closes the session after every call. That cause is still being confirmed, and there's no fix yet.
Does it work?
For the latency and speech fixes, yes: they run in staging and pass real test calls. The second-call bug isn't fixed yet, and I don't have production confirmation or numbers across many calls.