The Latency Number On The Pricing Page Is A Lie Of Omission

There is a moment about 1.3 seconds into a bad voice-AI call where you can hear the customer decide to hang up. The agent has finished processing, the response is coming, but the silence has already lasted long enough that a human brain registers it as "no one is there." By the time the audio starts, the caller is reaching for the end-call button. That gap is the single most important number in a voice agent, and it is almost never the number a vendor puts in front of you.

Here is what the marketing usually shows: a sub-250-millisecond text-to-speech figure, or a "500ms" headline pulled from an ideal single-turn test. Here is what production actually looks like, from this year's benchmarks across real deployments: median end-to-end response latency around 680 milliseconds at p50, but p95 stretching to roughly 1,180 milliseconds — and in cascaded speech-to-text to language-model to text-to-speech stacks, practical total pipeline latency runs 1.5 to 3 seconds from the end of the caller's speech to the start of the agent's audio. The recommended targets people quote — p50 under 1.5 seconds, p95 under 3.5 seconds — sound comfortable until you notice the quieter finding: latency above about 800 milliseconds already feels sluggish, and beyond 1.5 seconds callers start to check out.

Why the average lies so effectively

A voice call is not one turn, it is dozens, and abandonment is decided by the worst ones, not the typical ones. A pipeline that spends 100 to 300 milliseconds on speech-to-text, 350 to 1,000 milliseconds on the language model, 90 to 200 milliseconds on text-to-speech, and another 50 to 200 milliseconds on network round trips has a comfortable median. But add a database lookup mid-turn — checking an appointment slot, verifying an account — and that single turn blows past two seconds. Add a language switch, a barge-in where the caller interrupts, a retry after a mishearing, and you have a handful of turns per call that each land in the "caller checks out" zone. The customer does not experience your p50. They experience your worst three turns.

Latency is a retention metric, not an engineering one

This is why "fast enough" is the wrong frame. Engineering asks "what is our median time-to-first-token?" Retention asks "on what fraction of calls did at least one turn exceed the abandonment threshold, and did the caller stay?" Those are different questions with different owners. The first is a dashboard; the second is revenue.

If you are evaluating or building a voice agent, the honest measurement is not a single latency number at all — it is a distribution, measured on your real traffic, with the tail called out explicitly. A vendor who will only quote you a p50 is either not measuring the tail or hoping you will not ask.

Architecture follows from the measurement

The architecture choices flow from this. Speech-to-speech models are the fastest path and increasingly the default for latency-sensitive conversation, because they collapse the chain. Cascaded stacks are cheaper at scale and far easier to control and instrument — you can see exactly which component ate the time, swap a faster speech-to-text, or cache a slow lookup. The right answer is genuinely workload-dependent: a simple, high-volume flow may not justify the control tax of a cascade, while a flow with mid-turn database work and strict compliance logging often does. What is not defensible is choosing the architecture first and discovering the latency profile in production, which is the order most teams accidentally do it in.

The design lever most teams underuse

There is also a lever that has nothing to do with model choice: hiding latency with conversational cover. A human receptionist who needs three seconds to find something does not go silent — they say "let me pull that up." A voice agent can do the same, filling the lookup gap with a natural acknowledgment so the caller never hits the silence that triggers abandonment. It does not make the pipeline faster; it makes the slow turns survivable. When people say a voice agent "feels human," a large part of what they are responding to is that its slow moments were covered, not that its fast moments were fast.

If you take one thing from this: put the p95 in the eval harness and watch it every week, because it drifts. A new model version, a slower database, a traffic spike, and your tail quietly gets worse while your median looks fine. The teams shipping voice agents that people actually finish talking to are not the ones with the lowest number on a slide. They are the ones who know their worst number, measure it honestly, and design around the silence.