A voice agent feels natural when it answers quickly and never drops out. Tried approaches both from three sides: a fast default pipeline, provider fallbacks when a vendor fails mid-call, and measurements on every call so you can see what happened.
How a turn works#
Each conversational turn on the default (cascading) pipeline goes through four stages:
| Stage | Default provider | What adds time |
|---|---|---|
| End of turn | LiveKit turn detector (multilingual model) with VAD | Waiting to be sure the caller finished speaking |
| Speech to text | Deepgram Nova-3 | Time from end of speech to the final transcript |
| LLM | OpenAI gpt-5.6-terra |
Time to the first token of the reply |
| Text to speech | Premium voice (eleven_v4_turbo) |
Time to the first audio byte |
The runtime already overlaps these where it safely can:
- Preemptive generation. The LLM starts on the interim transcript before the turn is committed. Tools only run once the turn is committed, so no side effect happens on a guess. TTS waits for confirmation, so no wasted audio is played.
- Warm connections. STT, LLM and TTS sockets open while the call is ringing, and the LLM connection is primed with the agent's prompt and tools, so the first turn does not pay the handshake.
- Fast lanes.
model_high_priorityandmodel_ultrafastrequest the provider's priority tier; ultrafast also lowers reasoning effort on GPT-5 models and turns off thinking on Gemini.
Speech-to-speech engines (gpt-realtime, gpt-realtime-mini) replace the three middle stages with one model.
Where to see latency#
| Where | What it shows | Page |
|---|---|---|
| Agent builder header, Latency | An estimate from the selected models, before any call | Estimated latency and cost |
Call drawer, latency popover, and the call object's latency |
Measured per-turn values with percentiles | Actual latency |
Analytics (GET /analytics/summary) and alerting |
End-to-end p50, p90, p99 and average across calls; alert rules on e2e_latency_p50 and e2e_latency_p90 |
Actual latency |
When a provider fails#
| Failure | What happens | Configure |
|---|---|---|
| Voice (TTS) errors | The agent's Fallback Voices, then the workspace fallback voice, are tried in order | Agent: Security & Fallback Settings. Workspace: Reliability settings |
| LLM errors or sends no first token in time | The turn is retried on the workspace LLM fallback model | Reliability settings |
| Speech recognition fails | The workspace Speech recognition fallback provider takes over | Reliability settings |
| A failure with no fallback | The call ends with error_llm, error_asr or error_tts |
Debug disconnections |
Troubleshooting guides#
Troubleshoot latencySettings that make the agent answer faster.Debug disconnectionsEvery
disconnection_reason and what to do.Debug outbound callsWhy an outbound call was refused or never connected.Debug transfersCold, warm and agentic transfers that fail.Transcription accuracyMisheard words, names and numbers.Wrong responsesThe agent says the wrong thing or calls the wrong tool.Prevent abuseRate limits, public key checks, spend controls and fraud.Account limits that affect reliability#
Calls are refused, not dropped, when the workspace is out of concurrency, credit or budget. See Limits and concurrency and Billing.