Troubleshoot latency

Start from measurements. Open a slow call, read the latency breakdown and find the stage that dominates. Then change the matching setting below, place a few test calls and compare the medians.

Model and prompt#

The LLM's time to first token is usually the largest single stage.

Choose a faster model. The builder's estimate ranges show the relative speed of each model:

Faster models are also cheaper per minute, but may follow complex prompts and tools less reliably. Test with simulations before switching a live agent.

Turn on a fast lane. model_high_priority requests the provider's priority tier (service_tier: priority on OpenAI). model_ultrafast adds the lowest reasoning effort and short answers on GPT-5 models, and disables thinking on Gemini. Both bill the LLM line at 1.5 times (the llm_priority line). Claude 5 models already run at low effort in the voice pipeline.

Shorten the prompt. Every turn sends the system prompt, tools and recent history. Long prompts take longer to process and, above 4,000 tokens, also raise the LLM cost linearly. The builder's Tokens estimate shows the size. Move reference material into a knowledge base, and cut tool descriptions to what the model needs.

Long calls. History grows each turn. The workspace LLM token limit (default 32,768) caps it by dropping the oldest turns; a lower limit keeps late turns fast. Replies are capped at 300 completion tokens.

Voice#

Text to speech adds its time to first audio to every turn.

Keep the first sentence of each reply short. Speech starts after the first chunk of text is ready, so a short opening clause reaches the caller sooner. Phone calls use 16 kHz PCM output, so no decode step is added.

Transcription and turn-taking#

Most of the time between the caller finishing and the agent starting is the decision that the caller has finished.

Responsiveness. responsiveness (0 to 1, default 1) sets how long the runtime waits after speech stops:

min_delay = 0.3 + (1 - responsiveness) x 1.2   seconds
max_delay = 2.0 + (1 - responsiveness) x 3.5   seconds

At 1 the wait starts at 0.3 seconds. The turn detector extends it toward max_delay when the words look unfinished. Lower values wait longer, which suits callers who pause mid-sentence but adds delay to every turn.

Dynamic responsiveness. enable_dynamic_responsiveness lowers responsiveness for a caller who keeps resuming after a pause, and gives it back as their turns end cleanly. It avoids cutting off slow talkers without slowing everyone.

Transcription mode. stt_mode changes Deepgram's endpointing:

Use accurate only when number and date formatting in transcripts matters more than speed. With custom, keep endpointing_ms low.

Turn detector. The runtime uses LiveKit's multilingual turn detector by default. Operators can change it with TRIED_TURN_DETECTOR (multilingual, english, audio or none for VAD only). With none, end of turn relies on silence alone, which is slower for the same rate of cut-offs.

Preemptive generation is always on in the cascading pipeline: the LLM starts on the interim transcript, for utterances up to 10 seconds long. Tools never run and no audio plays until the turn is committed.

Tools and knowledge base#

A turn that calls a tool waits for the tool before the agent can answer.

• Keep custom function endpoints fast and close to the runtime. Return only what the model needs.
• Use speak_during_execution on slow tools so the agent says something while waiting.
• Large knowledge bases add retrieval time each turn. The knowledge_base latency field shows it.
• For a Custom LLM, host the WebSocket server in the same region as the runtime. The llm_websocket_network_rtt field shows the round trip.

First turn#

• Pause Before Speaking (begin_message_delay_ms, 0 to 5,000 ms) delays the greeting on purpose. Set it to 0 unless you need it.
• A static welcome message (begin_message text) is spoken directly, with no LLM call. An unset begin_message makes the LLM generate the greeting.
• Provider connections open while the phone rings, and the LLM is primed with the agent's prompt, so the first reply is not slower than later ones.

Region#

Network distance is added to every stage. For Australian callers:

• Run the voice runtime and media in Sydney (deployment).
• Set DEEPGRAM_BASE_URL=https://api.au.deepgram.com to use Deepgram's AU endpoint.
• Run python -m latency_lab.net_probe and latency_lab.vendor_probe from the runtime host to see the round trip to each vendor.

Speech-to-speech#

Speech-to-speech engines (gpt-realtime 500-900 ms, gpt-realtime-mini 400-700 ms estimated) skip separate transcription and synthesis, but they are not faster than a tuned cascading pipeline and cost $0.15 to $0.45 per minute for the engine alone. Try them for their voice quality, not for speed.

Checklist#