Skip to content
TriedDocs
Custom LLM

Custom LLM best practices

Keep custom LLM calls fast and stable: stream early, honour response_id supersession, keep the socket alive and host near the media region.

title: Custom LLM best practices nav: Best practices description: Keep custom LLM calls fast and stable: stream early, honour response_id supersession, keep the socket alive and host near the media region. order: 5#

These practices follow from how the Tried worker uses your server. Each one names the behaviour in the worker that it relies on.

Stream the first words fast#

The worker starts text to speech as soon as the first response chunk arrives. The caller hears nothing until then, so the time to your first chunk is added directly to every turn.

  • Stream model tokens straight through as response frames with content_complete: false. Do not wait for the full reply.
  • Send a short first chunk ("Sure,", "Let me check.") when your model or retrieval is slow to start.
  • Send content_complete: true on the last frame. An empty final frame ("content": "") is fine.
{ "response_type": "response", "response_id": 6, "content": "Sure,", "content_complete": false }
{ "response_type": "response", "response_id": 6, "content": " I can help with that.", "content_complete": false }
{ "response_type": "response", "response_id": 6, "content": "", "content_complete": true }

Treat response_id as a cancellation signal#

When the caller interrupts or keeps talking, the worker sends a new request with a higher response_id and drops every frame for older ids. Work for an old id is wasted.

  • Keep one current id per connection. When a new response_required or reminder_required arrives, cancel the model call for the previous id.
  • Never reuse an id, and never answer with a lower id than the last one you received.
  • Build the reply from the transcript in the request, not from update_only frames. Updates are sent without waiting and can cross with the request.

Send config first, every time#

The worker waits 0.5 seconds after opening the socket for your config frame, then decides whether to send call_details and start pings. It does this again after each reconnect.

  • Send config before any other frame, including the opening line.
  • Turn on auto_reconnect in production. Without it, any socket close ends the call with error_llm_websocket_lost_connection.
  • Turn on call_details if you need the caller's number, metadata or dynamic variables.

Keep the keepalive cheap#

With auto_reconnect, a missing ping_pong reply for 5 seconds triggers a reconnect, and the third failure in a call ends it.

  • Answer ping_pong on the same event loop that reads the socket, and never behind a model call. If your server blocks while generating, pings queue up and the worker reconnects in the middle of a turn.
  • Echo the exact timestamp. The echo is what gives you the llm_websocket_network_rtt latency on the call.

Handle reconnects#

After a reconnect the worker connects to the same {url}/{call_id}, waits for config, resends call_details if asked, and resends the in-flight response_required with the same id and transcript.

  • Key per-call state by call_id (the last path segment) in a store that outlives one socket, so a reconnect to another instance can continue.
  • Make a repeated response_required with the same id safe to answer again.

Write replies for speech#

Your text goes straight to text to speech.

  • Keep each turn to one or two short sentences and ask one question at a time.
  • Write numbers, dates and amounts the way they should be spoken.
  • Avoid markdown, lists and emoji.

Use reminders deliberately#

reminder_required arrives after reminder_trigger_ms of silence (default 10000), up to reminder_max_count times (default 1). Configure both on the agent. Reply with a short check-in that refers to the last question. Use end_call: true on the reply if you want to hang up after the last reminder.

The agent's end_call_after_silence_ms setting still applies. It ends the call with reason inactivity without asking your server.

Use no_interruption_allowed sparingly#

Set no_interruption_allowed: true only for text the caller must hear in full, such as a legal disclosure. It applies to the opening line, agent_interrupt and reminder replies. Long uninterruptible speech feels unresponsive.

Host close to the media#

Every turn crosses the network between the voice worker and your server at least once. For Australian numbers, run your server in Sydney next to the worker. Check the round trip with latency.llm_websocket_network_rtt on finished calls.

Validate frames#

The worker ends the call with error_llm_websocket_corrupt_payload when a frame is not valid JSON, has an unknown response_type, or lacks a required field. Build frames with a JSON library, and validate them against the outbound shapes on WebSocket server before sending.

Log per call#

Log the call_id, every response_id you receive and the time to your first chunk for each. Compare with the call's transcript and latency in Call History when a turn felt slow.