WebSocket server protocol

This page is the full reference for the custom LLM WebSocket protocol as implemented by the Tried voice worker (services/agent, custom_llm.py). The frame shapes are defined in packages/shared as CustomLlmInbound (Tried to your server) and CustomLlmOutbound (your server to Tried). They match Retell's custom LLM protocol.

Connection#

For every call, the worker opens one WebSocket client connection to:

{llm_websocket_url}/{call_id}

A trailing slash on llm_websocket_url is removed before the call id is appended. For wss://llm.example.com/llm-websocket and call call_8f2c..., the worker connects to wss://llm.example.com/llm-websocket/call_8f2c.... Route on the last path segment to get the call id.

• Every frame is a JSON text frame. Binary frames are ignored.
• Tried frames carry interaction_type. Your frames carry response_type.
• The connection is opened before the agent joins the call. If it cannot be opened, the call ends with error_llm_websocket_open.
• Plain WebSocket control pings are answered automatically. The application-level keepalive is the ping_pong frame below.

Sequence of a typical call#

worker  -> connects to {url}/{call_id}
server  -> config { auto_reconnect: true, call_details: true }
worker  -> call_details { call: {...} }
server  -> response { response_id: 0, content: "Hi, this is Ava...", content_complete: true }
worker  -> ping_pong (every 2 s)          server -> ping_pong (echo)
worker  -> update_only { transcript, turntaking: "user_turn" }
worker  -> update_only { transcript, turntaking: "agent_turn" }
worker  -> response_required { response_id: 1, transcript }
server  -> response { response_id: 1, content: "Sure", content_complete: false }
server  -> response { response_id: 1, content: ", which day?", content_complete: true }
...

Frames Tried sends#

ping_pong#

Sent every 2 seconds, only after your server has turned on auto_reconnect in a config frame.

{ "interaction_type": "ping_pong", "timestamp": 1760851200123 }

Reply with your own ping_pong that echoes the same timestamp. The worker measures the round trip from it (see Latency).

call_details#

Sent once right after the connection opens, only if your config frame (received within 0.5 seconds of connecting) set call_details: true. It is sent again after a reconnect.

{
  "interaction_type": "call_details",
  "call": {
    "call_id": "call_8f2c4b1e9d0a7c6b5e4f3a21",
    "call_type": "phone_call",
    "agent_id": "agent_46117b4b00fe0358e861a6d3",
    "direction": "inbound",
    "from_number": "+61411111111",
    "to_number": "+61355505030",
    "metadata": {},
    "retell_llm_dynamic_variables": { "first_name": "Sam" },
    "collected_dynamic_variables": {},
    "transcript": ""
  }
}

update_only#

Sent every time an utterance becomes final, for both the caller and the agent. It never needs a reply.

{
  "interaction_type": "update_only",
  "transcript": [
    { "role": "agent", "content": "Hi, this is Ava from Acme. How can I help?" },
    { "role": "user", "content": "I need to move my appointment." }
  ],
  "turntaking": "agent_turn"
}

Updates are sent without waiting, so they can arrive just before or just after a response_required for the same turn. Treat the transcript in response_required as the source of truth for a reply.

response_required#

Sent when the caller has finished a turn and the agent must reply.

{
  "interaction_type": "response_required",
  "response_id": 3,
  "transcript": [
    { "role": "agent", "content": "Hi, this is Ava from Acme. How can I help?" },
    { "role": "user", "content": "I need to move my appointment to Thursday." }
  ]
}

reminder_required#

Sent when the caller has been silent for the agent's reminder_trigger_ms (default 10000) and fewer than reminder_max_count (default 1) reminders have been sent. The shape is the same as response_required.

{
  "interaction_type": "reminder_required",
  "response_id": 4,
  "transcript": [
    { "role": "agent", "content": "Which day suits you?" }
  ]
}

Reply with a short nudge ("Are you still there?") using the same response_id. The reply is spoken and its end_call, transfer_number and digit_to_press flags are applied.

response_required versus reminder_required#

Supersession by response_id#

Only the newest request is live. When a new response_required or reminder_required goes out, any response frame that still carries an older response_id is dropped. This happens when the caller interrupts or keeps talking while your server is still generating. Stop generating for an old id as soon as a newer one arrives.

Frames your server sends#

config#

Optional. Send it as the first frame after the connection opens. The worker waits up to 0.5 seconds for it before it decides whether to send call_details and start pings.

{
  "response_type": "config",
  "config": {
    "auto_reconnect": true,
    "call_details": true,
    "transcript_with_tool_calls": false
  }
}

ping_pong#

Reply to each ping_pong from Tried, echoing its timestamp.

{ "response_type": "ping_pong", "timestamp": 1760851200123 }

Every reply resets the 5 second reconnect timer.

response#

The words the agent speaks for a request, streamed as one or more frames with the same response_id.

{ "response_type": "response", "response_id": 3, "content": "No problem.", "content_complete": false }
{ "response_type": "response", "response_id": 3, "content": " Thursday at 2:30 is free.", "content_complete": true }

The worker starts speaking as soon as the first chunk arrives. Action flags can be set on any frame of the response; they are applied after the last frame. If both are set, transfer_number wins over end_call. digit_to_press is sent first, after the agent finishes speaking.

{
  "response_type": "response",
  "response_id": 5,
  "content": "I'll put you through to our billing team now.",
  "content_complete": true,
  "transfer_number": "+61290000000",
  "show_transferee_as_caller": true
}

{ "response_type": "response", "response_id": 2, "content": "", "content_complete": true, "digit_to_press": "2" }

The opening line (response_id 0)#

To make the agent speak first, send a response with response_id: 0 right after the connection opens, before the first response_required. The chunks are buffered until content_complete: true, then spoken. Its action flags are applied too.

{ "response_type": "response", "response_id": 0, "content": "Hi, this is Ava from Acme.", "content_complete": false }
{ "response_type": "response", "response_id": 0, "content": " How can I help?", "content_complete": true }

If you send nothing, the agent waits for the caller to speak. A response_id: 0 frame that arrives after the first request is ignored.

agent_interrupt#

Makes the agent speak without a request, for example when a long lookup finishes or a supervisor pushes a message. The text is buffered per interrupt_id until content_complete: true, then spoken.

{
  "response_type": "agent_interrupt",
  "interrupt_id": 1,
  "content": "Sorry for the wait, I found your booking.",
  "content_complete": true,
  "no_interruption_allowed": false
}

update_agent#

Changes turn-taking settings for the rest of the call.

{
  "response_type": "update_agent",
  "agent_config": {
    "responsiveness": 0.4,
    "interruption_sensitivity": 0.7,
    "reminder_trigger_ms": 15000,
    "reminder_max_count": 2
  }
}

tool_call_invocation and tool_call_result#

Record tool calls that your server ran, so they appear in the call transcript. See Function calling.

{ "response_type": "tool_call_invocation", "tool_call_id": "t1", "name": "lookup_booking", "arguments": "{\"phone\":\"+61411111111\"}" }
{ "response_type": "tool_call_result", "tool_call_id": "t1", "content": "{\"booking\":\"Tue 14:00\"}" }

metadata#

{ "response_type": "metadata", "metadata": { "crm_lead_id": "L-1042" } }

Merged into the call's metadata (top-level keys, later frames win), so it appears on the call object, in webhooks and in Call History. The merged object is capped at 50,000 characters of JSON; a frame that would pass the cap is ignored.

Reconnects#

With auto_reconnect: true:

1. The worker sends ping_pong every 2 seconds.
2. If no ping_pong reply has arrived for 5 seconds, or the socket closes, the worker closes the socket and connects again to the same URL.
3. After a reconnect it waits for your config again and resends call_details if asked.
4. If a request was in flight, it resends response_required with the same response_id and the last transcript.
5. After 2 reconnects in one call, the next failure ends the call with error_llm_websocket_lost_connection.

Without auto_reconnect, a socket closed by your server ends the call with error_llm_websocket_lost_connection.

Errors and disconnection reasons#

These values appear in the call's disconnection_reason and in the call_ended webhook. See Troubleshooting.

Latency measurement#

Each ping_pong reply you send is timed against the timestamp it echoes, and the round trip is recorded as latency.llm_websocket_network_rtt on the call. Only replies that echo a timestamp count, so always echo the value you received.

Minimal server#

import { WebSocketServer } from "ws";

const wss = new WebSocketServer({ port: 8080 });

wss.on("connection", (ws, req) => {
  const callId = req.url!.split("/").pop();
  const send = (m: object) => ws.send(JSON.stringify(m));

  send({ response_type: "config", config: { auto_reconnect: true, call_details: true } });
  send({ response_type: "response", response_id: 0, content: "Hi, thanks for calling. How can I help?", content_complete: true });

  ws.on("message", async (raw) => {
    const msg = JSON.parse(raw.toString());
    switch (msg.interaction_type) {
      case "ping_pong":
        send({ response_type: "ping_pong", timestamp: msg.timestamp });
        break;
      case "call_details":
        console.log("call", callId, msg.call.from_number);
        break;
      case "update_only":
        break;
      case "reminder_required":
        send({ response_type: "response", response_id: msg.response_id, content: "Are you still there?", content_complete: true });
        break;
      case "response_required": {
        const last = msg.transcript.at(-1)?.content ?? "";
        const bye = /\bbye\b/i.test(last);
        send({ response_type: "response", response_id: msg.response_id, content: bye ? "Goodbye!" : `You said: ${last}`, content_complete: true, end_call: bye });
        break;
      }
    }
  });
});

import json
from aiohttp import web, WSMsgType

async def handler(request: web.Request) -> web.WebSocketResponse:
    call_id = request.match_info["call_id"]
    ws = web.WebSocketResponse()
    await ws.prepare(request)
    await ws.send_json({"response_type": "config", "config": {"auto_reconnect": True, "call_details": True}})
    await ws.send_json({"response_type": "response", "response_id": 0,
                        "content": "Hi, thanks for calling. How can I help?", "content_complete": True})
    async for msg in ws:
        if msg.type != WSMsgType.TEXT:
            continue
        data = json.loads(msg.data)
        t = data["interaction_type"]
        if t == "ping_pong":
            await ws.send_json({"response_type": "ping_pong", "timestamp": data["timestamp"]})
        elif t == "reminder_required":
            await ws.send_json({"response_type": "response", "response_id": data["response_id"],
                                "content": "Are you still there?", "content_complete": True})
        elif t == "response_required":
            last = data["transcript"][-1]["content"] if data["transcript"] else ""
            bye = "bye" in last.lower()
            await ws.send_json({"response_type": "response", "response_id": data["response_id"],
                                "content": "Goodbye!" if bye else f"You said: {last}",
                                "content_complete": True, "end_call": bye})
    return ws

app = web.Application()
app.router.add_get("/llm-websocket/{call_id}", handler)
web.run_app(app, port=8080)

Point the agent's llm_websocket_url at wss://your-host/llm-websocket. For a real agent, replace the echo with a streaming call to your model and send each token chunk as a response frame (see Best practices).