Testing a custom LLM

A custom LLM agent is tested differently from a dashboard agent. The Test LLM playground, the simulated user and simulation test cases run the built-in prompt engine, not your server, so the custom LLM builder does not show the test panel or the Simulation tab. Use the tools on this page instead.

Test connection#

The LLM WebSocket card in the custom LLM builder has a Server URL field and a Test connection button.

What the test does#

The API (POST /test-custom-llm-connection) does this:

1. Validates the URL and rejects anything that is not ws:// or wss://.
2. Resolves the host and refuses private, loopback and reserved addresses, the same rule as other outbound requests. Set EGRESS_ALLOW_PRIVATE=1 on the API to allow them in local development.
3. Opens {url}/test_connection_<24 hex chars>.
4. Sends a call_details frame with call_type: "web_call", agent_id: "test_connection" and call_status: "registered", then a ping_pong frame.
5. Waits up to 5 seconds for any frame from your server.

Your server passes if it sends any frame within 5 seconds, for example the config frame it sends on open.

Test connection errors#

The endpoint needs the admin or developer role:

curl -X POST https://api.tried.com.au/test-custom-llm-connection \
  -H "Authorization: Bearer $TRIED_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "llm_websocket_url": "wss://llm.example.com/llm-websocket" }'

{
  "ok": true,
  "latency_ms": 184,
  "first_frame": "{\"response_type\":\"config\",\"config\":{\"auto_reconnect\":true,\"call_details\":true}}",
  "error": null
}

Live calls#

Needs keys Live audio needs LiveKit configured on the API and a running voice worker. Phone calls also need a telephony provider.

Three ways to talk to a custom LLM agent:

• Voice Orb link. Use the share button in the agent header to open Share with a Voice Orb link, choose Create link, and open it in a browser. Calls are billed to the workspace. See Testing with audio.
• Web call through the API. Call create web call with the agent id and join the returned room with the web SDK.
• Phone call. Attach the agent to a number and call it, or use Make an outbound call on Phone Numbers. See Phone testing.

During a live call, check:

• The opening line plays if you send response_id: 0.
• The agent stops talking when you interrupt, and your server receives a new response_required with a higher id.
• A long silence produces reminder_required.
• end_call, transfer_number and digit_to_press work (transfers and digits need a phone call).

Afterwards, open the call in Call History. The transcript shows your tool call events, and the disconnection reason tells you whether the socket failed. The WebSocket round trip is stored on the call as latency.llm_websocket_network_rtt (available from the API).

Text simulator#

The voice worker ships a text simulator that runs the same engine as a live call, without audio, LiveKit or keys. With a custom LLM snapshot it connects to your server and drives it with typed lines.

Create snapshot.json:

{
  "call": { "call_id": "call_local_test_0001", "call_type": "phone_call", "direction": "inbound",
            "from_number": "+61411111111", "to_number": "+61355505030" },
  "agent": { "agent_id": "agent_local" },
  "engine": { "type": "custom-llm", "llm_websocket_url": "ws://127.0.0.1:8080/llm-websocket" },
  "voice": { "provider": "elevenlabs", "provider_voice_id": "x", "model": "eleven_flash_v2_5" },
  "knowledge_base_ids": [], "dynamic_variables": {}, "api_base": "http://api.local"
}

Write the caller's lines in lines.txt, one per line. Special lines:

Run it from services/agent:

.venv/bin/python -m tried_agent.simulate --snapshot snapshot.json --script lines.txt --events -

Without --script it reads lines interactively. --events - prints every runtime event (utterances, tool calls, latency, call_ended with the disconnection reason) as JSON. Simulator runs stop at 10 minutes or 400 utterances.

The simulator waits about one second for your server to send the opening line (response_id: 0) before the first caller line. Connection failures end the run with the same error_llm_websocket_* reasons as a live call.

Testing checklist#

• config is the first frame and arrives within 0.5 seconds.
• Every ping_pong is echoed with the same timestamp.
• Responses for an old response_id stop when a newer request arrives.
• The last frame of every response has content_complete: true.
• No frame gap within one response reaches 30 seconds.
• reminder_required gets a short reply.
• Unknown interaction_type values are ignored, not treated as errors.
• The server survives a reconnect: it can receive call_details and the in-flight response_required again.