Tried gives you several ways to test an agent. They trade speed for realism: a text chat takes seconds and costs nothing per minute, a phone call exercises the whole path from carrier to voice.
| Method | Where | What it exercises | Needs keys |
|---|---|---|---|
| Test LLM | Test panel in the agent builder | Prompt, tools (with mocks), state and node transitions, knowledge base | OpenAI for real replies; mock replies without |
| Simulated user | Test panel, Simulated user tab | A whole conversation driven by an AI caller | OpenAI for real replies |
| Simulation test cases | Agent builder, Simulation | Repeatable scenarios graded by an LLM judge | OpenAI for the simulated caller and judge |
| Batch tests | Simulation, Batch Testing History, or the API | Many test cases at once, pass, fail and error counts, CI | OpenAI |
| Test Audio | Test panel, Test Audio tab | Speech recognition, voice, turn-taking, latency in the browser | LiveKit, plus speech and voice providers |
| Voice Orb link | Share button in the agent header | The agent in anyone's browser | LiveKit |
| Phone testing | Phone Numbers | The full telephony path, transfers, keypad digits | Telephony provider and LiveKit SIP |
| Debugging responses | Test LLM turn actions | Why a turn came out the way it did, and how stable it is | OpenAI |
Every provider has a mock, so the dashboard and every test method above run without API keys. In mock mode, replies are deterministic ("Mock reply to: ...") and the judge is a keyword rule. Use mock mode to check wiring and flows; use real keys to judge the quality of answers.
Where testing lives in the builder#
Single prompt, multi-prompt and conversation flow agents have a test panel on the right of the builder with four tabs: Test Audio, Test LLM, a Simulated user tab (robot icon) and a Tool mocks tab (wrench icon). The { } button next to them sets Dynamic variables for every test in the panel.
The Create and Simulation switch in the builder header (the Simulation button in the left rail for conversation flow agents) opens the simulation test cases for the same agent.
Custom LLM agents do not have the test panel or the Simulation view, because those run the built-in engine and would not reach your server. See Testing a custom LLM.
What a test runs against#
- Every test in the panel saves your pending edits first, then runs against the agent's latest (highest numbered) version, normally the draft you are editing.
- Switching to Simulation also saves pending edits. Test cases and batch tests run against the agent's latest version too.
- Phone calls and web calls use the version the number or call resolves to. See Phone testing.
A testing workflow#
Chat through the happy path#
Use Test LLM to walk the main conversation. Add Tool mocks for every function so tool results are realistic.
Probe the edges#
Use the Simulated user tab with personas that push the agent off script: impatient callers, off-topic questions, requests for a human.
Turn what you learned into test cases#
In Simulation, add a test case for each scenario, with a user prompt, success criteria and function mocks.
Run the batch before every publish#
Select the cases and choose Run batch. Fix failures, run again, then publish.
Call it#
Run a Test Audio call to hear the voice and timing, and call the number once from a real phone.
Keep testing in production#
Watch Call History, and turn failing calls into new test cases. Conductor can write and run test cases for you.
Costs#
- Test LLM, the simulated user, test cases and batch tests use LLM tokens only. There is no per-minute charge.
- Test Audio, Voice Orb calls and phone calls are real calls and are billed like any other call.
- Conductor messages use Conductor credits. See Conductor overview.