Simulation testing

Simulation testing replaces you with an AI caller. You describe who is calling and what they want; the simulated user holds the conversation with your agent over text; a judge reads the transcript and decides whether your success criteria were met.

There are two ways to use it:

• Simulated user in the test panel: a one-off conversation that streams into the panel. Nothing is saved.
• Test cases in the Simulation view: saved scenarios you run alone or in batches, each with a pass, fail or error verdict.

Both run the same engine as Test LLM against the agent's latest version, so they work for single prompt, multi-prompt and conversation flow agents. They do not reach a custom LLM server.

Needs keys The simulated user and the judge use OpenAI. Without a key they run in mock mode (see Mock mode).

Simulated user in the test panel#

The run uses the panel's Dynamic variables and Tool mocks. A status line explains how it ended:

When you test a single flow component, a run that reaches the component's exit reports "The subflow finished".

Test cases#

Open the agent and switch the header from Create to Simulation (for a conversation flow agent, choose Simulation in the left rail). The Test Cases tab lists the agent's cases with Name, User prompt, Success criteria and Mocks, and the result of the last batch.

Add a test case#

Choose Add a Test Case. The modal reads "A simulated caller follows the persona; a judge checks the transcript against the success criteria."

Add function mock lists the agent's functions (general tools and every state's tools, or the flow's tools). If the agent has none, the modal says "This agent has no functions to mock."

Write good cases#

• One goal per case. "You want to move your Tuesday cleaning to Thursday afternoon" is better than a caller with three requests.
• Give the caller facts. Names, dates, account numbers the agent will ask for.
• Tell the caller how to end. "Say goodbye once it's booked" stops the run cleanly.
• Make criteria observable. Describe what the agent says or does ("confirms the new time", "calls book_appointment"), not how it feels.
• Mock every tool the scenario reaches. Unmocked tools return a placeholder (below), which usually makes the agent fail.

Run one case#

Choose Test on a row. The API creates a batch with that one case and the run opens in a drawer that refreshes every second until it finishes: "The simulated caller is talking to the agent…"

Case actions#

The row menu has Edit, Duplicate, Export JSON and Delete. Import loads a JSON file with one case or an array of cases (each needs name, user_prompt and success_criteria); Export downloads all cases as test-cases.json. Use export and import to copy cases between agents or keep them in source control.

How a run works#

Tool calls without a mock#

Tools never call your real endpoints in simulation. Simulations are text only, so transfer and SMS tools are recorded as tool calls in the transcript: no call is transferred and no message is sent. A tool call without a matching mock is recorded with this result, and the agent does not continue on it:

{ "mocked": false, "note": "No function mock for this tool" }

Limits#

Runs that hit a limit still go to the judge.

Read the results#

The run drawer shows the case name, start time, duration and status (Pass, Fail or Error), the Judge explanation (or the Error text), and the transcript with Agent and Simulated user turns, expandable Tool Invocation and Tool Result (mocked) items, and transitions.

Mock mode#

Without an OpenAI key:

• The simulated user turns the user prompt into first-person lines, one per turn ("You are Sam Taylor." becomes "I am Sam Taylor."), then says "Thanks, that's all. Goodbye."
• The agent replies with Mock reply to: "...", except for static welcome messages.
• The judge is a keyword rule. If the criteria contain quoted phrases, every phrase must appear in the transcript. Otherwise at least 60% of the significant words in the criteria must appear. The explanation starts with "Mock judge:" and lists what was found and missing.

Mock mode checks that cases, mocks and flows are wired correctly. It does not tell you whether the agent is good.

API#

See the API reference: create test case definition and create batch test.

curl -X POST https://api.tried.com.au/create-test-case-definition \
  -H "Authorization: Bearer $TRIED_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "agent_id": "agent_46117b4b00fe0358e861a6d3",
    "name": "Reschedules a cleaning",
    "user_prompt": "You are Sam Taylor. You booked a cleaning for Tuesday at 2pm but need to move it to Thursday afternoon. You are polite but short on time. Say goodbye once it is moved.",
    "success_criteria": "The agent moves the appointment to '\''Thursday'\'' and confirms the caller'\''s name.",
    "dynamic_variables": { "business_name": "Harbourside Dental" },
    "tool_mocks": [{ "tool_name": "check_availability", "output": "{\"slots\":[\"14:30\",\"16:00\"]}" }]
  }'

The simulated-user stream takes user_prompt (required), dynamic_variables, tool_mocks, llm_model, max_utterances (2 to 400, default 40) and start_component_id. It sends {"type":"item","item":...} for each transcript item and ends with {"type":"done","ended_by":...,"utterances":...} or {"type":"error","message":...}. ended_by is one of user_ended, agent_ended, script_finished, utterance_limit, time_limit or aborted.

Related#

• Batch tests to run many cases and use them in CI.
• Conductor: test and improve to have test cases written for you.