Conductor can write simulation test cases for an agent and start a batch test, from one chat message. The cases are ordinary test cases: you can edit, run and export them in the agent's Simulation view like any you wrote yourself.
Needs keys With an OpenAI key, Conductor writes cases from the agent's prompt and your request. Without one, it writes three template cases. Running the cases also uses OpenAI for the simulated caller and the judge (or the mock judge without a key).
Write test cases#
Open a Conductor conversation#
Use the floating Conductor button or Home.
Attach the agent#
Attach the agent as context: Agents in Attach context, from the side panel or the + button on Home. Conductor writes tests for the first attached agent. Without one it replies "Attach an agent so I know which agent to write tests for."
Ask#
For example "Write test cases for this agent and run them", or be specific: "Write tests for callers who want to cancel, including one who is angry".
On Home, the reply lists the names of the cases it created and, if it started a batch, "Batch test ... is running."
| Behaviour | Detail |
|---|---|
| Cases per message | Up to 10 are saved. With an OpenAI key, Conductor is asked for 2 to 5. |
| Fields filled | name, user_prompt, success_criteria. Dynamic variables and function mocks are empty; add them in Simulation, Edit. |
| Simulated user model | The default, gpt-5.6-terra. |
| Running them | Started when the model decides to (with a key), or when your message contains the word "run" (mock mode). |
Mock mode templates#
Without a key, Conductor creates three cases named after the agent:
| Case | Caller | Success criteria |
|---|---|---|
<agent>: happy path |
A polite caller with a simple goal guessed from the prompt (booking, an order, support, a question) | The agent helps with that goal and confirms the next step. |
<agent>: asks for a human |
An impatient caller who asks for a real person early | The agent acknowledges and offers a transfer or a callback. |
<agent>: off-topic question |
A caller who asks about the weather and football first | The agent stays polite and steers back. |
Run existing tests#
Ask "Run the tests" with the agent attached. Conductor starts a batch of the agent's test cases, up to the first 20 by creation date, and replies "Running N test cases (batch ...)". If the agent has none it says "This agent has no test cases yet. Ask me to write some."
Check the results#
Batches run in the background. Open the agent, switch to Simulation and choose Batch Testing History to see pass, fail and error counts and each run's judge explanation.
Improve the agent from failures#
- Open a failed run and read the judge's explanation and the transcript.
- Open the agent's builder and choose Conductor in the header.
- Describe the failure: "In the refund test the agent never says the refund will be processed. Add an instruction to confirm refunds are processed within 5 business days."
- Review the diff and choose Accept (see Build and refine).
- Ask Conductor to run the tests again, or select the cases and choose Run batch.
Tips#
- Attach a flow node to ask for tests that target one part of a conversation flow.
- Attach a few failed calls together with the agent and ask for test cases based on them. With an OpenAI key, Conductor sees the calls' summaries.
- Review generated success criteria before you rely on them in CI. Make them specific and observable.
- Each message costs credits by tier. Writing and running tests in one message costs one message.
Related#
- Simulation testing
- Batch tests
- Conductor overview for credits.