Skip to content
TriedDocs
Test

Batch tests

Run many simulation test cases at once, read pass, fail and error counts in Batch Testing History, and gate releases on them from CI.

A batch test runs a set of an agent's simulation test cases and records a verdict for each. Use it before every publish to catch regressions, and from CI to block a release when a case fails.

Needs keys Real verdicts need OpenAI for the simulated caller and the judge. Without a key, batches run in mock mode with the keyword judge.

Run a batch in the dashboard#

Open Simulation#

Open the agent and switch the header to Simulation (for a conversation flow agent, choose Simulation in the left rail).

Select cases#

On Test Cases, tick the cases to run, or tick the header box to select them all.

Choose Run#

The button reads Run N selected. A toast confirms "Running N tests" and the view switches to Batch Testing History.

Test on a single row also creates a batch, with one case.

Batch Testing History#

Each batch is a row with its start time, the number of tests, and three counters: passed, failed and errors. While it runs, the row shows "N of M finished" and a Running badge, and the list refreshes every 1.5 seconds. When every run has finished, the badge reads Complete.

Batch Testing History with a run open in the drawer

Expand a batch to see each run with its status and the start of the judge's explanation or error. Choose a run to open it in the drawer with the full transcript.

The newest batch opens expanded. The Test Cases tab also shows the last batch's result next to the case count, for example "last batch 10/17/2026 16:55: 3 passed, 1 failed".

How batches execute#

  • Runs execute one after another, in the order of the ids you sent, in the background on the API.
  • Each run is a full simulated conversation followed by the judge.
  • A run that cannot finish is marked Error and counted in error_count. It never stops the rest of the batch.
  • Each agent turn loads the agent's latest version when it runs. Avoid editing the agent while a batch runs, or later runs test the edited version.
  • Deleting a test case does not delete its past runs. A run that had not started yet ends with "The test case was deleted before it ran".
Limit Value
Test cases per batch 1 to 50
Batches returned by list-batch-tests 200, newest first
Runs returned by list-test-runs 200, newest first

API#

Method and path Description
POST /create-batch-test Start a batch. Body: agent_id, test_case_definition_ids (1 to 50). Returns 201.
GET /list-batch-tests?agent_id= List batches.
GET /get-batch-test/{id} A batch with its test_runs (without transcripts).
GET /list-test-runs?batch_test_id=&test_case_definition_id= List runs (without transcripts).
GET /get-test-run/{id} One run with its transcript.

API reference: create batch test, get batch test, get test run.

Batch object#

{
  "batch_test_id": "batch_test_6a1f0c2e9b4d7e3a5c8f1b20",
  "agent_id": "agent_46117b4b00fe0358e861a6d3",
  "status": "complete",
  "total_count": 4,
  "pass_count": 3,
  "fail_count": 1,
  "error_count": 0,
  "created_timestamp": 1760680551000,
  "completed_timestamp": 1760680612000
}
Field Description
status in_progress until every run has finished, then complete.
total_count Number of runs in the batch.
pass_count, fail_count, error_count Finished runs by verdict.

Run object#

{
  "test_run_id": "test_run_0d9e8c7b6a5f4e3d2c1b0a99",
  "batch_test_id": "batch_test_6a1f0c2e9b4d7e3a5c8f1b20",
  "test_case_definition_id": "test_case_3b2a1c0d9e8f7a6b5c4d3e21",
  "test_case_name": "Wants a refund",
  "agent_id": "agent_46117b4b00fe0358e861a6d3",
  "status": "fail",
  "transcript": [
    { "role": "agent", "content": "Hi, thanks for calling Harbourside Dental. How can I help?" },
    { "role": "user", "content": "I was charged twice for my last visit." },
    { "role": "agent", "content": "I'm sorry about that. Let me check your account." }
  ],
  "explanation": "The agent apologised but never said the refund would be processed.",
  "error": null,
  "duration_ms": 8421,
  "created_timestamp": 1760680551000,
  "completed_timestamp": 1760680560000
}
Field Description
status pending, running, pass, fail or error.
transcript Items with role agent, user, tool_call_invocation, tool_call_result or node_transition. Only on get-test-run.
explanation The judge's reason. Null on errors.
error Why the run could not finish. Null otherwise.

Use batch tests in CI#

Keep your cases in the repo (use Export in the dashboard, or create them with the API), then in your pipeline:

  1. Update the agent's draft with the new prompt or flow.
  2. Start a batch with every case id.
  3. Poll get-batch-test until status is complete.
  4. Fail the job if fail_count or error_count is above zero, and print each failing run's explanation.
  5. Publish only when the batch passes.
BATCH=$(curl -s -X POST https://api.tried.com.au/create-batch-test \
  -H "Authorization: Bearer $TRIED_API_KEY" -H "Content-Type: application/json" \
  -d "{\"agent_id\":\"$AGENT_ID\",\"test_case_definition_ids\":$CASE_IDS}" | jq -r .batch_test_id)

until [ "$(curl -s https://api.tried.com.au/get-batch-test/$BATCH \
  -H "Authorization: Bearer $TRIED_API_KEY" | jq -r .status)" = "complete" ]; do sleep 5; done

curl -s https://api.tried.com.au/get-batch-test/$BATCH -H "Authorization: Bearer $TRIED_API_KEY" \
  | jq -e '.fail_count == 0 and .error_count == 0'
const base = "https://api.tried.com.au";
const headers = { Authorization: `Bearer ${process.env.TRIED_API_KEY}`, "Content-Type": "application/json" };

const cases: { test_case_definition_id: string }[] = await (await fetch(`${base}/list-test-case-definitions?agent_id=${agentId}`, { headers })).json();
const batch = await (await fetch(`${base}/create-batch-test`, {
  method: "POST", headers,
  body: JSON.stringify({ agent_id: agentId, test_case_definition_ids: cases.slice(0, 50).map((c) => c.test_case_definition_id) }),
})).json();

let b = batch;
while (b.status !== "complete") {
  await new Promise((r) => setTimeout(r, 5000));
  b = await (await fetch(`${base}/get-batch-test/${batch.batch_test_id}`, { headers })).json();
}
for (const r of b.test_runs.filter((r: { status: string }) => r.status !== "pass")) console.error(`${r.status}: ${r.test_case_name}: ${r.error ?? r.explanation}`);
if (b.fail_count || b.error_count) process.exit(1);
import os, time, requests

base = "https://api.tried.com.au"
h = {"Authorization": f"Bearer {os.environ['TRIED_API_KEY']}"}

cases = requests.get(f"{base}/list-test-case-definitions", params={"agent_id": agent_id}, headers=h).json()
batch = requests.post(f"{base}/create-batch-test", headers=h, json={
    "agent_id": agent_id, "test_case_definition_ids": [c["test_case_definition_id"] for c in cases][:50],
}).json()

while True:
    b = requests.get(f"{base}/get-batch-test/{batch['batch_test_id']}", headers=h).json()
    if b["status"] == "complete":
        break
    time.sleep(5)

for r in b["test_runs"]:
    if r["status"] != "pass":
        print(r["status"], r["test_case_name"], r["error"] or r["explanation"])
raise SystemExit(1 if b["fail_count"] or b["error_count"] else 0)
Tip

LLM verdicts can vary between runs. For important cases, make the success criteria specific and observable, and rerun a failing batch once before you investigate.