AI Quality Assurance reviews calls for you. You define a cohort: which ended calls to look at, how many to sample, the resolution criteria a good call meets, and optional metric judges such as hallucination, word error rate or latency. Every matching call is scored, gets a QA sheet with a pass or fail per criterion and metric, and rolls up into cohort charts and a list of the questions callers ask most.
Use it to watch quality as call volume grows, to compare agent versions, to find the topics your agent struggles with, and to catch regressions after a prompt change.
Open it from MONITOR > AI Quality Assurance.
Concepts#
| Term | Meaning in Tried |
|---|---|
| Cohort | A saved set of filters, a sample rate, a weekly cap, criteria, metric judges and a pass threshold. See QA cohorts. |
| Resolution criteria | Named conditions in plain language, judged pass or fail on each call, each with a weight. See Resolution criteria. |
| Score | The share of criteria passed, 0 to 100. |
| Resolved | The judge's overall verdict on whether the caller's issue was resolved. Shown as Resolved or Unresolved. |
| Metric judges | Automatic per-call measurements scored 0 to 100, each with a weight and a pass threshold. See QA metrics. |
| Weighted score | Criteria (pass = 100, fail = 0) and applicable metric scores averaged by weight, 0 to 100. |
| Pass threshold | A call passes when its weighted score is at or above this value (default 70). |
| Override | A human Mark as passed or Mark as failed on a result. Overrides win over the AI verdict. |
| Calibrate | Picks the pass threshold that best agrees with your overrides. |
How scoring works#
A call ends and is analysed#
After post-call analysis, Tried checks every cohort in the workspace.
The call is matched#
The call must be ended, match the cohort's agents, dates, duration and disconnection reasons, fall inside the sample, match every extraction filter, and have a transcript. The cohort must not have reached its weekly maximum.
The call is judged#
The criteria are judged against the transcript, the metric judges run, and the weighted score and pass or fail are computed. The questions the caller asked are collected for Top questions.
Results roll up#
The cohort's Overview, Calls and Top questions tabs update.
Creating a cohort also scores existing calls that match, in the background. Run now scores any eligible calls that have not been scored yet.
Judges in real and mock mode#
| Part | With an OpenAI key | Without (mock mode) |
|---|---|---|
| Resolution criteria | gpt-5.6-terra judges each criterion, the resolved verdict and a one-sentence summary. |
A keyword scorer: quoted phrases in the criterion must all appear; otherwise at least half of its significant words must appear. |
| Hallucination, knowledge base recall, naturalness, tool call accuracy, transition accuracy | LLM judge with deterministic fallbacks. | Deterministic heuristics. |
| Word error rate, overlapping speech, sentiment, transfer success, transfer wait time, latency | Always computed from call data. | Same. |
| Top questions labels | LLM names each question group. | The most frequent question in the group. |
If the LLM request fails, the call is scored with the mock scorer and heuristics instead of failing.
Needs keys LLM judging uses OpenAI (OPENAI_API_KEY).
Page layout#
The header has a cohort selector, "Last scored ..." (or "Not scored yet"), edit and delete buttons, Calibrate, Run now and Create cohort. Under it, badges summarise the cohort: agents, sample rate, weekly maximum, criteria count, disconnection reasons, filter and metric counts, and the pass threshold.
The tabs are:
- Overview: KPIs, metric results, score and resolution trend, criteria pass rate. See QA results.
- Calls: every scored call; click one for its QA sheet. See QA results.
- Top questions: grouped caller questions with how often they were asked and how often those calls were resolved. See Top questions.
Pricing#
QA is billed on the minutes of calls scored: the first 100 minutes each month are free, then $0.15 per minute. Each result is charged once, so re-scoring a call does not charge it again. GET /qa/usage returns minutes_this_month, free_minutes_remaining, cost_this_month and price_per_minute. The metric picker also shows the free allowance and price.
QA API#
| Endpoint | Description |
|---|---|
POST /qa/cohorts |
Create a cohort (201) and start scoring matching calls. |
GET /qa/cohorts |
List cohorts with calls_analyzed and average_score. |
GET /qa/cohorts/:id, PATCH /qa/cohorts/:id, DELETE /qa/cohorts/:id |
Read, update, delete. Deleting removes its results and overrides; calls are not affected. |
POST /qa/cohorts/:id/run |
Score eligible calls now. Returns { "scored": N }. |
GET /qa/cohorts/:id/overview |
Cohort KPIs and charts (tz optional). |
GET /qa/cohorts/:id/results |
Every result with a call summary. |
GET /qa/results/:id |
One QA sheet with the call transcript. |
PATCH /qa/results/:id |
{ "override": "pass" | "fail" | null }. |
PUT /qa/results/:id/wer-corrections |
Save transcript corrections and re-score. |
POST /qa/cohorts/:id/calibrate |
Fit the pass threshold (apply: false previews). |
GET /qa/cohorts/:id/top-questions |
Grouped questions (limit 1 to 50, default 20). |
GET /qa/metrics |
Metric names with default thresholds, filter operators and pricing. |
GET /qa/usage |
QA minutes and cost this month. |
QA results also feed the qa_score_average and qa_failure_count alert metrics. See Alerting.