QA results

Once a cohort has scored calls, the Overview and Calls tabs show how the agent is doing, and each call has a QA sheet with every criterion and metric. You can mark results as passed or failed by hand and let Tried fit the pass threshold to your judgement.

Overview tab#

Below the tiles:

• Metric results (when the cohort has metric judges): per metric, Calls where it applied, Average score, Average value and Pass rate.
• Score and resolution trend: daily average score and resolution rate, by the day the call started.
• Criteria pass rate: pass rate per criterion of the cohort.

The overview refreshes every 15 seconds. Days use the Australia/Sydney time zone unless you pass tz to GET /qa/cohorts/:id/overview.

Calls tab#

Lists every scored call, newest first. The selector filters All calls, Resolved or Unresolved.

Click a row to open its QA sheet.

The QA sheet#

The header shows the score out of 100, the verdict, a Manual badge when overridden, and Weighted N · Pass or Fail. Under it: agent, start time and duration, and Open call to see the call in Call History.

The body has:

• The judge's one-sentence summary.
• Criteria: each criterion with a pass or fail icon and the explanation.
• Metrics: each metric judge with pass, fail or not applicable, its value and its score.
• Transcription accuracy: the WER badge and Correct transcript.
• Tiles for Disconnection, Sentiment and E2E latency p50.
• Transcript with agent and user turns and tool names.

Override the verdict#

Mark as passed and Mark as failed set a manual override. Clear override removes it.

An override changes the effective verdict everywhere: the Resolved or Unresolved badge, the resolution rate, the pass rate, qa_failure_count alerts, and top question resolution rates. The AI's own resolved, passed and scores are kept on the result.

In the API, PATCH /qa/results/:id with { "override": "pass" }, "fail" or null. The result returns effective_resolved and effective_passed with the override applied.

Transcription accuracy#

Correct transcript lists each user utterance with what was heard. Type what the caller really said and click Save corrections. Only changed lines are saved. The call is re-scored at once, so the word error rate and the weighted score update.

API: PUT /qa/results/:id/wer-corrections with { "corrections": [{ "index": 3, "text": "I'd like a check-up on Thursday" }] }. index is the position in the call's transcript_object and must point at a user utterance (400 otherwise). Up to 500 corrections. Saving re-runs the criteria and metric judges for that call.

Calibrate the pass threshold#

After you have marked at least 2 calls as passed or failed, Calibrate fits the cohort's pass threshold to your marks.

The dialog Calibrate pass threshold shows Now (the current threshold and how often it agrees with your marks) and Calibrated (the best threshold and its agreement), with the number of marked calls. Apply saves the new threshold and recomputes pass or fail on every result.

How it is computed: every threshold from 0 to 100 is tried against the marked results that have a weighted score; the one that agrees with the most marks wins, and ties go to the threshold closest to the current one.

API: POST /qa/cohorts/:id/calibrate with { "apply": false } to preview or true (default) to apply. With fewer than 2 marks it returns 400 "Mark at least 2 calls as passed or failed before calibrating". The cohort keeps the last calibration (threshold, accuracy, previous values, samples, timestamp).

Result fields#

Related#