Metric judges are automatic measurements that run on every call a cohort scores, next to the resolution criteria. Each metric returns a raw value, a score from 0 to 100 (higher is better), and passes when the score reaches its threshold. Metric scores are blended into the call's weighted score with the criteria.
The metrics are defined in apps/api/src/services/qa-metrics.ts.
Turn metrics on#
In Create QA cohort or Edit QA cohort, the Metric judges list has one checkbox per metric: "Automatic metrics scored per call and blended into the weighted score with the criteria." Tick a metric to show its Weight (0 to 100, default 1) and Pass at (0 to 100; empty uses the default shown as the placeholder).
In the API, set metrics on the cohort:
"metrics": [
{ "metric": "hallucination", "weight": 2, "threshold": 80 },
{ "metric": "wer", "weight": 1 },
{ "metric": "latency" }
]GET /qa/metrics lists every metric with its default_threshold.
Summary#
| Dashboard label | metric |
Value | Default pass at | Judge |
|---|---|---|---|---|
| Hallucination | hallucination |
Number of issues found | 70 | LLM, heuristic fallback |
| Knowledge base recall | kb_recall |
Share of retrievals used (0 to 1) | 60 | LLM, heuristic fallback |
| Word error rate | wer |
WER (0 to 1 or more) | 80 | Computed |
| Overlapping speech | overlapping_speech |
Number of overlaps | 70 | Computed |
| Sentiment | sentiment |
Sentiment score | 50 | Computed |
| Naturalness | naturalness |
Average words per agent turn | 60 | LLM, heuristic fallback |
| Tool call accuracy | tool_call_accuracy |
Share of good tool calls (0 to 1) | 80 | LLM, heuristic fallback |
| Transition accuracy | transition_accuracy |
Share of good transitions (0 to 1) | 80 | LLM, heuristic fallback |
| Transfer success | transfer_success |
1 or 0 | 100 | Computed |
| Transfer wait time | transfer_wait_time |
Milliseconds | 50 | Computed |
| Latency | latency |
Median end-to-end latency in ms | 60 | Computed |
"LLM" judges use gpt-5.6-terra when an OpenAI key is set: one request per call covers all five LLM metrics, given the agent's prompt or flow, the call's variables, knowledge base passages, tool results, tool and node events, and the transcript. Without a key, or if that request fails, the deterministic heuristics below are used. Each result's details.judge says llm or heuristic.
Not applicable results#
When a metric does not apply to a call, it returns score: null and passed: null with a details.reason, and it is left out of the weighted score. The QA sheet shows n/a with an empty circle.
| Metric | Not applicable when |
|---|---|
| Knowledge base recall | "No knowledge base retrievals in this call" |
| Word error rate | "No user speech" |
| Sentiment | "No post-call sentiment yet" |
| Naturalness | "The agent did not speak" |
| Tool call accuracy | "No tool calls" |
| Transition accuracy | "No node transitions" |
| Transfer success, Transfer wait time | "No transfer attempted" |
| Transfer wait time | Also when the transfer did not connect, or there is no bridge timestamp |
| Latency | "No latency samples" |
Metric details#
Hallucination#
Finds agent statements that are not supported by what the agent was given or told.
- LLM judge: lists issues of type
fabrication(not supported by the context),contradiction(contradicts an earlier statement) orconfusion, eachmajor(could mislead the caller) orminor. - Heuristic: every fact-like token the agent says (prices, times such as 4pm, numbers of 3 or more digits, amounts with percent, dollars, minutes, hours or days) must appear in the agent configuration, variables, knowledge base passages, tool results, or what the user said. Ungrounded facts are major fabrications. Phrases such as "sorry, I didn't understand" are minor confusion. Two agent turns sharing 3 or more significant words where only one has a negation count as a major contradiction.
- Score:
100 - 30 x major - 10 x minor, floored at 0. Value: number of issues. The heuristic keeps up to 20 issues indetails.issues.
Knowledge base recall#
How well the agent used the passages its knowledge base returned.
- Heuristic: for each retrieval, a retrieval counts as recalled when the number of the passage's significant words that the agent also said, divided by the passage's significant word count (capped at 20), is at least 0.2. Value is recalled retrievals divided by all retrievals; score is that times 100.
- LLM judge: rates use of relevant passages 0 to 100.
Up to 50 retrievals per call are considered.
Word error rate#
How accurately the caller's speech was transcribed, measured against human corrections.
- For each user utterance, the word-level edit distance between the corrected text (the reference) and the transcribed text is summed and divided by the number of reference words.
- Score:
(1 - WER) x 100, floored at 0. - Without corrections, the reference is the transcript itself, so WER is 0 and
details.estimatedis true. Add corrections on the QA sheet with Correct transcript: see QA results. Saving corrections re-scores the call.
Overlapping speech#
How often the agent and caller talked over each other.
- Uses per-word timings: an overlap is a turn that starts before the previous speaker's last word ended.
- Value: number of overlaps.
detailsalso has the total overlap in ms and as a percent of the call. - Score:
100 - 10 x overlaps - 2 x overlap percent. - Needs word timings from speech to text. Without them, no overlaps are detected and the score is 100.
Sentiment#
Maps post-call user_sentiment to a score: Positive 100, Neutral 70, Unknown 50, Negative 20. Value and score are the same.
Naturalness#
How human and fluent the agent sounded.
- Heuristic: starts at 100 and subtracts 15 per repeated agent turn, 25 per robotic phrase ("as an AI", "as a language model" and similar), 8 per turn over 60 words, and 1.5 for each word the average turn length is above 35.
- LLM judge: rates naturalness 0 to 100.
- Value: average words per agent turn.
Tool call accuracy#
Whether tools were called correctly.
- Heuristic: a tool call is good when its arguments are valid JSON, it has a result, and the result does not look like a failure (a JSON
errororerrorskey,"success": false, a status of 400 or more, or text starting with error, failed or failure). Value is good calls divided by all calls. - LLM judge: rates whether tools were called at the right time with correct arguments.
Transition accuracy#
Whether a conversation flow moved between nodes sensibly.
- Heuristic: a transition is bad when a node transitions to itself, or when the flow enters the same node more than twice. Value is good transitions divided by all transitions.
details.problemslists them. - LLM judge: rates whether transitions matched the conversation.
Transfer success#
Whether an attempted transfer connected. A transfer counts as attempted when there is a transfer_started event or a tool call with "transfer" in its name. It succeeded when there is a transfer_bridged event or the call ended with call_transfer. Score 100 or 0.
Transfer wait time#
Time from transfer_started to transfer_bridged. Score is 100 up to 10 seconds and falls linearly to 0 at 70 seconds.
Latency#
The median (p50) of the call's end-to-end latency samples. Score is 100 at 1000 ms or less and falls linearly to 0 at 4000 ms. details.e2e holds the full percentiles.
Results#
Each metric result on a QA result has this shape:
{
"metric": "latency",
"value": 1180,
"score": 94,
"passed": true,
"details": { "threshold": 60, "judge": "heuristic", "e2e": { "p50": 1180, "p90": 1520 } }
}Scores are rounded to one decimal. The cohort overview shows, per metric, the number of calls where it applied, the average score, the average value and the pass rate. See QA results.