Skip to content
TriedDocs
Post-call analysis and QA

Resolution criteria

Write the named pass or fail conditions AI QA checks on every call, weight them, and understand how they become a score.

Resolution criteria are the checklist a QA judge applies to each call in a cohort. Each criterion has a short name and a plain-language condition. On every scored call, each one passes or fails with a one-sentence explanation, and the results become the call's score and part of its weighted score.

Add criteria#

In Create QA cohort or Edit QA cohort, the Resolution criteria section lists criteria: "Each criterion is a named condition the judge checks against the transcript."

Each criterion has:

Field API Limits Description
Name name 1 to 80 characters Shown on the QA sheet and in Criteria pass rate. Keep names unique within a cohort: results are matched by name.
Condition prompt 1 to 2000 characters What must be true for the criterion to pass. The placeholder reads "The agent ...".
Weight weight 0 to 100, default 1 How much this criterion counts in the weighted score.

Add criterion adds a row (up to 20). The trash icon removes one. A cohort needs at least one criterion with a name and a condition before you can save.

How criteria are judged#

With an OpenAI key#

gpt-5.6-terra reads the transcript and the list of criteria and returns, for each criterion, passed and a one-sentence explanation. In the same request it decides whether the caller's issue was resolved and writes a one-sentence quality summary. A criterion the judge does not return is marked failed with "The reviewer did not return this criterion."

The judge sees the transcript text (agent and user turns). Tool calls and node transitions are not part of the criteria prompt, so write criteria about what was said.

Mock mode#

Without an OpenAI key, or if the LLM request fails, a keyword scorer runs instead:

  • If the condition contains quoted phrases, every quoted phrase must appear in the transcript or the tool call data. The explanation names any phrase that was not found.
  • Otherwise the condition's significant words are checked, and the criterion passes when at least half appear.
  • A condition with no usable keywords passes unless post-call analysis marked the call unsuccessful.

The mock resolved verdict is the call's call_successful from analysis, or a score of at least 50 when there is none.

Tip

Put exact required wording in quotes, for example The agent says "this call may be recorded". Both the LLM judge and the mock scorer handle it well.

From criteria to scores#

Value How it is computed
Score (score) Criteria passed divided by criteria judged, times 100, rounded. Weights do not apply.
Resolved (resolved) The judge's verdict, independent of the score.
Weighted score (weighted_score) Each criterion counts 100 if passed and 0 if failed, multiplied by its weight; applicable metric scores are added with their weights; the total is divided by the sum of weights. Rounded to one decimal.
Pass (passed) Weighted score at or above the cohort's pass threshold.

A weight of 0 leaves a criterion out of the weighted score but keeps it on the QA sheet and in the plain score.

Example: three criteria with weights 2, 1, 1, where the first and third pass, and a latency metric (weight 1) scoring 80:

(100*2 + 0*1 + 100*1 + 80*1) / (2 + 1 + 1 + 1) = 380 / 5 = 76

With the default pass threshold of 70 the call passes, while its plain Score is 67 (2 of 3 criteria).

Writing criteria that judge well#

  • One behaviour per criterion. "Confirmed the time" and "Offered a reminder" are easier to judge and to act on than one criterion that checks both.
  • Observable in the transcript. "The agent read the appointment date back to the caller" works; "The agent was confident" does not.
  • Say what counts as passing in edge cases. "If the caller hangs up before a time is agreed, this passes."
  • Match the agent's job. Pull wording from the agent's prompt so the judge checks what you asked the agent to do.
  • Use weights for priorities. Give compliance items a higher weight than nice-to-haves.

Examples#

Name Condition
Greeted the caller The agent greeted the caller and named the business in its first turn.
Disclosed recording The agent says "this call may be recorded" before collecting details.
Confirmed details The agent confirmed the caller's name and phone number by repeating them back.
Booked or moved the appointment The call ends with a new or changed appointment that the caller agreed to.
Escalated correctly If the caller asked for a person, the agent offered a transfer or took a message. Passes if the caller never asked.

Changing criteria#

Edits apply to calls scored afterwards. Existing results keep their criteria results until the call is scored again (saving WER corrections on a call re-scores it). The Criteria pass rate chart uses the cohort's current criteria names, so a renamed criterion shows only results scored after the rename.

QA cohortsThe rest of the cohort settings.QA metricsAutomatic metrics blended into the weighted score.QA resultsRead criteria results on the QA sheet.