A cohort is the unit of AI QA. It says which calls to score and how to score them. You can have several cohorts, for example one per agent or one for calls that booked an appointment, and a call can be scored in more than one.
Create a cohort#
Click Create cohort in AI Quality Assurance. The dialog is titled Create QA cohort: "Ended calls that match these filters are scored against the resolution criteria." Click Create and score to save. Matching calls are scored in the background right away.
Settings#
| Dashboard label | API field | Type | Default | Description |
|---|---|---|---|---|
| Name | name |
string, 1 to 120 characters | required | Shown in the cohort selector. |
| Agents | agent_ids |
string[] | [] |
Leave all unticked to include every agent. |
| From | from_timestamp |
epoch ms | null | start of the day 29 days ago (dashboard) | Calls created at or after this date. |
| To | to_timestamp |
epoch ms | null | none | Calls created up to the end of this date. Leave empty for no end, so new calls keep being scored. |
| Min duration (s) | min_duration_ms |
integer | null | none | Entered in seconds in the dashboard, stored in ms. |
| Max duration (s) | max_duration_ms |
integer | null | none | Same. |
| Sampling | sampling_percent |
integer 1 to 100 | 100 | Share of matching calls to score. |
| Weekly max | weekly_max |
integer 1 to 10000 | 100 | Calls scored per rolling 7 days. |
| Disconnection reasons | disconnection_reasons |
string[] | [] |
Leave all unticked to include every reason. |
| Resolution criteria | criteria |
1 to 20 criteria | two examples (dashboard) | See Resolution criteria. |
| Extraction filters | extraction_filters |
up to 20 filters | [] |
Only score calls whose analysis, metadata or variables match every filter. |
| Metric judges | metrics |
up to 11 metrics | [] |
See QA metrics. |
| Pass threshold | pass_threshold |
number 0 to 100 | 70 | A call passes when its weighted score is at or above this. |
A new cohort in the dashboard starts with two criteria you can edit: "Resolved the request" and "Confirmed details".
Which calls are eligible#
A call is scored for a cohort when all of these hold:
call_statusisended(calls that never connected are not scored).- It is in the workspace and matches Agents, From, To, Min duration, Max duration and Disconnection reasons. Dates use the call's creation time.
- It falls in the sample.
- It matches every Extraction filter.
- It has a transcript.
- The cohort has scored fewer than Weekly max calls in the last 7 days.
- It has not already been scored for this cohort (for Run now).
New calls are checked as soon as their post-call analysis finishes. Calls that do not fit the weekly cap at that moment are not queued; use Run now later to score them.
Sampling#
Sampling is deterministic: whether a call is in the sample depends only on the cohort ID and the call ID. The same call is always in or out of a given cohort's sample, so re-running a cohort never picks a different set. With 100% every matching call is in.
Weekly max#
The cap counts results created in the last 7 days, not calendar weeks. Run now scores the newest eligible calls first and stops at the cap.
Extraction filters#
Each filter row has a source, a key, an operator and a value.
| Source | Key refers to |
|---|---|
| Analysis | A custom extraction field (appointment_booked), or a preset such as call_successful, user_sentiment or call_summary. Dotted paths work. |
| Metadata | A key in the call's metadata, dotted paths allowed. |
| Variable | A dynamic variable passed in, or one collected during the call. |
| Operator label | API op |
|---|---|
| equals | eq |
| is not | neq |
| contains | contains |
| does not contain | not_contains |
| >, ≥, <, ≤ | gt, gte, lt, lte |
| is set | exists |
| is not set | not_exists |
| is one of | in (comma separated in the dashboard, an array in the API) |
Text comparisons ignore case. true and false typed in the dashboard are sent as booleans. Numbers are sent as numbers for equals, is not and the comparison operators. A value is "set" when it is not missing, null or empty.
{
"name": "Booked calls",
"agent_ids": ["agent_3f9c0a1b2c3d4e5f6a7b8c9d"],
"sampling_percent": 50,
"weekly_max": 200,
"criteria": [
{ "name": "Confirmed the time", "prompt": "The agent read the appointment date and time back to the caller.", "weight": 2 },
{ "name": "Offered a reminder", "prompt": "The agent offered an SMS reminder." }
],
"extraction_filters": [
{ "source": "analysis", "key": "appointment_booked", "op": "eq", "value": true }
],
"metrics": [{ "metric": "latency", "weight": 1 }, { "metric": "hallucination", "weight": 2, "threshold": 80 }],
"pass_threshold": 75
}Edit, run and delete#
- The pencil opens Edit QA cohort. Saving changes the settings for future scoring. Existing results keep their scores until the call is re-scored.
- Run now scores eligible calls that have not been scored yet and reports "Scored N calls" or "No new calls to score".
- The trash icon deletes the cohort with its scores and overrides. Calls are not affected.
Gotchas#
- A cohort with a To date stops picking up new calls after that date.
- Changing sampling or filters does not remove results that no longer match.
- Calls without a transcript (for example voicemail with no speech) are skipped.
- Each scored call counts toward QA minutes. See Pricing.