A/B testing

Split calls on a phone number across agents or agent versions by weight, and split widget chats across chat agents.

A/B testing sends a share of live traffic to each of two or more agents so you can compare them on real callers. Tried supports weighted splits in three places:

Where	What is split	Configure
Inbound calls on a number	Agents, optionally pinned to versions	Inbound Call Agent, A/B Testing, or `inbound_agents`
Outbound calls from a number	Agents, optionally pinned to versions	Outbound Call Agent, A/B Testing, or `outbound_agents`
Chats from a public key (widget)	Chat agents	Chat agents, Add an A/B variant, or `chat_agents`

Each call or chat picks one entry at random, with a probability proportional to its weight. The pick happens once per call, so a caller stays with one agent for the whole call.

Split calls on a number#

Turn on A/B Testing#

On Phone Numbers, select the number. Under Inbound Call Agent or Outbound Call Agent, switch on A/B Testing next to Call Agent.

Set the weights#

The A/B testing weights dialog opens with two agents at 50% each. Pick an agent per row and enter a percentage. Add agent adds a row; Split evenly spreads 100% across the rows. The total must be exactly 100% and each agent may appear once.

Save#

Click Save. The card shows each agent with a bar and its percentage. Click Edit weights to change them later. Switching A/B Testing off keeps only the first agent, at 100%.

Inbound A/B split of 70% Receptionist and 30% Appointment booking on a number

Split with the API#

inbound_agents and outbound_agents are arrays of AgentWeight:

Field	Type	Description
`agent_id`	string	The agent.
`agent_version`	integer, optional	Pin this entry to a version. Without it the entry uses the number's version binding, or the agent's latest version.
`weight`	number, greater than 0 and at most 1	Relative weight.

curl -X PATCH "https://api.tried.com.au/update-phone-number/%2B61255501234" \
  -H "Authorization: Bearer $TRIED_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "inbound_agents": [
      { "agent_id": "agent_8e0c...", "agent_version": 4, "weight": 0.7 },
      { "agent_id": "agent_8e0c...", "agent_version": 5, "weight": 0.3 }
    ]
  }'

The example tests two versions of one agent: version 4 gets 70% of calls and version 5 gets 30%. The dashboard edits agents and percentages only; use the API to pin entries to versions.

Weights do not have to add up to 1 in the API. Entries are drawn in proportion to their weight, so 0.5 and 0.25 split two to one. Each agent and version is checked when you save; an unknown one returns an error.

How a split interacts with other settings#

Inbound webhook. The weighted pick is sent to your webhook as agent_id and agent_version in the call_inbound payload. If the webhook returns an override agent, the override wins. See Inbound call webhook.
Version binding. If the picked entry has no agent_version, the number's Agent version binding applies. For outbound calls, the binding applies when the number has exactly one outbound agent or the request names override_agent_id. See Environments.
Overrides. override_agent_id on POST /v2/create-phone-call, or override_agent_id on a batch task, skips the split.
SMS. SMS chats started on a number also use a weighted pick: outbound_agents for chats you start, inbound_agents for chats the customer starts.

On Chat Agents, Website widget, edit a public key and click Add an A/B variant under Chat agents. Give each chat agent a weight; the dialog shows the share as a percentage. Keys with a split show an A/B badge. Up to 10 chat agents per key.

Add public key dialog with two chat agents weighted 100 and 50 (67% and 33%)

The chosen chat agent is stored on the chat in metadata.ab_test. Chat splits are across chat agents, not across versions of one chat agent.

Read the results#

Every call records the agent_id and agent_version that handled it. To compare variants:

In Call History, filter by agent and compare success, sentiment, duration and disconnection reasons.
With the API, POST /v2/list-calls with filter_criteria.agent_id, then group by agent_version.
In Analytics, break charts down by agent.

Give a split enough calls before you judge it. A 70/30 split on a quiet number can take days to collect a useful sample for the smaller side.