How to design call scoring criteria for AI analysis

Automated call scoring is exactly as good as the criteria it scores against. The good news: if your quality team already has a scorecard today, most of the work is done. It just needs to be translated into formats an AI can score unambiguously — with no room for “it depends”.

In this guide we walk through three criteria formats, show with examples the difference between a scorable and an unscorable criterion, provide a mini table of model phrasings, and describe calibration — the stage that decides whether your team will trust the results.

Start with the goal, not the form

Before you rewrite your current sheet, answer one question: what is this evaluation supposed to guard? Usually it is three things at once — compliance, sales or service effectiveness, and the communication standard. Each of these goals is best served by a different scoring format.

In practice, the emphasis depends on the campaign. On a sales line, evaluation mostly guards the process: needs discovery, offer presentation, proposing the next step. In complaint handling, accuracy of information and deadlines come to the fore. In debt collection, compliance carries the most weight, because a single prohibited phrase can cost more than the effectiveness of the whole campaign. That is why criteria are designed per campaign, not per company — one universal scorecard for every team ends in criteria so generic they measure nothing.

Three formats, three uses

1. Checklist — compliance and the call standard

A list of “happened / didn’t happen” items: greeting and self-introduction, informing about call recording, identity verification, summarizing what was agreed. The checklist works wherever the answer is binary — and in a typical scorecard it makes up 50–70% of the items.

A simple test for choosing the format: if you catch yourself adding “partially” while scoring, it is not a checklist criterion but a scale. “The agent summarized what was agreed” — binary, stays on the checklist. “The agent explained the contract terms clearly” — gradable, so it moves to a scale with described thresholds.

2. Point scale — quality, not just presence

Where how matters, not just whether — needs discovery, objection handling, clarity of explanations — a point scale with described thresholds works better. The key is the level descriptions: what exactly 2 points mean, and what 5 points mean.

An example for needs discovery on a 1–5 scale: 1 — the agent asked no questions about the customer’s situation; 3 — asked questions, but only closed ones, without confirming understanding; 5 — asked at least three open questions and paraphrased the answers before presenting the offer. Such a threshold description is long — and that is exactly why it works: the coach and the model look for the same thing in the transcript.

3. Critical errors — red lines

A separate category for events that invalidate the call regardless of the rest of the score: no consent to recording, misleading the customer, an unbacked promise, a skipped mandatory clause. A critical error does not lower the score — it triggers an alert.

The operational difference is fundamental: a checklist result is read in the weekly report, but a supervisor needs to know about a critical error immediately, with the specific fragment of the call as evidence. That is how critical error detection in CallSea works — an alert with a transcript quote reaches the supervisor the same day, not during the monthly sample review.

How many criteria should a call scorecard have?

Fewer than intuition suggests. Scorecards grow for years — every incident adds a new row, and after three years the sheet has 40 items, half of which nobody has analyzed for quarters. A sensible starting point for AI analysis is 10–15 criteria: 6–8 checklist items, 3–4 point scales and 2–3 critical errors.

Such a scorecard has three advantages. It can be calibrated in a week instead of a month. The dashboard shows readable trends instead of noise from 40 bars. And feedback for an agent fits into a single conversation — it is easier to fix 3 specific behaviours than to “raise the overall score by 12 points”.

Sample call scoring criteria — a mini table

Four criteria in three formats — with wording an AI can score unambiguously, and the typical phrasing mistake that ruins that clarity:

Note the mistakes column: each of these phrasings sounds reasonable in a spreadsheet, yet requires interpretation — and interpretation is exactly where two coaches (and the model) will drift apart in their scores.

Write criteria so that two people score them the same way

The test is simple: if two coaches could score the same recording differently, the criterion is too general — for AI as well. Compare:

Bad: “The agent was polite and professional.”

Good: “The agent introduced themselves with their name and the company name within the first 30 seconds of the call and did not interrupt the customer mid-sentence.”

A good criterion has three components: an observable behaviour (what exactly should be said in the call), a moment (when), and a pass condition (what counts as met). That level of detail makes the score repeatable and easy to defend in a feedback session — an agent will argue with “you weren’t professional enough”, but not with the fact “you didn’t introduce yourself in 4 out of 10 calls”.

What AI should not score: emotions and tone of voice

Criteria like “the agent was engaged and positive” have two problems. The first is craft: such a criterion cannot be scored repeatably — by a human or by a model. The second is legal: Art. 5(1)(f) of the EU AI Act prohibits inferring employees’ emotions in the workplace from biometric data, and voice is biometric data.

That is why CallSea’s models evaluate only the transcript text, and a configuration validator blocks criteria that infer an agent’s emotions — technically, not merely in the terms of service. Instead of “enthusiasm”, score behaviours visible in the text: did the agent answer every customer question, refer back to what the customer said earlier, propose a next step. We cover the red lines in more detail in our article on the EU AI Act in the call center.

Calibrate on real calls

Before you switch scoring on for all traffic, test the scorecard on 30–50 historical recordings. The procedure is simple:

1. Pick recordings from different result bands — not only model calls, but also those that ended in complaints and recordings from agents with extreme scores.
2. Score them independently: coaches according to their usual practice, the AI according to the new scorecard.
3. Compare the results criterion by criterion, not just in total — the divergence usually sits in 2–3 specific items, not in the whole scorecard.
4. Refine the descriptions of the divergent criteria and repeat the trial on the same set of recordings.

Divergences are usually not an “AI error” but an imprecise criterion. A good reference point: if on a given criterion two coaches disagree with each other as often as a coach disagrees with the AI, the problem lies in the definition, not in the model. Calibration does not end at launch — a human can verify and correct every AI score, and once you refine a contested criterion, the system scores subsequent calls against the corrected definition. It is worth repeating the trial after every change of script, offer or promotion.

From scorecard to dashboard

Well-designed criteria pay off at the management level: checklist and scale results build into trends per agent and per campaign, and critical errors land in alerts with a transcript quote. Instead of a discussion like “I have a feeling the team is getting weaker at needs discovery”, you have a chart that shows it — and a list of calls that confirm it.

On top of the scorecard comes an overall AI score on a 0–100 scale with a rationale — a first filter suggesting which calls to look at more closely; you will find a full description of the formats on the call quality scoring page. From every result on a chart you can drill down to the specific call and a transcript quote, and summaries arrive automatically — we show what that looks like on the dashboards and email reports page.

The starting point — why it is worth scoring all calls rather than a sample in the first place — is covered in: Manual reviews cover 1–2% of calls.