The EU AI Act in the call center: what it really means for agent evaluation

Regulation (EU) 2024/1689 — the EU AI Act — has stopped being a conference topic. It is binding law that directly affects call centers using AI to evaluate agents’ work. The good news: the duties are concrete and manageable. The bad news: “we’ll figure it out later” is no longer a strategy.

Why call evaluation is a high-risk system

Annex III of the AI Act, point 4(b), lists systems “intended to be used to monitor and evaluate the performance and behaviour of persons in work-related relationships”. A system that transcribes agents’ calls, scores them and aggregates results per person matches that definition literally — regardless of what the vendor calls it.

Be suspicious of vendors claiming their employee-evaluation tool is “not a high-risk system”. That classification is not a marketing choice; it follows from the product’s intended purpose.

Your duties as a deployer

A company deploying an AI evaluation system is a “deployer” in AI Act terms — with its own obligations that cannot be delegated to the vendor:

• Inform employees before going live — agents and their representatives must know they are subject to an AI system’s evaluation (Art. 26(7); in Poland, Art. 22³ of the Labour Code adds to this).
• Human oversight — evaluation results are reviewed by trained people with real authority to question and correct them (Art. 26(2)). AI scoring supports management decisions; it does not replace them.
• Adequate input data — recordings fed into the system should be of appropriate quality and consistent with the system’s intended purpose (Art. 26(4)).
• Logs — keep system usage logs for at least 6 months (Art. 26(6)).
• DPIA — a data protection impact assessment before switching monitoring on (Art. 35 GDPR); the provider must supply the necessary information (Art. 26(9) AI Act).
• AI literacy — people working with the results (managers, HR) must understand how the system works and where its limits are (Art. 4).

The red line: emotions from voice

Art. 5(1)(f) of the AI Act bans inferring employees’ emotions in the workplace from biometric data — and voice is biometric data. The practical takeaway: a call evaluation system should not analyze agents’ tone of voice or prosody, and configuring a criterion like “rate the agent’s emotional attitude” should be technically impossible, not merely contractually forbidden.

A safe architecture looks like this: language models receive transcript text only. Evaluating an agent’s emotions or state of mind isn’t “switched off” — it is unfeasible.

What to demand from your vendor

• An explicit classification of the system under the AI Act with reasoning — not an avoidance strategy.
• Explainable scores — every result must trace back to a specific call and a transcript quote, so a human can verify it.
• A technical block on emotion analysis of employees (a configuration validator), in line with Art. 5.
• A provider compliance plan — from labelling AI-generated content, through a risk register and technical documentation, to conformity assessment, the EU declaration and CE marking.
• Support with your own duties — an employee notice template, DPIA input data, training materials.

How CallSea approaches this

CallSea deliberately classifies itself as a high-risk system and follows a phased compliance plan mapped all the way to CE marking and registration in the EU AI database. Art. 5 guardrails are active today: a validator blocks criteria inferring agents’ emotions, and models evaluate transcript text only. Clients receive a deployment pack: the employee notice template, DPIA input data and AI literacy materials.