Automatic quality scoring with generative AI

Query-based quality assessment gets you a long way. Because most companies have a standard call flow, you can build searches that flag calls which do or do not follow it. But rules struggle with nuance — did the agent really understand the issue, or just tick a box? Generative AI closes that gap.

Automatic Quality Score is a Large Language Model (LLM) based assessment of call conversations. It evaluates interactions in a simple, structured way, focusing on the fundamentals of good communication, and produces a consistent score for every call — not a sample.

How the automated quality assessment works

The main users are team leaders and quality coaches who assess the performance of agents and teams on phone calls. Automatic, objective scoring across every call streamlines training, onboarding and QA.

The pipeline is deliberate:

1. Load and process the calls into the environment.
2. Transcribe them with purpose-built speech-to-text and take statistical measurements.
3. Classify and enrich with the relevant evaluations and metadata.
4. Anonymise using a combination of named entity recognition (NER) and black/whitelisting.
5. Score by feeding the anonymised conversation to the LLM with a prompt that assesses each key aspect and assigns a final score.

Crucially, the LLM works only with anonymised conversations, so no personally identifiable information (PII) is processed. Quality and security go together.

What are the key aspects?

The score focuses on five aspects of a service conversation: Greeting, Mapping the customer's needs, Explaining the next steps, Solving the issue, and Closing.

These are not box-ticking checks for saying "hi" or "bye". There are real expectations for each stage:

• Greeting — create a welcoming environment.
• Mapping needs — use open questions to understand the issue.
• Explaining next steps — help the customer understand what is being done and why.
• Solving the issue — resolve it, or escalate and explain why.
• Closing — recap what was discussed and done, so the customer leaves without confusion.

If the issue is not resolved, the conversation is additionally labelled with the reason why — useful data for analytics well beyond QA.

How to use the scores

Each call is scored per aspect and given a combined final score. That lets team leaders and quality coaches:

• Compare an agent's average per aspect against the team or overall average.
• Locate the calls with clear improvement opportunities, aspect by aspect.
• Decide which part of the conversation to treat as a strength or a weakness.
• Select specific calls to review and discuss with agents.
• Filter for topics or unresolved issues to find training needs.
• Monitor how a chosen area changes over time.

Dedicated views and templates help team leads and coaches filter their teams and agents, with easy access to average scores, the distribution of events per score, and monitoring methods.

Who else benefits?

Looking at NPS and conversations side by side shows that customer satisfaction hinges most on whether the issue was solved during the call. That makes resolution a focal point of the score — and the reason categories for unresolved cases are so valuable. Sometimes the process simply needs multiple contacts; sometimes it points to miscommunication or a training need.

Combine that with the reason for the call from automatic summaries and the assessment produces insight for process owners, product managers and others — not just QA. Scores can also be cross-referenced with existing metadata, classifiers and text searches. Today the score applies to phone calls, and it can extend to chats where needed.

Frequently asked questions

How is generative-AI scoring different from rule-based checks?

Rules confirm whether steps happened; an LLM evaluates how well — whether the agent understood the issue, used open questions and closed clearly — producing a more nuanced, explainable score.

Is customer data safe during scoring?

Yes. Conversations are anonymised with NER and black/whitelisting before the LLM sees them, so no PII is processed. See data security.

What does the final score include?

A score for each of the five aspects and a combined final score, plus a reason category when an issue is left unresolved.

Does it work for channels other than calls?

The score currently applies to phone calls and can be implemented for chats if the need arises.

Where to go next

• The pillar guide: Automated call-centre quality assurance
• The QA product: Automatic quality assurance
• The benefits, summarised: Benefits of automatic quality scoring
• Trust and security: Data security in conversation analysis

Curious how generative-AI scoring performs on your calls? Book a demo and we will score your own conversations.