Last updated: November 8, 2026
Want this ready-made? See the Call QA Kit, or get the free sampler.
Before the session
- Pick 2 or 3 interactions. Include one clear pass, one clear fail and one borderline case. The borderline case teaches the most.
- Invite every evaluator, and the team lead if they coach from QA scores. Four to six people is the maximum that works.
- Remove agent names. Calibration is about the scorecard, not the person.
- Ask everyone to score alone, before the meeting, using the same scorecard and without discussing the call first.
Run the session in five steps
- Collect the scores in one table, criterion by criterion, for each evaluator.
- Find the gaps. Mark every criterion where evaluators differ. For a 1 to 5 rating, treat one point apart as agreement.
- Discuss the biggest gap first. Ask each evaluator for the exact moment in the call that made them choose their answer.
- Agree on one answer and write down why. If you cannot agree, the criterion is unclear. Rewrite it.
- Record the decisions in a short log: the call, the criterion, the agreed answer and any change to the scorecard wording.
What to measure
Keep it simple. For each session, note the share of criteria where all evaluators agreed, and which criteria caused the most disagreement. Over several months, the agreement rate should rise and the same criteria should stop appearing at the top of the list.
| Sign | What it usually means |
|---|---|
| One criterion keeps causing disagreement | The wording is vague. Add an example of a pass and a fail. |
| One evaluator is always stricter | They read the standard differently. Review it together with a real call. |
| Everyone agrees but agents dispute the result | Agents may not know the standard. Share the scorecard with them. |
Common mistakes
- Discussing the call before scoring it, so the first speaker sets everyone's answer.
- Using only easy calls, so no one learns anything.
- Skipping the written record, so the same argument returns next month.
- Counting calibration scores in agent results. Keep them separate.
Frequently asked questions
How often should we calibrate?
Monthly is a good default. Do it more often when you add evaluators or change the scorecard.
How many evaluators should score each call?
At least two. Three or four give a better picture of where the scorecard is unclear.
Should agents take part?
Some teams invite agents to a session as observers, which builds trust in the process. Start with evaluators only.
Skip the setup
The Call QA Kit includes a calibration sheet that compares evaluators criterion by criterion, plus a playbook for running the session. Scorebook QA does the comparison for you.