# How to score candidates consistently across interviewers

> Two interviewers can hear the same answer and give it a 2 and a 4. Why that happens, what the research says about how much interviewers agree, and a calibration routine a small team can run in half an hour, with a worked example.

Jennifer at SortTheStack, November 1, 2026. Original: https://sortthestack.com/blog/consistent-interview-scoring

**To score candidates consistently across interviewers, give everyone the same
questions and the same written description of a strong, acceptable and weak
answer, practise rating a sample answer together before the first candidate,
have each person rate alone, and then look at where your ratings differ.** Even
then, expect some disagreement. A 2013 meta-analysis found that ratings from
separate interviews conducted by different interviewers agreed much less than
ratings from panel members watching the same interview.

This article assumes you already have a structured interview: fixed questions,
in a fixed order, with rating anchors. If you don't, start with
[structured interviews](https://sortthestack.com/blog/structured-interviews), which shows how to build
one. This is about getting several people to use it the same way.

## Why interviewers disagree

When two interviewers rate the same candidate differently, it is usually for
one of five reasons. The list is our synthesis; the research below bears on
several of them.

| Reason                       | What it looks like                                                        | Fix                                                  |
| ---------------------------- | ------------------------------------------------------------------------- | ---------------------------------------------------- |
| Different questions          | One interviewer improvised and asked something else                       | Same questions and follow-ups for everyone           |
| Different standards          | One person's 4 is another's 3, for every candidate                        | Written anchors, and calibration before you start    |
| Different reading of anchors | The anchor is vague, so each person fills the gap with their own view     | Rewrite the anchor where the gap showed up           |
| One impression colours all   | A strong first answer lifts every rating that follows                     | Rate each answer straight after it is given          |
| Different information        | Separate interviews: each heard different answers from the same candidate | Accept some of this; compare notes, not just numbers |

The last one is not a mistake. Candidates answer differently on different days
and to different people. That is part of why separate interviews agree less
than panels.

## What the research says

**How much interviewers agree.**
[Huffcutt, Culbertson and Weyhrauch (2013)](https://doi.org/10.1111/ijsa.12036)
pooled 125 reliability estimates covering 32,428 people. Mean agreement
between interviewers (interrater reliability, where 1 would be perfect
agreement) was .74 for panel interviews but .44 when different interviewers
conducted separate interviews. They also flagged "lower than expected
reliability for highly structured interviews conducted separately by
different interviewers". Structure helps, but it doesn't make separate
interviewers agree on its own.

**Rater training works when it teaches the standard.** Research on rating
accuracy comes mostly from performance appraisal rather than hiring, so treat
it as a strong hint rather than a direct finding about interviews. A
quantitative review by
[Woehr and Huffcutt (1994)](https://doi.org/10.1111/j.2044-8325.1994.tb00562.x)
compared four kinds of rater training, and an update by
[Roch and colleagues (2012)](https://doi.org/10.1111/j.2044-8325.2011.02045.x),
with over four times as many studies, concluded that frame-of-reference
training "is an effective method of improving rating accuracy".
Frame-of-reference training means showing raters examples of performance at
different levels and discussing why each deserves its rating, so everyone
shares the same picture of a 2, a 3 and a 4. The calibration session below is
a small version of it.

**Combine by rule, then discuss.** A meta-analysis by
[Kuncel and colleagues (2013)](https://doi.org/10.1037/a0034156) found
"consistent and substantial loss of validity when data were combined
holistically", even by experts. Averaging ratings by a rule you set in advance
beats talking your way to a number.

## Step 1: Share everything before the first interview

Every interviewer needs the same questions, the same allowed follow-ups and the
same anchors, in writing, before anyone meets a candidate. Agree on the scale
too. A 1–5 scale with anchors at 1, 3 and 5 is common; what matters is that
everyone uses the same one.

Decide who asks what. You can have each interviewer ask every question, or
split the questions between interviewers so each covers different
requirements. Splitting saves candidates' time, but then no two people rate the
same answer and you can't check agreement at all. If you split, calibration
(Step 2) matters even more.

## Step 2: Calibrate on a sample answer (30 minutes)

Before the first candidate, run this with everyone who will interview. It is
our recommendation, adapted from frame-of-reference training, not a
standard protocol.

1. **Write three sample answers to your most important question.** One
   strong, one middling, one weak, a paragraph each. Write them as a candidate
   would actually talk, with some vagueness. Don't label them.
2. **Everyone rates each sample alone**, in writing, using the anchors.
3. **Reveal the ratings together.** Where you agree, move on.
4. **Where you differ by more than a point, find the words.** Each person
   points to the part of the answer that drove their rating, and to the part of
   the anchor they matched it to.
5. **Fix the anchor, not the person.** If two people read the same anchor
   differently, the anchor is ambiguous. Rewrite it so the next answer like
   this gets the same rating from both.

Thirty minutes once per role is usually enough. Repeat it for a new
interviewer joining partway through.

## Step 3: Rate alone, straight away, with evidence

During each interview, take notes on what the candidate said. After each
answer, rate it against the anchor and write a line of evidence: "Owned the
close calendar; handed consolidation to manager." Submit ratings before
talking to the other interviewers. If the first confident opinion is heard
before others have committed, it anchors everyone else; our guide to
[making the final hiring decision](https://sortthestack.com/blog/making-the-final-hiring-decision)
explains why order matters.

## Step 4: Look at where you differ

Put the ratings side by side for each candidate, one row per question. Look
for two patterns:

- **A spread of more than one point on a question.** Go back to the notes. Did
  both interviewers hear the same answer? If they did and still disagree, the
  anchor needs work. If they heard different answers, the difference is real
  information, not error.
- **One interviewer consistently higher or lower.** If one person is a point
  above everyone else on every candidate, that's a standard, not a judgement
  about candidates. Talk about it with the anchors in front of you.

Resist adjusting ratings statistically. Methods that convert each
interviewer's ratings to a common scale need many ratings per interviewer to be
stable. With five finalists, one unusual candidate moves the adjustment more
than the interviewer's habits do. Fix the shared standard instead.

## Step 5: Combine by a rule, then decide

Average the ratings for each question, then add up the questions (or use the
weights you set beforehand). Use the discussion for the rows where ratings
still differ and for anything someone heard that others didn't. The
[candidate scorecard template](https://sortthestack.com/blog/candidate-scorecard-template) is built for
this step.

## A worked example

Larkspur Partners, the fictional accounting firm in our guides, is hiring a
Senior Accountant. A partner, the controller and a senior accountant each run
a separate 45-minute interview with the same five questions from our
[structured interview guide](https://sortthestack.com/blog/structured-interviews). The ratings below
are invented for illustration.

Before the first candidate, they calibrated on question 2: "It's day three of
close. A bank reconciliation is out by $14,000 and the partner wants the
numbers tomorrow morning. What do you do, in order?" On the middling sample
answer, the partner gave a 4 and the senior accountant a 2. The disagreement
was about one sentence: "I'd let the partner know straight away that the
numbers might be late." The partner read it as good judgement; the senior
accountant read it as giving up before investigating. The anchor said nothing
about escalation, so they added: "A strong answer investigates first (bank
feed, timing items, duplicates) and tells the partner early, with what is
known and when an answer is expected."

Their ratings for one finalist, Amara Nwosu:

| Question                 | Partner | Controller | Senior | Spread | Average |
| ------------------------ | ------- | ---------- | ------ | ------ | ------- |
| 1. Last month-end close  | 5       | 4          | 5      | 1      | 4.7     |
| 2. Reconciliation out    | 4       | 4          | 3      | 1      | 3.7     |
| 3. Audit working papers  | 3       | 5          | 4      | 2      | 4.0     |
| 4. New accounting system | 4       | 4          | 4      | 0      | 4.0     |
| 5. Junior's entries      | 4       | 3          | 4      | 1      | 3.7     |

Only question 3 differs by more than a point. The notes show why: in the
controller's interview, Amara described a year-end audit file in detail; in
the partner's, she gave a shorter answer about a different engagement. The
interviewers heard different answers, so the spread is information, not error.
They keep the average and note the stronger example. No anchor needed
changing.

## Where SortTheStack fits

SortTheStack doesn't run interviews or store interview scores. It does the step
before: it reads every application against your requirements and quotes the
evidence it found, so every interviewer starts from the same facts about each
candidate, and the requirements marked unclear are often the best interview
questions you have. Team members can comment on each candidate, and when
someone changes the product's recommendation they add a reason, which is saved
to the candidate's decision record. [See how the screening works](https://sortthestack.com/product).

## Sources

- Huffcutt, A. I., Culbertson, S. S., & Weyhrauch, W. S. (2013). Employment
  interview reliability: New meta-analytic estimates by structure and format.
  _International Journal of Selection and Assessment, 21_(3), 264–276.
  [doi.org/10.1111/ijsa.12036](https://doi.org/10.1111/ijsa.12036)
- Woehr, D. J., & Huffcutt, A. I. (1994). Rater training for performance
  appraisal: A quantitative review. _Journal of Occupational and Organizational
  Psychology, 67_(3), 189–205.
  [doi.org/10.1111/j.2044-8325.1994.tb00562.x](https://doi.org/10.1111/j.2044-8325.1994.tb00562.x)
- Roch, S. G., Woehr, D. J., Mishra, V., & Kieszczynska, U. (2012). Rater
  training revisited: An updated meta-analytic review of frame-of-reference
  training. _Journal of Occupational and Organizational Psychology, 85_(2),
  370–395.
  [doi.org/10.1111/j.2044-8325.2011.02045.x](https://doi.org/10.1111/j.2044-8325.2011.02045.x)
- Kuncel, N. R., Klieger, D. M., Connelly, B. S., & Ones, D. S. (2013).
  Mechanical versus clinical data combination in selection and admissions
  decisions: A meta-analysis. _Journal of Applied Psychology, 98_(6),
  1060–1072. [doi.org/10.1037/a0034156](https://doi.org/10.1037/a0034156)