How to score candidates consistently across interviewers

To score candidates consistently across interviewers, give everyone the same questions and the same written description of a strong, acceptable and weak answer, practise rating a sample answer together before the first candidate, have each person rate alone, and then look at where your ratings differ. Even then, expect some disagreement. A 2013 meta-analysis found that ratings from separate interviews conducted by different interviewers agreed much less than ratings from panel members watching the same interview.

This article assumes you already have a structured interview: fixed questions, in a fixed order, with rating anchors. If you don’t, start with structured interviews, which shows how to build one. This is about getting several people to use it the same way.

Why interviewers disagree

When two interviewers rate the same candidate differently, it is usually for one of five reasons. The list is our synthesis; the research below bears on several of them.

Reason	What it looks like	Fix
Different questions	One interviewer improvised and asked something else	Same questions and follow-ups for everyone
Different standards	One person’s 4 is another’s 3, for every candidate	Written anchors, and calibration before you start
Different reading of anchors	The anchor is vague, so each person fills the gap with their own view	Rewrite the anchor where the gap showed up
One impression colours all	A strong first answer lifts every rating that follows	Rate each answer straight after it is given
Different information	Separate interviews: each heard different answers from the same candidate	Accept some of this; compare notes, not just numbers

The last one is not a mistake. Candidates answer differently on different days and to different people. That is part of why separate interviews agree less than panels.

What the research says

How much interviewers agree. Huffcutt, Culbertson and Weyhrauch (2013) pooled 125 reliability estimates covering 32,428 people. Mean agreement between interviewers (interrater reliability, where 1 would be perfect agreement) was .74 for panel interviews but .44 when different interviewers conducted separate interviews. They also flagged “lower than expected reliability for highly structured interviews conducted separately by different interviewers”. Structure helps, but it doesn’t make separate interviewers agree on its own.

Rater training works when it teaches the standard. Research on rating accuracy comes mostly from performance appraisal rather than hiring, so treat it as a strong hint rather than a direct finding about interviews. A quantitative review by Woehr and Huffcutt (1994) compared four kinds of rater training, and an update by Roch and colleagues (2012), with over four times as many studies, concluded that frame-of-reference training “is an effective method of improving rating accuracy”. Frame-of-reference training means showing raters examples of performance at different levels and discussing why each deserves its rating, so everyone shares the same picture of a 2, a 3 and a 4. The calibration session below is a small version of it.

Combine by rule, then discuss. A meta-analysis by Kuncel and colleagues (2013) found “consistent and substantial loss of validity when data were combined holistically”, even by experts. Averaging ratings by a rule you set in advance beats talking your way to a number.

Step 1: Share everything before the first interview

Every interviewer needs the same questions, the same allowed follow-ups and the same anchors, in writing, before anyone meets a candidate. Agree on the scale too. A 1–5 scale with anchors at 1, 3 and 5 is common; what matters is that everyone uses the same one.

Decide who asks what. You can have each interviewer ask every question, or split the questions between interviewers so each covers different requirements. Splitting saves candidates’ time, but then no two people rate the same answer and you can’t check agreement at all. If you split, calibration (Step 2) matters even more.

Step 2: Calibrate on a sample answer (30 minutes)

Before the first candidate, run this with everyone who will interview. It is our recommendation, adapted from frame-of-reference training, not a standard protocol.

1. Write three sample answers to your most important question. One strong, one middling, one weak, a paragraph each. Write them as a candidate would actually talk, with some vagueness. Don’t label them.
2. Everyone rates each sample alone, in writing, using the anchors.
3. Reveal the ratings together. Where you agree, move on.
4. Where you differ by more than a point, find the words. Each person points to the part of the answer that drove their rating, and to the part of the anchor they matched it to.
5. Fix the anchor, not the person. If two people read the same anchor differently, the anchor is ambiguous. Rewrite it so the next answer like this gets the same rating from both.

Thirty minutes once per role is usually enough. Repeat it for a new interviewer joining partway through.

Step 3: Rate alone, straight away, with evidence

During each interview, take notes on what the candidate said. After each answer, rate it against the anchor and write a line of evidence: “Owned the close calendar; handed consolidation to manager.” Submit ratings before talking to the other interviewers. If the first confident opinion is heard before others have committed, it anchors everyone else; our guide to making the final hiring decision explains why order matters.

Step 4: Look at where you differ

Put the ratings side by side for each candidate, one row per question. Look for two patterns:

• A spread of more than one point on a question. Go back to the notes. Did both interviewers hear the same answer? If they did and still disagree, the anchor needs work. If they heard different answers, the difference is real information, not error.
• One interviewer consistently higher or lower. If one person is a point above everyone else on every candidate, that’s a standard, not a judgement about candidates. Talk about it with the anchors in front of you.

Resist adjusting ratings statistically. Methods that convert each interviewer’s ratings to a common scale need many ratings per interviewer to be stable. With five finalists, one unusual candidate moves the adjustment more than the interviewer’s habits do. Fix the shared standard instead.

Step 5: Combine by a rule, then decide

Average the ratings for each question, then add up the questions (or use the weights you set beforehand). Use the discussion for the rows where ratings still differ and for anything someone heard that others didn’t. The candidate scorecard template is built for this step.

A worked example

Larkspur Partners, the fictional accounting firm in our guides, is hiring a Senior Accountant. A partner, the controller and a senior accountant each run a separate 45-minute interview with the same five questions from our structured interview guide. The ratings below are invented for illustration.

Before the first candidate, they calibrated on question 2: “It’s day three of close. A bank reconciliation is out by $14,000 and the partner wants the numbers tomorrow morning. What do you do, in order?” On the middling sample answer, the partner gave a 4 and the senior accountant a 2. The disagreement was about one sentence: “I’d let the partner know straight away that the numbers might be late.” The partner read it as good judgement; the senior accountant read it as giving up before investigating. The anchor said nothing about escalation, so they added: “A strong answer investigates first (bank feed, timing items, duplicates) and tells the partner early, with what is known and when an answer is expected.”

Their ratings for one finalist, Amara Nwosu:

Question	Partner	Controller	Senior	Spread	Average
1. Last month-end close	5	4	5	1	4.7
2. Reconciliation out	4	4	3	1	3.7
3. Audit working papers	3	5	4	2	4.0
4. New accounting system	4	4	4	0	4.0
5. Junior’s entries	4	3	4	1	3.7

Only question 3 differs by more than a point. The notes show why: in the controller’s interview, Amara described a year-end audit file in detail; in the partner’s, she gave a shorter answer about a different engagement. The interviewers heard different answers, so the spread is information, not error. They keep the average and note the stronger example. No anchor needed changing.

Where SortTheStack fits

SortTheStack doesn’t run interviews or store interview scores. It does the step before: it reads every application against your requirements and quotes the evidence it found, so every interviewer starts from the same facts about each candidate, and the requirements marked unclear are often the best interview questions you have. Team members can comment on each candidate, and when someone changes the product’s recommendation they add a reason, which is saved to the candidate’s decision record. See how the screening works.

Sources

• Huffcutt, A. I., Culbertson, S. S., & Weyhrauch, W. S. (2013). Employment interview reliability: New meta-analytic estimates by structure and format. International Journal of Selection and Assessment, 21(3), 264–276. doi.org/10.1111/ijsa.12036
• Woehr, D. J., & Huffcutt, A. I. (1994). Rater training for performance appraisal: A quantitative review. Journal of Occupational and Organizational Psychology, 67(3), 189–205. doi.org/10.1111/j.2044-8325.1994.tb00562.x
• Roch, S. G., Woehr, D. J., Mishra, V., & Kieszczynska, U. (2012). Rater training revisited: An updated meta-analytic review of frame-of-reference training. Journal of Occupational and Organizational Psychology, 85(2), 370–395. doi.org/10.1111/j.2044-8325.2011.02045.x
• Kuncel, N. R., Klieger, D. M., Connelly, B. S., & Ones, D. S. (2013). Mechanical versus clinical data combination in selection and admissions decisions: A meta-analysis. Journal of Applied Psychology, 98(6), 1060–1072. doi.org/10.1037/a0034156