How to review an AI shortlist without rubber-stamping it

Reviewing an AI shortlist means forming your own view, not agreeing with the tool’s. In practice that means four things. Decide on a few applications yourself before you look at the tool’s verdicts. Check the evidence behind its verdicts, not just the labels. Spend your time on the applications where its mistakes hide. Write down why you decided, especially where you disagreed. Research on people working with AI recommendations shows that, without deliberate effort, review tends to drift into agreement.

This article is about the review step itself: what a person does between the tool producing results and anyone being shortlisted or turned away. For which steps to automate in the first place, see what to automate in hiring. For checking a single result in detail, see why an AI tool ranked a candidate.

Approving is not judging

A person can be “in the loop” in two very different ways.

Approving	Judging
Reads the tool’s label and summary	Reads the evidence, and sometimes the application itself
Asks “is there a reason to disagree?”	Asks “what would I have decided?”
Spends the same few seconds on everyone	Spends time where the tool is most likely to be wrong
Disagrees rarely, without a written reason	Disagrees when the evidence says so, and records why
Can’t say afterwards why someone advanced	Can say what the decision rested on

Both look the same in a log: a person clicked a button. Only the second one catches the tool’s mistakes.

European data protection regulators draw the same line. Guidance from the Article 29 Working Party on automated decision-making, later endorsed by the European Data Protection Board, says an organisation “cannot avoid the Article 22 provisions by fabricating human involvement”. Oversight must be “meaningful, rather than just a token gesture”, carried out by “someone who has the authority and competence to change the decision”, who should “consider all the relevant data” (WP251 rev.01). That is written for the GDPR, but it’s a good test anywhere.

Why review drifts into agreement

Three findings explain why “a person reviews it” is weaker than it sounds. None of them studied hiring, so treat them as evidence about how people work with AI recommendations generally.

Explanations increase agreement, right or wrong. In user studies by Bansal and colleagues (2021), people worked with an AI about as accurate as they were. Explanations didn’t improve the team’s accuracy. Instead, they “increased the chance that humans will accept the AI’s recommendation, regardless of its correctness.” A tidy reason next to a verdict makes the verdict more persuasive, not more correct.

Forming your own view first helps, and people dislike it. In an experiment with 199 participants, Buçinca, Malaya and Gajos (2021) found that people often accepted an AI suggestion even when it was wrong, and that explanations alone didn’t fix this. Designs that made people think before accepting, such as deciding on their own before seeing the AI’s suggestion, “significantly reduced overreliance”. The trade-off was that participants rated the designs that worked best least favourably. The task was about nutrition, not hiring.

Requiring a human can give false comfort. Green (2022) surveyed 41 government policies that require human oversight of algorithms. He concluded that evidence suggests people are unable to perform the oversight functions these policies expect, and that such requirements can provide “a false sense of security”.

Taken together: a review step only works if it’s designed to make the reviewer think. The routine below is our recommendation, built on these findings. It is not a standard.

A review routine for one job

1. Decide on five applications before you look

Before opening the tool’s results, pick five applications at random and read them against your requirements. For each, write your own call: shortlist, maybe or no, with one line of reason. This takes about 20 minutes.

Then compare with the tool. Where you agree, move on. Where you disagree, work out who is right. Sometimes it will be the tool, because it noticed a line you skimmed past. Sometimes it will be you. Either way, you now know how this tool reads this job, which is the thing you need to know before trusting the rest.

2. Read in full where mistakes hide

A tool’s errors aren’t spread evenly. Read these applications yourself:

• Every applicant marked unclear on a must-have. “The application doesn’t say” is a question, not an answer.
• Applicants on the boundary between two outcomes.
• A sample of everyone turned away, whether by the tool or by a knockout rule. These are the mistakes nobody would otherwise see. Our screening audit shows how to size the sample.

3. Check the evidence on the rest

For the applicants the tool put forward, you don’t need to re-read everything. Check the evidence for the requirements that decide the job. Is each quote in the application? Does it say what the verdict claims? “Supported the month-end close” is not “ran the month-end close”.

4. Write down your decision in your own words

For everyone you shortlist or turn away, record the reason, in a sentence, in terms of your requirements. Copying the tool’s summary is not a reason. Where you disagree with the tool, say so and why. That record is what lets you, or anyone else, check the decision later. Our guide to writing a candidate recommendation covers the longer version for finalists.

5. Look at your own pattern

After a few jobs, look at how often you disagreed with the tool. A reviewer who never disagrees is either working with a perfect tool or not really reviewing. The five-application check in step 1 will tell you which.

A worked example

Larkspur Partners, the fictional accounting firm in our guides, is hiring a Senior Accountant. Must-haves include a CPA designation and owning a month-end close end to end; familiarity with CaseWare and corporate tax preparation are nice to have. The applicants, evidence and notes are invented for illustration.

The hiring manager read five applications before opening the results, and agreed with the tool on four. The fifth was an applicant the tool marked unclear on the close. The manager had read “managed the close checklist” as ownership. Re-reading, they decided the tool was right that it didn’t say who owned the close, so that became a phone screen question rather than a shortlist place.

The manager’s shortlist review for three applicants:

Applicant	Tool’s outcome	Evidence checked	Reviewer’s note
Amara Nwosu	Recommended	Close: “Comfortable owning the close end to end and explaining the numbers to owners.” Quote found, says it	Agree. Shortlist. Owns the close; CaseWare shown. Nothing to probe beyond the standard questions
Sofia Marchetti	Recommended	Corporate tax (nice to have): Not shown, “Prepare compilation engagements and supporting working papers”	Agree. Shortlist. Compilation work isn’t corporate tax, so Not shown is right. It’s a nice-to-have; ask about it in the interview
Tomás Herrera	Possible	Close: Unclear, “Present monthly financial packages to owners and advise on cash flow and budgeting”	Agree it’s unclear: presenting the numbers isn’t owning the close. Two roles have overlapping dates. Phone screen on both before deciding

None of these notes overrules the tool, and that’s fine. Each says what the reviewer checked and why they agreed. If Tomás’s phone screen shows he ran the close, they’ll move him forward and record why the evidence changed.

What to keep

For each job, keep:

• The requirements as applied, including any changes made during the review.
• Your five-application check: your calls, the tool’s, and what you concluded where they differed.
• A reason for each decision, in your words, with who made it.
• Every case where you overruled the tool, and why.

That is enough to answer the questions that matter later: why was this person turned away, who decided, and on what evidence.

Where SortTheStack fits

SortTheStack is built to be reviewed rather than approved. Every application gets an outcome (Recommended, Possible, Needs review or Requirements not met) with each requirement marked met, not met or unclear and the text from the application quoted. Every quote is checked against the resume before you see it, and the compare view marks the rows where finalists differ. Setting aside clear mismatches automatically is off by default, and no rejection is sent unless someone chooses to send one.

When you change the product’s recommendation, you add a reason, which is saved to the candidate’s decision record with who made the change, and team members can comment on each candidate. For the five-application check in step 1, read those applications from the original files before opening their results. See how the screening works.

Sources

• Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. (2021). Does the whole exceed its parts? The effect of AI explanations on complementary team performance. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, 1–16. doi.org/10.1145/3411764.3445717
• Buçinca, Z., Malaya, M. B., & Gajos, K. Z. (2021). To trust or to think: Cognitive forcing functions can reduce overreliance on AI in AI-assisted decision-making. Proceedings of the ACM on Human-Computer Interaction, 5(CSCW1), 1–21. doi.org/10.1145/3449287
• Green, B. (2022). The flaws of policies requiring human oversight of government algorithms. Computer Law & Security Review, 45, 105681. doi.org/10.1016/j.clsr.2022.105681
• Article 29 Data Protection Working Party, Guidelines on automated individual decision-making and profiling for the purposes of Regulation 2016/679 (WP251 rev.01, adopted 6 February 2018; endorsed by the European Data Protection Board, Endorsement 1/2018), accessed 2 November 2026.