How to audit your resume screening, whether a person or AI does it

To audit your resume screening, take a random sample of the applications you turned down, have a second person read them against the same requirements without seeing the original decision, and trace every disagreement to its cause. Where you have the data, also compare how often different groups pass the screen. The first check finds qualified people you’re losing. The second finds whether the losses fall unevenly. Neither needs special software, and the first takes about an hour.

This applies whoever does the first read: you, a colleague, a knockout question or an AI tool.

Why rejected applications need checking

A screen’s mistakes come in two kinds. Letting through someone who doesn’t fit is visible: you meet them and find out. Turning away someone who did fit is invisible: they never come back, and nothing tells you they were there.

These invisible mistakes are common. In a survey of 2,275 employers, Harvard Business School and Accenture (2021) found that 88% of employers using software to filter or rank applicants said it screened out qualified high-skills candidates at least some of the time. Applicants worry about it too. In iHire’s 2026 survey of 1,054 U.S. job seekers, 44.3% viewed employers’ growing use of AI in hiring negatively, and of those, 83.3% worried that qualified candidates may be unfairly screened out.

People make the same mistakes as software, more slowly. A reader who is tired, skimming for familiar job titles or thrown by an unusual career path will turn away people who meet every requirement.

Part 1: Re-read a sample of rejections

Step 1: Draw a random sample

From the applications that were turned down, take a random sample: every fifth one, or those whose names fall on a random number list. Twenty is a practical size for a small team. Don’t choose which ones to check; the interesting mistakes are in the applications nobody would think to pick.

Include the applications removed by knockout questions, and any marked “unclear” that were never followed up.

Step 2: Re-read them blind

Give a second person the requirements and the sampled applications, without the original decisions. Ask them to mark each requirement met, not met or unclear, with the evidence, exactly as the first reader should have done. Our guide to what counts as evidence has a simple way to judge it.

Step 3: Sort the disagreements

Second reader’s verdict	What it means
Agrees: clearly misses	The screen worked
Unclear on a must-have	Should have been asked, not rejected
Meets every must-have	A false rejection: a qualified person was turned away

Contact anyone in the third group if the job is still open. Then work out why each one happened.

Step 4: Trace each mistake to its cause

The cause matters more than the count. Common ones:

• A knockout question worded badly. Qualified people answered “no” because the question was ambiguous. See screening questions.
• A requirement that is really a proxy. A degree or a years-of-experience threshold removed someone who can clearly do the work.
• Title or keyword matching. The evidence was there under a different job title or wording.
• Reader fatigue. Check whether the mistakes come from late in the pile, or from a long session of reading.
• The tool’s settings. An AI tool set to reject when it was meant only to rank, or a rule switched on that nobody remembered.

Fix the cause, then re-check the rest of the rejected pile for the same problem. If a knockout question was wrong, every applicant it removed needs a second look, not only the ones in your sample.

What a clean sample does and doesn’t tell you

Finding no mistakes in twenty is good news, but it doesn’t prove the screen is perfect. A quick rule of thumb from statistics, the “rule of three”, says that if you find no errors in a sample, the true error rate could still be as high as about three divided by the sample size, with 95% confidence. No errors in 20 is consistent with an error rate of up to about 14%. No errors in 50, up to about 6%. A small audit catches big problems. It can’t certify a small one away.

Part 2: Check whether rejections fall unevenly

The second question is whether your screen passes some groups of applicants at a lower rate than others. That’s called adverse impact, and it can be unlawful even when nobody intended it.

The four-fifths rule

In the United States, federal enforcement agencies use a rule of thumb from the Uniform Guidelines on Employee Selection Procedures: “A selection rate for any race, sex, or ethnic group which is less than four-fifths (4/5) (or eighty percent) of the rate for the group with the highest rate will generally be regarded by the Federal enforcement agencies as evidence of adverse impact.”

An invented example: if 40 of 100 applicants from one group pass your screen (40%) and 25 of 100 from another pass (25%), the ratio is 25 ÷ 40 = 0.63, below 0.8.

The same section warns against treating the rule mechanically. Smaller differences can still be adverse impact when they are significant “in both statistical and practical terms”, and larger ones may not be when “the differences are based on small numbers and are not statistically significant”. For a small employer with a few dozen applicants per job, that second warning applies almost every time. Pool several jobs, or several months, before drawing conclusions.

You need data most small employers don’t have

You can’t calculate selection rates by group without knowing applicants’ groups, and collecting that is its own decision. It has to be voluntary, kept separate from the applications, and never seen by the people screening. In Ontario, the Human Rights Commission’s guidance says application forms should not ask about protected grounds. For employers who want to increase the diversity of their workforce, it suggests asking people to self-identify, with appropriate safeguards. Get advice before you start collecting it.

If you don’t have the data, Part 1 is still worth doing, and look at the causes it finds. A screen that removes people over unexplained gaps, unfamiliar schools or foreign job titles is likely to fall unevenly on someone.

If you hire in New York City with an automated tool

New York City’s Local Law 144 goes further for automated tools. An employer using an “automated employment decision tool” to assess candidates for jobs in the city must have had it bias-audited by an independent auditor within the past year, publish a summary of the results, and notify candidates. According to the city’s FAQ, the audit must at a minimum calculate “selection or scoring rates and the impact ratio across sex categories, race/ethnicity categories, and intersectional categories”. The law itself doesn’t require any particular action on the results, but anti-discrimination law still does.

Using a vendor’s tool doesn’t move the responsibility

If an AI tool or a test is doing part of the screening, ask the vendor what they’ve checked. But the EEOC’s guidance is plain: “While a test vendor’s documentation supporting the validity of a test may be helpful, the employer is still responsible for ensuring that its tests are valid.” Our article on whether AI can identify the best applicants has a test you can run on any tool before trusting it.

How often

• For the first few jobs you screen with a new process or tool, after each one.
• After you change anything: a new knockout question, new requirements, a new tool or a new setting.
• Otherwise, a few times a year, pooling recent jobs so the numbers mean something.

Write down each audit: the date, the job, the sample size, what you found and what you changed. If anyone ever asks how you know your screening is fair, that record is your answer.

Where SortTheStack fits

SortTheStack makes Part 1 straightforward, because every decision comes with its reason. Each applicant is marked Recommended, Possible, Needs review or Requirements not met, with each requirement marked met, not met or unclear and the evidence quoted, and a knocked-out applicant shows the rule that removed them. Setting aside clear mismatches automatically is off by default. When someone overrides the product’s recommendation, they add a reason, which is saved to the candidate’s decision record, and that record can be exported.

It can’t do Part 2 for you. SortTheStack doesn’t collect applicants’ race, sex or other demographic information, and its assessments are instructed not to infer them, so it can’t calculate selection rates by group. See how the screening works.

Sources

• Fuller, J. B., Raman, M., Sage-Gavin, E., & Hines, K. (2021). Hidden workers: Untapped talent. Harvard Business School Project on Managing the Future of Work and Accenture.
• iHire (2026). 2026 State of Online Recruiting Report. Survey of 318 employers and 1,054 job seekers in the United States, July and August 2026. Accessed 26 October 2026.
• New York City Department of Consumer and Worker Protection, Automated employment decision tools: Frequently asked questions. Accessed 26 October 2026.
• Ontario Human Rights Commission, Designing application forms, Human rights at work. Accessed 26 October 2026.
• U.S. Equal Employment Opportunity Commission, Employment tests and selection procedures. Accessed 26 October 2026.
• Uniform Guidelines on Employee Selection Procedures, 29 C.F.R. § 1607.4. Accessed 26 October 2026.