Precision and Recall in Candidate Shortlists: Measuring the Candidates You Never Saw

Every candidate search makes two kinds of mistake. It includes people who should not be there, and it excludes people who should. These errors are not symmetrical in cost, and — more importantly — they are not symmetrical in visibility. One generates constant complaints. The other generates silence. Teams optimise against the one that complains.

Definitions, stated precisely

Borrowing from information retrieval, where these terms have exact meanings:

Term	Formula	Question it answers
Precision	true positives / (true positives + false positives)	Of the profiles I reviewed, how many were worth reviewing?
Recall	true positives / (true positives + false negatives)	Of the qualified people who exist, how many did I find?

A search returning three perfect profiles out of three has precision of 100%. If there were four hundred qualified people in the market, its recall is 0.75%. Both statements are true simultaneously, and only the first one is visible from inside the process.

The asymmetry of visibility

A false positive is a profile on your list that fails on review. You see it, you reject it, you feel the cost as wasted minutes. Over a week, false positives produce a clear and loud signal: "this list is noisy".

A false negative is a person who would have been an excellent hire and who your search never surfaced. There is no artefact. No one reviews them, no one rejects them, no counter increments. The cost is real — a longer search, a weaker final slate, possibly a worse hire — but it is never attributed to its cause.

This asymmetry has a predictable consequence: teams tune their searches to reduce visible noise, which means adding filters, which means narrowing, which means recall falls further. The process feels like it is improving while the actual candidate pool shrinks.

Why boolean filters fail in both directions

Structured filters are exact on the fields they cover. Location, seniority, current title, years of experience, school — these are database columns and a filter on them is unambiguous.

The difficulty is that hiring criteria are rarely fully expressible in those columns. Consider a realistic brief for a senior engineering hire:

• Based in the EU — structured, filterable
• Ten or more years of experience — structured, filterable
• Has led a team through a period of rapid headcount growth — unstructured
• Has owned infrastructure at meaningful scale, not just contributed to it — unstructured
• Has worked somewhere without a large platform team to fall back on — unstructured

The first two criteria are handled perfectly. The last three are the ones the hiring manager actually cares about, and no filter expresses them. So the search returns everyone matching the structured criteria — which is a large, mostly irrelevant set — and the burden of applying the real criteria falls entirely on manual review.

That produces the worst of both errors at once. Precision is poor because the unstructured criteria are unapplied. Recall is also poor, because recruiters compensate for the noise by tightening the structured filters — restricting to specific titles, specific companies, narrower experience bands — which excludes qualified people whose structured fields happen to read differently.

The title trap

Job titles are the clearest illustration. Filtering on "Head of Infrastructure" excludes the person titled "Principal Engineer" who did exactly that job, and includes the person at a fifteen-person company whose title is aspirational. Titles are weakly correlated with the underlying reality, and filtering on them hard trades a large amount of recall for a small amount of precision.

Estimating recall when you cannot see the misses

The standard objection to taking recall seriously is that it is unmeasurable: computing it requires knowing the size of the qualified population, which is exactly what you do not know.

There is a workaround, adapted from the capture-recapture method ecologists use to estimate wildlife populations without counting every animal. The logic transfers cleanly.

Run two independent searches for the same role, using genuinely different methods — different platforms, different query strategies, or two different sourcers who do not see each other's work. Then measure the overlap.

Let A = qualified profiles found by search 1
Let B = qualified profiles found by search 2
Let O = qualified profiles found by both

Estimated population  P ≈ (A × B) / O
Recall of search 1    ≈ A / P  =  O / B
Recall of search 2    ≈ B / P  =  O / A

The intuition: if two independent searches keep finding the same people, the pool is small and both searches cover it well. If they barely overlap, both are drawing small samples from a much larger pool, and both have low recall.

Worked example

Two sourcers work the same brief independently for one day.

Quantity	Value
Qualified profiles found by sourcer A	40
Qualified profiles found by sourcer B	35
Found by both	7
Estimated qualified population	(40 × 35) / 7 = 200
Recall of sourcer A	40 / 200 = 20%
Recall of sourcer B	35 / 200 = 17.5%
Union of both searches	68 / 200 = 34%

Two competent sourcers, working a full day each, between them reached roughly a third of the estimated pool. That is a typical and sobering result — and it is invisible to any process that does not deliberately look for it.

Caveats worth stating

The estimate assumes the two searches are genuinely independent and that every qualified person is equally likely to be found by either. Neither assumption holds perfectly. If both sourcers use the same platform and similar heuristics, their overlap will be inflated and the method will understate the true population — meaning your real recall is worse than the estimate suggests. Treat the output as an optimistic bound and a directional signal, not a precise measurement.

Run it once per quarter on a live role. The absolute number is less useful than the change over time as you alter your sourcing method.

Deciding which error to optimise

The two errors trade off against each other, and the correct balance depends on the role.

Situation	Dominant risk	Implication
Large candidate pool, common skill set	False positives	Tighten criteria; missing people is cheap because there are many
Small pool, specialised or senior role	False negatives	Widen aggressively; every missed person is a material loss
Highly competitive market	False negatives	The people you miss are being hired by someone else
High-volume repeat hiring	False positives	Review cost compounds across many searches

The general rule: the scarcer the profile, the more expensive a false negative becomes relative to a false positive. For senior and specialised roles — precisely where outbound recruiting is most used — false negatives usually dominate, and yet the tooling most teams use is tuned to minimise false positives by narrowing. The instincts and the incentives point the wrong way.

Escaping the trade-off rather than tuning it

The trade-off between precision and recall is only fixed if the filtering mechanism is fixed. It exists because structured filters are the only cheap way to reduce a large set, and structured filters are a poor proxy for the real criteria.

Breaking it requires separating the two jobs that filtering currently conflates:

1. Retrieval should be wide. Use structured criteria only for genuinely hard constraints — the ones where a violation is disqualifying, such as location or work authorisation. Every soft criterion applied at this stage costs recall permanently.
2. Evaluation should be deep, and applied to the wide set. This is where the unstructured criteria get assessed — by reading what a profile actually evidences, not by matching a keyword.

Done manually, this is exactly the expensive configuration described in our unit economics model: wide retrieval means a large S, and review cost scales with S. It is the right approach that most teams cannot afford, which is why they narrow instead.

What changes the calculus is making evaluation cheap rather than making retrieval narrow. If each profile arrives with the specific evidence for each criterion attached — the sentence, the role, the link that supports or refutes it — then verification takes seconds instead of minutes, and wide retrieval becomes economically viable. That is the design premise behind Hiris: apply the unstructured criteria at evaluation time, at scale, and hand the recruiter the proof rather than the raw profile.

A practical protocol

Four steps, none of which require new tooling to start:

1. Write criteria in two columns. Hard constraints on the left, judgement criteria on the right. Only the left column is allowed to become a filter.
2. Record rejections with reasons. This is what makes precision measurable. Without rejection data you have no denominator.
3. Run an overlap test quarterly. Two independent searches on one live role. Compute estimated recall.
4. Review the excluded set occasionally. Take fifty profiles your filters removed and read them. If more than a handful were qualified, your filters are too tight — and you have just made a false negative visible, which is the only way that error ever becomes actionable.

The fourth step is the one teams skip and the one that changes minds. Reading fifty rejected profiles is uncomfortable in a specific and useful way.