Four ideas make that constraint real rather than aspirational:
Structured, per-job criteria.
Every job gets its own requirements-based rubric, built from the job description and frozen once the job is created.
The model judges, code counts.
The LLM assesses the evidence for each requirement; deterministic code turns those judgments into the score and label, so the same judgments always produce the same number, and the arithmetic never drifts.
Test before trust.
No job can accept applications until its evaluator passes an automated verification suite, including deliberate manipulation attempts.
Governance by default.
Fair defaults, immutable criteria, audited exceptions and least-privilege access are built into the workflow, not written into a policy document.
The problem: hiring is a data-quality process
Speech and language datasets are only as good as the people who correct them. At Volga Partners, machine transcription produces a first draft, and native-speaking annotators correct it in a purpose-built editor. The quality of the data we deliver is bounded by who those annotators are.
That makes hiring a first-order quality process, not an administrative one. As we scaled to several languages in parallel, an inbox-and-spreadsheet process broke in three predictable ways:
Inconsistency.
Applicants for the same role were asked different questions and judged against criteria that were never written down.
Silence.
Candidates rarely knew where their application stood.
Reading load.
Careful reading of every application did not scale for a team that also ran interviews and onboarding.
LLMs are an obvious answer to the third problem. Used carelessly, they make the first problem worse. The rest of this article is about how we got the benefit minimizing the caveats LLM bring along.
Why “just ask the LLM for a score” fails
The naive design is tempting: paste the job description and a resume into a prompt and ask for a number out of 100. It demos well but would fail in production on four fronts.
LLM judges are unstable.
Research on LLM-as-a-judge shows that model evaluators are sensitive to answer order, length and self-preference, and that a single holistic judgment can shift between runs (Zheng et al., 2023; Wang et al., 2024). A number that moves when nothing about the candidate has changed is not a basis for hiring.
Resumes are attacker-controlled documents.
Any text an LLM reads can try to instruct it. Indirect prompt injection is well documented (Greshake et al., 2023) and is the top risk on the OWASP Top 10 for LLM Applications. In hiring, every applicant has a motive to try it.
Opaque scores hide bias.
Automated hiring tools can encode historical bias, and vendor claims of mitigation are rarely specified or validated (Raghavan et al., 2020). A single number gives a reviewer nothing to check.
Regulation now treats this as high-risk.
The EU AI Act classifies AI used for recruitment and candidate evaluation as high-risk, and New York City’s Local Law 144 requires bias audits for automated employment decision tools. Quietly automating a hiring decision is no longer only an ethical question.
Our conclusion: the model is useful as a careful reader, not as a decision-maker. Everything that follows is engineering in service of that line.
Six design principles
We turned the problems above into six requirements before writing any code. Each one is enforced by a mechanism in the system, not by guidance that reviewers are expected to remember.
The most important word in that table is enforced. A principle written in a policy document is followed on a quiet day. A principle built into the state machine is followed every day.
The system at a glance
The hiring module is part of the platform that we use, our workforce platform for transcription and annotation. It has four faces: a job editor and applicant pipeline for HR, a public careers board, a candidate portal, and an asynchronous evaluation pipeline (Fig. 1).
Policy layer
- Authentication, roles, per-job access
- Job state machine (Draft → … → Archived)
- Form schema validation + branching
- Optimistic concurrency (version + row lock)
- Stage moves, offer gate, override audit
Create-job pipeline
- Ranker builder → verifier → freeze
- Interview selection / generation
Rate limits & cost guards
- Per-route limits, per-job build quota
Two architectural choices matter more than the rest. First, all policy lives in one place: authentication, per-job access, state transitions and validation are enforced centrally, and no client talks to the data layer directly. Second, anything slow or untrusted, such as file handling and model evaluation, runs asynchronously in isolated workers, away from the request path.
A job moves through a short lifecycle: Draft → Created → Live → Off → Archived. Only drafts are editable. Creating a job is the moment its evaluator is built and verified. Turning a job off stops new applications while letting interviews and offers already in progress finish (Fig. 2).
Archive from any state · Restore returns it to where it was
An application moves through Applied → Assessment → Interview → Offer → Hired. The model contributes in exactly two places, and both are advisory:
- It scores the submission in the background against the job’s criteria.
- It conducts a short, structured voice interview, so every applicant receives the same questions under the same conditions.
Everything else is a human action: moving between stages, rejecting, sending an offer and granting the hired role. The candidate accepts or declines.
The Candidate Ranker: let the model judge, let code count
The core of the module is a per-job evaluator we call the Candidate Ranker. Its design rests on one separation: the model reads and judges evidence, and ordinary code does the arithmetic.
5.1From a job description to a requirement model
Every job is specified in five parts, each answering one question:
From these, the system builds a structured requirement model for that job. It separates must-haves from plus points, weights each requirement, marks the ones that are genuine dealbreakers as critical, and records which parts of an application can serve as evidence for each. The model is validated against the job’s actual form and frozen when the job is created.
This replaces criteria that lived in a reviewer’s head with criteria that are explicit, consistent across every applicant and fixed for the life of the job.
5.2One requirement at a time, with quoted evidence
At evaluation time, the model does not produce an overall impression. It works through the requirement model one item at a time and, for each, returns whether the requirement is met, how directly the candidate’s own words support that judgment, a short quote as evidence and a one-line rationale.
It also returns what a reviewer actually needs before an interview: the strongest matches, the gaps, what remains unknown, any red flags, and suggested follow-up questions that target exactly that uncertainty.
5.3Deterministic scoring
The model never outputs the final score. Code maps each per-requirement judgment to points, applies the weights and aggregates must-haves and plus points into a 0–100 role-fit score with an alignment label.
This has three practical consequences:
- Reproducibility. Two candidates with the same judged evidence always receive the same number.
- Auditability. Every point in a score can be traced to a requirement, a judgment and a quote.
- Honesty about evidence. Alongside the score, we report how much of it rests on clearly stated evidence rather than inference. A high score built on thin evidence looks different from a high score built on solid evidence.
5.4Dealbreakers cannot be averaged away
Weighted averages have a known flaw: enough strength elsewhere can hide a missing essential. We close that gap with a critical cap. If a critical must-have is not met, the alignment label is capped, whatever the numeric score.
In one test case, a candidate scored 78/100 on the strength of excellent plus points but lacked a critical requirement. The reviewer saw the 78, saw the label capped at limited, and saw a one-sentence explanation of why. Nothing was hidden, and the dealbreaker could not be outvoted.
5.5What a reviewer sees
Every evaluation opens with a fixed statement that it supports human review and does not make decisions. We found that saying so on every screen matters more than saying it once in onboarding.
Test before trust: verifying every evaluator before it goes live
A requirement model can be perfectly well-formed and still be wrong. It might weight the wrong things, ignore the application form, or be easy to manipulate. So we treat each job’s evaluator the way we treat code: as a deployable artifact that must pass a test suite before it ships (Fig. 3).
When a recruiter clicks Create job, the system runs a verification protocol in three stages:
Structural checks.
The requirement model is valid, every piece of evidence it relies on exists on the job’s form, and the full evaluation fits the model’s limits.
Synthetic candidates.
The system generates a set of test applicants from the job’s own description, spanning strong fits, weak fits and deliberate manipulation attempts.
Behavioral expectations.
The evaluator must rank and treat those candidates the way a sensible reviewer would, use the application form rather than ignoring it, and catch every manipulation attempt.
Two properties make this more than a smoke test. First, the synthetic candidates go through the same evaluator, prompts, and parsing path as real applicants, so we test the deployed system rather than an approximation of it. Second, manipulation resistance stops being a claim on a slide and becomes a per-job regression check that runs before any real candidate is scored.
If an evaluator fails, it is rebuilt automatically. If it fails again, the job stays in draft and HR sees a single plain-language reason. Separately, whenever we change evaluation prompts, an offline regression harness re-scores a stored set of cases and compares them against a baseline.
Verification is not free: creating a job takes a few minutes instead of a few seconds. It runs in the background, so nobody waits on it. We consider that an easy trade against discovering a mis-specified evaluator after real candidates have been scored by it.
Candidate documents are untrusted input
We designed the evaluator on the assumption that some applicants will try to manipulate it. Hidden instructions in a résumé, text addressed to “the AI reviewer” and similar tricks are treated as expected input, not edge cases.
Our stance has three parts:
Never obeyed.
Candidate text is data. The evaluator’s instructions and the candidate’s content are kept strictly separate, and candidate content is cleaned before evaluation.
Detected in layers.
Manipulation attempts are caught by more than one independent mechanism, and anything reported is verified against the candidate’s actual submission before it is shown.
Never hidden.
When an attempt is detected, the reviewer sees exactly what the candidate wrote, as plain text, with a clear warning. The decision about what that means for the application stays with a person.
That last point is deliberate. Silently discarding suspicious text would protect the score but hide useful information from the hiring team. Showing it turns an attack into a signal.
Data minimization
The model only sees what it needs to assess role fit. Name, email, phone number, address, photo and other identifying or non-textual fields are never sent to it. This narrows the room for the model to pick up on signals that have nothing to do with the job, and it keeps personal data out of third-party processing wherever it is not required.
Governance: defaults carry the ethics
Fairness is usually decided on a busy day, by whatever the system does when nobody changes a setting. So we put our governance into defaults and workflow rules.
Interview before offer.
An offer cannot be sent until the candidate has completed an interview.
Frozen criteria.
Once a job is created, its criteria and its interview are fixed. Later changes to a shared interview template never alter an interview candidates are already taking. Every candidate is assessed against exactly what they applied to.
Least-privilege access.
Each job lists the recruiters who may see it, and the system reveals nothing about jobs or applicants outside that list. Evaluation internals are never exposed to reviewers, which keeps the method consistent and prevents criteria from being gamed by people who can see them. Automated tests check that they do not leak.
Transparency for candidates.
Applicants get a portal showing each application’s progress, their interview, any offer to accept or decline, and messages from the hiring team. They see the exact wording and version of the data-processing consent they give, and joining a talent pool is optional.
We also applied established human–AI interaction guidelines literally: every automatic action explains itself in one sentence, and every disabled control states why it is disabled.
Engineering for production
An LLM pipeline that handles untrusted files and paid APIs fails in mundane ways long before it fails in interesting ones. Much of the work in this module went into controls that never appear in a demo.
None of these is novel on its own. Together, they are the difference between a prototype that impresses and a system an HR team can rely on during a hiring surge.
What we have validated
We want to be precise about the kind of evidence we have. What follows is engineering validation: proof that the system behaves as designed. It is not yet a study of predictive validity or fairness, which we cover in the next section.
What we haven’t proven yet, and what’s next
We think an honest account of open questions is part of building trustworthy systems, so here are ours.
Agreement with expert reviewers.
We have not yet measured how closely the evaluator’s judgments match those of experienced human reviewers. Until we have, the score is a reading aid, not a quality claim.
Adverse-impact monitoring.
We do not yet report outcomes across demographic groups. This kind of monitoring is standard practice under US selection guidelines and expected for high-risk systems, and it comes before any claim about fairness.
The limits of synthetic testing.
Our test candidates are generated from the same job description the evaluator is built from. The protocol reliably catches gross failures, but it cannot prove that the evaluator measures what truly matters for the role. That requires real outcomes.
Dependence on third-party models.
Model behavior can change between versions. Frozen criteria and regression testing reduce this risk but do not remove it.
Our roadmap follows directly from that list:
- A blinded agreement study between the ranker outputs and experienced reviewers.
- Routine adverse-impact reporting on stage transitions.
- Calibrating alignment labels against real reviewer outcomes.
Lessons for anyone putting LLMs into high-stakes workflows
Test the model the way you test code.
The most valuable component we built was not the evaluator but the protocol that decides whether an evaluator may be used. Per-job synthetic and adversarial cases turned vague confidence into a pass/fail gate.
Let the model judge, consolidate deterministically.
Moving arithmetic out of the model made results reproducible and explainable, and made rules like the critical cap trivial to enforce.
Defaults carry the ethics.
Submission order, interview-before-offer and reasoned overrides did more for consistent treatment than any guidance document.
Explanation is a feature.
Screens that said why, such as why an interview was scored the way it was, why a score was capped, were trusted more and generated fewer support questions.
Conclusion
LLMs can take on the slow, careful reading that makes fair hiring expensive. They can only be trusted with it if the system around them ensures the model cannot decide, cannot silently change the criteria and is tested before it is used.
The hiring module combines frozen per-job criteria, deterministic scoring from evidence-quoted judgments, pre-deployment verification that includes adversarial cases, and governance built into the workflow. We think it is a practical pattern for any team bringing AI into decisions that affect people, and we’ll keep sharing what we learn as we measure it.
If you’re building AI into a workflow where the stakes are high and the inputs can’t be trusted, we’d like to hear from you. Contact Us – Volga Partners
Further reading
- European Union, Artificial Intelligence Act, Annex III (high-risk AI systems, including recruitment)
- New York City, Automated Employment Decision Tools (Local Law 144)
- OWASP, LLM01: Prompt Injection
- Greshake et al., Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection (2023)
- Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (2023)
- Wang et al., Large Language Models are not Fair Evaluators (2024)
- Raghavan et al., Mitigating Bias in Algorithmic Hiring: Evaluating Claims and Practices (2020)
- Amershi et al., Guidelines for Human-AI Interaction (2019)