Every AI evaluation vendor has a good-looking dashboard. What it can’t show you is who made the judgments, how your data was handled, or whether the operation can scale. Here are the four questions to ask before you sign.
Every AI evaluation vendor has a good-looking dashboard. The charts are clean, the throughput is high, and the quality score creeps up week after week.
But a dashboard can’t tell you who made the judgments behind those numbers. It can’t tell you how your data was protected, or whether the vendor can handle ten times the volume next quarter. Those answers are hard to spot in a demo, and they’re where vendors really differ. Here’s what to ask before you sign.
Human-in-the-loop evaluation means real people reviewing, scoring, and correcting what an AI model produces. It’s how teams find out where a model is weak, what it gets wrong, and what it should refuse. But “human in the loop” is only a selling point if the humans are the right ones, doing the right tasks, with your data handled properly. A dashboard can’t show you any of that.
What does “human in the loop” actually mean for evaluation?
Automated metrics can tell you how fast a model responds. They can’t always tell you whether a translation sounds natural, whether a financial explanation holds together, or whether a response crosses a line it shouldn’t. Those calls need human judgment, and that’s the “human” in human-in-the-loop.
The catch is that the loop is only as strong as the people in it. A vague promise of “human review” tells you very little. What matters is who the reviewers are, how qualified they are, how they were vetted, and how they were matched to the task. That’s what the rest of this checklist covers.
01Who is actually in your human-in-the-loop evaluation team?
Most vendors describe their workforce as a number, like “thousands of vetted linguists.” How those people got in matters more than how many there are.
At Volga, we keep a standing pool of roughly three to five thousand contributors, and it keeps growing. Most projects start by drawing on people we already know, so we’re not hiring cold. When a project needs expertise the existing pool lacks, we source through Workable and LinkedIn. For harder roles, we also use professional and language-specific communities, referrals, and direct outreach.
Every pipeline is built around the project. Budget shapes it, and so do location, data-residency, and compliance rules, which decide who is even eligible to do the work. You rarely see any of that on a dashboard.
How are evaluators vetted?
Good vetting has layers, and ideally it changes with the role. It starts with the resume and application: language and proficiency level, location and eligibility, and academic or domain background. From there, depending on the work, it can add:
- A short recorded introduction
- A subject-matter test of their expertise
- A language check against CEFR levels (B2, C1, or C2)
- A live interview on camera
The live interview does two jobs. It shows how someone communicates without a script, and it tests how deep their experience really goes. Ask a finance candidate about which valuation methods they’ve used directly, and you’ll quickly learn whether they’ve practiced the craft or only read about it.
Why don’t we publish a single pass rate? Our quality bar changes from project to project. A conversational translation role and a medical reasoning role screen for very different things, so one blanket percentage would hide what matters most. Plenty of people apply to any given call, and fewer clear the bar for the work in front of them. We vet for the work in front of us rather than placing whoever happens to be available.
How are the right people matched to each task?
Hiring good people is one problem. Putting the right person on the right task is another, and it’s where a lot of quality is quietly won or lost. Someone excellent at colloquial translation may be the wrong choice for a chain of financial reasoning or a clinical summary.
So matching starts with what the task requires, then lines candidates up against it:
- Language and proficiency level (graded L1 to L3)
- Specific language pair and regional variant
- Domain and professional background
- Localization experience
- Results on any project-specific tests
We also keep the roles separate. Recruitment finds, screens, and qualifies people, and a separate coordination team places them on workflows. Because both teams work from the same written requirements, nobody is skimming resumes for keywords. When you talk to a prospective vendor, ask whether they do any of this or just hand out the next task to whoever is free to complete the given volume.
02How does the vendor treat the people reviewing harmful content?
Not all of this work is comfortable. Some of it means reviewing harmful content, the material a model has to learn to recognize before it can refuse it. Someone has to read and label that content, and deciding where harmful ends and acceptable begins is part of how a model learns to behave. This is human-in-the-loop work at its most demanding: a person must make the call a model can’t yet make reliably on its own. The work is demanding because it matters.
That’s why ethical sourcing is no longer a footnote in these deals. Buyers now screen for it, as they should. It deserves a real conversation, not just a reassuring line in a proposal. Ask how the work is sourced, how it’s structured, and what support surrounds the people doing it. We’d rather have that conversation directly and in specifics.
03Where does your data go?
When you send an evaluation vendor your prompts and model outputs, you’re handing over some of your most revealing material: how your system behaves, where it’s weak, and what you’re trying to fix. Protecting that material is a condition of doing the work, not an add-on.
At Volga, our security program rests on two certifications:
- ISO/IEC 27001: an information security management system
- ISO/IEC 42001: an AI management system, aimed specifically at how AI-related data and outputs are governed
In practice, client data, prompts, model outputs, and evaluation results are controlled across their whole lifecycle. Access follows need-to-know and least privilege, so contributors see only what their task requires. Everyone who touches client work (employees, annotators, and contractors alike) passes a background check, signs a non-disclosure agreement, and completes security-awareness training before even starting. Controlled work environments, access monitoring, and set procedures for sensitive material back all of it up.
04Can the vendor scale without cutting corners?
The last question is whether the operation can grow without cracking, and whether it starts in the right direction at all. The costliest mistake in this work is moving fast in the wrong direction.
Why start with a pilot?
Most engagements should open with a small batch, and ours do. We take the task and guidelines, run a limited set of items (often across several languages), and put that work in front of you before scaling anything. Sometimes clients ask for a pilot, and often we suggest one.
The math behind it is simple. Imagine processing 1,500 items and then discovering that a certain type of case should have been handled differently. That could mean reworking all 1,500 items.
Identifying and addressing the same issue during a pilot comes at a fraction of the cost and saves significant time. We can also rework the pilot batch based on your feedback and preferences, ensuring that the scaled-up volume follows the agreed requirements and meets the expected quality standards.
How should scaling work?
After the pilot, scaling is really about planning and execution, not just making promises. We map our capacity against your timeline and stay realistic about what we can actually deliver.
If something is urgent, we start sourcing right away and ramp up at a pace we can sustain, rather than committing to a number upfront and then struggling to maintain it halfway through. Once the capacity is in place, the volume can follow smoothly. Throughout the process, we’ll keep you updated on our progress, any potential risks or challenges, and the opportunities we see along the way. Our goal is to make sure you’re always in the loop and have a clear view of where things stand.
The bottom line: look behind the dashboard
A dashboard is really the last step in a much longer process. By the time a number appears on it, everything that determines whether you can trust that number has already happened: who was hired and how, whether they were a good fit for the task, how your data was handled, and whether the ramp-up was properly planned or simply improvised. None of that is going to show up in a chart.
That’s the real point. Human-in-the-loop is only as reliable as the people and processes behind it.
A vendor can show you a perfect score, but if the hiring process is weak, the matching isn’t right, or data is being handled carelessly, those issues will eventually show up in the actual work.
So, spend less time evaluating the dashboard and more time looking at what feeds it: the hiring process, how people are matched to tasks, how your data is handled, and how scaling is planned. Ask for specifics. Don’t settle for a reassuring statement when you need a clear answer. Start with one simple question: What happens to our prompts and model outputs once they’re in your hands? A clear, specific answer shows you the operation behind the screen. A vague one tells you everything you need to know.