Many candidates now tailor their CVs to each advert with AI writing tools. That is a reasonable thing for a job seeker to do. It creates a problem for agencies that screen with AI. A tool that rewards overlap between a CV and the advert will rank a mirrored CV highly whether or not the evidence behind it is strong. Vendor demos rarely show this case. The only way to know how your tool behaves is to test it on your own roles, with CVs whose outcomes you already know.
Here is a test any agency can run in an afternoon.
Build a small test set
Start with one role you filled recently. A closed role matters, because you already know who was placed, who reached interview and who was rejected.
Take 10 to 15 CVs from that campaign across all three outcomes. Remove names, contact details, photos, dates of birth and anything else that identifies the person.
For each CV, create a mirrored twin. Keep the same jobs, dates, qualifications and achievements, but rewrite the wording to echo the advert. Add no new evidence. A recruiter can do this by hand, or an AI writing tool can do it under strict instructions; either way, check every twin against its original to confirm nothing was invented.
Add two or three plain but strong CVs: well qualified candidates who described their experience in their own words, without the advert's vocabulary.
Add two or three decoys: CVs that copy the advert's language but lack one stated essential requirement, such as a required licence or qualification.
Before you run the tool, ask two recruiters to rank the original CVs independently, without seeing the tool's output. That gives you a human baseline.
What to look for
Rank shift. Compare each mirrored twin with its original. If a twin jumps several places with no new evidence, the tool is scoring vocabulary rather than experience.
Decoy placement. Check whether any decoy outranks a qualified plain CV. A decoy near the top means the tool can miss an essential requirement when the wording looks right.
Consistency. Run the same set twice, then again after any vendor update. Rankings that move between identical runs are hard to defend to a client.
Explanations. Ask whether the tool can say why a CV scored highly, in terms a recruiter can check against the CV itself. “Strong match” is not an explanation. “Lists the required forklift licence” is.
Agreement with your recruiters. Note where the tool and the human baseline agree and where they part company. Each disagreement is a case worth reading closely.
Keep the data protection side tight
Past CVs are personal data, and UK data protection law applies to a test just as it applies to live screening. Use the minimum data you need and anonymise before anything enters the tool. Confirm whether your vendor stores uploaded files or uses them to train its models. Speak to your data protection lead before you start, and record the purpose of the test. If you have no lawful basis to reuse past candidate CVs, build synthetic CVs modelled on real ones instead. Delete the test set, or store it securely with access limited, once the test is done.
Keep a human in the loop
The test will almost certainly show that the tool is good at some things and weak at others. That is the point. Use the results to decide where human review has to sit. For example, a recruiter might read every CV whose rank shifts sharply with wording alone, or review a sample from the bottom of every long list before it is discarded. Candidates also have rights around decisions made solely by automated means, so a meaningful human check protects the agency as well as the candidate. A check is only meaningful if the reviewer has the time and authority to overturn the tool.
What the findings tell you
A match score measures similarity to the advert. It does not measure whether the claims are true, or whether the candidate can do the job. The mirrored CV test makes that gap visible on your own data. It also gives you specific questions for your vendor, a schedule for rechecking and a written record of how you tested your screening. When a client or candidate asks how a shortlist was made, that record carries more weight than a vendor brochure.
Run the test once, keep the set, and rerun it whenever the tool or the role changes.
About the author
Dr. Ijaz ud din is an Assistant Professor with a PhD in Computer Science from the University of Nottingham. He is a former LLM trainer and evaluator and the founder of CVGennie, an AI job application tool.
