A convincing demo is not enough. Ask to inspect the work, the evaluation logic, and the operating boundaries before you ask candidates to participate.
What exactly does the assessment measure?
Ask for the role competencies, task boundaries, permitted tools, and scoring criteria. Request an example of a task that does not fit the product.
Can we inspect the evidence?
Ask the vendor to open the code, test output, or transcript behind a finding. Examine a conflicting or uncertain case, not only the strongest demo candidate.
What has actually been validated?
Distinguish integration tests from expert agreement, repeatability, fairness analysis, and predictive validity. Ask what population and workflow the results apply to.
What do candidates experience?
Review the invitation, time expectation, tool policy, accessibility, support, and data notice. A good buyer interface cannot compensate for an unclear candidate experience.
What happens when the system is wrong?
Ask about retries, incomplete sessions, human review, correction, deletion, and escalation. Agree on pilot success criteria and commercial rules before you start.