Independent AI test and evaluation

How to check an AI vendor's claims before you buy

Benchmark scores are easy to show. Your mission isn't a benchmark. Five questions to ask, and a test you can run yourself.

By Scott Mastin · 12 November 2026

Vendors show benchmark scores because they are easy to show. A benchmark is a standard test; your mission is not standard. A model can top a public leaderboard and still fail on your documents, your imagery or your users.

Five questions to ask before you sign

1. What was it tested on? If the answer is only public benchmarks, ask for results on data like yours, held out from training.
2. Who ran the test? A number the vendor produced and nobody else has reproduced is a claim, not a result.
3. What does a failure look like? Ask for the worst cases, not the average. Averages hide the cases that matter most.
4. What changes after delivery? Models get updated. Ask how you will know when performance shifts, and who checks.
5. Will they let you test it your way? A vendor confident in the product says yes. Resistance is information.

Then test it yourself, briefly

Take a small set of real tasks from your own work and run them through the system. Have the people who do that work judge the output without knowing which system produced it. Compare what you asked for with what you got.

This won't replace a full evaluation, but it will tell you quickly whether the vendor's numbers describe your situation.

Watch for the metric that gets gamed

When a vendor is paid or ranked on one number, the system tends to get better at that number, not necessarily at the job. Ask what the number leaves out, and look at the work directly.

When the purchase is large or the mission is serious, get an independent evaluation before you commit.