Independent AI test and evaluation

Know whether your AI works before you bet on it.

I manage AI test and evaluation programs for the Pentagon. I give program offices, defense primes and AI companies the same independent answer: does this AI work well enough to trust, and if not, why not? I have no stake in which vendor wins.

Book a free 30-minute call

30 minutes with me, not a sales team. No preparation needed.

Prefer email? scott@aieval.org

Scott Mastin

Track record

Army Research LaboratorySensor and imagery AI testing, 2007 to 2019

The PentagonAI test and evaluation programs, 2019 to present

AI Eval CorporationPresident, independent AI test and evaluation

When to call me

An outside answer, before the decision is made.

Before you buy or renew.

Find out whether the system does what you need, with your data and your users, before the contract is signed.

When the numbers don't match.

The vendor's results look better than what your users see. I find out which one is right, and why.

Before fielding or an ATO.

Leaders get one plain sentence: what the system can be trusted to do, under what conditions, and what to watch.

When you built it yourself.

Your team tested its own work. A second, independent look catches what the builders can't see.

How I judge

Is it the model, the data or the workflow?

What was asked for, against what was delivered.

I start with the job the AI was bought to do, then look at what it actually does on real work.

Checked at least two different ways.

A finding that rests on one test is a guess. Every conclusion I give a leader has been checked more than one way.

A plain conclusion with the reasoning.

Ready, ready with conditions, or not ready. No pile of scores to decode.

The cause, so the fix lands.

When the answer is no, the reason is usually the data or the workflow, not the model. I say which.

Diagram of an AI capability that is not working. Independent testing sorts the cause into the model, the data, or the workflow. The model fix is to specialize an open model with a smaller, cheaper model tuned to the task, and then test it. The data fix, marked most common, is to trim inputs to what belongs in the product and add examples, or change how the AI reads it. The workflow fix is to redesign around what AI does well and use plain software for steps that do not need AI.
I check the model, the data and the workflow, so the fix lands in the right place.

What this looks like.

A team built an AI to write its reports. It read everything in the database, most of it unrelated, and the reports weren't usable. The model wasn't the problem; the data was. I cut its input to only what belongs in the report and gave it examples of a finished one. The output cleaned up, and the team started relying on it.

Before and after for an AI tool that writes summary reports. Before, everything it could find goes in, most of it unrelated, and the output is not usable. After, only the data that belongs in that product goes in, plus examples of the finished product, and the output is clean enough that end users can rely on it.
Before, the reports weren't usable. After, the team started relying on them.

Who this is for.

  • Government program offices deciding whether to buy, field or keep an AI system.
  • Defense primes that need an independent read on AI they are integrating or delivering.
  • AI companies selling into government that want to know how their system will hold up under an outside evaluation.
  • Smaller teams watching an AI bill grow: see AI spend.

Writing

Questions I get asked.

How it works.

01

Talk.

30 minutes. Tell me what the AI is supposed to do, what it's doing now, and what decision is coming.

02

Find the question.

I'll tell you what I'd test first and what a useful answer looks like for your decision.

03

Decide together.

If it makes sense to keep going, we work out what that looks like. If it doesn't, you still leave with a clearer picture.

About

Scott Mastin

Scott Mastin

President, AI Eval Corporation

I manage AI test and evaluation programs for the Pentagon. For nearly two decades I've answered one question for senior leaders: does this AI work well enough to trust? When the answer is no, the reason is usually the data or the workflow, not the model.

I'm a service-disabled veteran, and I lead AI Eval Corporation, which does this work for the Department of War.

  • Army Research Laboratory. AI testing for sensor and imagery systems, 2007 to 2019.
  • Project Maven. Former AI test and evaluation program manager, 2019 to 2023.
  • OUSD(R&E) and the Chief Digital and AI Office (CDAO). I manage AI test and evaluation programs for the Pentagon, 2023 to present.
  • AI Eval Corp
  • ARL
  • SDVOSB
  • Tradewinds

More on where this judgment comes from. AI Eval Corporation takes on this work under contract; learn more at aieval.org.

Common questions

What does an independent AI evaluation give me?

A plain conclusion (ready, ready with conditions, or not ready), the reasoning behind it, where the system breaks, and whether the problem is the model, the data or the workflow.

Do you work with vendors, or only with buyers?

Both. Buyers get an independent read before they commit. AI companies get an honest look at how their system will hold up when someone else tests it.

What happens on the first call?

You tell me what the AI is supposed to do and what decision is coming. I tell you what I'd test first and whether I can help.

What does it cost?

We'll talk about that once we both know whether I can help. The first call is free.

If this sounds useful, let's talk.

Tell me what the AI is supposed to do and what decision is coming. In 30 minutes we'll know whether I can help.

Book a free 30-minute call

Prefer email? scott@aieval.org