How to test AI value before a bigger commitment

Testing before committing is sensible. It is also where a lot of AI money disappears.

The common failure is not a pilot that fails. A failed pilot is useful: you learn something and stop. The expensive failure is the pilot that never ends, never quite works, and never gets a decision.

A good test is designed to produce a decision. Here is how.

Why do so many pilots go nowhere?

• Nobody measured how the work performed before the pilot, so improvement cannot be shown
• The scope was too broad to finish or too vague to judge
• Success was never defined, so every result is “promising”
• The pilot ran on clean demo data, not real cases
• The people who do the work were not involved
• There was no date on which someone had to decide

Each of these is avoidable before the test starts.

Start with a baseline

Measure the current process before changing it:

• Volume: how many cases in a typical week
• Time: how long each case takes, including corrections
• Quality: how often it goes wrong, and what that costs
• Experience: what the people doing it find hardest

Rough numbers from the people who do the work are fine. No numbers at all are not.

Keep the scope narrow

Test one workflow, one team, one type of case. Exclude the awkward exceptions at first, and say so explicitly. You are testing whether the core idea works, not building the finished system.

Narrow scope is also what makes a test affordable and quick enough to reach a decision.

Write acceptance criteria before you begin

Agree in advance what result would justify going further. For example:

• Handles a stated share of typical cases without manual rework
• Reduces time per case by an amount the team considers worthwhile
• Produces no errors of a defined serious type
• Is something the team says they would keep using

Set the thresholds with the people who do the work. Write them down. Do not move them once results arrive.

Build in review gates

During a test, a person should check output before it reaches a customer, a record or a payment. Review gates protect the business while the system is unproven, and they generate the evidence you need: how often the reviewer had to intervene, and why.

Set an evaluation period and a decision date

Agree how long the test runs and the date the decision will be made. The period should be long enough to see a representative mix of cases, and short enough that momentum survives.

Put the decision meeting in the calendar on day one.

Make a real go/no-go decision

At the decision date there are three honest outcomes:

• Go: the criteria were met. Plan the full version, including support, ownership and running costs.
• Adjust: close, with a clear and specific reason. Run one more defined round, with a new decision date.
• Stop: the criteria were not met, or the value does not justify the cost. Record what you learnt and stop spending.

Stopping is a successful outcome of a test. It is the outcome a pilot that goes nowhere never reaches.

What should you pay for a test?

Price depends on scope, systems and how much discovery is needed, so beware anyone quoting before they understand the work. What you should expect for your money is clear: a defined scope, written acceptance criteria, a measured result against your baseline, and a recommendation you can act on, including a recommendation to stop.

It is worth paying more when the test uses your real data and real cases, involves your team, and leaves you with documentation you keep regardless of the outcome.

Questions to ask before you sign

• What exactly is in scope, and what is deliberately excluded?
• What are the acceptance criteria, and who agreed them?
• How will the result be measured against our baseline?
• When is the decision date?
• What do we keep if we decide to stop?
• What would the full version cost to build and to run?

A pilot without a decision date is not a test. It is a hobby.