How do you plan an AI pilot you can actually decide on?

Most AI pilots do not fail — they simply never end. A demo is run, it looks good on ten examples, someone asks "but what about the complicated cases", and there is no answer because nobody defined what is being tested. This guide is a business test plan: how to define a pilot so that at the end there is a table with numbers, and a decision follows from the table. It does not explain what a language model is or how to build RAG — it explains what the process owner should demand before anyone starts building.

A pilot tests one action performed by one user, and compares it with how it is done today. The three parts must be written in a single sentence before moving on:

Before moving on to a pilot, one question: can the existing alternative be improved without AI? Better document search, an updated policy document or a structured form sometimes solve the same problem cheaply and predictably. The four questions that decide whether AI fits the need — required quality, data, cost and the alternative — are set out on the service page.

Success conditions are set before the pilot, in writing, with the process owner — the person responsible for the business outcome, not for the technology. Four measures, each with a target the process owner fills in:

The pilot is tested on a scenario set — a collection of real or representative cases, with the answer or outcome a person would accept. The set is built before the system is, by people who know the domain rather than those building it, and deliberately contains what demos tend to leave out:

Alongside the set, define boundaries: which sources the system may access, and under which permission — the same permission as the user it serves, no more; which data may be sent to the model provider and which may not (and what its contractual terms are on data use); what is kept in logs and for how long. If the data includes personal information, these definitions are closed with whoever is responsible for privacy in the organization before the pilot, not after.

An AI system will fail in some cases — not a question of whether but of how. The difference between a pilot that can be approved and one that is dangerous to continue is what the system does at the moment of failure. Three mechanisms, each tested in the scenario set:

Everything above converges into one sheet, signed by the process owner before starting. This is the template; the examples are filled in for illustration only.

The pilot ends on the date that was set, not when the results are good. On that date the process owner makes one of four decisions, according to the resulting table:

Before AI goes into a product we define the experiment sheet with you: action, user and alternative; success conditions with the process owner; a scenario set and boundaries; and failure behavior. Then we build the pilot at the defined scope, run it with the trial group, and deliver the table — scenarios, score, cost — on which the decision is made. What the pilot includes, how quality and failures are tested, and what it takes to turn it into a stable product — on the service page, including exactly what we built in the published cases.

When the AI is part of a new product that has not launched yet — the pilot is part of defining the first version, and the decision after it determines whether the capability goes into the MVP or is deferred. When it is added to an existing business system — it enters the same anatomy of a system: permissions, data, integrations and monitoring.

What a business system includes beyond its screens — and where an AI capability fits in it — in the anatomy section of the software development page.