Why most AI pilots stall before production

The demo answers the wrong question

A pilot answers whether something is possible. Production answers whether it is dependable on a Tuesday afternoon when the person who understands it is on leave, the input arrives in a format nobody anticipated, and the vendor API is returning errors.

Those are different questions, and the second one is where the work is. Most pilots that never ship were not technically wrong. They were simply never asked the second question, so nobody costed the answer.

What actually blocks the move

Permissions. A demo runs on a dataset someone exported. Production has to respect who is allowed to see what, enforced at the point of retrieval rather than in the prompt. That single requirement often rewrites the architecture.

Error handling. A demo has a happy path. Production needs defined behaviour for a missing record, an ambiguous request, a timeout, and a confidently wrong answer. Deciding what the system does when it does not know is design work, not an afterthought.

Evaluation. Without a written set of cases and expected behaviours, nobody can say whether a change made things better. Teams end up judging by vibe, which does not survive a stakeholder asking for evidence.

Cost. Per-call pricing that is trivial at demo volume becomes a line item at real volume. It is worth modelling before the build, not after finance notices.

A more honest sequence

Start with the process, not the technology. Establish what the work is, who does it now, which system holds the record, and what happens when it goes wrong today. That map is what tells you whether a model is even the right instrument.

Then run a paid, fixed-scope discovery phase whose deliverable is a decision rather than a prototype: target architecture, integration points named, risks written down, and a cost model. It should be capable of concluding that the work should not happen. A discovery phase that can only say yes is marketing.

Only then build, against acceptance criteria agreed in advance. The criteria are what turn sign-off from an opinion into a check against a list.

The uncomfortable part

Some processes should not be automated. They are low volume, high judgement, or so poorly defined that the automation would encode a mess. The correct output of discovery is sometimes a recommendation to fix the process on paper first and revisit the software later.

Firms that cannot deliver that recommendation will build the thing anyway. It is worth knowing which kind you are working with before the invoice arrives.