Why most AI pilots don’t become systems

An insurer tests an assistant that summarizes claims files. A bank uses AI to prepare credit reviews. A revenue team generates account research in minutes.

The results are promising. Users want access. There is a plausible business case.

Six months later, the tool still needs the people who built it to keep it running.

The explanation usually involves data quality, integration, governance, or adoption. All four deserve attention. But they can obscure a more specific problem: the pilot was supported by work that never appeared in its evaluation.

Someone chose the documents. Someone knew which records were outdated. Someone checked an ambiguous answer with a colleague. Someone copied the result into the correct system and made sure it reached the right person.

That work helped the pilot succeed. Scaling it means deciding which parts to automate, which require judgment, and who will do the rest.

Until those decisions are made, the business case is incomplete.

What the pilot actually proved

A pilot can establish that a model performs a task well under a particular set of conditions. Those conditions matter as much as the result.

If an experienced analyst selects the inputs and checks every answer, the result demonstrates the performance of the analyst and the model together. It says less about what will happen when other users encounter unfamiliar cases without that support.

This is easy to miss because the assistance feels incidental. The analyst spends a few minutes finding the correct document or explaining an exception. Nobody considers it a separate workstream.

Across thousands of cases, those minutes become staffing requirements. The explanations become business rules. The exceptions become queues.

The distinction is familiar in machine learning research. Data dependencies, feedback loops, and surrounding infrastructure can create maintenance costs far beyond the model itself. More capable models do not make those dependencies disappear. Google Research, Hidden Technical Debt in Machine Learning Systems.

A useful pilot should therefore produce two things: evidence of model performance and a record of the work required to achieve it.

For every manual intervention, note what happened and why. Was the source incomplete? Was a business rule missing? Did the case require specialist judgment? Was the task unsuitable for the proposed workflow?

That record gives the implementation team something more actionable than a folder of good outputs.

Correct information can still lead to the wrong action

A policy clause can be quoted accurately and still have no bearing on a claim. It may come from the wrong policy period or have been superseded by an endorsement.

A financial figure can be correct and belong to the wrong entity.

An account signal can be genuine and already known to the salesperson responsible for the relationship.

These are problems of applicability. Adding more documents to a retrieval system may give the model more information without establishing which information should govern the decision.

The requirements differ by industry, but the underlying questions are recognizable.

Consider a claims recommendation prepared on Tuesday. On Wednesday, another document arrives.

Does anyone know the recommendation needs to be revisited? Does it remain in a queue looking complete? Has a downstream user already acted on it?

A production workflow needs an answer. It should retain the evidence and dates behind a recommendation, identify changes that invalidate it, and route the case for reassessment when necessary.

The same applies when a borrower submits revised financial statements or a target account enters an active sales process.

An answer has a useful lifetime. The system needs to know when that lifetime ends.

The review queue changes the economics

AI can accelerate a task without improving the process that contains it.

Faster claims preparation sends more work to adjusters. Faster credit analysis sends more reviews to an approval team. More account research creates more recommendations for sellers to investigate.

If the next step is already constrained, the immediate result may be a longer queue.

That does not make the productivity gain worthless. It means the organization has to decide how to use it. Perhaps preparation costs fall. Perhaps specialists can spend more time on difficult cases. Perhaps capacity needs to move between teams.

None of those outcomes follows automatically from faster generation.

Take an illustrative document-processing workflow:

These figures illustrate the calculation; they are not industry benchmarks.

Nothing has become less accurate. The referral rate is unchanged. But a manageable pilot now requires 100 specialist hours a day, before accounting for quality assurance, interruptions, or unusually difficult cases.

“Human review” is too vague to capture that requirement.

Who receives the cases? How much work can they absorb? Which cases take priority? What happens when the queue exceeds capacity?

The cases that pass without review also need inspection through an appropriate sampling process. Otherwise, reducing the referral rate can make the dashboard look better while allowing more errors through.

This is why the exception workload belongs in the original business case. Discovering it after rollout leaves the operating team to choose between delays, additional staffing, and weaker review.

An answer that is expensive to verify may save little time

Suppose an AI tool writes a credit review in two minutes. The reviewer then spends twenty minutes checking the figures, reopening documents, and identifying unsupported statements.

The generation time is impressive. Whether the tool saves work depends on what the review required before.

The design of the output makes a substantial difference. Material figures should be traceable to their sources. Missing evidence should be visible. A documented fact should be distinguishable from an interpretation.

A claims handler should be able to locate the passage supporting a coverage statement. A credit officer should be able to inspect the source behind a material calculation. A salesperson should be able to see the event behind an account recommendation and when it occurred.

A source link alone may be insufficient. Sending a reviewer to a 200-page document still leaves considerable work to do.

Useful evidence includes the relevant passage or record, its date and version, any unresolved conflicts, and the checks already performed. Where an action follows, the workflow should record what was proposed, who approved it, and what actually happened.

The measure worth tracking is the time it takes someone to reach an informed decision. Generation, verification, correction, and escalation all count.

The second pilot should cost less to operate

Claims, credit, and GTM workflows need different domain rules. They also share recurring requirements.

Each may need to identify an entity, check access, preserve evidence, assign work, record a decision, and recover from an unsuccessful update.

When every pilot builds its own version, inconsistencies accumulate. One application stores the source behind a recommendation; another retains only the summary. Two tools disagree about which legal entity a record belongs to. A failed update is retried differently across applications.

These differences become harder to manage as the tools interact.

There is a practical case for sharing capabilities such as identity, access controls, source metadata, evaluation records, and action logging. Domain decisions still need explicit ownership: a common logging service cannot determine whether a claim should be referred or a credit exception accepted.

The first workflow should leave behind capabilities that reduce the effort of operating the next one. Otherwise, the enterprise is funding a series of separate experiments with recurring setup costs.

What to settle before expanding a pilot

For one bounded workflow, document what happens from the arrival of a case to its accepted outcome. Keep it specific enough that an operating team could use it.

This also makes governance easier to implement. Oversight has a place in the workflow. Monitoring has defined measures and responses. NIST’s Generative AI Profile treats governance, evaluation, and monitoring as responsibilities throughout the system lifecycle. NIST, Generative Artificial Intelligence Profile.

The evaluation should follow the same boundary. Include preparation, review, rework, and operating effort when calculating cost per accepted outcome. Measure cycle time through the queues, rather than stopping the clock when the model responds.

Business measures should reflect the workflow. An insurer might track handling time alongside reopened cases. A bank might track review turnaround and material corrections. A GTM team might track qualified opportunities per seller hour and avoidable outreach.

Minutes saved become financial value only when the organization can use the released capacity. The business case should explain how that happens.

Questions that come up before rollout

Will a better model solve these problems?

It may reduce errors and review effort enough to change the economics. It will not assign decision authority, clarify an undefined policy, or staff a downstream team. Reevaluate the full workflow when the model changes; a different constraint may become the limiting factor.

Does the system need to act autonomously?

No. An assistant embedded in a human workflow can be a mature system. It needs dependable performance, clear responsibilities, and measurable outcomes. Human participation is compatible with all three.

Should the enterprise build a common platform first?

Establish known shared requirements early, particularly access, evidence, and evaluation records. Use actual workflows to discover what else should be shared. A large platform built ahead of that evidence can make assumptions expensive to change.

When is a pilot ready to expand?

When the intended operating team can run it under realistic conditions, including difficult cases and failures, at an understood cost. The people who built it should no longer be supplying unrecorded assistance that the business depends on.

Let the operating team run it

Before expanding, give the workflow to its intended users for an agreed period. Keep normal support and accountable reviewers in place, and record every occasion when the builders have to intervene.

Those interventions reveal what remains unfinished.

Perhaps a source has no clear owner. Perhaps a reviewer cannot find the evidence needed to make a decision. Perhaps the business has never agreed how to resolve a recurring exception.

Some gaps will require engineering. Others will require staffing or an explicit business decision. A few may show that the use case is not worth pursuing.

A pilot earns its value by making those facts visible. An impressive answer is a promising start; the work is complete when the people responsible for the process can depend on it.

Related reading: Data lineage, entity resolution, data orchestration, and intent data.