How to design Artificial Intelligence (AI) pilots that lead to production decisions instead of endless experiments.
Pilot purgatory is the most common way AI programs lose momentum. The Massachusetts Institute of Technology (MIT) NANDA study published in August 2025 found that most enterprise generative AI pilots were not producing measurable impact, and that solutions bought from specialized vendors or built through partnerships succeeded about two thirds of the time, while internal builds succeeded about a third as often. Gartner had already predicted that at least 30% of generative AI projects would be abandoned after proof of concept by the end of 2025.
This is Part 7 of my series on running a n AI Transformation Program. The discipline described here works in banking, insurance, healthcare, logistics, retail and government.
In this article, you will learn:
Why pilots get stuck
How to run discovery workshops that frame the right problem
What belongs in a pilot charter
How to combine Agile, Kanban, Waterfall and Lean in one program
How to make and present the scale, pivot or stop decision
Why Pilots Get Stuck
A pilot designed only to impress will rarely be built to last.
Pilots stall for reasons that are visible on day one, if anyone looks.
No scale decision defined: Nobody agreed what result would justify production.
Wrong success measures: The pilot proves the technology works, not that the business outcome improves.
No production path: Security, data, integration and support needs were never planned.
No owner or funding: Nobody in the business owns the solution after the pilot team leaves.
Unchanged workflow: The tool was added on top of the old process instead of changing it, which is the learning gap the MIT study describes.
Most stuck pilots were never designed to move; build the path to production into the pilot from the start.
Discovery Workshops That Frame the Right Problem
The best pilots begin with a sharp problem, not a shiny tool.
A focused discovery workshop with business users, technologists, data owners and risk partners prevents most pilot failures. A practical agenda looks like this.
Problem framing: What is the problem, who has it and what does it cost today?
Current process walk: Walk through the real workflow step by step, including workarounds.
Value hypothesis: State how AI would change the outcome and by how much.
Feasibility check: Confirm data availability, integration points and vendor options.
Risk tier: Agree the initial risk level and the controls it implies.
Pilot shape: Agree scope, users, duration and success measures.
One well-run workshop saves months of building the wrong thing well.
The Pilot Charter
Agree on what success looks like before anyone writes a line of code.
The pilot charter is a short document signed by the business owner and the program lead. It turns good intentions into commitments.
Hypothesis: The specific improvement the pilot is expected to prove.
Success metrics: Measurable targets with baselines, such as cycle time, accuracy or capacity released. (OKRs and KPIs)
Exit criteria: The conditions under which the pilot stops early.
Path to production: The platform, controls, integration and support model if the pilot succeeds.
Owner and funding: Who owns the solution and who pays to scale it.
Timebox: A fixed duration, typically six to twelve weeks.
A charter signed before the build turns a pilot from a demonstration into a decision.
Hybrid Delivery: Agile (Scrum), Kanban, Waterfall and Lean
Choose the method that fits the work, not the method that fits the habit.
AI programs mix very different kinds of work, so one method rarely fits all of it.
Agile and Scrum for build and learn: Short sprints suit prompt design, retrieval tuning and user feedback loops.
Kanban for intake: A visible flow board suits the continuous stream of use case ideas and small requests.
Waterfall-style stage gates for controls, contracts and platforms: Waterfall-style gates suit risk approvals, vendor contracts and platform releases with fixed dates.
Lean for the workflow: Remove waste from the process before automating it, because automating a broken process only makes it fail faster.
Keeping vendors inside one plan
Shared milestones: Vendor deliverables appear in the program's integrated plan.
Joint RAID: Vendor Risks, Assumptions, Issues and Dependencies (RAID) are tracked in the same log as internal ones.
Exit and portability terms: Contracts allow you to change vendors or models without starting over.
Integrating two firms' treasury systems after a major bank acquisition taught me that mixed methods work only when everyone shares one plan and one set of gates.
Agile builds, Kanban flows, gates protect and Lean simplifies; the skill is knowing which to use where.
The Scale, Pivot or Stop Decision
Stopping a pilot with evidence is a success; continuing one without evidence is a cost.
At the end of the timebox, the program brings a clear recommendation to the sponsor.
Scale: The hypothesis is proven, the production path is funded and risks are acceptable.
Pivot: The value is real but the approach needs to change, such as a different vendor, scope or workflow design.
Stop: The evidence does not support the investment, and lessons are captured and shared.
Presenting the decision
Results versus metrics: What the pilot set out to prove and what it actually showed.
Cost to scale: One-time and ongoing costs, including people and controls.
Risks: What remains and how it will be managed.
Recommendation: Scale, pivot or stop, with the reasoning in plain language.
A program that scales winners, fixes promising ideas and stops weak ones with confidence will outrun one that keeps every pilot alive.
Key Takeaways
Pilots matter only when they lead to decisions.
Design for production: Define the scale decision, the owner and the production path on day one.
Start with discovery: Frame the problem and value before choosing tools.
Sign a charter: Hypothesis, metrics, exit criteria and timebox make pilots accountable.
Use hybrid delivery: Combine Agile, Kanban, stage gates and Lean to fit each kind of work.
Decide with evidence: Scale, pivot or stop, and celebrate a well-evidenced stop.
Discipline turns pilots from expensive experiments into the engine that moves AI from promise to production.
What is the most important thing you have learned about getting AI pilots into production?
Previous Articles in the AI Transformation Strategy Series:
Strategy to Execution: Standing Up an AI Transformation Program
The Governance Cadence: OKRs, KPIs and RAID for AI Programs
Responsible AI by Design: Controls at Every Stage of Delivery
AI in the Program Office: Agentic Tools for Transformation Delivery
Next in this series, Part 8: AI for the Workforce: HR Use Cases and Their Guardrails.
Explore more: Start Here · Topics · Archive · About