# Why AI Projects Fail > AI projects rarely fail because the model was wrong. They fail because the organization asked an Experience-layer question of a company whose lower layers could not support the answer. This module walks through the five failure modes we see most, maps each to the Pyramid layer it actually comes from, and describes what a project that is set up to succeed looks like. - Series: Knowledge Module (SKM-004) - Published: 2026-10-22 - Updated: 2026-10-22 (v0.1.0) - Author: Summit ML - Tags: ai, architecture, decision-making - Canonical: https://summitml.com/knowledge/why-ai-projects-fail **The model is almost never the problem. The layers beneath it almost always are.** > **Audience:** CTOs · Product Leaders · Engineering Managers · Solution Architects > "How can we use AI?" is an Experience-layer question. The honest answer usually starts three layers lower. ## Executive summary The pattern is familiar enough to be a genre. A team ships an impressive demo in a week. Leadership is energized. A pilot is funded. Six months later the assistant gives inconsistent answers, the automation is paused after a bad week, the data science team is rebuilding a pipeline that already existed, and the project is quietly reframed as a learning exercise. The post-mortem usually blames the model, the vendor, or the prompt. In our experience, those are rarely the cause. The model did what it was asked with the data it was given. The failure was decided earlier, when an Experience-layer ambition was pointed at an organization whose capabilities, operational data, and automation could not carry it. This module names the five failure modes we see most often, locates each on the Summit Architecture Pyramid, and describes what the successful projects had in common. The short version: they all started lower than the demo. ## Five failure modes ### 1. The assistant has three answers An AI assistant for customers or staff is asked about an order, a balance, or an entitlement. It answers from whichever system it was connected to, and that system disagrees with the one the customer is looking at. Trust collapses on the first contradiction. **Pyramid layer: Operational Data.** The assistant surfaced a disagreement that already existed between systems. Retrieval and prompting cannot resolve it, because the organization never decided which system is right. SKM-003 covers why the warehouse does not help here. ### 2. The automation works until the schema moves An agent or workflow performs a multi-step task reliably in testing. In production, an upstream field is renamed, a picklist value is added, an integration starts returning a slightly different shape, and the automation either fails loudly or, worse, continues with wrong assumptions. **Pyramid layer: Operational Data and Automation.** The automation was built on undeclared transformation rules living inside the pipeline. When the rules were implicit, there was nothing to version, test, or observe. SKM-002 is the architectural answer: transformation as governed policy that automation reads rather than re-implements. ### 3. Nobody can say what "good" looks like The pilot runs. Usage is modest. The team cannot say whether it worked because the success measure was "adopt AI," which is not an outcome. Without an Experience-layer result tied to a business capability, the project cannot be defended at budget time and is cut. **Pyramid layer: Business Capability.** The project was never attached to a capability the business already values and measures. SKF-001's first and fifth questions exist to catch this before funding. ### 4. The demo was the product The demo was built on a small, hand-cleaned data sample. It handles one happy path and makes one model call per request. Then the team treats it as the production design. Making it real means solving login, permissions, speed, cost, and what happens when things break. All of that gets solved for the first time, on a deadline, after everyone has already seen the demo and set their expectations. **Pyramid layer: Applications.** A demo is an application without the application work. The successful projects we have been part of treated the demo as a hypothesis and budgeted the real application separately, with the lower layers checked first. ### 5. The team is solving the wrong layer's problem A data science team is asked to make the assistant more accurate. They improve the model, tune retrieval, and write evaluations. Accuracy improves on the test set and stays flat in production, because the production errors come from source data that is wrong, not from the model misreading it. **Pyramid layer: misdiagnosis.** Effort is being spent at layer five on a layer two problem. This is the most expensive failure mode because it looks like progress. The glue index in SKF-001 is the fastest way to notice it. ## What the successful ones had in common Across the AI work that has stuck, four things were true before the model was chosen. **The capability was named first.** Reduce order-status contacts. Shorten quote turnaround. Resolve catalog data issues before they reach the storefront. Each was already measured, so the project inherited a success metric instead of inventing one. **A system of record existed for everything the AI would read or write.** Where it did not, establishing one became the first phase, and the AI phase waited. This was unpopular for about a month and decisive for the following year. **Transformation rules lived somewhere governed.** The assistant and the automation read policy from one place. When the business changed a rule, it changed once. **Approval was designed in.** Writes to systems of record went through a preview and a human decision until trust was earned, and the record of those decisions became the evaluation set. This is the same guardrail pattern we use internally in Forge. > **Avoid:** Let's start with the assistant and clean up the data as we go. > **Prefer:** Let's find the capability, fix the entity it depends on, then put the assistant on top of data it can trust. ## A different first question The Pyramid replaces "how can we use AI" with "what architectural conditions let intelligence consistently create value here." The second question is less exciting and far more useful. It produces a short list of lower-layer work, most of which the organization needed anyway, and it produces an AI project that is small, specific, and attached to a number someone already cares about. Those projects do not make for dramatic demos. They do make it past month six. ## Key takeaways - AI projects fail at the Business Capability, Operational Data, and Automation layers far more often than at the model. - Three answers to one question, automation that breaks when a schema moves, and accuracy work that never lands in production are all layer two symptoms. - Name the capability, establish the system of record, govern the transformation rules, and design in approval before choosing a model. - A demo is a hypothesis about the Experience layer, not evidence that the layers below can support it. ## Related knowledge - SKM-001 — The Summit Architecture Pyramid (the model behind the diagnosis) - SKM-002 — The Agentic Data Layer (governed transformation as the fix for failure mode two) - SKM-003 — Operational Data vs Analytical Data (why the warehouse cannot fix failure mode one) - SKF-001 — The Architecture Pyramid as a Decision Framework (the five questions to run before funding) ## Changelog - 0.1.0 (2026-10-22) — First public draft.