95, 80, 28: the war of AI failure numbers

Three numbers run most board conversations about AI this year. MIT's NANDA report says 95% of GenAI pilots show no measurable P&L impact. RAND is quoted for over 80% of AI projects failing, and Gartner reports that 28% of AI use cases fully succeed. They do not contradict one another, because they never counted the same object. The choice of what to count was made before any data was collected, and that choice is the story.

Failure statistics about AI are a publishing genre with its own economy. When you assess which questions were asked and how the results were presented, factor in the publisher's interest. Nobody had to lie for the numbers to land where they did.

A companion piece takes one of these apart in detail: what the Gartner 28% actually says. This one is about the genre.

Who makes the numbers, and what they sell

Start with the business model. Most of Gartner's revenue comes from recurring subscriptions to decision-grade research, with conferences and advisory adding the rest. The quotable number in the press release is the free sample; the product sits behind it. That is a description of a funnel. It is not the pay-to-play claim, which was litigated in NetScout v. Gartner — filed in 2014 and finally rejected by the Connecticut Supreme Court in 2020. Nobody is buying a result. The point is subtler and harder to defend against: the number that reaches you has already been selected for being quotable.

The selection is visible in one producer's own output. Inside thirteen months Gartner published three figures. In June 2025 it forecast that more than 40% of agentic AI projects would be cancelled by the end of 2027. Days later it published a survey result: 45% of organisations with high AI maturity keep AI projects operational for at least three years. In April 2026 came the 28%. The first and third travelled everywhere. The 45% has almost no media life at all. The pessimistic figures were not pushed harder than the optimistic one; they were picked up harder. Selection happens on the receiving end too.

The university lab has a product as well. The 95% figure came out of MIT Project NANDA — Networked Agents and Decentralized AI, based at the MIT Media Lab, with Ramesh Raskar as principal investigator. NANDA builds infrastructure for AI agents: a decentralised registry described as a DNS for agents, agent-to-agent protocols, identity and reputation verification. Its mission statement draws the analogy directly, comparing the missing agent layer to DNS and HTTP in the early web.

Now read the report's diagnosis next to that. Enterprise deployments fail, the report argues, because the systems lack memory and cannot learn from feedback. That is the same gap the NANDA infrastructure exists to fill. The report was not peer reviewed.

The consultant's version is the oldest one in the genre. BCG's October 2024 study, "Where's the Value in AI?", found that 74% of companies struggle to achieve and scale value from AI. Only 26% had the capabilities to move beyond proof of concept, and 4% were described as cutting-edge. The sample was 1,000 CxOs and senior executives across 59 countries and more than 20 sectors. Read the instrument carefully. It is a self-assessment of capability maturity, and it never claimed to measure project outcomes. It nonetheless circulates as a 74% failure rate, and BCG sells AI transformation.

The last exhibit is what happens when one of these numbers stops being a talking point. On 19 and 20 August 2025 AI-linked equities sold off: the Nasdaq fell more than 1.2%, Nvidia about 3.5%, Palantir close to 10%. Market commentary at the time named the MIT report alongside Sam Altman's bubble remarks as the proximate triggers. Both were in the same week, and the sell-off should not be pinned on the report alone. Still, a non-peer-reviewed working paper from a lab building agent infrastructure moved public markets, and its methodology did not travel with the headline number.

The number arrived. The denominator stayed home.

The five settings that decide the answer

Before any respondent is contacted, a study of failure has already chosen five things. Each choice moves the result by tens of points, and none of them is visible in a headline.

Unit of analysis. Use case, pilot, proof of concept, project, firm, or forecast. Six different denominators are in active circulation, and they are not convertible into one another.

Success threshold. Measurable P&L impact, reaching production, staying in production for three years, meeting a manager's ROI expectations, or matching the original estimate. Each threshold has its own literature and its own number.

Population. Infrastructure and operations leaders, C-suite respondents, interviewed practitioners, a fintech's customer base, or a nationally representative government sample.

Horizon. A project younger than the survey window counts as a failure by construction. An eighteen-month payback period registers as nothing at month nine.

Self-report versus measurement. A manager's judgement of their own ROI is not an audit, and the two can point in opposite directions. More on that below.

Set those five knobs and the answer is largely determined. Headlines usually leave these differences out, although the result cannot be read correctly without them.

Every row has its own denominator and its own threshold. Stacking them into a single "AI failure rate" is a category error, and the detailed dissection of the first row lives in the companion essay.

The genre is older than AI

None of this started in 2023. The template — an alarming percentage of technology projects fail — has been running since the mid-1990s, and it has already been taken apart twice by people with data.

The Standish Group's CHAOS report defines success as on time, on budget and on scope against the original estimate. Everything that overruns becomes "challenged"; everything cancelled becomes "failed".

Two independent critiques matter, and most citations of CHAOS acknowledge neither.

Moløkken-Østvold and Jørgensen, writing in Information and Software Technology 48(4) in 2006, examined the flagship 1994 figure: an average cost overrun of 189%. They found it far higher than comparable estimation studies of the same period, and traced the gap to a sampling method skewed toward failing projects. Their conclusion was that 189% is very likely much too high for typical 1990s projects. Continued use of the figure as a benchmark, they argued, produces bad decisions and holds back estimation practice.

Eveleens and Verhoef, in IEEE Software 27(1) in 2010, worked from 5,457 forecasts across 1,211 projects and raised four objections. A definition of success built purely on estimate accuracy is misleading. The measure is one-sided, so successes are systematically undercounted. Managing to the measure actively damages estimation practice, because the cheapest route to "success" is to inflate the original forecast. And averaging data of unknown bias produces figures with no interpretation.

That third objection has a name. Goodhart's law, in Marilyn Strathern's 1997 formulation, holds that "when a measure becomes a target, it ceases to be a good measure". Apply it forward. If your board starts reporting the percentage of successful AI use cases, your teams will start filing only the safe ones. The number will improve while nothing else does.

Methodological problems similar to those behind 95, 80 and 28 were described earlier for the CHAOS reports.

How a number mutates in transit

The genre has a transmission mechanism, and it is easiest to see on a single page.

Folio3 AI, a firm selling custom AI development and consulting, maintains a statistics page on AI project failure rates. Three of its entries repay a close look, purely as artefacts.

First: "80% of AI projects fail to deliver their intended business value", attributed to RAND's analysis of more than 2,400 enterprise AI initiatives. RAND analysed no such thing. The RAND report is a qualitative study built on interviews with practitioners, and its 80% appears as an estimate cited from elsewhere, hedged with "by some estimates". The figure acquired a sample it never had.

Second: "42% of companies abandoned at least one AI initiative in 2025", attributed to S&P. In the S&P data, 42% is the share of firms abandoning most of their initiatives, up from 17% the year before. Swapping the quantifier changes the construct entirely, and makes the number sound both smaller and more universal.

Third: "85% of AI project failures trace back to poor data quality (Gartner, 2025)". The origin is a Gartner forecast from 2018, which predicted that through 2022, 85% of AI projects would deliver erroneous outcomes because of bias in data, algorithms or the teams managing them. A forecast about wrong answers became a measured cause of failure, and 2018 became 2025. Three mutations on one page, all pointing the same way.

The canonical number of the genre never had a study behind it at all. "87% of data science projects never make it into production" comes, according to secondary accounts, from a July 2019 VentureBeat article which was sponsored content reporting a panel discussion at that publication's own conference. There was no sample and no source paper. The 78% figure that travels beside it came from a Dimensional Research survey commissioned by Alegion, a data-labelling vendor. Dozens of MLOps vendors still cite it as the evidence base for buying their tooling.

The mechanism has been named since 1979. Beverly Houghton coined the Woozle effect for evidence by citation: a claim gains credibility through the frequency with which it is repeated rather than through any evidence for it. Her own description was reification by accretion. It remains in current methodological use: a 2024 paper in Electronic Commerce Research applies it directly to belief persistence in research literatures.

Repetition is not replication. It only sounds like it.

One counterexample from inside the same commercial genre is instructive. Pertama Partners, an AI training and advisory firm, runs a page headlined with an 80% failure rate for 2026. The body then states that RAND's study is qualitative and built on interviews. It is best read as "the large majority fail" rather than as a precise percentage, the page says, and it links the RAND report. The fine print is correct; the headline is alarmist; the page sells readiness audits and workshops. The incentive lives in the title and the call to action, not in the analysis.

Why the extreme number is the one that travels

It is tempting to explain the selection with editorial malice. The evidence points at something duller and harder to fix.

Soroka, Fournier and Nir, in PNAS 116(38) in 2019, ran a psychophysiological experiment across 17 countries on six continents using real news material. The average person shows stronger arousal responses to negative content than to positive content. It is the broadest cross-national demonstration of negativity bias available, and the authors note substantial individual variation.

Trussler and Soroka, in the International Journal of Press/Politics 19(3) in 2014, showed the behavioural half. In a controlled setting, readers select negative and cynical stories, regardless of what they say they prefer. Stated preference and revealed preference come apart.

No editor needs to conspire. An outlet optimising for clicks and a conference optimising for attendance will converge on the most extreme available figure by ordinary means. The optimistic 45% loses to the pessimistic 40% forecast for the same reason a quiet quarter loses to a scandal.

Then there is the one control that would settle any of this, and that the genre almost never uses. The MIT NANDA report did not go through peer review. Kevin Werbach of Wharton, reading it repeatedly, reported that he could not reconstruct where the 95% came from, and set a clean condition: publish the full supporting data, or retract the report. The report mentions 52 interviews and hundreds of data points without disclosing sample demographics or collection method. As of this essay's writing, there is no trace of either a data release or a retraction. Werbach's wording is quoted as reported; the original was not verified. Absence of review proves nothing about the number either way. It only removes the mechanism that would have caught an error.

What measurement gives you instead

Replace the survey with a measurement and the numbers do not converge. They fragment differently, and the fragments are more useful.

METR ran a randomised controlled trial published in July 2025: 16 experienced open-source developers, 246 real tasks in their own mature repositories, using early-2025 tooling. Developers forecast that AI would make them 24% faster. After finishing, they estimated they had been 20% faster. Measured, they were 19% slower. Self-assessment and measurement pointed in opposite directions, on the same tasks, for the same people.

That is the strongest available argument against reading any self-report survey as a measurement of return. It is also a small study, in a narrow context, with tools from the first half of 2025, and METR itself revised the experimental design in February 2026. It does not show that AI slows developers down. It shows that asking people and measuring them give different answers.

Measurement also runs strongly positive, in the same period, on different work. Brynjolfsson, Li and Raymond, in the Quarterly Journal of Economics 140(2) in 2025, used a staggered rollout of a GenAI assistant across 5,172 customer support agents. Productivity, measured as issues resolved per hour, rose 15% on average — 34% for novices and close to nothing for experienced agents. Customer sentiment improved and employee retention rose.

Side by side, the two studies look contradictory, but they describe different work and different people. Experienced open-source developers working in their own repositories were 19% slower with AI. In customer support, the assistant raised productivity most for novices and barely changed it for experienced agents. Neither study yields a failure rate; each measures an effect in one specific work context.

The same mechanism shows up even in adoption. The US Census Bureau's Business Trends and Outlook Survey, a nationally representative government instrument, puts AI use at 17–20% of firms between mid-December 2025 and early May 2026, on a question reworded in November 2025. Ramp's AI Index, built from card and bill-pay transactions of more than 70,000 customers of one fintech, crossed 50% of firms paying for AI tools in March 2026. The thirty-point gap comes from a different population and a different definition of adoption: Ramp counts paid commitment, on a sample skewed toward technology firms.

Application: build your own reference class

The productive move is to stop importing anyone's number as a forecast for your own organisation.

In 2006 Bent Flyvbjerg described reference class forecasting in three steps:

1. Identify a reference class of past, similar undertakings.
2. Establish a probability distribution for that class on the parameter you are forecasting.
3. Compare your specific undertaking with that distribution to establish the most likely outcome.

Behind the method sits Kahneman and Tversky's distinction between the inside view, built on the details of your project, and the outside view, built on the distribution of outcomes for similar projects. Flyvbjerg added the observation that the outside view corrects two different errors at once: optimism bias, which is unconscious, and strategic misrepresentation, which is the deliberate understatement of cost to get a project approved. The method helps most with non-routine undertakings, which is what enterprise AI is for an organisation attempting it the first time.

That distinction transfers cleanly to the producers of failure numbers. Some of the skew is cognitive. Some of it is an editorial choice with a business behind it. The remedies differ.

Hold two base rates while you build your own.

Flyvbjerg's database covers more than 16,000 large projects. According to How Big Things Get Done (2023), 8.5% of them come in on budget and on time, and 0.5% come in on budget, on time and deliver the promised benefits. These figures come from a trade book rather than a peer-reviewed paper. The implication is uncomfortable for the whole genre. "Most initiatives fail to deliver what was promised" is the base rate for projects in general, not a peculiarity of AI.

Flyvbjerg and Budzier, in Harvard Business Review 89(9), September 2011, studied 1,471 IT projects. Average cost overrun: 27%. But one project in six was a black swan, with an average cost overrun of 200% and a schedule overrun near 70%. The mean describes almost nobody. The tail decides whether the company survives the project.

The entire war over 95 versus 80 versus 28 is being fought over a parameter that was the wrong one to ask for.

Here is the procedure I would put in front of a board, in order.

1. Define the unit before looking at any number. Use case, pilot, project or firm. Without this, nothing you read is comparable to anything you run.
2. Define the success threshold ex ante and in writing. Measurable P&L impact? Reaching production? Still running after three years? Each threshold has a different number attached to it in the literature, and picking one afterwards is how projects get graded on a curve.
3. Collect your own reference class. Ten to thirty closed initiatives from the last three to five years inside your own organisation — not only AI, but any change project of comparable complexity and novelty.
4. Build a distribution, not an average. Cost, time and benefit against the original forecast. Record the median and the P80 or P90.
5. Locate the new project inside that distribution and write down the resulting uplift on cost and on time. This is step three of reference class forecasting, and it is the step people skip.
6. Separate optimism bias from strategic misrepresentation. The first is treated with data. The second is treated by changing incentives: check whether the person promoted for getting a project approved is the person accountable for delivering it.
7. Price the tail before the mean. If one IT project in six overruns by 200%, the first question is what you do if this is that one, not what the average looks like.
8. Use external numbers only as an external reference class, with the denominator stated out loud, and never as a forecast for yourself.

Steps three and four are the whole exercise. The rest is bookkeeping.

Most organisations cannot execute step three, and that is the real finding. Suppose you cannot list thirty closed initiatives with their original forecasts and their actual outcomes. Then you have no institutional memory of your own delivery, which is why someone else's percentage looks attractive. It fills a gap that should not exist.

Limits, and the counterargument that matters

Four objections have force, and the third is the one that should change how you read everything above.

One: incentive analysis is not falsification. A number produced by an interested party can still be correct, and pointing at a business model does not refute a finding. Gartner sells subscriptions and may still be right about 28%. NANDA sells a protocol thesis and may still be right that memory is the binding constraint. This essay has a producer too, with an interest in appearing more careful than the sources it examines. Incentive analysis tells you where to check. It does not tell you what you will find.

Two: what we have here are disclosure gaps, not proven errors. For Gartner, BCG, McKinsey, Wharton and Ramp we have no published sampling frame, no weighting scheme and no question wording. That is undisclosed methodology, and it is a reason to hold the numbers loosely. It is not evidence that any of them is wrong.

Three: the direction converges, and the convergence is real. Take the numbers pointing the other way first, and apply the same scepticism to them. Wharton's Human-AI Research initiative, working with the consultancy GBK Collective, surveyed more than 800 senior leaders at firms with over 1,000 employees and $50m in revenue in October 2025. It found 74% already seeing positive ROI from GenAI, 72% tracking formalised ROI metrics, and 82% of leaders using GenAI at least weekly. That is the same class of evidence as the 95%: executive self-assessment, a sample selected by an interested party, and no independent audit of return. Scepticism that only runs in the pessimistic direction is not scepticism. It is a mood.

Now hold the convergence. McKinsey's State of AI published a wave in November 2025, with 1,993 respondents across 105 countries. According to secondary reports, it shows 88% of organisations using AI in at least one function, but only 39% attributing any EBIT impact to it. Most of those put the impact below 5%, and roughly 6% qualify as high performers. The earlier March 2025 wave had more than 80% of respondents reporting no measurable enterprise-level EBIT impact.

Then leave the commercial producers entirely. Daron Acemoglu, in "The Simple Macroeconomics of AI", Economic Policy 40(121), 2025, applies a task-based model and Hulten's theorem. The upper bound he derives is about +0.66% total factor productivity over ten years. Using a method with no survey in it, he too arrives at a small number.

A consultant, an analyst house, a university lab, a survey of 1,993 executives and an academic macroeconomist measure five different things by five different methods and point the same way. Most AI initiatives do not deliver measurable return quickly. The dispersion of the values justifies rejecting the specific figures. It does not justify rejecting the conclusion. And rejecting every number is cheaper than building your own, which is why that move deserves suspicion.

Four: "pilot did not scale" is not the same as "pilot failed", and nobody has good data on the difference. Arnon Shimoni argued in August 2025 that MIT counts as a failure every pilot that did not reach full production. He offered his own breakdown, in which only a quarter of pilots are genuine failures. The rest are learning exercises, vendor evaluations, and discoveries that a problem was not worth solving. Those are private observations with no published methodology, from someone working in the industry, and they are not data. As an argument about construct validity they still stand. A portfolio of cheap, reversible experiments should have a high termination rate by design, and any metric that scores termination as failure will punish exactly the behaviour you want.

Closing

The useful question for your next board meeting is how many of your own closed initiatives you have ever counted, and against which forecast. Until that list exists, every percentage you quote is somebody else's organisation, measured on somebody else's terms, for somebody else's purpose.

Sources

1. primaryBent Flyvbjerg, "From Nobel Prize to Project Management: Getting Risks Right," Project Management Journal 37(3), August 2006, pp. 5–15 (DOI 10.1177/875697280603700302); "Curbing Optimism Bias and Strategic Misrepresentation in Planning: Reference Class Forecasting in Practice," European Planning Studies 16(1), 2008, pp. 3–21 (DOI 10.1080/09654310701747936); and, with Alexander Budzier, "Why Your IT Project May Be Riskier Than You Think," Harvard Business Review 89(9), September 2011, pp. 23–25 — the three steps of reference class forecasting, the optimism-bias versus strategic-misrepresentation distinction, and the 1,471-project fat tail. The 16,000-project database figures (8.5% and 0.5%) come from the trade book How Big Things Get Done (2023) and are quoted as reported.
2. primaryKjetil Moløkken-Østvold & Magne Jørgensen, "How large are software cost overruns? A review of the 1994 CHAOS report," Information and Software Technology 48(4), 2006, pp. 297–301 (the 189% figure as an outlier; a sampling frame skewed toward failing projects), and J. Laurenz Eveleens & Chris Verhoef, "The Rise and Fall of the Chaos Report Figures," IEEE Software 27(1), 2010, pp. 30–36 (5,457 forecasts across 1,211 projects; the four objections). Marilyn Strathern's 1997 formulation of Goodhart's law names the third objection.
3. primaryMETR, "Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity" (10 July 2025; arXiv:2507.09089) — 16 developers, 246 tasks; forecast −24%, post-hoc estimate −20%, measured +19%; design revised February 2026. With Erik Brynjolfsson, Danielle Li & Lindsey Raymond, "Generative AI at Work," Quarterly Journal of Economics 140(2), 2025, pp. 889–942 — 5,172 support agents; +15% average, +34% for novices, near zero for the experienced.
4. primaryDaron Acemoglu, "The Simple Macroeconomics of AI," Economic Policy 40(121), 2025, pp. 13–58 (NBER WP 32487, May 2024) — task-based model plus Hulten's theorem; no more than +0.66% TFP over ten years.
5. primaryStuart Soroka, Patrick Fournier & Lilach Nir, "Cross-national evidence of a negativity bias in psychophysiological reactions to news," PNAS 116(38), 2019, pp. 18888–18892 (17 countries, six continents), and Marc Trussler & Stuart Soroka, "Consumer Demand for Cynical and Negative News Frames," International Journal of Press/Politics 19(3), 2014, pp. 360–379 (revealed preference diverges from stated preference).
6. primaryUS Census Bureau, "Large Firms With at Least 20 Employees Biggest AI Users" (BTOS, 26 May 2026) — 17–20% of firms; 37% at 250+ employees; 18% unweighted against 32% weighted by employment; question rewritten in November 2025.
7. primaryThe sources behind the headline numbers. MIT Media Lab, "NANDA — Project Overview" (the agent registry, agent-to-agent protocols and the DNS-and-HTTP analogy) plus MIT NANDA, "The GenAI Divide: State of AI in Business 2025" (the 95%, not peer reviewed); RAND, Ryseff, De Bruhl & Newberry, "The Root Causes of Failure for AI Projects" (RRA2680-1, 2024) — qualitative interviews, with "more than 80%" appearing as an estimate cited from elsewhere; S&P Global Market Intelligence, "Voice of the Enterprise: AI & Machine Learning, Use Cases 2025" — 1,006 respondents, 46% of PoCs killed, 42% of firms abandoning most initiatives.
8. primaryBCG, "AI Adoption in 2024: 74% of Companies Struggle to Achieve and Scale Value" (24 October 2024) — 1,000 executives, 59 countries; a self-assessment of capability maturity. Wharton Human-AI Research with GBK Collective, "Accountable Acceleration: Gen AI Fast-Tracks Into the Enterprise" (October 2025) — 800+ senior leaders; 74% reporting positive ROI, on the same class of evidence.
9. secondaryGartner press-release titles and figures, retrieved via search index because the newsroom returns a 403 to automated access: "Predicts Over 40% of Agentic AI Projects Will Be Canceled by End of 2027" (25 June 2025, a forecast); "45% of Organizations With High AI Maturity Keep AI Projects Operational for at Least Three Years" (30 June 2025, a survey result); "Generative AI is Now the Most Frequently Deployed AI Solution" (7 May 2024, 48% reaching production over eight months); the 2018 forecast that 85% of AI projects would deliver erroneous outcomes through 2022. Gartner FY2024/FY2025 revenue and contract value via stockanalysis.com (SEC EDGAR returns 403). NetScout v. Gartner (filed 2014, Connecticut Supreme Court 2020) rejected the pay-to-play claim.
10. secondaryFortune, "An MIT report finding 95% of AI pilots fail spooked investors…" (21 August 2025) — the 19–20 August sell-off, with Altman's bubble remarks in the same week. Futuriom, "Why We Don't Believe MIT NANDA's Weird AI Study" (August 2025) — the source for Kevin Werbach's data-or-retraction condition, quoted as reported since the page returns a 403. Arnon Shimoni, "MIT's 95% AI failure rate is wrong" (27 August 2025) — a practitioner's decomposition, an argument rather than data.
11. secondaryMcKinsey QuantumBlack, "The State of AI" (November 2025 wave; 1,993 respondents, 105 countries) — 88% using AI, 39% attributing any EBIT impact, ~6% high performers; March 2025 wave, over 80% with no measurable EBIT impact. Figures as reported through secondary outlets, since the page and PDF time out. Ramp, "AI Index" and "AI Index March 2026" — card and bill-pay data across 70,000+ businesses; 47.6% in February 2026, above 50% in March, ~35% a year earlier; no published sampling frame.
12. secondaryProvenance and mutation. Beverly D. Houghton's Woozle effect (1979), described by the author as reification by accretion, with current methodological use in Electronic Commerce Research (Springer, 2024). VentureBeat, "Why do 87% of data science projects never make it into production?" (July 2019) — sponsored content reporting a conference panel; provenance confirmed through secondary accounts as the page returns a 403. Exhibits, cited as artefacts and not as data: Folio3 AI, "AI Project Failure Rate in 2026" (the RAND sample of 2,400+ initiatives that does not exist, the S&P quantifier swap, the 2018 Gartner forecast redated to 2025) and Pertama Partners, "AI Project Failure Rate 2026: 80% Fail" (correct qualitative caveat in the body, alarmist headline above it).