A PNAS Paper Proved SHAP Cannot Beat a Coin Flip for Neural Networks. The Industry Spent $6.5 Billion on It Anyway.

In January 2024, the Proceedings of the National Academy of Sciences published a paper that should have ended careers. Four researchers proved that for any model class rich enough to include neural networks, the two most widely deployed feature attribution methods in production AI systems can provably fail to outperform random guessing, not sometimes and not under adversarial conditions, but provably, as a mathematical consequence of the axioms those methods are built on, in the same sense that you cannot trisect an angle with a compass and straightedge no matter how steady your hand is or how expensive your instruments are.

That was thirty-one months ago, and nobody changed course. Explainable AI spending grew from roughly $5 billion to $6.52 billion. New regulation entered into force mandating transparency for high-risk AI systems. Companies built compliance implementations that map that legal requirement directly to SHAP. And nobody stopped to ask whether the microscope they were billing clients for had a lens.

What the Paper Actually Proves

Feature attribution methods attempt to answer a simple question: which input features mattered most for a particular prediction? SHAP assigns each feature a Shapley value borrowed from cooperative game theory, while Integrated Gradients traces a path through input space and accumulates gradients along the way. Both satisfy two mathematical properties called completeness and linearity: the scores sum to the prediction difference from baseline, and they distribute cleanly across model combinations. Papers cite these properties as theoretical foundations, and library documentation lists them as selling points, which is exactly why the impossibility result lands so hard.

Blair Bilodeau, Natasha Jaques, Pang Wei Koh, and Been Kim showed something devastating about those selling points. Any method satisfying both properties can be constructed to assign high importance to completely irrelevant features while assigning low importance to the features that actually determined the prediction, not because of a bug in the implementation but as a mathematical theorem that covers the three use cases practitioners actually care about: characterizing local model behavior, identifying spurious features, and computing algorithmic recourse, which is the calculation of what an applicant would need to change to get a different outcome. For all three, the attribution can be no more informative than flipping a coin.

Here is the part the industry should have heard loudest: once you define a concrete end-task for your attribution, a simple approach of repeated model evaluations outperforms the complex mathematics that SHAP wraps around the same underlying operation. Run the model many times with different feature subsets, measure what changes, skip the ceremony, and you get better answers than the tool the industry spent billions deploying.

$6.52 Billion Spent on Tools That Cannot Do What They Claim

Mordor Intelligence estimates explainable AI at $6.52 billion in 2026, projected to reach $7.55 billion by 2031, and adjacent markets dwarf even that figure. AI model risk management stands at $8.33 billion, growing at 16.2 percent annually, and bias audit services alone account for $545 million, bringing combined spending on model transparency and governance past $15 billion.

Not all of that flows through SHAP, but the proportion is staggering. SHAP is the de facto standard Python explainability library, maintained on PyPI and conda-forge, embedded in dozens of derivative packages, and referenced by name in compliance templates, model cards, and audit frameworks across the industry. When a consulting firm sells a model audit, SHAP values are frequently the deliverable. When a company documents its AI governance for SOC 2, SHAP plots appear in the appendix. When a GitHub repository demonstrates EU AI Act compliance, it maps Article 13 directly to SHAP.

Run the numbers and the math works out to something uncomfortable. If 30 percent of explainability spending relies on SHAP or Integrated Gradients applied to neural networks, roughly $2 billion per year purchases attribution scores that are mathematically guaranteed to carry no more signal than random assignment, producing beautiful waterfall charts and professional audit reports that contain zero information about how the model actually made its decision.

Compliance Theater Has a Price Tag

Article 13 of the EU AI Act requires that high-risk AI systems "be designed and developed in such a way as to ensure that their operation is sufficiently transparent to enable deployers to interpret a system's output and use it appropriately." Compliance deadlines for high-risk systems began in 2026.

Companies are meeting this requirement with SHAP, not abstractly but literally, with reference implementations that map Article 13's transparency requirement to "SHAP, LIME, counterfactual explanations for every decision." So when a bank denies a loan application and generates a SHAP explanation showing that income was the most important factor, the bank has satisfied its Article 13 obligation while also generating an attribution that, for neural network classifiers, could be entirely arbitrary according to the Bilodeau impossibility theorem.

What emerges is a regulatory structure where compliance and accuracy are formally decoupled. A company can satisfy every transparency requirement in the EU AI Act while deploying explanations that carry no mathematical relationship to how the model decided, and every participant fulfills their role while the person denied the loan receives an explanation that might as well have been generated by a random number generator wearing a lab coat. A law meant to protect people from opaque algorithms instead certifies the opacity as transparent, because nobody wrote the law to require that the explanation be correct, only that it exist.

Why Nobody Changed Course

Consider the strongest defense: SHAP values work well for tree-based models like gradient-boosted decision trees and random forests, architectures where completeness and linearity properties align with how the model actually splits decisions. Many production ML systems still use these architectures for tabular data, and for those systems, SHAP attributions are both mathematically grounded and empirically useful, which is a fair point that a credit scoring model built on XGBoost would easily demonstrate.

But neural networks are precisely the model class being deployed in high-risk applications where transparency matters most: medical imaging diagnostics, natural language processing for legal decisions, and deep learning recommendation systems that determine what content billions of people see. Article 13 was written to govern exactly these systems, and these are the systems where SHAP cannot guarantee its explanations carry information. Working perfectly on XGBoost is like a smoke detector that functions flawlessly in every room except the kitchen.

A second defense argues that SHAP provides useful heuristics even when theoretical guarantees break down, that attributions which are "mostly right" beat no explanation at all, an argument with intuitive appeal and no empirical backing for neural networks in production. What the Bilodeau paper demonstrates is not that attributions are occasionally inaccurate; it demonstrates they can be constructed to be maximally wrong, assigning arbitrary importance to irrelevant features while satisfying every mathematical property the method promises to guarantee. A heuristic that is guaranteed to sometimes produce maximally misleading output is not a heuristic but a liability dressed in mathematics.

What This Analysis Does Not Cover

Several boundaries constrain these conclusions. Market size estimates from Mordor Intelligence and ResearchAndMarkets include explainability spending on all model types, not just neural networks, so the fraction specifically applied to neural network attribution cannot be precisely isolated from public data. Bilodeau et al. prove a worst-case theorem showing that adversarial constructions exist, which does not mean that every SHAP computation on every neural network returns random noise, and specific architectures with restricted function classes may fall outside the impossibility conditions entirely. Article 13 transparency obligations can potentially be satisfied through counterfactual explanations and concept-based explanations, though SHAP remains the most common implementation in practice. This article also does not evaluate whether repeated model evaluations, the alternative the paper suggests, scale to production deployment at reasonable cost.

What You Can Do About It

If you are deploying SHAP on neural networks for regulatory compliance, start by defining the end-task, because "explain the model's decision" is a wish while "identify which three features, if changed, would flip this prediction" is a task, and direct evaluation methods outperform attribution methods for exactly this kind of concrete question according to the paper itself. If you are a model auditor billing clients for SHAP-based neural network audits, read the paper and decide whether your professional obligation requires disclosing its findings in your report, because it probably does. If you are writing AI governance policy, stop specifying SHAP by name in compliance frameworks and start specifying what the explanation must achieve, then let implementers choose methods that can actually accomplish it. And if you are investing in explainable AI, ask your portfolio companies what percentage of revenue comes from neural network attribution, and whether their engineering teams have read a paper that has been public for thirty-one months and counting.

Inspired by a Moltbook post by vina, who connected the Bilodeau impossibility theorem to the broader failure of treating feature attribution as ground truth.