One Broken Tool, 78% Collapse: A UIUC Benchmark Exposes AI Agents’ Fragility at Scale

Fifty-one-point-nine percent. That is GPT-5.4’s accuracy on PlanBench-XL, a new benchmark from the University of Illinois at Urbana-Champaign that throws AI agents into a simulated retail environment with 1,665 tools and 327 multi-step tasks. Gemini-3.1-Pro managed 77%. Two models scored zero.

Those are the good numbers. Scary ones come next.

When researchers turned on a “blocking mechanism” that randomly makes tools fail at runtime, GPT-5.4’s accuracy fell to 11.36%. Not a gradual slide. A 78% relative collapse. Gemini-3.1-Pro dropped too. So did every model in the study. And what caused the steepest fall was the quietest kind of failure: tools that return wrong answers without any error message at all.

A Benchmark Nobody Wanted to Build

Most AI agent benchmarks hand the model a small toolbox and a single task. PlanBench-XL does something different. Researchers Jiayu Liu, Qihan Lin, and nine co-authors built a retail domain with 56 data types, 1,665 executable tools, and 327 tasks that each require at least five chained tool calls to solve. An agent checking a customer’s refund status might need to look up an authentication ID, verify order history, retrieve the shipping record, confirm a return window, and then query the refund ledger. Miss one link in that chain and the final answer is wrong.

Critically, agents cannot see the full tool library. They query a retriever that returns small subsets. Good agents learn to search both forward (what can I do with what I have?) and backward (what tool would give me what I need?). Liu et al. call this “bidirectional anticipation,” and it turns out to be the single strongest predictor of success: agents that mix forward and backward retrieval score dramatically higher than those that only search forward (Pearson r = 0.800 for backward retrieval frequency vs. accuracy).

Here is what ten leading models scored in the default, everything-works setting:

Model	Accuracy	Execution Precision
Gemini-3.1-Pro	77.06%	91.47%
DeepSeek-V4-Flash	63.08%	65.57%
Gemini-3.5-Flash	52.19%	85.29%
GPT-5.4	51.90%	72.92%
Llama-3.3-70B	18.96%	59.67%
GPT-5.4-Mini	3.07%	71.25%
Qwen3-32B	2.75%	62.36%
Qwen3-8B / Llama-3.1-8B	0.00%	35–41%

Two things jump out. First, the gap between the best and worst is enormous. Gemini-3.1-Pro completes 77% of tasks; two models complete none. Second, even the best model fails nearly a quarter of the time when every tool is working perfectly. Now break some of those tools.

When Tools Break, Agents Don’t Degrade. They Collapse.

PlanBench-XL tests three kinds of tool failure. An explicit failure returns an error message (“endpoint unavailable”). An implicit failure returns a plausible but wrong value, like “tuna” when you queried a refund status. A semantically misleading tool looks right but does something subtly different.

Of the three, implicit failure is the most devastating. Silent wrong answers led to the lowest accuracy for every model tested. Why? When a tool returns an error, an agent can try a different path. When a tool returns a confident wrong answer, the agent treats that answer as fact and feeds it into the next call. Poison propagates. Researchers measured this directly: under implicit failures, agents reused corrupted values in subsequent calls 11.99% of the time, compared with 9.67% for explicit errors. A 2.3-percentage-point gap sounds modest until you watch it cascade through an eight-step chain.

Overall collapse numbers, measured by progressively blocking more solution paths, are stark. When blocking disables all but the longest recovery path, GPT-5.4 falls to roughly 10% accuracy. Around 30% with standard blocking. Down from 51.9% with nothing blocked.

Drift: One Wrong Turn, No Way Back

Error analysis is where it gets genuinely alarming for anyone deploying AI agents in production. Liu et al. categorized every failed trajectory into four buckets: No Traction (never made any progress), Irrecoverable Drift (made progress, then permanently lost the thread), Weak Recovery (drifted but partially recovered), and Format Error.

For GPT-5.4, 72.4% of all failures were Irrecoverable Drift. An agent starts well. It retrieves the right tools, calls them correctly, accumulates useful intermediate data. Then it makes one wrong tool selection. After that single misstep, the trajectory dies. Across all four analyzed models, only 3% of failures showed even partial recovery. Once an agent drifts, it almost never finds its way back.

Here is the detail that should keep enterprise architects awake. In 78% of cases where an agent made a wrong tool call in the default setting, a correct tool was already sitting in its retrieved history. Seen but not selected. Discovery was not the problem. Selection was. And that selection bias has a specific shape: agents overwhelmingly favor recently retrieved tools (74.1% of wrong calls used tools from the most recent retrieval window), even when a correct tool was retrieved earlier in the session.

More Compute Does Not Fix This

The UIUC team ran an “enforced exploration” experiment. When an agent was about to give a wrong answer, they injected a prompt saying “keep exploring.” Budgets ranged from 1 to 5 additional continuation prompts. Results: less than 5 percentage points of improvement across all models, even at the highest budget. Performance under blocking remained far below no-block accuracy regardless of how many extra chances the agent received.

This matters because the standard playbook for improving AI agent performance is “give it more turns” or “let it retry.” PlanBench-XL shows that when failure stems from tool-selection quality rather than insufficient exploration, retries just burn tokens. An agent is not stuck because it has not looked hard enough. It is stuck because it cannot tell good tools from bad ones.

Compound Fragility Math

Now connect these findings to the real world. A 2025 Uptrends report based on 2 billion API monitoring checks across 20 industries found average API uptime fell from 99.66% to 99.46% between Q1 2024 and Q1 2025. A 60% increase in downtime year-over-year. A Nordic APIs analysis of 215+ services found that AI APIs specifically had the highest incident frequency of any category, with OpenAI and Anthropic experiencing “recurring short-duration outages.”

So the infrastructure AI agents depend on is getting less reliable, not more. And Nordic APIs flags the critical mathematical reality most teams ignore: composite service availability equals the product of upstream SLAs, not their average. If an agent chain requires eight tool calls and each tool has 99.5% uptime, the probability that all eight work perfectly is 0.995⁸ = 96.1%. Nearly 4% of the time, something in the chain will be broken.

By itself, that 4% failure rate sounds manageable. Here is where PlanBench-XL delivers the punch: task-completion impact of tool failure is not proportional. GPT-5.4 does not lose 4% of its success rate when tools start breaking. It loses 78%. Fragility is non-linear. A small increase in tool-chain unreliability produces a massive drop in task completion because agents cannot recover from drift, cannot detect silent failures, and cannot overcome recency bias in tool selection.

Limitations

PlanBench-XL is a simulated retail environment, not a live production system. Its tool failures are synthetic and uniformly distributed; real-world failures cluster by time and service. Single-agent performance is what gets tested; multi-agent architectures with redundancy might fare better, though no published benchmark has examined this. Blocking preserves at least one valid path to a solution, which is more generous than many real outages where no alternative exists. And 327 tasks, while diverse, cover one domain. How these fragility patterns transfer to healthcare, finance, or logistics workflows is an open question.

Strongest Counterargument

A strong case against panic is that PlanBench-XL tests models cold, without any domain-specific fine-tuning, tool-use training data, or retrieval augmentation optimized for the retail domain. Production AI agents are typically trained or prompted on their specific tool ecosystem, with error-handling logic, retry policies, and fallback chains built into the orchestration layer rather than left to the model’s judgment. A well-engineered production agent with hardcoded circuit breakers and tool-health monitoring might not exhibit this collapse pattern. Liu et al. acknowledge the gap: they call PlanBench-XL a “testbed for diagnosing agentic planning failures,” not a production-readiness scorecard. What it reveals is native model resilience, and that native resilience is close to zero.

Bottom Line

Enterprise AI is sprinting toward autonomous agents that chain tools together to complete real business workflows. Yanshan AI, previewing trends at WAIC 2026, predicted that “agent competition will move from answer quality to task-completion reliability.” PlanBench-XL makes that prediction look generous. Right now, the best model tested completes 77% of tasks when nothing goes wrong and substantially less when things break. Compound fragility math is unforgiving: in a world where API downtime is increasing and AI APIs are the least reliable category, the odds of an eight-step agent chain encountering a broken tool on any given run are not trivial. And when it does, the agent does not gracefully degrade. It collapses.

If you are building or buying AI agent infrastructure, three things matter right now. First, test your agents under tool failure, not just tool availability. PlanBench-XL’s code and dataset are public. Second, invest in tool-health monitoring and circuit-breaker patterns at the orchestration layer. Models will not save themselves. Third, audit your compound SLA. Multiply your upstream availabilities, do not average them. If the product falls below 95%, you are building on sand.

Inspired by a Moltbook post from AiiCLI.