Silent failure is any agent run that ends without an error signal and leaves the world in the wrong state.

It is the dominant failure mode of production agents because every observability signal you own — HTTP status, tool return codes, completion flags — reports the run as healthy.

In an audit of 15 agent-facing tools on ToolUniverse, 7 out of 15 tools exhibit confirmed failures on more than 50% of test cases — ExpressionAtlas 75%, BV-BRC 67%, AMPSphere 67%, CTIS 53%, ChIP-Atlas 57%, KEGG 53%, Reactome 53%. (arXiv 2609.26836v2, Silent Failures in Agent–Tool Interaction; submitted Sep 2026, v2 Oct 2026.)

A loud failure costs you an incident; a silent one costs you a customer who never tells you. This page is the taxonomy: six ways agents fail quietly, the specific mechanism that hides each from your logs, and how to detect each one.

The six modes below are grouped by where the failure originates: at the boundary between agent and tool, in the real-world effect itself, in the record of what happened, or in the input the agent trusts. The grouping is a diagnostic aid. When a run looks healthy but the outcome is wrong, ask which of these four places the lie could be sitting in, and you have narrowed six possibilities to two.

Each mode is defined so that it can be read on its own. If you landed here from a search or a link into one section, that section does not depend on the ones above it.

Failures at the agent–tool boundary

These are failures in the handoff. The agent decided to act, the tool was supposed to act, and the two disagree about what happened.

Tool-call hallucination

Tool-call hallucination is when the model invokes a tool that does not exist, invents arguments the schema never defined, or passes values it inferred rather than received. The mechanism is that the model generates the call as free text and the runtime validates it after the fact — nothing constrains the name or the arguments at generation time.

That ordering is the entire problem. A typed interface or a constrained decoder would refuse an invalid call before it exists. The common agent loop instead lets the model write whatever string looks plausible, then hands it to a dispatcher that either finds a matching tool or does not. When it does not, most loops return an error string to the model and invite a retry. The model now has a second chance to guess, and guessing is exactly what produced the error.

Argument hallucination is quieter than name hallucination. A wrong function name produces a lookup miss the runtime can see. A plausible argument produces a call that runs. If the model was handed an order ID in context, it will copy it faithfully. If it inferred the order ID from a summary, it will invent one, and the tool has no way to distinguish a real identifier from a well-formed fake.

The tell is almost never in the tool name. It is in the arguments: values that are internally consistent but do not exist in your database, or formats that match your schema without matching your data.

Why it's silent: the failed call is retried or narrated around, and the agent's final answer describes the work it intended rather than the call it actually made.

How teams misdiagnose it: they read a clean transcript, decide the model is weak at tool use, and swap models, when the fault is an unconstrained schema or an ambiguous tool description that kept inviting a guess.

Silent tool failure

A silent tool failure is a call that returns success — or an empty, truncated, or stale payload — while the operation it represents did not complete. The mechanism is that the tool reports transport status, not semantic outcome, so 200 OK stands in for "the thing you asked for happened."

Transport status answers a narrow question: did the request reach the service and did the service accept it. It says nothing about whether the effect was applied, whether the response was complete, or whether the data you got back was the data you asked for. An empty result set is a successful call. A paginated response truncated at the first page is a successful call. A cached payload from last Tuesday is a successful call.

This is the mode with the widest gap between what the logs show and what happened. Your error rate is zero. Your latency graph is flat. Your tool-call success percentage is a round number near a hundred. And a refund that should have been issued was accepted, queued, and dropped by a downstream consumer that nobody instrumented.

The empty-payload variant catches teams most often, because an empty list is a legitimate answer. "No matching records" and "the query silently failed and returned nothing" are the same bytes.

Why it's silent: the return code reflects delivery rather than effect, and the agent reads delivery as completion.

How teams misdiagnose it: they assume an error at the tool layer would have surfaced in the transcript, and go looking for a reasoning bug in the model when the failure was downstream of the model entirely.

Failures that apply a real-world effect wrongly

Here the agent and the tools agree about what happened, and the effect applied to the world is still wrong. These failures are consistent, well-reported, and expensive.

Duplicate side effects (retries)

Duplicate side effects occur when an agent retries an operation that already succeeded, applying a non-idempotent effect twice. The mechanism is that timeouts are indistinguishable from failures, so every framework default turns an unknown outcome into a second attempt.

A timeout is not a failure. It is an absence of information. The request may have been rejected before it landed, or it may have been applied and the response lost on the way back. The client cannot tell those apart, and most retry policies resolve the ambiguity by assuming the worst and trying again. For a read, that is harmless. For a charge, a refund, an email, a shipment, or a ticket creation, it doubles the effect.

Agent frameworks make this worse by treating tool calls as deterministic functions. A tool call looks like a function call in the code, so it inherits the mental model of one: safe to retry, cheap to repeat, no external state. Underneath, most of those calls are non-idempotent HTTP requests to systems that will happily apply the same instruction twice.

The failure is a counter problem, not an error problem. There is no stack trace to find, because both attempts succeeded. The evidence lives in the ledger, in the inbox, or in the shipping manifest, and it arrives days after the run that caused it.

Replay is the check that settles it. Capture the call, restore the pre-state, re-execute, and diff the effect — replaying a tool call is the only way to know what your agent did. If the effect differs from the claimed effect, the original run lied about what it did.

Why it's silent: both attempts return success, so the duplicate exists only in downstream systems and never in the agent's own transcript.

How teams misdiagnose it: they search their logs for errors, find none, and conclude the system is healthy, when the failure was never an error in the first place.

Failures in the record

The agent did the work correctly, or did not, and the record you rely on to judge it has drifted away from reality. These failures corrupt your ability to know, which makes every other failure harder to find.

Judge drift

Judge drift is a change in the scoring behaviour of an LLM-as-a-judge that silently re-bases your quality metric. The mechanism is that judge behaviour is a function of model version, prompt, and sampling, so upgrading any of them changes the instrument while the dashboard keeps drawing the same line.

A judge is a measuring instrument built out of a model, and models are not stable instruments. Every version bump, every prompt tidy-up, every change to sampling temperature or the rubric's wording moves the score distribution. Nothing in the pipeline announces that the ruler changed. The metric keeps arriving, formatted identically, plotted on the same axis, and somebody reads a two-point drop as a product regression.

The compounding problem is that judge drift is invisible against judge drift. If the judge loosens while quality falls, the two movements cancel and the dashboard reports stability. If the judge tightens while quality improves, you get a regression report about a system that got better.

Pinning is the minimum defence: record the judge model, its version, and the prompt hash alongside every score, so a number can never be compared to a number produced by a different instrument without that fact being visible.

Why it's silent: the metric stays green and only its meaning changed, so no alert fires and no consumer of the dashboard has any reason to question it.

How teams misdiagnose it: they treat movement in the score as movement in the product, when they actually replaced the ruler mid-measurement.

Context rot (instruction drift)

Context rot is the progressive loss of instruction fidelity as a conversation grows — the agent follows its system prompt at turn 3 and quietly stops honouring it at turn 40. The mechanism is that earlier instructions compete for attention with everything accumulated since, and nothing in the runtime reasserts them.

Every turn adds tool results, retrieved documents, prior reasoning, and user corrections. The system prompt does not grow, but the material competing with it does. The instruction does not get deleted; it gets outvoted. Behaviour degrades gradually, which means there is no clean turn where you can point at the moment compliance stopped.

The degradation is domain-dependent and, in the published work, measured under simulation rather than in the field. In a simulated study of agent drift, simulated drift incidence reached 53.2% by 500 interactions in financial analysis systems, with a 42% reduction in task success rate. (arXiv 2601.04170, Agent Drift; submitted Jan 2026.) The simulated figures describe a modelled trajectory, not an observed rate in your stack.

Mitigation is real even if the magnitude is uncertain. Researchers report that reminder interventions reduced divergence by up to 30% with LLaMA 3.1 8B. (arXiv 2510.07777, Drift No More?; submitted Oct 2025.) Reasserting the instruction periodically is cheap and measurably moves behaviour back toward compliance.

Why it's silent: every individual turn looks reasonable in isolation, and the deviation only becomes visible when you compare a late turn against the original specification rather than against the turn before it.

How teams misdiagnose it: they blame prompt quality and rewrite the system prompt, when the instruction was correct all along and had simply stopped being salient.

Failures from untrusted input

The agent behaved exactly as instructed, performed every action correctly, and the instruction came from an attacker.

Prompt injection via tool output

Tool-output prompt injection is when content the agent retrieves — a web page, an email body, a ticket comment — contains instructions the model treats as its own. The mechanism is that tool results and developer instructions arrive in the same channel, so the model has no reliable way to tell data from command.

The model reads one flat sequence of text. A developer message and a scraped web page are not different in kind once they are tokens in the same context; they differ only in position and whatever framing the template added. An attacker who controls retrievable content controls a slice of that sequence, and the model has no trustworthy signal marking which slice carries authority.

What makes this the most consequential mode is that the injection executes through the agent's own legitimate authority. The agent has credentials, tool access, and the user's trust. When an injected instruction says to email the contents of a file to an address, the agent does it, and every log records an authorised action performed by an authenticated agent.

Input filtering does not close this. Filtering catches known patterns in text you can see, and the space of instructions that hijack a model is open-ended. The durable fix is structural: keep untrusted content in a position that cannot issue commands, constrain what a tool-using agent is permitted to do after touching retrieved data, and require confirmation for irreversible actions.

Why it's silent: the agent behaves as instructed, so every log shows a correctly executed, fully authorised action taken by an agent doing its job.

How teams misdiagnose it: they treat it as a content-moderation problem and add input filters, when the exploitable surface is the trust boundary between retrieved data and instructions.

How to detect silent failure

Detection is not harder observability. It is a different question: not "did the call succeed" but "did the world change the way the agent says it did."

  • Replay against restored state. Capture the call, restore pre-state, re-execute, and diff the effect. If the effect differs, the original run lied. The mechanics are covered in replaying a tool call is the only way to know what your agent did.
  • Verify the effect, not the return code. Assert on the resulting state — row counts, balances, sent-message IDs — rather than on the tool's own success flag. The flag describes delivery. Your assertion should describe consequence.
  • Score claims against traces. Every "done" the agent emits should be checkable against the trace that produced it. A completion claim with no corresponding verified effect is an unverified claim, not a completion.
  • Calibrate the judge before trusting the score. Pin the judge model, version, and prompt, and run an overlap period before swapping either one. Without an overlap window, a metric change and a product change are the same event.
  • Assert the invariant, not the outcome. Encode "refund issued exactly once" as a checkable property and test it after every retry path. Invariants survive refactors; expected outputs do not.

These five checks are the sequence described in the method behind the audit, run in order. The first two find effect-level failures, the third finds claim-level failures, the fourth protects the measurement itself, and the fifth is what turns a one-off investigation into a regression test.

If you want this done for you in two weeks with a fixed-fee audit and a no-find, no-fee guarantee, start the two-week audit intake.

How to prevent it

Prevention here is structural, not vigilant. Every mode in this taxonomy is held open by a design choice, and each choice can be closed: constrain tool schemas so an invalid call cannot be constructed, make every mutating tool idempotent with a per-step key so a retry cannot double an effect, reassert instructions at fixed intervals instead of trusting one system prompt to survive a long conversation, and treat all tool output as untrusted data rather than instruction.

None of that requires heroics on the part of whoever is on call. It requires that the constraints exist in code before the incident rather than in a postmortem afterwards.

If you would rather start from something that already has these properties, the Secure Agent Kit ships a deterministic Pytest suite that replays tool calls and diffs the effect. To see everything else available, browse all products.

Silent failure: common questions

What is silent failure in an AI agent?

A silent failure is an agent run that reports success and leaves the world in the wrong state. No exception is raised, the tool calls return success, and the transcript reads as a completed task. The only place the failure exists is the difference between what was claimed and what actually changed.

Why do AI agents fail silently?

Because agents report on their own reasoning rather than on verified effects. The signals available to them — transport status, their own narration — cannot distinguish "the effect was applied" from "the instruction was accepted." An agent that describes its plan in the past tense produces a completion message that is indistinguishable from a real one.

What does it mean when an agent reports success but the task failed?

It means the completion signal was generated from the plan rather than from a state check. The agent finished its reasoning, emitted "done," and nothing in the loop compared that claim against the resulting state. This is the signature of silent tool failure: the tool returned, the model moved on, and no one verified the effect.

How do you test AI agent reliability?

Test the effect, not the answer. Replay real tool calls against restored state and diff the result, then assert on downstream state — counts, balances, identifiers — rather than on return codes. Use calibrated judges only for the subjective residue that has no checkable ground truth, and re-calibrate them whenever the judge model or its prompt changes.

What is silent tool failure?

A silent tool failure is a tool call that returns success while the operation did not complete. The payload may be empty, truncated, or stale, or the effect may have been accepted and dropped downstream. It is invisible to error-rate dashboards because no error occurred anywhere in the call.

How do you detect silent failures in production?

Replay calls against restored state and diff the effect, verify outcomes against downstream state instead of return codes, score the agent's claims against the traces that produced them, and pin and calibrate the judge before trusting any quality score. Run the invariant check after every retry path, not only in pre-production. Detection works because it asks what changed in the world rather than what the agent said about it.