Last month, KPMG pulled one of its flagship reports from circulation. The report, titled “Redefining Excellence in the Age of Agentic AI,” had been distributed by the firm’s consulting groups across multiple countries, with local marketing contacts attached. Research outfit GPTZero then conducted a forensic review of its 45 citations. Of those, only five accurately pointed to real sources.

At least 16 were hallucinations. Another 12 were too ambiguous or flawed to determine whether a source existed at all. The errors were not random. Most citations had at least one hallucinated component, typically the title. Many also had fabricated dates or authors.

The bitter detail: this was a report about the promise of AI, written with AI, undone by AI.

It was not the first. Deloitte Australia had already agreed to partially refund the Australian government for a $290,000 report containing AI-generated errors. The document, a 237-page independent assurance review of a welfare compliance system, included references to multiple academic papers that did not exist, citations attributed to a real professor at the University of Sydney for work she had never published, and a fabricated quote attributed to an actual federal court judgment. The judge’s name was even misspelled.

EY withdrew a cybersecurity report on loyalty programs the same month after GPTZero identified fabricated footnotes and hallucinated references following the same pattern. By mid-2026, three of the four largest professional services firms in the world had published reports riddled with AI-generated errors, through the same mechanism, within months of each other.

GPTZero CEO Edward Tian told the Financial Times that error-riddled publications by firms of this stature “poison the well of information,” increasing what he called “the risk of second-hand hallucinations” as their findings get cited across industry publications, fed into AI models, and treated as verified fact.

An AI hallucination works differently from a human error. Understanding the distinction requires a brief look at what these systems actually do, because the mechanism matters for everything that follows.

Every large language model, regardless of provider, operates on a single core principle: given the text that has come before, predict the most statistically probable next piece of text. The model does not retrieve facts from a database. It generates, token by token, the continuation that its training across billions of documents tells it is most likely to follow the input it has been given.

This process is extraordinarily good at producing fluent, contextually appropriate, stylistically coherent text. The process is also, by design, indifferent to factual accuracy. If the context suggests that a citation belongs at this point in a report, the model produces a citation. Whether that citation corresponds to something real is a separate question the architecture does not automatically ask.

This is why the Deloitte report looked exactly like a properly researched document. The paper titles were plausible. The author names were formatted correctly. The dates fell within credible ranges.

In 2024, researcher Xu et al. formalized this mathematically, establishing that eliminating hallucination in large language models is impossible given the fundamental architecture. Any system that generates text by predicting probable sequences from learned statistical distributions will, by mathematical necessity, sometimes produce outputs ungrounded in fact. More compute, better training data, and more careful fine-tuning move the rate, but the floor is above zero.

Karpowicz, working at Samsung AI Center Warsaw in 2025, approached the same question from three independent mathematical frameworks and reached the same conclusion each time. No LLM inference mechanism can simultaneously satisfy all four of these properties:

The implication is that the architecture has a structural ceiling on truthfulness that does not move regardless of how capable the model becomes at everything else.

The finding that complicates this further, and that most enterprise AI strategies have not yet absorbed, is that the models being marketed most aggressively as more capable are in several measurable ways worse on this specific dimension.

In 2025, benchmarking against the PersonQA dataset, which tests factual recall about specific individuals, produced a result that cut against the prevailing assumption that more reasoning capability means fewer errors.

A reasoning model builds longer internal chains of inference before producing an answer. That process is genuinely useful for complex, multi-step problems. But those longer chains also create more opportunities for the model to generate plausible-sounding intermediate steps ungrounded in fact, arriving at wrong answers with the full structural confidence of a well-reasoned conclusion.

OpenAI’s own system card for o3 and o4-mini, published in April 2025, described the dynamic: the models “tend to make more claims overall, leading to more accurate claims as well as more inaccurate/hallucinated claims.” Common leaderboard benchmarks also penalize “I don’t know” responses, which means models are implicitly trained toward confident answers over calibrated uncertainty.

General-purpose hallucination rates on well-supported factual queries have fallen across leading models, and the headline numbers vendors publish reflect real progress. The problem is sharpest precisely where the stakes are highest. Select a domain below.

The vendors are not misrepresenting the progress when they report falling hallucination rates. The improvements are real. But there is a meaningful difference between reducing frequency and eliminating the underlying cause, and the interventions that actually work make that distinction visible.

The gap between those numbers tells the story. Prompt engineering, the most common intervention and the one most organizations reach for first, produces the weakest result by a factor of five to six. Telling the model to be more careful does not change the underlying generative process. The interventions that work operate at the architecture level, constraining what the model can draw on and verifying what it produces against grounded sources.

Anthropic’s research on internal concept vectors, published in its work on tracing the thoughts of large language models, demonstrated how models like Claude can be trained to learn when to decline an answer, turning refusal into a learned policy. That is a genuine architectural advance. Even so, that research does not claim to eliminate hallucination. It reduces confident wrong answers in cases where the model has learned to recognize its own uncertainty. Cases it has not learned to recognize remain.

Retrieval-augmented generation is not a complete solution either. Even with retrieval grounding, hallucination rates stay above zero. Models can misread retrieved documents, over-generalize from them, or fabricate claims about content that was present but ambiguous. The Karpowicz impossibility proof applies to all architectures, including hybrid ones.

The organizational lesson from KPMG and Deloitte is that the verification infrastructure around AI has to be designed to match the deployment context. In both cases it was not.

In the Deloitte case, GPT-4o via Azure OpenAI was used as part of the technical workstream for a 237-page independent assurance review of a government welfare compliance system. No organizational contingency existed for validating citation quality before publication. A single Sydney University researcher, reading the report out of professional interest, identified fabrications that had passed through Deloitte’s entire production chain undetected.

The structural requirement this creates for enterprise leaders is straightforward to describe, if not always simple to implement. AI deployment in high-stakes output contexts requires verification infrastructure proportionate to the consequence of a wrong answer. That infrastructure takes the form of a process, with defined ownership, not a prompt instruction added at the end of a workflow.

A document can read coherently and be comprehensively wrong. Coherence review does not catch hallucinations. Citation verification does.

For organizations using AI in document generation, research synthesis, compliance analysis, or any output that will be cited by others, the minimum viable architecture involves grounding generation to verified source documents, span-level citation verification that checks each generated claim against retrieved evidence, and human review structured to catch fabrications rather than overall coherence.

For procurement and vendor evaluation, the question that is currently underasked is what this model’s hallucination rate is on the specific type of output being generated, in the specific domain, with the specific verification infrastructure in place. The general benchmark number is a different question entirely, and the one least relevant to operational risk.

The research establishes that a better model alone does not solve this. The ceiling is structural. The organizations that manage hallucination risk most effectively are those that have designed their AI deployments around the fact that the model will sometimes be wrong, that it will not always signal when it is wrong, and that the confidence with which it is wrong is indistinguishable on the surface from the confidence with which it is right. Building for that reality is the condition under which AI can be trusted with consequential work.