Last month, KPMG pulled one of its flagship reports from circulation. The report, titled “Redefining Excellence in the Age of Agentic AI,” had been distributed by the firm’s consulting groups across multiple countries, with local marketing contacts attached. Research outfit GPTZero then conducted a forensic review of its 45 citations. Of those, only five accurately pointed to real sources.

At least 16 were hallucinations. Another 12 were too ambiguous or flawed to determine whether a source existed at all. The errors were not random. Most citations had at least one hallucinated component, typically the title. Many also had fabricated dates or authors.

The bitter detail: this was a report about the promise of AI, written with AI, undone by AI.

It was not the first. Deloitte Australia had already agreed to partially refund the Australian government for a $290,000 report containing AI-generated errors. The document, a 237-page independent assurance review of a welfare compliance system, included references to multiple academic papers that did not exist, citations attributed to a real professor at the University of Sydney for work she had never published, and a fabricated quote attributed to an actual federal court judgment. The judge’s name was even misspelled.

EY withdrew a loyalty rewards program report the same month after GPTZero identified fake footnotes and errors following the same pattern. Three of the four largest professional services firms in the world. Same failure mode. Same cause. Same tool.

Publications by firms like these poison the well of information, because their work is treated as credible and referenced by other publications, creating chains of second-hand hallucination that are almost impossible to trace. — Edward Tian, GPTZero CEO

An AI hallucination is not a mistake in the way a human makes a mistake. Understanding why requires a brief look at what these systems actually do, because the mechanism matters for everything that follows.

Every large language model, regardless of provider, operates on a single core principle: given the text that has come before, predict the most statistically probable next piece of text. The model does not retrieve facts from a database. It generates, token by token, the continuation that its training across billions of documents tells it is most likely to follow the input it has been given.

This process is extraordinarily good at producing fluent, contextually appropriate, stylistically coherent text. It is also, by design, indifferent to truth in any foundational sense. If the context suggests that a citation belongs at this point in a report, the model produces a citation. Whether that citation corresponds to something real is a separate question the architecture does not automatically ask.

This is why the Deloitte report looked exactly like a properly researched document. The paper titles were plausible. The author names were formatted correctly. The dates fell within credible ranges.

In 2024, researcher Xu et al. formalized this mathematically: eliminating hallucination in large language models is not a matter of more compute, better training data, or more careful fine-tuning. It is mathematically impossible given the fundamental architecture. Any system that generates text by predicting probable sequences from learned statistical distributions will, by mathematical necessity, sometimes produce outputs not grounded in fact.

Karpowicz, working at Samsung AI Center Warsaw in 2025, approached the same question from three independent mathematical frameworks and reached the same conclusion each time. No LLM inference mechanism can simultaneously satisfy all four of these properties:

The implication is not that AI is useless or unreliable in all contexts. It is that the architecture has a structural ceiling on truthfulness that does not move regardless of how good the model becomes at everything else.

The finding that complicates this further, and that most enterprise AI strategies have not yet absorbed, is that the models being marketed most aggressively as more capable are in several measurable ways worse on this specific dimension.

In 2025, benchmarking against the PersonQA dataset, which tests factual recall about specific individuals, produced a result that cut against the prevailing assumption that more reasoning capability means fewer errors.

A reasoning model builds longer internal chains of inference before producing an answer. That process is genuinely useful for complex, multi-step problems. But those longer chains also create more opportunities for the model to generate plausible-sounding intermediate steps that are not grounded in fact, and to arrive at wrong answers with the full structural confidence of a well-reasoned conclusion.

OpenAI’s own research paper on the subject, published in September 2025, described the incentive structure clearly: next-token prediction training objectives and common leaderboard benchmarks penalize “I don’t know” responses, which means models are implicitly trained to bluff rather than abstain. The result is models architecturally disposed toward confident wrong answers rather than calibrated uncertainty.

General-purpose hallucination rates on well-supported factual queries have fallen across leading models, and the headline numbers vendors publish reflect real progress. The problem is sharpest precisely where the stakes are highest. Select a domain below.

The vendors are not misrepresenting the progress when they report falling hallucination rates. The improvements are real. But there is a meaningful difference between reducing frequency and eliminating the underlying cause, and the interventions that actually work make that distinction visible.

The gap between those numbers tells the story. Prompt engineering, the most common intervention and the one most organizations reach for first, produces the weakest result by a factor of five to six. Telling the model to be more careful does not change the underlying generative process. The interventions that work are architectural, not behavioral.

Anthropic’s research on internal concept vectors, published in its work on tracing the thoughts of large language models, demonstrated how models like Claude can be trained to learn when not to answer, turning refusal into a learned policy rather than a fragile prompt instruction. That is a genuine architectural advance. But even that research does not claim to eliminate hallucination. It reduces confident wrong answers in cases where the model has learned to recognize its own uncertainty. Cases it has not learned to recognize remain.

Retrieval-augmented generation is not a complete solution either. Even with retrieval grounding, hallucination rates are non-zero. Models can misread retrieved documents, over-generalize from them, or fabricate claims about content that was present but ambiguous. The Karpowicz impossibility proof applies to all architectures, including hybrid ones.

The organizational lesson from KPMG and Deloitte is not that AI cannot be used in high-stakes work. It is that the verification infrastructure around AI has to be designed to match the deployment context, and in both cases it was not.

In the Deloitte case, GPT-4o via Azure OpenAI was used as part of the technical workstream for a 237-page independent assurance review of a government welfare compliance system. No organizational contingency existed for validating citation quality before publication. A single Sydney University researcher, reading the report out of professional interest, identified fabrications that had passed through Deloitte’s entire production chain undetected.

The structural requirement this creates for enterprise leaders is straightforward to describe, if not always simple to implement. AI deployment in high-stakes output contexts requires verification infrastructure proportionate to the consequence of a wrong answer. That infrastructure is not a prompt. It is a process.

A document can read coherently and be comprehensively wrong. Coherence review does not catch hallucinations. Citation verification does.

For organizations using AI in document generation, research synthesis, compliance analysis, or any output that will be cited by others, the minimum viable architecture involves grounding generation to verified source documents rather than open-ended generation, span-level citation verification that checks each generated claim against retrieved evidence, and human review structured to catch fabrications rather than overall coherence.

For procurement and vendor evaluation, the question that is currently underasked is not what this model’s hallucination rate is on general benchmarks, but what its hallucination rate is on the specific type of output being generated, in the specific domain, with the specific verification infrastructure in place. Those are four different questions and only the last one is relevant to operational risk.

The answer to AI hallucinations is not a better model. The research establishes that clearly. The answer is an organization that has designed its AI deployments around the fact that the model will sometimes be wrong, that it will not always know it is wrong, and that the confidence with which it is wrong is indistinguishable, on the surface, from the confidence with which it is right. Building for that reality is not a limitation on what AI can do. It is the condition under which AI can be trusted to do anything at all.