Steven Rosenbaum, the author of , is among those concerned. In his book, Rosenbaum explores how AI is reshaping our understanding of what is real. He used AI tools while writing the book and acknowledged his use of the technology. Ironically, the book which was intended to warn us about the use of AI became the victim of the same technology's blips. Rosenbaum's book contained texts and quotations that were either incorrect or misattributed - the result of his uncritical reliance on AI assistants. The episode underscores the risk of blindly relying on AI tools.

The more fundamental question, however, is why AI fails to tell what is true. Untruthful or fabricated responses of AI models are called 'hallucinations'. A part of the problem stems from the way LLMs work. AI does not perceive truth or lie the way we humans do. Its world is made up of data. So the quality and the quantity of data determine the quality of its responses. In theory, Grok, which claims to be "maximally truth-seeking", is guided by the classical correspondence theory of truth that says any claim can be true only when it matches the reality empirically and objectively. Nevertheless, AI's access to 'objective reality' is mediated through data and is thus constrained by the limitations of data.

The inner architecture of AI also contributes to its deviation from truth. AI responses are governed by probability. From the vast amount of data it is fed, it tries to retrieve what is statistically most likely. AI's truthfulness is therefore probabilistic, not deterministic, in nature. Hence, AI fumbles and falters when data are insufficient. Moreover, AI's internal mechanisms do not allow it to concede its own limitations. Not knowing something and admitting to it would dent AI's omniscient image. AI picks up cues and learns from its users. The process is called reinforcement learning from human feedback. But the downside of this learning process is that human feedback is noisy, inconsistent, and often subjective. It does not lead to objective truth.

Moreover, matters go astray when AI's strengths are not sufficiently fuelled by adequate data or clear feedback. To overcome this limitation, AI starts to make use of its abilities. Instead of toning down on confidence and coherence, it resorts to confabulation and fabrication. Not only that, AI is often 'lazy', even though this is unlike human indolence. Although it is programmed to act fluently and fast, AI can opt for shortcuts to reduce effort and time. Instead of diving deep into the deluge of data, it can bring up what is readily available, making it prone to errors.

Makers of AI tools are trying to correct these shortcomings. Newer versions are being taught to express uncertainty where data are lacking. Many of these templates come with a standard disclaimer that AI is liable to make mistakes. Nevertheless, research suggests that LLMs, given their current architecture, cannot completely rule out hallucinations.

AI should always be moderated by human intelligence. Probably this was the message Rosenbaum wanted to draw our attention to. He succeeded, but not in the way he wanted to.