AI chatbots are quickly replacing traditional search engines for millions of users, but the answers they deliver are often wrong, incomplete, or entirely fabricated. Research from multiple institutions shows that even the most popular models produce unreliable citations, fall for hidden manipulation, and confidently present false information as fact. For anyone relying on these tools to answer everyday questions, the message from researchers is blunt, verify everything.
When Chatbots Confidently Get It Wrong
AI systems do not flag their own errors the way a broken link or a “page not found” message might. Instead, they present fabricated details with the same polished tone they use for accurate ones. A guide from the University of Maryland’s library system lays out the core problem: AI can give the wrong answer, omit information by mistake, and make up completely fake people, events, and articles. That last failure mode, often called “hallucination,” is especially dangerous because it leaves no obvious trail for a casual user to detect. A response that sounds fluent and authoritative can be entirely invented, and without a habit of checking, most people will never notice.
The scale of the problem is not small. Benchmarking research highlighted by Stanford’s Institute for Human-Centered Artificial Intelligence found that legal AI models hallucinate in 1 out of 6 or more queries. That rate was measured in retrieval-augmented generation settings, where models are specifically designed to pull from source documents before answering. If systems built to ground their responses in real text still fail at that frequency, general-purpose chatbots operating without such guardrails face even steeper odds.
For everyday users, the effect is subtle but pervasive. A chatbot might give a mostly accurate explanation of a medical condition but slip in an incorrect dosage, or summarize a news story while misidentifying the key actors. Because the wrong detail is wrapped in correct context and confident language, it can be harder to spot than an obviously broken web page. Over time, repeated exposure to such polished but unreliable answers can erode a person’s sense of what counts as a trustworthy source.
Citations That Lead Nowhere
One common defense of AI search tools is that many now include links to their sources, giving users a way to check the underlying material. That feature sounds reassuring, but the citations themselves are often unreliable. A verifiability audit framework developed for generative search engines and published on arXiv found that many generated sentences are not fully supported by the citations attached to them, and a non-trivial fraction of those citations do not actually support their associated statements. In plain terms, the footnotes look real but do not back up what the chatbot claimed.
The problem is not just that links may be weakly related. Sometimes the cited material contradicts the chatbot’s summary, or covers only a small portion of what is asserted. Users who glance at the presence of a link but never click it can walk away with a false sense of security. The visual cue of a citation mimics academic or journalistic standards without reliably meeting them.
A separate empirical study assessed the performance of eight AI chatbots specifically on bibliographic reference retrieval. The results, also published on arXiv, showed that while Grok and DeepSeek outperformed ChatGPT in that task, none of the eight were fully accurate. A large fraction of the references they produced were partially correct, erroneous, or outright fabricated. For students, journalists, lawyers, or anyone else who needs to trace a claim back to a real document, that failure rate turns a time-saving tool into a liability.
Library guides from institutions like Fairmont State University and East Los Angeles College have responded by urging users to follow every provided link and confirm it leads to a real, relevant source. As Fairmont State’s AI literacy guide puts it, the first step is checking for citations, then actually clicking through them. Some AI tools include references or links to original sources, which makes fact-checking easier in theory, but as ELAC’s research guide warns, those references sometimes point to sources that do not actually exist.
That means users cannot treat a bibliography-style answer as evidence that the underlying content is solid. Instead, they have to treat every reference as a claim to be tested: Does this article exist? Does it say what the chatbot implies? Is it current and credible? The more serious the decision (whether about health, finances, or legal rights), the more important it becomes to answer those questions directly rather than trusting the chatbot’s formatting.
Manipulation Beyond Random Errors
Hallucination is not the only risk. The errors AI search tools produce are not always accidental; they can be deliberately induced. Testing of ChatGPT’s search function revealed that the tool is vulnerable to manipulation and deception through techniques like prompt injection, where hidden instructions embedded in web pages can steer the chatbot’s output. A bad actor could, for example, plant invisible text on a product review page that instructs the chatbot to give a glowing summary regardless of what the visible reviews actually say.
This kind of attack differs from a simple factual mistake. It means the tool can be weaponized by anyone who understands how it reads web content. Instead of merely misremembering a statistic, the system can be pushed to promote a particular brand, viewpoint, or conspiracy theory while suppressing competing information. Users who assume they are getting a neutral synthesis of the web may instead be seeing the results of someone else’s hidden instructions.
OpenAI itself includes an on-page disclaimer noting that ChatGPT can make mistakes, but a small-print warning does little to protect users who treat chatbot answers with the same trust they once gave a curated search-results page. The gap between how reliable these tools appear and how reliable they actually are is where real harm happens. People may base purchases, votes, or medical choices on text that has been quietly steered by invisible prompts or adversarial content.
High-Profile Failures in Public View
The problem is visible enough that it has already embarrassed major companies in front of massive audiences. Google was forced to edit its Super Bowl ad for an AI product after the advertisement featured false information. The company remade the spot after the error was identified, but the original version had already aired during one of the most-watched broadcasts of the year. If a tech giant’s own marketing team cannot catch AI-generated inaccuracies before a Super Bowl spot goes live, expecting individual users to do so on every query is unrealistic without deliberate habits.
That incident also exposes a tension in how these products are sold versus how they perform. Companies market AI search as faster and more convenient than traditional web browsing, promising instant answers instead of a list of links. But the very speed and fluency that make chatbots attractive also make their errors more dangerous. When a search engine shows ten blue links, users expect to compare and evaluate them; when a chatbot presents one smooth paragraph, it feels like an answer rather than a starting point.
Public failures serve as a warning that even well-resourced organizations can be lulled into overtrusting AI output. If marketing teams and product managers can miss glaring inaccuracies in a high-stakes ad, individual users working alone at their laptops are even more likely to accept a plausible-sounding paragraph at face value. The lesson is not that AI tools are useless, but that they must be treated as unverified drafts, not as final authority.
Building Better Habits for AI Search
None of this means people should abandon AI tools entirely. Used carefully, they can still help summarize long documents, generate ideas, or point to starting places for further research. But the burden is on users to build habits that compensate for the technology’s flaws. That starts with assuming that any specific fact, quote, or reference might be wrong, no matter how confidently it is presented.
Practical steps are straightforward but require discipline. Click through citations and skim the original source before repeating a claim. Cross-check important facts with at least one independent outlet, especially for health, legal, or financial topics. Watch for signs of manipulation, such as oddly one-sided praise for a product or source, and be wary when an answer discourages you from looking elsewhere. When something matters, treat the chatbot’s response as a lead to investigate, not a verdict to accept.
AI chatbots are reshaping how people find information, but the research record is clear: they hallucinate, mis-cite, and can be manipulated in ways that traditional search engines are not. Until the underlying technology and safeguards improve, the safest stance is skepticism. Verification is no longer an optional extra step; it is the price of using AI search at all.
More from Morning Overview
*This article was researched with the help of AI, with human editors creating the final content.
