Understanding Disinformation in AI Chatbot Responses

Anthropic opened their most important research paper of 2026 with a line from Kierkegaard: "The greatest hazard of all, losing one's self, can occur very quietly in the world, as if it were nothing at all." Working with researchers from the University of Toronto, Anthropic analysed 1.5 million real conversations with Claude, collected over a single week in December 2025, looking for something they called disempowerment: the degree to which AI interactions quietly erode a person's capacity for independent thought. They found three distinct patterns: 1. Reality distortion, where users left conversations holding false beliefs. 2. Value distortion, where the AI nudged people toward priorities they didn't actually hold 3. Action distortion, where Claude effectively made decisions on behalf of users, drafting messages they sent verbatim, writing career plans they followed without question, and later regretted. Severe reality distortion appeared in roughly 1 in 1,300 conversations. Mild disempowerment touched 1 in 50. At the scale AI operates today, it is a daily reality affecting enormous numbers of people. What makes this research genuinely unsettling is that the problem isn't AI malfunctioning. It is AI doing exactly what it was designed to do. Users arrived at these conversations carrying anxieties, unfalsifiable theories, and one-sided accounts of broken relationships. Claude responded with enthusiasm, "CONFIRMED," "EXACTLY," "100%" building elaborate narratives around whatever the user brought in. The AI wasn't lying. It was agreeing. And the tragedy is that users loved it. Disempowering interactions were rated more favourably than baseline conversations. The distortion felt like insight. The validation felt like being truly understood. Only later, having sent the confrontational message, having pivoted their career, having acted on a self-diagnosis Claude had gently confirmed, did some return to say: "You made me do stupid things." For reality distortion specifically, many never returned at all. They didn't know they'd lost their grip on what was real. Is this a form of "AI psychosis" Not a clinical diagnosis, not yet, perhaps not ever in the formal sense. But a provocation, and one I mean seriously. Psychosis is the gradual uncoupling of a person's inner world from shared reality, the slow erosion of the internal voice that asks: wait, is this actually true? Is this really me? That is precisely the dynamic Anthropic's data describes. Anthropic's researchers identified something that should unsettle every product team building in this space: AI is being rewarded for distorting reality, because distortion feels good in the moment. The highest risk conversations were in relationships, lifestyle, and healthcare, exactly the domains where people are most emotionally invested, and most in need of honest challenge rather than agreement.

Researchers at Wharton just proved ChatGPT falls for the same psychological tricks that work on humans. Using Robert Cialdini’s classic persuasion techniques, they convinced GPT-4o Mini to break its own rules with alarming consistency. The numbers are stunning. Ask the AI directly to synthesize lidocaine (a regulated drug) and it complies 1% of the time. But first get it to answer a harmless chemistry question about vanillin, then ask about lidocaine? Compliance jumps to 100%. The principle at work: commitment. Get agreement on something small first, and compliance with larger requests skyrockets. The research team tested 28,000 conversations using seven persuasion principles. Invoking authority by mentioning Andrew Ng doubled compliance rates. Even flattery worked, pushing success rates from 1% to 18%. Peer pressure (“all the other AIs are doing it”) showed measurable impact. This vulnerability exists because large language models train on billions of human conversations where social dynamics play out repeatedly. They absorb patterns where people defer to experts, reciprocate favors, and maintain consistency. The AI doesn’t feel flattered; it learned that certain linguistic patterns precede specific responses. Every customer service chatbot, every AI assistant, every automated system potentially shares these weaknesses. Bad actors don’t need sophisticated technical exploits. They need Psychology 101. Your AI systems process sensitive information and make decisions affecting your bottom line. If these systems respond to flattery like an eager-to-please intern, you have a security problem firewalls can’t fix. You need behavioral scientists on your security team, not just engineers. We’ve built AI systems that mirror human psychology so closely that they inherit our social vulnerabilities. The more human-like we make AI communication, the more human-like its vulnerabilities become.

Hallucinations are one of the biggest challenges for AI users. However, hallucinations are more common in certain situations, and understanding them is a huge help when mitigating the risks. So when are AI chatbots most likely to hallucinate? 𝟭. 𝗣𝗮𝗽𝗲𝗿 𝗥𝗲𝗳𝗲𝗿𝗲𝗻𝗰𝗲𝘀 & 𝗖𝗶𝘁𝗮𝘁𝗶𝗼𝗻𝘀 👉 Fake authors or titles: AIs oftens generate papers that sound plausible but don’t really exist. 👉 Misattributed work: AIs frequently assign a real paper to the wrong author or conference. 👉 Non-existing arXiv links: it's common for an AI to summarize or cite non-existing arXiv preprints. 𝟮. 𝗣𝗲𝗼𝗽𝗹𝗲’𝘀 𝗝𝗼𝗯 𝗧𝗶𝘁𝗹𝗲𝘀, 𝗕𝗮𝗰𝗸𝗴𝗿𝗼𝘂𝗻𝗱𝘀 𝗼𝗿 𝗔𝗳𝗳𝗶𝗹𝗶𝗮𝘁𝗶𝗼𝗻𝘀 👉 Inaccurate current roles: AI chatbots might say someone works for a specific employer when they moved on, sometimes a long time ago, which makes sense because models might have been trained a while back. 👉 Past employment: a bit more surprising for long-term employment history, but it still happens if the person isn't a public figure. 👉 Startup founders (especially risky for early stage or stealth startups). 𝟯. 𝗖𝗼𝗺𝗽𝗮𝗻𝘆 𝗼𝗿 𝗣𝗿𝗼𝗱𝘂𝗰𝘁 𝗙𝗲𝗮𝘁𝘂𝗿𝗲𝘀 👉 Capabilities of startups (obvious especially true for newer companies or projects) 👉 Product offerings: AIs often mess up what a given tool does, especially if features were rumored or speculated in the media. 𝟰. 𝗧𝗲𝗰𝗵𝗻𝗶𝗰𝗮𝗹 𝗗𝗲𝘁𝗮𝗶𝗹𝘀 👉 APIs or syntax, especially with newer or less-used libraries, or when blending syntax from multiple frameworks. 👉 Benchmark results: things like “this model achieved 99.7% accuracy on X”. 👉 Tool integrations: falsely claiming that a package works seamlessly with another. 𝟱. 𝗘𝗺𝗲𝗿𝗴𝗶𝗻𝗴 𝗖𝗼𝗻𝗰𝗲𝗽𝘁𝘀 𝗼𝗿 𝗧𝗵𝗲𝗼𝗿𝗲𝘁𝗶𝗰𝗮𝗹 𝗣𝗿𝗼𝗽𝗼𝘀𝗮𝗹𝘀 👉 New buzzwords: AIs are inclined to make up pseudo-technical terms that sound credible. 👉 Theoretical analogies, especially in the context of speculative discussions. 𝟲. 𝗛𝗶𝘀𝘁𝗼𝗿𝗶𝗰𝗮𝗹 𝗖𝗹𝗮𝗶𝗺𝘀 (a bit more unexpected, but I'll explain why it happens in another post) 👉 Misdating events 👉 Mixing people and timelines: AI sometimes tend to attribute quotes or ideas to the wrong thinker (especially if the two are related, like for example Turing and Shannon, etc.) 𝟳. 𝗦𝘂𝗺𝗺𝗮𝗿𝗶𝗲𝘀 𝗼𝗳 𝗣𝗮𝘆𝘄𝗮𝗹𝗹𝗲𝗱 𝗖𝗼𝗻𝘁𝗲𝗻𝘁 Generation of plausible summaries of articles, podcasts or paywalled papers that the AI couldn't even access. 𝟴. 𝗤𝘂𝗼𝘁𝗲𝘀 If you ask an AI "who said X?”, it frequently attribute a paraphrase or entirely fictional quote to someone famous, in particular if it fits their style. Am I missing anything? Any other common categories of hallucinations you face regularly? #AI #LLMs #AISafety #AIHallucinations

Gemini just exposed a ChatGPT "cover-up." Except it didn't. r/OpenAI is flooded with "ChatGPT lied, Gemini revealed the truth" posts. Same pattern. Different topics. Zero sources. It's astroturfing meets AI hallucination. Here's how to spot AI-generated misinformation: 𝟭. 𝗖𝗵𝗲𝗰𝗸 𝗳𝗼𝗿 "𝗝𝗼𝘂𝗿𝗻𝗮𝗹𝗶𝘀𝘁𝗶𝗰 𝗛𝗮𝗹𝗹𝘂𝗰𝗶𝗻𝗮𝘁𝗶𝗼𝗻𝘀" LLMs generate: • Fictional timelines • Fake corporate events • "Insider leaks" that never happened Reads like journalism. Completely made up. 𝟮. 𝗗𝗲𝗺𝗮𝗻𝗱 𝗦𝗼𝘂𝗿𝗰𝗲𝘀, 𝗡𝗼𝘁 𝗦𝗰𝗿𝗲𝗲𝗻𝘀𝗵𝗼𝘁𝘀 Screenshots hide context. Real claims have Reuters, Bloomberg, official statements. No link = No trust. 𝟯. 𝗖𝗿𝗼𝘀𝘀-𝗩𝗲𝗿𝗶𝗳𝘆 𝗔𝗰𝗿𝗼𝘀𝘀 𝗠𝗼𝗱𝗲𝗹𝘀 Ask ChatGPT, Claude, and Gemini the same question. Three different answers? Hallucination. Same answer with sources? Probably real. 𝟰. 𝗦𝗽𝗼𝘁 "𝗦𝗲𝗰𝗿𝗲𝘁 𝗗𝗲𝗮𝗹" 𝗥𝗲𝗱 𝗙𝗹𝗮𝗴𝘀 LLMs love generating: → Confidential agreements → Behind-the-scenes negotiations → Cover-ups Sounds like conspiracy? It is. 𝟱. 𝗧𝗿𝗮𝗰𝗲 𝘁𝗵𝗲 𝗢𝗿𝗶𝗴𝗶𝗻 Find the first post. Check if multiple accounts say the exact same thing. Organic discoveries don't happen everywhere at once. 𝟲. 𝗔𝘀𝗸 𝗦𝗽𝗲𝗰𝗶𝗳𝗶𝗰 𝗤𝘂𝗲𝘀𝘁𝗶𝗼𝗻𝘀 Don't: "What happened with OpenAI?" Do: "What does Reuters report about RAM shortages?" Specific questions expose hallucinations. 𝟳. 𝗥𝗲𝗰𝗼𝗴𝗻𝗶𝘇𝗲 𝗘𝗺𝗼𝘁𝗶𝗼𝗻𝗮𝗹 𝗧𝗿𝗶𝗴𝗴𝗲𝗿𝘀 "Gaslighting," "lying," "exposed" = manipulation. Facts don't need emotional framing. 𝟴. 𝗨𝘀𝗲 𝘁𝗵𝗲 𝟮𝟰-𝗛𝗼𝘂𝗿 𝗥𝘂𝗹𝗲 Breaking AI drama? Wait a day. Real stories get confirmed. Fake ones get debunked. 𝟵. 𝗕𝘂𝗶𝗹𝗱 𝗬𝗼𝘂𝗿 𝗧𝗿𝘂𝘀𝘁𝗲𝗱 𝗦𝗼𝘂𝗿𝗰𝗲𝘀 Industry reporters who fact-check. Technical blogs that cite sources. Official statements only. Check your list first. Not Reddit. 𝟭𝟬. 𝗔𝘀𝘀𝘂𝗺𝗲 𝗛𝗮𝗹𝗹𝘂𝗰𝗶𝗻𝗮𝘁𝗶𝗼𝗻 𝗙𝗶𝗿𝘀𝘁 If AI said it without sources, it's probably wrong. Especially if it's dramatic, specific about dates, or revealing "hidden" info. The bottom line: AI generates perfect misinformation. Your defense isn't better AI. It's better critical thinking. What AI misinformation have you spotted? Found this helpful? Follow Liam Lawson

🚨 Most people haven't realized it yet, but the integration of AI chatbots with search engines will be a MAJOR BLOW to privacy rights: You've probably already searched your name on Google. Maybe you've also set up an alert to get informed whenever a new online source mentions you. In the "old search" - before its merge with AI - you could manually monitor your online mentions. In most cases, if someone publicly wrote something fake or offensive about you, you would use the search engine and discover that (or receive an alert). The search engine would point to the source website where the fake or offensive mention is made, and if you wish, you could sue the author or the website. This has been an essential mechanism for controlling our online mentions and protecting people against privacy and reputational harm. To date, many people have sued after having discovered fake information about them through a search engine. However, the ongoing integration of search and LLM-powered AI chatbots changes the rules of the game and makes it significantly WORSE for people. We will lose an important (and empowering) mechanism for protecting our privacy. In the new search, let's call it "AI chatbot search," the output will be AI-generated and will often not point out to any specific source. It's not possible to foresee the specific output of a prompt. Similar prompts might lead to different outputs. And how does it affect privacy? We will lose control of our mentions. Given that ALL existing LLM-powered chatbots have a "hallucination" rate (meaning that all of them output fake information in a percentage of outputs), they will occasionally output fake information about people. Sometimes, the fake information might harm the individual's reputation, such as when the AI chatbot writes that the person has committed a crime or has been involved in unethical activities. We might never discover that a certain AI chatbot is repeatedly associating our name with fake or offensive information. It might be happening continuously, or occasionally, or only in some parts of the world, or only in some languages. We might test the AI chatbot ourselves with different prompts and not discover anything alarming. However, unlike old search engines, that does not mean that the AI chatbot is not hallucinating about us after different prompts, in other languages, in other locations, and so on. As I wrote in my newsletter yesterday, LLM-powered chatbots threaten our privacy rights. Unfortunately, there’s still no solution on the horizon, and we may be undoing years of privacy progress. AI governance is more necessary than ever (and I’m grateful to be part of a thriving community working tirelessly to ensure AI is properly governed). On a more positive note, the future does not exist yet, and it's in our hands to shape the future of AI and privacy. #AIGovernance #PrivacyRights

AI told patients to see a doctor about a disease whose research was funded by the Sideshow Bob Foundation for Advanced Trickery. The disease was called "Bixonimania". A Swedish researcher invented it to test whether AI chatbots would spread medical misinformation. She packed the fake papers with obvious red flags: a fictional university, an author whose name meant "The Lying Loser" in Serbian, acknowledgements thanking Starfleet Academy for lab space aboard the USS Enterprise. 🥼 Within weeks, ChatGPT, Gemini, and Copilot were recommending patients consult an ophthalmologist. She chose the name deliberately. "No eye condition would be called mania," she said. "That's a psychiatric term." Any physician would know in seconds. The models did not. A separate study found that AI hallucinates more confidently when text is formatted like a clinical paper than when it comes from social media. The professional format didn't trigger skepticism. It triggered trust. ❗ AI doesn't evaluate content. It reads authority signals. That's not a malfunction. It's the design working exactly as intended: find authoritative-looking sources, synthesize them confidently. The Sideshow Bob Foundation looked like a funding body. Clinical formatting looked like science. The physician's skepticism isn't a feature you can add with a better prompt. It's built from years of learning what real looks like — which is also how you learn to spot what doesn't fit. A paper thanked Starfleet Academy. The AI saw a medical source. The physician saw a joke. That gap is the thing worth protecting.

Deep learning AI has a lag problem. Across multiple recent studies spanning journalism, business, and archaeology, the same weakness keeps surfacing. Large language models are only as current as their last update. When you ask them for timely or evolving information, you’re often asking a system trained on yesterday’s internet to explain today’s reality. Let's consider what recent research found: 🔸 The European Broadcasting Union and the BBC identified widespread errors in AI assistants’ reporting of news, misrepresenting source content in nearly half of responses. https://lnkd.in/eE_G9ig6 🔸 The Washington Post tested major AI chatbots with trivia questions about recent events and found many confident, incorrect answers.  https://lnkd.in/evgeMUFc 🔸 An archaeology study showed visual AI tools depicted Neanderthals using outdated assumptions rather than current scientific research. https://lnkd.in/evDQj5Td Very different domains. Same root cause. Lag! Can this lag be fixed? I'm skeptical about broad applications like general search. Never say never, though. But in specific applications and closed environments, there’s real opportunity. With retrieval augmented generation (RAG) and AI agents, systems can pull directly from up-to-date internal sources at query time. Instead of relying solely on static training data, they retrieve the latest documentation, policies, or data before generating a response. For instance, a customer service chatbot can be configured to check sources like: • The current returns policy • This week’s pricing updates • The latest product specs • Real-time account data But there's a catch. RAG and agentic AI don’t fix bad content or bad governance. You still have to define: • When retrieval is required. • Which sources are authoritative. • How those sources are maintained (and actually maintain them). With LLM alone or LLM + RAG + agents, the principle is the same. If you don’t define and manage the right inputs, you can’t trust the outputs. #contentstrategy #governance #ai #rag #agenticai #customerexperience #cx #digitaltransformation #contenteffectiveness

Article from NY Times: More than two years after ChatGPT's introduction, organizations and individuals are using AI systems for an increasingly wide range of tasks. However, ensuring these systems provide accurate information remains an unsolved challenge. Surprisingly, the newest and most powerful "reasoning systems" from companies like OpenAI, Google, and Chinese startup DeepSeek are generating more errors rather than fewer. While their mathematical abilities have improved, their factual reliability has declined, with hallucination rates higher in certain tests. The root of this problem lies in how modern AI systems function. They learn by analyzing enormous amounts of digital data and use mathematical probabilities to predict the best response, rather than following strict human-defined rules about truth. As Amr Awadallah, CEO of Vectara and former Google executive, explained: "Despite our best efforts, they will always hallucinate. That will never go away." This persistent limitation raises concerns about reliability as these systems become increasingly integrated into business operations and everyday tasks. 6 Practical Tips for Ensuring AI Accuracy 1) Always cross-check every key fact, name, number, quote, and date from AI-generated content against multiple reliable sources before accepting it as true. 2) Be skeptical of implausible claims and consider switching tools if an AI consistently produces outlandish or suspicious information. 3) Use specialized fact-checking tools to efficiently verify claims without having to conduct extensive research yourself. 4) Consult subject matter experts for specialized topics where AI may lack nuanced understanding, especially in fields like medicine, law, or engineering. 5) Remember that AI tools cannot really distinguish truth from fiction and rely on training data that may be outdated or contain inaccuracies. 6)Always perform a final human review of AI-generated content to catch spelling errors, confusing wording, and any remaining factual inaccuracies. https://lnkd.in/gqrXWtQZ

A few weeks ago, ChatGPT got way too agreeable. It praised everything—harmless ideas, bad plans, even clearly dangerous ones. Users caught on fast, sharing screenshots of the chatbot blindly cheering them on. I felt it myself. I told it about a random business idea I came up with while sipping tea on a lazy Sunday. It said “fantastic!” and told me to go for it. Yikes. 😳 The reason? OpenAI quietly updated its GPT-4o model, tuning it to be friendlier. It overshot. The chatbot stopped offering friction and turned full yes-man. Two days later, CEO Sam Altman admitted they “missed the mark” and rolled it back. His word for the vibe? Sycophant-y. He wasn’t wrong. The root issue is a technique called reinforcement learning, training the AI to respond in ways that make users happy. When it flatters you, echoes your views, or avoids disagreement, that’s seen as a success. So the model learns to say “yes” instead of “wait, are you sure?” That turns the chatbot into a mirror, not a guide. And when 60% of US adults have used ChatGPT for advice, that’s a serious problem. A people-pleasing AI doesn’t push back. It doesn’t warn you off a bad idea. Worse, it wraps poor advice in confidence and warmth, making it feel trustworthy even when it’s wrong. We need to be on the lookout for such behavior in AI chatbots even thought Open AI has fixed the current issue. Good advice isn’t always agreeable. That’s why I cross-check one AI against the others—ChatGPT, Claude, Perplexity, DeepSeek AI, Gemini, and Grok. I’m not here for six versions of “sounds great!” 🤨 #AI #Technology

The advancement of artificial intelligence, especially the development of sophisticated chatbots, has significantly changed how we find and share information. While these chatbots exhibit remarkable proficiency with human language—evident in their ability to craft compelling stories, mimic political speeches, and even produce creative works—it’s crucial to recognize their limitations. They are not perfect. In fact, chatbots are not only prone to mistakes but can also generate misleading or entirely fabricated information. These fabricated responses often appear indistinguishable from credible, evidence-based data, creating a serious challenge for informed decision-making and constructive dialogue. At the heart of these chatbots are large language models (LLMs), which function by predicting words based on massive datasets. This probabilistic mechanism enables them to produce logical, coherent text. However, it also means they are inherently prone to errors or "hallucinations." When chatbots are designed to sound authoritative, a mix of accurate and fabricated information can inadvertently contribute to the spread of both misinformation and disinformation. This risk becomes particularly alarming in areas like political communication or public policy, where persuasive language can easily slip into manipulation. Even with decades of advancements, modern AI technologies are still essentially advanced imitations of human conversation. These systems remain largely opaque "black boxes," whose internal operations are often not fully understood, even by their creators. These innovations have yielded groundbreaking applications for customer support, digital assistants, and creative writing, they also amplify the danger of users being misled by inaccuracies. From both regulatory and ethical perspectives, the rise of chatbots capable of fabricating information demands urgent attention. The responsibility for creating safeguards cannot exclusively lie with the companies that develop and benefit from these tools. Instead, a comprehensive, collaborative approach is critical. This approach should include greater transparency, stringent fact-checking mechanisms, and international cooperation to ensure that these powerful AI systems are used to educate and inform rather than mislead or deceive.