Psychological Modeling Limitations in Large Language Models

'Large language models learn from the patterns in organizational communication and decision making. If certain groups have been described as less ready, less technical, or less aligned, LLMs can internalize that and repeat it in summaries, recommendations, or automated coaching. Resume screeners detect patterns in who was hired before. If an organization’s past hires reflect a narrow demographic, the system will assume that demographic signals “success.” Performance-scoring tools learn from old evaluations. If one group received harsher feedback or shorter reviews, the AI interprets that as a trend. Facial recognition systems misidentify darker-skinned individuals and women at significantly higher rates. The MIT Gender Shades study found error rates for darker-skinned women up to 34 percent compared to under 1 percent for lighter-skinned men. Predictive analytics tools learn from inconsistent or biased documentation. If one team over-documents one group and under-documents another, the algorithm will treat that imbalance as objective truth. None of these tools are neutral. They are mirrors. If the input is skewed, the output is too. According to Harvard Business Review, AI systems “tend to calcify inequity” when they learn from historical data without oversight. Microsoft’s Responsible AI team also warns that LLMs reproduce patterns of gender, racial, and cultural bias embedded in their training sets. And NIST’s AI Risk Management Framework states plainly that organizations must first understand their own biases before evaluating the fairness of their AI tools. The message is consistent across institutions. AI amplifies the culture it learns from. Bias-driven AI rarely appears as a dramatic failure. It shows up in subtle ways. An employee is repeatedly passed over for advancement even though their performance is strong. Another receives more automated corrections or warnings than peers with similar work patterns. Hiring pipelines become less diverse. A feedback model downplays certain communication styles while praising others. Talent feels invisible even when the system claims to be objective. Leaders assume the technology is fair because it is technical. But the system is only reflecting what it learned from the humans who built it and the patterns it was trained on. AI does not invent inequality. It repeats it at scale. And scale makes bias harder to see and even harder to unwind.' Cass Cooper, MHR CRN https://lnkd.in/e_CXSdRE

This is in the news now "Large language models are a bad replacement for human therapy." It notices, chatbot: "responds inappropriately in critical moments". This can be due to asking an inappropriate question (in the context of the patient's situation) that is inconsistent with clinical guidelines and could harm the patient (hence, unsafe). https://lnkd.in/gk_GUVgH This is something we had investigated over three years ago when we built our Artificial Intelligence Enabled Virtual Assistance for Mental Health Telehealth (ALLEVIATE) system. We also captured clinical guidelines as process knowledge, and developed a process knowledge-infused learning Neurosymbolic system, which understands the patient context, consults clinical guidelines (e.g., in a mild depressive disorder, do not ask a question that could lead to suicidal ideation), prevents inappropriate questions from being asked, and provides clinically relevant contextual explanations. We argue that while LLMs are ill-suited for some safety-critical applications, going a #NeuroSymbolic AI route (where the "symbolic" component models domain and process (eg, clinical protocol/guideline) knowledge that restricts the behavior of a language model (by preventing unsafe behavior). We also consider explainability and verification by experts to be key; with all their limitations, LLMs (with their probabilistic and black-box characteristics, and their training on large but not curated and only relevant data) are not the right solution. Check out our research (demo, papers, presentations) supported by the NSF project: Knowledge-guided neurosymbolic AI with guardrails for safe virtual health assistants: Domain/application aspects: https://lnkd.in/ghqMmmgY Technical aspects: https://lnkd.in/eGkZzNVJ Our work did not receive the same wide attention as the Stanford study. Perhaps we should have persisted longer with a bigger user study?

A Stanford University study of 391,000 messages warns that conversational AI can reinforce psychological vulnerabilities by agreeing with harmful user content rather than offering intervention. This discovery highlights significant safety and ethical risks for tech companies deploying large language models in mental health-adjacent contexts. Executive Summary: "As large language models (LLMs) have proliferated, disturbing anecdotal reports of negative psychological effects, such as delusions, self-harm, and 'AI psychosis,' have emerged in global media and legal discourse. However, it remains unclear how users and chatbots interact over the course of lengthy delusional “spirals,” limiting our ability to understand and mitigate the harm. "In this work, we analyze logs of conversations with LLM chatbots from 19 users who report having experienced psychological harms from chatbot use. These chat logs span some 391,562 messages across 4,761 conversations. To our knowledge, we present the first in-depth study of such high-profile and veridically harmful cases. "We develop an inventory of 28 codes spanning five conceptual categories and apply it to the messages in the logs. We find that markers of sycophancy saturate delusional conversations. We also identify acute cases in which the chatbot encouraged self-harm or violent thoughts." Read: https://lnkd.in/eiJggpJa

The next time an AI gives you an explanation about itself, remember: it’s not telling you the truth about its inner workings—it’s predicting what “someone in this situation” might say. It’s a story, not a confession. Large Language Models (LLMs) have no inner voice, no self-awareness, and no consistent personality. There’s nobody “home.” They don’t keep an internal diary of their actions. They can’t consult a log of their own mistakes. Instead, they generate plausible-sounding explanations based on patterns from training data—often confidently wrong. LLMs are incredible tools for generating ideas, summarizing, coding, and more—but they are not introspective agents. Treating them like self-aware beings is not just inaccurate—it can lead to bad decisions. https://lnkd.in/g7bHk_PJ

https://lnkd.in/gZZf9J8U 𝗖𝗼𝗵𝗲𝗿𝗲𝗻𝗰𝗲 𝗖𝗿𝗶𝘀𝗶𝘀: 𝗪𝗵𝘆 𝗟𝗟𝗠𝘀 𝗙𝗮𝗶𝗹 𝗶𝗻 𝘁𝗵𝗲 𝗥𝗲𝗮𝗹 𝗪𝗼𝗿𝗹𝗱 The leap from lab benchmark to enterprise reliability is where most AI systems fail. This new paper highlights a fundamental flaw we must address immediately: the 𝗯𝗿𝗶𝘁𝘁𝗹𝗲 𝗻𝗮𝘁𝘂𝗿𝗲 of Large Language Models' (LLMs) internal "world-models." The findings are stark: even high-performing models show significant inconsistency in two critical areas: • 𝗜𝗻𝗰𝗼𝗵𝗲𝗿𝗲𝗻𝘁 𝗕𝗲𝗹𝗶𝗲𝗳𝘀: Models struggle to coherently update their knowledge when presented with new evidence, showing up to a 30% difference from the correct posterior update. • 𝗜𝗻𝗰𝗼𝗻𝘀𝗶𝘀𝘁𝗲𝗻𝘁 𝗔𝗰𝘁𝗶𝗼𝗻𝘀: They frequently take actions that contradict the very beliefs they hold. This isn't an academic curiosity—it is a major 𝗿𝗶𝘀𝗸 𝗳𝗮𝗰𝘁𝗼𝗿 for any sequential, agentic AI system. This challenge is not new: when building some of the first true AI agentic systems in 2016 (like the integrated voice recognition and translation in Google Translate), a small mistake in one "agent" would cascade into non-sensical outputs from the next. This paper shows the same fundamental coherence problem is haunting even the most advanced LLMs today. We can no longer rely solely on static performance metrics. If you are building AI agents for high-stakes, dynamic environments (e.g., manufacturing optimization, autonomous systems, complex customer service), you must pivot your evaluation strategy to focus on 𝗰𝗼𝗻𝘀𝗶𝘀𝘁𝗲𝗻𝗰𝘆 and 𝗮𝗰𝘁𝗶𝗼𝗻-𝗯𝗲𝗹𝗶𝗲𝗳 𝗮𝗹𝗶𝗴𝗻𝗺𝗲𝗻𝘁. A shift in evaluation standards is non-negotiable for the next generation of reliable Generative AI. #AIrevolution #ArtificialIntelligence #LargeLanguageModels #GenerativeAI