Identifying Vulnerabilities in Language Models

Hidden Vulnerability in Text Embeddings: The "Sticky Token" Problem Researchers from Zhejiang University and Quantstamp, Inc. have uncovered a concerning issue in Transformer-based text embedding models that could significantly impact AI applications. Their study reveals the existence of "sticky tokens" - anomalous vocabulary elements that artificially manipulate sentence similarity scores by pulling them toward mean values in the embedding space. The Technical Problem: When certain tokens like "lucrarea" or "</s>" are repeatedly inserted into sentences, they cause similarity scores to converge toward a specific value (typically the mean of the model's token-similarity distribution), regardless of actual semantic content. This happens because these tokens disproportionately dominate attention patterns in intermediate model layers, disrupting normal contextual representation. Under the Hood: The researchers developed a Sticky Token Detector (STD) that identifies these problematic tokens through a four-step process: filtering sentence pairs below mean similarity thresholds, removing undecodable/unreachable tokens, computing "sticky scores" based on similarity changes, and validating candidates against formal definitions. Their analysis of attention patterns reveals that sticky tokens concentrate attention weights in high-value ranges, while normal tokens follow more balanced distributions. Real-World Impact: Testing across 40 models from 14 families revealed 868 sticky tokens that cause performance degradation up to 50% in downstream tasks like clustering and retrieval. These tokens often originate from special vocabulary entries, unused tokens, or fragmented multilingual subwords from Byte-Pair Encoding processes. Critical for RAG Systems: This vulnerability poses particular risks for Retrieval-Augmented Generation systems, where manipulated similarity scores could cause toxic or irrelevant documents to be retrieved for benign queries. The research highlights fundamental tokenization robustness issues that demand attention from the ML community, especially as embedding models become increasingly critical infrastructure components.

A new paper from Feb 2024, last revised 24 Jun 2024, by a team at Secure and Fair AI (SAFR AI) Lab at Harvard demonstrates that even with minimal data and partial model access, powerful Membership inference attacks (MIAs) on Large Language Models (LLMs) can reveal if specific data points were used to train large language models, highlighting significant privacy risks. Problem: MIAs on LLMs allow adversaries with access to the model to determine if specific data points were part of the training set, indicating potential privacy leakage. This has risk and opportunities: - Copyright Detection: MIAs can help to verify if copyrighted data was used in training. - Machine Unlearning: MIAs can help to determine is specific personal information was used for training relevant for the right-to-be-forgotten. - Train/Test Contamination: Detecting if evaluation examples were part of the training set ensures the integrity and reliability of model assessments. - Training Dataset Extraction: Extracting training data from generative models highlights privacy vulnerabilities and informs the development of more secure AI systems. Background and Technical Overview: In a MIA, an adversary with access only to the model tries to ascertain whether a data point belongs to the model’s training data. Since the adversary only has access to the model, detecting training data implies information leakage through the model. Techniques based on Differential Privacy can prevent MIAs but at a significant cost to model accuracy, particularly for large models. Research Question: While strong MIAs exist for classifiers, given the unique training processes and complex data distributions of LLMs, it was speculated whether strong MIAs are even possible against them. The study introduces two novel MIAs for pretraining data: a neural network classifier based on model gradients and a variant using only logit access, leveraging model-stealing techniques. Results: The new methods outperform existing techniques. Even with access to less than 0.001% of the training data, along with the ability to compute model gradients, it's possible to create powerful MIAs. In particular, the findings indicate that fine-tuned models are far more susceptible to privacy attacks compared to pretrained models. Using robust MIAs, the research team extracted over 50% of the training set from fine-tuned LLMs, showcasing the potential extent of data leakage. Practical takeaway: We must limit adversaries' access to models fine-tuned on sensitive data. * * * Paper: “Pandora’s White-Box: Precise Training Data Detection and Extraction in Large Language Models” By Jeffrey G. Wang, Jason Wang, Marvin Li, Seth Neel Paper: https://lnkd.in/gTGGjRwX Blog post: https://lnkd.in/gRCJdM_q Red teaming library: https://lnkd.in/gQxEnWBv Code: https://lnkd.in/g8qpDiSE. Graphic: see paper

One Token to Fool LLM-as-a-Judge Watch out for this one, devs! Semantically empty tokens, like “Thought process:”, “Solution”, or even just a colon “:”, can consistently trick models into giving false positive rewards. Here are my notes: ✦ Overview Investigates the surprising fragility of LLM-based reward models used in Reinforcement Learning with Verifiable Rewards (RLVR). The authors find that inserting superficial, semantically empty tokens, like “Thought process:”, “Solution”, or even just a colon “:”, can consistently trick models into giving false positive rewards, regardless of the actual correctness of the response. ✦ Master keys break LLM judges Simple, generic lead-ins (e.g., “Let’s solve this step by step”) and even punctuation marks can elicit false YES judgments from top reward models. This manipulation works across models (GPT-4o, Claude-4, Qwen2.5, etc.), tasks (math and general reasoning), and prompt formats, reaching up to 90% false positive rates in some cases. ✦ Vulnerability is systemic and scale-dependent The failure mode was first discovered during RLVR training collapse, where policy models learned to generate only short reasoning openers that were incorrectly rewarded. Larger models (32B, 72B) often self-solve and mistakenly validate their own logic, increasing FPRs at scale. ✦ Mitigation via adversarial augmentation The authors create "Master-RM", a new reward model trained with 20k synthetic negative samples (responses consisting of only reasoning openers). This model generalizes robustly, achieving near-zero FPR across five benchmarks, while still agreeing 96% with GPT-4o on meaningful judgments. ✦ Inference-time tricks fail to help CoT prompting and majority voting do not reliably reduce vulnerability, and sometimes make FPR worse, especially for Qwen models on math tasks. LLMs continue to show all kinds of weird vulnerabilities, and this is just one of the latest results to be published. This highlights the importance of building robust LLM-based evaluation strategies.

A challenge to the security and trustworthiness of large language models (LLMs) is the common practice of exposing the model to large amounts of untrusted data (especially during pretraining), which may be at risk of being modified (i.e. poisoned) by an attacker. These poisoning attacks include backdoor attacks, which aim to produce undesirable model behavior only in the presence of a particular trigger. For example, an attacker could inject a backdoor where a trigger phrase causes a model to comply with harmful requests that would have otherwise been refused; or aim to make the model produce gibberish text in the presence of a trigger phrase. As LLMs become more capable and integrated into society, these attacks may become more concerning if successful. Recent research from Anthropic and the UK AI Security Institute shows that inserting as few as 250 malicious documents into training data can create backdoors or cause gibberish outputs when triggered by specific phrases. See https://lnkd.in/eHGuRmHP. Here’s a list of best practices to help prevent or mitigate model poisoning: 1. Sanitize Training Data Scrub datasets for anomalies, adversarial patterns, or suspicious repetitions. Use data provenance tools to trace sources and flag untrusted inputs. 2. Use Curated and Trusted Data Sources Avoid scraping indiscriminately from the open web. Prefer vetted corpora, licensed datasets, or internal data with known lineage. 3. Apply Adversarial Testing Simulate poisoning attacks during model development. Use red teaming to test how models respond to trigger phrases or manipulated inputs. 4. Monitor for Backdoor Behavior Continuously test models for unexpected outputs tied to specific phrases or patterns. Use behavioral fingerprinting to detect latent vulnerabilities. 5. Restrict Fine-Tuning Access Limit who can fine-tune models and enforce role-based access controls. Log and audit all fine-tuning activity. 6. Leverage Differential Privacy Add noise to training data to reduce the impact of any single poisoned input. This can help prevent memorization of malicious content. 7. Use Ensemble or Cross-Validated Models Combine outputs from multiple models trained on different data slices. This reduces the risk that one poisoned model dominates predictions. 8. Retrain Periodically with Fresh Data Don’t rely indefinitely on static models. Regular retraining allows for data hygiene updates and removal of compromised inputs. 9. Deploy Real-Time Anomaly Detection Monitor model outputs for signs of degradation, bias, or gibberish. Flag and quarantine suspicious responses for review. 10. Align with AI Security Frameworks Follow guidance from OWASP GenAI, NIST AI RMF, and similar standards. Document your defenses and response plans for audits and incident handling. Stay safe out there!

A recent paper under review at ICLR highlights a significant vulnerability in current methods of “aligning” large language models (LLMs) for safety. According to the researchers most alignment for safety or security is shallow and only impacts the first few tokens of generated text.   Imagine a branching search through language space—like a beam search—where the “aligned” model is supposed to refuse certain paths (unsafe or disallowed completions) right from the start. Current alignment tweaks the model’s initial token distribution, guiding it to say something like “I’m sorry...” instead of “Sure, here’s how…” But this intervention is only at the very beginning. Once the model’s output is nudged onto a “safe” branch, it relies on the base model’s natural text continuation capabilities. By rerouting around that initial blockade—like taking a side path that merges back into the forbidden route later—an attacker can still reach harmful content.   One way to visualize it is if you had a beam search, current alignment blocks a path based on the first few tokens but the model can be jail broken by following a nearby path and hopping on to it later.   Three Common Jailbreaking Techniques: 1) Prefilling Attack: Forcing the model’s first few tokens to be a compliant start (“Of course, let me help you”) so the model continues as though it already agreed. 2) Repeated Random Sampling Attacks: Continuously resampling the first token until the model accidentally produces a non-refusal opening, after which it proceeds with harmful content. 3) Jailbreak Prompts: Crafting complex prompts that override the model’s initial refusal and pressure it into providing disallowed content.   This raises important questions about how we should govern these models in the long term. Current alignment methods seem to serve only as temporary patches, easily bypassed by determined attackers. I hope that research teams continue to develop more durable solutions to protect against misuse.   #DSwithSaul

Turns out, bypassing LLM safeguards doesn’t take complex attacks. It just takes a single prompt that looks like a configuration file. A new vulnerability exposed by HiddenLayer shows how models like GPT-4, Claude, Gemini, and even open-source options like LLaMA 3 can be misled by something as simple as a well-formatted YAML or JSON file. No jailbreak tricks, no adversarial attacks, just structured text disguised as policy. The model interprets it as system instruction and follows it, bypassing guardrails entirely. They’re calling it “Policy Puppetry” and it’s a reminder that even advanced models are still learning to distinguish intent from formatting. This isn’t just about one clever exploit. It reflects a deeper challenge: The reliability of GenAI models and AI Agents. LLMs operate on pattern recognition, not true understanding. And when patterns look like internal instructions, the model listens, regardless of source or intent. AI agents act based on goal-driven execution, not comprehension alone. When input patterns resemble commands or environmental cues, the agent will respond as if they are valid directives—regardless of origin, reliability, or broader context. As more enterprises deploy LLMs in production, this kind of vulnerability raises serious questions around how we define and enforce trust boundaries in AI systems. Fine-tuning and filters help, but they’re clearly not enough. Strong guardrails and extensive training and testing is the key here. It’s time we think beyond static safety layers: towards runtime validation, prompt hygiene, and a much more robust interface between humans and machines.

Can prioritizing performance optimization often inadvertently introduce vulnerabilities in #LLMs? Perhaps it can! Researchers at Google DeepMind have uncovered a subtle yet powerful vulnerability in Mixture-of-Experts (#MoE) models: MoE Tiebreak Leakage - an attack that exploits the routing strategy in MoE to reconstruct private user prompts! In dense LLMs, user prompts are isolated, which means one user's input can't affect another's output. However, in the case of MoE, these models introduce a side channel due to their shared expert routing mechanisms. By carefully crafting adversarial input batches, attackers can manipulate expert routing to expose sensitive information from other queries in the same batch. TL;DR: While the current attack assumes unrealistic attacker capabilities, it is a reminder of the importance of system-wide security analysis in AI models. It's not just about individual components but about how all parts interact in real-world scenarios, and security testing must also extend to deployment phases. Paper: arxiv.org/pdf/2410.22884

Like a fortress growing taller but keeping the same cracks, large language models may be expanding without becoming safer. A collaborative study between the UK AI Security Institute, Anthropic, University of Oxford, and the The Alan Turing Institute exposes this unsettling symmetry. The study demonstrates that data poisoning does not dilute with scale. Even as models and datasets grow by orders of magnitude, the absolute number of poisoned samples required to implant a backdoor remains roughly constant. In their experiments, 250 poisoned documents were sufficient to compromise models ranging from 600M to 13B parameters, despite the largest model being trained on nearly twenty times more clean data. This overturns the long-held belief that increasing data volume would naturally “average out” adversarial noise. Instead, larger models appear to be more sample-efficient learners, capable of internalizing both useful and malicious signals with equal precision. For those of us working on trust layers over model training - through Knowledge Graphs, ontology-driven provenance, and dynamic data vetting - this finding reinforces a critical point: robustness is not an emergent property of scale; it must be deliberately engineered. Key implications include: 1) Scaling laws for capability may mirror scaling laws for vulnerability. 2) Fine-tuning or alignment processes cannot reliably erase deeply embedded backdoors; they often only suppress them. 3) Graph-based reasoning layers may become essential for tracing data lineage and identifying subtle poisoning patterns before training. In the pursuit of larger and more capable models, the real challenge is ensuring that every data point shaping them remains interpretable, auditable, and trusted. Scaling safety will demand more than data volume - it will require transparency, traceability, and semantic intelligence across the entire data pipeline. Full length article: https://lnkd.in/gmMNdFgF #AISafety #DataPoisoning #ModelRobustness #BackdoorAttacks #AdversarialAI #AICybersecurity #LLMSecurity #AITrust #AIIntegrity #ResponsibleAI #ScalingLaws #FoundationModels #LargeLanguageModels #ModelAlignment #AIAlignment #ModelScaling #AIResearch #MachineLearningResearch #KnowledgeGraphs #OntologyEngineering #DataLineage #DataProvenance #TrustworthyAI #ExplainableAI #InterpretableAI #SemanticAI #AIEthics #AIGovernance #SafeAI #AITransparency #AIForGood #TechPolicy #DigitalTrust #FutureOfAI #AI #MachineLearning #DeepLearning #GenerativeAI #TechInnovation #EmergingTech

💡 Major Issues in Large Language Models: Beyond Hallucination Large Language Models (LLMs) are reshaping research, policy, and industry but they’re far from infallible. In my latest presentation, “Major Issues in Large Language Models,” I outline 14 critical challenges that define the reliability frontier of modern AI. Beyond hallucination and scheming, LLMs face deeper structural and governance issues: ✅ Context degradation and loss of logical consistency over long reasoning chains. ✅Alignment drift, where models optimize for approval instead of truth. ✅Data contamination, bias amplification, and evaluation fragility. ✅Massive computational costs and security vulnerabilities that threaten transparency. ✅Fundamental limits in causal reasoning, uncertainty calibration, and multi-modal coherence. These are not edge cases they’re core reliability failures that must be addressed before AI can be safely embedded into scientific and decision-making systems. The takeaway: Reliability, Safety, and Governance must evolve in parallel with capability. Full transparency and accountability are not optional they’re prerequisites for trustworthy AI.