How Problem Structure Affects Large Language Models

LLMs are impressive at language. But they struggle with something humans find easy: spatial reasoning. A new paper shows why, and offers a surprising solution. Researchers tested LLMs on spatial planning tasks: stacking bricks, navigating environments, manipulating objects in space. Using standard Chain-of-Thought prompting (where the model reasons step-by-step in natural language), performance was poor. On some tasks, accuracy was only 31.8%. The problem: natural language is verbose and imprecise for representing spatial relationships. So the researchers tried something different: Chain-of-Symbol (COS) prompting. Instead of describing spatial relationships in words, they used condensed symbolic representations. For example: → Instead of: "Brick A is on top of Brick B, which is on top of Brick C" → Use symbols: "A/B/C" The results were dramatic: → Accuracy jumped from 31.8% to 92.6% on complex spatial tasks → Token usage dropped by 66% (from 407 to 139 tokens) → Consistent improvements across multiple spatial reasoning benchmarks Here's what makes this interesting: LLMs are trained on language. But language isn't always the best representation for the problems we're trying to solve. Spatial relationships, mathematical operations, logical structures → these often have more efficient symbolic representations. The study showed that when you match the representation to the problem structure, performance improves dramatically. This has implications beyond spatial reasoning: For domains with structured relationships (clinical pathways, anatomical hierarchies, diagnostic flowcharts) symbolic or semi-symbolic representations might work better than pure natural language. We've been pushing LLMs to reason about everything through language because that's what they're trained on. But maybe the key to better performance isn't just bigger models or more data, it's smarter prompting that uses the right representation for the task. The researchers also found something interesting about model size: larger models showed stronger benefits from symbolic prompting, suggesting an emergent ability to understand abstract symbols. This matters because it points toward a future where we're not just building bigger language models, but building systems that can reason across multiple modalities and representations. *** What other problem domains might benefit from symbolic rather than natural-language representations? Source: "Chain-of-Symbol Prompting for Spatial Reasoning in Large Language Models" (Hu et al., COLM 2024) #AI #LLM

If you’re an AI engineer trying to understand how reasoning actually works inside LLMs, this will help you connect the dots. Most large language models can generate. But reasoning models can decide. Traditional LLMs followed a straight line: Input → Predict → Output. No self-checking, no branching, no exploration. Reasoning models introduced structure, a way for models to explore multiple paths, score their own reasoning, and refine their answers. We started with Chain-of-Thought (CoT) reasoning, then extended to Tree-of-Thought (ToT) for branching, and now to Graph-based reasoning, where models connect, merge, or revisit partial thoughts before concluding. This evolution changes how LLMs solve problems. Instead of guessing the next token, they learn to search the reasoning space- exploring alternatives, evaluating confidence, and adapting dynamically. Different reasoning topologies serve different goals: • Chains for simple sequential reasoning • Trees for exploring multiple hypotheses • Graphs for revising and merging partial solutions Modern architectures (like OpenAI’s o-series reasoning models, Anthropic’s Claude reasoning stack, DeepSeek R series and DeepMind’s AlphaReasoning experiments) use this idea under the hood. They don’t just generate answers, they navigate reasoning trajectories, using adaptive depth-first or breadth-first exploration, depending on task uncertainty. Why this matters? • It reduces hallucinations by verifying intermediate steps • It improves interpretability since we can visualize reasoning paths • It boosts reliability for complex tasks like planning, coding, or tool orchestration The next phase of LLM development won’t be about more parameters, it’ll be about better reasoning architectures: topologies that can branch, score, and self-correct. I’ll be doing a deep dive on reasoning models soon on my Substack- exploring architectures, training approaches, and practical applications for engineers. If you haven’t subscribed yet, make sure you do: https://lnkd.in/dpBNr6Jg ♻️ Share this with your network 🔔 Follow along for more data science & AI insights

Prompt formatting can have a dramatic impact on LLM performance, but it varies substantially across models. Some pragmatic findings from a recent research paper: 💡 Prompt Format Significantly Affects LLM Performance. Different prompt formats (plain text, Markdown, YAML, JSON) can result in performance variations of up to 40%, depending on the task and model. For instance, GPT-3.5-turbo showed a dramatic performance shift between Markdown and JSON in code translation tasks, while GPT-4 exhibited greater stability. This indicates the importance of testing and optimizing prompts for specific tasks and models. 🛠️ Tailor Formats to Task and Model. Prompt formats like JSON, Markdown, YAML, and plain text yield different performance outcomes across tasks. For instance, GPT-3.5-turbo performed 40% better in JSON for code tasks, while GPT-4 preferred Markdown for reasoning tasks. Test multiple formats early in your process to identify which structure maximizes results for your specific task and model. 📋 Keep Instructions and Context Explicit. Include clear task instructions, persona descriptions, and examples in your prompts. For example, specifying roles (“You are a Python coder”) and output style (“Respond in JSON”) improves model understanding. Consistency in how you frame the task across different formats minimizes confusion and enhances reliability. 📊 Choose Format Based on Data Complexity. For simple tasks, plain text or Markdown often suffices. For structured outputs like programming or translations, formats such as JSON or YAML may perform better. Align the prompt format with the complexity of the expected response to leverage the model’s capabilities fully. 🔄 Iterate and Validate Performance. Run tests with variations in prompt structure to measure impact. Tools like Coefficient of Mean Deviation (CMD) or Intersection-over-Union (IoU) can help quantify performance differences. Start with benchmarks like MMLU or HumanEval to validate consistency and accuracy before deploying at scale. 🚀 Leverage Larger Models for Stability. If working with sensitive tasks requiring consistent outputs, opt for larger models like GPT-4, which show better robustness to format changes. For instance, GPT-4 maintained higher performance consistency across benchmarks compared to GPT-3.5. Link to paper in comments.

What if giving an LLM less Knowledge Graph information leads to better SPARQL? That thought stayed with me after reading a recent paper on natural language to SPARQL translation over domain-specific Knowledge Graphs. Most KGQA systems follow the same reflex. If the model struggles, we add more structure. More triples. More ontology. More context. This work questions that reflex. Instead of feeding LLMs reduced RDF graphs, the authors show that ontology vocabulary alone can be sufficient. Classes, properties, instances. No serialized graphs. No labeled NL–SPARQL pairs. And it still works. Across a real, domain-heavy railway knowledge graph, vocabulary-only prompts achieved accuracy comparable to prompts augmented with RDF triples. In practice, this means fewer tokens, less prompt complexity, and fewer hallucinated properties. What really stood out to me was the shift in perspective. The bottleneck in domain KGQA is not always reasoning over triples. Often it is simply knowing which terms exist and how they are allowed to connect. Once the model is grounded in the domain’s vocabulary, it can already do much of the rest. The comparison between OpenAI GPT-3.5 and Google Gemini reinforces this. Larger context windows and stronger code generation capabilities translate directly into more reliable SPARQL, especially as queries become multi-step. This paper does not argue for bigger prompts or heavier pipelines. It argues for semantic restraint. Teach the model the language of the graph, not the entire graph itself. That idea feels quietly important, especially for enterprise and scientific KGs where token budgets, ontology churn, and missing training data are the norm. Some papers don’t announce a new technique. They just change how you design the next system. Full length article: https://lnkd.in/ggBiWQJT #KnowledgeGraphs #KGQA #SPARQL #SemanticWeb #Ontologies #OntologyEngineering #GraphAI #GraphRAG #LargeLanguageModels #LLMs #PromptEngineering #NL2SPARQL #NaturalLanguageInterfaces #EnterpriseAI #IndustrialAI #DomainSpecificAI #DataInteroperability #LinkedData #AIResearch #AppliedAI #AIInfrastructure

Most large language models are trained to predict the next token in a left-to-right (L2R) manner. However, Apple researchers discovered that right-to-left (R2L) models can significantly outperform L2R models on specific multiple-choice question (MCQ) tasks! I just read this new paper "Reversal Blessing: Thinking Backward May Outpace Thinking Forward in Multi-choice Questions" that challenges our assumptions about how language models process information. This "reverse thinking" approach uses Bayesian inference to evaluate answer choices based on their likelihood of generating the question, rather than the traditional approach of evaluating questions to predict answers. Surprising Results: The researchers trained both L2R and R2L models with identical data and computational resources across different model sizes (2B-8B parameters). - R2L models consistently outperformed L2R on logical reasoning tasks (LogiQA) - R2L excelled at commonsense understanding tasks (OpenbookQA, CommonsenseQA) - R2L showed dramatic improvement on truthfulness assessment (TruthfulQA - 51% better!) What's fascinating is that these improvements held across different model sizes, datasets, and random seeds, suggesting this isn't just statistical noise. Why Does This Work? The researchers explored three hypotheses for why R2L performs better on certain tasks: 1. Calibration - R2L naturally "auto-normalizes" different answer choices, avoiding the "surface competition" issue where semantically similar answers (like "dog" and "puppy") split probability mass 2. Computability - Different directional factorizations have varying computational complexity 3. Conditional Entropy - The optimal reasoning direction corresponds to lower conditional entropy Through controlled simulation studies with arithmetic tasks, they found strong evidence supporting the conditional entropy hypothesis - the direction with lower conditional entropy tends to perform better. Implications This research suggests exciting possibilities for future language model development: - We might benefit from models that can reason in multiple directions - Alternative factorizations beyond L2R and R2L could further enhance LLM capabilities - Task-specific reasoning directions could boost performance on targeted applications The study suggests that our default assumptions about "forward thinking" might not always be optimal.

𝗜 𝗵𝗮𝘃𝗲 𝗯𝗲𝗲𝗻 𝗶𝗻 𝘁𝗵𝗲 𝗡𝗟𝗣 𝘀𝗽𝗮𝗰𝗲 𝗳𝗼𝗿 𝗮𝗹𝗺𝗼𝘀𝘁 𝟭𝟬 𝘆𝗲𝗮𝗿𝘀 𝗻𝗼𝘄, and I know the first-hand challenges of building text-based models in the pre-GPT era! So, I am a 𝗽𝗿𝗼-𝗟𝗮𝗿𝗴𝗲 𝗟𝗮𝗻𝗴𝘂𝗮𝗴𝗲 𝗠𝗼𝗱𝗲𝗹 (𝗟𝗟𝗠) 𝗲𝗻𝘁𝗵𝘂𝘀𝗶𝗮𝘀t, but I don’t believe they will replace humans or solve all our problems, especially when it comes to highly complex reasoning in industries like Finance. 𝗧𝗵𝗶𝘀 𝘄𝗲𝗲𝗸𝗲𝗻𝗱, I spent reading two compelling papers, and I’m convinced we’re bumping into real reasoning ceilings: 𝗜> "𝗧𝗵𝗲 𝗜𝗹𝗹𝘂𝘀𝗶𝗼𝗻 𝗼𝗳 𝗧𝗵𝗶𝗻𝗸𝗶𝗻𝗴: 𝗨𝗻𝗱𝗲𝗿𝘀𝘁𝗮𝗻𝗱𝗶𝗻𝗴 𝘁𝗵𝗲 𝗦𝘁𝗿𝗲𝗻𝗴𝘁𝗵𝘀 𝗮𝗻𝗱 𝗟𝗶𝗺𝗶𝘁𝗮𝘁𝗶𝗼𝗻𝘀 𝗼𝗳 𝗥𝗲𝗮𝘀𝗼𝗻𝗶𝗻𝗴 𝗠𝗼𝗱𝗲𝗹𝘀 𝘃𝗶𝗮 𝘁𝗵𝗲 𝗟𝗲𝗻𝘀 𝗼𝗳 𝗣𝗿𝗼𝗯𝗹𝗲𝗺 𝗖𝗼𝗺𝗽𝗹𝗲𝘅𝗶𝘁𝘆" (Apple) Apple researchers rigorously tested 𝗟𝗮𝗿𝗴𝗲 𝗥𝗲𝗮𝘀𝗼𝗻𝗶𝗻𝗴 𝗠𝗼𝗱𝗲𝗹𝘀 (𝗟𝗥𝗠𝘀), LLMs that explicitly generate chain-of-thought reasoning, using controlled puzzles like Tower of Hanoi and River Crossing Key insights: 1. 𝗧𝗵𝗿𝗲𝗲 𝗿𝗲𝗮𝘀𝗼𝗻𝗶𝗻𝗴 𝗿𝗲𝗴𝗶𝗺𝗲𝘀: ▪️Low complexity: standard LLMs outperform LRMs ▪️Medium complexity: LRMs excel ▪️High complexity: 𝗯𝗼𝘁𝗵 𝗰𝗼𝗹𝗹𝗮𝗽𝘀𝗲, accuracy plummets 2. Fascinating observation, 𝗟𝗥𝗠𝘀 “𝗴𝗶𝘃𝗲 𝘂𝗽” as puzzle complexity increases, their reasoning effort declines rapidly, even with enough tokens 3. Even when provided an exact algorithm (e.g., Tower of Hanoi strategy), the 𝗺𝗼𝗱𝗲𝗹𝘀 𝘀𝘁𝗶𝗹𝗹 𝗳𝗮𝗶𝗹𝗲𝗱 𝘁𝗼 𝗴𝗲𝗻𝗲𝗿𝗮𝗹𝗶𝘇𝗲 and mostly outputs based on past observed data pattern it is trained on 𝗜𝗜> "𝗥𝗲𝗮𝘀𝗼𝗻𝗶𝗻𝗴 𝗼𝗿 𝗢𝘃𝗲𝗿𝘁𝗵𝗶𝗻𝗸𝗶𝗻𝗴: 𝗘𝘃𝗮𝗹𝘂𝗮𝘁𝗶𝗻𝗴 𝗟𝗮𝗿𝗴𝗲 𝗟𝗮𝗻𝗴𝘂𝗮𝗴𝗲 𝗠𝗼𝗱𝗲𝗹𝘀 𝗼𝗻 𝗙𝗶𝗻𝗮𝗻𝗰𝗶𝗮𝗹 𝗦𝗲𝗻𝘁𝗶𝗺𝗲𝗻𝘁 𝗔𝗻𝗮𝗹𝘆𝘀𝗶𝘀" (Dimitris Vamvourellis & Dhagash Mehta, Ph.D., BlackRock) This study tested major 𝗟𝗟𝗠𝘀 (𝗚𝗣𝗧‐𝟰𝗼, 𝗚𝗣𝗧‐𝟰.𝟭, 𝗼𝟯‐𝗺𝗶𝗻𝗶, 𝗙𝗶𝗻𝗕𝗘𝗥𝗧 𝘃𝗮𝗿𝗶𝗮𝗻𝘁𝘀) on financial sentiment classification using: - "𝗦𝘆𝘀𝘁𝗲𝗺 𝟭" (𝗳𝗮𝘀𝘁/𝗶𝗻𝘁𝘂𝗶𝘁𝗶𝘃𝗲) - "𝗦𝘆𝘀𝘁𝗲𝗺𝟮" (𝘀𝗹𝗼𝘄/𝗱𝗲𝗹𝗶𝗯𝗲𝗿𝗮𝘁𝗲) 𝗽𝗿𝗼𝗺𝗽𝘁𝗶𝗻𝗴 Key takeaways: ▪️𝗥𝗲𝗮𝘀𝗼𝗻𝗶𝗻𝗴 𝗽𝗿𝗼𝗺𝗽𝘁𝘀 𝗱𝗶𝗱 𝗻𝗼𝘁 𝗶𝗺𝗽𝗿𝗼𝘃𝗲 𝗽𝗲𝗿𝗳𝗼𝗿𝗺𝗮𝗻𝗰𝗲 ▪️Surprisingly, straightforward, intuitive prompts with GPT-4o (no chain-of-thought) outperformed all others  ▪️More reasoning led to overthinking, reducing alignment with human-labeled sentiments 💡 Why it matters for builders and researchers in Finance and every industry: ❎ 𝗕𝗶𝗴𝗴𝗲𝗿 𝗺𝗼𝗱𝗲𝗹𝘀 + 𝗺𝗼𝗿𝗲 “𝘁𝗵𝗶𝗻𝗸𝗶𝗻𝗴” = 𝗯𝗲𝘁𝘁𝗲𝗿 𝗼𝘂𝘁𝗰𝗼𝗺𝗲𝘀. Sometimes it’s actively worse ❎ We’re not seeing a soft plateau — these are 𝗵𝗮𝗿𝗱 𝗰𝗲𝗶𝗹𝗶𝗻𝗴𝘀 𝗶𝗻 𝗿𝗲𝗮𝘀𝗼𝗻𝗶𝗻𝗴 𝗰𝗮𝗽𝗮𝗰𝗶𝘁𝘆 ❎ For real-world systems, agents, and financial tools: design for 𝗿𝗲𝗮𝘀𝗼𝗻𝗶𝗻𝗴 𝗲𝗰𝗼𝗻𝗼𝗺𝘆, not just reasoning depth. #LLMs #ReasoningLimits #LLMChainofthought #LLMReasoningDecline

📏 NOLIMA: Long-Context Evaluation Beyond Literal Matching This paper challenges how we test large language models' ability to handle long texts by revealing they often rely on simple word matching rather than true understanding. → Current benchmarks are too easy: Most tests let models find answers by matching words between questions and text. NOLIMA removes these matches, forcing models to actually understand relationships. → Models struggle more than we thought: Even top models like GPT-4 and Llama 3.3 70B see huge performance drops with longer texts. When tested on NOLIMA, 10 out of 12 models lose half their accuracy at 32K tokens. → Position matters less than length: In complex tasks, the bigger problem isn't where information sits in the text - it's how long the text is. Models get overwhelmed trying to make connections in longer documents. → Chain-of-thought helps but doesn't solve it: Making models explain their reasoning improves results but doesn't fix the core problem. Even "reasoning-focused" models still struggle with long texts. My take: It's nice to see more work around long-context evaluations. I think this is something that users observed for a long time but isn't necessarily properly reflected in current benchmarks. Yes, we have a lot of LLMs with context windows of >128k tokens, but they're rarely accurate. For real applications like search engines or document analysis, this is a big deal: models might miss crucial information just because it's worded differently. 📝 Paper: https://lnkd.in/eAyizAiM

Apple’s machine learning team just released a paper that takes aim at one of the core assumptions behind Chain-of-Thought (CoT) prompting—a technique used to help large language models (LLMs) “think out loud” to solve complex problems. What they found? Many CoT-based models collapse when applied to complex reasoning tasks like the advanced levels in Tower of Hanoi (e.g., with more than 8 disks to place), despite performing well on traditional benchmarks. Why? Because these tasks go well beyond the narrow prompting examples used during fine-tuning and require longer sequences of precise reasoning than a CoT model can handle. An interesting observation from the paper is that, for the simple cases, the raw LLMs actually perform slightly better than LRMs, though LRMs significantly outperform raw LLMs in medium-level cases. This indicates that if we can decompose a long/difficult reasoning task into several medium-level tasks, we can still make the best use of existing LRMs, and if we can decompose them further into many simple-level tasks, a standard LLM would even be better than LRMs. Considering the fact that the response lengths of LRMs are usually much longer than standard LLMs (LRMs need to generate its reasoning process explicitly), we are actually not only solving the problem better, but also at a cheaper cost. What does this mean for users? If you’ve been relying on a single model to handle multi-step reasoning—like planning, logic puzzles, or simulations—this paper suggests you might want to rethink your approach. Here’s my take: - While I’ve always been skeptical of CoT-style large reasoning models (LRMs), I don’t think we should write them off completely. They’re specialists—and they can outperform on tough tasks like coding or niche benchmarks. But they are constrained by their inherent imprecision that emerges as tasks scale. - For broader, more general-purpose use cases, LLMs paired with multi-agent systems are a more robust path forward. Instead of pushing a single model to its limits, we can distribute reasoning across agents—each focused, each efficient—working together to scale intelligence more reliably. Worth a read: Apple’s study via The Guardian: https://lnkd.in/gEq2hYhK Cognizant, Xin Qiu, Elliot Meyerson

Large language models that score well on medical exams lose accuracy when answer patterns are disrupted, showing they rely more on pattern matching than true reasoning. 1️⃣ Researchers modified MedQA questions by replacing the correct answer with "None of the other answers" (NOTA), while keeping the reasoning unchanged. 2️⃣ Six LLMs (DeepSeek-R1, o3-mini, Claude-3.5 Sonnet, Gemini-2.0, GPT-4o, Llama-3.3) were tested with chain-of-thought prompts. 3️⃣ All models showed significant drops in accuracy when faced with NOTA-modified questions. 4️⃣ Accuracy losses ranged from 8.8% (DeepSeek-R1) to 38.2% (Llama-3.3-70B). 5️⃣ Even reasoning-focused models like o3-mini fell by over 16%. 6️⃣ The results suggest models depend on answer pattern familiarity, not consistent medical reasoning. 7️⃣ A system that drops from 80% to 42% accuracy under minor format changes is unreliable for clinical use. 8️⃣ Current benchmarks overstate readiness of LLMs for medical deployment. 9️⃣ Authors call for new tests that separate reasoning from memorization. 🔟 Clinical deployment should remain supportive and supervised until models show stable reasoning under novel scenarios. ✍🏻 Suhana Bedi, Yixing J., Philip Chung, Sanmi Koyejo, Nigam Shah. Fidelity of Medical Reasoning in Large Language Models. JAMA Network Open. 2025. DOI: 10.1001/jamanetworkopen.2025.26021

Hundreds of millions of people are now learning technical and scientific concepts through general-purpose large language models (LLMs), yet these models operate with an accuracy rate of only around 60%. At first glance, these small inaccuracies may seem inconsequential, but they are quietly embedding themselves into our collective knowledge systems, posing a hidden and compounding risk. If left unaddressed, this slow accumulation of errors will inevitably surface over the next 10–15 years, manifesting in systemic failures across precision-critical fields like engineering, medicine, and infrastructure—potentially leading to devastating technogenic disasters. A common misconception is that upgrading to larger, more advanced models—often at significantly higher costs—will inherently produce better results. In reality, a model’s effectiveness is not defined by its scale or cost alone but by how well it aligns with the complexity and specificity of the task at hand. High-capacity models excel at tackling high-entropy problems—emergent reasoning, unstructured data analysis, or complex symbolic dynamics. However, for simpler, structured tasks, smaller models often outperform them: they are more efficient, less prone to overfitting, and faster at inference due to their reduced parameter space and computational overhead. Moreover, the principle of diminishing returns applies here. Once a model meets a task’s performance threshold, increasing its size yields negligible improvements and may even introduce unnecessary complexity. Beneath this lies a deeper limitation: LLMs fundamentally operate within their own latent vector spaces, lacking the ability to dynamically map to the real-world physical feedback systems. This creates a gap between high-dimensional abstractions and the lower-dimensional reality they aim to represent—a gap that breeds information distortion, undermining decision-making in critical applications. If these issues persist, we risk cascading into three interconnected crises: First, a crisis of trust in technology, as flawed knowledge is applied to real-world systems, eroding their reliability. Second, an innovation stagnation crisis, where entrenched inaccuracies hinder true breakthroughs in science and engineering. Third, a collapse of our knowledge frameworks, as unchecked, unverifiable information spreads, blurring the lines between truth and illusion. The solution lies not in an unquestioned pursuit of “bigger and more expensive,” but in returning to a task-driven, rational approach—matching model architectures to the inherent complexity and demands of specific tasks. Equally crucial is the creation of robust verification mechanisms and domain-specific oversight, where general-purpose models work in tandem with high-precision, specialized systems to ensure accuracy and reliability.