LLM Prompt Challenges in Complex Reasoning

✨ NEW Paper from Apple: The Illusion of Thinking in LLMs Apple researchers discuss the strengths and limitations of reasoning models. Apparently, reasoning models "collapse" beyond certain task complexities. Lots of important insights on this one. (bookmark it!) Here are my notes: ⭐ Paper Overview Investigates the capabilities and limitations of frontier Large Reasoning Models (LRMs) like Claude 3.7, DeepSeek-R1, and OpenAI’s o-series by systematically analyzing their performance on reasoning tasks as a function of problem complexity. Rather than relying on conventional math benchmarks, which suffer from contamination and lack structure, the authors evaluate LRMs using four controllable puzzles (Tower of Hanoi, Checker Jumping, River Crossing, and Blocks World) that allow fine-grained complexity scaling and transparent trace analysis. ⭐ Three complexity regimes The study identifies distinct performance phases. In low-complexity tasks, non-thinking LLMs outperform LRMs due to more efficient and direct computation. In medium complexity, reasoning models show an advantage, leveraging longer chain-of-thoughts to correct errors. However, in high complexity, all models, regardless of their reasoning scaffolds, collapse to near-zero accuracy. ⭐ Counterintuitive reasoning collapse Surprisingly, LRMs reduce their reasoning effort (i.e., number of tokens used in thoughts) as problem complexity increases beyond a threshold. This suggests an internal scaling failure not caused by token limits but by intrinsic model behavior. ⭐ Reasoning trace inefficiencies LRMs frequently “overthink” on simple problems, finding correct answers early but continuing to explore incorrect paths. For moderate tasks, they correct late; and for complex ones, they fail to find any valid solution. Position-based accuracy analysis of thoughts reveals systematic shifts in when correct solutions are generated within the trace. ⭐ Failure to execute explicit algorithms Even when supplied with correct pseudocode (e.g., Tower of Hanoi recursion), models still failed at similar complexity points. This indicates that LRMs don’t just struggle to find solutions; they can’t reliably execute logical instructions either. ⭐ Inconsistent behavior across puzzles Models could perform >100 correct steps in Tower of Hanoi (N=10) but fail after 4 steps in River Crossing (N=3), suggesting performance correlates more with training data familiarity than inherent problem complexity. Overall, this paper challenges the assumption that LRMs are progressing steadily toward generalizable reasoning. It argues that existing “thinking” enhancements provide local, not scalable, benefits, raising critical questions about inference-time scaling, symbolic reasoning, and robustness of these emerging systems.

One of the biggest barriers to deploying LLM-based agents in real workflows is their poor performance on long-horizon reasoning. Agents often generate coherent short responses but struggle when a task requires planning, tool use, or multi-step decision-making. The issue is not just accuracy at the end, but the inability to reason through the middle. Without knowing which intermediate steps helped or hurt, agents cannot learn to improve. This makes long-horizon reasoning one of the hardest and most unsolved problems for LLM generalization. It is relatively easy for a model to retrieve a document, answer a factual question, or summarize a short email. It is much harder to solve a billing dispute that requires searching, interpreting policy rules, applying edge cases, and adjusting the recommendation based on prior steps. Today’s agents can generate answers, but they often fail to reflect, backtrack, or reconsider earlier assumptions. A new paper from Google DeepMind and Stanford addresses this gap with a method called SWiRL: Step-Wise Reinforcement Learning. Rather than training a model to get the final answer right, SWiRL trains the model to improve each step in a reasoning chain. It does this by generating synthetic multi-step problem-solving traces, scoring every individual step using a reward model (Gemini 1.5 Pro), and fine-tuning the base model to favor higher-quality intermediate steps. This approach fundamentally changes the way we train reasoning agents. Instead of optimizing for final outcomes, the model is updated based on how good each reasoning step was in context. For example, if the model generates a search query or a math step that is useful, even if the final answer is wrong, that step is rewarded and reinforced. Over time, the agent learns not just to answer, but to reason more reliably. This is a major departure from standard RLHF, which only gives feedback at the end. SWiRL improves performance by 9.2 percent on HotPotQA, 16.9 percent on GSM8K when trained on HotPotQA, and 11 to 15 percent on other multi-hop and math datasets like MuSiQue, BeerQA, and CofCA. It generalizes across domains, works without golden labels, and outperforms both supervised fine-tuning and single-step RL methods. The implications are substantial: we can now train models to reason better by scoring and optimizing their intermediate steps. Better reward models, iterative reflection, tool-assisted reasoning, and trajectory-level training will lead to more robust performance in multi-step tasks. This is not about mere performance improvement. It shows how we can begin to train agents not to mimic outputs, but to improve the quality of their thought process. That’s essential if we want to build agents that work through problems, adapt to new tasks, and operate autonomously in open-ended environments.

Large Reasoning Models (LRMs) show promising gains on math and coding benchmarks, but we found their fundamental limitations are more severe than expected. In our latest work, we compared each “thinking” LRM with its “non-thinking” LLM twin. Unlike most prior works that only measure the final performance, we analyzed their actual reasoning traces—looking inside their long "thoughts". Our study reveals several interesting results: 1 - Three distinct performance regimes🟡🔵🔴: Under equal inference-compute budgets, standard LLMs outperform reasoning models on low-complexity tasks, reasoning models excel at medium complexity, and both collapse to zero accuracy on high-complexity tasks. 2 - Counterintuitive scaling limits 🔄: As problems get more difficult, reasoning models initially think more (good!) but then START THINKING LESS despite having plenty of token budget left. They give up right when they should work harder! 3 - Looking inside the "thoughts" 🔍: By replaying every intermediate move in our simulators, our puzzle setup shows that LRM find answers early but then "overthink" on simple problems, eventually reach correct solutions only after exploring wrong paths on medium problems, and completely fail on hard problems. 4 - Catastrophic failure on exact computation ⚠️: Even when given explicit solution algorithms, reasoning models still collapse at the same complexity thresholds—revealing fundamental symbolic manipulation limits and erratic performance across tasks, as shown by Claude 3.7 flawlessly handling ~100 Tower of Hanoi moves yet floundering after just four steps in the River Crossing puzzle. 5 - Scaling compute is helpful, but not enough to close the reasoning gaps 🧠: Our findings challenge assumptions about LRM capabilities. Despite sophisticated self-reflection mechanisms from RL training, our results suggest that these models can't follow algorithm steps and importantly can't generalize algorithmic reasoning beyond certain complexity thresholds. #Paper: https://lnkd.in/g3XJC-cX Work done with my colleagues at Apple: Parshin Shojaee, keivan alizadeh vahid, Maxwell Horton, Samy Bengio, and Mehrdad Farajtabar #AI #MachineLearning #LLM #reasoning

𝗜 𝗵𝗮𝘃𝗲 𝗯𝗲𝗲𝗻 𝗶𝗻 𝘁𝗵𝗲 𝗡𝗟𝗣 𝘀𝗽𝗮𝗰𝗲 𝗳𝗼𝗿 𝗮𝗹𝗺𝗼𝘀𝘁 𝟭𝟬 𝘆𝗲𝗮𝗿𝘀 𝗻𝗼𝘄, and I know the first-hand challenges of building text-based models in the pre-GPT era! So, I am a 𝗽𝗿𝗼-𝗟𝗮𝗿𝗴𝗲 𝗟𝗮𝗻𝗴𝘂𝗮𝗴𝗲 𝗠𝗼𝗱𝗲𝗹 (𝗟𝗟𝗠) 𝗲𝗻𝘁𝗵𝘂𝘀𝗶𝗮𝘀t, but I don’t believe they will replace humans or solve all our problems, especially when it comes to highly complex reasoning in industries like Finance. 𝗧𝗵𝗶𝘀 𝘄𝗲𝗲𝗸𝗲𝗻𝗱, I spent reading two compelling papers, and I’m convinced we’re bumping into real reasoning ceilings: 𝗜> "𝗧𝗵𝗲 𝗜𝗹𝗹𝘂𝘀𝗶𝗼𝗻 𝗼𝗳 𝗧𝗵𝗶𝗻𝗸𝗶𝗻𝗴: 𝗨𝗻𝗱𝗲𝗿𝘀𝘁𝗮𝗻𝗱𝗶𝗻𝗴 𝘁𝗵𝗲 𝗦𝘁𝗿𝗲𝗻𝗴𝘁𝗵𝘀 𝗮𝗻𝗱 𝗟𝗶𝗺𝗶𝘁𝗮𝘁𝗶𝗼𝗻𝘀 𝗼𝗳 𝗥𝗲𝗮𝘀𝗼𝗻𝗶𝗻𝗴 𝗠𝗼𝗱𝗲𝗹𝘀 𝘃𝗶𝗮 𝘁𝗵𝗲 𝗟𝗲𝗻𝘀 𝗼𝗳 𝗣𝗿𝗼𝗯𝗹𝗲𝗺 𝗖𝗼𝗺𝗽𝗹𝗲𝘅𝗶𝘁𝘆" (Apple) Apple researchers rigorously tested 𝗟𝗮𝗿𝗴𝗲 𝗥𝗲𝗮𝘀𝗼𝗻𝗶𝗻𝗴 𝗠𝗼𝗱𝗲𝗹𝘀 (𝗟𝗥𝗠𝘀), LLMs that explicitly generate chain-of-thought reasoning, using controlled puzzles like Tower of Hanoi and River Crossing Key insights: 1. 𝗧𝗵𝗿𝗲𝗲 𝗿𝗲𝗮𝘀𝗼𝗻𝗶𝗻𝗴 𝗿𝗲𝗴𝗶𝗺𝗲𝘀: ▪️Low complexity: standard LLMs outperform LRMs ▪️Medium complexity: LRMs excel ▪️High complexity: 𝗯𝗼𝘁𝗵 𝗰𝗼𝗹𝗹𝗮𝗽𝘀𝗲, accuracy plummets 2. Fascinating observation, 𝗟𝗥𝗠𝘀 “𝗴𝗶𝘃𝗲 𝘂𝗽” as puzzle complexity increases, their reasoning effort declines rapidly, even with enough tokens 3. Even when provided an exact algorithm (e.g., Tower of Hanoi strategy), the 𝗺𝗼𝗱𝗲𝗹𝘀 𝘀𝘁𝗶𝗹𝗹 𝗳𝗮𝗶𝗹𝗲𝗱 𝘁𝗼 𝗴𝗲𝗻𝗲𝗿𝗮𝗹𝗶𝘇𝗲 and mostly outputs based on past observed data pattern it is trained on 𝗜𝗜> "𝗥𝗲𝗮𝘀𝗼𝗻𝗶𝗻𝗴 𝗼𝗿 𝗢𝘃𝗲𝗿𝘁𝗵𝗶𝗻𝗸𝗶𝗻𝗴: 𝗘𝘃𝗮𝗹𝘂𝗮𝘁𝗶𝗻𝗴 𝗟𝗮𝗿𝗴𝗲 𝗟𝗮𝗻𝗴𝘂𝗮𝗴𝗲 𝗠𝗼𝗱𝗲𝗹𝘀 𝗼𝗻 𝗙𝗶𝗻𝗮𝗻𝗰𝗶𝗮𝗹 𝗦𝗲𝗻𝘁𝗶𝗺𝗲𝗻𝘁 𝗔𝗻𝗮𝗹𝘆𝘀𝗶𝘀" (Dimitris Vamvourellis & Dhagash Mehta, Ph.D., BlackRock) This study tested major 𝗟𝗟𝗠𝘀 (𝗚𝗣𝗧‐𝟰𝗼, 𝗚𝗣𝗧‐𝟰.𝟭, 𝗼𝟯‐𝗺𝗶𝗻𝗶, 𝗙𝗶𝗻𝗕𝗘𝗥𝗧 𝘃𝗮𝗿𝗶𝗮𝗻𝘁𝘀) on financial sentiment classification using: - "𝗦𝘆𝘀𝘁𝗲𝗺 𝟭" (𝗳𝗮𝘀𝘁/𝗶𝗻𝘁𝘂𝗶𝘁𝗶𝘃𝗲) - "𝗦𝘆𝘀𝘁𝗲𝗺𝟮" (𝘀𝗹𝗼𝘄/𝗱𝗲𝗹𝗶𝗯𝗲𝗿𝗮𝘁𝗲) 𝗽𝗿𝗼𝗺𝗽𝘁𝗶𝗻𝗴 Key takeaways: ▪️𝗥𝗲𝗮𝘀𝗼𝗻𝗶𝗻𝗴 𝗽𝗿𝗼𝗺𝗽𝘁𝘀 𝗱𝗶𝗱 𝗻𝗼𝘁 𝗶𝗺𝗽𝗿𝗼𝘃𝗲 𝗽𝗲𝗿𝗳𝗼𝗿𝗺𝗮𝗻𝗰𝗲 ▪️Surprisingly, straightforward, intuitive prompts with GPT-4o (no chain-of-thought) outperformed all others  ▪️More reasoning led to overthinking, reducing alignment with human-labeled sentiments 💡 Why it matters for builders and researchers in Finance and every industry: ❎ 𝗕𝗶𝗴𝗴𝗲𝗿 𝗺𝗼𝗱𝗲𝗹𝘀 + 𝗺𝗼𝗿𝗲 “𝘁𝗵𝗶𝗻𝗸𝗶𝗻𝗴” = 𝗯𝗲𝘁𝘁𝗲𝗿 𝗼𝘂𝘁𝗰𝗼𝗺𝗲𝘀. Sometimes it’s actively worse ❎ We’re not seeing a soft plateau — these are 𝗵𝗮𝗿𝗱 𝗰𝗲𝗶𝗹𝗶𝗻𝗴𝘀 𝗶𝗻 𝗿𝗲𝗮𝘀𝗼𝗻𝗶𝗻𝗴 𝗰𝗮𝗽𝗮𝗰𝗶𝘁𝘆 ❎ For real-world systems, agents, and financial tools: design for 𝗿𝗲𝗮𝘀𝗼𝗻𝗶𝗻𝗴 𝗲𝗰𝗼𝗻𝗼𝗺𝘆, not just reasoning depth. #LLMs #ReasoningLimits #LLMChainofthought #LLMReasoningDecline

🧠 When Reasoning Fails: Why More Steps Can Hurt LLMs' Inductive Abilities It’s often assumed that giving large language models more reasoning steps—via Chain-of-Thought (CoT) prompting—leads to better outcomes. But in our latest study, we found the opposite. 📄 Reasoning Can Hurt the Inductive Abilities of Large Language Models https://lnkd.in/gYghxWYz We designed a suite of diagnostic tasks—chess, poker, dice, and blackjack—where each game followed hidden rules unknown to the model. No labels. No prior rulebook. Just transcripts. 💡 The task: infer latent rules from sparse examples—a test of inductive reasoning, not memorization. Despite the hype around “reasoning-enabled” LLMs, our findings show: ❌ Reasoning often degrades inductive accuracy. ✅ Non-reasoning models like GPT-4o outperform reasoning-heavy ones like GPT-o3. Why? We developed a formal model of multi-step inference and found that reasoning introduces three compounding failure modes: Breakdown Error – the model asks the wrong sub-question. Solving Error – even a good sub-task is answered noisily. Summary Error – the model doesn’t know when to stop. Together, these create a U-shaped tradeoff: more steps help—until they don’t. 🛠️ To fix this, we propose interventions at each failure point: – Structured decomposition – Solving constraints that avoid math overuse – Token-based stopping rules These fixes consistently improve performance, across all games, without retraining. 📌 Implication: Reasoning isn’t magic. It’s powerful only when aligned, constrained, and intentional. — If you're working on reasoning agents, symbolic abstraction, or test-time prompting strategies, I’d love to connect. #LLMs #Reasoning #AgenticAI #InductiveBias #TrustworthyAI #ComputationalCognition #LanguageModels #AIresearch

This paper has gained popularity in tech circles lately and for a good reason. It goes against the narrative that LLMs keep getting smarter with scale. Published by Apple researchers, it hits on something many have suspected: Most benchmarks might be gamed, and we need a better way to measure actual intelligence. The study investigates Large Reasoning Models (LRMs), those models that generate long Chain-of-Thought (CoT) outputs and appear to “think.” They perform well on popular benchmarks, but crack under real complexity. The paper asked: - Do “thinking” models actually reason, or are they just verbose? - How does their reasoning evolve with problem difficulty? - Do they use their token budget wisely? - Can they generalize logic, or are they still just pattern matches? How they tested it: - Instead of math problems (which often suffer from data contamination), they used puzzle environments like the Tower of Hanoi and River Crossing, where complexity can be precisely controlled and evaluated. They compared: - “Thinking” models (Claude 3.7 Thinking, DeepSeek-R1) vs. Standard, non-thinking LLMs. And crucially, they didn’t just look at final answers, they analyzed every reasoning step. Key findings: 1. Three degrees of complexity - Low Complexity: Non-thinking models outperform. They’re faster and more accurate. - Medium Complexity: LRMs show their strength, more reasoning helps. - High Complexity: Both collapse. Even with ample token budgets, no correct answers emerge. 2. Reasoning effort doesn’t scale with difficulty - As problems get harder, models initially increase reasoning effort. But beyond a threshold, they start thinking less, even though they have tokens to spare. This suggests a fundamental scaling limit in current reasoning architectures. 3. Overthinking On simple problems, LRMs often find the right answer early, then keep “thinking” and talk themselves out of it. Even when given a correct algorithm, they still fail at execution as the complexity increases. This isn’t reasoning. It’s a failure to follow steps. Why this matters: The current generation of LRMs with CoT and self-reflection are not general reasoners. Their performance holds only in narrow complexity bands, and more tokens or better prompts don’t fix it. To move forward, we likely need new architectures, not just more scale. My key takeaways as someone building with AI: - Use lightweight LLMs for simple tasks. Don’t pay the “thinking tax” unless there's clear benefit. Non-CoT models are often faster and more accurate for straightforward problems. - Test across a difficulty spectrum. Don’t just benchmark your product on average cases, include edge cases at both ends. Know where your model breaks and set guardrails accordingly. - Monitor reasoning effort. If your agent starts giving quicker answers to harder questions, that’s a red flag. Explore ways to enforce more consistent “thinking” through RL or prompt design.

What can we learn from Apple's recent "Illusion of Thinking" paper on reasoning tasks? Apple’s recent paper (Shojaee et al., "The Illusion of Thinking") explores Large Reasoning Models (LRMs)—specialized LLMs built to reason step-by-step. The authors identify three distinct regimes that significantly impact model performance. While these regimes aren't easily defined solely by task characteristics, we can recognize them through clear symptoms in LRMs, such as overthinking or prematurely giving up. 🟢 Low-Complexity Regime: Symptoms: Surprisingly, standard LLMs outperform LRMs, which tend to "overthink" by exploring unnecessary paths and missing straightforward solutions. Strategy: Opt for LLMs to prevent needless complexity. 🟠 Medium-Complexity Regime: Symptoms: LRMs outperform standard LLMs by reasoning step-by-step to reach accurate solutions. Strategy: Improve outcomes by regularly validating intermediate solutions, possibly using another LLM or alternative evaluation methods. 🔴 High-Complexity Regime: Symptoms: Both LRMs and LLMs perform poorly, with LRMs paradoxically giving up prematurely as complexity grows. Strategy: Simplify problems by breaking them down into smaller, more manageable components. Have you encountered these phenomena in your projects? I'd love to hear about your experiences and how you've tackled these challenges.