Multimodal Reasoning Techniques

Exciting New Research on Multimodal Retrieval-Augmented Generation (MRAG)! I just finished reading a fascinating survey paper on Multimodal Retrieval-Augmented Generation (MRAG) from researchers at Huawei Cloud. This cutting-edge technology represents a significant advancement in enhancing large language models by integrating multimodal data like text, images, and videos into both retrieval and generation processes. Traditional Retrieval-Augmented Generation (RAG) systems primarily rely on textual data, which limits their ability to leverage rich contextual information available in multimodal sources. MRAG addresses this limitation by extending the RAG framework to include multimodal retrieval and generation, enabling more comprehensive and contextually relevant responses. The paper outlines the evolution of MRAG through three distinct stages: >> MRAG1.0 ("Pseudo-MRAG") This initial stage extended RAG by converting multimodal data into textual representations. The architecture consisted of three key components: - Document Parsing and Indexing: Processing multimodal documents using OCR and specialized models to generate captions for images and videos - Retrieval: Using vector embeddings to find relevant information - Generation: Synthesizing responses using LLMs While effective, this approach suffered from information loss during modality conversion and retrieval bottlenecks. >> MRAG2.0 ("True Multimodal") This stage preserved original multimodal data within the knowledge base and leveraged Multimodal Large Language Models (MLLMs) for direct processing. Key improvements included: - Using unified MLLMs for captioning instead of separate models - Supporting cross-modal retrieval to minimize data loss - Employing MLLMs for generation to directly process multimodal inputs >> MRAG3.0 (Advanced Integration) The latest evolution introduces: - Enhanced document parsing that retains document screenshots to minimize information loss - Multimodal Search Planning that optimizes retrieval strategies through retrieval classification and query reformulation - Multimodal output capabilities that combine text with images, videos, or other modalities in responses The technical architecture includes sophisticated components like multimodal retrievers (using single/dual-stream and generative structures), rerankers (fine-tuning or prompting-based), and refiners (hard or soft prompt methods) to optimize the information flow. What's particularly impressive is how MRAG outperforms traditional text-modal RAG in scenarios where both visual and textual information are critical for understanding and responding to queries. The researchers have systematically analyzed essential components, datasets, evaluation methods, and current limitations to provide a comprehensive understanding of this promising paradigm.

VLMs are not enough for healthcare. LLMs changed how we reason with text. VLMs brought vision into the loop. Healthcare is multimodal by nature. Voice, imaging, ECGs, EHRs, genomics, no doctor looks at just one. That is why LMMs - Large Multimodal Models are the real breakthrough. How they work: → Each modality (voice, image, time-series, text) is encoded into embeddings. → These embeddings are projected into a shared latent space. → Cross-attention layers force the model to align across modalities, not just co-locate them. → Training combines contrastive alignment (match cough to X-ray), generative prediction (fill missing labs from history), and multimodal instruction tuning (answer prompts that mix text + imaging + signals). Why they are better: LLMs: great at reasoning with text, but blind to medical signals like coughs or ECGs. VLMs: can align images and text, but brittle when asked to integrate time-series, genomics, or audio. LMMs: unify every modality into one reasoning pipeline, enabling predictions grounded in the full patient context. Practical impact: → Alzheimer’s detection from subtle voice changes, validated against neuro exams. → Respiratory diagnosis by fusing cough audio with chest X-rays and oxygen levels. → Cardiology risk prediction by integrating heartbeat audio, ECGs, and longitudinal vitals. This is why models like MedGemma from Google are so exciting. It is one of the first open-source medical multimodal models. It shows that unifying diverse data types, not just scaling text is the way forward. At Frekil (YC X25), we are building the data backbone that makes LMMs truly powerful: clean, multimodal, longitudinal patient datasets. LLMs were step one. VLMs were step two. But LMMs trained on aligned medical data are the leap that will make healthcare AI clinically reliable and globally impactful.

🧠 Pixels Aren’t Enough – The Case for Truly Multimodal EO Models Satellite imagery shows us the surface. But if we want to understand why something is happening—or predict what’s next—we need more than pixels. What multimodal models add beyond imagery: 🗂️ Textual labels – land cover types, hazard annotations, infrastructure info 📊 Tabular data – crop yields, emissions inventories, population density 🗺️ Geographic priors – soil maps, elevation bands, administrative boundaries 🌦️ Weather data – reanalysis and forecasts for temperature, precipitation, wind Each of these brings context that satellite imagery alone can’t provide. They anchor image patterns in meaning—physical, social, and policy-relevant. For instance, combining imagery with crop yield stats and precipitation data can help assess food security more accurately than pixels alone. 🧭 Why this matters: - Enables semantic alignment—connecting image patterns to real-world categories - Supports advanced tasks like language-based retrieval, scenario analysis, or causal inference - Powers vision-language models that respond to prompts like “find expanding agricultural zones near wetlands” - Enables EO agents—models that combine perception, reasoning, and retrieval to assist analysts like intelligent copilots - Improves interpretability, especially in decision-making contexts (e.g., urban planning, disaster response) ⚠️ Design challenges to watch out for: - Misalignment in scale or timing across modalities - Models learning to cheat with metadata (e.g., associating labels with regions) - Loss of transparency when multiple data sources are stacked blindly But adding more data isn’t always better—it depends on what the model is learning, and why. 📌 Multimodal EO models bring us closer to real-world insight—but only when the added signals are relevant to the task, aligned in time and space, and tied to the real-world outcomes we care about. 👇 Have you tried adding tabular, textual, or weather data to your EO models? What did it unlock—or complicate? #EarthObservation #MultimodalLearning #GeospatialAI #RemoteSensing #FoundationModels #DataFusion #AIForClimate #AIForGood #VisionLanguageModels #EOAgents #CausalAI — Subscribe to 𝘊𝘰𝘮𝘱𝘶𝘵𝘦𝘳 𝘝𝘪𝘴𝘪𝘰𝘯 𝘐𝘯𝘴𝘪𝘨𝘩𝘵𝘴 — weekly briefings on making vision AI work in the real world → Click "View my newsletter" under my name above

Critic-V: framework to enhance feedback quality in visual perception and reasoning processes of Vision-Language Models (VLMs) VLMs often generate inaccurate or irrelevant responses due to issues like hallucinated image understandings or unrefined reasoning paths. To address it, this paper introduce Critic-V, a novel framework inspired by the Actor-Critic paradigm to boost the reasoning capability of VLMs. This framework decouples the reasoning process and critic process by integrating two independent components: the Reasoner, which generates reasoning paths based on visual and textual inputs, and the Critic, which provides constructive critique to refine these paths. 𝗞𝗲𝘆 𝗰𝗼𝗻𝘁𝗿𝗶𝗯𝘂𝘁𝗶𝗼𝗻𝘀: - propose a Reasoner-Critic framework that can integrate seamlessly with existing VLMs, significantly improving their performance in multimodal reasoning tasks - introduce a comprehensive dataset including 29,012 multimodal question-answer pairs with critiques generated by VEST and ranked by Rule-based Reward (RBR) 𝗠𝗲𝘁𝗵𝗼𝗱 i) Reasoner - responsible for generating reasoning actions a based on the current state, typically via a policy function - reasoner’s policy update is no longer based solely on traditional parameterization but instead on the evolution of text prompt - uses TextGrad framework for computing gradients of text-based policies ii) Critic Model - serves a crucial role in providing evaluative feedback on the reasoning and generation processes of the model - offers natural language feedback that is more nuanced and context-sensitive - for more useful feedback, shifted from scalar rewards fashion of policy gradient to preference-based training via DPO - To generate preference data for training the Critic with DPO, applied vision error insertion technique (VEST) to question-image pairs from VQA datasets - leveraged a Rule-based Reward (RBR) mechanism to evaluate quality of critiques to construct preference relationship iii) Reasoner-Critic Framework - aims to improve Reasoner’s output by utilizing feedback from Critic to guide its adjustments - process begins with Reasoner generating an initial response to a given query based on the input prompt - Critic then evaluates response and provides feedback in form of a critique - Reasoner then revises its response based on the Critic’s suggestions - This cycle continues until a predefined maximum number of iterations is reached, or until the Critic determines that the Reasoner’s response meets a satisfactory level of quality. 𝗥𝗲𝘀𝘂𝗹𝘁𝘀 - Critic-V framework significantly outperforms existing methods, including GPT-4V, on 5 out of 8 benchmarks, especially regarding reasoning accuracy and efficiency 𝗣𝗮𝗽𝗲𝗿: https://lnkd.in/eEvaMETU 𝗧𝗿𝗮𝗶𝗻𝗶𝗻𝗴 𝗱𝗮𝘁𝗮𝘀𝗲𝘁: https://lnkd.in/evMwZ6dz

#ALERT Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Models This is another huge review of all multimodal reasoning models. it is absolutely extensive and includes models from the last 5 years and that revolution. it's a must read. ➡️ LMRMs integrate text, image, audio, and video to support complex reasoning abilities in AI systems operating in diverse environments. ➡️ The field has evolved through a four-stage roadmap: Perception-Driven Modular Reasoning, Language-Centric Short Reasoning (System-1), Language-Centric Long Reasoning (System-2), and Native Large Multimodal Reasoning. ➡️ Early efforts relied on task-specific modules, with reasoning implicitly embedded in processing pipelines. ➡️ Recent advancements leverage Multimodal Large Language Models (MLLMs) and techniques like Multimodal Chain-of-Thought (MCoT) for more explicit and structured reasoning. ➡️ Significant challenges persist in achieving omni-modal generalization, sufficient reasoning depth, and robust agentic behavior for real-world applications. This survey provides a structured overview of Large Multimodal Reasoning Models (LMRMs), analyzing their historical development from modular systems to unified, language-centric architectures. The paper highlights the progression through stages of increasing complexity in reasoning capabilities. Despite progress in using MLLMs and explicit reasoning chains like MCoT, current models show limitations in handling all modalities simultaneously and performing deep, long-horizon reasoning required for complex tasks and adaptive agent interactions. The work introduces Native Large Multimodal Reasoning Models (N-LMRMs) as a prospective direction. These models aim to move beyond retrofitting capabilities onto language models, focusing instead on inherently integrating omnimodal perception, understanding, generation, and agentic reasoning for enhanced scalability and real-world adaptability. #LMRMs #MultimodalAI #AIReasoning #FoundationModels

#DeepSeek has released some of the best OSS LLMs. But they’re blind! What if we could turn any text-only LLM into a competitive multimodal reasoner without modifying its parameters? (Fig 1) Excited to introduce #BeMyEyes: a modular multi-agent framework that extends LLMs to new modalities by letting them collaborate with lightweight vision perceivers. Core idea💡: decouple perception from reasoning. (Fig 2) ▸A lightweight VLM acts as the eyes: it inspects images and sends textual evidence. ▸A powerful frozen LLM acts as the brain: it reasons over that evidence. They talk via natural-language multi-turn conversation, so the LLM can ask follow-ups and the perceiver replies with targeted visual details. ✓ No joint multimodal training. No architecture changes. Just agents collaborating through text, and you can swap in better eyes/brains anytime. 1️⃣Results (Table 1): equipping DeepSeek-R1 with a small perceiver (e.g., #Qwen2.5-VL-7B) unlocks strong multimodal reasoning. BeMyEyes + DeepSeek-R1 reaches #MMMU-Pro 57.2, #MathVista 72.7, #MathVision 48.5, which are competitive with, and often surpassing, large proprietary VLMs like GPT-4o on knowledge-intensive benchmarks. 2️⃣BeMyEyes generalizes to different model families. (Table 2) Beside DeepSeek-R1 and Qwen-VL, we consider different model families: #InternVL3-8B as the eyes, and #GPT-4, a large proprietary LLM, as the brain. Our BeMyEyes framework also shows very strong results. 3️⃣BeMyEyes generalizes to specialized domains. (Table 3) With a medical perceiver (Lingshu-7B), the same framework stays competitive on multimodal medical reasoning benchmarks, without needing to rebuild or retrain a giant medical VLM from scratch. ⭐Takeaway: multimodality doesn’t have to mean a monolithic unified model. A scalable alternative is agent collaboration through language: small, adaptable “eyes” + strong “brains,” improving independently over time. Blog: https://lnkd.in/gkBV_QdD Paper: https://lnkd.in/g-SAykZT Joint work by James Yipeng Huang, Sheng Zhang, Qianchu (Flora) Liu, Guanghui Qin, Tinghui Zhu, Tristan Naumann, Muhao Chen, Hoifung Poon

I spent this weekend breaking down Google DeepMind's Gemini 3 pro. Here are my big takeaways on multi modal AI, especially on how it relates to building AI for healthcare. Most of the information that drives real decisions today does not sit in paragraphs. It lives in charts, tables, diagrams, forms, dashboards, and mixed media that combine numbers, shapes, colors, and text. These complex visuals are how people in finance, healthcare, manufacturing, logistics, and research make sense of the world. For AI systems to be truly useful, they must do more than look at a picture of a document. They must understand it. They must track relationships inside a table, follow the logic implied by arrows in a diagram, compare two lines on a chart, and reconcile the visual story with the written one. This is the difference between extraction and reasoning, and it is becoming one of the most important challenges in multimodal AI. Rohan Doshi points out that complex visuals contain subtle cues humans notice automatically a tiny sub segment in a pie chart, a nested row in a table, a faint trend line in a plot, a color code that changes meaning across pages. These details matter because they change how a decision maker interprets the information. A table means nothing without the labels. A chart means nothing without the legend. A diagram means nothing without understanding how the arrows relate. The intelligence is in the structure. Especially in healthcare. People often assume that medical AI is about detecting objects. In reality, the hard part is reasoning across complex visuals. Radiologists never rely on a single slice or a single feature. They look across dozens of images, compare patterns, integrate anatomy with patient history, and draw conclusions from the relationships between visual elements. The meaning emerges from context. At Turmerik, we have worked with the documents that power clinical research and patient care. Protocols filled with diagrams. Lab panels packed with nested tables. Imaging reports that mix visuals and prose. These documents slow down clinicians because they are long AND because their visuals require deep interpretation. Most work today focuses on whether a model can answer a question correctly. That is important, but it is only the foundation. The real potential lies in models that can check whether a chart contradicts the text flag surprising outliers explain why two visuals tell different stories or guide a user toward the most important pattern even if they did not know to ask. #MultimodalAI #VisualReasoning #MedicalAI #AIinHealthcare #DocumentAI #FrontierAI

Scientific research demands sophisticated reasoning over multimodal data, a challenge especially prevalent in biology. Despite recent advances in multimodal large language models (MLLMs) for AI-assisted research, existing multimodal reasoning benchmarks only target up to college-level difficulty, while research-level benchmarks emphasize lower-level perception, falling short of the complex multimodal reasoning needed for scientific discovery. To bridge this gap, we introduce MicroVQA, a visual-question answering (VQA) benchmark designed to assess three reasoning capabilities vital in research workflows: expert image understanding, hypothesis generation, and experiment proposal. MicroVQA consists of 1,042 multiple-choice questions (MCQs) curated by biology experts across diverse microscopy modalities, ensuring VQA samples represent real scientific practice. In constructing the benchmark, we find that standard MCQ generation methods induce language shortcuts, motivating a new two-stage pipeline: an optimized LLM prompt structures question-answer pairs into MCQs; then, an agent-based `RefineBot' updates them to remove shortcuts. Benchmarking on state-of-the-art MLLMs reveal a peak performance of 53\%; models with smaller LLMs only slightly underperform top models, suggesting that language-based reasoning is less challenging than multimodal reasoning; and tuning with scientific articles enhances performance. Expert analysis of chain-of-thought responses shows that perception errors are the most frequent, followed by knowledge errors and then overgeneralization errors. These insights highlight the challenges in multimodal scientific reasoning, showing MicroVQA is a valuable resource advancing AI-driven biomedical research. MicroVQA is available at https://lnkd.in/egsPaVG7, and project page at https://lnkd.in/eQQkKaKD. Interesting preprint publication by James Burgess and larger teams at Stanford University, Tsinghua University, University of North Carolina at Chapel Hill, Princeton University, KTH Royal Institute of Technology, Chan Zuckerberg Biohub Network Link to full paper: https://lnkd.in/ewzSgMWt

How AI Models Process Text and Images Together? 👀 Attention across multimodality in models is all about how models combine and reason over different kinds of data like text, images, audio, or video inside the transformer architecture. Since transformers were originally built for text, multimodal extensions had to figure out how to align and fuse very different inputs. Broadly, there are two main strategies: 1. Unified Embedding (Concatenation) Approach 👉 Each modality (say, text and image) is first turned into embeddings of the same size.  👉 So while text tokens represent words/subwords, image tokens represent visual patches/features. Both end up as vectors of the same dimension. 👉 Once both are in the same embedding space, the model treats them like one long sequence and processes them with standard self-attention. 👉 Attention doesn’t care if a token came from text or image, it just looks at relationships across the whole sequence. Advantage: Simple, minimal architectural changes. Limitation: Long image sequences blow up the context window and can be computationally heavy. Example: Fuyu, Molmo, Pixtral. 2. Cross-Modality Attention Approach 👉 Instead of dumping all embeddings into one sequence, each modality is encoded separately. 👉 The text sequence is processed normally, but at certain layers, it can “look at” image features via cross-attention: - Queries (Q) come from one modality, say, text. - Keys (K) and Values (V) come from another modality, say, image. 👉 This way, text tokens can selectively pull in relevant visual information without being flooded by raw image tokens. Advantage: More efficient, keeps text-only performance intact. Limitation: Requires modifying the LLM architecture. Example: LLaMA 3.2 multimodal, NVLM-X, Aria. #MultiModality #LLM #AI