Multimodal Language Generation Techniques

Exciting New Research on Multimodal Retrieval-Augmented Generation (MRAG)! I just finished reading a fascinating survey paper on Multimodal Retrieval-Augmented Generation (MRAG) from researchers at Huawei Cloud. This cutting-edge technology represents a significant advancement in enhancing large language models by integrating multimodal data like text, images, and videos into both retrieval and generation processes. Traditional Retrieval-Augmented Generation (RAG) systems primarily rely on textual data, which limits their ability to leverage rich contextual information available in multimodal sources. MRAG addresses this limitation by extending the RAG framework to include multimodal retrieval and generation, enabling more comprehensive and contextually relevant responses. The paper outlines the evolution of MRAG through three distinct stages: >> MRAG1.0 ("Pseudo-MRAG") This initial stage extended RAG by converting multimodal data into textual representations. The architecture consisted of three key components: - Document Parsing and Indexing: Processing multimodal documents using OCR and specialized models to generate captions for images and videos - Retrieval: Using vector embeddings to find relevant information - Generation: Synthesizing responses using LLMs While effective, this approach suffered from information loss during modality conversion and retrieval bottlenecks. >> MRAG2.0 ("True Multimodal") This stage preserved original multimodal data within the knowledge base and leveraged Multimodal Large Language Models (MLLMs) for direct processing. Key improvements included: - Using unified MLLMs for captioning instead of separate models - Supporting cross-modal retrieval to minimize data loss - Employing MLLMs for generation to directly process multimodal inputs >> MRAG3.0 (Advanced Integration) The latest evolution introduces: - Enhanced document parsing that retains document screenshots to minimize information loss - Multimodal Search Planning that optimizes retrieval strategies through retrieval classification and query reformulation - Multimodal output capabilities that combine text with images, videos, or other modalities in responses The technical architecture includes sophisticated components like multimodal retrievers (using single/dual-stream and generative structures), rerankers (fine-tuning or prompting-based), and refiners (hard or soft prompt methods) to optimize the information flow. What's particularly impressive is how MRAG outperforms traditional text-modal RAG in scenarios where both visual and textual information are critical for understanding and responding to queries. The researchers have systematically analyzed essential components, datasets, evaluation methods, and current limitations to provide a comprehensive understanding of this promising paradigm.

Cross-attention is a fundamental idea that is heavily used by multi-modal LLMs. Let’s learn how it works from the ground up… The original transformer architecture has two components: an encoder and a decoder. The encoder and decoder contain repeated blocks of: 1. Self-attention: transforms each token vector based on the other tokens that are present in the sequence. 2. Feed-forward transformation: transforms each token vector individually. Basics of self-attention. Given a list of token vectors as input, self-attention transforms these vectors by considering all other tokens in the sequence. To do this, self-attention creates three (linear) projections of our input–the keys, queries and values. We use the keys and queries to compute an attention score between every pair of tokens in the sequence. Then, we multiply these attention scores by the value vectors to obtain the final output. Transformer cross-attention. In addition to self-attention and a feed-forward transformation, the transformer’s decoder has an extra cross-attention module in each of its blocks. Whereas self-attention computes attention over the tokens in a single sequence, cross-attention considers two sequences of tokens—the tokens from the encoder and the tokens from the decoder—and computes attention between the tokens of these two sequences. This allows the decoder to consider the encoder’s representations when generating output! How does cross-attention work? Cross-attention is not much different than self-attention. The key difference is how we compute the key, query and value matrices. Instead of computing all three of these matrices by linearly projecting a single sequence of token vectors, we linearly project two different sequences of vectors. The first sequence produces the queries, while the second sequence produces the keys and values. As a result, our attention matrix contains all pairwise attention scores between tokens in the first and second sequence. Application to multi-modal LLMs. Cross-attention is used constantly in multi-modal LLM research. We can use cross-attention to fuse representations of images–or other modalities like speech as long as they are represented as a sequence of vectors–produced by a vision model (e.g., CLIP is a very common image encoder for this purpose) into a text-based LLM. In other words, we incorporate visual information into an LLM as it generates its output, allowing the model to ingest and interpret images (or other modalities of data) as input in addition to just text!

How far is an LLM from not only understanding but also generating visually? Our new work shows not very far! Introducing MetaMorph---a multimodal understanding and generation model. MetaMorph only needs very moderate generation data to elicit visual generation from an LLM, when trained together with visual understanding. We propose Visual-Predictive Instruction Tuning (VPiT), a simple extension of visual instruction tuning. It tunes the LLM to predict both discrete text tokens and continuous visual tokens, purely autoregressively. Two central findings about this unified autoregressive framework: 1) When trained jointly with visual understanding, the model quickly gains visual generation ability, with as little as 200K data points on generation. 2) Visual understanding and visual generation benefit each other, but visual understanding data contributes much more to both abilities. MetaMorph, trained with VPiT on LLaMA 3.1-8B, is competitive on both visual understanding and generation tasks. MetaMorph has LLM's knowledge and semantic understanding, and also possesses certain reasoning capabilities. This work was led by our amazing intern Peter Tong, and jointly with David Fan, Jiachen Zhu, Yunyang Xiong, Xinlei Chen, Koustuv Sinha, Michael Rabbat, Yann LeCun, and Saining Xie "MetaMorph: Multimodal Understanding and Generation with Instruction Tuning" arXiv: arxiv.org/abs/2412.14164 Project page: https://lnkd.in/gWDDp7Sh

Apple announces MM1 Methods, Analysis & Insights from Multimodal LLM Pre-training In this work, we discuss building performant Multimodal Large Language Models (MLLMs). In particular, we study the importance of various architecture components and data choices. Through careful and comprehensive ablations of the image encoder, the vision language connector, and various pre-training data choices, we identified several crucial design lessons. For example, we demonstrate that for large-scale multimodal pre-training using a careful mix of image-caption, interleaved image-text, and text-only data is crucial for achieving state-of-the-art (SOTA) few-shot results across multiple benchmarks, compared to other published pre-training results. Further, we show that the image encoder together with image resolution and the image token count has substantial impact, while the vision-language connector design is of comparatively negligible importance. By scaling up the presented recipe, we build MM1, a family of multimodal models up to 30B parameters, consisting of both dense models and mixture-of-experts (MoE) variants, that are SOTA in pre-training metrics and achieve competitive performance after supervised fine-tuning on a range of established multimodal benchmarks. Thanks to large-scale pre-training, MM1 enjoys appealing properties such as enhanced in-context learning, and multi-image reasoning, enabling few-shot chain-of-thought prompting.

MAmmoTH-VL: method to create large-scale multimodal instruction-tuning dataset with intermediate rationales to elicit CoT reasoning in MLLMs. This paper introduces a streamlined yet scalable approach to enhancing MLLM performance through the strategic use of open-source models to generate diverse, high-quality training data that reflects human preferences and real-world complexity. 𝗞𝗲𝘆 𝗰𝗼𝗻𝘁𝗿𝗶𝗯𝘂𝘁𝗶𝗼𝗻𝘀: - introduce a simple, scalable, and cost-effective methodology for constructing instruction-tuning datasets at scale designed to elicit multimodal CoT reasoning - develop a dataset comprising 12 million entries using only open-weight LLMs and MLLMs, named as MAmmoTH-VL-Instruct-12M - trained an MLLM model, MAmmoTH-VL-8B, based on the LLaVA-OneVision architecture, using the 12M dataset created with this approach 𝗠𝗲𝘁𝗵𝗼𝗱 introduce a simple, scalable, and cost-effective data generation pipeline that produces 12 million high-quality samples, involves three key steps: i) Dataset Collection and Categorization - sourced data from 153 publicly available multimodal instruction datasets - reorganized the training data into ten major categories: General, OCR, Chart, Caption, Domain-specific, Code&Math, Language, Detection, Multi-Image, and Video - conducted an initial manual screening of the 153 data sources, by randomly sampling 1,000 data points and a subsequent rapid evaluation to gauge overall quality ii) Instruction Data Rewriting - approach involves transforming the original multimodal data into diverse instruction-response pairs enriched with detailed rationales - for this transformation, designed custom prompts to generate responses that align with real-world applications while encouraging critical thinking and reasoning - results in a rich dataset of instruction-response pairs, characterized by detailed rationales and diverse real-world scenarios iii) Self-data Filtering - following the "Model-as-a-judge" paradigm, leveraged InternVL2-Llama3-76B model, the same model used during data rewriting process, to evaluate logical consistency of each question-answer pair against the corresponding image 𝗘𝘅𝗽𝗲𝗿𝗶𝗺𝗲𝗻𝘁𝗮𝗹 𝗥𝗲𝘀𝘂𝗹𝘁𝘀 - MAmmoTH-VL-8B trained with 12M dataset created with this approach significantly improves reasoning capabilities, achieving state-of-the-art performance on benchmarks such as MathVerse(+8.1%), MMMU-Pro (+7%), and MuirBench(+13.3%). - Also, model demonstrates notable improvements of up to 4% on non-reasoning-based benchmarks 𝗣𝗮𝗽𝗲𝗿: https://lnkd.in/eQB2ZkNm 𝗖𝗼𝗱𝗲: https://lnkd.in/eRcKewHU 𝗗𝗮𝘁𝗮𝘀𝗲𝘁: https://lnkd.in/eig--Fk7 𝗠𝗼𝗱𝗲𝗹: https://lnkd.in/e5Etx_mE

🧐 Multimodal with just fine-tuning? ByteDance makes it happen with VoRA. The new Vision as LoRA (VoRA) paper introduces a bold, streamlined approach to building multimodal LLMs—no vision encoder, no connector, no architecture bloat. Just LoRA. Instead of bolting a vision tower onto an LLM, VoRA injects vision understanding directly into the LLM using Low-Rank Adaptation (LoRA). That means: 🔹 No new inference overhead- LoRA layers are merged into the LLM after training. 🔹 Frozen base LLM- Only LoRA + visual embeddings (~6M params) are trained, preserving language ability and ensuring stability. 🔹 Image inputs at native resolution- No resizing, no tiling hacks—VoRA leverages the LLM’s flexible token handling. 🔹 Bidirectional attention for vision- Instead of using causal masks across all tokens, VoRA allows vision tokens to attend freely—boosting context modeling. To teach the LLM visual features: 🔹 VoRA uses block-wise distillation from a pretrained ViT, aligning intermediate hidden states across layers. This improves visual alignment while keeping the LLM’s core untouched. 🔹 The training objective combines distillation loss (cosine similarity between ViT + LLM visual features) and standard language modeling loss over image-caption pairs. What does this get you? 🔹 A modality-agnostic architecture ready for extension to audio, point clouds, and beyond. This might be one of the most efficient takes yet on vision-language modeling. Excited to see how this evolves. 📄 Paper + Code: https://lnkd.in/gwZr33Vj Follow Aman Chadha and I for more updates!

Excited to share our latest work on multimodal pre-training MoMa! When working with early fusion mixed modal models (natively multimodal  in/out), Chameleon showed the text LLM architecture (e.g. Llama) doesn’t scale to early fusion. While Chameleon proved the significant quality and efficiency benefits of early fusion models, there is a core issue with the inherent differences in information across text and image tokens being treated uniformly. We propose MoMa (Mixture of Modality-aware experts), a novel adaptive architecture that explores adaptivity across modality, width, and depth, delivering up to 4x FLOPs savings (4x smaller or 4x cheaper for the same quality)! Building on the success of MoE in Text LLMs (e.g. Mixtral, Deepseek-MoE,  Grok), MoMa leverages a MoE (mixture-of-experts) block for each Modality showing for early fusion LLMs this is a much more efficient approach compared to dense or MoE alternatives. We further show empirical scaling of MoMa, extensions with MoD (mixture-of-depths), and Upcycling to propose our core training recipe. MoMa is our first step in re-thinking the core architecture primitives when building multimodal early fusion LLMs, we believe there’s a lot more potential in further exploring adaptive compute. This is joint work with amazing co-first authors Victoria Lin, Armen Aghajanyan and co-authors – Liang Luo, Srini Iyer, Mike Lewis, Gargi Ghosh and Luke Zettlemoyer. Paper: https://lnkd.in/gvKBj8SQ Twitter: https://lnkd.in/gj8TG7iR

🚀 Fine-Tuning Multimodal Large Language Models (MLLMs): A Deep Dive into Efficiency 🌐 Adapting large models to specific multimodal tasks has become more efficient with Parameter-Efficient Fine-Tuning (PEFT) techniques like LoRA, QLoRA, and LLM-Adapters. Here's how these approaches are driving innovation: ✅ LoRA: Reduces parameter requirements using matrix factorization, enabling models to adapt with fewer resources while maintaining performance. ✅ LLM-Adapters: Introduce modular, task-specific tuners for flexibility and precision in fine-tuning. 🔍 Advanced Techniques in Fine-Tuning: DyLoRA: Dynamically prioritizes representations during training for smarter updates. LoRA-FA: Freezes specific matrix components to optimize efficiency. Efficient Attention Skipping (EAS): Lowers computational costs of attention mechanisms without sacrificing accuracy. MemVP: Accelerates training and inference by integrating visual prompts, allowing models to seamlessly combine image data into their processing pipelines. These advancements make fine-tuning large language models faster, more resource-efficient, and better equipped for complex multimodal tasks. By leveraging innovations like EAS and MemVP, we're not only streamlining training but also enhancing the model's ability to process and utilize visual data effectively. The result? Smarter, faster, and more adaptable models ready to tackle diverse real-world challenges. 🌟 What excites you most about the future of multimodal AI? Let’s discuss! 👇 #AI #MachineLearning #MultimodalAI #FineTuning #Innovation

Brain Boost Drop #21 𝐌𝐮𝐥𝐭𝐢𝐦𝐨𝐝𝐚𝐥 𝐑𝐀𝐆 𝐄𝐱𝐩𝐥𝐚𝐢𝐧𝐞𝐝 𝐕𝐢𝐬𝐮𝐚𝐥𝐥𝐲 ! Retrieval-Augmented Generation (RAG) is revolutionizing AI-powered search and retrieval systems, but it's no longer limited to just text! With the integration of multimodal capabilities, we can now combine both text and images to enhance the retrieval process, making AI systems more context-aware and capable of providing richer, more accurate responses. How does Multimodal RAG work? 1️⃣ A custom knowledge base is built using both text and images. 2️⃣ Images are converted into embeddings using specialized image embedding models and stored in a vector database. 3️⃣ Similarly, text is processed using text embedding models and indexed for retrieval. 4️⃣ When a query is made, it is converted into embeddings using text embedding models. 5️⃣ A similarity search is performed in the vector database to fetch the most relevant images and text. 6️⃣ The retrieved content is combined and used as context to prompt a multimodal large language model (LLM). 7️⃣ The LLM generates a response, leveraging both textual and visual data to provide a more accurate and contextualized answer. Why does this matter? Multimodal RAG enables AI to go beyond traditional text-based retrieval and integrate visual understanding, making it ideal for applications such as: ✅ AI-powered search engines ✅ Advanced chatbots with better context awareness ✅ Medical and scientific research assistance ✅ E-commerce and recommendation systems ✅ Legal and financial document analysis The future of knowledge retrieval is multimodal! If you're building AI applications that rely on enhanced retrieval mechanisms, Multimodal RAG is something you should explore. What are your thoughts on the future of AI-powered retrieval? Let's discuss! Follow Nikhil Kassetty for more Brain Boost Drops. #AI #MachineLearning #MultimodalRAG #LLM #KnowledgeRetrieval #AIInnovation #DeepLearning