Information Retrieval Methods

Exciting breakthrough in Information Retrieval! A team of researchers at Walmart Global Tech has developed a novel approach to enhance diversity in Differentiable Search Indexing (DSI). The innovation lies in modifying the DSI architecture to balance both relevance and diversity during the training phase itself, eliminating the need for post-processing steps. The team introduced a clever dual-component loss function that combines traditional cross-entropy for relevance with a document similarity metric inspired by Maximal Marginal Relevance (MMR). Under the hood, their implementation: - Uses BERT-based classification model as the foundation - Implements a custom loss function that considers top-K logits for diversity calculation - Leverages cosine similarity to measure document overlap - Introduces an α parameter to balance relevance-diversity trade-off The results are impressive - their approach maintains high retrieval accuracy (Hits@1: 0.67, MRR@10: 0.75 on NQ320K dataset) while significantly reducing document redundancy in search results. Most importantly, this comes with no additional inference latency compared to vanilla DSI. Tested on both NQ320K and MSMARCO datasets, the method shows robust performance across different types of queries. A fascinating example from their paper shows how for a simple query like "California", their system returns a more diverse set of results covering history, demographics, specific regions, and even cultural references. This work opens up exciting possibilities for building more effective search systems that can provide comprehensive coverage while maintaining relevance. The approach is particularly valuable for sub-topic retrieval and can be extended to handle streaming document updates.

Over the past year, Retrieval-Augmented Generation (RAG) has rapidly evolved—from simple pipelines to intelligent, agent-driven systems. This visual compares the four most important RAG architectures shaping modern AI design: 1. 𝗡𝗮𝗶𝘃𝗲 𝗥𝗔𝗚 • This is the baseline architecture. • The system embeds a user query, retrieves semantically similar chunks from a vector store, and feeds them to the LLM. • It's fast and easy to implement, but lacks refinement for ambiguous or complex queries. 𝗨𝘀𝗲 𝗰𝗮𝘀𝗲: Quick prototypes and static FAQ bots. 2. 𝗔𝗱𝘃𝗮𝗻𝗰𝗲𝗱 𝗥𝗔𝗚 • A more precise and thoughtful version of Naive RAG. • It adds two key steps: query rewriting to clarify user intent, and re-ranking to improve document relevance using scoring mechanisms like cross-encoders. • This results in more accurate and context-aware responses. 𝗨𝘀𝗲 𝗰𝗮𝘀𝗲: Legal, healthcare, enterprise chatbots where accuracy is critical. 3. 𝗠𝘂𝗹𝘁𝗶-𝗠𝗼𝗱𝗲𝗹 𝗥𝗔𝗚 • Designed for multimodal knowledge bases that include both text and images. • Separate embedding models handle image and text data. The query is embedded and matched against both stores. • The retrieved context (text + image) is passed to a multimodal LLM, enabling reasoning across formats. 𝗨𝘀𝗲 𝗰𝗮𝘀𝗲: Medical imaging, product manuals, e-commerce platforms, engineering diagrams. 4. 𝗔𝗴𝗲𝗻𝘁𝗶𝗰 𝗥𝗔𝗚 • The most sophisticated approach. • It introduces reasoning through LLM-based agents that can rewrite queries, determine if additional context is needed, and choose the right retrieval strategy—whether from vector databases, APIs, or external tools. • The agent evaluates the relevance of each response and loops until a confident, complete answer is generated. 𝗨𝘀𝗲 𝗰𝗮𝘀𝗲: Autonomous assistants, research copilots, multi-hop reasoning tasks, real-time decision systems. As AI systems grow more complex, the method of retrieving and reasoning over knowledge defines their real-world utility. ➤ Naive RAG is foundational. ➤ Advanced RAG improves response precision. ➤ Multi-Model RAG enables cross-modal reasoning. ➤ Agentic RAG introduces autonomy, planning, and validation. Each step forward represents a leap in capability—from simple lookup systems to intelligent, self-correcting agents. What’s your perspective on this evolution? Do you see organizations moving toward agentic systems, or is advanced RAG sufficient for most enterprise use cases today? Your insights help guide the next wave of content I create.

If you are an AI Engineer building production-grade GenAI systems, RAG should be in your toolkit. LLMs are powerful for information generation, but: → They hallucinate → They don’t know anything post-training → They struggle with out-of-distribution queries RAG solves this by injecting external knowledge at inference time. But basic RAG (retrieval + generation) isn’t enough for complex use cases. You need advanced techniques to make it reliable in production. Let’s break it down 👇 🧠 Basic RAG = Retrieval → Generation You ask a question. → The retriever fetches top-k documents (via vector search, BM25, etc.) → The LLM answers based on the query + retrieved context But, this naive setup fails quickly in the wild. You need to address two hard problems: 1. Are we retrieving the right documents? 2. Is the generator actually using them faithfully? ⚙️ Advanced RAG = Engineering Both Ends To improve retrieval, we have techniques like: → Chunk size tuning (fixed vs. recursive splitting) → Sliding window chunking (for dense docs) → Structured data retrieval (tables, graphs, SQL) → Metadata-aware search (filtering by author/date/type) → Mixed retrieval (hybrid keyword + dense) → Embedding fine-tuning (aligning to domain-specific semantics) → Question rewriting (to improve recall) To improve generation, options include: → Compressing retrieved docs (summarization, reranking) → Generator fine-tuning (rewarding citation usage and reasoning) → Re-ranking outputs (scoring factuality or domain accuracy) → Plug-and-play adapters (LoRA, QLoRA, etc.) 🧪 Beyond Modular: Joint Optimization Some of the most promising work goes further: → Fine-tuning retriever + generator end-to-end → Retrieval training via generation loss (REACT, RETRO-style) → Generator-enhanced search (LLM reformulates the query for better retrieval) This is where RAG starts to feel less like a bolt-on patch and more like a full-stack system. 📏 How Do You Know It's Working? Key metrics to track: → Context Relevance (Are the right docs retrieved?) → Answer Faithfulness (Did the LLM stay grounded?) → Negative Rejection (Does it avoid answering when nothing relevant is retrieved?) → Tools: RAGAS, FaithfulQA, nDCG, Recall@k 🛠️ Arvind and I are kicking off a hands-on workshop on RAG This first session is designed for beginner to intermediate practitioners who want to move beyond theory and actually build. Here’s what you’ll learn: → How RAG enhances LLMs with real-time, contextual data → Core concepts: vector DBs, indexing, reranking, fusion → Build a working RAG pipeline using LangChain + Pinecone → Explore no-code/low-code setups and real-world use cases If you're serious about building with LLMs, this is where you start. 📅 Save your seat and join us live: https://lnkd.in/gS_B7_7d Image source: LlamaIndex

RAG just got smarter. If you’ve been working with Retrieval-Augmented Generation (RAG), you probably know the basic setup: An LLM retrieves documents based on a query and uses them to generate better, grounded responses. But as use cases get more complex, we need more advanced retrieval strategies—and that’s where these four techniques come in: Self-Query Retriever Instead of relying on static prompts, the model creates its own structured query based on metadata. Let’s say a user asks: “What are the reviews with a score greater than 7 that say bad things about the movie?” This technique breaks that down into query + filter logic, letting the model interact directly with structured data (like Chroma DB) using the right filters. Parent Document Retriever Here, retrieval happens in two stages: 1. Identify the most relevant chunks 2. Pull in their parent documents for full context This ensures you don’t lose meaning just because information was split across small segments. Contextual Compression Retriever (Reranker) Sometimes the top retrieved documents are… close, but not quite right. This reranker pulls the top K (say 4) documents, then uses a transformer + reranker (like Cohere) to compress and re-rank the results based on both query and context—keeping only the most relevant bits. Multi-Vector Retrieval Architecture Instead of matching a single vector per document, this method breaks both queries and documents into multiple token-level vectors using models like ColBERT. The retrieval happens across all vectors—giving you higher recall and more precise results for dense, knowledge-rich tasks. These aren’t just fancy tricks. They solve real-world problems like: • “My agent’s answer missed part of the doc.” • “Why is the model returning irrelevant data?” • “How can I ground this LLM more effectively in enterprise knowledge?” As RAG continues to scale, these kinds of techniques are becoming foundational. So if you’re building search-heavy or knowledge-aware AI systems, it’s time to level up beyond basic retrieval. Which of these approaches are you most excited to experiment with? #ai #agents #rag #theravitshow

🚀 𝐄𝐧𝐡𝐚𝐧𝐜𝐢𝐧𝐠 𝐒𝐞𝐚𝐫𝐜𝐡 𝐟𝐨𝐫 𝐌𝐨𝐫𝐞 𝐑𝐞𝐥𝐞𝐯𝐚𝐧𝐭 𝐑𝐀𝐆 𝐑𝐞𝐬𝐮𝐥𝐭𝐬. . . Retrieval-augmented generation (RAG) systems depend on retrieval and generation to produce high-quality responses. However, if the retrieval process isn’t effective, even the best LLMs will struggle to generate useful outputs. The Solution? 𝐄𝐧𝐡𝐚𝐧𝐜𝐞𝐝 𝐑𝐞𝐭𝐫𝐢𝐞𝐯𝐚𝐥 𝐓𝐞𝐜𝐡𝐧𝐢𝐪𝐮𝐞𝐬 Instead of relying on a basic retrieval system, we can refine queries and retrieval strategies to improve accuracy and relevance. Here are four techniques that could enhance retrieval performance: 📌 𝐄𝐧𝐭𝐢𝐭𝐲-𝐀𝐰𝐚𝐫𝐞 𝐑𝐞𝐭𝐫𝐢𝐞𝐯𝐚𝐥 Use named entities (e.g., people, locations, organizations) to refine search queries. ✅ Benefits: Improves precision by focusing on domain-specific terminology and reducing ambiguity. 📌 𝐇𝐲𝐛𝐫𝐢𝐝 𝐒𝐩𝐚𝐫𝐬𝐞-𝐃𝐞𝐧𝐬𝐞 𝐑𝐞𝐭𝐫𝐢𝐞𝐯𝐚𝐥 For better relevance, combine sparse retrieval (e.g., BM25) with dense vector search (embeddings). ✅ Benefits: Balances precision and recall, covering keyword-based and semantic search techniques. 📌 𝐌𝐮𝐥𝐭𝐢-𝐒𝐭𝐞𝐩 𝐃𝐨𝐜𝐮𝐦𝐞𝐧𝐭 𝐑𝐞𝐭𝐫𝐢𝐞𝐯𝐚𝐥 Retrieves documents iteratively, refining queries and filtering results in multiple stages. ✅ Benefits: Increases relevance for complex queries and eliminates noisy or duplicate results. 📌 𝐇𝐲𝐩𝐨𝐭𝐡𝐞𝐭𝐢𝐜𝐚𝐥 𝐃𝐨𝐜𝐮𝐦𝐞𝐧𝐭 𝐄𝐦𝐛𝐞𝐝𝐝𝐢𝐧𝐠 (𝐇𝐲𝐃𝐄) Generates a pseudo-document from the query before retrieval, improving search results. ✅ Benefits: Helps when queries are short, vague, or lack sufficient context. 🛠 How These Techniques Improve RAG 1️⃣ They increase recall, ensuring important documents aren’t missed. 2️⃣ They reduce noise, preventing irrelevant or duplicate context from misleading the generation step. 3️⃣ They handle complex queries better, allowing for better reasoning and improved search expansion. 💡 Key Takeaways 🔑 Better retrieval leads to better generation—fix retrieval first! 🔑 Simple techniques like entity-aware retrieval can drastically improve RAG results. ✍️ Want to dive deeper? Read the full article here: https://lnkd.in/gYv9UWuy 🔗RAG-To-Know Repository: https://lnkd.in/gQqqQd2a What are your thoughts? Have you used any of these techniques before? Let’s discuss this in the comments!👇👇👇

Many companies have started experimenting with simple RAG systems, probably as their first use case, to test the effectiveness of generative AI in extracting knowledge from unstructured data like PDFs, text files, and PowerPoint files. If you've used basic RAG architectures with tools like LlamaIndex or LangChain, you might have already encountered three key problems: 𝟭. 𝗜𝗻𝗮𝗱𝗲𝗾𝘂𝗮𝘁𝗲 𝗘𝘃𝗮𝗹𝘂𝗮𝘁𝗶𝗼𝗻 𝗠𝗲𝘁𝗿𝗶𝗰𝘀: Existing metrics fail to catch subtle errors like unsupported claims or hallucinations, making it hard to accurately assess and enhance system performance. 𝟮. 𝗗𝗶𝗳𝗳𝗶𝗰𝘂𝗹𝘁𝘆 𝗛𝗮𝗻𝗱𝗹𝗶𝗻𝗴 𝗖𝗼𝗺𝗽𝗹𝗲𝘅 𝗤𝘂𝗲𝘀𝘁𝗶𝗼𝗻𝘀: Standard RAG methods often struggle to find and combine information from multiple sources effectively, leading to slower responses and less relevant results. 𝟯. 𝗦𝘁𝗿𝘂𝗴𝗴𝗹𝗶𝗻𝗴 𝘁𝗼 𝗨𝗻𝗱𝗲𝗿𝘀𝘁𝗮𝗻𝗱 𝗖𝗼𝗻𝘁𝗲𝘅𝘁 𝗮𝗻𝗱 𝗖𝗼𝗻𝗻𝗲𝗰𝘁𝗶𝗼𝗻𝘀: Basic RAG approaches often miss the deeper relationships between information pieces, resulting in incomplete or inaccurate answers that don't fully meet user needs. In this post I will introduce three useful papers to address these gaps: 𝟭. 𝗥𝗔𝗚𝗖𝗵𝗲𝗸𝗲𝗿: introduces a new framework for evaluating RAG systems with a focus on fine-grained, claim-level metrics. It proposes a comprehensive set of metrics: claim-level precision, recall, and F1 score to measure the correctness and completeness of responses; claim recall and context precision to evaluate the effectiveness of the retriever; and faithfulness, noise sensitivity, hallucination rate, self-knowledge reliance, and context utilization to diagnose the generator's performance. Consider using these metrics to help identify errors, enhance accuracy, and reduce hallucinations in generated outputs. 𝟮. 𝗘𝗳𝗳𝗶𝗰𝗶𝗲𝗻𝘁𝗥𝗔𝗚: It uses a labeler and filter mechanism to identify and retain only the most relevant parts of retrieved information, reducing the need for repeated large language model calls. This iterative approach refines search queries efficiently, lowering latency and costs while maintaining high accuracy for complex, multi-hop questions. 𝟯. 𝗚𝗿𝗮𝗽𝗵𝗥𝗔𝗚: By leveraging structured data from knowledge graphs, GraphRAG methods enhance the retrieval process, capturing complex relationships and dependencies between entities that traditional text-based retrieval methods often miss. This approach enables the generation of more precise and context-aware content, making it particularly valuable for applications in domains that require a deep understanding of interconnected data, such as scientific research, legal documentation, and complex question answering. For example, in tasks such as query-focused summarization, GraphRAG demonstrates substantial gains by effectively leveraging graph structures to capture local and global relationships within documents. It's encouraging to see how quickly gaps are identified and improvements are made in the GenAI world.

🗄️ Retrieval Augmented Generation (RAG) • http://rag.aman.ai - RAG combines information retrieval with LLMs for enhanced response generation using an external knowledge base.  - This RAG primer delves into various facets of RAG encompassing chunking, embedding creation, indexing strategies, and evaluation. ➡️ For more AI primers, follow me on X at: http://x.aman.ai 🔹 Neural Retrieval 🔹 RAG Pipeline 🔹 Benefits of RAG  🔹 RAG vs. Fine-tuning  🔹 Ensemble of RAG 🔹 Choosing a Vector DB using a Feature Matrix 🔹 Building a RAG Pipeline - Ingestion - Chunking - Embeddings - Sentence Embeddings - Retrieval (Standard/Naive Approach, Sentence Window Retrieval Pipeline, Auto-merging Retriever) - Retrieve Approximate Nearest Neighbors - Response Generation / Synthesis (Lost in the Middle, The “Need in a Haystack” Test) 🔹 Component-Wise Evaluation - Retrieval Metrics (Context Precision, Context Recall, Context Relevance) - Generation Metrics (Groundedness/Faithfulness, Answer Relevance) - End-to-End Evaluation: Answer Semantic Similarity, Answer Correctness 🔹 Multimodal RAG 🔹 Improving RAG Systems - Re-ranking Retrieved Results - FLARE Technique - HyDE 🔹 Related Papers - Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks - MuRAG: Multimodal Retrieval-Augmented Generator - Active Retrieval Augmented Generation (FLARE) - Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs - Dense X Retrieval: What Retrieval Granularity Should We Use? - ARES: an Automated Evaluation Framework for Retrieval-Augmented Generation Systems - Hypothetical Document Embeddings (HyDE) ✍🏼 Primer written in collaboration with Vinija Jain #artificialintelligence #machinelearning #deeplearning #neuralnetworks 

This paper, “Closing the Retriever Gap in Agentic Search,” was featured in a Substack I follow on top IR papers of the week. It really stood out to me as one of the more thoughtful takes on retrieval for agentic systems. Most retrievers today are trained for a fairly simple setting: given a query, return locally relevant documents. That works well for one-shot RAG. But in agentic search, retrieval and reasoning are interleaved across multiple steps, and what matters is not just local relevance, but whether a retrieved document actually helps the system reach a correct final answer. What I especially like about this paper is that the proposed approach reminded me of A*, a classic algorithm from search theory used for efficient pathfinding in graphs. For a bit of context, A* is a goal-directed search algorithm. Instead of exploring paths blindly, it combines: - the cost of getting to the current node, and - a heuristic estimate of how close that node is to the goal. This balance allows A* to prioritize paths that are not just locally cheap, but globally promising. Agentic-R applies a very similar idea to information retrieval. Rather than training a retriever purely on query–document similarity, the paper trains the retriever to account for how much a passage contributes to final answer correctness within an agent’s multi-step reasoning process. In other words, retrieval is guided by progress toward the goal, not just surface relevance at a single step. Seen through this lens, Agentic-R reframes retrieval as a goal-directed search problem: - Local relevance still matters. - But passages are preferred if they move the agent closer to a correct outcome, even if they are not the most obviously similar to the current query. This helps explain why similarity-only retrievers can struggle in agentic settings, and why aligning retrieval with downstream reasoning can lead to meaningful gains. More broadly, this paper fits a pattern I’ve been noticing across recent IR work: strong systems emerge when retrieval is designed in service of the end objective, rather than optimized in isolation. Highly recommend this one if you’re thinking about retrieval for agents, multi-step QA, or the next evolution of RAG. #Search #InformationRetrieval #AgenticSearch #RAG #LLMs #SearchSystems https://lnkd.in/e63ans88

Learn problem framing before AI. Learn data curation before RAG. Learn ground truth before “LLM-as-a-judge.” Learn context engineering before multi-agent AI. Learn observability before deployment. Learn evaluation before scaling anything. RAG isn’t just retrieval + generation. It’s how you turn unstructured knowledge into a governed reasoning loop. Here’s the blueprint that actually ships. 1. Problem → Retrieval Objective Every strong RAG starts with defining what you’re retrieving and why. ↳ Clarify the intent: lookup, reasoning, or synthesis. ↳ Identify which data sources truly hold the answer. ↳ Define the expected output form: citation, snippet, summary, or decision aid. ↳ Then design your retrieval to serve that goal Without this alignment, every downstream step is noise. 2. Data Curation > Vectorising Internal Docs My first RAG, I dumped every internal wiki and doc into the pipeline, and it failed miserably. The information was there, but it wasn’t usable. ↳ Stitch related docs and close knowledge gaps before ingestion. ↳ Rewrite ambiguous text into task-relevant form. ↳ The best retrieval quality starts with curated structure, not volume. You don’t feed raw knowledge, you model it. 3. Chunking is Context Engineering Chunking isn’t about tokens, it’s about meaning boundaries. ↳ Segment by semantic units: definitions, procedures, FAQs, decisions. ↳ Preserve hierarchy: titles, headers, and relationships. ↳ Add connective tissue: short summaries that give each chunk standalone meaning. ↳ Test retrieval overlap: too small loses context, too large dilutes it. 4. Retrieval that actually retrieves ↳ Hybrid search (BM25 + vectors) → rerank. ↳ Domain-tuned embeddings when language is specialised. ↳ Routing/sub-queries for multi-facet questions. ↳ Tune your retriever to return diverse evidence; each chunk should add context the model didn’t already see. 5. Prompts as a lifecycle, not text ↳ Version in Git. ↳ Unit + regression tests tied to eval sets. ↳ A registry for safe rollout. You don’t YOLO prompts into prod. 6. Evals: the chicken-and-egg you must solve Most RAG metrics don’t help on day one, “LLM-as-a-judge” can grade a rubric, but without ground truth the score is noise. ↳ Start small: manually curate a seed Q/A set for your real tasks. ↳ Avoid synthetic Q/A from your own chunks as the only source (train-test contamination risk). ↳ Grow ground truth from user feedback (thumbs, edits, selected citations). ↳ Track per-query traces: input → sub-queries → retrieved chunks → final answer → citation correctness. Observability, Guardrails, Cost/Latency ↳ Log retrieval coverage, overlap, and dead-ends. ↳ Validate citations point to supporting text. ↳ Cache/rerank to cut tokens without cutting truth. ↳ Fail safe: when unsure, ask for clarification, don’t hallucinate. Stop wiring demos. Engineer retrieval, Then earn your evals. ♻️ Repost to help your team stop guessing and start measuring.

Most teams think RAG is solved. It’s not. What if the real breakthrough is not bigger models… But smarter retrieval? 𝐇𝐞𝐫𝐞 𝐚𝐫𝐞 12 𝐚𝐝𝐯𝐚𝐧𝐜𝐞𝐝 𝐑𝐀𝐆 𝐭𝐞𝐜𝐡𝐧𝐢𝐪𝐮𝐞𝐬 𝐫𝐞𝐬𝐡𝐚𝐩𝐢𝐧𝐠 𝐡𝐨𝐰 𝐀𝐈 𝐫𝐞𝐚𝐬𝐨𝐧𝐬, 𝐯𝐞𝐫𝐢𝐟𝐢𝐞𝐬, 𝐚𝐧𝐝 𝐬𝐜𝐚𝐥𝐞𝐬. → Mindscape Aware RAG • Builds a high level summary before retrieval • Connects scattered evidence like a human reader → Bidirectional RAG • Writes verified answers back into the corpus • Expands knowledge safely without hallucination drift → Graph O1 • Agent based GraphRAG with MCTS and reinforcement learning • Reasons efficiently over large graphs within context limits → QuCo RAG • Triggers retrieval using pretraining statistics • Detects rare or suspicious entities early → MegaRAG • Uses multimodal knowledge graphs for long documents • Enables global reasoning across text and images → Hybrid RAG for Multilingual QA • Handles noisy historical and OCR heavy documents • Grounds answers despite language drift → Multi Step RAG with Hypergraph Memory • Stores facts as structured hypergraphs • Supports deep multi step reasoning → TV RAG • Time aware retrieval for long videos • Aligns visuals audio and subtitles → SignRAG • Zero shot road sign recognition • Combines vision with retrieval → HiFi RAG • Multi stage document filtering • Reduces noise before generation → AffordanceRAG • Multimodal RAG for robotics • Selects actions grounded in physical reality → RAGPart and RAGMask • Lightweight protection against corpus poisoning • Defends systems without changing the LLM RAG is no longer just retrieval. It is reasoning architecture. Follow Umair Ahmad for more insights