Boosting LLM Accuracy Using Graph Neural Networks

Exciting breakthrough in Retrieval Augmented Generation (RAG): Researchers have developed GFM-RAG, the first Graph Foundation Model for enhancing LLM knowledge retrieval. >> Key Innovations Novel Architecture: GFM-RAG introduces a query-dependent Graph Neural Network that can process complex knowledge relationships in a single step, dramatically improving both efficiency and accuracy compared to traditional multi-step approaches. Under the Hood - Constructs a knowledge graph index from documents to capture relationships between information - Uses a 6-layer query-dependent message passing neural network with 512-dimensional hidden states - Implements DistMult message functions and sum aggregation for graph processing - Pre-trains on 60 knowledge graphs containing over 14M triples and 700k documents >> Performance Highlights The system achieves state-of-the-art results across multiple datasets, outperforming existing methods by significant margins: - Up to 19.8% improvement in retrieval accuracy - 10x faster processing compared to multi-step approaches - Demonstrates strong zero-shot generalization across 7 different domains >> Impact This breakthrough by researchers from Monash University, Nanjing University of Science and Technology, and Griffith University represents a significant step forward in making LLMs more knowledgeable and efficient. The system's ability to scale and transfer across domains makes it particularly valuable for real-world applications.

GraphRAG: Teaching LLMs to Connect the Dots 📚 Ever felt like your AI assistant just doesn't get the big picture? Traditional RAG systems are like that friend who remembers random facts but can't quite piece them together. Meet GraphRAG, Microsoft's clever solution to help LLMs see the forest, not just the trees. Imagine trying to solve a puzzle with pieces scattered across different rooms. That's what traditional RAG does - it finds individual pieces but struggles to put them together. GraphRAG creates a map of how all the information fits together. This means LLMs can now understand connections and context in ways they never could before. What all GraphRAG can do?  1. Uncover Hidden Connections GraphRAG is like a detective, finding links between facts even when they're spread out. It helps LLMs tackle complex questions that require understanding how different pieces of info relate to each other. 2. Pinpoint Accuracy GraphRAG uses its knowledge map to find answers that are spot-on and make sense in context. Plus, you can trace each part of an answer back to its source. 3. Unlock Meaningful Insights GraphRAG doesn't just fetch facts, it sees the big picture. It can spot trends, identify themes, and offer insights that would be near impossible to find otherwise. Why This Matters for You? Think about how often you've asked an AI a question and gotten a response that's... close, but not quite right. Or worse, an answer that's just plain wrong. GraphRAG could change all that. It's about making AI assistants that truly understand what you're asking and can give you answers that actually help. What's Next? As GraphRAG like developments mature, we might see: • More intuitive AI assistants that can handle complex, multi-step questions • Better automated research tools that can draw insights from vast databases • AI systems that can explain their reasoning, making them more trustworthy and useful in fields like medicine or law.

Agentic systems don't just benefit from Small Language Models. They architecturally require them, paired with knowledge graphs. Here's the technical reality most teams miss. 🎯 The Workload Mismatch Agents execute 60-80% repetitive tasks: intent classification, parameter extraction, tool coordination. These need <100ms latency at millions of daily requests. Physics doesn't negotiate. Model size determines speed. But agents still need complex reasoning capability. 🧠 The Graph Solution The breakthrough: separate knowledge storage from reasoning capability. LLMs store facts in parameters. Inefficient. Graph-augmented SLMs externalize knowledge to structured triples (entity-relationship-entity), use 3-7B parameters purely for reasoning. Knowledge Graph of Thoughts: Same SLM solves 2x more tasks when querying graphs vs. processing raw text. Cost drops from $187 to $5 per task. Multi-hop reasoning becomes graph traversal, not token generation. Token consumption drops 18-30%. Hallucination reduces through fact grounding. 💰 The Economics At 1B requests/year: GPT-5 approach: $190K+ 7B SLM + graph infrastructure: $1.5-19K One production system: $13M annual savings, 80%→94% coverage by caching knowledge as graph operations. ⚡ The Threshold Below 3B parameters: Models can't formulate effective graph queries Above 3B: Models excel at coordinating retrieval and synthesis over structured knowledge Modern 7B models (Qwen2.5, DeepSeek-R1-Distill, Phi-3) now outperform 30-70B models from 2023 on graph-based reasoning benchmarks. 🏗️ The Correct Architecture Production agents converge on this pattern: Query → Classifier SLM → Graph construction/update → Specialist SLMs query graph → Multi-hop traversal → Response synthesis → (5% escalate to LLM) The graph provides: External memory across reasoning steps Fact grounding to prevent hallucination Reasoning scaffold for complex inference 🔐 Why This Matters Edge deployment: 5GB graph + 7B model runs locally on laptops Privacy: Medical/financial data never leaves premises Latency: Graph queries are deterministic <50ms operations Updates: Modify graph triples without model retraining Real case: Clinical diagnostic agent on physician laptop. Patient symptoms → graph traversal → diagnosis in 80ms. Zero external transmission. 🎓 The Separation of Concerns Graphs handle: relationship queries, continuous updates, auditability SLMs handle: query formulation, reasoning coordination, synthesis LLMs conflate both functions in one monolith. This drives their size and cost. Agent tasks follow this pattern: understand intent → retrieve structured knowledge → reason over relationships → execute action → update knowledge state. Graphs make each step explicit. SLMs provide coordination intelligence. Together, they outperform larger models on unstructured data at 10-36x lower cost. Are you still processing agent tasks with 70B+ models on raw text, or have you separated knowledge (graphs) from reasoning (SLMs)?

How can we further increase the accuracy of LLM-powered question answering systems? Ontologies to the rescue! That is the conclusion of the latest research coming from the data.world AI Lab with Dean Allemang. Based on our previous Knowledge Graph LLM Accuracy benchmark research, our intuition is that accuracy can be further increased by 1) leveraging the ontology of the knowledge graph to check for errors in the generated queries and 2) using the LLM to repair incorrect queries. We ask ourselves the following two research questions  1️⃣  To what extent can the accuracy increase by leveraging the ontology of a knowledge graph to detect errors of a SPARQL query and an LLM to repair the errors? 2️⃣ What types of errors are most commonly presented in SPARQL queries generated by an LLM? 🧪 Our hypothesis: An ontology can increase the accuracy of an LLM powered question answering system that answers a natural language question over a knowledge graph. 📏 Our approach consists of - Ontology-based Query Check (OBQC): checks deterministically if the query is valid by applying rules based on the semantics of the ontology. The rules check the the body of the query (i.e. WHERE clause) and the head of query (i.e. the SELECT clause). If a check does not pass, it returns an explanation.  - LLM Repair: repair the SPARQL query generated by the LLM. It takes as input the incorrect query and the explanation and sends a zero-shot prompt to the LLM. The result is a new query which can then be passed back to the OBQC. 🏅Results: Using our chat with the data benchmark and GPT-4 - Our OBQC and LLM Repair approach increased the accuracy to 72.55%. If the repairs were not successful after three iterations, an unknown result was returned, which occurred 8% of the time. Thus the final error rate is 19.44%. “I don’t know” is a valid answer which reduces the error rate. - Low complex questions on low complex schemas achieves an error rate of 10.46%, which is now arguably at levels deemed to be acceptable by users. - All questions on high complex schemas substantially increased the accuracy. - 70% of the repairs where done by rules checking the body of the query. The majority were rules related to the domain of a property.  Putting this all together with our previous work, LLM Question Answering accuracy that leverages Knowledge Graphs and Ontologies is over 4x the SQL accuracy! These results support the main conclusion of our research: investment in metadata, semantics, ontologies and Knowledge Graph are preconditions to achieve higher accuracy for LLM powered question answering systems. Link to paper in comments. We are honored that we get to work with strategic customers to push the barrier of the data catalog and knowledge graph industry, and the data.world product. We are proud that our research results are a core part of the the data.world AI Context Engine. Thanks for all the valuable feedback we have received from colleagues across industry and academia 

🤔 Vanilla-RAG struggles with structured knowledge sources like knowledge graphs. GNN-RAG is a very neat idea to fix this! ⛳ Vanilla-RAG struggles with structured inputs like KGs because it relies heavily on LLMs for retrieval, which are not adept at handling the complex graph information inherent in KGs. This leads to suboptimal performance, especially on multi-hop and multi-entity questions that require traversing multiple relationships in the graph. ⛳ GNN-RAG integrates the strengths of both LLMs and and Graph Neural Networks (GNNs) to solve this issue: 💡 GNN: Excels at processing and reasoning over graph structures. It reasons over a dense KG subgraph to retrieve answer candidates for a given question. 💡LLM: Leverages its natural language processing abilities to further reason over the information provided by the GNN. 👉 Here's the workflow: 🔺 A GNN processes the KG to identify and retrieve candidate answers. 🔺The shortest paths connecting question entities to answer candidates in the KG are extracted to represent reasoning paths. 🔺These paths are verbalized and provided as input to the LLM for final reasoning and answer generation. GNN-RAG achieves state-of-the-art results on two widely used KGQA benchmarks, WebQSP and ComplexWebQuestions (CWQ) and outperforms existing methods, including GPT-4, particularly on multi-hop and multi-entity questions. Link to the paper: https://lnkd.in/euC7N85K

TL;DR: There has been a dramatic uptick in interest in Knowledge Graphs (KGs). Combined with LLMs, KGs can provide better insights into organizational data while reducing or even eliminating hallucinations just like some ideas in 𝗡𝗲𝘂𝗿𝗼-𝗦𝘆𝗺𝗯𝗼𝗹𝗶𝗰 𝗔𝗜. A long time ago I wrote about how Symbolic AI and Neural AI will come together to unlock new value while lowering enterprise risk. (https://bit.ly/3WZQ11q). We are definitely headed down that path with some interesting startups like Elemental Cognition (https://lnkd.in/eFUhFYEZ) and Amazon Web Services (AWS) using symbolic techniques for security scanning of LLM generated code in Q Developer (https://lnkd.in/ecJTSSaS). Another variant albeit not Neuro-Symbolic AI is the 𝗶𝗻𝘁𝗲𝗴𝗿𝗮𝘁𝗶𝗼𝗻 𝗼𝗳 𝗞𝗚𝘀 𝗮𝗻𝗱 𝗟𝗟𝗠𝘀. KGs are inherently symbolic and integrating with LLMs is a no-brainer for specific use cases. A great writeup of the 𝗯𝗲𝗻𝗲𝗳𝗶𝘁𝘀 by the excellent Neo4j team (Philip Rathle, Emil Eifrem): https://lnkd.in/ebR6tMD8 which itself builds on some great work by the Microsoft GraphRAG team (https://lnkd.in/enRpA6Y7). Benefits summary: 1. 𝗛𝗶𝗴𝗵𝗲𝗿 𝗔𝗰𝗰𝘂𝗿𝗮𝗰𝘆 & More Useful Answers  • A KG combined with an LLM improved accuracy by 3x • LinkedIn showed that KG integrated LLMs outperforms the baseline by 77.6% (https://lnkd.in/eNvvQaeq) 2. 𝗜𝗺𝗽𝗿𝗼𝘃𝗲𝗱 𝗗𝗮𝘁𝗮 𝗨𝗻𝗱𝗲𝗿𝘀𝘁𝗮𝗻𝗱𝗶𝗻𝗴, 𝗙𝗮𝘀𝘁𝗲𝗿 𝗜𝘁𝗲𝗿𝗮𝘁𝗶𝗼𝗻 3. 𝗚𝗼𝘃𝗲𝗿𝗻𝗮𝗻𝗰𝗲: 𝗘𝘅𝗽𝗹𝗮𝗶𝗻𝗮𝗯𝗶𝗹𝗶𝘁𝘆, 𝗦𝗲𝗰𝘂𝗿𝗶𝘁𝘆, and More 𝗔𝗻𝗱 𝗵𝗲𝗿𝗲 𝗶𝘀 𝘁𝗵𝗲 𝗶𝗻𝘁𝗲𝗿𝗲𝘀𝘁𝗶𝗻𝗴 𝘁𝘄𝗶𝘀𝘁: KGs and ontologies have historically been hard to create and maintain. Turns out you can use LLMs+ to simplify that process!! Great research work here: https://lnkd.in/eTyGjSe5 and actual implementation by the Neo4J team (https://bit.ly/3WIJxmd). If you want to try this using AWS services give it a whirl here: https://go.aws/3T8FK0L 𝗔𝗰𝘁𝗶𝗼𝗻 𝗳𝗼𝗿 𝗖𝘅𝗢𝘀: Consider adding Knowledge Graphs to your enterprise Data and GenAI strategy.

GNNs + RAG = Graph Neural Retrieval ✨ Knowledge Graphs (KGs) are a powerful way to represent factual knowledge, but querying them with natural language is challenging. And Graph Neural Networks (GNNs) excel at reasoning over KGs, something Large Language Models (LLMs) are still struggling with. There has been a lot of work in combining these two approaches lately, but it doesn't feel like we've found the right recipe yet. By leaning into the popular Retrieval-augmented Generation (RAG) trend, GNN-RAG tries to change this. The idea is for the GNN to handle the complex graph structure, while the LLM leverages its language understanding to produce the final answer. GNN-RAG achieves state-of-the-art performance on two major KGQA benchmarks, even outperforming GPT-4 in some cases. There have been quite a few graph related papers in May. Might be time for a review article? ↓ Liked this post? Join my newsletter with 25k+ readers that breaks down all you need to know about the latest LLM research: llmwatch.com 💡