Knowledge Management System Development

Building useful Knowledge Graphs will long be a Humans + AI endeavor. A recent paper lays out how best to implement automation, the specific human roles, and how these are combined. The paper, "From human experts to machines: An LLM supported approach to ontology and knowledge graph construction", provides clear lessons. These include: 🔍 Automate KG construction with targeted human oversight: Use LLMs to automate repetitive tasks like entity extraction and relationship mapping. Human experts should step in at two key points: early, to define scope and competency questions (CQs), and later, to review and fine-tune LLM outputs, focusing on complex areas where LLMs may misinterpret data. Combining automation with human-in-the-loop ensures accuracy while saving time. ❓ Guide ontology development with well-crafted Competency Questions (CQs): CQs define what the Knowledge Graph (KG) must answer, like "What preprocessing techniques were used?" Experts should create CQs to ensure domain relevance, and review LLM-generated CQs for completeness. Once validated, these CQs guide the ontology’s structure, reducing errors in later stages. 🧑⚖️ Use LLMs to evaluate outputs, with humans as quality gatekeepers: LLMs can assess KG accuracy by comparing answers to ground truth data, with humans reviewing outputs that score below a set threshold (e.g., 6/10). This setup allows LLMs to handle initial quality control while humans focus only on edge cases, improving efficiency and ensuring quality. 🌱 Leverage reusable ontologies and refine with human expertise: Start by using pre-built ontologies like PROV-O to structure the KG, then refine it with domain-specific details. Humans should guide this refinement process, ensuring that the KG remains accurate and relevant to the domain’s nuances, particularly in specialized terms and relationships. ⚙️ Optimize prompt engineering with iterative feedback: Prompts for LLMs should be carefully structured, starting simple and iterating based on feedback. Use in-context examples to reduce variability and improve consistency. Human experts should refine these prompts to ensure they lead to accurate entity and relationship extraction, combining automation with expert oversight for best results. These provide solid foundations to optimally applying human and machine capabilities to the very-important task of building robust and useful ontologies.

Sad but true. The closer the agent gets to production, the more the cracks begin to show. Teams make two mistakes at this point. They either look for a bigger/smarter LLM or endlessly iterate on prompting and RAG. Neither works. Successful agents start with the workflow, not the LLM. The more detailed the description of the workflow and outcomes, the less the agent needs to rely on AI. Every time the agent must guess what the next step is or what tools and information to use at this step, it creates an opportunity for small mistakes. They compound across multiple steps into much larger failures. Next, the workflow and the domain expertise required to deliver the outcome must be built into a knowledge graph. Trying to stuff everything into markdown files is a recipe for hallucination pie. The longer the file, the harder it is for LLMs to keep things straight. They lose focus and lose sight of what information is important. Knowledge graphs fix this by giving the agent exactly the information it needs at exactly the step it needs it. When agents get lost, and uncertainty metrics rise, the knowledge graph can deliver examples and metrics that define success and refocus the agent on iterating until it builds an acceptable output. Knowledge graphs can deliver guardrails that prevent agents from falling into endless loops. The goal is to build agents that rely on LLMs as little as possible and only deploy LLMs for what they are good at. Use the smallest models possible, and open-source models should handle over 80% of the workflow. Finally, agents need real-world feedback to improve. Version 1 is never perfect, and it takes multiple improvement cycles to be ready for deployment. Agents and knowledge graphs must be architected to benefit from improvement cycles. Every mistake creates the data required to ensure it never happens again.

We have been deploying RLM-style architectures for enterprise clients over the past months, and the implementation lessons are significant. The use cases driving adoption include:- - Regulatory compliance:- Organizations are analyzing thousands of pages across evolving frameworks such as GDPR, AI Act, and NIST AI RMF. Traditional approaches often hit context limits or hallucinate. Recursive patterns allow us to trace every conclusion back to source clauses. - Enterprise knowledge work:- Teams are overwhelmed by documentation, codebases, and institutional knowledge. RLMs effectively handle what RAG systems struggle with: multi-hop reasoning across massive, heterogeneous datasets. - Security audits:- Analyzing entire codebases for vulnerabilities is now possible. The ability to recursively decompose and reason over 100K+ line repositories transforms automated review capabilities. Key lessons learned from implementing these systems include:- - Architecture beats brute force:- Using larger context windows can be costly and often ineffective. Teaching systems to intelligently decompose problems is more efficient and effective. - Observability is crucial:- When an AI makes multiple sub-queries to answer a single question, serious instrumentation is needed. We have developed custom tracing to understand decision flows, which is essential for governance and debugging. - The prompt evolves into a framework:- Instead of simple prompts, we are creating meta-cognitive frameworks that guide the system's exploration. This requires a different skill set. - Cost dynamics change:- Initial implementation may be heavier than basic LLM calls, but at scale, selective context loading can reduce costs by 3-5 times compared to naive long-context approaches. The governance aspect is vital:- Recursive systems with code execution create auditable reasoning chains. When AI decisions impact compliance, procurement, or risk assessment, the ability to trace the logic and criteria used is essential. However, there are hard truths to acknowledge:- - Not every problem requires recursion; some tasks genuinely need dense attention across the full context. - Failure modes are different. A single bad sub-query can cascade. Error handling and validation become critical. - Latency can be an issue. Synchronous recursive calls add up. We're exploring async patterns. Where this is heading:- The shift from LLMs as 'smart text generators' to 'cognitive orchestrators' is accelerating. The research from Massachusetts Institute of Technology MIT Computer Science and Artificial Intelligence Laboratory (CSAIL) validates what we're seeing in production, the next wave of AI systems won't just process information; they'll actively manage computational workflows. What patterns are you finding for orchestrating multi-step AI reasoning? Are you seeing similar cost/performance tradeoffs? #AgenticAI #AIArchitecture #AIGovernance #EnterpriseAI #BuildingAI

Most companies treat “knowledge management” like a filing cabinet. Centralize everything → hope people find it. But doing it well is very different. It’s not just about managing knowledge—it’s about: 1/ Managing diverse content types across departments – Support writes post-sales content, Marketing creates pre-sales content, Product ships product documentation and release notes, HR builds onboarding and employee enablement, Sales shares playbooks and sales resources, Channels shares partner enablement content. Each team produces content in a different format. 2/ Distributing knowledge and content across touchpoints – Knowledge and content are needed in customer portals, support pages, AI assistants, in-app help, CRMs, ticketing systems, intranets, partner portals, and even sales and service partner's websites, portals, and ticketing systems. 4/ Designing experiences for specific audiences – What customers expect in self-service is different from what employees need on an intranet, or what distributors need in a portal. Each application must feel purpose-built to the user, but be created and evolved using nocode by the creator. 5/ Capturing feedback & measuring performance – Customers downvote unhelpful articles, partners flag missing details, employees leave comments. Without feedback loops, content gets stale and trust erodes. 6/ Blending traditional and AI-based experiences – Some users prefer to search, filter, and browse while others want to have a conversation with an AI Assistant, and often the best solution is both working together. The result isn’t a “central library.” It’s a living system that: - Bridges departments and silos - Serves every audience where they are - Improves with each interaction That’s how knowledge can be leveraged to help customers, partners, and employees be more successful with your products and services. #knowledgemanagement, #customerenablement, #partnerenablement, #employeeenablement, #AI, #selfservice

For enterprises Knowledge as a Service (KaaS) is getting crucial for AI readiness. The knowledge layer needs to sit on top of existing enterprise systems, making organizational knowledge accessible, maintainable, and AI-ready while preserving existing operational capabilities and governance. Let me try to bring clarity to KaaS Knowledge Discovery and Mapping Map all operational databases and their relationships Identify data warehouses and their current analytical models Document unstructured data sources (documents, emails, process documentation, pictures, videos etc.) Catalog existing business intelligence reports and dashboards Knowledge Flow Analysis Map how data flows between different systems Identify key business processes and their data dependencies Document decision points that require knowledge access Knowledge Structure Development Categorize data based on business context and usage Identify critical knowledge areas and their relationships Create taxonomy for organizing enterprise knowledge Establish metadata framework for knowledge assets Knowledge Model Creation Design knowledge graphs connecting different data sources Create semantic relationships between business concepts Develop ontology for business domain knowledge Map data lineage across systems Technical Implementation Deploy knowledge management platform Implement connectors to operational databases and data warehouses Set up real-time data synchronization mechanisms Create APIs for knowledge access and retrieval Processing Pipeline Develop ETL processes for knowledge extraction Implement AI-powered categorization systems Create automated tagging and classification workflows Set up validation and quality control mechanisms Knowledge Transformation Enrich operational data with business context Create relationships between different knowledge components Implement version control and lifecycle management Integration Layer Connect knowledge platform with existing BI tools Enable knowledge discovery through search interfaces Implement role-based access control Create audit trails for knowledge usage AI Readiness Knowledge Componentization Break down complex information into AI-digestible components Create training datasets for AI models Implement RAG (Retrieval Augmented Generation) capabilities Develop knowledge validation workflows AI Integration Set up AI models for knowledge processing Implement machine learning for continuous improvement Create feedback loops for knowledge refinement Enable automated knowledge updates Operational Excellence Monitoring Setup Implement usage tracking and analytics Create performance dashboards Set up alerting for knowledge quality issues Monitor system performance and utilization Governance Implementation Establish knowledge management policies Define roles and responsibilities Create maintenance procedures Implement compliance controls #GenerativeAI #EnterpriseAI #LLMIntegration #AIImplementation #Innovation

Continuing with the GenAI series, I am excited to share how we revolutionised the knowledge management system (KMS) for a leading client in the manufacturing industry. R&D teams in manufacturing often face the tedious task of manually sifting through complex engineering documents and standard operating procedures to ensure compliance, uphold safety standards, and drive innovation. This manual process is not only time-consuming but also prone to errors. To address this, we collaborated with our client to automate their R&D function’s KMS using Generative AI (GenAI). By allowing precise querying of specific sections of documents, our solution sped up access to critical information, reducing search time from hours to mere seconds. Our Generative AI team processed over 110 R&D-related documents, leveraging Large Language Models (LLMs) to generate accurate responses to complex queries. Hosted on a leading cloud platform with an Angular-based UI, the solution delivered remarkable benefits, including: - Significant accuracy in generated answers - Faster and more accurate data search and summarisation - Enhanced decision-making with easier access to critical R&D information - Improved overall employee productivity By implementing GenAI for knowledge management, the client's R&D function was also able to improve its competitive edge by tracking and responding quickly to market trends and consumer behavior. With plans to scale the solution to process over 1,500 documents across multiple departments, the client is creating a centralised hub for all their information needs. Taking advantage of GenAI can revolutionize knowledge management by delivering the right information to the right person on demand and enabling strategic impact. #GenAI #ManufacturingInnovation #KnowledgeManagement #GenAIseries #GenAIcasestudy #Innovation #R&D #DigitalTransformation #AI #Deloitte

Focus on Knowledge Management NOW I have been working on the ServiceNow platform for over six years, and one common mistake organizations make is neglecting to mature their knowledge bases and articles. A poor ServiceNow knowledge base can make your entire platform feel bleak. It can be very frustrating when you want to introduce new capabilities, like the virtual agent, or improve your service catalog, but your knowledge bases lack sufficient articles. Organizations need to invest time in building a strong knowledge base before they can successfully develop more comprehensive IT Service Management workflows. Here are several ways organizations can build a strong knowledge base: 1. Conduct an audit of existing knowledge articles to identify which articles should be retired or updated. 2. Hire a dedicated Knowledge Manager responsible for updating existing knowledge articles and creating new ones. 3. Develop a knowledge management governance process for creating new articles to ensure consistency in formatting, a clear content strategy, and proper meta tagging. Create a knowledge article template for this purpose. 4. Establish a review and approval process involving the Knowledge Manager, subject matter experts, and key stakeholders. 5. Ensure that knowledge articles are appropriately linked within service catalog items, virtual agents, and other relevant ServiceNow portals.  6. Gather valuable feedback from end users to ensure that knowledge articles are useful and effectively address their requests and incidents. 7. Review the knowledge management data to identify which articles are viewed the most. This will help you understand how to improve other ITSM workflows related to your service catalog items and request forms. 8. Knowledge Management is not a one-time task; become comfortable with making continuous improvements. Listen to your end users, as they can help you make your knowledge bases better. How do you improve your knowledge articles? Comment below #ITSM #ServiceNow #KnowledgeManagement #ITIL

Just built a working version of Andrej Karpathy's "LLM Knowledge Bases" idea with Spring AI. Karpathy recently shared how he uses LLMs to turn research materials into a self-maintaining personal wiki. Plain Markdown files, Obsidian on the front end, and the LLM handling summaries, backlinks, and cleanup. I loved the pattern, so I built it in Spring. karpathy-wiki is a local Spring Boot CLI app with three Spring AI agents: ✅ WikiCompilerAgent takes raw notes and URLs and turns them into a clean, linked wiki with articles, concepts, summaries, and an index. ✅ ResearchAgent answers questions against your wiki, generates new content like notes and Marp slides, and files it back in. ✅ WikiLinterAgent finds orphans, broken links, stubs, and contradictions to keep things tidy. Everything lives as plain .md files on disk. No vector DB, no black boxes. Just files you can version with Git. Built with: • Spring AI 2.0 and spring-ai-agent-utils • Spring Shell for the CLI • Java 26 The whole wiki schema and agent behavior is defined in one SCHEMA.md file, so it's easy to extend. Repo: https://lnkd.in/gkAZ7cdi If you're a Java or Spring developer looking at AI agents, personal knowledge management, or alternatives to RAG, I'd love your feedback. Give it a star, try it out, or let me know what you think. How do you build knowledge bases with LLMs? #SpringAI #Java #SpringBoot #AIAgents

If you don’t manage knowledge properly, AI can’t deliver real value. Here’s how to go from beginner to pro in knowledge management: Managing knowledge is just like any other skill. Level 1 systems give level 1 results... And learning how to manage knowledge effectively is a process. Most companies are stuck at level 1, maybe level 2 if they're trying. But the real power of knowledge management lies in levels 3 and 4. That’s when you stop wasting time and start: Harnessing valuable insights at scale Ensuring your data is accessible and actionable Making AI actually work for your business Here are the 4 levels of knowledge management broken down: Level 1: The Collector “Store knowledge in documents and chats.” Goal: Keep everything in one place. Mindset: Gathering data, no structure. This is where most companies stay. They store everything but it’s hard to find or use. How to improve: Organize documents based on themes, not random storage. Start creating some basic structure (folders, categories). Level 2: The Organizer “Classify knowledge by topics, departments, and workflows.” Goal: Add clarity and context. Mindset: Structuring data so it’s easier to retrieve. You’ve moved beyond simply storing knowledge. You start to define where and how data is kept. How to improve: Use simple structures: Category → Subcategory → Actionable Insight. Make sure the content can be easily updated and retrieved. Level 3: The Strategist “Link data with context, making it actionable.” Goal: Create a system where context meets knowledge. Mindset: Context-driven knowledge retrieval and application. This is where results compound. You turn stored knowledge into actionable insights. How to improve: Use feedback loops: categorize → review → refine → apply. Start building systems where knowledge is automatically applied to real tasks. Level 4: The Master “Integrate AI into your knowledge system to automate insights.” Goal: Make the system intelligent and adaptive. Mindset: AI seamlessly integrates with your knowledge base. At this level, AI works with your knowledge to deliver insights instantly. Your system evolves and improves continuously. How to improve: Build smart systems that learn and adapt with each data point. Ensure the system becomes part of your everyday workflow. My team and I use Level 3 and 4 knowledge management every single day. It’s how we scale insights and create smarter AI systems faster. How much do you know about managing knowledge for AI? Drop a comment below to discuss. If you want your business to thrive with AI, you need to optimize your knowledge management system. That’s exactly what we do at Thunai.ai. Learn more here: Thunai.ai ♻️ Repost if you believe AI can only be powerful if knowledge is properly structured. ➕ Follow Aditya for more actionable insights on optimizing your AI-driven knowledge systems.

Is your team making the most of what it knows? AI can help unlock that knowledge. Knowledge Graphs = 10X'ing your team capabilities. (FYI: Part of my talent intelligence work involves mapping out centers of knowledge. This is my basic scoring system.) It's a 9-level framework for team knowledge management. It generally applies to these six categories 🟠 Talent Intelligence 🟣 Team Meetings 🟡 Account Management 🟢 Sales Enablement 🔵 Community Engagement ⚫ Workforce Management ***The first three stages are where a majority of orgs are trapped (in red first column) It starts simple at Level 1 (no AI). Then, teams begin recording info (Level 2) and using AI for notes (Level 3). As you progress to levels 4 through 9 it enables two groups to leverage the data: your 🔴 Executive team 🔵 R&D/innovation team. Higher levels use AI to structure knowledge (Level 4). AI finds key insights and tasks (Level 5). It can even assign tasks with human help (Level 6). The top levels (7-9) see AI running tasks itself, acting like a team member, and helping the whole organization adapt. Understanding these levels helps teams see where they are and where they can go with AI. ⚠️ If your team hasn't progressed out of steps 1 to 3 - you are leaving invaluable and priceless knowledge on the table. See the attached graphic for a full look at each level. What level is your team on? Would love to hear if your team uses a different approach to structured systems for knowledge management. #TalentIntellignece #PeopleAnalytics #KnowledgeManagement