Yu (Jason) Gu, PhD

REFRAG ( Representation for RAG) represents a fundamental shift for enterprise RAG systems.A Smarter Way to Scale RAG Using RL-Guided Chunk Compression: ➡️ Not “bigger context windows,” but smarter context routing ➡️ Not “retrieve everything” from sources as part of Retrieval step but retrieve and compress strategically before passing to Generation (decoder) step . Everyone in the LLM ecosystem is chasing longer context windows — but REFRAG offers a more elegant, scalable alternative. Here’s the breakthrough: 1️⃣ Compression Happens BEFORE passing to LLM Decoding For each retrieved chunk as part of retrieval step in RAG pipeline : • The encoder creates a semantic embedding • A trained RL policy decides: Compress this chunk → replace with embedding vector? Or keep it fully tokenized? This is the only place RL is applied. 2️⃣ What the Decoder Actually Sees The decoder receives a hybrid input: • Real tokens (for important chunks) • Single embedding vectors (for compressed chunks) Crucially: • ❌ The decoder does not decompress • ❌ There is no expansion of embeddings back to text • ✔ The decoder is pretrained to interpret compressed vectors as dense summarizations 3️⃣ Why It Works Because RL optimizes: • Which chunks matter • How many tokens to save • How to maximize task accuracy The result: • ⚡ Faster inference • 💡 Better use of limited context • 💰 Lower cost per query • 📈 Better relevance and grounding As agentic AI and on-device intelligence grow, techniques like this will be essential to scale GenAI without bloating context windows.

19

In physical autonomy, evaluating a plan based on its semantic plausibility guarantees failure. The architectural requirement is proving that an action is safe under bounded uncertainty. Language models generate plans through token prediction, a probabilistic process optimized for semantic coherence rather than physical safety. When those plans activate motors and servos, failure modes shift from hallucinated text to physical collision, drift, and hardware damage. The mismatch is structural: token prediction cannot validate the real-time state space. Autonomous systems fail due to accumulated sensor error, dynamic overshoot, and latency-driven desynchronization. A model that proposes "move the gripper 15cm left" cannot verify whether that instruction violates joint limits, exceeds payload capacity, or intersects an obstacle that appeared moments earlier. Scaling model parameters improves pattern recognition, not constraint enforcement. Safe autonomous architecture pairs probabilistic cognition with deterministic execution controls. Four boundaries enforce that separation: ↳ State reconciliation gate: The control system continuously reconciles intended motion with measured reality using proprioception, vision, and Inertial Measurement Unit (IMU) telemetry at control-rate frequencies (10–1000 Hz+, depending on the embodiment). If the measured state diverges beyond tolerance, the system halts or replans. ↳ Deterministic feasibility gate: A validation engine gates every proposed motion through deterministic checks before execution—forward kinematics for reachability, inverse dynamics for torque limits, and collision checks (e.g., swept-volume constraints). If any constraint fails, the gate rejects the motion unconditionally. ↳ Independent safety interlocks: Firmware-level circuit breakers and emergency-stop logic override model outputs when sensors detect force spikes, proximity violations, or thermal limits. These mechanisms operate independently of the probabilistic stack; a crashed inference server cannot disable them. ↳ Platform decoupling: The autonomy stack must treat physical embodiment as a swappable dependency. Architectures that bind the reasoning layer directly to proprietary hardware interfaces create structural fragility. When vendors deprecate interfaces, tightly coupled stacks break. Hardware abstraction layers are load-bearing constraints, not optimization targets. This separation builds on established engineering foundations. Traditional motion planning and control already rely on deterministic methods such as RRT* and Model Predictive Control (MPC) to enforce feasibility. Generative models add valuable intent synthesis, inferring semantic goals from ambiguous instructions, but the final arbiter of safe execution must remain verifiable. Architects must treat the large language model as an untrusted planner, requiring deterministic evidence before committing any actuator. #SystemsArchitecture #EnterpriseAI #AIGovernance

5

2 Comments

Google has recently launched their AlphaEvolve: a Gemini-powered coding agent for algorithm discovery. It shows clear progression from discovering algorithms for definite industry domains to developing more complex algorithms for a wide range of real-world challenges. AlphaEvolve orchestrates an autonomous pipeline of LLMs, which brings along automatic improvement of an algorithm by making direct changes to the code. It’s able to: ✅ Design faster matrix multiplication algorithms ✅ Find new solutions to open math problems ✅ Make data centers, chip design and AI training more efficient across ✅ Efficient data center scheduling and helping to improve hardware design Read a summary about it at https://lnkd.in/ga2jic2A Read Complete Research paper : https://lnkd.in/gr3CYrVn #LLMs #chatGPT #GENAI #gemini #gcp

47

1 Comment

Claude just released a memory import tool: https://lnkd.in/eM8c7Gf5 You paste a prompt into ChatGPT (or whatever you're using), it spits out your context — preferences, projects, how you work — and you drop it into Claude's settings. That's it. Claude picks it up from there. The switching cost has always been the real barrier. Not the AI itself, just the fact that you've spent months building context somewhere else. This is Anthropic admitting that out loud and actually doing something about it. Quiet release. Worth knowing about.

2

Excellent note on selecting the right AI models for particular tasks. The best AI-native products often leverage a collection of large language models (LLMs). https://lnkd.in/ejAG8ayV

16

Thrilled to share, "Devex Ambient Agent: Advanced Knowledge Graph," just published on DZone! Moving beyond RAG and dashboards in developer experience (DevEx). The real game-changer is giving AI agents a "memory and a map." The DevEx Ambient Agent uses a Knowledge Graph to solve a critical pain point: data silos and context-switching. (1) The Problem: Your code, Jira tickets, and documentation are often disconnected. Developers waste time hunting for the why behind a task. (2) The Solution: The Ambient Agent actively builds a dynamic, interconnected web—a Knowledge Graph—linking every development artifact. (3) The Value: This rich context enables the Agent's Intelligence Core to use ML Pattern Detection, identifying your team’s specific anti-patterns and offering proactive, highly relevant suggestions. It’s context-at-your-fingertips. Give it a read and let me know your thoughts on the future of DevEx and knowledge graphs! https://lnkd.in/gWY5R525

28

Dealing with chargeback evidence is a classic noisy data problem. You are handling a chaotic mix of PDFs, low res screenshots, and diverse document layouts where precision is the entire product. We recently moved our extraction pipeline to a GPU accelerated architecture using NVIDIA Nemotron Parse. The results were a practical masterclass in the value of specialized LLMs. And its now open source on Hugging Face! The Engineering Shift: Instead of relying on traditional OCR followed by a separate LLM pass, which often leads to context drift, we integrated a pipeline that handles layout aware parsing and semantic analysis in a more unified flow. The Impact: 🎯 25% Error Reduction: By improving the first mile of data extraction, we significantly reduced downstream hallucinations 🧠 Context Aware Mapping: The ability to map complex fields dynamically (as seen in our IDP flow) has made our evidence generation much more robust. ⚡Scalability: Moving to a GPU accelerated pipeline means we can process massive volumes of evidence without the latency bottlenecks of CPU bound parsing. For those interested in the technical architecture of how we are scaling Intelligent Document Processing (IDP), NVIDIA published a detailed breakdown of our stack. Full Case Study: https://lnkd.in/dYMDbY8d Thanks Gal Mizan and Lior Cohen for excellent collaboration! #AIArchitecture #MachineLearning #LLMs #DocumentParsing #Engineering #justt

28

1 Comment

Thank you Frank Sui for sharing the report and your perspectives. Important for Financial Service (FS) professionals tapping into the power of Machine Learning (ML) and GenAI in order to leverage the large volumes of data at our disposal in our quests to serve clients and mitigate risks more efficiently and effectively. My takeaways are: * The ratio of registered vs in-experimentation models in FS is the lowest among industries as the stakes are high and subject to more rigorous testing and data preparation. Historical challenge of data fragmentation and quality concerns makes the task all the more time consuming (remember the regulatory findings in the news about FS data quality and governance?). * Unstructured data is ubiquitous, making Natural Language Processing (NLP) a critical tool to derive meaning. If your organization has not rolled out NLP, there is much to catch up... Email surveillance and client contact notes review are the immediate areas for application. * Retrieval augmented generation (RAG) enables organizations to use their own proprietary data to better customize LLMs and deliver high-quality GenAI apps. It ensures models give more accurate answers and are less likely to hallucinate. That's why off-the-shelf models without any proper training are prohibited by most firms. * Common FS LLMs support business-critical use cases such as fraud detection and wealth management client sales pitch. It's not just about automating tasks, but also dynamically recalibrating parameters and synthesizing data to improve decision making and next best action. This is indeed an exciting area to further explore! #MachineLearning #AI #fraud #supervision #controls #data

8

Insightful HFS session on Sclaing AI with purpose! Practical approach to typical challenges like business value, data, risk approvals, and extensible architecture!

21