Sai Teja Chava

Compressing LLMs but worried about accuracy? 🎯 EoRA uses eigenspace low-rank approximation to compensate for errors—no retraining needed. A promising direction for scalable, task-adaptive LLMs. New paper from #NVIDIAResearch.

1

Compressing LLMs but worried about accuracy? 🎯 EoRA uses eigenspace low-rank approximation to compensate for errors—no retraining needed. A promising direction for scalable, task-adaptive LLMs. New paper from #NVIDIAResearch.

1

Building a Production-Ready RAG System for Persian Language Processing I’m excited to share insights from developing an advanced Retrieval-Augmented Generation system specifically engineered for Persian/Farsi document processing, leveraging cutting-edge ML infrastructure and architectural patterns. 🔬 Technical Architecture Highlights: Advanced RAG Implementation: ∙ Self-Corrective Agentic RAG with adaptive query rewriting and iterative refinement ∙ Hybrid search architecture combining dense embeddings with SPLADE sparse representations ∙ Multi-agent orchestration enabling context-aware document interaction and general conversation modes Production Infrastructure: ∙ NVIDIA Triton Inference Server deployment on H100 GPUs with vLLM backend ∙ Milvus vector database for high-performance similarity search ∙ PostgreSQL for structured data with MinIO object storage for scalable file management ∙ Model ensemble: Qwen2.5-72B-Instruct-AWQ, DeepSeek-OCR, and sentence-transformers ML Pipeline Innovations: ∙ Automated synthetic QA pair generation using LangGraph workflows ∙ Cross-encoder reranking (ms-marco-MiniLM-L6-v2) for precision optimization ∙ Dynamic prompt management system with A/B testing capabilities and versioning ∙ Fine-tuning pipeline with quality-scored synthetic datasets Engineering Principles: ∙ Clean Architecture with dependency injection and separation of concerns ∙ Docker containerization for reproducible deployments ∙ Comprehensive observability with metrics tracking and performance monitoring This project demonstrates how to architect enterprise-grade RAG systems that combine theoretical ML advancements with practical production considerations, particularly for low-resource languages. Open to discussing technical implementation details, architectural decisions, or challenges in deploying LLMs at scale. #MLEngineering #RAG #NLP #LLMs #VectorSearch #ProductionML #AI #DeepLearning #Persian #TritonInference #CleanArchitecture

1

Quantization in Large Language Models (LLMs) As LLMs scale, intelligence is no longer the bottleneck — memory, latency, and cost are. Quantization turns out to be one of the most powerful (and misunderstood) techniques enabling real-world deployment of large models. In this blog, I break down: • Why LLMs are surprisingly quantization-friendly • INT8 vs INT4 vs NF4 — what actually works in practice • How modern architectures absorb low-bit noise • Where quantization fails and how to design around it Quantization is no longer an optimization step — it’s foundational infrastructure for production LLM systems. Link: https://lnkd.in/gqGje75G #LLMs #AIEngineering #Quantization #MachineLearning #GenAI #MLOps

9

2 Comments

🚨 Amazon. Google. Microsoft. Meta. All are spending hundreds of billions on AI. All are quietly dependent on the same constraint. H-1B talent! And US policy is tightening it at the exact moment AI demand is peaking. 🔍 What the data actually shows • In FY2025, 80%+ of new H-1B filings at Amazon, Google, Microsoft, Meta, and Apple were for AI-related roles • Roughly 70% of graduate students in AI fields are international • These roles are not cost arbitrage. US wage rules make that argument collapse fast ⚠️ At the same time, policy is moving the other direction • Visa renewal interviews delayed into 2027 • A proposed $100K fee on new H-1B applications • Shift from lottery to wage-weighted selection favoring senior roles • Pressure on OPT and student visa pathways This is not a political contradiction. It’s a strategic one. 📉 Why this matters for AI leadership When immigration tightens: • Companies do not slow AI investment • They relocate where AI work gets done • Talent follows policy • Capital follows talent • Capability follows both We have decades of evidence that restricting high-skill immigration pushes work offshore rather than protecting domestic jobs. 🤔 The real question Does the US want to build the next generation of AI systems, or manage them from a distance while the work migrates elsewhere? #AILeadership #FutureOfWork #H1B #TalentStrategy #DigitalLabor #TechPolicy #CHRO #CIO #AIEconomy

4

1 Comment

Google Research Director Denny Zhou, who founded the LLM Reasoning Team at DeepMind, recently gave a great talk at Stanford’s CS25 class: Large Language Model Reasoning. This was one of the most intuitive talks about reasoning model training and application. He outlined new directions for training models to handle questions where answers aren’t easily verifiable. My team is currently working on an enterprise LLM project focused on using reasoning to extract relevant context and generate responses for unverifiable cases. It’s a tough but promising area, and it’s an interesting time to be building. [Link to Denny Zhou's talk: https://lnkd.in/gdJ6Mzi8]

1,026

8 Comments

Fine-tuning LLMs on a Mac? Yes, it’s finally real-time possible. Meet mlx_lm. As a software engineer working, I've always wanted to fine-tune language models locally, without firing up expensive cloud GPUs. If you're on a Mac with Apple Silicon, this is where mlx_lm changes the game. What is mlx_lm? mlx_lm is a lightweight, Apple-optimized LLM training and inference library built on top of the MLX framework. It supports running quantized models efficiently on Mac’s unified memory and Neural Engine, making it perfect for experimentation, inference, and fine-tuning small to medium models like Mistral, LLaMA, etc. But here's the catch  Ref: https://lnkd.in/gUH-K4_Z ➡️ As of now, MLX doesn't support multi-GPU (even across CPU/Neural Engine), ➡️ Training very large models or with very large dataset is still memory-limited (~8–16B is okay), Personally I tested with 7B Mistral-7B-Instruct-v0.2. This worked with dataset where the size of each question and response is short. Unfortunately, Mac crashed for the large response scenario. ➡️ Some tools (e.g. advanced tokenizers, distributed training) still require workarounds. Why Quantization Matters Quantization reduces the size and compute of LLMs by converting high-precision weights (FP32) into lower-precision (e.g., int4/int8). Ref:  https://lnkd.in/gCFESgFe https://lnkd.in/gzq2Bw7h This leads to: Smaller memory footprint Faster inference Feasible fine-tuning even on consumer-grade hardware On MLX, quantized models are seamlessly supported, and the performance is surprisingly good. Getting Models from Hugging Face You can directly load quantized models from Hugging Face using: bash CopyEdit mlx-lm pull mistralai/Mistral-7B-Instruct-v0.1 For Mac: from mlx_lm import load, generate model, tokenizer = load("mlx-community/Mistral-7B-Instruct-v0.2-4bit") These models run inference-ready on your Mac without a single CUDA driver. How to Fine-Tune with MLX Fine-tuning is as simple as: Preparing your dataset (e.g., using HuggingFace datasets) Adapting hyperparams: learning_rate, batch_size, num_epochs Using mlx_lm train CLI or script interface for LoRA (Low Rank Adaption) or full fine-tuning Parameter tuning is key. For Apple chips, you’ll want: Lower batch sizes (2–8) Learning rates around 1e-5 to 2e-4 Optional: Use 4-bit LoRA if memory is tight What Can You Expect? With the right setup: ✅ Fine-tune a 7B model in a few hours ✅ Achieve task-specific improvements (e.g. customer support, Q&A, or internal docs) If you want to know more, please comment down below. I will create a separate post on each of these. #LLM #FineTuning #MacML #MLX #AppleSilicon #MachineLearning #AIEngineering #HuggingFace #OpenSourceAI

11