Kriti Goyal

⚡ ML inference continues to fascinate me, and I've been diving deep into LLM inference optimization techniques - something I genuinely enjoy working on! Recently, I've been exploring the intricate challenges of serving large language models efficiently, and it led me to build TorchWeave LLM - an open-source inference compiler that achieves 2-5x throughput improvements through some compelling optimization strategies. What makes ML inference so interesting is how it sits at the intersection of systems engineering and machine learning theory. Every optimization decision has cascading effects on memory, compute, and user experience. 🔄 Continuous Batching: One technique I found particularly rewarding to implement - merging concurrent requests into shared decode steps to maximize GPU utilization. The scheduling algorithms here are genuinely fascinating. ⚡ Per-Request KV-Cache Management: Exploring how attention masks can enable efficient memory management for variable sequence lengths opened up entirely new optimization possibilities. 📡 Real-time SSE Streaming: Building Server-Sent Events with time-to-first-token metrics taught me so much about balancing performance with user experience. The exploration paid off: 98% throughput improvement with 4 concurrent requests Average TTFT of 0.860s Production-ready architecture that scales What I love about working on inference optimization is that every bottleneck you solve reveals three new interesting problems. The rabbit holes of memory management, scheduling algorithms, and distributed systems design never get old. The technical implementation spans everything from PyTorch model interfaces to FastAPI async patterns - each component was a learning opportunity that deepened my appreciation for the complexity of production ML systems. 🔗 Check it out on GitHub: https://lnkd.in/gCZdZ5Pm I'm curious - what aspects of ML inference optimization do you find most compelling? Always eager to learn from others tackling similar challenges! #MachineLearning #LLM #MLInference #PyTorch #Optimization #OpenSource #MLOps #AI #SoftwareEngineering #TechnicalExploration Built with Python 3.12, FastAPI, PyTorch, and genuine curiosity about making models faster ⚡

49

6 Comments

Thrilled to share today’s progress in my journey to build Large Language Models (LLMs) from scratch! Instead of just reading formulas, I implemented self-attention with trainable weights and Encapsulated the logic in a clean SelfAttention class, handling input embeddings, projecting Q/K/V vectors, and computing attention outputs - a key step toward mastering transformers. ✨ Each token creates queries, keys, and values - and the model learns the weights that decide what to focus on. ✨ The attention mechanism then combines these into context vectors, dynamically adjusting focus across the sequence. ✨ What struck me: the code is compact, but the behavior it enables is incredibly powerful - scaling from simple toy text to the backbone of modern LLMs. Regular updates to come as I dive deeper into LLMs! If you’re into PyTorch, tokenization, or building AI from scratch, let’s connect and share insights! Check out my GitHub for full-stack and AI : https://lnkd.in/gvz8WtKu #AI #LLM #Attention #SelfAttention #PyTorch #DeepLearning #MachineLearning #LearningJournal #AIEngineer #CareerGrowth #OpenToWork #PyTorch #WebDev #Transformers #Learning #NeuralNetworks #Tokenization #LearningJourney #AIEngineering #AIJobs #MachineLearningJobs #TechCareers #DeveloperJourney #CareerGrowth #FullStackAI

9

✨ Excited to be speaking at the Women in AI Meetup today hosted by Voxel51! 🎤👩‍💻 I’ll be diving into "Towards Reliable Clinical AI," discussing our work on FactEHR (in collaboration with Stanford Medicine) and the critical need for evaluating factuality and robustness in medical LLMs 🏥🤖 , and BEACON, a framework to improve NER (in collaboration with Ai2). As I prepare to graduate with my PhD in Summer 2026, I’m looking forward to connecting with fellow researchers and industry leaders! 🎓🚀 📢 Note: I am officially on the job market! If your team is looking for someone passionate about NLP, Clinical AI, and LLM reliability, Agents, RAG and so on, I’d love to chat 🤝 Huge thanks to the organizers for putting this together. Looking forward to some great conversations! 💬🌍 #WomenInAI #MachineLearning #ClinicalAI #NLP #Voxel51 #LLMs #PhDLife #JobMarket #TechTalks #GenAI ✨

21

Launching an Open-Source Research Initiative on LLMs I’m excited to initiate a collaborative open-source research project centered around Large Language Models (LLMs), where we explore cutting-edge topics like fine-tuning, model compression, quantization, quantum-enhanced architectures, and inference optimization. This initiative is open to passionate individuals — students, researchers, engineers — who are serious about AI and want to contribute to impactful, peer-reviewed, and community-driven research. Having been involved in research internships at esteemed institutions like IIT Guwahati, Samsung Research, and others—where I was appointed to work on projects such as Quantum-Enhanced Transformers—I firmly believe that open, interdisciplinary collaboration can lead to meaningful breakthroughs in LLM research. If you’re interested in joining this initiative, please fill out this brief form to help us get connected and aligned: Link: https://lnkd.in/dtVm4Zki Let’s build something impactful, together. #LLM #OpenSourceResearch #ArtificialIntelligence #TransformerModels #QuantumAI #LLMCompression #ResearchCommunity #MachineLearning #Collaboration #NLP

26

I published a system design write-up on rate limiter design — focusing on trade-offs, failure modes, and what actually breaks in production. Instead of treating rate limiting as a textbook problem, I tried to reason about: choosing the right strategy for real workloads distributed rate limiting with Redis failure scenarios like hot keys, clock skew, and Redis outages how I’d approach this in a production system Article: 👉 https://lnkd.in/gZzBrvwX I’m documenting more real engineering work publicly this year.

3

Excited to share that we've had three papers accepted at #EMNLP2025! At Amazon Nova Inference Science team, we have been working on making large language models run faster and more efficiently. Our accepted works: 📝 "Model-free Speculative Decoding for Test-Time Scaling" (https://lnkd.in/g9bmgZRM) - A simple model-free drafter based on n-grams can outperform sophisticated approaches like Eagle-2 for accelerating multi-step reasoning 📝 "Mamba Drafters" (https://lnkd.in/gA9rGBc9) - State-space models as efficient drafters for speculative decoding, specially for longer context 📝 "Think Clearly: Redundant Token Pruning for Reasoning" (https://lnkd.in/gxBtvQhd) - Training-free method to remove redundant tokens from long reasoning chains Thankful to my amazing collaborators Woomin Song, Daewon Choi, Jinwoo Shin, Aram Galstyan, Sravan Bodapati (hiring), Amazon Science and others.

137

4 Comments

🚀 Excited to share our NeurIPS acceptance Our paper “Explanation Driven Counterfactual Testing for Faithfulness in Vision Language Model Explanations” by Sihao Ding, Santosh V. and Aditi Ramadwar (Mercedes-Benz Research & Development North America, Inc.) has been accepted to the NeurIPS 2025 Regulatable ML Workshop, and we will be presenting our poster next month. 👉 EDCT provides a structured protocol for verifying VLM explanations and helps close the gap between a "model sounding correct" and a "model actually reasoning correctly" about the scene. Read the full paper here: https://lnkd.in/gpsYhTsm 💡 Why this work matters: Vision Language Models are starting to play a larger role in high stakes domains like autonomous driving, robotics and safety critical perception. These models often provide natural language explanations for their decisions, but those explanations can sound correct even when they do not reflect the model’s actual reasoning. This creates major reliability and safety challenges. 💡 What we introduce: We propose Explanation Driven Counterfactual Testing (EDCT), a method that treats the model’s own explanation as a testable hypothesis. We extract key visual concepts from the explanation, generate targeted counterfactual edits to the image and then re-query the VLM to evaluate whether both the answer and the explanation respond in a logically consistent way. This gives us a measurable Counterfactual Consistency Score (CCS) that captures how faithful and causally aligned a model’s explanation truly is. 💡 Why this is important for autonomous safety: As teams explore VLM based end to end driving systems, auditing the reasoning behind a model’s decisions becomes essential. For example, if a system claims it slowed down because a cyclist signaled left, we should be able to minimally change that signal and verify that the model’s answer and explanation change accordingly. If not, there is a safety gap. Looking forward to presenting this work at NeurIPS and connecting with others working on regulatable, multimodal models and autonomous safety. #NeurIPS2025 #RegulatableML #VisionLanguageModels #VLMs #AIExplainability #SafetyCriticalAI #AutonomousVehicles

69

6 Comments