Why Use Deterministic LLM Testing Methods

Many people think temperature = 0 guarantees determinism in LLMs. It doesn’t. Even with the same prompt and same settings, production systems can still generate different outputs. The reason lies in how inference actually runs on GPUs. LLM inference runs on massively parallel GPU computations. In production systems, requests are often batched together dynamically. Small floating-point variations during these computations can slightly change token probabilities. If two tokens have nearly identical probabilities, even a tiny numerical difference can flip which token gets selected. Once a single token changes, the rest of the response diverges. That’s why temperature = 0 does not guarantee true determinism. So how do you ensure the same answer every time? You remove every source of randomness: 📍Use deterministic decoding (greedy decoding) 📍Fix a random seed 📍Disable sampling 📍Or cache responses for identical prompts In many real-world LLM systems, response caching is actually the most reliable solution. It guarantees identical outputs while also reducing latency and cost. #ai #llms #aiengineering #datascience #latency #inference Follow Sneha Vijaykumar for more...😊

Ask OpenAI models the same question twice at temperature=0, and you’d expect the same output every time. After all, greedy decoding should be deterministic. Yet in practice, it isn’t. The common explanation has been “floating-point math plus GPU concurrency.” But as Horace He and the team at Thinking Machines Lab argue in their recent deep dive, that story is incomplete. The real culprit is batch invariance. Most inference engines tie your output not only to your request, but also to the server’s load at that instant. A different batch size changes the reduction strategy inside kernels like RMSNorm, matmul, or attention. That shift cascades into subtle, yet real, differences in outputs - even under greedy decoding. Their work demonstrates batch-invariant kernels that restore bitwise reproducibility. The impact is far from academic: 1) It enables true on-policy reinforcement learning, where training and inference remain perfectly aligned. 2) It reframes non-determinism from an unavoidable nuisance into a fixable systems-design issue. Too often, the default reaction in Machine Learning is to relax tolerances and move on. This work reminds us that non-determinism isn’t just noise; it’s a bug we can eliminate with careful engineering. If reproducibility is the bedrock of science, deterministic inference should be the foundation of reliable AI. This research makes that vision tangible. Full article: https://lnkd.in/gPUxX8xE #ArtificialIntelligence #MachineLearning #DeepLearning #AIResearch #LLM #MLOps #SystemsDesign #DistributedSystems #HighPerformanceComputing #GPUComputing #DeterministicAI #ReproducibleAI #Determinism #Reproducibility #ReliableAI #AICommunity #DataScience #OpenSourceAI #EngineeringExcellence #ResearchInnovation #FutureofAI #AIEngineering #AIforScience #TechInnovation #NextGenAI

Thinking Machines Lab just solved a major problem in LLM inference: they identified that nondeterminism comes from batch size variations affecting kernel computations, not the commonly blamed "floating point + concurrency" issues. They built batch-invariant implementations of key operations (RMSNorm, matrix multiplication, attention) and demonstrated true deterministic inference with open-source code. This is exceptional work 🔥 Goes way beyond the usual hand-waving explanations to pinpoint the real culprit: lack of batch invariance. When server load changes batch sizes, it actually changes the numerical results for individual requests - something most people don't realize. The writing is clear, they provide working implementations, run real experiments, and show how to solve the problem. This is how industry research should be shared - not buried in PR papers (who say OPENAI? )

LLM-as-a-Judge is a dangerous way to do evaluation. In most cases, it just doesn't work. While it gives a false impression of having a grasp on your system's performance, it lures you with general metrics such as correctness, faithfulness, or completeness. They hide several complexities: - What does "completeness" mean for your application? In the case of a marketing AI assistant, what characterizes a complete post from an incomplete one? If the score goes higher, does it mean that the post is better? - Often, these metrics are scores between 1-5. But what is the difference between a 3 and a 4? Does a 5 mean the output is perfect? What is perfect, then? - If you "calibrate" the LLM-as-a-judge with scores given by users during a test session, how do you ensure the LLM scoring matches user expectations? If I arbitrarily set all scores to 4, will I perform better than your model? However, if LLM-as-a-judge is limited, it doesn't mean it's impossible to evaluate your AI system. Here are some strategies you can adopt: - Online evaluation is the new king in the GenAI era Log and trace LLM outputs, retrieved chunks, routing… Each step of the process. Link it to user feedback as binary classification: was the final output good or bad? Then take a look at the data. Yourself, no LLM. Take notes and try to find patterns among good and bad traces. At this stage, you can use an LLM to help you find clusters within your notes, not the data. After taking this time, you'll already have some clues about how to improve the system. - Evaluate deterministic steps that come before the final output Was the retrieval system performant? Did the router correctly categorize the initial query? Those steps in the agentic system are deterministic, meaning you can evaluate them precisely: Retriever: Hit Rate@k, Mean Reciprocal Rank, Precision, Recall Router: Precision, Recall, F1-Score Create a small benchmark, synthetically or not, to evaluate offline those steps. It enables you to improve them later on individually (hybrid search instead of vector search, fine-tune a small classifier instead of relying on LLMs…) - Don't use tools that promise to externalize evaluation Your evaluation system is your moat. If you externalize it, not only will you have no control over your AI application, but you also won't be able to improve it. This is your problem, your users, your revenue. Evaluate your system. Not a generic one. All problems are different. Yours is unique as well. ... Those are some unequivocal ideas proposed by the AI community. Yet, I still see AI projects relying on LLM-as-a-judge and generic metrics among companies. Being able to evaluate your system gives you the power to improve it. So take the time to create the perfect evaluation for your use case.

We’re starting to make 𝗟𝗟𝗠𝘀 𝗺𝗼𝗿𝗲 𝗱𝗲𝘁𝗲𝗿𝗺𝗶𝗻𝗶𝘀𝘁𝗶𝗰 - and that’s a big deal. Normally, when you ask the same LLM the same question multiple times, you can still get slightly different answers. Even at temperature 0, results are often non-deterministic. This isn’t because of the nature of the hardware (GPU, CPU, or TPU), nor it is of the LLM. It comes down to how inference requests are orchestrated. Every request gets batched with others in unpredictable ways to optimize hardware efficiency. Recently, Mira Murati and her company Thinking Machines Lab proposed a set of solutions: • Batch-invariant RMSNorm • Batch-invariant matrix multiplication • Batch-invariant attention These approaches achieved deterministic outputs over 𝟭,𝟬𝟬𝟬 𝘁𝗼𝗸𝗲𝗻𝘀 (vs. ~100 with traditional systems) - though with 𝗻𝗲𝗮𝗿𝗹𝘆 𝟮𝘅 𝘁𝗵𝗲 𝗶𝗻𝗳𝗲𝗿𝗲𝗻𝗰𝗲 𝘁𝗶𝗺𝗲. 𝗪𝗵𝘆 𝘁𝗵𝗶𝘀 𝗺𝗮𝘁𝘁𝗲𝗿𝘀 Here’s how this could reshape how we think about AI agents: 1️⃣ 𝗥𝗲𝗴𝘂𝗹𝗮𝘁𝗲𝗱 𝗶𝗻𝗱𝘂𝘀𝘁𝗿𝗶𝗲𝘀 → Determinism is a game-changer for finance, healthcare, or legal use cases where consistency and reliability are non-negotiable. 2️⃣ 𝗦𝗶𝗺𝗽𝗹𝗲𝗿 𝗮𝗴𝗲𝗻𝘁 𝗱𝗲𝘀𝗶𝗴𝗻 → With deterministic outputs, you don’t need as many guardrails to enforce consistency. 3️⃣ 𝗨𝘀𝗲𝗿 𝘁𝗿𝘂𝘀𝘁 → People trust systems more when they deliver the same result every time. If latency isn’t a dealbreaker, deterministic LLMs could become the default choice for mission-critical applications. Link to article in the comments below! #ai #llm #aipm

Defeating Nondeterminism in LLM Inference - Thinking Machines just released their first blog and I think it is very good! Thinking Machines is a new AI lab founded in early 2025 by former OpenAI CTO Mira Murati and several OpenAI alums. Backed by $2B in seed funding, their mission is bold: rebuild the AI stack to be more transparent, deterministic, and customizable; starting from the kernels up. The blog is not a product announcement. No hype. Just a surgical teardown of a core flaw in LLM infrastructure: nondeterministic inference. Even with temperature 0, LLM outputs can change between runs. Most blame floating point math or GPU randomness. But the real culprit? Batch-size-dependent numerics. Your output can shift based on how many other users hit the server. That’s not randomness; it’s poor kernel design. Their fix: make key ops like matmul, RMSNorm, and attention batch-invariant. It costs a bit of performance, but you gain something more important: trust. If this is how Thinking Machines opens, I’m watching what comes next. Read the blog here: https://lnkd.in/gnuxvveX