Simulation-based Experimentation

What if A/B tests are already too late? A/B tests are the gold standard of decision-making. But by the time we run them, they have already affected real users. So here's the question: "Can we experiment before the experiment even starts?" Lately, I've been working at an internal Python package (a Back-Testing Engine) that simulates the impact of code changes before they ever reach production. The engine "replays" past campaigns using different production logic and ML models, letting us estimate the impact of new ideas on our key business metrics. It sits at the intersection of: - Engineering. - Machine Learning. - Optimization. - And ultimately… decision quality. Instead of just asking "Did this work?" after the test, we can ask "Is this worth testing at all?" before we ship. This doesn't replace A/B testing, it upgrades it. Fewer risky launches, better candidates, higher signal experiments. In the Yelp Engineering Blog, I shared a deep dive into how this system works and how it's helping us make safer, smarter decisions. If you care about experimentation, simulation, or building better decision systems at scale, I hope you enjoy the deep dive. ☕ Article link in the first comment. #YelpEngineering #MachineLearning #Experimentation #Simulation #DataScience

Trial and error at 7,000 Liters is an expensive mistake. When product quality is sensitive to every RPM, "guessing" the right agitation isn't an option. Moving from a 0.7 L lab flask to a 7,000 L production tank isn't just a change in size—it's a complete change in physics. When you are scaling by a factor of 10,000, here is how engineers tackle the three fundamental questions of scale-up: 1️⃣. Which reactor configuration is optimal? The goal is often to use existing assets to reduce capital expenditure. Simulation allows you to compare various available vessels and agitation systems virtually, identifying which configuration best replicates the laboratory environment without needing to build new infrastructure. 2️⃣. How do you scale agitation for sensitive products? Laboratory tests often show that product quality is highly sensitive to mixing. Too much agitation can shear the product or cause degradation. Too little agitation can lead to poor yields or inconsistent quality. The solution lies in maintaining mixing power per unit mass. By modeling the lab setup, we establish a baseline. We then simulate the production tank and adjust the impeller speed until the power density matches the lab results exactly. 3️⃣ What is the impact on mixing time? Mixing time changes drastically as volume increases. Predicting this time accurately ensures that the transition to the 7,000 L tank remains predictable, ensuring that homogeneity is achieved within the necessary process windows. By using a mathematical and simulation-based approach, what used to be a risky "leap of faith" becomes a predictable, calculated transition. Scale Up Methodology for the Fine Chemical Industry - The Influence of the Mixing in the Process: https://lnkd.in/dSSjRZTx #ProcessEngineering #ChemicalEngineering #ScaleUp #VisiMix #Agitation #MixingTechnology #ManufacturingOptimization #TechTransfer #ResearchAndDevelopment #ChemicalManufacturing #SimulationSoftware

AgentA/B is a fully automated A/B testing framework that replaces live human traffic with large-scale LLM-based agents. These agents simulate realistic, intention-driven user behaviors on actual web environments, enabling faster, cheaper, and risk-free UX evaluations, even on real websites like Amazon. Key Insights: • Modular agent simulation pipeline – Four components—agent generation, condition prep, interaction loop, and post-analysis—allow plug-and-play simulations on live webpages using diverse LLM personas. • Real-world fidelity – The system parses live DOM into JSON, enabling structured interaction loops (search, filter, click, purchase) executed via LLM reasoning + Selenium. • Behavioral realism – Simulated agents show more goal-directed but comparable interaction patterns vs. 1M real Amazon users (e.g., shorter sessions but similar purchase rates). • Design sensitivity – A/B test comparing full vs. reduced filter panels revealed that agents in the treatment condition clicked more, used filters more often, and purchased more. • Inclusive prototyping – Agents can represent hard-to-reach populations (e.g., low-tech users), making early-stage UX testing more inclusive and risk-free. • Notable results: - Simulated 1,000 LLM agents with unique personas in a live Amazon shopping scenario. - Agents in the treatment condition spent more ($60.99 vs. $55.14) and purchased more products (414 vs. 404), confirming the utility of interface changes. - Behavioral alignment with humans was strong enough to validate simulation-based testing. - Only the purchase count difference reached statistical significance, suggesting further sample scaling is needed. AgentA/B shows how LLM agents can augment — not replace — traditional A/B testing by offering a new pre-deployment simulation layer. This can accelerate iteration, reduce development waste, and support UX inclusivity without needing immediate live traffic.

𝙒𝙝𝙖𝙩 𝙞𝙛 𝙮𝙤𝙪𝙧 𝙛𝙖𝙘𝙩𝙤𝙧𝙮 𝙤𝙣𝙡𝙮 𝙬𝙤𝙧𝙠𝙨 𝙗𝙚𝙘𝙖𝙪𝙨𝙚 𝙧𝙚𝙖𝙡𝙞𝙩𝙮 𝙝𝙖𝙨𝙣’𝙩 𝙩𝙚𝙨𝙩𝙚𝙙 𝙞𝙩 𝙮𝙚𝙩? Most plants look stable— until demand shifts, a resource slips, or variability shows up where no one expected it. That’s when leaders realize the system wasn’t designed for reality. It was designed for assumptions. This is why simulation-based decision making—especially Discrete Event Simulation (DES)—has become essential for smart plants. Not to predict the future. But to stress-test the system before the system is forced to respond. Here’s what DES actually validates—end to end: 1️⃣ 𝙋𝙧𝙤𝙘𝙚𝙨𝙨 𝙁𝙡𝙤𝙬 𝙊𝙥𝙩𝙞𝙢𝙞𝙯𝙖𝙩𝙞𝙤𝙣 DES shows how material and information truly move—not how the routing sheet claims they do. 2️⃣ 𝙀𝙦𝙪𝙞𝙥𝙢𝙚𝙣𝙩 𝙐𝙩𝙞𝙡𝙞𝙯𝙖𝙩𝙞𝙤𝙣 𝘼𝙣𝙖𝙡𝙮𝙨𝙞𝙨 High utilization can hide starvation and blocking. DES exposes when assets look busy but flow is unhealthy. 3️⃣ 𝘽𝙤𝙩𝙩𝙡𝙚𝙣𝙚𝙘𝙠 𝙄𝙙𝙚𝙣𝙩𝙞𝙛𝙞𝙘𝙖𝙩𝙞𝙤𝙣 Constraints aren’t static. DES reveals where the bottleneck migrates under different conditions. 4️⃣ 𝙋𝙧𝙤𝙙𝙪𝙘𝙩𝙞𝙤𝙣 𝘾𝙖𝙥𝙖𝙘𝙞𝙩𝙮 𝙋𝙡𝙖𝙣𝙣𝙞𝙣𝙜 Capacity isn’t a fixed number. DES models how throughput behaves under variability, downtime, and mix changes. 5️⃣ 𝘽𝙪𝙛𝙛𝙚𝙧 𝙎𝙞𝙯𝙞𝙣𝙜 Too much buffer masks instability. Too little amplifies it. DES finds the point where flow stays resilient. 6️⃣ 𝘾𝙮𝙘𝙡𝙚 𝙏𝙞𝙢𝙚 𝘿𝙞𝙨𝙩𝙧𝙞𝙗𝙪𝙩𝙞𝙤𝙣 Averages lie. DES reveals the spread—and where volatility is introduced. 7️⃣ 𝙍𝙚𝙨𝙤𝙪𝙧𝙘𝙚 𝘼𝙡𝙡𝙤𝙘𝙖𝙩𝙞𝙤𝙣 People, machines, and automation interact as a system. DES tests the balance before locking it in. 8️⃣ 𝘿𝙚𝙢𝙖𝙣𝙙 𝙁𝙡𝙤𝙬 𝙊𝙥𝙩𝙞𝙢𝙞𝙯𝙖𝙩𝙞𝙤𝙣 DES connects demand patterns to execution reality—without overloading the system. 9️⃣ 𝙏𝙧𝙞𝙖𝙡 𝘽𝙪𝙞𝙡𝙙 𝙎𝙘𝙚𝙣𝙖𝙧𝙞𝙤 𝘼𝙣𝙖𝙡𝙮𝙨𝙞𝙨 Instead of learning after launch, DES lets teams explore “what if” scenarios before they become problems. 🔟 𝘿𝙖𝙩𝙖-𝘿𝙧𝙞𝙫𝙚𝙣 𝙄𝙣𝙫𝙚𝙨𝙩𝙢𝙚𝙣𝙩 𝘿𝙚𝙘𝙞𝙨𝙞𝙤𝙣𝙨 Every capex decision is validated against system behavior—not isolated ROI logic. This is the real shift leaders are making: 𝙁𝙧𝙤𝙢 𝙩𝙧𝙞𝙖𝙡 𝙗𝙪𝙞𝙡𝙙𝙨 → 𝙩𝙤 𝙫𝙖𝙡𝙞𝙙𝙖𝙩𝙚𝙙 𝙨𝙘𝙚𝙣𝙖𝙧𝙞𝙤𝙨 𝙁𝙧𝙤𝙢 𝙤𝙥𝙞𝙣𝙞𝙤𝙣𝙨 → 𝙩𝙤 𝙚𝙫𝙞𝙙𝙚𝙣𝙘𝙚 𝙁𝙧𝙤𝙢 𝙛𝙞𝙧𝙚𝙛𝙞𝙜𝙝𝙩𝙞𝙣𝙜 → 𝙩𝙤 𝙙𝙚𝙨𝙞𝙜𝙣𝙚𝙙 𝙨𝙩𝙖𝙗𝙞𝙡𝙞𝙩𝙮 Simulation doesn’t improve factories. It reveals whether the system was ever ready. 𝙄𝙛 𝙮𝙤𝙪’𝙧𝙚 𝙨𝙘𝙖𝙡𝙞𝙣𝙜 𝙥𝙧𝙤𝙙𝙪𝙘𝙩𝙞𝙤𝙣, 𝙞𝙣𝙩𝙧𝙤𝙙𝙪𝙘𝙞𝙣𝙜 𝙖𝙪𝙩𝙤𝙢𝙖𝙩𝙞𝙤𝙣, 𝙤𝙧 𝙧𝙚𝙗𝙖𝙡𝙖𝙣𝙘𝙞𝙣𝙜 𝙘𝙖𝙥𝙖𝙘𝙞𝙩𝙮— 𝙩𝙝𝙚 𝙦𝙪𝙚𝙨𝙩𝙞𝙤𝙣 𝙞𝙨𝙣’𝙩 𝙘𝙖𝙣 𝙩𝙝𝙚 𝙡𝙞𝙣𝙚 𝙧𝙪𝙣?

📊 Can we discover new therapeutics entirely in silico? A new study in Nature Computational Science "In silico biological discovery with large perturbation models" https://lnkd.in/eitcTVBs introduces the Large Perturbation Model (LPM), a deep learning framework that learns from thousands of perturbation experiments across CRISPR, drug, and transcriptomic datasets to predict unseen biological outcomes. Every perturbation experiment captures how a cell changes when a gene is silenced or a compound is applied. But integrating these diverse data types has long been a bottleneck. The same gene perturbed in two different cell types or time points often leads to incomparable results. LPM solves this by separating three essential dimensions — Perturbation (P), Readout (R), and Context (C) — and then learning how they interact. The result: a model that doesn’t just interpolate data but learns causal rules connecting interventions to outcomes. 🔍 Key findings: → LPM consistently outperforms leading models (CPA, GEARS, scGPT, Geneformer) in predicting post-perturbation gene expression. → By placing drugs and gene knockouts in a shared latent space, it links compounds to their molecular targets — and flags off-target effects. For example, pravastatin grouped with anti-inflammatory agents, aligning with known biology. → In a virtual screen for polycystic kidney disease (ADPKD), LPM predicted that simvastatin could increase PKD1 expression — a prediction later validated in real-world patient data showing slower disease progression. → When its “virtual experiments” were added to real datasets, causal gene-gene network inference became more accurate. 💡 Why it matters: LPM shows that it’s possible to simulate biology before running an experiment. By training across heterogeneous datasets, it builds a foundation model for biological discovery , one that can generalize across cell types, perturbations, and modalities. 🧠 Potential applications: • Virtual drug discovery and mechanism-of-action prediction • Detecting off-target or synergistic drug effects • Filling in missing perturbation data to strengthen causal network mapping • Prioritizing compounds or pathways for wet-lab validation • Personalizing therapeutic hypotheses based on molecular context This isn’t just about prediction — it’s about transforming how experimental biology is done. Models like LPM turn large-scale perturbation data into a continuous, learnable system. Instead of testing one gene or one compound at a time, researchers can now explore millions of possibilities computationally, then focus resources where the model predicts meaningful biology. As perturbation datasets grow, from CRISPR screens to chemical libraries and multi-omic assays, this approach will redefine discovery pipelines in pharma, functional genomics, and personalized medicine alike. 🧬 Digital models are becoming the new laboratories of discovery.

AI is starting to challenge how we approach Design of Experiments (DoE) in industrial R&D. Traditional DoE requires you to define everything up front… factors, ranges, interactions. ML-guided optimization learns as it goes. Instead of fixing an experimental plan, the model watches results and decides where to test next, focusing on regions where behaviour is changing or still uncertain. Take Monolith AI's "Next Test Recommender" tool. After each batch, they fit a surrogate model, estimate uncertainty, and select the next points to maximize information gain. Classical DoE is planned exploration. ML-guided optimization is adaptive. But this only works if simulations are cheap. If runs are slow, licence-constrained, or organisationally painful, you don’t get enough data to train a useful surrogate. You fall back to intuition. That’s where infrastructure matters. Containerised FMUs (by the Modelica Association) remove licence bottlenecks. API-driven simulations let you parallelise, store, and replay every run. Tools like the Quix FMU runner are built for exactly this: turning simulations into something you can scale and automate. Now you have enough data for the model to actually learn. At that point, experimentation stops being manual. It becomes software that improves with every run. Less tweaking. More exploration. And once exploration is cheap, engineers stop asking “what should I tweak next?” and start asking “what parts of the design space haven’t we explored yet?”

AI Agents Are Now Simulating Human Behavior. Should We Be Excited or Alarmed? At Stanford Institute for Human-Centered Artificial Intelligence (HAI) researchers just built something extraordinary: 1,000 AI agents modeled after real people, each trained on a two-hour interview with an actual U.S. participant. The agents weren’t just role-playing. They replicated survey responses with 85% accuracy - close to how well humans replicate their own answers after two weeks. On personality tests like the Big Five, they outperformed demographic models. In social science experiments, they behaved strikingly like the original participants. This isn’t just a technical feat. It’s a signal. → Simulated people can now stand in for real ones in surveys, economic games, even behavioral experiments. → We’re entering an era where “what if” questions in policy, marketing, and governance can be tested at scale without touching a single human respondent. → And yes, it’s faster, cheaper, and (so far) more equitable: these agents reduced bias across race, gender, and ideology. But there’s a catch. → What happens when the simulation fails quietly? → Who governs consent when a digital proxy mimics your beliefs - accurately or not? → And what safeguards stop someone from fabricating controversy in your name? Stanford’s team made a key decision: no public release. Access is API-gated, privacy-controlled, and monitored. It’s a model of restraint. Others won’t be so careful. The future of social science may be synthetic. But the risk is real. 📌 𝐒𝐚𝐯𝐞. 🔁 𝐒𝐡𝐚𝐫𝐞. 💬 𝐃𝐢𝐬𝐜𝐮𝐬𝐬. 𝐅𝐨𝐥𝐥𝐨𝐰 𝐦𝐞 𝐟𝐨𝐫 𝐧𝐨-𝐟𝐥𝐮𝐟𝐟 𝐢𝐧𝐬𝐢𝐠𝐡𝐭𝐬 𝐨𝐧 𝐞𝐧𝐭𝐞𝐫𝐩𝐫𝐢𝐬𝐞 𝐀𝐈, 𝐚𝐠𝐞𝐧𝐭𝐬, 𝐚𝐧𝐝 𝐥𝐞𝐚𝐝𝐞𝐫𝐬𝐡𝐢𝐩.

The testing process typically follows a sequential order, starting with the least complex and moving toward the most complex and hardware-involved stages. Here's the usual order: 1️⃣MIL (Model-in-the-Loop) Testing: ✳️Done first to validate the model's logic and behavior in a simulated environment. ✳️Ensures the algorithm works as intended before any code is generated. 2️⃣SIL (Software-in-the-Loop) Testing: ✳️Conducted after MIL, once the model is converted into code. ✳️Validates that the generated code behaves the same as the model on a host machine. 3️⃣PIL (Processor-in-the-Loop) Testing: ✳️Performed after SIL, once the code is ready to be tested on the target processor. ✳️Ensures the code runs correctly on the actual hardware or an equivalent emulator. 4️⃣HIL (Hardware-in-the-Loop) Testing: ✳️Conducted last, after PIL, when the software is integrated with the actual hardware. ✳️Validates the full system in a real-time environment with simulated inputs/outputs. ⭕Summary of Order: MIL → SIL → PIL → HIL ✳️This sequence ensures that issues are caught early in the development process, reducing costs and risks as the system moves closer to deployment. Here's a brief overview of the differences between HIL, SIL, PIL, and MIL testing in the context of embedded systems and control software development: ✅MIL (Model-in-the-Loop) Testing: ⏩Tests the model of the system (e.g., Simulink) in a simulation environment. ⏩Focuses on verifying the algorithm and logic of the model. No actual code is generated or executed; it's purely simulation-based. ✅SIL (Software-in-the-Loop) Testing: ⏩Tests the generated code (e.g., C/C++) on a host machine (PC). ⏩Compares the behavior of the generated code with the model to ensure consistency. ⏩No hardware is involved; it validates the software functionality. ✅PIL (Processor-in-the-Loop) Testing: ⏩Tests the generated code on the target processor or an equivalent emulator. ⏩Validates that the code runs correctly on the actual hardware platform. ⏩Focuses on compiler optimizations, memory usage, and processor-specific behavior. ✅HIL (Hardware-in-the-Loop) Testing: ⏩Tests the entire system with real hardware components and simulated inputs/outputs. ⏩Validates the interaction between software, hardware, and the physical system. ⏩Used for final validation before deployment, often in real-time environments. Summary: ✔️MIL: Tests the model in simulation. ✔️SIL: Tests generated code on a host machine. ✔️PIL: Tests generated code on the target processor. ✔️HIL: Tests the full system with real hardware and simulated environments. Each step increases in complexity and hardware involvement, ensuring robustness at every stage of development. #automotive #iso26262 #sil #hil #pil #mil

I’ve spent a lot of time building and studying agent competitions and simulations like Power TAC (The Power Trading Agent Competition) where autonomous agents trade in complex markets under constraints. They are incredibly useful for stress‑testing ideas: you can turn up competition, change rules overnight, and observe emergent behavior at scale. But there’s a trap. It’s easy to assume that strategies that win in a clean simulation will transfer directly into a bank. In practice, once you add regulation, conduct risk, legacy systems, and human workflows, a lot of neat ideas simply don’t survive contact with reality. The way I think about it now: simulation is a wind tunnel, not the real world. It’s a great place to explore design space, debug agent behaviors, and understand failure modes, but you still need a serious layer of governance, observability, and human‑in‑the‑loop control before anything belongs in production. At Artian AI, we use that experience to design agentic systems that behave well not just in a sandbox, but in messy, regulated environments where decisions need an audit trail and a clear owner. Curious how others are using simulation today: is it feeding your operating model, or just your slideware?