Accelerate Hypothesis Testing Using LLM Prototypes

AgentA/B is a fully automated A/B testing framework that replaces live human traffic with large-scale LLM-based agents. These agents simulate realistic, intention-driven user behaviors on actual web environments, enabling faster, cheaper, and risk-free UX evaluations, even on real websites like Amazon. Key Insights: • Modular agent simulation pipeline – Four components—agent generation, condition prep, interaction loop, and post-analysis—allow plug-and-play simulations on live webpages using diverse LLM personas. • Real-world fidelity – The system parses live DOM into JSON, enabling structured interaction loops (search, filter, click, purchase) executed via LLM reasoning + Selenium. • Behavioral realism – Simulated agents show more goal-directed but comparable interaction patterns vs. 1M real Amazon users (e.g., shorter sessions but similar purchase rates). • Design sensitivity – A/B test comparing full vs. reduced filter panels revealed that agents in the treatment condition clicked more, used filters more often, and purchased more. • Inclusive prototyping – Agents can represent hard-to-reach populations (e.g., low-tech users), making early-stage UX testing more inclusive and risk-free. • Notable results: - Simulated 1,000 LLM agents with unique personas in a live Amazon shopping scenario. - Agents in the treatment condition spent more ($60.99 vs. $55.14) and purchased more products (414 vs. 404), confirming the utility of interface changes. - Behavioral alignment with humans was strong enough to validate simulation-based testing. - Only the purchase count difference reached statistical significance, suggesting further sample scaling is needed. AgentA/B shows how LLM agents can augment — not replace — traditional A/B testing by offering a new pre-deployment simulation layer. This can accelerate iteration, reduce development waste, and support UX inclusivity without needing immediate live traffic.

Bayesian Optimization (BO) struggles with cold starts. #ChemBOMAS gives it memory, intuition, and foresight, via LLMs. BO is a trusted method for optimizing chemical reactions, but it can be difficult to get started when there's little to no experimental data or a clear sense of where to explore. Early iterations are often inefficient and slow to converge, especially in complex, high-dimensional spaces. ChemBOMAS, a new LLM-enhanced multi-agent framework, augments BO to behave more like a chemist: informed by knowledge, guided by experience, and strategic in exploration. 🧠 First, memory At the outset, ChemBOMAS uses LLMs to read and synthesize chemical literature to identify which parameters matter most. This knowledge is used to decompose the search space intelligently and prioritize directions, giving BO a meaningful head start. 💡 Next, intuition LLMs cluster reaction components based on physicochemical properties (like steric and electronic effects), guiding the optimizer toward chemically plausible regions instead of random combinations. 🧭 Finally, foresight A fine-tuned LLM predicts outcomes for hypothetical conditions, generating pseudo-experiments that warm start the BO process, leading to faster convergence with fewer real trials. In wet-lab validation, ChemBOMAS significantly outperformed domain experts using traditional methods, especially when rich literature was available for the LLM to leverage. It’s another strong reminder that LLMs aren’t just summarizing papers anymore. They’re shaping how we design and run experiments. 📄 ChemBOMAS: Accelerated BO in Chemistry with LLM-Enhanced Multi-Agent System, arXiv, Sep 10, 2025 🔗 https://lnkd.in/e5CfyPkm

We rebuilt our entire outbound system from scratch. Before sharing what it looks like, a short story: A few months ago I started working with a Series B AI company. Hot product. Huge TAM. The kind of company VCs love. But they had a problem: too many possibilities. Their product worked for dozens of use cases, across multiple personas, in countless industries. They knew what was working - but they had no idea what they were missing. What other industries might resonate? What use cases hadn't they explored? So we started testing. And I immediately hit a wall. The old way - building lists in Clay, writing copy, launching campaigns - couldn't match the pace we needed. We weren't trying to run 1-2 campaigns a month. We wanted to test 10+ hypotheses. Fast. So we rebuilt everything from scratch: 1. Context Repo A single source of truth: company narrative, product positioning, personas, campaign angles, offers. All documented. All version-controlled. 2. Micro-List Building Instead of 5,000-person blasts, we build targeted lists of a few hundred people per hypothesis programmatically. Each list tests specific traits we have theories about. 3. Programmatic Copywriting We feed the LLM the exact context it needs—campaign brief, persona, offer—pulled directly from the repo. No manual copywriting bottleneck. 4. Cost Optimization Rapid experimentation = high token volume. So we built prompt caching and batching that saves 60-80% on costs. 5. Audit Trails Every decision is documented. If we targeted the wrong pain point, then we can simply tweak the campaign brief, rerun. The system learns. The result: We now scientifically test outbound hypotheses with full traceability. We're not guessing. We're not hoping our instincts are right. We're not rerunning a playbook that once worked for a similar company. We're systematically exploring the space to find what actually works - not just what worked before for someone else. The difference between a local maximum and a global maximum. And it's why I believe the future of GTM isn't more SDRs running their own experiments in silos. It's a centralized system that controls the testing - so you're not leaving hypothesis exploration to chance. --- If you're in a similar situation as our client - drowning in hypotheses and frustrated with how slow it is to actually test them or if any of this resonates with challenges you're facing - happy to chat, DM me. I'm always down to go deeper on how we're thinking about this and how we've set things up.

Don't ask an LLM to do your evals. Instead, use it to accelerate them. LLMs can speed up parts of your eval workflow, but they can’t replace human judgment where your expertise is essential. Here are some areas where LLMs can help: 1. First-pass axial coding: After you’ve open coded 30–50 traces yourself, use an LLM to organize your raw failure notes into proposed groupings. This helps you quickly spot patterns, but always review and refine the clusters yourself. Note: If you aren’t familiar with axial and open coding, see this faq: https://lnkd.in/gpgDgjpz 2. Mapping annotations to failure modes: Once you’ve defined failure categories, you can ask an LLM to suggest which categories apply to each new trace (e.g., “Given this annotation: [open_annotation] and these failure modes: [list_of_failure_modes], which apply?”). 3. Suggesting prompt improvements: When you notice recurring problems, have the LLM propose concrete changes to your prompts. Review these suggestions before adopting any changes. 4. Analyzing annotation data: Use LLMs or AI-powered notebooks to find patterns in your labels, such as “reports of lag increase 3x during peak usage hours” or “slow response times are mostly reported from users on mobile devices.” However, you shouldn’t outsource these activities to an LLM: 1. Initial open coding: Always read through the raw traces yourself at the start. This is how you discover new types of failures, understand user pain points, and build intuition about your data. Never skip this or delegate it. 2. Validating failure taxonomies: LLM-generated groupings need your review. For example, an LLM might group both “app crashes after login” and “login takes too long” under a single “login issues” category, even though one is a stability problem and the other is a performance problem. Without your intervention, you’d miss that these issues require different fixes. 3. Ground truth labeling: For any data used for testing/validating LLM-as-Judge evaluators, hand-validate each label. LLMs can make mistakes that lead to unreliable benchmarks. 4. Root cause analysis: LLMs may point out obvious issues, but only human review will catch patterns like errors that occur in specific workflows or edge cases—such as bugs that happen only when users paste data from Excel. Start by examining data manually to understand what’s going wrong. Use LLMs to scale what you’ve learned, not to avoid looking at data. Read this and other eval tips here: https://lnkd.in/gfUWAjR3

A new preprint introduces a Socratic Method Agent that guides LLMs through structured questioning, i.e., definition, analogy, hypothesis elimination, and more following lessons learned from thousands of years of philosophy and reasoning. Why it matters: ➡️ The prompted LLM saw significantly improved performance, achieving SOTA (97.15%) on the ARC Challenge without fine-tuning or external tool usage. ➡️ The approach showed consistent gains in reasoning depth, clarity, and domain-specific insight across chemistry and materials science. ➡️ You can implement this approach with a single, reusable system prompt to deliver a deeper “chain-of-reasoning”. Links to the paper and prompt in the comments. Add your thoughts there! Wonderful work by the team at Argonne National Laboratory, University of Illinois Chicago, and Northwestern University including Hassan Harb, Rajeev Assary, Brian Ingram and others

Stanford Researchers Developed POPPER: An Agentic AI Framework that Automates Hypothesis Validation with Rigorous Statistical Control, Reducing Errors and Accelerating Scientific Discovery by 10x Researchers from Stanford University and Harvard University introduced POPPER, an agentic framework that automates the process of hypothesis validation by integrating rigorous statistical principles with LLM-based agents. The framework systematically applies Karl Popper’s principle of falsification, which emphasizes disproving rather than proving hypotheses. POPPER was evaluated across six domains: biology, sociology, and economics. The system was tested against 86 validated hypotheses, with results showing Type-I error rates below 0.10 across all datasets. POPPER demonstrated significant improvements in statistical power compared to existing validation methods, outperforming standard techniques such as Fisher’s combined test and likelihood ratio models. In one study focusing on biological hypotheses related to Interleukin-2 (IL-2), POPPER’s iterative testing mechanism improved validation power by 3.17 times compared to alternative methods. Also, an expert evaluation involving nine PhD-level computational biologists and biostatisticians found that POPPER’s hypothesis validation accuracy was comparable to that of human researchers but was completed in one-tenth the time. By leveraging its adaptive testing framework, POPPER reduced the time required for complex hypothesis validation by 10, making it significantly more scalable and efficient..... Read full article: https://lnkd.in/gmz_REHS Paper: https://lnkd.in/gHvMqxV8 GitHub Page: https://lnkd.in/g3xNGEcZ Stanford University Kexin Huang Jure Leskovec

Remember when I said your senior engineers should be a big part of discovery? Here's what actually happens when they are. They run experiments. Ugly ones. Fast ones. Brutally honest ones. And that's precisely why it works. See, engineers have a superpower that nobody talks about. They can smell expensive failure from a mile away. So when you put them in charge of discovery, they don't build elaborate anything. They ask one question: "What's the fastest way to find out if this will actually work?" The answer is always an experiment. But not the kind you're thinking of. Here's the manifesto for AI experiments that actually accelerate discovery: 1. Embrace beautiful ugliness Use anything - API playgrounds, messy notebooks, and scripts held together with digital duct tape. If you're thinking about code quality, you're thinking too far ahead. This is discovery, not delivery. 2. One question to rule them all Each experiment answers exactly one question. Not "Can we build a support bot?" But "Can Claude 4 reliably extract invoice numbers from our top 20 email formats with less than 5% error?" Surgical precision or go home. 3. Data should be your love language A messy CSV showing 92% success rate? That's gold. You're not hunting for applause or beautiful demos. You're hunting for truth. Because truth is what you build on. This approach transforms discovery from a months-long mystery into a series of rapid revelations. Monday: "Can we do this?" Wednesday: "Here's the data." Friday: "Now we know exactly what we're building and why." You get cost estimates based on reality, not hope. You can compare different models in an afternoon. This is how you kill bad ideas while they're still cheap. (And trust me, in the world of LLMs, bad ideas get expensive fast.) The counterintuitive truth? These quick, ugly experiments don't slow you down - they're your fast-forward button. Every failed experiment saves you from a failed feature. Every surprising result redirects you toward what actually works. We've seen two-day experiments save six-month projects. We've watched ugly code prevent beautiful disasters. The messier the experiment, the cleaner the final product. So next time someone says "we need to validate this idea," hand it to your dev team and ask them to make it ugly. The uglier the experiment, the prettier your time-to-market. What's your take? Beautiful fiction or ugly truth? --- If you liked it, follow along for more tales from the LLM trenches. #LLMLessonsLearned #QuantumLemon

Our AI plan for our engineering team this month 👇 Sharing this to remove some of the “magic” around AI and make the process more transparent. In our view, there isn’t enough practical sharing of how agencies actually use LLMs in day-to-day operations. There’s plenty of “AI will change the world,” and not enough “this is how we run it in production right now.” So here’s ours: ▪️ We listed ~150 common scenarios Dreamers of Day developers perform when building large-scale websites. The data for these tasks came from commit messages in the code repository, Trello cards, and other historical project artifacts ▪️ We test AI agents (mostly Opencode with OpenAI Codex 5.2 + Claude Code) against each of the 150 scenarios, for example, “add an accordion block with accessibility checks”, to measure how consistently they reach the intended outcome. ▪️ When a scenario fails, we improve the agent documentation, add clearer examples, and expand its context. It’s a continuous loop: test → refine context → retest, repeated until the agent produces the result we’re aiming for. ▪️ Success? We move on to the next scenario. What we believe ▪️ Most LLM failures are due to context issues, not fundamental model limitations. ▪️ The stronger our internal building blocks (pre-checked and vetted website components) and context are, the better the outputs. It’s like giving the agent a big box of well-organized LEGO pieces instead of a random pile. ▪️ Clear instructions, rich context, and strong human-made examples dramatically improve reliability. ▪️ Same thought process / potential gains apply to marketing, business development, project management, and operations inside our agency. Also important ▪️ A large portion of our automation/acceleration at Dreamers of Day isn’t LLM-driven at all. Many wins come from straightforward API integrations, simple scripts, and other “boring” robotic workflows. I wouldn't sleep on these basics. Would love to hear what other agencies are actually running in production!

A Head-to-Head Look: Frontier AI Models Tackle Cancer Research Prototyping 🔬 Direct comparisons of frontier AI models on complex tasks aren't common. Curious about their use in cancer research, I challenged four leading models—Gemini 2.5 Pro (just released), Claude Sonnet 3.7, o1 Pro, and DeepSeek R1—to create a multi-agent prototype for accelerating early lung cancer research. The goal: AI agents collaborating with human experts. Here’s a brief look at how they performed: 🥇 Claude Sonnet 3.7 stood out, delivering the most extensive code (2200+ lines!!) for its 6 core AI agents (plus a Human Controller interface). Remarkably, the code worked immediately (!!), produced seemingly meaningful output in the form of research hypotheses, included direct LLM API integration logic, provided a clear visual schematic, and offered a detailed roadmap. 🥈 Gemini 2.5 Pro provided substantial code (400+ lines) implementing 7 distinct agents. Its structure executed correctly but needed future implementation for LLM calls (marked as 'TODOs'). It included a very detailed Mermaid diagram schematic and a thorough roadmap. 🥉 o1 Pro generated moderate, working code (260+ lines) outlining a workable structure with 6 agents. Like Gemini, LLM integration was simulated or conceptual. It also provided a Mermaid diagram schematic and a solid roadmap outline. 🏅 DeepSeek R1 offered minimal code (160+ lines) for its 5 agents, requiring troubleshooting. It integrated LlamaIndex for literature tasks (local files initially) but other LLM interactions were simulated. It included a text-based schematic and a roadmap outlining necessary future work. Disclaimer: This wasn't intended as a rigorous benchmark. It was an initial attempt at real-world testing where all models received the exact same prompt. Better prompting strategies would likely improve results across the board. 💡 Overall Reflections: All four models conceptualized the multi-agent framework for cancer research well, providing varied schematics to visualize their designs. However, implementation depth differed greatly. Claude Sonnet 3.7 delivered a more complete, functional prototype with coded LLM API logic. DeepSeek integrated a relevant tool (LlamaIndex), while Gemini and o1 Pro provided structural starting points requiring more development for core AI/external interactions. Interestingly, none used existing agent libraries (like Autogen or Crew AI); instead, they all built their custom frameworks. This comparison highlights the differing capabilities of current AI in generating complex scientific code, particularly regarding functional completeness and API integration readiness. The potential for AI to become a genuine partner in cancer research is clearly growing fast.