Alternatives to A/b Testing Methods

Not every user interaction should be treated equally, yet many traditional optimization methods assume they should be. A/B testing, the most commonly used approach for improving user experience, treats every variation as equal, showing them to users in fixed proportions regardless of performance. While this method has been widely used for conversion rate optimization, it is not the most efficient way to determine which design, feature, or interaction works best. A/B testing requires running experiments for a set period, collecting enough data before making a decision. During this time, many users are exposed to options that may not be effective, and teams must wait until statistical significance is reached before making any improvements. In fast-moving environments where user behavior shifts quickly, this delay can mean lost opportunities. What is needed is a more responsive approach, one that adapts as individuals utilize a product and adjusts the experience in real time. Multi-Armed Bandits does exactly that. Instead of waiting until a test is finished before making decisions, this method continuously tests user response and directs more people towards better-performing versions while still allowing exploration. Whether it's testing different UI elements, onboarding flows, or interaction patterns, this approach ensures that more users are exposed to the most optimal experience sooner. At the core of this method is Thompson Sampling, a Bayesian algorithm that helps balance exploration and exploitation. It ensures that while new variations are still tested, the system increasingly prioritizes what is already proving successful. This means conversion rates are optimized dynamically, without waiting for a fixed test period to end. With this approach, conversion optimization becomes a continuous process, not a one-time test. Instead of relying on rigid experiments that waste interactions on ineffective designs, Multi-Armed Bandits create an adaptive system that improves in real time. This makes them a more effective and efficient alternative to A/B testing for optimizing user experience across digital products, services, and interactions.

Is A/B testing really the "gold standard" in data science? We've all heard it: A/B testing is the gold standard for measuring impact. It's what data scientists preach and what execs use to make big investment decisions. But recent research reveals a crucial blind spot in this conventional wisdom. Abadie and Zhao (2025) show: when you're working with smaller datasets - think a handful of cities or store locations rather than millions of users - A/B testing loses its edge. This smaller data scenario is often the norm in marketing and data science. 𝗪𝗵𝘆 𝗿𝗮𝗻𝗱𝗼𝗺𝗶𝘇𝗮𝘁𝗶𝗼𝗻 𝗯𝗿𝗲𝗮𝗸𝘀 𝗱𝗼𝘄𝗻 With only a few big cities, a random draw can easily pick quirky "treated" geos that don't look like the rest. Randomization is unbiased on average across many draws, but in the one assignment you actually run, baseline imbalances can be huge. 𝗪𝗵𝗮𝘁 𝘁𝗵𝗲𝘆 𝗳𝗶𝗻𝗱 Using Walmart sales data, they find synthetic control experiments dramatically outperform randomization: 1-5% error vs standard A/B testing: 20-45% error. 𝗪𝗵𝗮𝘁 𝗺𝗮𝗸𝗲𝘀 𝘀𝘆𝗻𝘁𝗵𝗲𝘁𝗶𝗰 𝗰𝗼𝗻𝘁𝗿𝗼𝗹 𝗲𝘅𝗽𝗲𝗿𝗶𝗺𝗲𝗻𝘁𝘀 𝗱𝗶𝗳𝗳𝗲𝗿𝗲𝗻𝘁 Instead of hoping randomization creates balanced groups, synthetic control experiments take a deliberate approach. They carefully select treatment locations that represent your target market, then weight the control group to match that treated mix. Traditional synthetic control (also by Abadie) takes treatment assignment as given - this new approach optimizes which units get treated. A/B testing works at scale. But with small data, there's a better way.

Struggles of doing data science in the real world 🤦: What do you do when there’s no A/B test but you still need insights? I recently faced that challenge (again): 👉 The growth team asked me to evaluate the impact of a new mobile app feature on conversions (a week after it launched) In the real world, data is messy, and A/B tests aren’t always an option. As a Data Scientist, you need to learn to be resourceful Here’s how I approached it: 1️⃣ Segmented analysis: I created pre- and post-launch groups based on user signup dates. 2️⃣ Exploratory data analysis (EDA): Visualized conversion trends, layering in cohort and seasonal comparisons. 3️⃣ Statistical testing: Ran an independent t-test to validate observed changes, carefully checking assumptions like normality and variance equality. Result? A clear signal of increased conversions on iOS, while Android showed minimal impact. 💡 Key takeaway: T-tests (or similar methods) can still deliver actionable insights outside traditional A/B testing, but validating assumptions and adding context is critical to making reliable conclusions. I broke down my full workflow and the lessons learned in my latest newsletter article (If you’re curious, check the link in the comments👇) What’s your go-to method for analyzing feature impacts without a perfect experimental setup?

I've refused A/B price tests for 16 years. But even Shopify's easy feature hides a brutal truth: Showing different customers different prices for the same product at the same time is an ethical minefield. In the EU, it can get you fined. A French customer paying more than a Belgian customer for identical products violates consumer protection law, even in random A/B tests. In the US, the Robinson-Patman Act creates restrictions most consumer businesses think they avoid. The legal grey area alone should make any brand leader pause. Then there's trust. When customers discover they paid more than someone else for the exact same product at the exact same time, resentment builds fast. And once lost, that trust is expensive to rebuild. Most consumers view price discrimination as fundamentally unfair, even when it's legal. So what works instead? Pricing research. Van Westendorp, conjoint analysis, and Gabor-Granger all reveal willingness to pay without charging real customers different prices. One telecom company used segmentation research to discover their "mid-market" was actually three distinct groups with wildly different price sensitivity. The insight drove 10%+ revenue growth. Zero customers felt cheated. A/B testing prices gives you data. Pricing research gives you data without burning trust.

📈 Econometric Corner: #18 📊 A/B tests have proven to be highly valuable for firms. However, the limitations of these simple A/B tests have been increasingly recognized, of which the two most prominent being addressing interference and estimating heterogeneous effects. In this citied paper, the authors address these two challenges by developing a theoretical framework for the optimal design and analysis of switchback experiments with minimal assumptions. In these experiments, a unit is sequentially exposed to random treatments, and its response is measured over a fixed period. Administering alternate treatments to the same unit enables direct estimation of individual-level causal effects and mitigates interference challenges. 𝗦𝗲𝘁𝘂𝗽: The authors use the variation from the random assignment path for inference (i.e., design-based inference). They focus on regular switchback experiments and exclude adaptive treatment assignments. 𝗧𝘄𝗼 𝗮𝘀𝘀𝘂𝗺𝗽𝘁𝗶𝗼𝗻𝘀 that limit the dependence of the potential outcomes on assignment paths: 𝗔𝘀𝘀𝘂𝗺𝗽𝘁𝗶𝗼𝗻 𝟭 (non-anticipating potential outcomes) states that the potential outcomes at time, 𝘵, do not depend on future treatments. 𝗔𝘀𝘀𝘂𝗺𝗽𝘁𝗶𝗼𝗻 𝟮 (m-carryover effects) restricts the order of the carryover effect. 𝗘𝘀𝘁𝗶𝗺𝗮𝗻𝗱: the average lag-𝘱 causal effect of consecutive treatments on the outcome, where 𝘱 reflects the experimental designer's knowledge of the order of the carryover effect. 𝗗𝗲𝘀𝗶𝗴𝗻 𝗼𝗳 𝗥𝗲𝗴𝘂𝗹𝗮𝗿 𝗦𝘄𝗶𝘁𝗰𝗵𝗯𝗮𝗰𝗸 𝗘𝘅𝗽𝗲𝗿𝗶𝗺𝗲𝗻𝘁𝘀:    • To find the optimal design of the regular switchback experiment, the authors use a minimax framework to derive the best possible design for the worst-case set of potential outcomes. 𝗞𝗲𝘆 𝗥𝗲𝘀𝘂𝗹𝘁𝘀: 1️⃣ 𝗢𝗽𝘁𝗶𝗺𝗮𝗹𝗶𝘁𝘆 𝗼𝗳 𝗙𝗮𝗶𝗿 𝗖𝗼𝗶𝗻 𝗙𝗹𝗶𝗽𝗽𝗶𝗻𝗴. Under p=m (i.e., perfect knowledge of carryover effects), the optimal randomization probabilities should be 1/2. 2️⃣ 𝗢𝗽𝘁𝗶𝗺𝗮𝗹 𝗗𝗲𝘀𝗶𝗴𝗻. Under p=m, when m=0, the optimal randomization frequency is {1, 2, 3,…,T}. When m>0, and if there exists , s.t. T=nm, then the optimal randomization frequency is {1, 2m+1, 3m+1, …, (n-2)m+1}. 3️⃣ 𝗧𝘄𝗼 𝗺𝗲𝘁𝗵𝗼𝗱𝘀 𝗳𝗼𝗿 𝗶𝗻𝗳𝗲𝗿𝗲𝗻𝗰𝗲. Under p=m, Exact Inference (randomization-based test) and Asymptotic Inference (a finite population conservative test). The inference is still valid when p m, though. 4️⃣ 𝗧𝗵𝗲 𝗽𝗿𝗼𝗰𝗲𝗱𝘂𝗿𝗲 𝘁𝗼 𝗲𝘀𝘁𝗶𝗺𝗮𝘁𝗲 𝘁𝗵𝗲 𝗼𝗿𝗱𝗲𝗿 𝗼𝗳 𝘁𝗵𝗲 𝗰𝗮𝗿𝗿𝘆𝗼𝘃𝗲𝗿 𝗲𝗳𝗳𝗲𝗰𝘁. Define a hypothesis testing procedure that, when combined with a search method, provides an estimate of the magnitude of the carryover effect. Finally, the authors demonstrate their approach using a simulated study and conclude by discussing its practical implications and limitations. Check it out! 𝗥𝗲𝗳𝗲𝗿𝗲𝗻𝗰𝗲: Bojinov, I., Simchi-Levi, D., & Zhao, J. (2023). Design and analysis of switchback experiments. 𝘔𝘢𝘯𝘢𝘨𝘦𝘮𝘦𝘯𝘵 𝘚𝘤𝘪𝘦𝘯𝘤𝘦, 69(7), 3759-3777.

Experimentation lies at the core of effective digital product strategies. Vanguard’s recent tech blog explores how A/B testing and multi-armed bandit (MAB) algorithms each bring value to web optimization—and why choosing the right method matters for delivering fast, impactful results. The article presents a simulation study comparing three approaches: traditional A/B testing, Adaptive Allocation MAB, and Thompson Sampling MAB. For three or fewer variations, a properly powered A/B test often identifies the winner more quickly and is easier to implement and interpret. But once you move beyond four variations, bandit strategies like Thompson Sampling begin to outperform A/B testing—both in terms of speed and in minimizing “regret,” or lost opportunity cost. Thompson Sampling also tended to edge out Adaptive Allocation across most simulated scenarios, though the gap narrows when there’s significant performance uplift or a large number of variants. In short: use A/B when you want clarity and simplicity with a small set of variants; turn to MAB when you need efficiency at scale or rapid optimization. Of course, as this was a simulation-based study, some nuances and real-world dynamics may not be fully captured. Still, this analysis offers a practical rule of thumb for experimentation design—especially for teams looking to improve the efficiency and impact of their testing strategies. #DataScience #MachineLearning #Analytics #Experimentation #ABTest #MultiArmBandit #Measurement #SnacksWeeklyonDataScience  – – –  Check out the "Snacks Weekly on Data Science" podcast and subscribe, where I explain in more detail the concepts discussed in this and future posts:    -- Spotify: https://lnkd.in/gKgaMvbh   -- Apple Podcast: https://lnkd.in/gFYvfB8V    -- Youtube: https://lnkd.in/gcwPeBmR https://lnkd.in/gnfN4bGa

Most tech companies excel at A/B testing—but what if the structure of your product makes a standard A/B test impossible? My latest blog post provides a non-technical introduction to switchback experiments—a powerful alternative when interference, single-unit constraints, or marketplace dynamics make traditional approaches unworkable. I break down: - What switchback experiments are - How to design them (length, number, and randomization of periods) - The challenge of carryover effects (and strategies to handle them) - How to analyze results with modern, non-parametric methods Check out the blog post here: https://lnkd.in/eKuJh9iE I also updated my website: ibojinov.com

While user-level A/B testing is often ideal for causal inference, it can struggle with challenges like SUTVA violations—where the treatment for one group impacts the control group. In tech, this might happen when a rideshare algorithm prioritizes treated users, leaving control users with fewer resources. Public policy can face similar issues; for example, treatments assigned to only some students in a school may affect control students in a variety of ways. Geo-experimentation offers a solution. By randomizing entire geographic units—metro areas, markets, schools—rather than individual users, it contains spillovers within regions, allowing for cleaner comparisons. While it has tradeoffs, like smaller sample sizes, its benefits have made geoX a very popular tool for marketing, product, and growth data scientists. I introduce the topic in a little more detail in this 5-minute tutorial (best viewed on mobile): https://lnkd.in/ebCcw2Eb #datascience #datasciencetutorials #causalinference #abtesting

📋 New Research: Optimizing Patient Outreach Methods Our team's study out today in Nature's npj Digital Medicine journal examines methodological improvements for patient engagement programs. While many organizations currently use A/B testing (simple randomized trials) for message optimization, our analysis demonstrates specific advantages of Sequential Multiple-Adaptive Randomized Trials (SMART): • Increased statistical power with smaller sample sizes • Structured approach to message personalization • Enhanced efficiency for LLM-based communications Relevant for Medicaid programs and health plans managing large-scale patient outreach initiatives, e.g., for HEDIS gap closure. Here's the paper: https://lnkd.in/gphUxnnf