Split Testing Techniques

Most “advanced A/B testing” content is either: 1) too academic to apply, or 2) too simplified to be useful. My favourite way to actually learn advanced methods: → read how top tech companies run experiments at scale. Here are 4 approaches you’ll see again and again in real experimentation platforms 👇 1) CUPED (variance reduction) Use pre-experiment behavior as a covariate to reduce noise → narrower CI → shorter tests. Nubank’s practical lessons from implementing CUPED: https://lnkd.in/dKHGktNt 2) CUPAC (ML-powered variance reduction) Same idea as CUPED, but the covariate comes from a prediction model (e.g., predicted revenue, ETA, or spend). Because the model can use many features, it often explains more variance than a single pre-period metric → even less noise and faster tests. Works great when the raw pre-period metric is weak/noisy. DoorDash’s deep dive (they coined “CUPAC”): https://lnkd.in/dB9Yevkw 3) Sequential testing (solve the “peeking” problem) Want to monitor results daily without inflating false positives? Sequential frameworks let you “peek” safely and stop earlier when evidence is strong enough. Spotify’s guide to choosing a sequential testing framework: https://lnkd.in/dwyzmzvK 4) Switchback tests (when classic A/B breaks in marketplaces) If there are network effects / interference (dispatch, pricing, matching), user-level randomization can lie. Switchback: randomize time × region blocks and compare periods. DoorDash’s introduction to switchback testing under network effects: https://lnkd.in/dffDCVEX In practice, how much do these methods speed up your tests?

Classic A/B testing relies on SUTVA (Stable Unit Treatment Value Assumption), which assumes one user’s decision doesn’t influence another’s. But what if your product is a social network, marketplace, or delivery service? Imagine you’ve improved the post-ranking algorithm on LinkedIn. Users in Group A (new algorithm) creates more content now. But this content spreads to Group B (old algorithm), distorting the results due to network effects. Here are two main ways to tackle this: 1. 𝐂𝐥𝐮𝐬𝐭𝐞𝐫𝐢𝐧𝐠-𝐛𝐚𝐬𝐞𝐝 𝐞𝐱𝐩𝐞𝐫𝐢𝐦𝐞𝐧𝐭𝐬: Randomize groups of users (clusters) instead of individual users. For social networks, the most popular approach is to define clusters based on interaction frequency — those who engage more often stay together in one cluster. 2. 𝐒𝐰𝐢𝐭𝐜𝐡𝐛𝐚𝐜𝐤 𝐭𝐞𝐬𝐭𝐬: In this approach, everyone in the network receives the same treatment at any given time. Over time, we flip between test and control groups, compare metrics, and evaluate the impact. This is especially useful for location-based services (e.g., taxis or delivery). Even if you’re not working with a product that has potential network effects, understanding these methods will help you in future interviews!

Day 5 - CRO series Strategy development ➡A/B Testing (Part 1) What is A/B Testing? A/B testing, also known as split testing, is a method used to compare two versions of a marketing asset, such as a webpage, email, or advertisement, to determine which one performs better in achieving a specific goal. Most marketing decisions are based on assumptions. A/B testing replaces assumptions with data. Here’s how to do it effectively: 1. Formulate a Hypothesis Every test starts with a hypothesis. ◾ Will changing a call-to-action (CTA) button from green to red increase clicks? ◾ Will a new subject line improve email open rates? A clear hypothesis guides the entire process. 2. Create Variations Test one element at a time. ◾ Control (Version A): The original version ◾ Variation (Version B): The version with a change (e.g., a different CTA color) Testing multiple elements at once leads to unclear results. 3. Randomly Assign Users Split your audience randomly: ◾ 50% see Version A ◾ 50% see Version B Randomization removes bias and ensures accurate comparisons. 4. Collect Data Define success metrics based on your goal: ◾ Click-through rates ◾ Conversion rates ◾ Bounce rates The right data tells you which version is actually better. 5. Analyze the Results Numbers don’t lie. ◾ Is the difference in performance statistically significant? ◾ Or is it just random fluctuation? Use analytics tools to confirm your findings. 6. Implement the Winning Version If Version B performs better, make it the new standard. If no major difference? Test something else. 7. Iterate and Optimize A/B testing isn’t a one-time task—it’s a process. ◾ Keep testing different headlines, images, layouts, and CTAs ◾ Every test improves your conversion rates and engagement Why A/B Testing Matters ✔ Removes guesswork – Decisions are based on data, not intuition ✔ Boosts conversions – Small tweaks can lead to significant growth ✔ Optimizes user experience – Find what resonates best with your audience ✔ Reduces risk – Test before making big, irreversible changes Part 2 tomorrow

🚀 Beyond A/B Testing: When Classic Experiments Fall Short 🚀 A/B testing has long been the gold standard for measuring feature impact. Clear results, clean comparisons, randomized traffic, and the ability to quantify long term effects—it's a powerful tool in any product or analytics team's toolbox. But as products get smarter, more personalized, and more real-time, traditional A/B tests struggle to keep up. Let’s break it down 👇 ❌ The Limits of A/B Testing: ⚠️ Requires long run times due to the need for large sample sizes ⚠️ Not well-suited for ranking problems like search results (e.g., “Which movie to watch”) or personalized recommendations (e.g., “Items you may like”) ⚠️ Doesn’t adapt in real time—once traffic is split (e.g., 50/50), users may continue to see a subpar experience even if early results show it’s underperforming ⚠️ Struggles in environments with user-to-user interactions, such as marketplaces or social networks, where one user’s experience can influence another’s 💡 So… What Do You Do When A/B Test Falls Short? That’s exactly what this series is about 👇 Welcome to Part 1 of a multi-part series on modern experimentation strategies —and we're kicking things off with a deep dive into Sequential Testing. 🔍 What’s Inside: ✅ A layman explanation of the method ✅ When and why to use it ✅ Tips on influencing stakeholders ✅ A real-world business use case ✅ Python implementation 👥 Who Is This Series For? 👩💻 Product Data Scientists exploring advanced techniques beyond traditional A/B testing 🧠 Product Managers looking to design smarter, more efficient experiments 📊 Risk Managers & Analysts aiming to detect early signals of performance issues 🤝 I’m Collaborating With... I’ve teamed up with Bhavnish Walia, who specializes in product risk management and experimentation. Together, we’ll be sharing practical insights, drawing from real-world experience at companies like Amazon, Walmart, and more.

"We tried Cold Email, but didn't see results." Has to be one of the most common challenges I hear. Let me explain. Over the course of 2024, I’ve spoken with many B2B SaaS Founders, Marketing Directors, Sales Directors, and GTM Leaders. They all share one problem in common: They’ve tried Cold Outreach, but they don’t get any results. So naturally, I start asking questions and offer to have a look at what they’re doing. When I review their campaigns, one thing becomes crystal clear: They understand how to build prospect lists, but there's little to no split testing happening. Here’s the reality: If you’re only sending 100-200 emails without testing different angles, you’re gambling on the success of your campaign, and in most cases, that gamble doesn’t pay off. Let’s break this down. There are two types of companies: 1️⃣ The 1% that doesn’t need to split test (they already know their ICP and what works for them). 2️⃣ The 99% that absolutely MUST split test to find what works best. If you’re part of the 99% (and most of us are), here’s how to do it effectively: Step 1: Test Pain Points Start by identifying the key problems your target audience is facing. Let’s say you’re an agency targeting e-commerce brands. You could test angles like: → High customer acquisition costs → Low lifetime value → Low return on ad spend Each email script stays consistent, only the pain point changes. 💡 Example: If you’re targeting a Sales Director, one angle might focus on the challenge of getting unqualified leads filling up their pipeline, while another might highlight how their team spends too much time on lead nurturing rather than closing. Allocate a set number of leads to each angle (e.g., 1,000 leads per angle) and track results. Step 2: Analyze & Scale Winners Once you’ve sent out the emails, review your data. Ask yourself: → Which angle is getting the most positive replies? → Are certain pain points resonating more than others? If one angle shows promise, double down. If another flops, drop it. Step 3: Test Offers After narrowing down the best angles, shift your focus to your offer. Split test variations of your offer to see which drives the most engagement and demo bookings. Forget vanity metrics like open rates (for now). Instead, track the ratio of PRRs. Many B2B companies: ❌ Send a small volume of Cold Emails (100-200) and expect big results. ❌ Focus too much on minor variables like subject lines before testing major factors like pain points or offers. ❌ Don’t analyze campaign performance enough to refine their approach. 💡 Pro tip in the PDF below👇 💬 Drop a comment below, or DM me for a free campaign audit.