A/B Testing for App Marketing

I’ve spent 15+ years and $100m+ learning mobile advertising → I’ve put my biggest learnings about creative testing into a playbook for mobile marketers. I’m giving it away for free. The 3 big mistakes I see advertisers make are: - Not testing creative at all - Not testing enough - Not testing right I’ve seen these mistakes cost advertisers millions - in cost and opportunity costs. This is why I put this guide together to help advertisers test right - even in a post-identifier world with its data limitations. What it contains: 🔄 How to play nice with algorithms: Learn to work with algorithms rather than against them to optimize your ad performance. 🔧 How algos really work ‘under the hood’: Understand the mechanics of algorithms and why this knowledge can make or break your testing strategy. 🧪 AB tests or Bayesian tests?: Get insights into which testing method is right for your campaigns. 🌍 Where should you test?: Find out the best geos and platforms for your creative tests. 💰 How much should you spend on testing?: Learn to budget effectively for your creative testing efforts. ⚠️ Dealing with creative saturation: Discover strategies to combat creative fatigue and keep your ads fresh. 📊 Actual examples and calculations from testing setups for real apps: See real-world examples and data to guide your testing strategy. …and much more. After years of working in the mobile advertising industry and spending over $100 million on campaigns, I’ve distilled my biggest learnings about creative testing into this comprehensive guide. It’s packed with actionable insights and strategies that can help you improve your own creative testing process. Want the playbook? See link in comments.

founder learnings! part 8. A/B test math interpretation - I love stuff like this: Two members of our team (Fletcher Ehlers and Marie-Louise Brunet) - ran a test recently that decreased click-through rate (CTR) by over 10% - they added a warning telling users they’d need to log in if they clicked. However - instead of hurting conversions like you’d think, it actually increased them. As in - Fewer users clicked through, but overall, more users ended up finishing the flow. Why? Selection bias & signal vs. noise. By adding friction, we filtered out low-intent users—those who would have clicked but bounced at the next step. The ones who still clicked knew what they were getting into, making them far more likely to convert. Fewer clicks, but higher quality clicks. Here's a visual representation of the A/B test results. You can see how the click-through rate (CTR) dropped after adding friction (fewer clicks), but the total number of conversions increased. This highlights the power of understanding selection bias—removing low-intent users improved the quality of clicks, leading to better overall results.

A 6% revenue lift. 99% statistical significance. Ship it. It couldn't go wrong, could it? 🫣 In 2016, I was leading a product analytics team at Credit Karma. We ran an A/B test for a personal loans redesign. The results looked fantastic: - 𝗔𝗽𝗽𝗿𝗼𝘃𝗮𝗹𝘀 𝘄𝗲𝗿𝗲 𝘂𝗽 (good for users). - 𝗥𝗲𝘃𝗲𝗻𝘂𝗲 𝘄𝗮𝘀 𝘂𝗽 𝟲% (good for business). - 𝗦𝘁𝗮𝘁𝗶𝘀𝘁𝗶𝗰𝗮𝗹 𝘀𝗶𝗴𝗻𝗶𝗳𝗶𝗰𝗮𝗻𝗰𝗲: 𝟵𝟵%. We should have ramped it up to 100% of users and closed out the test. However, we couldn't roll it out immediately due to other constraints. Over the next few weeks, I watched that 6% revenue lift drift down to 3%. It was still positive. It was still 99% significant. But the downward trend didn't sit right with me. I dug into the segments and found the reality: 𝗨𝘀𝗲𝗿𝘀 𝗻𝗲𝘄 𝘁𝗼 𝘁𝗵𝗲 𝗲𝘅𝗽𝗲𝗿𝗶𝗲𝗻𝗰𝗲: +10% revenue. 𝗨𝘀𝗲𝗿𝘀 𝗿𝗲𝘁𝘂𝗿𝗻𝗶𝗻𝗴 𝘁𝗼 𝘁𝗵𝗲 𝗲𝘅𝗽𝗲𝗿𝗶𝗲𝗻𝗰𝗲: -5% revenue. The aggregate number was positive only because the traffic was initially heavy with people seeing the design for the first time. Over time, as those people returned to the page, they fell into the negative bucket. 𝗜𝗳 𝘄𝗲 𝗵𝗮𝗱 𝘀𝗵𝗶𝗽𝗽𝗲𝗱 𝗯𝗮𝘀𝗲𝗱 𝗼𝗻 𝘁𝗵𝗲 𝗮𝗴𝗴𝗿𝗲𝗴𝗮𝘁𝗲, 𝘄𝗲 𝘄𝗼𝘂𝗹𝗱 𝗵𝗮𝘃𝗲 𝗲𝘃𝗲𝗻𝘁𝘂𝗮𝗹𝗹𝘆 𝗹𝗼𝘀𝘁 𝗺𝗼𝗻𝗲𝘆. We wouldn't have even known that it was due to a negative A/B test. Because we caught this, we redesigned the experience to address the issues for the returning users before rolling it out. Don't just blindly follow A/B tests and their implied results. While I love A/B testing, you need to be very careful to understand what you are truly measuring. (we did end up fixing the experience for returning users and deploying a win-win)

Day 6 - CRO series Strategy development ➡  A/B Testing (Part 3) Common Pitfalls in A/B Testing (And How to Avoid Them) A/B testing can unlock powerful insights—but only if done right. Many businesses make critical mistakes that lead to misleading results and wasted effort. Here’s what to watch out for: 1. Testing Multiple Variables at Once If you change both a headline and a CTA button color, how do you know which caused the impact? Always test one variable at a time to isolate its true effect. 2. Using an Inadequate Sample Size Small sample sizes lead to random fluctuations instead of reliable trends. ◾ Use statistical significance calculators to determine the right sample size. ◾ Ensure your audience size is large enough to draw meaningful conclusions. 3. Ending Tests Too Early It’s tempting to stop a test the moment one variation seems to be winning. But early spikes in performance may not hold. ◾ Set a minimum duration for each test. ◾ Let it run until you reach statistical confidence. 4. Ignoring External Factors A/B test results can be influenced by: ◾ Seasonality (holiday traffic may differ from normal traffic). ◾ Active marketing campaigns. ◾ Industry trends or unexpected events. Always analyze results in context before making decisions. 5. Not Randomly Assigning Users If users aren’t randomly split between Version A and B, results may be biased. Most A/B testing tools handle randomization—use them properly. 6. Focusing Only on Short-Term Metrics Click-through rates might rise, but what about conversion rates or long-term engagement? Always consider: ◾ Immediate impact (CTR, sign-ups). ◾ Long-term effects (retention, revenue, lifetime value). 7. Running Tests Without a Clear Hypothesis A vague goal like “Let’s see what happens” won’t help. Instead, start with: ◾ A clear hypothesis (“Changing the CTA button color will increase sign-ups by 15%”). ◾ A measurable outcome to validate the test. 8. Overlooking User Experience Optimizing for conversions shouldn’t come at the cost of usability. ◾ Does a pop-up increase sign-ups but frustrate users? ◾ Does a new layout improve engagement but slow down the page? Balance performance with user satisfaction. 9. Misusing A/B Testing Tools If tracking isn’t set up correctly, your data will be flawed. ◾ Double-check that all elements are being tracked properly. ◾ Use A/B testing tools like Google Optimize, Optimizely, or VWO correctly. 10. Forgetting About Mobile Users What works on desktop may fail on mobile. ◾ Test separately for different devices. ◾ Optimize for mobile responsiveness, speed, and usability. Why This Matters ✔ More Accurate Insights → Reliable data leads to better decisions. ✔ Higher Conversions → Avoiding mistakes ensures real improvements. ✔ Better User Experience → Testing shouldn’t come at the expense of usability. ✔ Stronger Strategy → A/B testing is only valuable if done correctly. See you tomorrow!

I used to think a simple “Sign up now” button was enough. But let me tell you something. I’ve seen conversions double (and die) based on the tiniest CTA change. Not the offer. Not the product. Just how you ask for action. That’s the magic of A/B testing. I’ve tested it all: ✅ Animated vs static CTAs ✅ One-click vs multi-step forms ✅ Geo-targeted vs global buttons ✅ Voice-based vs typed call-to-actions ✅ AR previews vs static images ✅ Mood-based color schemes (yes, that’s real) Sometimes the results surprised me. Sometimes they humbled me. But every test taught me something new about how people think and click. This list? It’s not theory. It’s battlefield data. It’s what I wish someone handed me when I started. If you’re not testing, you’re guessing. 👉 Start with one element. 👉 Test it. Track it. Improve it. 👉 Repeat. Because the smallest tweaks often unlock the biggest wins.

Marketing is one of the biggest investments for a company like Tripadvisor, yet unlike product changes, there hasn’t been a standardized way to run A/B tests in this space. In a recent tech blog, Tripadvisor’s data science team shared how they built a self-serve experimentation platform to democratize marketing experimentation. At its core, the system employs causal inference methods—specifically, the difference-in-difference (DiD) approach—to measure marketing effectiveness. It uses Designated Market Areas (DMAs) as the unit of randomization and employs repeated randomization to identify control and treatment groups that satisfy the parallel trends assumption, which is critical for DiD to work properly. Once these groups are established, the system integrates with campaign data to surface the causal impact of marketing initiatives. The outcome is a scalable and trustworthy experimentation framework that allows Tripadvisor to evaluate marketing spend with the same rigor and confidence it applies to product testing. It’s a great example of how data science can bridge messy real-world challenges with structured methodologies, ultimately transforming how organizations make decisions. #DataScience #MachineLearning #Analytics #Experimentation #CausalInference #Marketing #SnacksWeeklyonDataScience – – –  Check out the "Snacks Weekly on Data Science" podcast and subscribe, where I explain in more detail the concepts discussed in this and future posts:    -- Spotify: https://lnkd.in/gKgaMvbh   -- Apple Podcast: https://lnkd.in/gFYvfB8V    -- Youtube: https://lnkd.in/gcwPeBmR https://lnkd.in/g4Zn5JCP

Are you confused by 𝗔/𝗕 𝗧𝗲𝘀𝘁𝗶𝗻𝗴? Don't worry, I've got you! 𝗔/𝗕 𝘁𝗲𝘀𝘁𝗶𝗻𝗴 (𝘢𝘭𝘴𝘰 𝘤𝘢𝘭𝘭𝘦𝘥 𝘴𝘱𝘭𝘪𝘵 𝘵𝘦𝘴𝘵𝘪𝘯𝘨) is like a science experiment for your deliverables. You compare two versions of something to see which one people like better. 𝗛𝗲𝗿𝗲'𝘀 𝘁𝗵𝗲 𝟲-𝗦𝘁𝗲𝗽 𝗔/𝗕 𝗧𝗲𝘀𝘁𝗶𝗻𝗴 𝗙𝗿𝗮𝗺𝗲𝘄𝗼𝗿𝗸 1. Identify the problem (use Analytics to find drop-offs) 2. Form a hypothesis ("Blue CTA will increase clicks by 20%") 3. Create test versions 4. Run the test (1-2 weeks minimum) 5. Analyze results (check statistical significance) 6. Document & share learnings 𝗦𝗸𝗶𝗹𝗹𝘀 𝗬𝗼𝘂 𝗡𝗲𝗲𝗱 𝗳𝗼𝗿 𝗔/𝗕 𝗧𝗲𝘀𝘁𝗶𝗻𝗴 🟢Basic Statistics: p-values, confidence intervals, and sample sizes. 🟢Data Tools: Use Excel, Google Sheets, or Python to analyze data. 🟢Critical Thinking: Ask, “Do these results make sense?”. 🟢Storytelling: Explain your findings in simple terms. 𝘌𝘹𝘢𝘮𝘱𝘭𝘦: A data analyst noticed a test increased clicks but hurt sales. They recommended keeping the original design because sales mattered more. 𝗧𝗲𝗰𝗵𝗻𝗼𝗹𝗼𝗴𝘆 𝗮𝗻𝗱 𝗧𝗼𝗼𝗹𝘀 🟢Google Optimize: Free for basic tests. 🟢Optimizely: Best for large companies. 🟢Statsig: Easy to get started and scale. 🟢Mixpanel (for app tests). 🟢Python 𝗪𝗵𝘆 𝗔/𝗕 𝗧𝗲𝘀𝘁𝗶𝗻𝗴 𝗠𝗮𝘁𝘁𝗲𝗿𝘀 𝙎𝙩𝙤𝙥 𝙂𝙪𝙚𝙨𝙨𝙞𝙣𝙜: Use data to choose the best design. 𝙎𝙖𝙫𝙚 𝙈𝙤𝙣𝙚𝙮: Avoid costly mistakes by testing small changes first. 𝗜𝗺𝗽𝗿𝗼𝘃𝗲 𝗦𝘁𝗮𝗸𝗲𝗵𝗼𝗹𝗱𝗲𝗿 𝗘𝘅𝗽𝗲𝗿𝗶𝗲𝗻𝗰𝗲: Fix problems users complain about. 𝗔/𝗕 𝘁𝗲𝘀𝘁𝗶𝗻𝗴 helps you make smarter choices. Start with small tests (like button colors), learn the tools, and always check your math. Even if a test fails, you’ll discover what not to do next time. ➡️ Follow Jess Ramos, MSBA, a Sr. Data Analyst who regularly performs A/B testing. You've got this! #statistics #dataanalytics #sql