Importance of A/b Testing for Apps

I spent almost a decade writing ineffective software because I didn't do this... Enabling our product team to make data-driven decisions. Specifically, having an A/B experimentation framework in place to test different ways features could be presented to users. Life without A/B Experiments: - Endless debates about whose copy is better - Designing based on gut feelings instead of evidence - Risky launches without knowing what works - Assuming "what worked before" will work again - Struggling to explain decisions to stakeholders Life with A/B Experiments: - Clear, data-driven decisions - Discovering unexpected user preferences - Testing small changes before committing to big ones - Building a culture of curiosity and iteration - Making stakeholders confident in your choices If you're on a team that regularly conducts A/B experiments, maybe this seems obvious to you. If you're not, maybe this sounds too good to be true. I know that now that I've been on teams that embrace experimentation, it'd be hard to go back to an environment that lacks it. How has your experience with A/B experiments and data driven software development been?

founder learnings! part 8. A/B test math interpretation - I love stuff like this: Two members of our team (Fletcher Ehlers and Marie-Louise Brunet) - ran a test recently that decreased click-through rate (CTR) by over 10% - they added a warning telling users they’d need to log in if they clicked. However - instead of hurting conversions like you’d think, it actually increased them. As in - Fewer users clicked through, but overall, more users ended up finishing the flow. Why? Selection bias & signal vs. noise. By adding friction, we filtered out low-intent users—those who would have clicked but bounced at the next step. The ones who still clicked knew what they were getting into, making them far more likely to convert. Fewer clicks, but higher quality clicks. Here's a visual representation of the A/B test results. You can see how the click-through rate (CTR) dropped after adding friction (fewer clicks), but the total number of conversions increased. This highlights the power of understanding selection bias—removing low-intent users improved the quality of clicks, leading to better overall results.

🚨 Your A/B test results are not the real impact. A happy PM runs an A/B test → sees a +15% lift in revenue → scales the feature to all users → shares the big win in Slack 🎉 But… once the feature is fully rolled out, the KPI impact isn’t there. Why? Because test results often don’t reflect the true long-term effect. Here are a few reasons why this happens: 1️⃣ Confidence intervals matter → That “+15%” is actually a range. The lower bound might be close to zero. 2️⃣ Novelty effect → Users are excited at first, but the effect fades as they get used to the change. 3️⃣ Experiments aren’t additive → Three +15% lifts don’t stack to +45%. There’s a ceiling, and improvements often cannibalize each other. 4️⃣ Sample ≠ population → The test group might not represent your entire user base. For example, you have more high-intent users in the variant. 5️⃣ Time-to-KPI effects → We see that a lot, especially in conversion experiments. The experiment could improve the time to conversion, so when you close the experiment, it seems like you’re winning, but actually if you monitor the users a few days/weeks after the experiment ends, there are no differences in total conversions between the variant and the control. 6️⃣ Type I error → With P-value=0.05 (or worse, 0.1), there’s still a decent chance the “win” is a false positive. 👉 That’s why tracking post-launch impact is just as important as running the experiment itself. Methods like holdout groups, simple correlation tracking, or causal inference models (building synthetic control) help reveal the real sustained effect.

A 6% revenue lift. 99% statistical significance. Ship it. It couldn't go wrong, could it? 🫣 In 2016, I was leading a product analytics team at Credit Karma. We ran an A/B test for a personal loans redesign. The results looked fantastic: - 𝗔𝗽𝗽𝗿𝗼𝘃𝗮𝗹𝘀 𝘄𝗲𝗿𝗲 𝘂𝗽 (good for users). - 𝗥𝗲𝘃𝗲𝗻𝘂𝗲 𝘄𝗮𝘀 𝘂𝗽 𝟲% (good for business). - 𝗦𝘁𝗮𝘁𝗶𝘀𝘁𝗶𝗰𝗮𝗹 𝘀𝗶𝗴𝗻𝗶𝗳𝗶𝗰𝗮𝗻𝗰𝗲: 𝟵𝟵%. We should have ramped it up to 100% of users and closed out the test. However, we couldn't roll it out immediately due to other constraints. Over the next few weeks, I watched that 6% revenue lift drift down to 3%. It was still positive. It was still 99% significant. But the downward trend didn't sit right with me. I dug into the segments and found the reality: 𝗨𝘀𝗲𝗿𝘀 𝗻𝗲𝘄 𝘁𝗼 𝘁𝗵𝗲 𝗲𝘅𝗽𝗲𝗿𝗶𝗲𝗻𝗰𝗲: +10% revenue. 𝗨𝘀𝗲𝗿𝘀 𝗿𝗲𝘁𝘂𝗿𝗻𝗶𝗻𝗴 𝘁𝗼 𝘁𝗵𝗲 𝗲𝘅𝗽𝗲𝗿𝗶𝗲𝗻𝗰𝗲: -5% revenue. The aggregate number was positive only because the traffic was initially heavy with people seeing the design for the first time. Over time, as those people returned to the page, they fell into the negative bucket. 𝗜𝗳 𝘄𝗲 𝗵𝗮𝗱 𝘀𝗵𝗶𝗽𝗽𝗲𝗱 𝗯𝗮𝘀𝗲𝗱 𝗼𝗻 𝘁𝗵𝗲 𝗮𝗴𝗴𝗿𝗲𝗴𝗮𝘁𝗲, 𝘄𝗲 𝘄𝗼𝘂𝗹𝗱 𝗵𝗮𝘃𝗲 𝗲𝘃𝗲𝗻𝘁𝘂𝗮𝗹𝗹𝘆 𝗹𝗼𝘀𝘁 𝗺𝗼𝗻𝗲𝘆. We wouldn't have even known that it was due to a negative A/B test. Because we caught this, we redesigned the experience to address the issues for the returning users before rolling it out. Don't just blindly follow A/B tests and their implied results. While I love A/B testing, you need to be very careful to understand what you are truly measuring. (we did end up fixing the experience for returning users and deploying a win-win)

Most teams are drowning in optimization test ideas... but starving for real impact. I've seen this pattern destroy more optimization programs than poor execution ever could. The problem isn't lack of creativity. It's lack of strategy. Before you run another A/B test, ask yourself four critical questions: ↳ Is this strategically important to your business goals? ↳ Are you confident the change won't harm the user experience? ↳ Can you reach statistical significance in a reasonable timeframe? ↳ Do you have the technical capability to execute properly? If any answer is "no," you have better options: ↳ De-prioritize non-strategic tests. Add them to your backlog for later consideration. ↳ Run rapid sentiment tests or task completion analysis for quick validation. Only commit to full experimentation when all four criteria align. Or implement proven solutions directly when you're confident in the outcome. This decision framework has helped our clients at The Good generate over $100 million in additional revenue by focusing their testing efforts where they matter most. Your optimization program isn't measured by how many tests you run. It's measured by how much value you create.

A/B testing is not a new method at all, in behavioral science, this logic has been around for a very long time, even if we do not usually call it A/B testing. We call it experiments, condition comparisons, randomized studies, or controlled designs. The basic idea is the same: compare versions, observe responses, and learn from the difference. But in behavioral science, that comparison was never restricted to metrics alone. Depending on the question, researchers have long used quantitative data, qualitative data, or a combination of both to understand how people respond to different conditions. That broader view makes a lot of sense to me, especially because human behavior is rarely just a number. In UX, though, A/B testing is still usually framed in a much narrower way. Most of the time, it is treated as a purely quantitative exercise: which version got more clicks, which flow converted better, which option increased engagement. That is valuable, and I am not arguing against it. I use behavioral metrics too, and they matter. But across projects, one thing has become very clear to me: a metric can tell you that something changed without telling you what that change actually meant to users. I have seen cases where a version performed better, but the real insight was not the lift itself. The real insight was that users felt less hesitation, understood the next step faster, or trusted the experience more. Without talking to them, observing them, or giving them space to explain their thinking, that deeper layer would have remained invisible. That is why I think qualitative methods can make A/B testing far more insightful. Interviews, think aloud sessions, usability observation, and even short open ended follow ups can reveal the mechanism behind the outcome. Instead of stopping at Version B won, we can start understanding whether it reduced confusion, lowered cognitive effort, aligned better with expectations, or made the interface feel more credible. To me, that is where the real value is. A/B testing should not only help us choose between options. It should help us learn something meaningful about perception, attention, trust, decision making, and friction. Otherwise, we risk becoming very good at measuring outcomes while staying relatively shallow in how we interpret them. I also think qualitative methods are useful at more than one stage of the process. Before a test, they help generate stronger variants because the changes are grounded in actual user problems rather than assumptions. During a test, they can capture reactions that behavioral logs cannot fully explain. After a test, they help interpret both positive and null results. Sometimes a version does not win because the change was weak, because users did not notice it, or because the thing the team cared about was not what users cared about. Those are important lessons, and they rarely come from the dashboard alone. Perceptual User Experience Lab

A multi-million-dollar mistake many brands make: treating A/B testing as something you do after launching a new site. They pour millions into a redesign and cross their fingers on launch day. Not only are they missing a huge opportunity, they’re taking a huge risk. Launching a redesigned storefront that hasn’t been tested can materially damage key metrics like conversion rate, revenue per visitor, and AOV. Fixing it can take weeks or months. All the while, you’re losing millions in sales you would’ve otherwise gotten. Smart brands do it differently. Daniel Wellington is a great example. They’re currently working on some major theme updates. While rebuilding parts of their Shopify theme, they're using Shoplift to A/B test every major design decision before it goes into the new build. Think about that. Instead of guessing which PDP layout will convert, they already know. Instead of hoping the new gallery doesn't tank engagement, they've validated it. They're not testing to optimize an existing site. They're testing to build a better site from day one. This is the underutilized superpower of A/B testing: using it as a development tool, not just an optimization tool. The result? Daniel Wellington won't just launch a new theme; they'll launch a pre-validated, data-proven winner. The lesson: Stop treating A/B testing as post-launch homework. Make it part of how you build.

We have tracked A/B tests from Airbnb for years. Airbnb tests at a high volume, dozens per month, and each change has high consequences. Close to 100 million visitors hit Airbnb’s site each month. A few patterns emerge… 1) More segmentation wins. Whether it is Airbnb, streaming companies, or B2B SaaS, brands with more segmentation and personalization create better affinity, and these A/B versions drive more clicks. 2) Know your weaknesses and address them. For a brand like Airbnb, they recently tested showing full pricing upfront, versus a lower price displayed (but then adding fees, cleaning, etc. after). No surprise, full upfront pricing is the A/B they kept, as this addresses a major criticism of the brand (hidden fees). Similarly, if you look at A/B tests from brands like Salesforce, you find the tests that emphasize simplicity and ease of use win out. Today, they use the words “easy” or “simple” a dozen times on their homepage. One criticism they get is that they are overcomplicated, so they got ahead of it. 3) Lean into your brand differentiation. For a brand like Airbnb, they want to give a sense of “less touristy”, more “local”. Copy that leans into this distinction tends to win in A/B tests. It’s the same as Klaviyo tests winning when they lean more into eCommerce, B2C, and niche-specific messaging. For a brand like Apple, putting their “design” block ahead of “features” or “tech specs” might have felt non-intuitive, but it won out, because design is a core part of what makes Apple unique and stand out. Are there any brands (or competitors of yours) that you would love to see more about? Happy to connect you with our team.

Many non-gaming apps miss a crucial #optimization step by not prioritizing A/B testing for their app #icons. While they engage in various forms of A/B testing, overlooking the icon can lead to missed opportunities for significant #improvements. 💡 Through consistent A/B testing of our clients’ app icons, we’ve observed substantial #success, typically seeing #conversionrate improvement around 4-6%. 🚀 This emphasizes the crucial role that even small changes in app presentation can play in attracting and retaining #users. This approach to tweaking app icons is gaining popularity. For instance, Duolingo, a leading app, often adjusts its character’s expression on the icon to match specific events or themes. 📗 Similarly, the Tandem app changed its icon nine times in 2022, aligning with events like the start of the academic year, holidays, and New Year’s. Each new icon was used for about two weeks before going back to the original. (source: SplitMetrics report) These examples show how simple icon adjustments can make a big impact. Take the Ten Percent app, for instance. They added a subtle touch of light to their icon, proving that small changes can create a significant difference. Using this strategy can help your app stand out in #search results. Customizing your icon for #holidays or events grabs users’ attention and can boost #engagement. Even minor changes like zooming in, adding highlights, or changing the background color can significantly improve user engagement. #appicon #abtesting #conversionrateoptimization #mobilemarketing