Quick ABC's to A/B Testing

Sites frequently undergo changes to stay relevant and engaging. People want more and more with their online experience, but how can we keep up with the demand? How do we know which feature works best with our users? Is it adding more messaging or simplifying the online order process? A/B testing helps you figure that out. We can compare a control group versus a treatment group to isolate the key variable that affects your site activity. Below are three simple steps which I refer to as the ABC’s of A/B split testing.

ASK

What business goal is your new site feature trying to achieve? The goals you define guide you to success metrics that measure your test’s performance. Some might say “increase in sales” and others “higher engagement on site.” These goals measure as “conversion rates” and “pages per visitor” in web analytics. You can then focus on these primary metrics to evaluate the test’s success. For a more thorough analysis, you can also utilize more metrics that can support the test results like amount of traffic.

BALANCE

Having a balanced comparison between your control and treatment groups allows you to distinguish the cause of your results. Try to make sure your user base is as close to an apples-to-apples comparison as possible. Sites have many elements that can subtly affect your results, such as simultaneous promotions or even a peak traffic period led by an external campaign. Try to limit those variables that are not the focus of your test. It can easily skew your results and make them unusable. Your test won’t be perfect, but do your best to isolate the test’s focus on the changing feature. You can always do more and more tests to watch for consistencies. This will ensure that you can be confident in the new feature leading to an increase of sales.

CONFIDENCE

You’ll need enough data to reach confidence in your results where it is guaranteed after the test. Think about how long to run the test. For sites with larger traffic, you can easily gather a good sample size within a few weeks. For sites that don’t have as much traction, it will take more time to gain enough data to make a strong conclusion. You can look into automated A/B tools that can easily help you evaluate the confidence levels for your test.

Reach out if you need help to structure your A/B tests! It’s always fun to see the results for all types of scenarios.