A/B Testing

Introduction to A/B Testing

What is A/B Testing at a high level?

It’s a statistical way of comparing two or more versions, such as version A or version B to determine not only which version performs better but also to understand if a difference between two versions is statistically significant.

Why do businesses conduct A/B test?

Businesses have to take a data-driven approach. A common dilemma that companies face is that they think they understand the customer. But in reality, customers would behave much differently than you would think, consciously or subconsciously users don’t often even know why they make the choices they make, they just do, but when running an experiment, or an A/B test, you might find out otherwise, and the results can often be very humbling and customers can behave much differently than you would think. So especially conduct tests, rather than relying on intuition.

Let’s Visualize

For example, in marketing, or web design, you might be comparing two different landing pages with each other, or two different newsletters. Let’s say you take the layout of the page, you move the content body to the right now versus the left,

or maybe you change the call to action from green to blue,

or your newsletter subject line has the word “promotion” in version A and the word “free” in version B.

In order for A/B testing to work, you must call out your criteria for success before you begin your test.

Criteria for Success — Define your Metric

What is your hypothesis, or rather, What do you think will happen by changing to version B?

Maybe you’re hoping to increase:

Conversion rate

or Newsletter signups

or increase open rate

Split the traffic or User base

Traffic split doesn’t have to be 50-50, but you will want to figure out what is the minimum number of people I need to run my A/B test on to achieve statistically significant results.

You can do this with multiple versions such as two buttons that are blue and two that are orange.

Consider one blue and one orange button say RSVP, and another blue and orange button says sign up. This would be called a multivariate test, or a full factorial test since you’re comparing different factors.

Factors we can test on when running an A/B test

Consider changing the layout of the page and shifting where certain items are such as moving the content body to the right, the navigation to the left.

You can also compare two different images with each other, to see if one has a higher conversion rate or higher click-through rate.