AB Split Testing, Does It Really Prove Anything?

What is AB Split Testing?

AB testing or split testing is a way of measuring the success of one thing vs another. 

Imagine you own an ice cream shop on the high street and you can't decide to paint the exterior of the shop mellow yellow or candy floss pink. Will the colour of the shop front make any difference to sales? To split test it you would need to keep altering the colour to see the effect it had. Ideally you would like the shop yellow for half of the passers by and pink for the other half. Then at the end of the test period, with the result of which passers by actually entered the shop, made a purchase, how long they stayed in the store and how much they spent, you could decided which colour to keep the shop.

In the real world, this would be very difficult to actually implement unless you kept repainting the store and unless the colours were run side by side at the same time, other variables such as weather could cloud the accuracy of your testing.

With ecommerce, split testing is much easier. We can create two versions of the same webpage and send half the visitors to one page and half to the other. At the end of the test we can decide which page was the most successful based on whatever criteria that we find important. The criteria could be a goal such as email subscription, making a sale or it could be how long the visitor stayed on the page. 

Split Testing Our Garden Furniture Covers Page

One simple split test that we have recently run was on our garden furniture covers page.

We already knew from customer tracking using other tools which parts of the covers page customers used (clicked on) most often. We wanted to see what would happen if we altered the page so that the most used elements were presented to the customer straight away at the top of the page. This meant removing a landscape image of one of the covers from the top of the page, this is the only real image on the page showing one of the covers.

So in essence we wanted to test the impact of removing a visual element from the page so that the functional elements were presented sooner. 

We created two versions of the page and started the split testing using Google Analytics. By placing the analytics code on our site a percentage of visitors are presented with one version of the page whilst the others are presented with the alternate new page and each visitor is tracked through the site.

These are the two versions, with or without the header. 

It's important to note that the second page loads faster as the large image is missing and on smaller devices the main content is displayed on the screen sooner without scrolling. 

After setting the test up we left it and moved on to other things whilst it ran long enough to provide any meaningful data. After 2,000 visitors we started to see some positive data in one direction.

Although the bounce rate and pages per visit were all very close, the conversion rate for the original page was consistently better, with an average conversion rate nearly 20% higher than the version without the slider and the average spend per visitor around 20% higher also. However the data during the Black Friday weekend shows that data swings the other way and the new page without the landscape image performs better.

As long as the data settles back into the original pattern we will declare the page with the image the victor and the image will stay. Then we will progress onto the next split test, should the image be purely aesthetic or should it promote a range of covers as it does now. 

The test is still running, click on the link to see which version of the page you are presented with..Garden Furniture Covers

I hope this article explained about AB split testing and gave you an insight into what happens in the background at an ecommerce company.