When should I start split testing?

Many businesses in South Africa aren't doing split testing at all. Well established brands are making website design and user journey decisions based on boardroom conversations or team lead opinions rather than structured user research, analytics and experimentation data.

A large reason for this is because many people simply don't know the power of a structured, iterative research and split testing programme. Some do know, but are plagued with the question of whether or not now is the right time to begin. When you're looking to introduce UX Research & testing into your online strategy you can easily be confronted by a few questions:

When is the right time to split test? Should it be an always on strategy? A sprint strategy? Are there periods of the year when testing should be switched off? And others.

As I‘ve spoken and worked with different eCommerce, Product and Marketing Managers across industries locally and internationally it’s become clear that there are quite a range of opinions on the timing and consistency of split testing. Here’s what I think. 

Should I do split testing always on or in sprints?

Sometimes the idea seems good to do split testing in sprints. Something like 3 months on, 3 months off. Split testing on your website should without a doubt be an always on strategy for your business. The primary reason for this is that your traffic mix & market conditions are evolving constantly and testing helps you keep your finger on the pulse.

If you're reading an article on split testing, you know enough to know that running an online environment means dealing with a constant state of flux. Your campaign managers are making updates to their targeting, Facebook is updating its algorithm, market conditions are changing, geopolitical events are affecting sentiment etc. etc. Constantly testing your online environments means that you're constantly learning what works and what doesn't work for your users today.

One of the benefits people see in switching testing off for a while is that this gives them time to develop in the direction the tests pointed in. The problem is that after 3 months of building you may get back to your next cycle of testing and realise that things have changed since last quarter and your findings from 4 months ago no longer apply. This can be a painful lesson.

Should I do split testing before everything is stable?

In short, there will never be a perfect time to hit the gym, start that diet or making experimentation a part of your business culture (too hard a segway?). There is no future date when things are stable and easy and you're looking for a project at 14h25pm because you've already knocked off your to do list.

The primary reason we see decision makers not making experimentation a priority is because it's very difficult to understand the value it brings to a business until you've done it.

The good news is that it actually takes a lot less than most people think to get started with split testing, especially if you outsource to a trusted agency that can drive these things forward for you in the background.

Should I do split testing during promotions or campaigns?

This is an interesting one and one of the most valid reasons I've seen that businesses cite as a reason that they don't want to begin running split tests on their sites. I still believe the thinking is flawed though.

Many marketers don't want to test during big promotions or campaigns for 2 reasons:

1. They're worried that the results won't be useful post-promotion

Let's say you run the following ab testing during your Black Friday campaign.

• Version a) The top of your home page is a big Black Friday banner that drives to a landing page with a listing of all of the different product categories on sale.
• Version b) The top of your home page is a selection of the top selling categories which are on sale in the form of tiles.

Let's say you run the test over the first 2 weeks of your Black Friday sale and find that version (b) above leads to a 12% higher revenue per home page user.

What's useful here?

• You can change the production version of your home page to Version 2 for the remainder of your Black Friday sale, thereby increasing the overall results for your campaign.
• You can leverage this insight into the way in which you structure and plan user journeys for future campaigns

The counter argument is that you may then change the home page to include category tiles for the remainder of the year, but what if none sale traffic behaves differently. This is a valid concern, and so we usually always retest test wins post promotion or sale to ensure that the insight or win can be replicated in none promotion times.

If the test doesn't win outside of the promotion it's no less valid, it's just now an insight for us to only leverage during promotions or sales.

2. They just don't want to 'mess with anything' during the critical promotion times

This is actually an easy one to solve: do proper Quality Assurance on your tests.

Whenever we run split tests any time of the year we do QA across a range of browsers and devices to ensure that no bugs have been created before any real user sees that variant. This is critical to any healthy testing programme because without it your test results will never be reliable and you'll be moving around in the dark.

If you have a QA process that's robust, carefully thought out and executed every single time, you'll find that there's not much to worry about during critical times of the year.

Should I do split testing if I'm building a new site?

This is a nuanced question and the answer is: it depends.

You should do split testing on a site that you're ultimately going to abandon if:

• You're going to use the old site front end as a base for the new site front end.

• The new site build will take more than 3 months - sometimes you can have major test wins in the first few weeks of a split testing programme that can help to fund the new site or build support from stakeholders for your work.
• You are looking for learnings about what you should cut and what you should keep from the old site when designing the new. Without these learnings you can easily inadvertently cut a feature, layout or user journey that was helping the business tremendously on the old site because of your own sentiments to the old site and all of its baggage.

You shouldn't do split testing on a site that you're ultimately going to abandon if:

• The traffic mix or product set are going to be tremendously different on the new site when compared to the old site.

That's really the only context within which I can see little to no value testing a site that you're leaving. Particularly with redesigns and rebuilds, the most important thing is to make sure that the new site is going to perform better than the old site. It's very difficult to achieve that objective if you're not entirely sure what is good and what is bad about the old site. The only way to be sure is through research and testing.

Should I split testing if I'm planning on changing backends?

A much easier one to answer. You can always choose the depth of your testing. If you're going to heavily overhaul 'deeper' aspects of the site like search, recommendation engines or eCommerce backends then the learning doesn't have to stop. Shift your testing programme to be front end focused. Potentially spend some time on customer research and psychological aspects of your messaging and design. That being said, the same principles of the previous point apply regarding learning from the old to inform the new.

In short, there are very few conditions that would make testing in an always on, iterative fashion anything but a hugely value adding activity for online teams.

Start small. Start anywhere, but start today.

Conversion Rate Pros is a pure CRO agency based in Cape Town, South Africa. Interested in getting started with your own research and split testing strategy? Let's chat.