Starting Foundations for A/B Testing

To give you a little background, I've been executing A/B tests for over 15 years at this point. I first started with paid media but quickly moved onto testing digital experiences, printed collateral and other marketing pieces. At this point in my career, I've probably tested over 1000 different experiences and pieces of collateral.

Many marketers don't realize it but almost anything can be tested as long as there is a way to randomly sample a targeted audience. Back when I was more focused on direct marketing, we would send out near a million direct mail pieces at time. With this sort of scale, I wanted to make sure that we tested around 20% of audience at any given time.

Nowadays, more of my testing is focused on digital experiences but the same practices hold true. When building and maintaining digital experiences, there is no excuse why you shouldn't be testing a new experience to around 20% of your web audience.

Here are some of the requirements that I have come across when determining what makes up a great A/B test:

Scale

Often times, test ideation is centered around an assumption that something is could be better. That is honestly a great starting point for any test planning. However, if what is being tested has limited scale, it may be difficult to understand if a test is successful and what can be learned from the results of the test. Keep in mind that you want to test to improve results. Testing on content or marketing collateral that has limited scale means that the impact of the results will have limited reach. Any successful testing plan will prioritize based on the potential impact.

When considering scale, consider the following red flags:

• Testing on a webpage or website with limited traffic.
• Testing on an audience that is small in size and is difficult to randomly sample.
• Testing too many variations within a limited universe.

Measurable

One of the biggest issues I've come across with test ideation and test design is how to measure success of a test. Often times, it will be difficult to measure success because success was never defined from the beginning. This results in one of two scenarios:

1. No one clear metric exists to measure success.
2. Too many metrics exists to measure success, so they will all be included in a report which often results in conflicted information.

When setting up tests, make sure to outline what success is and equate to either a KPI or Metric. When reporting on the results of a test, limit your reporting to one primary KPI and at max two supporting metrics that could perhaps explain the results better but won't be used to conflict the results.

Here is a sentence to live by when setting up measurable goals for an A/B test:

By changing _____ into _____, I can get more _____ to _____ and thus increase _____.

What you are asking yourself here are the foundations of: Who, What, Where, When, Why and How. Here is that template with an example:

By changing the CTA of "Learn more" into "Buy now", I can get more prospects to click the CTA and thus increase the number of leads I get on my website.

Of course, this is a very simple hypothesis in this scenario, but you get the drift. Tests have to drive something, otherwise it's not testing is it?

Impactful

Impact goes hand-in-hand with both Scale and Measurable. An Impactful test means testing content, creative, experiences or an audience that we know will have a meaningful difference in the result. We need Scale to make sure the test we are executing is large enough to drive the business forward. Creating a Measurable test is important to make sure the changes that are being optimized have a positive influence on the business. Where, Impact comes into play is to make sure the test meets the goals of the organization.

Here are some examples of test ideas that are not impactful:

• Testing content that is rarely seen by users.
• Testing outside-the-funnel actions while measuring that they will impact lower-funnel metrics.
• Testing supporting content that is non-action oriented and expecting an measurable action in response.

This is a drastic example, but gets the point across. You wouldn't test some of the links or the copy at the bottom of your page because they don't drive your business goals. In this example on CNN, links like "Terms of Use" and "Privacy Policy" are legally required to be there and don't have a meaningful impact on your site.

The same thing can be said about testing two different shades of light blue for a button. Unless there is a clear usability or accessibility benefit, it likely won't have much impact.

The same can be said about testing long-form copy changes. Users normally scan on both printed and digital experiences so most users won't notice the changes. Focus your testing on headlines, call-to-actions, bullet-points and other copy that sticks out. Alternatively, test on making copy and call-to-actions stand out more.

Repeatable

When mentioning repeatable as a core requirement of a test, it's not always about being able to repeat the test. Repeatable as a requirement means one of the following things and could at times mean all of them:

Activation

Activation is the step of finalizing a test and pushing the test live for all to see. This is a common difficulty for organizations because testing practices and technology teams don't typically communicate. Running a test that won't get activated (either initially or through future iterations) is little more than a high school science experiment. It's a lot of fun at the time, but it isn't going to change your big picture. Consider this during ideation, if you know that your technology team struggles to update core parts of your experiences, focus on items that can be supported for activation by your technology team.

Iterations

So you didn't knock it out of the park for that first test, consider how a test can be iterated as time goes on for improvements. Iterative testing is the foundation of a strong optimization program. Take the insights that you gained for the first test and determine what fell short and how the next experience can be improved.

Gaining Insights

While it's great to run tests, the ultimate end goal is to gain insights that can be used during both activation and future iterations. If you aren't able to gain insights from your test, your test likely didn't meet one of the first three criteria that we covered (Scale, Measurement, Impactful). It's also great to adapt any insights you have obtained and use them for future iterations. In many tests, you may have to continue to iterate in order to learn.

Conclusion

So no matter your testing platform, it's key to have these core foundations as part of your testing program. It's essential to test scenarios that have scale and can be impactful. Make sure to set out any measure plans before hand to make sure you understand success. Finally, once the test is complete, work towards activation and if that isn't possible based on results, keep on iterating.

Questions or Comments?

Matthew Simpson

msimpson [ @ ] criticalmass

@pitchblende on twitter

Feel free to comment