A/B Testing: Review

I spent last week reading and realizing how adaptable A/B testing can be for young companies and how its importance is only going to get intense in the near future as the marketing decisions of the company become more data driven and online media focussed.

One of the very crucial requirements of A/B testing is that we need to a have enough observations to infer results which are actionable. A significant results backed by good power (95% confidence or above) can mostly start from 10,000 conversions. Then there are phases consequently based on number of observations beyond this number which will provide even better results. While we test our hypotheses on changes observed in KPIs, this selection of KPIs can almost change the game. While number of completed transactions can be a good KPI, average order size isn’t because few big orders can completely change the picture and would give wrong interpretations of progress. Similarly, the test should be consistent over a time and we should keep in mind that power of test will directly relate the outcomes in real world to the one we got in the test results.

A/B test is one tool that can aid our conversion optimization process like nothing else. Probably without it, it’s difficult to achieve those conversion figures which just puts the company to break-even. While a company’s growing, it needs to conduct a lot of tests through hypothesis testing methods which mainly comprises of A/B testing. One example of A/B testing can be like that we take the current page and then put a ‘testing page’ next to it which is different, may be in terms of content, design, page architecture etc. depending on what’s the purpose of testing. These tests are time constrained, can be from a week to a month, and results of which are analysed for detecting the desired changes or affects. It may sound easier to read than to do it practically.

There is great formula to see if we are doing the right tests or not.

A/B testing success formula = Relevant location * relevant hypothesis * chance there’s an effect. 

Thus it clearly implies that we should prioritize our A/B testing based on what impact the KPIs have for which we are testing. It’s important to identify which KPIs are a true indication of effect and which are just vanity metrics. Following identifies what do I mean by each of the term in the above formula.

Relevant Location-It means is selecting and prioritizing pages which has high alienation with the hypothesis being tested, and is potentially highly impactful for the core business metrics. For instance, it can ‘no of actual transactions’, ‘number of email signup’ etc. In a very rare situation when a strong and significant effect fails because it might be running in a wrong location. It has then an extra risk of invalidating the correct hypothesis due to undesired results.

Therefore it is very important that the hypotheses is backed up with a proper research. We should keep that in mind that we don’t fall in the trap of testing templates because of certain norms for industry, but testing product performance & only those associations which impact the right KPIs directly.

Relevant Hypotheses- But how to zero down to one hypothesis. Actually, it’s a good sign to come up with many hypotheses initially. This shows that we’ve tested multiple possibilities already and the final result is going to precise. Generally it is a good idea to directly talk to as many consumers as possible to design our hypotheses because this will let us know about as many possibilities, consumer feedbacks, user behaviours, doubts and challenges. Growth marketing presents 6V conversion canvas. The following paras describe what exactly this framework is and how each of these areas are full of insights waiting to be unravelled.

1)   Value: What are the company’s values which it wants to retain in its pursuit of capturing market? What focus would then deliver most impactful change?

2)   Versus: What competitor analysis and market best practices can be fraud?

3)   View: What sights can be found from web analytics and web data?

4)   Validated: What analyses have already been validate or rejected in the previous tests?

5)   Voice: What scientific researches, insights and models are available?

6)   Verified: What scientific research and insights are already available?

 After brainstorming with a good number of hypotheses, we can start hunting for specific ones based on location of tasting. And then choosing which one to go with based on potential effect.

Potential effect: It is equally important then to check for the potential effect of each of our tests. Yes it’s obvious knowledge but just to reiterate, it would be a huge loss of time and money to run experiments based on hypotheses which are targeting low impact KPIs. Potential impact also depends on the important KPIs which vary according to the seasons, business needs and all. But once we identify it, we can gain significant insights because these KPIs in most probability will be directly related to the revenues. Sometimes, it’s profitable even to optimize the not-the-most important potential affect but which can lead to a great change in the KPI which directly leads to revenue changes. For example, if a company optimizes the checkout pages (if it sees major downturn there), it can lead to more completed transactions.

The next step is to create a ‘experiments roadmap’ once we have these 3 components i.e. location, relevant hypothesis and high potential effect. Roadmap gives a broad idea about what all tasks need to be done. Each test has locations, hypotheses derived from some psychological determinants and a stage of consumer journey. Each test thus should be calculated for MDE- Minimum Desired Effect and once the experiments are live and running, on actual results. This thus would lead to create a priority order and timeline of A/B tests to be run.

I would like to thank CXL Institute for coming up with such a wonderful blend of courses which has continuously been a great source of learning for me till now. The above article is overview of topics on ‘A/B testing mastery’. I’ll be maintaining the flow of these weekly articles in future as well. Stay tuned!