A/B Testing: Are you doing it right?
In a world which is going mobile at a rapid rate, statistical techniques like A/B Testing are quickly gaining popularity. A time when managers used to take decisions based on their guts is long gone. The consumer is the king and techniques like A/B Testing give them a voice through data.
Unfortunately, most of the tools available in the market do not offer a comprehensive solution. There is always a trade-off between speed and accuracy in any machine learning technique. However, these tools do not give a clear picture about the pros and cons of these trade-offs in A/B Testing. They make you believe that their out-of-the-box solution boils it down to one or two metrics. Features can be judged just by comparing these parameters. As a result, most companies do not experience the explosive growth suggested by these sequential A/B Tests. And worse is the aftermath in which developers spend so much time to understand what went wrong.
One of the most glaring errors that I found with most of these tools is:
Distinguishing between New and Old Users: Think about it, A new user who sees the 'B variant' of the product feels that this is the default behaviour because he/she has never seen the 'A variant' before. However, an old user thinks of 'B variant' as a change in the product because he/she is already familiar with 'A variant'. In such scenario (which is very frequent), it's only fair that we analyse these two groups of users separately.
Even though most tools give you the option to break down the users by New and Old, they do all the analysis based on total population. Whether it is the calculation of sample size required or the calculation of infamous p-values, they do not take this bifurcation into consideration. The problem is not that these tools are just not intelligent enough to take care of these simple business rules. The problem is that these tools give no indication of such problems exist and give no information about uncertainty caused by this. Without assistance from data scientists, the managers have a very high chance of making bad decisions.
At Tuple we are aiming to create applications which are not only intelligent enough to understand these simple rules, they have the ability to talk & learn from other applications. You can easily connect your A/B Testing with your Customer Segmentation and see a complete picture of your business without running into pitfalls. You should see what you get, and you should get what you see.
More on that later. Feel free to drop me a line if you need help with any of this.
