AB Framework notes

A robust AB Framework needs to solve for the following:

• State Management

 The two obvious solutions are to keep state on the client using local storage or cookies, or to keep state on the server most likely in a memory cache (mapped to a smaller client-side state holding a unique identifier). Keeping state on the client is way easier so an encrypted and signed cookie should do the trick. 

• User distribution and cohort management (Even or nearly even distribution)

Random number generator Utility and and cookies should ensure that user A always sees version A and user B always sees version B across sessions 

Even distribution strategy 

Example - Generating 8 random bits repeatedly will produce an evenly distributed set of values between 0 and 255. If we want a random number between 0 and 200, we still need 8 bits (7 bits will only get us up to 127) but we might end up with values outside our desired range. We can’t just ignore the most-significant-bit when the number is over 200 because that will skew our even distribution. To get even distribution we have to throw out the entire 8 bits block and generate a new one, until we get a value within our 0-200 range. Let’s call this x 

Cohort Management 

Now based on the number of versions being AB tested – let’s say you have an A-B-C test running to test 3 versions, compute x modulo 3 to decide whether to serve A or B or C, store it as the value of persistent cookie. (y=x%3) 

• Analytics integration

Set this as value for a predefined variable in the analytics system(say GA/Omniture)

For eg. configure a variable named AB segment and as part of AB test always set the value of this variable as ‘y’. This is key for performance measurement for different segments. 

• Test configuration

AB tests can broadly be bucketed into two categories 

• Data points, order of results, turn on and turn off certain features

Eg. Order of social logins

Show only economy flights in first go (Filter turn on and off)

Play around with Popular search order on search results page.

Support in API using additional flags 

• UI layer –Typography, colors, grid

If using an MVC framework for client side such as Angular JS, use templates

            On top of this you need a test definition system, can use simple JSON based test definition system to define the test, this in turn turns on or off flags in API request or flips the template as discussed above. So the AB framework reads the test definition supplied in a JSON and handles it. This will greatly increase the velocity of experiments. Depending on the intended complexity of tests this test definition system can be fairly simple.

• Metrics

Defining right success metrics for every experiment is the key and also a prioritized stack rank of hypothesis which we believe will definitely create impact.