A/B Testing Framework as a Service

TL;DR - I did a really cool thing and here is the project link.

This past weekend was the 2018 hackathon on the SFSU campus. It was my first hackathon and was only a tiny bit overwhelming. After brainstorming for while on ideas, and looking at the API's provided, my team decided to go forward using none of them and instead build a baseline project as suggested at a Reddit panel brought on campus by the LeanIn chapter on campus. The Project was A/B testing, which we modified to be as a service, might as well be ambitious. I figured that it would be the perfect time to collaborate with my peers in a moment of downtime, as we were all ready for a change of pace. 24 hours of straight development might not have been the biggest change from 2 weeks of midterms, but a change in tools was a nice transition. You can see the project here.

The main goal of this project was to design a framework for testing components of applications involving a Server/Client paradigm. The overarching idea was to utilize node.js and express templating to create a modular, non-intrusive way of determining the best of multiple features by bucketing the delivery of files. By tracking simple metrics such as time between clicks, new visitors, and repeated visits, among other basic metrics through anonymous identifiers in the form of cookies, we are able to create much more complicated metrics including ratios summarizing user retention and similar ideas. We imagined a really impressive presentation moment where we could have all of the audience members access the site and do live data updating with Ajax calls, but that was a bit ambitious for 24 hours. We also envisioned the ability to interface another web-application from another group at the hackathons, but getting used to new technologies was a big bottleneck to get to those goals. Anything can seem easy on paper it seems.

Below is was an example of the Plotly API interacting with sample data from the dataset, due to the fact that even though we had data flowing into the database, we couldn't reasonably create enough unbiased material to work with. Sample seemed like the best way to approach. In the example, you can clearly see one example of clustering of A vs. B, which was the essence of the project.

The API’s presented from Macy’s, Twilio, and IBM created a lot of development space, and I would have liked to implement IBM’s AI library, but time ran short. 24 hours is relatively short, especially without sleep. Even though we didn’t get to include some of the features ballparked, having one of our team members able to get a crash course in Node, group development with git and laundry-list of technology we used was well worth the missing features. I felt very comfortable in a teaching role, with the PINC mentorship role on campus giving me that experience; a full day was a far more strenuous time period than a 1-2 hours meeting. I can definitively say that kind of development cycle is not for me. I felt comfortable in the group setting in both production and design settings.

One of the big challenges that I’m facing in both PINC mentorship and in general is how to cater to an individual’s needs while still meeting the requirements of the group. Ironically, the problem can be solved with the A/B framework we began to develop. The span of the use case of the project was staggering once we realized that we could use the framework to test the data display page itself. Going along with the advice from the MLH representative, I am looking to open source this project and hopefully make it available for rapid-prototyping of projects where you don't have time to create testing environments, or for the use-case where the individual does not have the resources to do the analysis and they want to learn how to get started. Looking at user-experience is an interesting change from the way that I looked at it while working at Blue Bottle because direct feedback is not accessible, we have to make inferences. I'm looking forward to exploring the project.

Feedback is greatly appreciated. Thanks for the read!

 Extension goals:

• Additions of Multiple Linear and Logistic Regressions
• More Complex single and Composite Metrics
• Variable Selection methods to optimize the analysis necessary
• Added complexity in cookies for metric analysis
• Full Client System for modularity in files. Plug and play functionality with tracking .ejs files through js calls, to be used as a service.
• Pinging the Server to determine of time user spent on the site
• Open sourcing the project and continuing development
• More Dynamic visitor bucketing and funneling of features, with an interface for the site owner.
• Use of IBMs AI library
• Using cookies to implement multipage statistics through conditional probabilities

Github link to the project

Technologies used:

Github, AWS ( EC2 ) , node.js, Plotly, Falcon2SQL client, SQL, HTML, CSS, EJS