Does quantifying performance in the Cloud drive you mad?

When I first discussed the challenges of performance management in the Cloud with Dr. Ng from Rice University we quickly agreed that it was a tricky issue with many different dynamics.

Both of us were familiar with traditional performance testing. However we also believed that those practices are only partially applicable to understanding cloud computing performance. We agreed that in order to fully understand performance in the cloud one cannot rely on a static report based a snapshot in place and time. The cloud changes too rapidly: New products, new locations, new software updates and new hardware emerge daily and are affecting performance accordingly.

We needed to do something quite different: to do it right one would have to measure all instances, all the time and everywhere. Unfortunately that is a lot of testing. Fortunately the cloud also presented the solution to the economic constraint: on demand computing.

The result of the collaboration with Rice is the industry’s first comprehensive and continuous price-performance benchmark. Burstorm tests all instances of our target providers, several times at random days during the week, continually (to emulate all the time), at locations in Europe, the USA and Asia (to emulate everywhere).

Using a high degree of automation the most recent scope shows the 7 major suppliers representing  153 instance types. Each instance type is tested in 3 locations for a total of 459 test runs per cycle. You might think this is a lot but we currently have 1075 product sets in our database with over 40,000 different products. We are just getting started.

We have been testing since June 2015 and are continuing to do so while adding new instance types and vendors. Today this represents more than 24,000 data points.

Another key differentiation to traditional performance measurement is the need to include the different pricing models which are a key innovation of cloud computing.  Therefore we include pricing in our data collection and the results are normalized to a 720-hour month since several providers use a different number of hours in a month.

Doing this turned out to be quite revealing by the way. Look at what happens when you look at 4 core systems by performance and then by price performance:

If you’d only look at performance you would have selected a G2 from Azure, however you can spin up almost 22 Linode-4GB instances for the same price. Its like cars, they might all have 4 wheels, an engine, a steering wheel and brakes but that does not make a Ferrari the same as a Prius.

Quantifying performance in the cloud is hard and we believe its only going to get harder. That is why we are on a quest to measure all instances, all the time and everywhere. We are also expanding the tests themselves to provide more visibility in additional workload performance profiles. 

To make it even more powerful we also added the capability for members of the Burstorm application to add their own system’s price performance data and compare it to the published instances. 

Feel free to connect with me if you want to learn more and get access to this data so you don't have to pull out your hair out anymore and can do your own research.