Performance testing Story...

Infrastructure Audit.. The first step in starting any performance testing project....

…A small performance testing story…

Typical Software delivery scenario ... Two days remaining to go Live !

Final round of meeting to discuss the performance test results....

Scenario … Meeting room … Discuss final performance test results discussions… Business manager, Performance tester…

Business Manager: So, Do we have performance testing results?

Performance tester: Yes, We have completed the final rounds of tests. As of now, we are satisfying the response time NFR however, we are on edge!

Business Manager: What do you mean on edge?

Performance tester: Well, Target was to reach 100 Transactions per second at 200 Users with 2 second transaction response time.  However, When we did a test we found that we get 1.9 second response time at 200 users load.

Business Manager: Hmmm, Can we conclude that We are ready for production? We are there satisfying the NFRs.

Performance tester : Well , I am afraid that System has scalability issues.

Business Manager : What do you mean scalability issues? You are breaking my confidence. I am already running on low confidence because the development, System testing and User acceptance testing took my 95% of project efforts!  Can you elaborate those numbers?

Performance tester : While Analyzing the results during ramp up period , I see response time increases with User load ramping up.

Business Manager : What does that mean ?

Performance tester : It means that , if we increase the users now, Our Transactions per second is not going to increase in linearly however, We might see response time increases exponentially.  Whereas system utilization is going  not increasing linearly. This was confirmed by our single user test, where we see average response time is around 1.2 second.

Business Manager : Aha! Thats the problem! What do we do now?

Performance tester : Well, We might need to dig/profile in code, queries.. We need architect, lead developer and our “busy bee” that is breaking his head with multiple releases… the one and only "performance engineer".

Business Manager : Let’s call all of them in urgent meeting. We are running short of time. It's only two days pending before GO Live!

Meeting Room scene ..

Performance tester : Gents, We have challenge , Our application is not scaling.

Architect : We have invested in good amount of money to get quality infrastructure. System has to scale.

Performance tester : Well that’s the problem. System is not scaling. And here are the results…..

Lead developer : Did we analyze thread dumps , heap dumps? Did we analyze long running queries ?

Performance testers : I am sorry, I have not done however, i have collected the data for you to analyze. they are at this location <218>:/ABCD

Performance Engineer : Did we do any audit after the infrastructure hand over?

Performance tester : What do you mean  by audit?

Performance Engineer : Did we check , CPU, Memory , Network and Disk are according to specification mentioned in architectural diagram? Did we do any audit in general configuration of Web server, Application server, DB server and operating system at each layer?

performance tester : We did not do however, While handover the environment i was told it is according to specification.

Performance Engineer : Thats the first step , We need to do before starting any performance testing  work. let me tell you , CPU and memory size, number and model can be identified by  Operating command. Where as network and Disk we can use some open source tool like TTCP and IOzONE.

Busineess Manager : Hey , lead/Archi, Did you get anly clue on the bottleneck?

Architect : Well every things looks normal. Query execution plan is fine, queries using proper indexes , No threads are in block mode, No synchronous block in code.

Lead : Even, No clue in heap dump, JVM has sufficient memory to take further load.

Business manager: Then , that is bad news !

Performance Engineer : Well, I just now found some good news, based on the TTCP results, I found that there is only 100 Mbps Lan between Web server and Web application server. Based on the small test, It seems , network is chocking ... According to architect specification it has to be 1 Gbps network.

Performance Tester : So, There is bottleneck. And , We missed because, We did not have any specialized tool to monitor network at each and every stack of architecture...

Performance Engineer : Yes, Now let’s contact infrastructure lab guys and move these servers to 1 Gbps Lan.

After Server movement :

Performance Test : Aha! Look at the performance ..200 users 1.2 second response time and 166 Transactions per second !

Business Manager : Now, Thats a good news!  Now that will definately boost my confidence ! Good Job Performance Engineer.

Performance Tester : So, Always do infrastructure audit before starting any performance testing project.

Business Manager : I will initiate the similar exercise for production servers..