Performance Testing - 3 Things to Know

Like to share what I learned doing load and performance testing. It's one thing to script and run the test, it's quite another to prepare and complete an effective test.

1. Prepare the test scope and plan - before you test. This entails figuring out how much load your application should support without failing. Failing could mean a slow response time, outright failure (website not responding), or something worse. An example of worse is showing one user another user's data. These are all things that can only be checked during performance testing - and will likely not get caught during regular functional (system, acceptance, etc) testing. My best story about not setting up an appropriate scope - is when someone demanded the test be run to the full ability of the load testing license (which was 100,000 simulated users). The actual expected usage was only 10 per day. The lack of setting scope and planning was expensive as the web-server, login function, everything, stopped working long before ramp-up.
2. Now that we've planned... have appropriate support while running the test. This includes someone who can capture data during the run (system performance like CPU, database response, errors, application log files, user experience samples). By not having a mechanic watching the race car during the race - one ends up with a coroner examining the body later. My best example of doing this well - is when a DBA or a lead programmer observes something totally baffling and says - let me turn on some more logging and I'de like you to run that exact same test again. That's a win.
3. Following our planning and execution... analyze the results before signing off. A report including a summary of the tests, analysis from the load testing specialist, the application lead, and others (system admin, DBA, usability specialist, security, etc) should be gathered and presented to those who care. The people who care include those who are paying for the application, those who will support it (think about the Healthcare Signup issue a few years back), and the development team who owns the application. Failure to analyze the results is almost as bad as failing to do performance testing.