Continuous Performance Engineering Strategy - II

Please refer the part I of this article in which I discussed the "Where to start" and "Non-Functional requirements" with alluding to the  Software Performance Engineering Strategy/Model that I was referring to as follows,

So allow me to start the continuation,

Realistic Performance Scenario selection

Selecting the correct performance scenarios that simulate the real user behavior is the key focus of this phase. Also, the scope of the test scenario selection is based on the risk and the delivery timelines as well.

The performance scenario selection can be classified as follows for effective selection. There are other consideration factors as well, but this would be a guide to start.

• Business-critical scenario:

The workflows/features of an application which is very critical in terms of the high usage (both concurrent user load and frequency of using it or either of them). e.g :- Patient registration/admission of a health care system, Product check-out page, product search page in an E-commerce site and Student Registration in an E-learning platform.

Features\functions that generate high revenue to an organization

•  Critical dependency for one or more functionalities (may be low in concurrent user load or frequency of using).

E.g:- Course creation of an E-learning platform, though the usage is low very critical functionality of the overall system.

• Functions or features that are having a technical complexity or fragility of a system could cause a performance issue.

Tips:

As explained above some aspects such as no of users of a system, anticipated load and critical dependency, etc can be obtained from the NFRs as input for this stage.

 Does a performance scenario always need to be “end to end (system end to end flow)”?

• No, it depends on which application layer cover/test from the performance test. E.g – Performance testing APIs, performance testing a database with high volume through SQL scripts.
•  Also, it’s a good practice to think about the time it takes to complete a test depending on the steps involved in the end-to-end flow. Simulating too many steps through a performance script may introduce instability, maintenance and complexity issues. So, it is recommended to simplify the scripts as much as possible to cover the non-functional objectives.  

Deriving an Effective Performance Test Environment

 It is best if a production-like environment is available to conduct performance tests to identify issues effectively, however, this is very costly and can’t afford by most of the companies in the real world.

So, proposing the following approach to derive an internal test environment to simulate performance tests under production similar hardware, software and configuration set up. This leads to predicting performance behaviors more realistically.

Note:- This is a very simple form of a method called “Extrapolation Method” in performance engineering to test an application with a lesser number of users in a scaled-down environment. However, there are more scientific extrapolation methods that you can apply based on the maturity and the expertise you have on them.

• Derive a scale-down internal performance environment based on a ratio basis (of the live environment) as much as possible.

E.g.: 1:3 ratio basis = If the live/production environment has 9 application servers, then the test environment has 3 application servers. If the production environment has 3 load balancer servers, then the test environment has 1 load balancer server.

• One of the key focuses of this approach is to set up the scale down test environment identically (to the production environment) in terms of hardware and software specifications. However, the settings such as JVM heap memory and server connection limits, etc should be scaled down based on the ratio.
• Another key factor is that the intended user load or transaction per second etc needs to scale down to in line with the same ratio that the test environment was derived. E.g.: -1:3 ratio basis  = If the live/production environment’s concurrent user load or Transactions Per Second (TPS) is 90, then 30 users or TPS to be simulated in the internal test environment.
• This internal testing-related performance test environment should be fully dedicated (not sharable) in all means such as resided in a dedicated network, CPU and Memory etc are dedicated, the performance test environment should be in a dedicated cluster etc,. Residing the load generators inside the same network/environment is recommended unless there is a requirement to measure the end-user response time behavior when accessing from outside.

   Tips :-

Deriving a scale down performance test environment through a scientific extrapolation technic increases the ability in predicting/recreating performance behaviors similar to the production environment.   

Need to conduct a couple of trial executions to finalize that the test env is ready and no noise/issues in the environment on the test result.

What covers in the next episode/part?

Performance Testing Process backed by an engineering process

• Why and what is early performance testing – Code level, SQL profiling, Front end performance profiling, test during the functional test phase, single-user tests.
• How continuous performance testing enables through CI/CD (continuous integration and delivery)
• Testing methods – Comparison based and SLA based
•  What to include to derive an effective testing strategy and many more to include under this section