Performance testing experience in production like, scaled down & cloud environments.
The following tests will be conducted to ensure that the application is error free and will perform as expected in Production
SIT Environment - 70% of production size
Profiling:- Implement the diagnostic tools and capture the server metrics and provide recommendations to development
Sanity Test: - This is basic level testing (small scale tests )to perform sanity check and find out any technical and compatibility issues by adequate monitoring analysis
Business Verification Test & Design Validation Test: - The design validation test focuses on minimum load test (like 50% load) using VUgen protocols to ensure that there is complete coverage for all protocols across all business flows.
The following are the high level guidelines to be defined before acknowledging the scale down environment
- When determining the size of a scaled down environment for performance testing care must be taken to ensure every component/tier in the system is adequately sized down in such a way that no single component/tier becomes a bottleneck
- The configuration of an individual server (For ex; Application server) need to be identical to that of production environment even though the number of application servers is scaled down
- It is recommended to always run a baseline performance test (prior to current release changes) and then run the new code (with latest release changes) so as to measure the delta of the Application/System performance to get a realistic picture of performance
- Proper Environment configuration is crucial when determining scaled down environment. The number of connections, threads, sessions, processes and server specific configurable items
Risks of having scale down version
- Database server performance like other servers (Application server, Middleware) is not linear. Hence when testing in a scaled down environment with small database volumes the query response times and throughput are not true representatives of those in production environment
- In most performance test environments network configuration is one area that is often not appropriately sized. Proxy servers, Load Balancers, Firewalls and network devices that manipulate traffic have significant impact on the overall performance
UAT (Production like) / Cloud- Based environment
Real Time Test: - The real time testing focuses on measuring the Application health and Performance in terms of responsiveness of servers when there is realistic production like workload.
While this envisage the Perf lab concept. Your lab environment should be isolated and for DB servers try to use Physical servers instead of VM's. Also with virtualized environment make sure the hypervisor should be same as PROD.
useful information....
Good read. Agree with the risks you mentioned about using scaled down environment. But unfortunately, this has become the usual challenge in most of the projects. Here is where our industry benchmarks like TPC / SPEC helps us a lot. Majority of the time, to define the % difference of scaled down test environment with respect to production itself sometimes requires lot of analysis. In your case, all we need is to assess the computing power difference of CPU (in both environments) & thereby service demand differences for your chosen workload. That will help to derive your performance SLAs for your test environment.