Performance Benchmarking

Introduction 

This article’s primary purpose is to highlight the importance of performance testing, how to do it properly, and when you should consider doing it. With our experience at Portaltech Reply, we have performed many roll-outs of extensive transformation programs, platform upgrades & migrations, business capabilities enhancements, and more. 

Why Performance Test 

We all know the importance of performance testing. For us, it's the primary and first non-functional requirement on the sheet. However, it's usually the neglected child in the software development lifecycle, even though it always comes back to bite with a vengeance. It's traditionally forgotten until we get to the last stages of a project and then put it together in a rush. We’ll explain why this approach has its problems later in this article. 

Performance testing is a critical part of the software development and application lifecycle, designed to evaluate the speed, responsiveness, and stability of a software system under various conditions. The primary reasons to conduct performance testing are to ensure the software or system meets its performance objectives and to identify and address performance issues before they affect end-users.  

When Performance Test 

Any new application should have performance testing planned and scheduled as part of project implementation and maintained as an ongoing exercise. 

But when should you consider scheduling it? This is a judgment call, but we recommend doing a performance test for any of the following reasons:  

• Environment sizing and identifying scaling bottlenecks for growth projections 

• Preparing for significant events, such as Black Friday and any other high-traffic marketing periods 

• Quarterly Cadence as a BAU activity or a shorter frequency 

• Core Platform Upgrade 

• A new set of integrations to third parties  

• Introduction of new Functional capability 

• Third-party upgrades or library upgrades 

• Database upgrades or changes 

Type of performance test 

There are several types of performance tests, each designed to evaluate different aspects of a software system's performance. The type of performance test depends on the specific goals and requirements of the application or system under test. Here are some common types of performance tests: 

1. Load Testing - Load testing assesses how a system performs under expected loads. It helps determine if the system can handle the anticipated number of users, transactions, or data volume without degrading performance. The goal is to identify performance bottlenecks and ensure the system meets its performance objectives. 
2. Stress Testing - Stress testing evaluates a system's behaviour under extreme conditions, such as a sudden spike in user traffic, a rapid increase in data volume, or hardware failures. The goal is to understand how the system behaves when pushed beyond its limits and to identify failure points and vulnerabilities. 
3. Spike Testing - Spike testing is similar to stress testing but focuses on sudden and significant increases in user load. It helps assess how well the system can handle abrupt changes in user demand. 
4. Soak Testing - Soak testing involves subjecting the system to a sustained load for an extended period to identify performance degradation or resource leaks. It helps ensure the system remains stable and responsive in the long run. 
5. Scalability Testing - Scalability testing assesses the system's ability to scale horizontally (adding more servers or nodes) or vertically (upgrading hardware resources) to accommodate increased load. It helps determine how well the system can grow to meet future demands. 

Each type of performance test serves a specific purpose in evaluating a system's performance characteristics. Choosing the correct type of performance test or a combination of tests depends on the project's goals, requirements, and the potential performance risks associated with the application or system under test. 

Performance testing Accuracy 

We recommend that performance tests simulate the user's activities on the site in the most trustworthy way possible. Do the testing based on realistic customer scenarios in an environment that is identical to the production environment if you cannot use it? This also means having the external elements of your solution in place—for instance, CDN, WAF, etc. 

This requires an upfront analysis of the user's behaviour on your site. What are the most common Journeys? How do users get to the checkout? These are all questions that can be answered by using analytics tools like Google Analytics, Omniture, etc. 

Performance Testing Steps 

How early, you may ask? Before any development starts, this is also called a baseline or benchmark. If we know the starting point, we know what impact our custom code has had. In the case of a greenfield project, this would be the baseline code/accelerator. If you are developing an existing system, then this is the current system that is live today. 

Define user journeys 

With the client, define a set of user journeys and implement the scripts for these journeys. For most e-commerce sites, these are multiple permutations of the checkout journey, but you could also include journeys in the My Account areas and static content pages. You want a set of journeys that touch every area of the site but emphasise traffic on the most common journeys. 

Load test baseline site (Benchmarking), then run continuous tests 

Run a benchmark load test on the existing site using the scripts implemented in the previous step. At this point, you should know what level of traffic you want to load test with. The client should provide this information, which can be obtained via analytics tools like Google Analytics. 

These must be measurable and typically include estimating the number of cores required, a performance threshold per request (in milliseconds), and the maximum number of concurrent users your application can handle without central processing unit (CPU) saturation. 

Schedule continuous load tests; these should take place regularly so you can identify any issues introduced as soon as possible. 

Investigate bottlenecks 

In this step, you need to identify system bottlenecks and areas of inefficiency, such as slow database queries, inefficient code, or resource limitations. This allows developers to optimize these areas for improved performance. 

Following are a few common issues that typically surface on an SAP Commerce Cloud solution: 

Data maintenance/retention - We typically see that data is not always maintained and the size of the database grows, which can adversely affect DB query times. It’s essential to review table sizes occasionally to ensure the data is cleaned up. The usual suspects in SAP Commerce Cloud solution are props, JobLog, SavedValues and SavedValueEntry. 

DTO conversion - It's tempting to use "out of the box" converters and populators that populate a large amount of data when just a tiny percentage is required on the front end. There are two essential optimisations here. First and foremost, attempt to limit the payload data to what is used on the front end. This is especially useful if you have a headless architecture because the payload must be downloaded from the backend system. The second point that will be raised often is using server-side caching. ehCache regions may be configured to store any data. You should examine the data; if it's not customer-specific (dynamic), you should cache it. A 5- to 10-minute cache on a high-traffic site could benefit greatly. Before implementing this, try to use the existing caching mechanism setup on SAP Commerce, like DTO caching and CMS caching. 

Allocate jobs to the correct aspect - We have a backend processing aspect that handles all the backend tasks like business processes and cronjobs, which can be resource-hungry. These should not run on API or Storefront aspects, primarily server customer activity. 

Search engine optimisation - This usually comes up if you have an extensive product set; Solr instances may need to be configured to support more extensive indexes. This may also apply to external search engines. 

Use Solr properly when relying on Solr for Search - Solr is a form of caching, and the idea is to provide a quick mechanism for searching. Yet we see repeatedly that database calls are executed to populate the data to form the search results or category listing pages. Ideally, everything needed to render this page should come from Solr. Although this is the ideal implementation, we also understand that real-time data is necessary for things like stock and price, but these should be limited to data that is required to be current and correct. 

There is also a configuration element here that is configured from within the codebase. I’m referring to database connection pools, Tomcat thread configurations, etc. 

Sometimes, a third party can harm performance stats; for instance, an external search engine or price manager are APIs that must be invoked in real time and can not always be cached. It would help if there were visibility on third-party integrations, especially real-time API requests triggered by user activity. Identifying these issues will help improve the general performance of the site. 

As a result of this investigation, you may also need to resize certain areas of the infrastructure. This is an activity that needs to be conducted in conjunction with SAP. Using the statistics around the number of concurrent users and the results of your load tests, SAP should be able to size the environment to support this expected traffic level. 

Reporting 

Reporting in performance testing is a fundamental component that ensures transparency, communication, and data-driven decision-making. It helps stakeholders understand the system's performance, identifies issues, and provides a roadmap for improvement. Without comprehensive reporting, it would be challenging to manage and optimize the performance of software systems effectively. 

When can we mark Performance testing as completed? 

When you can replicate more than the expected level of traffic against the application and have it performing within the SLAs, then you can mark this item off your list for now. This should give you the confidence needed from a performance perspective to go live. 

Conclusion 

To minimise surprises later in the project, it is critical to monitor performance early on. Performance testing should be included from the start of your Continuous Integration Pipeline, and tests should be run on a fully loaded and realistic test instance. 

For project success, you should see performance testing as an investment with great returns. Investing in this early on means you are much more likely to identify issues that would have delayed the project's go-live. It's also crucial for the early discovery of any issues with third parties integrated with the overall solution. Load testing a fully integrated system is also indirectly pushing combined 3rd parties. 

The alternative is usually crossing your fingers and hoping everything will work well, which might save you some resources if you’re lucky. But if you’re not, it could mean a site outage, substantial revenue losses, a bad customer experience, and reputation tarnishing. 

 In this last section, we would like to introduce performance testing of any single-page application Storefront (SPA) with specific tools like JMeter. JMeter works on HTTP requests, which is fine when you get a page in the storefront that is not a SPA, but you don’t want to use this approach with any single-page application. This would be like putting the URL of each page in the user journey in the browser and requesting each page with a page refresh. This is not realistic at all. Overcoming this is a topic for which we will be putting an article together, so watch this space.