Performance Testing your Salesforce Application

Introduction

The Force.com platform serves Billions of transactions a day and is known to be highly scalable. Salesforce.com themselves tests their platform, maintains required excess capacity and ensures a consistent experience for all users of their system.

 If that’s the case, do we really need to load test a Salesforce application? Can’t we just develop our application and let Salesforce take care of the resources and performance bottlenecks? The biggest performance bottleneck that Salesforce.com (even with their infinite resources) has not yet been able to overcome is ‘BAD CODING’. An inefficient piece of code can literally break or become the weakest link in a Salesforce application.

It is also possible that a piece of code, that is performing just fine under current load conditions, may yield in future once the data volume increases. Normal unit testing may not be sufficient to identify such performance bottlenecks. You will have to ensure that your application is scalable and can meet demanding application conditions.

That is when and why Performance Testing becomes crucial.

What exactly is Performance Testing?

Performance refers to information regarding your application’s response time, throughput and resource utilization levels. Application Performance and Customer Loyalty goes hand in hand. Unpredictable Performance, inability to scale up to fluctuating demands, service disruptions etc. are serious limitations that directly impact any company’s revenue.

Performance Testing normally includes Stress and Load testing.

• Stress Testing is used to gauge performance under extreme conditions often caused by diminished resources or excessive requests.  
• Load Testing is used to gauge performance under expected conditions with varying loads.

Performance testing is a must if your code is highly customized or if you expect large transaction volumes.

Main Goals Of Performance Testing

The main goals of performance testing on Force.com platform are

• To ensure that the custom application meets the desired response times
• To determine if the transaction throughput of anticipated load is as estimated.

Recently an ‘Attinad Software’ client who is in Real Estate Domain, switched over to Salesforce for their CRM needs. Their system had capabilities to identify leads, assign agents for leads, have the leads converted etc. The system was working fine as per the business requirements.

But the client had a growing customer base and they had also done complex custom development on Salesforce side while integrating with many third party systems. They wanted to ensure that the system was capable of efficiently handling higher loads without performance depreciation. They approached us with this problem statement and in discussion with their IT team, we suggested performance testing of Salesforce web services and UI services using their expected maximum loads. Attinad Software was responsible for developing the end to end strategy and the execution of the same

The client identified 17 web service APIs and a UI service that needs to be load tested. This included services for creating Leads, agent assign, activity adds, updating agent info etc. Some of these were ‘POST’ services and some were ‘GET’s.

Sample load details for some of the services, that the client wanted tested, are as listed below.

It was decided that the Performance testing will be limited to load testing only for two reasons.

1. Even if the Client’s operations added a few million transactions daily, that would only be a negligible percent of the Salesforce.com capacity
2. Stress testing might trigger system governor limits added to ensure resource sharing for all Salesforce.com customers.

The following were the key highlights of our solution .

1. Using Apache JMETER: Performance tools evaluation and comparison was done against various decision factors like testing objectives and outcomes expected, commercial impacts etc and finally JMeter was decided on after evaluating other options like HP Loadrunner, LoadUI, CloudTest etc. Since JMETER was open source, it had some limitations and needed a more extensive scripting for achieving the required performance criteria.
2. Extensive Scripting and Configuration for Complex Load scenarios: The requirements itself was a bit complicated. We had to execute the various threads in a certain sequence and with some time delays in between. Also the required loads for different services were different which had to be accounted for. So we had to adjust the loop counters, put timers in between, multiply or eliminate some of the threads to achieve the loads etc.
3. Record and Playback of certain use-cases: The UI service had to be recorded using JMETER and for the actual test, the recorded event had to be played back in a loop to achieve the required loads. There were some issues in recording and playing this back since it was using a HTTPS protocol.
4. Execution using Cloud Server: Other than the above, due to the huge volumes of threads and data involved, it was near impossible to run the actual full load test using a normal server. So we decided to go for a 14 GB RAM cloud server.

The actual testing had to be properly planned to overcome the following constraints:

• There was another development team working on the same Salesforce Sandbox instance. The performance testing had to be coordinated with them.
• Salesforce does not allow Load Testing in production instances and we had to do that on a Full Sandbox which is a near perfect copy of actual production.

Testing Outcome and Actions Taken

The test results were captured in graphical as well as tabular format.

Screenshots of a few reports that were generated for this project are given below.

Most of the services remained stable even at peak loads and showed little variation. 

But there were few services which showed some performance fluctuations at higher loads. These webservices were analysed for possible reasons for the performance depreciation and the root cause was identified as

• There were a few SOQL statements which were not normalized properly and the filter conditions and nested queries was causing a performance depreciation at higher loads
• There were some nested loops which were not properly handled and this was also causing issues

Corrected these issues and reran the load testing process for these services. After 2 iterations, we were able to bring down the issues and improve the performance to the client expected values.

This testing and the later corrections made the system more stable, reliable and the CUSTOMER HAPPY.

“Don't lower your expectations to meet the performance. Raise your level of performance to meet the expectations “