How to create good test inventory and fastpath inventory assessment

A bad release history of application, wreck of the last project executed, project kept on hold due to code quality, bad quality test cases and lot of deferred defects. Project relaunched and we need to test the application base code and proposed customizations..

Bad introduction huh! In the current competitive multivendor run programs and projects, it may not be an uncommon situation. The above situation can be addressed easily, if we have a good and proven test inventory, however, in majority of the cases, we may not have one. In such cases, we may have to start from scratch by integrating the regression test strategy with customization / additional feature testing. This article makes an effort to explain how we need to handle similar situations and create a good test inventory for the future releases.

Let me try explaining this through a scenario (a bit complex though). Let’s assume,

1. Vendor 1 provides a solution to effectively manage one of the client’s portfolios.
2. Requirements are provided by the client’s line of business,
3. Tested by vendor 2 successfully. 
4. The product is rolled out with version X.0

Further,

1. The client asked the vendor 1 to initiate the development for some additional customizations requested by line of business and the vendor initiated the development of version Y.0.
2. No new requirements were created to address the needs of the line of business and the vendor 2 who was entrusted with the responsibility of the testing the new version Y.0 has created a regression suite based on their previous understanding of the application and the design document created by vendor 1.
3. Large number of defects were identified in version Y.0 and the version upgrade project was put on hold, owing to finding of significant number of defects. Vendor 1 was asked to come back with defect free code to continue the testing further.
4. Vendor 1 delivered the new code and the project was re-launched. The client vendor 3 for testing version Y.

Goal?

The primary objective here is to determine the quality (stability) of the base code and if all the defects are fixed and the code is clean, to apply the customizations

 On the date of entry, vendor 3 will have

• Bad release history for the application
• Deferred defects in defects module of HP Quality Center
• Possibly new faulty code
• Regression suite with unknown quality
• Zero domain & application knowledge
• Above all, little time to confirm that the base code is stable.

Where do we start? The biggest question in front of us will be, How do we create a good test inventory that will help us determine the stability of the application in a short period?

Best Way to start is to Build the Application Knowledge

The first and foremost need is to build the application knowledge in the team by working with SMEs in understanding,

• the business need for the application,
• the history of previous releases,
• important functionalities,
• technical specifications
• how the application interacts with the upstream and downstream applications.

The team also needs to, 

• interact with the development team,
• go through the design documents,
• release notes from the previous releases,
• play with the application in test environment to get some hands on and sought clarification from the SMEs & BAs as needed to better understand the functionalities. 

So by now, the team will be reasonably equipped with the application knowledge and atleast we know what we are looking at whenever we look at. So what do we do next?

Assessment & Due Diligence

The next important thing is to assess the inherited documentation and test cases. Also, when we go through the traceability matrix, there may be a scenario where test cases created earlier does not have good requirements linked to them or the requirements may be mere names of the functionalities

Moreover, as we now know the size and complexity of the application reasonably, we may have following questions in mind.

• Are these test cases needed for an application with this size and complexity?
• What is the quality of these test cases?
• Have the test cases been parameterized and modularized enough? If we had not done this, then there is every chance for having more number of test cases than required.
• Do I have enough time to assess all the test cases and execute all the test cases? 

What next?

To answer the above questions, we need to

1. Vet the functionalities with the business:  identify the core functionalities in the repository for which the above test cases were created. Vet those functionalities with the help of business and the subject matter experts. During this exercise, we need to mark the functionalities which are deemed to be extremely critical for business separately. Thereby we will slim down the number of functionalities from N to N-n and the total number of test cases related to them will be a smaller subset of regression suite.
2. The test cases related to deferred defects: Go through each defect including the attachments enclosed and then identify the test cases linked to these defects. Also identify the defects which had a very long turnaround time for defect fix based on the delta between the creation date and fixed date and the related test cases. While executing this step, we had few challenges which I will discuss a little later.

Now, if the above test cases pass, we can safely assume that the base code is reasonably stable. Then, take a deep dive into these test cases and review all the test cases pertaining to the above two categories to identify,

• whether more test cases were created than actually needed
• there is enough modularization and parameterization in the test cases.

We may conclude that there is a definite need for refinement of the test cases. 

Now what about quality of defects written in the past? To address this, we need to scrutinize,

• Whether every defect is linked to atleast one test case
• Whether the defects were linked to appropriate test case
• Whether the status of the defect and latest available comments are in sync
• Whether enough screenshots and attachments were available for analyzing the defects

Trust me. The findings may not be always positive 😊. And there may be a need for analyzing each of the defects, involving SME / BA wherever needed, adding the comments appropriately and bringing the test cases and linked defects into sync.

Are We Well Equipped Now

After the above due diligence and assessment, now we know -

• Reasonably well about the application
• The focus areas for testing the stability of the base code which are business critical functionalities and deferred defect related test cases
• The quality of the existing test cases and defects and need for refinement.

Wait! Did we miss something? What is in regression suite for this application? We have not looked at that yet. So our next step is to take a deep dive into existing regression folder for this application. We should review. Frequently, we will find that the project test cases are just copied and pasted into regression folder which is a blunder. No project will have either time or budget to execute all the test cases created during the previous releases in order to ensure that the new code has not affected the old code.

My mentor has always advised me that any testing must be driven by objective and the objective of regression testing must be well understood by us before determining the regression suite. According to me, a good regression suite must allow me to have confidence that if executed, there is a scope for less than 5% production defects. The suit we construct should cover the critical functionalities and defect prone areas and now we are sure that this suite under construction can be used in multiple releases to test the stability of the application.

• However, we need to note that the job does not end here. The regression suite needs to be enhanced once the test case creation for the customization is completed in case the client decided for applying the customizations. 

 The final clean up

Now that we know what to do, how to do, we have

• Updated the existing test cases to ensure test cases are modular and parameterized
•  Created new test cases for the uncovered business critical and defect related scenarios
•  Created the linkages between the defects and the test cases
•  Updated the defect comments and ensured that they are in sync with the test case execution status

 Conclusion

Key point here is to understand that a good test inventory and regression suite are equally (sometimes more) important than the project testing and we should factor enough time during the project execution irrespective of delivery model we work as it costs more to fix a missed critical regression defect in the production than the new code related incident !