Making Software Less Fallible

1.    Introduction

“If builders built building like programmers wrote programs, the first woodpecker that comes along will destroy civilisation”. - from the Fortune Cookie in Unix

Rather gloomy quote to begin with, but we seized to compare software systems with other mundane things like buildings, cars and aeroplanes mainly because software is more likely to fail than those above. Ironically, for reasons unknown, we developers assume that our programs WILL work. In that context, the bottom line is, to be a good developer, one should not be too confident of the program that one develops. Demoralising and ironical, but true!

Another factor for poor quality is the shorter life cycle of current projects. The fact that System Test is always done before UAT, makes the development team slightly complacent to test all possible scenarios. It is not sure if everyone understands the differences between Unit Test and System Test.

The old adage of Quality to identify ones “internal customers” is paramount in enforcing stricter quality standards to the software we deliver. System Test Team is the internal customer for the development team. Hence it is important that Development Team delivers their program for System Test as if it is delivered to the client.

2.    Reducing fallibility

2.1.          Why programs are fallible?

Software is not infallible. But fallibility can be reduced by proper testing. Many factors can be identified that attribute to the poor quality of the programs:

-         No testing guidelines have been completely laid out in the design document.

-         Design documents that are more process oriented. Hence it cannot be used effectively as a basis for testing.

-         No test script is followed by developers.

-         The test data is not comprehensive enough to cover all scenarios, especially the exception scenarios.

-         Developers tend to assume that their programs will work. As a result, scope of the testing gets limited to successful scenarios rather than failure scenarios.

-         Unit test is normally carried out with the mindset of a developer rather than the end user. This factor influences more for interactive programs like screens and menus.

-         Unit test results cannot be verified completely in situations where verification depends on other modules, which are either being developed or not fully tested.

-         Changes made to dependent modules and/or objects are not communicated to the development team, resulting in programs that are already unit tested, failing in subsequent tests. This could be configuration changes affecting interfaces and vice versa.

-         Failure to identify all objects for releasing.

-         Failure to identify areas requiring Regression test as a result of a fix done.

-         Designers fail to keep the design updated.

These are some of the reasons for failures encountered in System Test. Whereas failures encountered in UAT could be due to inadequate System Test and/or the following factors:

-         Business Analysts/Architects/Designers did not understand the requirement correctly.

-         Business Analyst did not pass on the requirements clearly to the design team.

-         Design documents are not clearly stating the requirement and the solution.

-         Sometimes, it may not have been possible to simulate the exact test environment for unit test and system test. This is more common in interface modules, where interaction with other external systems is involved.

2.2.          What it impacts?

Errors encountered in System Test impacts the project in variety of ways as given below. 

-         More time and effort required completing the System Test.

-         Rework done to fix the bug is often done in haste as the System Test is already behind schedule due to the errors encountered. This affects the quality.

-         Retest done by developers after the fix is not comprehensive, as the pressure is upon them to complete the fix. This results in unforeseen side effects in other areas.

-         It is possible that some of the developers are no more with the project, when the project moves into System Test. As a result, rework is sometimes done by a different developer. This requires more time to understand the program, than it would have taken to avoid this in the first place. This increases the time to fix the problem.

2.3.          How it can be prevented?

Obviously, it is by testing, testing and more testing. We all do testing, but it needs to be that little bit more complete and effective. To make that happen, a good design document and testing guidelines are required.

This section starts by explaining the differences between Unit and System tests. It then goes on to formulate a guideline that can be adopted to ensure that unit test is not something that is “also done” at the end of the development; but it is something that is conceived at the design stage and implemented during development.

2.3.1.     System Test Vs Unit Test

For those of you, who are not aware of the subtle differences between the two:

System Test, also known as the Black-box testing, looks at the system as a whole and it is done to ensure the functionality and the boundary conditions of the system meets the customer requirement. It does not test the internal working of the individual module.

Unit Test, also known as the White-box testing is done to ensure that:

-         all independent paths within a module have been executed at least once

-         all logical decisions are tested on their true and false sides

-         all loops at their boundaries and within their operational bounds are executed

-         all internal data structures are tested for their validity

-         all the user interfaces like screens, messages and other outputs are tested for validity, un-ambiguity and grammatical correctness.

2.4.          Where do I begin?

Any attempt at enforcing quality invariably involves an additional effort on documentation. This is a factor that we have to live with.

Testing ideas should be born in design phase. It is painful, but this will only help improve the design that you do as you start thinking in terms of the quality of the program in addition to the client requirements. A simple thing like visualising and listing all possible error messages that will be displayed will help in preparing your test conditions a great deal. 

2.4.1.     The design document

Following changes should be brought about to the design document:

• Business rules should be isolated from the actual solution.
• The solution section should explain the process flow and validations separately, instead of mixing these two together. This will help prepare the test scenarios better.
• For complex processes, it is always a better idea to include a process flow diagram. Remember: Pictures speak more volumes than words. (Of course, it does not mean that you should overshoot your design estimates by drawing fancy pictures!)
• Similarly, for complex business rules, a decision tree or decision table can be created.
• As a customer I will be more interested in what I see on the screen. Hence it should be mandatory to provide all screen shots and messages that the process will display. Since early in the design, screen shots may not available, a simple diagram can be created depicting the layout of the screen.
• All messages should be identified in the design document. This helps in two ways.
• Visualizing all potential error messages helps in preparing a good exception handling mechanism in the design, which the customer will appreciate.
• As mentioned earlier, as a customer I will be interested in knowing all messages I am likely to see when I start using the system. This helps in making the learning process easier.
• Messages should not be hard coded in the programs. Having soft coded messages will reduce maintenance effort in your program. If the message is found to be wrong during testing you do not need to change the programs, thereby reducing the fixing effort in the future.
• The messages identified in the design should be complemented by an explanation of the Cause and Action. An end user would greatly appreciate seeing this in the design document.
• Any technical details, which the customer need not be aware of, should not be put in the external design. For recording all technical details, a simple internal design can be created. This gives more flexibility to development team to change any technical details in the document at any time that does not affect the functionality. If these technical details were in the external design, it would have been difficult to do, because external designs are bound by the client.
• No formal template needs to be used for internal designs. If you think your program is too simple to have an internal design, at least have a one-page document to list all the objects you are developing so that nothing gets missed out during the release.
• Development should be based on the external and internal design documents and not other way around. That means, if you are maintaining a document, first change the document before you change the program. Remember: An outdated document is more dangerous than having no document.

2.4.2.      System testing guideline

The Testing Consideration section of the External Design can provide direction for functionality testing. It should be given as much importance as the solution itself. This section should contain guidelines on what needs to be tested from the functionality point of view. In addition to helping the testing activity, it will help in giving a better impression to the client, as it will be a clear reflection of the depth of testing and quality.

2.4.3.      Unit testing guideline

Since Unit Test cases will be too detailed, it cannot be included in the external design. Hence a simple checklist can be prepared separately that will list all Unit Test cases. There will be two components to it.

1. A standard set of test cases that can be used for all interfaces – This will not undergo change. But not all items need to be used every time. The designers can identify what is required from this list, based on the functionality of the program.

2. A second set of test cases that are specific to a particular interface – This should be prepared for each interface afresh during design time. This can be included in the Testing Consideration section of the external design. The standard test cases may be used as the basis to evolve these test cases.

3. Developers have a tendency to write Unit Test Plans after development completes. This is not a good practice. Write down your Unit Test Plans before you start development. This will help you visualise more testing scenarios, without being influenced by the way you coded your program. 

3.    Conclusion

This article is just a guideline. Neither these guidelines cover all testing scenarios, nor it guarantee that your programs will pass System Test in the first attempt without any tickets being raised. But it should definitely help reduce the occurrences of errors.

Some of the guidelines may look very obvious or trivial to an experienced developer. It is up to the individuals to decide how rigorous the unit testing should be, depending upon on the circumstances like available time, complexity of the programs, etc. A sample unit testing guideline is provided as an appendix. It is not an exhaustive list but something you could develop upon. For most of you to start the testing, these guidelines may not be required at all. But before you release, it may be a good idea to run through such a list to ensure that you did not miss anything.

Finally, my own version of an alternate quote to the one at the beginning:

“Had God designed the Universe the way Architects designed solutions, Big Bang would have failed with a Time Out error!”

Appendix - General Unit Testing Checklist

Parameters and Arguments

• Number and type of parameters should match the arguments
• If an argument is mandatory and if the corresponding parameter is not passed in, does the program handle the error properly?
• If the type of parameter passed in and the argument does not match, does the program have proper error handling or auto conversion (if such conversion is safe for the logic) built in?

Data Elements: Variables and Constants

• Are you initializing all your variables and constants appropriately?
• If you are outputting a variable, is the display readable in all circumstances. Test to see if your variable displays a junk value.
• Test cases when the variable does not receive any value from the source module, function or entity.
• Test cases when a wrong type of value is received by the variable

Nature of Transactions handled

• Test to see all different transaction types are processed
• If the transactions handled are date sensitive, test different chronological sequence of transactions to ensure if it can handle. E.g.: Pre-dated, Post-dated payments/adjustments etc.
• If the transactions handled can be cancelled, deleted, rejected, etc, then test after those operations are performed to see if there is any change in the expected behaviour of your program with respect to the above transactions.

Data manipulation

• If your program is performing any data manipulation like update, delete or insert, ensure consistency between all these operations and any output that is generated. The output here could be, file, display of statistics, reports, etc.
• If your program is performing multiple sets of data manipulations, ensure consistency between these sets. E.g.: Between inserting into a table and then updating another table, etc.

Exception handling

• Is there any exception for which there is no message being displayed? I.e. test for exceptions that go unnoticed.

Database Exceptions:

• If your program is performing data manipulations in conjunction with other I/O operations such as file creation or upload, the exception handling within the database should be consistent with other I/O operations.

For external input:

• If reject file is not created, database transactions should get rolled back and the entire file should be rejected.
• If reject file is created, erroneous transactions should be rolled back and valid transactions should be committed.

For external output:

• Database transactions should be rolled back, and any entities created (like file, report, etc) till then should be deleted.

Files

• Can the program handle a missing input/output directory?
• Can the program handle a missing input file?
• What if the output file already exists?
• What if the input/output directories/files have insufficient read/write permissions?
• In the above cases, does it display a meaningful message?
• Is the output file deleted when the program ends unsuccessfully?
• Is the output file generated with the expected file name?
• Is the output file generated in the expected directory?
• Is the file generated as per the layout?
• Are the individual data elements having the right format and alignment?
• Did you test to ensure all data elements appear in the file?
• Did you test to ensure all different types of transactions, expected in the file, appear?

Boundary Conditions

• The data manipulation carried out by the process was limited to the expected set of records, or did it affect other records? This is a difficult test to carry out on big and volatile database tables. But can be done otherwise.

GUI

• Is the drop-down list displaying only the required list of values or any required values missing?
• Check the spelling and correctness of the field labels
• Check the pop messages and the messages displayed on the task result screen for spelling, grammar and clarity.
• How is my screen behaving when I mouse-click randomly. E.g. click save before completing data entry or change the order in which you navigate around the screen to check if you are getting any inappropriate error messages.

Execution

• Should you allow concurrent execution of your process?
• If answer to above is “Yes” test with concurrent execution to ensure it does not trigger conflicts of any kind.
• If answer to above is “No” test to ensure your schedule type does not allow concurrent execution.

Post-execution

• Check if the transactions created by the process is accessible to a typical operator login. Test should include retrieving, updating and then saving the transaction through online screens, after having logged in using a typical operator login.
• If your process does data manipulation, use two test operator logins 1) having the right privileges associated to a typical operator who executes such processes and 2) having the right privileges associated to a typical operator who accesses the transactions that were created.
• run the process using the first operator login
• test if the data manipulated by the above process is accessible though the second operator login.
• Use a second test operator login, that has the right privileges associated to a typical operator who

Code Review

• Did I remove all debug messages?
• Did I remove all temporary statements?
• Did I remove any temporary hard coded values?
• Are my messages grammatically correct?
• Are my messages meaningful for the type of exception?
• Are my messages clear enough for the user to take any further action?
• Am I following the coding standards as the development handbook?
• Is there any potential for any performance problems with my SQL statements? Use a tool (like Explain Plan in Oracle) if required.
• Is my program easy to maintain in future?
• If your program has a complicated logic and/or a dormant problem, provide sufficient comments to make life of the maintenance programmer easier.

Release

• Did I miss any objects? Check with your internal design to ensure you have included all objects.