Continuous Integration for Verification
Introduction
In the previous post we talked about the importance of sanity testing. In this post, we’ll talk about how we can leverage sanity testing to implement a continuous integration flow that further improves team productivity. First, a couple of definitions:
• Continuous Integration (CI) is a development practice that requires developers to submit code into a shared repository frequently. Each check-in is then verified by an automated build (sanity test), allowing teams to detect problems early.
• Release - For the purpose of this post, we will be talking about an internal release – the time from when a team member submits to the repository, to the time that it is observable by all members of the team. We call each release a ‘baseline’.
Standard Verification Development (no CI)
In a standard verification flow, when team members sync their workspaces, they sync to the latest version of every file. In this situation, whenever a bug is introduced, it is observable by every member of the team. The bug should be detected quickly by the sanity test, but some damage has already been done – team members who sync out the bug will see failures in their tests and will need to make some manual effort to figure out what happened, when to sync back to or how to resolve the issue. Even worse, catastrophic bugs can kill full regressions if introduced around the time the regression is being launched. An example of introducing a bug in this flow in the example broken heart, indicating a release with a bug that all team members may see and be affected by.
The question is – can we avoid all the churn associated with a bug being introduced into the repository?
Recommended by LinkedIn
Continuous Integration Flow
As hinted above, we can avoid the churn by moving to a continuous integration flow. In this flow, changes submitted to the repository are not released to the team until they have passed a sanity test. Team members no longer sync to the head of the repository, but instead sync to the most recent released version of the code. This does come at a cost that there is a latency introduced from when a change is submitted until it is observable by the team, however that is more than made up for by the fact that team members can always be sure that when they sync, at a minimum they will sync to a state where they would pass a sanity test. An example of introducing a bug in the continuous integration flow is shown below. The main difference is the unbroken heart, meaning that when a bug is introduced into the repository, it is not observed by the rest of the team. The engineer who submitted the bug still needs to monitor the sanity results and fix the issue, but other team members can proceed with their work without interruption.
Protect the Trunk with Multiple Development Branches
The process shown above is good, but for some teams (especially larger ones) there is still a problem with this flow. The problem is that everyone is working from the same code base (the trunk branch). When a bug gets introduced, it makes it immediately into the trunk. Although that code isn’t released to the team until the bug can be fixed, the trunk is still broken for everyone, and no releases can be made until the bug can be fixed. This can block unrelated work from proceeding while waiting for issues to get resolved.
The solution to this problem is to have everyone work on their own development branch. Now, when someone submits something, it doesn’t go into the trunk immediately – instead it goes to the development branch and Jenkins will try to merge the change to the trunk and run the sanity test. If the test passes, the merge is submitted, and a new baseline gets created. If it fails, the submitter is notified of the failure and the code is not merged on to the trunk. This leaves the trunk unbroken and allows unrelated work to proceed uninterrupted.
Note that this is not something that we have implemented ourselves but is something that you may want to consider if you are facing issues where team members are stepping on each other’s toes with incorrect code submissions.
Conclusion
In this post we’ve talked about using Continuous Integration to reduce the churn and wasted effort teams face when bugs are introduced to the code base. In the next post, we’ll talk about how we implemented our releases (we call each release a ‘baseline’), and some other benefits of syncing to baselines.
Hi. I love these topics. Can you share about how do you, or your team control regression for both Sanity and Default regression status week by week? Do you have a specific tool to analyze regression result? Currently I only use EMAN and Excel with a simple script to dump data automatically. And, the important is Do your team configure Regression to reach coverage with just some regression round? Is there any useful tricks or tips applied for your project? I'm searching on Sharepoint and Confluence too, but If you can share it here, I think it's a good way also. :D
One of the challenges with CI the way you describe it is the compute resources and latency due to running the sanity checks often. Making code more unit testable, and thus minimizing the need for expensive whole-system tests is a good way to address the issue. It also encourages cleaner boundaries and tighter specs, where the unit tests essentially become the spec.