Problems of Continuous integration or "We only really use it when we are bored"

How Continuous Integration Should Work

The idea is simple and very straight forward: Each time you make a commit to the master branch (or /trunk in Subversion), a continuous integration server (or service) attempts to build the entire product. "Build" means compile, unit test, integration test, quality analysis, code coverage and much more

The result is "success" or "failure." If it is a success, we can say that "the build passed." If it is a failure, we say that "the build is broken." The build usually gets broken because someone breaks it by committing new code that turns previously passing tests into failing ones.

This is the technical side of the problem which always works. It may have its problems, like hard-coded dependencies, lack of isolation between environments or parallel build collisions but we can leave implementation problems aside for now. If the application is well written and its unit tests are stable, continuous integration is easy. Technically.

On the organizational side, continuous integration is not only a server that builds, but a management/organizational process that should "work." Being a process that works means: "If the build fails, the development team should stop whatever they are doing and fix the problem immediately." THIS is what doesn't work and really can't work.

Who Needs This?

Continuous integration is about setting the entire development team on pause and fixing the broken build. To rephrase it: Once the build is broken, everybody should focus on fixing it and making a commit that returns the build to the stable state.

Now, my question is—who, in an actively working team, may need this?

A product owner, who is interested in launching new features to the market as soon as possible? Or a project manager who is responsible for the deadlines? Or maybe programmers, who hate to fix bugs made by someone else, especially under pressure?

Who likes this continuous integration and who needs it?

Nobody.

What Happens In Reality?

I've seen it multiple times, the scenario is always the same: We just start to ignore that continuous integration build status. Either the build is clean or it is broken, and we continue to do what we were doing before. We don't stop and fix it. Instead, we ignore the information that's coming from the continuous integration server.

Eventually, maybe tomorrow or on Monday, we'll try to find some spare time and will try to fix the build. Only because we don't like that red button on the dashboard and want to turn it into a green one.

What About Discipline?

There is another side of this coin: we can try to enforce discipline in the team. We can make it a strict rule, that our build is always clean and whoever breaks it gets some sort of a punishment. Try doing this and you will get a fear driven development. Programmers will be afraid of committing anything to the repository because they will know that if they cause a build failure they will have to apologize, at least.

A strict discipline (which I'm a big fan of) in this case only makes the situation worse. The entire development process slows down and programmers keep their code to themselves for as long as possible, to avoid possibly broken builds. When it's time to commit, their changes are so massive that merging becomes very difficult and sometimes impossible. Not to mention peer review or code reviews, static analyses etc.

As a result you get a lot of throw-away code, written by someone but never committed to master, because of that fear factor.

OK, What Is The Solution?

It's called "Read only master branch".

It is simple—prohibit anyone from merging anything into master and create a script that anyone can call. The script will merge, test, and commit. The script will not make any exceptions. If any branch breaks at even one unit test, the entire branch will be rejected.

In other words: raise the red flag BEFORE the code gets into master.

This solves all problems.

First, the build is always clean. We simply can't break it because nobody can commit unless his code keeps the build clean.

Second, there is no fear of breaking anything. Simply because you technically can't do it. All you can do is get a negative response from a merging script. Then you fix your errors and tell the script to try again. Nobody sees these attempts, and you don't need to apologize. Fear factor is gone.