Continuous Integration in Large Software Projects

Those who have worked in a large software project have probably noticed how difficult it is to get all parts of the software to work together flawlessly when the release of a new version approaches. Changes to different parts in different branches should be integrated, and release procedures should be carried out, including tasks such as last minute system tests, composing the release notes and fixing any remaining defects. Usually this takes more time than was initially planned, and consequently the final stages of the project become a death march or the release is delayed.

The basic idea of continuous integration is to integrate changes into the software as early as possible. My experience is that the biggest advantage of continuous integration can be reached in large software projects. Unfortunately, those participating in large software projects often believe that continuous integration is not suitable to projects of this scale, or that it is too difficult and troublesome considering the advantages it offers.

Below I describe one example of this from my working life. The idea is to show the significance of continuous integration in a large project and to give some practical tips on how to make the best use of it.

Case: Large international software project

This example case involves a software product of a major international company, which is installed to the customer's own environment, and the customer takes care of its maintenance. The different versions of the product are in active and critical use around the world at all times. The product has a long history, but it is a new generation of the previous version. In practice this means that big parts of it have been re-written but there’s also a lot of old code. There are millions of lines of both old and new code written with several different programming languages and using a variety of technologies. There are hundreds of software developers and development teams are located in many different countries.

The initial situation was that the product had a daily build, meaning that a new version of the product was built and tested once a day. If everything went as planned, this took eight hours. The developers often worked for a good while in their own branches before releasing their changes. Because overall there were many changes and they were often based on an old version, there were plenty of integration problems.

Because the problems almost always went unnoticed at least until the next working day, fixing them was slow. Consequently the daily build reported the product to be more or less broken all the time. The developers were not that interested in these reports. They wanted to focus on the change they were working on, not on its effects elsewhere. Product management was also challenging: it was difficult to see how far the product was from being ready for release and how long it would take to fix the problems.

The right tools to support continuous integration

After these problems, the project decided to invest in continuous integration. First the tools were swapped for ones that supported continuous integration better. Source code was moved from ClearCase to Subversion and Ant was replaced as a build tool with Maven.

The old version control system supported a way of working where developers work without paying attention to the changes made by others. The new toolchain forced them to notice changes made by others. At the same time a decision was made that the number of different branches is minimized and all developers aim to develop a common branch – the latest version of the product.

The earlier build tool (Ant) supported a way of working in which all logic related to building the product had to be spelled out. In addition, both internal and external dependencies of the product had to be managed manually. A large software accumulates a lot of this kind of build logic over time. Managing the internal dependencies of the software manually is also challenging when a lot of changes are made. The new build tool (Maven) took care of managing the dependencies and included a lot of built-in build logic. The user only had to configure the tool. Changing the tool considerably decreased the volume of build instructions and made them easier to understand for everyone familiar with the new tool.

Changing the tools was a fairly massive operation. Many problem situations were encountered, but they were all solved. The operation revealed plenty of problems in the old model and forced people to solve them. Learning the new tools also took time and required new kind of thinking. Afterward almost all developers have agreed that the new tools were in fact an improvement over the old ones.

In connection with changing the tools, the daily build was replaced by an automated integration process based on the new tools. Changes in version control triggered the build, installation and testing of the new version. Both automated unit tests executed during the build and automated acceptance tests executed after installation were used. The situation improved, but continuous integration still reported the product to be broken all the time and feedback loop of build results was still too long.

Better quality, work motivation and results

The problems were solved in priority order. Simplifications, optimizations, parallelization and better hardware were applied to speed up test results. The more reliable and quicker continuous integration became, the bigger were its benefits. The build stayed green for longer and longer times, and the developers became interested in its results. Work motivation grew and issues with the product were fixed faster than before. Communication between development teams increased. People responsible for product and project management were satisfied when they received almost real time information on the product’s health. The quality of the product improved along with the results of continuous integration.

When the original eight hours of the daily build were cut down to one and a half hours and new results were received completely automatically every 35 minutes, the continuous integration of the product was talked about as an internal success story. The company's other large software projects also wanted to adopt continuous integration.

Significant advantages even with simple implementation

In traditional continuous integration a change in the source code triggers a build and a build often includes unit tests. A monitor in the development team's workspace shows whether or not all tests passed. Such a simple model does not scale, though, if the new version takes a long time to make and there are a lot of simultaneous changes.

In the example described above, continuous integration was a wide system in which changing different parts triggered the re-building of certain other parts. A process combining all parts was running in the background, and the new version it produced was automatically installed on test servers where it was further tested. In addition, some versions were also tested by installing them in different environments and executing wider tests on them. The whole system produced a lot of different reports and views and gathered interesting data during the process.

So continuous integration can be done in large scale, too. This does not mean that the continuous integration of a large software product should always be sophisticated. Even a very simple implementation can provide major advantages if it is reliable and fast enough. The continuous integration mentioned above started from a simple implementation that has been gradually improved based on the feedback received from using it.

Better quality faster

Continuous integration improves the developers’ communication and team spirit, gives visibility to the problems in the product, speeds up fixing faults and improves the quality of the product. At the same time, as many as possible of the work stages of releasing and testing a new version have to be automated.

My own experience is that fast and reliable continuous integration is extremely useful in large software projects. Adopting continuous integration may be laborious, but it is worth the trouble. Even minor improvements can have a big significance. If the amount of automation increases and the developers take pride in the product passing all of its tests, the effect on the development of the whole product is enormous. Investing in continuous integration easily pays for itself in easier development and product management and a shorter lead time for releasing the software.

Samuli Siivonen, Software Architect

Samuli Siivonen is an experienced software architect who likes challenging implementation tasks. In recent years he has worked closely with large-scale software projects and their development environments.