How Continuous Delivery helped a large organization deliver faster with increased quality

A major company in the IT business created an advanced scrum of scrums organisation, but ran into problems with delivery times and quality decay. The implementation of Continuous Delivery solved the problems rapidly.

The company – a major international player in the IT business with thousands of employees – had just performed an agile transformation project. Top brass was strongly convinced that new ways of working would improve efficiency. And indeed it was an impressive setup where seventy scrum teams worked in parallel, delivering software for multimedia, Linux and Android. Accordingly, the amount of code produced was enormous.

The quality assurance process created problems
There was, however, a conviction that extra quality assurance had to be put in place as a part of the agile transformation. The aim was to secure the quality of each delivery before being integrated to the main branch. As a consequence, a huge quality assurance process was put in place. Every code change had to go through multiple steps of test and merges before finally making it to the main branch. It may appear to be a sound ambition to safeguard the code quality in this way, but in this complex organisation problems soon arose.

Mikael Reinholdsson, seasoned project leader at Softhouse, participated in the process as a consultant. He witnessed how the delivery times quickly grew:
“At best, it took two weeks from the moment the code was ready until it was integrated into the main branch,” he says. “Another major issue was that the code quality of the main branch was very poor–in spite of our high ambitions to keep it flawless.”

The integration teams came to include lots of people who had to struggle to get a tag once per week which at least worked to some extent. Quality and integration teams kept complaining that code from development teams was of very poor quality and that they seemed to ignore integration quality issues.

“They demanded that we should introduce more tests before each delivery. The result was that we became stuck in a bad quality situation,” Reinholdsson says. “Every correction and delivery took an age to make it into the main branch.”

Continuous Delivery to the rescue
As the quality problems amassed and started to block progress in the entire organisation, something had to be done. The suggestion of trying out Continuous Delivery
was more or less a grassroots initiative. The managers of the organization kept believing in their setup where the process was strengthened by adding one more week of quality assurance before delivery to the main branch. Reinholdsson and his colleagues, on the other hand, were convinced that the ”quality assurance as you go” concept of Continuous Delivery would sort out the problems.
“In principle, we were forbidden by management to change,” Reinholdsson recalls. ”There was, however, one middle manager who supported us. So we decided to go for it during the Christmas holidays.”

Initially, there was some insecurity about whether this would actually work. Challenging the guidelines of the management of course led to some nervousness. As a matter of fact, the confidence in integration teams was quite low; the general expectation was that a transformation into Continuous Delivery routines would take a long time. Therefore the outcome was quite surprising:
“We just took the latest code in integration from every team and set out to do a first tag,” says Reinholdsson. “This took us two to three days—and then we started to get in sync! After just two weeks we were able to set one tag per day. For every day, we could see that quality was rapidly improving.”

The benefits of the change came in many areas. Apparently, a large part of the quality problems was created in the delivery queue. This was because each delivery was tested in conjunction with outdated code from other teams. A fact that seems almost contradictory is that less testing before delivery led to a great improvement in quality. This was because the visibility was vastly improved and the lead times much shorter.
“Quality improved very rapidly after these changes. The excellent visibility introduced gives us a very transparent view on where we are,” Reinholdsson says. “Since any correction is only one day away we can quickly improve quality and remove critical issues fast. We also have a daily visibility on test environment quality and deliver corrections each day to give more stable and repeatable results. Today, we can deliver corrections of 75% of customer issues the day after they are reported.”

Reinholdsson is careful to underline that the speed of the transformation process depended on some fortunate preconditions:
“The fast improvements we saw were due to the fact that we already had excellent tools for integration and build, as well as well working test automation that fitted right into the continuous delivery way of working.”

In hindsight, this facility to give swift response to customer feedback has actually had strategic implications:
“Today, the customer expectations are that we should be able to correct a majority of all issues within one day. The kind of lead times we had before Continuous Delivery was implemented would simply kill any chances of gaining large volume contracts,” Reinholdsson says.