{"id":997,"date":"2015-05-27T17:02:57","date_gmt":"2015-05-27T15:02:57","guid":{"rendered":"http:\/\/leanmagazine.net\/?p=997"},"modified":"2024-03-15T15:18:49","modified_gmt":"2024-03-15T14:18:49","slug":"continuous-delivery-quality","status":"publish","type":"post","link":"http:\/\/leanmagazine.net\/lean\/continuous-delivery-quality\/","title":{"rendered":"Continuous Delivery – Higher speed and higher quality"},"content":{"rendered":"

\"race<\/a><\/p>\n

How Continuous Delivery helped a large organization deliver faster with increased quality<\/h2>\n

A major company in the IT business created an advanced scrum of scrums organisation, but ran into problems with delivery times and quality decay. The implementation of Continuous Delivery solved the problems rapidly.<\/em><\/h4>\n

The company \u2013 a major international player in the IT business with thousands of employees \u2013 had just performed an agile transformation project. Top brass was strongly convinced that new ways of working would improve efficiency. And indeed it was an impressive setup where seventy scrum teams worked in parallel, delivering software for multimedia, Linux and Android. Accordingly, the amount of code produced was enormous.<\/p>\n

The quality assurance process created problems<\/strong>
\nThere was, however, a conviction that extra quality assurance had to be put in place as a part of the agile transformation. The aim was to secure the quality of each delivery before being integrated to the main branch. As a consequence, a huge quality assurance process was put in place. Every code change had to go through multiple steps of test and merges before finally making it to the main branch. It may appear to be a sound ambition to safeguard the code quality in this way, but in this complex organisation problems soon arose.<\/p>\n

\"Mikael<\/a>
Mikael Reinholdsson is a senior consultant at Softhouse, specializing in program management using Lean&Agile methods and Continuous Delivery.<\/figcaption><\/figure>\n

Mikael Reinholdsson, seasoned project leader at Softhouse, participated in the process as a consultant. He witnessed how the delivery times quickly grew:
\n\u201cAt best, it took two weeks from the moment the code was ready until it was integrated into the main branch,\u201d he says. \u201cAnother major issue was that the code quality of the main branch was very poor\u2013in spite of our high ambitions to keep it flawless.\u201d<\/p>\n

The integration teams came to include lots of people who had to struggle to get a tag once per week which at least worked to some extent. Quality and integration teams kept complaining that code from development teams was of very poor quality and that they seemed to ignore integration quality issues.<\/p>\n

\u201cThey demanded that we should introduce more tests before each delivery. The result was that we became stuck in a bad quality situation,\u201d Reinholdsson says. \u201cEvery correction and delivery took an age to make it into the main branch.\u201d<\/p>\n

Continuous Delivery to the rescue<\/strong>
\nAs the quality problems amassed and started to block progress in the entire organisation, something had to be done. The suggestion of trying out Continuous Delivery
\nwas more or less a grassroots initiative. The managers of the organization kept believing in their setup where the process was strengthened by adding one more week of quality assurance before delivery to the main branch. Reinholdsson and his colleagues, on the other hand, were convinced that the \u201dquality assurance as you go\u201d concept of Continuous Delivery would sort out the problems.
\n\u201cIn principle, we were forbidden by management to change,\u201d Reinholdsson recalls. \u201dThere was, however, one middle manager who supported us. So we decided to go for it during the Christmas holidays.\u201d<\/p>\n

Initially, there was some insecurity about whether this would actually work. Challenging the guidelines of the management of course led to some nervousness. As a matter of fact, the confidence in integration teams was quite low; the general expectation was that a transformation into Continuous Delivery routines would take a long time. Therefore the outcome was quite surprising:
\n\u201cWe just took the latest code in integration from every team and set out to do a first tag,\u201d says Reinholdsson. \u201cThis took us two to three days\u2014and then we started to get in sync! After just two weeks we were able to set one tag per day. For every day, we could see that quality was rapidly improving.\u201d<\/p>\n

The benefits of the change came in many areas. Apparently, a large part of the quality problems was created in the delivery queue. This was because each delivery was tested in conjunction with outdated code from other teams. A fact that seems almost contradictory is that less testing before delivery led to a great improvement in quality. This was because the visibility was vastly improved and the lead times much shorter.
\n\u201cQuality improved very rapidly after these changes. The excellent visibility introduced gives us a very transparent view on where we are,\u201d Reinholdsson says. \u201cSince any correction is only one day away we can quickly improve quality and remove critical issues fast. We also have a daily visibility on test environment quality and deliver corrections each day to give more stable and repeatable results. Today, we can deliver corrections of 75% of customer issues the day after they are reported.\u201d<\/p>\n

Reinholdsson is careful to underline that the speed of the transformation process depended on some fortunate preconditions:
\n\u201cThe fast improvements we saw were due to the fact that we already had excellent tools for integration and build, as well as well working test automation that fitted right into the continuous delivery way of working.\u201d<\/p>\n

In hindsight, this facility to give swift response to customer feedback has actually had strategic implications:
\n\u201cToday, the customer expectations are that we should be able to correct a majority of all issues within one day. The kind of lead times we had before Continuous Delivery was implemented would simply kill any chances of gaining large volume contracts,\u201d Reinholdsson says.
\n

\n

Continuous Delivery in large organisations\u2014three pieces of advice from Mikael Reinholdsson<\/strong><\/h4>\n

1. Keep integrating.<\/strong>
\n\u201cNever let code grow old before merging results with all other developers. Developers do more things right than you might think, given an efficient working environment.\u201d<\/p>\n

2. Set up an effective infrastructure.<\/strong>
\n\u201cInvestment in a high level of automation and efficient tools are prerequisites to get the visibility of quality of every patch before delivery. Test results at patch level need to be conveyed very quickly, preferably within one hour.\u201d<\/p>\n

3. Create visibility.<\/strong>
\n\u201cFor the organisation to know where it\u2019s at and where it should go, automation has to give very good visibility on a daily basis on the main branch quality. In our case, we had the benefit of already having good tools and high level of automation in tests. This made the change fairly smooth and we saw the benefits very fast.\u201d<\/div><\/p>\n","protected":false},"excerpt":{"rendered":"

How Continuous Delivery helped a large organization deliver faster with increased quality A major company in the IT business created an advanced scrum of scrums organisation, but ran into problems with delivery times and quality decay. The implementation of Continuous Delivery solved the problems rapidly. The company \u2013 a major […]<\/p>\n","protected":false},"author":32,"featured_media":998,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[77,72,5,83],"tags":[74,73,6,29,53,58],"_links":{"self":[{"href":"http:\/\/leanmagazine.net\/wp-json\/wp\/v2\/posts\/997"}],"collection":[{"href":"http:\/\/leanmagazine.net\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/leanmagazine.net\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/leanmagazine.net\/wp-json\/wp\/v2\/users\/32"}],"replies":[{"embeddable":true,"href":"http:\/\/leanmagazine.net\/wp-json\/wp\/v2\/comments?post=997"}],"version-history":[{"count":9,"href":"http:\/\/leanmagazine.net\/wp-json\/wp\/v2\/posts\/997\/revisions"}],"predecessor-version":[{"id":1159,"href":"http:\/\/leanmagazine.net\/wp-json\/wp\/v2\/posts\/997\/revisions\/1159"}],"wp:featuredmedia":[{"embeddable":true,"href":"http:\/\/leanmagazine.net\/wp-json\/wp\/v2\/media\/998"}],"wp:attachment":[{"href":"http:\/\/leanmagazine.net\/wp-json\/wp\/v2\/media?parent=997"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/leanmagazine.net\/wp-json\/wp\/v2\/categories?post=997"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/leanmagazine.net\/wp-json\/wp\/v2\/tags?post=997"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}