{"id":1237,"date":"2016-12-03T16:43:26","date_gmt":"2016-12-03T15:43:26","guid":{"rendered":"http:\/\/leanmagazine.net\/?p=1237"},"modified":"2024-03-15T15:18:49","modified_gmt":"2024-03-15T14:18:49","slug":"devops-at-large-moving-a-mountain","status":"publish","type":"post","link":"http:\/\/leanmagazine.net\/scaling\/devops-at-large-moving-a-mountain\/","title":{"rendered":"DevOps at large: Moving a mountain"},"content":{"rendered":"

\"gruver_mountain\"<\/h3>\n

Transformation expert Gary Gruver has received much praise for his book Leading the Transformation: Applying Agile and DevOps Principles at Scale<\/em>. Here he shares some of his experiences and reflects on how to \u2013 among other things \u2013 tackle organizational challenges, coach managers and succeed with a DevOps transformation.<\/h3>\n

\u201cJez Humble would put it like this: \u2018We independently discovered the deployment pipe line and started doing DevOps before people even considered the term!\u201d says Gary Gruver.<\/p>\n

The Jez Humble quote is part of astory describing a formidable challenge in 2008. It was then, in the middle of the economic downturn, that Gary Gruver and his colleagues at HP went on a journey to improve the productivity of the firmware development team of the LaserJet Printers by a staggering 900 percent!<\/p>\n

\u201cThis led to a lot of innovations, but one of the biggest was a very large-scale deployment pipeline with 20,000 hours of automated testing running every day,\u201d Gruver remembers. \u201cSome would say it wasn\u2019t really DevOps because there was no Ops team or data center deployments. That may be true but a lot of the practices of keeping the code base stable and releasable are the same.\u201d<\/p>\n

Executive workshops<\/strong><\/h4>\n

Today, Gary Gruver is a consultant working with large organizations around the world, helping them transform their software development processes. Through executive workshops, he helps ensure that management has a common understanding of DevOps principles and practices. He puts a lot of effort into making sure that they understand their role in leading the transformation and guides them so they can avoid common mistakes in their journey. In addition, he runs execution workshops for teams responsible for complex systems.<\/p>\n

\u201cIn these workshops we analyze their deployment pipeline, highlighting the biggest inefficiencies,\u201d says Gruver. \u201cWe then prioritize fixing things that will create positive momentum for the transformation. Finally we get the teams started on their ongoing continuous improvement process.\u201d<\/p>\n

Whenever people ask Gary Gruver about DevOps he usually starts by asking them what DevOps means to them.<\/p>\n

\u201cI start here because there are so many different ideas in the industry and most of them start with the practices they have heard referencing DevOps. I think this leads to a lot of confusion. I tend to like to start with the Principles and Gene Kim\u2019s definition that is focused on the outcomes\u201d.<\/p>\n

\u201cDevOps is the technical, process, and cultural changes that enable the organization to release code more frequently while maintaining all aspects (functional, security, performance) of quality.\u201d<\/span><\/h3>\n

Gene Kim, author of the bestseller \u201cThe Phoenix Project\u201d<\/em><\/span>.<\/h3>\n

This may sound like a pretty straight-forward definition but, according to Gruver, large corporations would be doing this already if they didn\u2019t have all the waste and inefficiencies that exist in their organizations which prevent them from releasing more frequently.<\/p>\n

\u201cSo to me DevOps is all about removing the waste and inefficiencies that exist in their organization so they can release more frequently with quality,\u201d he says. \u201cWhen you are not building, testing, or releasing on a very frequent basis the organization can brute force their way through all the inefficiencies. When you increase the frequency, brute force no longer works. You have to automate things and fix the ongoing problems that have existed and frustrated your organization for years. This has a dramatic impact on the productivity of the organization.\u201d<\/p>\n

Starts with the business objectives<\/strong><\/h4>\n

Gruver says he avoids starting any discussion with executives about DevOps. Instead, his starting point is what matters to them: their business objectives.<\/p>\n

If their current development and delivery processes are meeting the needs of their business then it probably doesn\u2019t make sense to invest in changing the processes.<\/p>\n

\u201cThe reality though for most executives is that their current processes are not meeting the needs of the business,\u201d he says. \u201cI start with how the needs are not being met to clarify the business objectives of the transformation.\u00a0Then and only then do we start looking in the toolbox to understand what tools would provide the most benefits. This usually comes around to DevOps because these objectives at some level involve increasing the efficiencies and effectiveness of their process. From my perspective, documenting their deployment pipelines and working to increase the frequency requires them to eliminate waste and inefficiencies, which improves productivity.\u201d<\/p>\n

The biggest misunderstanding that Gary Gruver sees in the industry is that people think DevOps practices should be the same regardless of the size or complexity of the organization. As he sees it, DevOps is rather about coordinating the work across different parts of the organization. If the organization is small, the practices are all about coordinating the work across small teams. This is also valid for a large organization that has an architecture which enables small teams independently to develop, qualify, and release code.<\/p>\n

\u201cThe challenge is that most large traditional organizations have architectures that require very large teams to work together to coordinate the development, qualification, and release of their code,\u201d says Gruver. \u201cAssuming that you would use the same practices to coordinate the work across hundreds or thousands of people as you would for small teams is the biggest misunderstanding I see in the industry.\u201d<\/p>\n

According to Gruver, one of the most important things you need to be doing to transform your software development and delivery processes is test automation. \u201cIt is also one of the things I most\u00a0frequently see being done wrong,\u201d says Gruver. \u201cOrganizations create a bunch of test automation using frameworks that are not easy to maintain or triage. This works fine for the first few tests but over time you are going to have thousands of automated tests and you are going to die under the weight of these tests if they are not designed correctly.\u201d<\/p>\n

\n

Gary Gruver on
\nIMPLEMENTATION OF DEVOPS IN LARGE CORPORATIONS<\/strong><\/h3>\n

The greatest challenges<\/strong><\/h4>\n

The cultural challenges of getting people to embrace different ways of working. You can add all the automation in the world but if you can\u2019t get developers to prioritize responding to the feedback from the deployment pipeline you won\u2019t get the improvements you expect. You can invest a lot in creating infrastructure as code but if people keep logging into servers and making manual changes you will still have issues. Et cetera …<\/p>\n

The greatest difficulties<\/strong><\/h4>\n

In large organizations it is getting everyone aligned on a common definition of DevOps, agreeing where to start improving, and how to scale it over time.<\/p>\n

The greatest mistakes<\/strong><\/h4>\n

Assuming the practices that worked well for coordinating work in small teams are the same practices that will work for coordinating development, qualification, and deployment across hundreds or thousands of people. The principles are the same but the practices can and should be different.<\/p>\n<\/div>\n

Tests on a frequent basis<\/strong><\/h4>\n

Gruver remembers working with one organization that had written over 1,000 automated tests that they were running at the end of each release cycle.<\/p>\n

\u201cI told them that is great but the reason you automate testing is not to reduce the cost,\u201d he says. \u201cIt is so you can afford to run them on a more frequent basis for feedback to developers. What we should do is have these tests running in a CI environment and the developers can\u2019t go home until they are all passing, but before we do that we need to make sure all the tests are rock solid and finding code issues instead of test issues.\u201d<\/p>\n

The organization started an experiment where they ran all the tests \u2013 over and over again \u2013 in random order, against the same build, and against different builds to see which ones were stable and ready for CI. At the end of six weeks they found out they did not have any stable tests that could be added to CI. They had to rewrite the tests in a new framework that was stable, maintainable, and triagable.<\/p>\n

\u201cDon\u2019t make that same mistake!\u201d Gruver warns. \u201cIf you have any automated tests, the first step is to see if they are stable enough to add to the CI process and gate code. If not, fix them or rewrite them so they are stable. Using them in CI will force you to fix your test automation framework and design patterns. If you are not using your automated tests to gate code then you are not doing automated testing.\u201d<\/p>\n

Small & large organizations<\/strong><\/h4>\n

Regarding the evolution of DevOps and the global software community \u2013 where are we at, and where are we going \u2013 Gruver tends to think about two different types of groups. The first is small organizations, alternatively large organizations with architectures that allow small teams independently to develop, qualify, and deploy code. \u201cThese are the organizations that have been really defining what is possible and pushing the technology in new and interesting ways,\u201d he says. \u201cThese teams will always be quicker and more efficient just because it is so much easier to coordinate smaller teams. They will continue to improve and create technology to solve problems in creative ways we can\u2019t even imagine yet.\u201d<\/p>\n

The second group is large organizations with tightly coupled architectures that require coordination across big groups of people.<\/p>\n

\u201cI think DevOps is going to expand rapidly for these organizations because of the impact it will have on their effectiveness and efficiency,\u201d says Gruver. \u201c They will never deploy and react as fast as the small teams but the opportunity for improvements is much larger just because the inefficiencies of coordinating work across large groups are so much more pronounced. This is where the majority of the software in the world exists and where DevOps can provide the biggest benefits so it is where we will see the most action. It will never be as sexy and interesting as what the small teams are doing but it will have the biggest impact on the productivity of software development around the world.\u201d<\/p>\n

\n

The three main benefits of DevOps according to Gary Gruver<\/strong><\/h3>\n

1. Making developers better developers by increasing the quality and frequency of feedback they receive.<\/strong><\/h4>\n

They all want to do a good job and think they have done a good job until they get feedback letting them know something needs to change. If this feedback comes weeks after they wrote the code they will have forgotten what they were doing and you are beating them up for something they don\u2019t even remember doing. If instead you can give them feedback while they are working on the code, it can help eliminate the waste of working on code that isn\u2019t secure, that doesn\u2019t work with the code others are writing, or that won\u2019t work in production.<\/p>\n

2. DevOps can help to dramatically improve the efficiency of the triage process.<\/strong><\/h4>\n

Lots of large organizations spend more time and energy creating and triaging large complex test environments at the end of long development cycles than they do writing the code. DevOps helps to improve the efficiency of this triage process because the increase in test cycles means there are fewer changes between when a test was passing and when it started failing. Additionally a well-defined deployment pipeline uses quality gates at the subsystems to localize the triage to the teams that created the problems and keeps these defects from causing instability in the complex enterprise test environments.<\/p>\n

3. Repetitive tasks like build, test, deploy, create an environment, configure a database, etc. get automated.<\/strong><\/h4>\n

The automation of these repetitive tasks has three main advantages. First, it is eliminating the work associated with manually doing something that can be automated. Second, because it is automated it can be run more frequently enabling fast feedback to developers and smaller batch sizes for more efficient triage. And lastly the automated process is more repeatable so there are less manual errors to triage and fix.
\n<\/div>\n","protected":false},"excerpt":{"rendered":"

Transformation expert Gary Gruver has received much praise for his book Leading the Transformation: Applying Agile and DevOps Principles at Scale. Here he shares some of his experiences and reflects on how to \u2013 among other things \u2013 tackle organizational challenges, coach managers and succeed with a DevOps transformation. \u201cJez […]<\/p>\n","protected":false},"author":32,"featured_media":1250,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[79,83,76],"tags":[],"_links":{"self":[{"href":"http:\/\/leanmagazine.net\/wp-json\/wp\/v2\/posts\/1237"}],"collection":[{"href":"http:\/\/leanmagazine.net\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/leanmagazine.net\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/leanmagazine.net\/wp-json\/wp\/v2\/users\/32"}],"replies":[{"embeddable":true,"href":"http:\/\/leanmagazine.net\/wp-json\/wp\/v2\/comments?post=1237"}],"version-history":[{"count":14,"href":"http:\/\/leanmagazine.net\/wp-json\/wp\/v2\/posts\/1237\/revisions"}],"predecessor-version":[{"id":1252,"href":"http:\/\/leanmagazine.net\/wp-json\/wp\/v2\/posts\/1237\/revisions\/1252"}],"wp:featuredmedia":[{"embeddable":true,"href":"http:\/\/leanmagazine.net\/wp-json\/wp\/v2\/media\/1250"}],"wp:attachment":[{"href":"http:\/\/leanmagazine.net\/wp-json\/wp\/v2\/media?parent=1237"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/leanmagazine.net\/wp-json\/wp\/v2\/categories?post=1237"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/leanmagazine.net\/wp-json\/wp\/v2\/tags?post=1237"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}