Transformation expert Gary Gruver has received much praise for his book Leading the Transformation: Applying Agile and DevOps Principles at Scale. Here he shares some of his experiences and reflects on how to – among other things – tackle organizational challenges, coach managers and succeed with a DevOps transformation.
“Jez Humble would put it like this: ‘We independently discovered the deployment pipe line and started doing DevOps before people even considered the term!” says Gary Gruver.
The Jez Humble quote is part of astory describing a formidable challenge in 2008. It was then, in the middle of the economic downturn, that Gary Gruver and his colleagues at HP went on a journey to improve the productivity of the firmware development team of the LaserJet Printers by a staggering 900 percent!
“This led to a lot of innovations, but one of the biggest was a very large-scale deployment pipeline with 20,000 hours of automated testing running every day,” Gruver remembers. “Some would say it wasn’t really DevOps because there was no Ops team or data center deployments. That may be true but a lot of the practices of keeping the code base stable and releasable are the same.”
Today, Gary Gruver is a consultant working with large organizations around the world, helping them transform their software development processes. Through executive workshops, he helps ensure that management has a common understanding of DevOps principles and practices. He puts a lot of effort into making sure that they understand their role in leading the transformation and guides them so they can avoid common mistakes in their journey. In addition, he runs execution workshops for teams responsible for complex systems.
“In these workshops we analyze their deployment pipeline, highlighting the biggest inefficiencies,” says Gruver. “We then prioritize fixing things that will create positive momentum for the transformation. Finally we get the teams started on their ongoing continuous improvement process.”
Whenever people ask Gary Gruver about DevOps he usually starts by asking them what DevOps means to them.
“I start here because there are so many different ideas in the industry and most of them start with the practices they have heard referencing DevOps. I think this leads to a lot of confusion. I tend to like to start with the Principles and Gene Kim’s definition that is focused on the outcomes”.
“DevOps is the technical, process, and cultural changes that enable the organization to release code more frequently while maintaining all aspects (functional, security, performance) of quality.”
Gene Kim, author of the bestseller “The Phoenix Project”.
This may sound like a pretty straight-forward definition but, according to Gruver, large corporations would be doing this already if they didn’t have all the waste and inefficiencies that exist in their organizations which prevent them from releasing more frequently.
“So to me DevOps is all about removing the waste and inefficiencies that exist in their organization so they can release more frequently with quality,” he says. “When you are not building, testing, or releasing on a very frequent basis the organization can brute force their way through all the inefficiencies. When you increase the frequency, brute force no longer works. You have to automate things and fix the ongoing problems that have existed and frustrated your organization for years. This has a dramatic impact on the productivity of the organization.”
Starts with the business objectives
Gruver says he avoids starting any discussion with executives about DevOps. Instead, his starting point is what matters to them: their business objectives.
If their current development and delivery processes are meeting the needs of their business then it probably doesn’t make sense to invest in changing the processes.
“The reality though for most executives is that their current processes are not meeting the needs of the business,” he says. “I start with how the needs are not being met to clarify the business objectives of the transformation. Then and only then do we start looking in the toolbox to understand what tools would provide the most benefits. This usually comes around to DevOps because these objectives at some level involve increasing the efficiencies and effectiveness of their process. From my perspective, documenting their deployment pipelines and working to increase the frequency requires them to eliminate waste and inefficiencies, which improves productivity.”
The biggest misunderstanding that Gary Gruver sees in the industry is that people think DevOps practices should be the same regardless of the size or complexity of the organization. As he sees it, DevOps is rather about coordinating the work across different parts of the organization. If the organization is small, the practices are all about coordinating the work across small teams. This is also valid for a large organization that has an architecture which enables small teams independently to develop, qualify, and release code.
“The challenge is that most large traditional organizations have architectures that require very large teams to work together to coordinate the development, qualification, and release of their code,” says Gruver. “Assuming that you would use the same practices to coordinate the work across hundreds or thousands of people as you would for small teams is the biggest misunderstanding I see in the industry.”
According to Gruver, one of the most important things you need to be doing to transform your software development and delivery processes is test automation. “It is also one of the things I most frequently see being done wrong,” says Gruver. “Organizations create a bunch of test automation using frameworks that are not easy to maintain or triage. This works fine for the first few tests but over time you are going to have thousands of automated tests and you are going to die under the weight of these tests if they are not designed correctly.”
Gary Gruver on
IMPLEMENTATION OF DEVOPS IN LARGE CORPORATIONS
The greatest challenges
The cultural challenges of getting people to embrace different ways of working. You can add all the automation in the world but if you can’t get developers to prioritize responding to the feedback from the deployment pipeline you won’t get the improvements you expect. You can invest a lot in creating infrastructure as code but if people keep logging into servers and making manual changes you will still have issues. Et cetera …
The greatest difficulties
In large organizations it is getting everyone aligned on a common definition of DevOps, agreeing where to start improving, and how to scale it over time.
The greatest mistakes
Assuming the practices that worked well for coordinating work in small teams are the same practices that will work for coordinating development, qualification, and deployment across hundreds or thousands of people. The principles are the same but the practices can and should be different.
Tests on a frequent basis
Gruver remembers working with one organization that had written over 1,000 automated tests that they were running at the end of each release cycle.
“I told them that is great but the reason you automate testing is not to reduce the cost,” he says. “It is so you can afford to run them on a more frequent basis for feedback to developers. What we should do is have these tests running in a CI environment and the developers can’t go home until they are all passing, but before we do that we need to make sure all the tests are rock solid and finding code issues instead of test issues.”
The organization started an experiment where they ran all the tests – over and over again – in random order, against the same build, and against different builds to see which ones were stable and ready for CI. At the end of six weeks they found out they did not have any stable tests that could be added to CI. They had to rewrite the tests in a new framework that was stable, maintainable, and triagable.
“Don’t make that same mistake!” Gruver warns. “If you have any automated tests, the first step is to see if they are stable enough to add to the CI process and gate code. If not, fix them or rewrite them so they are stable. Using them in CI will force you to fix your test automation framework and design patterns. If you are not using your automated tests to gate code then you are not doing automated testing.”
Small & large organizations
Regarding the evolution of DevOps and the global software community – where are we at, and where are we going – Gruver tends to think about two different types of groups. The first is small organizations, alternatively large organizations with architectures that allow small teams independently to develop, qualify, and deploy code. “These are the organizations that have been really defining what is possible and pushing the technology in new and interesting ways,” he says. “These teams will always be quicker and more efficient just because it is so much easier to coordinate smaller teams. They will continue to improve and create technology to solve problems in creative ways we can’t even imagine yet.”
The second group is large organizations with tightly coupled architectures that require coordination across big groups of people.
“I think DevOps is going to expand rapidly for these organizations because of the impact it will have on their effectiveness and efficiency,” says Gruver. “ They will never deploy and react as fast as the small teams but the opportunity for improvements is much larger just because the inefficiencies of coordinating work across large groups are so much more pronounced. This is where the majority of the software in the world exists and where DevOps can provide the biggest benefits so it is where we will see the most action. It will never be as sexy and interesting as what the small teams are doing but it will have the biggest impact on the productivity of software development around the world.”
The three main benefits of DevOps according to Gary Gruver
1. Making developers better developers by increasing the quality and frequency of feedback they receive.
They all want to do a good job and think they have done a good job until they get feedback letting them know something needs to change. If this feedback comes weeks after they wrote the code they will have forgotten what they were doing and you are beating them up for something they don’t even remember doing. If instead you can give them feedback while they are working on the code, it can help eliminate the waste of working on code that isn’t secure, that doesn’t work with the code others are writing, or that won’t work in production.
2. DevOps can help to dramatically improve the efficiency of the triage process.
Lots of large organizations spend more time and energy creating and triaging large complex test environments at the end of long development cycles than they do writing the code. DevOps helps to improve the efficiency of this triage process because the increase in test cycles means there are fewer changes between when a test was passing and when it started failing. Additionally a well-defined deployment pipeline uses quality gates at the subsystems to localize the triage to the teams that created the problems and keeps these defects from causing instability in the complex enterprise test environments.
3. Repetitive tasks like build, test, deploy, create an environment, configure a database, etc. get automated.
The automation of these repetitive tasks has three main advantages. First, it is eliminating the work associated with manually doing something that can be automated. Second, because it is automated it can be run more frequently enabling fast feedback to developers and smaller batch sizes for more efficient triage. And lastly the automated process is more repeatable so there are less manual errors to triage and fix.