How did they get to be running this impossible race? Well, better tools and working practices have meant a dramatic shift from annual software releases to a world where teams have the ability to deploy multiple times every day. But!☝️ DevOps teams in places like fintech, automotive and healthcare still have mandatory compliance obligations. And the change management processes that worked for annual releases aren’t working now.

The automation we’ve seen in build, test, and security has not been mirrored in change management. That means it’s becoming a bottleneck for regulated teams and it’s time to ask what we should be doing about it. How do we bring a compliance culture into our DevOps?

Key takeaways

• DevOps is producing volumes of change that IT can’t handle
• External approvals for releasing software are not only slow but also risky
• Building compliance into your DevOps means faster and better releases

A decade of DevOps - from silos to serverless

It’s hard to believe, but we’ve been doing DevOps for over a decade. In that time the way we build and deliver software has changed significantly. Tools that we already take for granted - Docker and Kubernetes, Jenkins and CircleCI - are all relatively recent inventions. But can you imagine life without them?

You don’t have to be old to remember when systems were built on large, custom managed hardware. And when you joined a software organization you’d find yourself siloed by specialty. There were different departments for development, analysis, QA, and ops. Software was delivered once a year, maybe once a quarter if you were at the cutting edge. Change was generally seen as a bad thing. It was large, risky and rare.

But we soon figured out that smaller, services-oriented architectures and independently deployable systems were much better ideas. We started to develop more dynamic and resilient designs. We realized we could use Conway’s Law to organize ourselves around these services.

And these trends continue with the growing adoption of serverless features and functions-as-a-service in the cloud. In a decade of DevOps we’ve come a long way. There’s every reason to believe that the next decade will see more innovation and even higher volumes of change. But how do we prepare for that?

What does DevOps mean for change management and compliance?

One of the challenges we have when it comes to understanding the impact of DevOps on technology organizations is that change management and compliance mean lots of different things.

Managing change and ensuring compliance has always been the overall responsibility of the IT department. They typically consult the ITIL framework for guidance on how to manage change in their IT systems. And there are lots of different types of changes to manage.

There’s changes to infrastructure, servers, and hardware. Data migrations and enterprise resource planning. Web updates and performance improvements. Traditionally, all of this was categorized in a very general way as change management.

Compliance is a related and similarly broad term covering a range of topics which are also IT’s responsibility. There’s open source compliance, policy compliance, compliance with industry standards, compliance with internal standards, regulatory compliance, etc.

Change management at the software release

But there’s one aspect of change management and compliance where DevOps has an enormous impact - the software release process. In regulated industries teams need to know what software they have running in production and how it got there. They need to track and document changes to production to ensure their process is in compliance.

Naturally, software releases were also part of the IT department’s change management responsibilities. And that made a lot of sense in the days before DevOps. You’d batch up a bunch of changes, make a change request to a change advisory board (CAB), get their approval, and then release the changes over a holiday weekend in case everything went sideways. There was a lot of risk involved and a prolonged, manual approval process to mitigate it.

And it’s somehow worse than no change process at all

But these processes aren’t even good at mitigating risk. Thanks to extensive DevOps research in Accelerate by Dr. Nicole Forsgren, et al. we now know that:

“External approvals were negatively correlated with lead time, deployment frequency, and restore time, and had no correlation with change failure rate. In short, approval by an external body (such as a manager or CAB) simply doesn’t work to increase the stability of production systems, measured by the time to restore service and change failure rate. However, it certainly slows things down. It is, in fact, worse than having no change approval process at all”

So, not only slow but risky too. Factor in the increasingly dynamic ways we have of delivering software and it’s clear that the old ways of managing change are not only inefficient, but potentially disastrous.

DevOps automation meets manual change processes

DevOps, more specifically continuous delivery, produces volumes of change that the IT department was never designed to handle. How is IT supposed to effectively manage software releases when DevOps teams are able to deliver hundreds of changes every day? 🤔 How is it supposed to ensure compliance in any meaningful way when it’s using manual processes to rubber stamp highly automated software delivery pipelines?

It’s clear that no amount of requests, manual approvals, CAB meetings and paperwork can manage change at these volumes. As we’ve explored in a previous post: ITIL just doesn’t work for DevOps.

So, what if we take responsibility for managing software releases away from the IT department and give it to DevOps teams? 🙌 Let IT do what IT does best - slow and risky tasks like managing servers, migrations and hardware. For dynamic ways of delivering software, DevOps teams need their own change management automation.

Building a DevOps culture around compliance

Modern technology organizations do not have traditional separations between software development, quality assurance, and IT Operations. That has been replaced by cross-functional DevOps teams responsible for the entire value stream of software systems. This new DevOps approach allows organizations to reduce handovers, increase collaboration, and ultimately deliver more innovation.

And as organizations change, the supporting technology approaches also change. Companies are adopting metered cloud infrastructure as a service, automated build, test and security tooling. With the support of this automated DevOps delivery, high performing teams can deliver changes 973 times more frequently than traditional teams. 973 times!

If we need to be compliant, but external approvals and management don’t work, what should we do?

Continuous Integration > Continuous Delivery > Continuous Compliance

This vast increase in the number of changes needs new and better approaches to change management and software process compliance. To support the efforts of DevOps teams, we need to move away from manual and gated checks towards continuous, automated checks. We need to take the lessons we learned about continuous integration and continuous delivery and apply them to compliance.

We have seen in practice that the adoption of Continuous Compliance methodology leads to the same sort of benefits we’ve seen with test automation.

Adopting test automation doesn’t make testers obsolete. Instead, it removes monotonous and repetitive work, freeing testers to work at higher-level exploratory testing.

In the same way, Continuous Compliance does not mean the end of compliance work, but enables compliance officers to work on higher value compliance investigations.

Conclusion - the benefits of a DevOps Compliance Culture

The DevOps principles around culture, automation, lean, measurement and sharing can help teams increase compliance activities while reducing delays and waste. Overall, the benefits from adopting a DevOps Compliance Culture are:

• Culture: gives teams agency to lead risk management responsibilities
• Automation: drives higher compliance conformance and reduces waste
• Lean: results in continuous improvement in risk control posture
• Measurement: ensures compliance becomes data-driven
• Sharing: makes compliance work visible to empower compliance and security functions

DevOps teams need to build a compliance culture, especially if they’re releasing software in a regulated industry. We can see that DevOps is producing high volumes of change that IT can’t handle. And we know that relying on IT for change management at the software release is both slow and risky.

If regulated teams want to release software with speed and compliance they need to automate change management as part of their DevOps.

That’s it. That’s the tweet.

These are such great analogies. Unless we’re running away from imminent danger🦁, humans have a really hard time processing that faster is safer. Usually, when we feel threatened or uncertain, we slow down, take our time and look at our options. 🤔

Even in relatively comfortable situations, we sense that increasing our speed will also increase our risk. And in the world of software these “deep-seated biological impulses” create human barriers to automation.

Yeah, why is that?

There are several reasons why tech leads and managers cling to these deep-seated notions of predictability, safety and routine. Continuous delivery scares them because…..

They don’t trust their teams to push desirable changes to PROD every 20 minutes ❌ They think batching changes means more oversight (also great for spreading any potential blame around 👍) They think slower deploys 🐌 make for higher quality and fewer bugs DevOps people know none of this is true. Low frequency deploys and batched changes aren’t good for productivity, but they do give managers the illusion of control. And because the idea that “faster is safer” doesn’t feel right, people will make all kinds of trade-offs to hold on to that illusion.

Wizadora, nothing’s surer!

As ever, DORA❤️ has findings that back up what we see in the field. In the 2018 report, they found that “misguided performers suffer from a cautious approach.” It’s worth quoting at length:

“We often hear from organizations that prefer to take a cautious approach to software development and delivery. They assure us—and their stakeholders—that releasing code infrequently can be an effective strategy, as they use the extra time between deployments for testing and quality checks to minimize the likelihood of failure. For the most part, they achieve that goal.”

Notice the positive reinforcement. They believe slow is safe and they’re even seeing results that confirm those beliefs. It’s all fun and games until someone loses an eye though. The report continues:

“Developing software in increasingly complex systems is difficult and failure is inevitable. Making large-batch and infrequent changes introduces risk to the deployment process. When failures occur, it can be difficult to understand what caused the problem and then restore service. Worse, deployments can cause cascading failures throughout the system. Those failures take a remarkably long time to fully recover from.” 😬

How long exactly⏰? DORA found that anywhere between 1 and 6 months was a reasonable ballpark for a full recovery. And it was all going so well! We were on the ice and inching along. Suddenly, we’re flat on our faces with no easy way of getting back up. Turns out slow wasn’t that safe after all.

Plus ca change!

We found something similar in a previous post on the FCA’s report on Implementing Technology Change. Asked to grade their own homework ✅, managers thought that their costly and time-consuming change management processes were effective, precisely because they were costly and time-consuming!

If it costs money and takes time it must be doing something, right? Managers were buying into that illusion even as they reported that change failure rates were the single biggest cause of incidents. Cognitive dissonance to the moon! 🚀 🌝

Charity’s tweet inspired me (thank you🙏!) because the idea that “speed is safety” applies to change management as well as continuous delivery. They’re part of the same value stream. If you can automate🤖 your change and release controls as part of your DevOps pipelines, there’s no reason to delay release candidates because some guy👨🏻‍💼 with a clipboard needs to eyeball 👀 them first.

Managers still see their job as a high wire act where a pinky toe on either side of the rope means disaster. We need to encourage them to go faster by thinking more like ice skaters and less like tightrope walkers.

First published here