The Yin and Yang of DevSecOps - The 3 Gotcha-s of a Successful DevSecOps Transformation
A lot is written and spoken about the benefits and positive outcomes (Yang) of DevSecOps. There is also a substantial body of opinions and advice about how to undertake a DevSecOps transformation. However, there is very little said about the over-the-rainbow, unknown (Yin) side of DevSecOps.
As an Agile and DevSecOps practitioner and Coach of 8 years, I have worked with organizations of different sizes and levels of commitment to DevSecOps adoption. I have seen the same patterns several times. Teams and organizations begin their transformations with tons of excitement and a little bit of pomp. But almost inevitably, especially if the transformation is done right, the teams come across a certain (usually short-lived) chasm of disillusion and disappointment. They run into the 3 Gotcha-s I am about to describe. The larger the organization, and the wider the swing between traditional roles and practices, and new/DevSecOps roles and practices, the more pronounced the 3 Gotcha-s and respective disappointments are.
Let's first qualify "Successful DevSecOps transformation":
DevSecOps, very much like Agile, is not a destination but a journey. It is not a binary and well-defined state of being, but a murky, mushy (and sometimes - a little messy) amalgamation of practices, tools, culture, and organizational design. For that reason, there is no distinct point in time when a team can claim "Success - we are now DevSecOps!". However, along the way, if all these practices, tools, cultural, and organizational changes are done right, teams start to noticeably behave in a different way. We start to see them taking ownership and wearing not one, but three hats for "Dev", "Sec" and "Ops" - all at the same time. This is also when the teams start experiencing the Gotcha-s of their new paradigm.
Gotcha#1 – “Oops, we must do Ops too!”
DevSecOps transformations typically start from the development shop in large organizations. The developers are accustomed to creating and testing new features and releasing them periodically to Prod. At that point, the new product version becomes the problem for the Ops organization, while the Dev team pivots to developing and testing the next set of features.
Under the new DevSecOps paradigm, the same development team continues to own the Prod version of the product, with all its messiness and "nuisance" such as operational issues, incidents, security vulnerabilities, end-user complaints, and service requests. As the saying goes, the developers are forced to start eating their own dog food... and they don't like it at first.
Additionally, all ad-hoc and urgent requests coming from Prod have no other place to go but the backlog, and more often than not - in the current sprint backlog. The teams start to experience some churn and unevenness in their burndown charts - something that they had worked so hard to overcome before becoming DevSecOps.
Top that with the implied 24x7 responsibility for production support, monitoring alerts, and triage.
All of a sudden, the same developers that thought CI/CD, test automation, and containers are super cool and they were two thumbs up for going DevSecOps, now they start to rumble and grumble a little. They are not fully happy to pay the price for wearing the "Sec" and "Ops" hats.
If the sprint churn is too much, consider switching to Kanban or Scrumban. Or at least, remove the % StoriesComplete/StoriesCommitted metric from your dashboards not to penalize the team.
Reserve capacity for enhancements vs ad-hoc issues. Within a few sprints, you will figure out the best ratio.
Lean on Ops automation and self-healing scripts, in addition to CI/CD and testing/scanning automation.
Gotcha#2 – “The Backlog Blew Up!”
A related side effect of undertaking Ops and security responsibilities is that now, the team may be dealing with multiple ticketing sources. Typically, large organizations have a separate IT Service Management (ITSM) system and related ITIL-aligned enterprise processes for Service/Change Request Management, Incident Management, and Problem Management. Each of the above - Change, Incident, and Problem tickets become work items for the same DevSecOps team that also works on product enhancements and bugs logged in an Agile Lifecycle Management (ALM) system. In many cases, the two systems are disconnected creating a multiplicity of queues, priorities, and workflows for the team. (If we are lucky, the Security Incident Response is handled by the same ITSM system and not a third enterprise system.) The team no longer has one backlog with clearly defined priorities and commitments, and an easy way to measure velocity. Work starts to fall between the cracks and the team starts to hear the rumble and grumble from users and leadership.
If consolidating the ITSM and ALM functions into a single system of record is not feasible, consider integrating the two systems. Tools such as ServiceNow and Jira provide powerful integration mechanisms.
If integration is not feasible (or not coming any time soon), even at the expense of manual sync-up - continue maintaining a single all-encompassing backlog in your ALM system.
Create internal processes for monitoring, triage, and prioritization so that urgent/high-severity production issues are not overlooked while in the weeds of feature development, and vice versa.
Gotcha#3 – “We are never done! Where is the end of it?”
This Gotcha is a little bit more subtle than the other two but can be just as unsettling.
Human brains are wired to crave new challenges, set new goals, and get satisfaction by achieving those goals. It is our evolutionary reward circuit in action. The bigger the goal/achievement, the bigger the reward (aka dopamine rush).
Unfortunately, this comes in a direct contradiction with the Agile tenets for reducing the batch size, and the DevSecOps mantra for continuous deployment/delivery. The smaller, more frequent, and less risky the product releases are - the more uneventful they become and the less dopamine rush we get. This may affect some team members more than others, but sooner or later "maintaining a constant pace indefinitely" starts to get a little boring and may breed complacency. Teams become the victims of their own success.
To avoid losing sight of the forest from the trees, periodically review and revisit your "Why-s" - the big-picture vision for your product and the mission of your organization.
Create and build in some small meaningful and achievable goals or milestones for the team to give you reasons to celebrate. It could be as simple as a small party for your every 10th sprint, a celebration for every 100,000th line of code, or a shoutout for every 100th code commit.
Welcome and embrace imperfection and chaos in small quantities. Occasional push to make the date for a larger release is okay - it will make your team stronger and more alert.
The 3 Gotcha-s discussed here are not necessarily bad, wrong, or something to avoid. They are only the underappreciated and kept-in-the-shadow flip side of the same coin.
If any of the above Gotcha-s sounds familiar, it only means you are being successful with your transformation and you are doing DevSecOps the right way. Embrace the challenge and use the coping strategies to adjust to your new paradigm.