You look at the alarm clock that’s proven to be amazingly resilient to abuse, particularly the snooze button, and it’s fifteen minutes after midnight on a Sunday morning. Your alarm is going off because it’s time to go into the office for the change window; your company’s Change Advisory Board (CAB) had voted on Thursday afternoon that your weekend was going to suck.
I worked for a managed services provider back in 2010, and this is an un-embellished recount of a typical change window. At the time, I was a managing network engineer. The fact that we ran our own data centers meant that someone had to be on-site in case connectivity was severed. My Friday afternoon was oftentimes taken up with writing network equipment configuration scripts, saved to text files, for the upcoming window. The time I actually spent working was minimal, most sessions. Ten or fifteen minutes pasting firewall or routing updates, maybe tweaking the configuration on a load balancer, and then it was a waiting game for the other teams to deploy. QA would log in and run the test suite near the end of the window, then we would all leave the office about the time the sun was coming up on Sunday morning.
This had become my life — losing my weekend nights to the drudgery, rinse and repeat. I was burnt out. And when I was ready to bounce after a couple of years of this, there was nothing that could convince me to stay.
The systems engineering team had been writing their own automation tools for provisioning virtual machines, and Puppet was being considered for orchestrating the services and applications inside the VMs. Software-defined networking could have made my life so much easier but it was too late. I had my eyes on the door long before our network hardware vendor started talking about OpenFlow.
I went into security after that gig — I wasn’t just tired of the job, I was burned out on the network engineering field altogether. Working on the vendor side during the advent of the security automation movement (if we can call it that), I get to observe the same story play out for friends on the practitioner side of the equation. History rhymes.
The CAB used to function as a sort of human shield for production: In this (one of the most expensive meetings in the entire company) you had representatives from every layer of the operations and development disciplines, together with security, product and project ownership. Now, many of those roles have been mechanized. Process-wise, Continuous Integration and Continuous Deployment processes facilitate the coordination of work between project and product management, development, and operations. The only human remaining (more or less un-augmented and un-automated) is the Security Engineer, who gets a limited amount of time to review changes before they are rolled into a deploy, and who then has to react to every mistake and oversight that flew under the radar on the way out the door.
Is it really a big surprise that security folks are still burning out and hopping jobs so frequently?
Throwing money at your security people can slow down the exodus for a little while, but money isn’t usually the biggest problem: It’s the stress that comes along with not being able to focus on the important stuff because you’re being suffocated by a flood of minutia and a real fear of failure because of it.
We can be quality-obsessed and we see our work as a representative, an extension, even, of ourselves. When we don’t have the time or tools at our disposal to produce the quality of work we can be proud of, our corner of the company culture gets dark and stormy. So how do we turn this ship around?
Securing production workloads is usually the easiest place to start. The efficacy of traditional tools diminishes the more agile your delivery process gets. This is especially true for network-based scanning tools which may never be able to complete a full scan across all production infrastructure before it changes with the next release. You need something that is built into the deployment process. Something that’s not a bolted-on afterthought. Something that will fit into an automation plan and not need broad access from outside for a scanner. But then what happens when you’ve got a bunch of tools, all with their own single panes of glass? How many single panes of glass do you really need?
Don’t suffer the hubris of security products with bad integration options. Like a SOAP-only public API, for example. You shouldn’t have to throw your time at integrating with some atavistic SOAP-based public API. It’s your data, and you should be able to extract it without the rigidity and strict formality of constructing SOAP envelopes. Career software engineers don’t usually have an issue using SOAP, and it makes sense more often for internal APIs. For integrators, however, this sort of unnecessary complexity is the enemy. A product that can easily integrate with your workflow helps end management clutter and becomes an integral part of your security automation platform, not just another source of eye strain. It’s a single pane of glass that doesn’t have to happen.
Pipeline security automation is the other side of the coin. DevOps teams get all sorts of fun tools to do the mundane work for them. Many of them can be used to deliver security functionality. One of the first security functions to find its way into the development pipeline is static analysis. When implemented in the Continuous Integration (CI) process, it gives the development crew quick feedback on the quality of their work, from a secure coding practices perspective. As more and more of the application stack becomes software-defined (networking, server infrastructure, containers), the tools that enable code quality assessment can be augmented with tools that analyze infrastructure configuration before it finds its way into production.
Do you see what we just did there? We went proactive. The security engineer who had to react to the stuff that found its way into production now gets a list of infrastructure configuration anomalies— all in one report— and can react accordingly. Why even ship a report? Have your security automation stop a deploy and file a ticket for anything egregious and pass-but-ticket anything that’s not super critical. Your security teams aren’t staring at their feet as they run anymore— they can see vulnerabilities and exposure even before they go out the door in a deploy. Your security automation becomes an automated coaching mechanism for your developers and DevOps folks, giving security-oriented feedback on code quality for infrastructure. So much for burn out. Now we’re learning and growing.
Security culture just found its way into the development group, but now we’re finally getting everyone on the same page.
