The Patch Problem

A number of organisations have recently asked for help with patching, specifically in response to vulnerabilities. They've found patching to be not just difficult, but very, very difficult - in fact a wicked problem - and have asked the industry for help in solving it.

Vulnerability scanning itself is an old technology, one that went through the hype cycle. When we first used it, we were very excited to finally be able to enumerate the vulnerabilities in the attack surface. Not so our colleagues in IT Ops, who's jobs suddenly went from fixing everything that went wrong - to fixing everything that went wrong and also remediating the millions of vulnerability instances that were suddenly enumerated. They viewed vulnerability scanning as a plague on all of mankind. The problem of vulnerabilities felt too big to tackle - hundreds of thousands of machines each with thousands of different vulnerabilities, different software stacks and each critical to some portion of the business. That becomes millions of unique regression tests, one for each combination of software stack and patch set, to find the correct order to apply them in order to both patch the vulnerabilities while maintaining a stable service. It feels like drinking from a firehose.

But as organisation after organisation has discovered, the fact that a problem feels too big to tackle isn't a good reason to do nothing. Maersk, the NHS, WPP, Rosneft, Deutche Post, Merck and Saint Gobain can all attest to varying levels of pain from the lack of patching vulnerable portions of the IT estate. The costs of failing to patch are in the hundreds of millions of pounds, just across those companies alone.

So, what to do?

First, don't shoot the messenger. Keep the scanning. It at least lets you know the shape and size of the problem, and you can begin tracking loss and outbreaks back to vulnerabilities you knew about, thus making the case for greater budget and investment in currency in the IT Ops world. Vulnerability management involves scanning the estate, discovering new boxes and images, identifying them correctly, placing them in the correct scan schedules, determining which scans to apply to those schedules, scanning them as often as is technically feasible and then informing the platform teams just how totally under the hammer they really are. Technology such as Tenable is great for performing the scan, but how to go about patching?

Second is to prioritise patches by risk to the organisation, so you're taking care of the most important patches first. There's new modules from places such as ServiceNow that can help to organise the information, which is a start. In the ServiceNow SecOps offering, there sits a handy Vulnerability Response application which will take the full list of vulnerabilities from a scanner such as Tenable and allow your organisation to begin to make a stab at prioritising. You can turn the handle on a vulnerability scoring system and, provided you have good criticality information from the business, you can start to prioritise which patches will give you the best bang for your buck.

Now at least if you're only going to patch a tiny portion of your estate and leave the rest vulnerable, you can take care of the biggest risks first.

Third, start to look at mechanisms for being able to connect patching with your automation and orchestration tools. Here at Adarma, we're fans of using generic automation execution engines to do this, such as Tachyon.

Finally, rethink your lean Pre-Prod strategy. Lean pre-production is the idea that, since changes in production are normally relatively rare, you don't need to have a full copy of each production system - or even a scaled down copy of one (frequently, disaster recovery and business continuity overprovisioning is scaled down for the pre-prod system). Instead, some enterprises have implemented a policy that, when you need your pre-prod environment, you spin one up in VM and work on it there. This probably isn't going to be wide enough to regression test the millions of patch and application stack combinations you have to test to be sure there won't be production outages from all these patches you're about to deploy.

One thing you can be sure of, though, is that if you don't measure your vulnerabilities and don't apply security patches, you'll end up in the news.