Developing a Patch Management Strategy for your Business

Note: This article is aimed more at large to Enterprise sized businesses, who need to protect their business, and need a strategy in place for how to approach patching. Focus is on Microsoft based environments.

Although a pretty dry topic, I hope that what I can share here from my personal experience in this space can help any business struggling to get something in place.

I have deliberately refrained from any product recommendations in this document, as it is about the development of a strategy, rather than the in depth detail of implementation. However, there are plenty of tool-sets out there that make this an easier job for the team, once a strategy has been laid out.

Introduction

As businesses, we are responsible for ensuring the confidentiality, integrity, and availability its data and that of customer data stored on its systems. Even more so now these days with legislation like GDPR, Australian Privacy Laws, and now our very own brand new New Zealand Privacy Law (2020).

As organizations, we have an obligation to provide appropriate protection of that customer data against threats, such as malware, viruses, trojans, and worms. Any of which could adversely affect the security of our systems or the data on those systems to which we have been entrusted.

As IT professionals and leaders, we should be using best practices to balance the needs of security and that of the business.

Effective patch management can mitigate a significant portion of this risk by applying standard processes across all systems to ensure that security and feature updates are applied proactively, and in a timely fashion, covering all areas of the business. This is a key player in protecting IT infrastructure and application software from disruption and security breaches.

Good processes and policies are essential for the success of any strategy such as this one. Defining policies and processes that everyone understands and adheres to helps make the environment more manageable and reduces risk.

In addition to addressing security, regular updates and software revisions provide staff with the ability to use improved feature sets and new functionality, which is an important point not to be overlooked.

Key Objectives of a Patch Management Strategy

The objectives of a Patch Management Strategy should strive to:

• Ensure that client & server operating systems and business productivity applications are kept within the mainstream support cycle of the respective vendors.
• Ensure that these business applications and operating systems are kept up to date with security patches to maintain compliance with governing security standards.
• Enable the business to leverage new productivity enhancements that are released through software updates.
• Establish an appropriate testing and release process that minimizes the business risk of software change.
• Produce and maintain a well-documented baseline software standard for business operating systems and applications.

Key Considerations

Businesses must meet their obligations in regards to regulatory requirements and protection of customer data. A breach of this information under these regulations may be publicly notifiable, and therefore can result in a loss of brand reputation. This ultimately can affect a company’s trading performance.

The organization's IT team need to be implementing solutions that can support the overall patch management strategy, and apply patches in a timely and proactive manner. This is imperative in preventing potential breaches related to discovered vunerabilities.

As a business grows in size, it becomes a bigger target, and also has a larger attack surface, therefore more likely subject to targeting by parties with malicious intent. Without handling patching appropriately and ensuring that even globally roaming users are being secured, we open our businesses to a larger extent of vulnerability.

Upgrading software versions and their subsequent feature sets also enables the users to leverage new tool-sets to stay ahead in a global market.

With the introduction of Windows 10 by Microsoft in 2015, the release method of new operating system versions has changed. There is now a twice-yearly feature release, with a 18-30 Month support life cycle before that release version is retired.

Especially initially, but even still today, there has been a lack of awareness of the impact of this new model, which has left a lot of Microsoft customers in non-supported states across their environments. It is ont sufficient to wait until the hardware that the operating system is installed on has become end of life before an operating system upgrade occurs. Patch management processes need to be adapted to accommodate this new release cadence, and keep up with the rate of change.

It is important to define a patch management strategy that does not just focus on the Microsoft suite of products, as that is not the only items in an environment. It must encompass all user and server applications and associated life cycles, as well as firmware. Firmware vulnerabilities are a widely used attack vector, and they are often deemed “too complicated” and therefore not patched.

Challenges

Windows 10 has brought significant changes to how the operating system is serviced and updated by Microsoft. Many organizations are struggling to keep up with the constant release of new patches and updates.

The challenge is to keep up with the pace of operating system updates across the environment to ensure vendor support. This needs to be in such a way that it does not significantly increase workload on the IT staff, or cause unwarranted disruption to business users.

In this new world of increased cyber-attacks, the time between the discovery of a vulnerability and the emergence of an exploit is getting shorter, sometimes only a matter of hours. This applies pressure to rapidly patch production systems. However, this directly conflicts with best practices and processes regarding testing before business wide deployment.

A roaming user base and flexible working policies apply greater pressure to provide 24/7 system availability.

The challenge, is to develop a process to get these systems patched without causing issues, and make sure that the systems are available for business users when they require them.

Baseline Configurations are Important!

A baseline configuration policy sets the minimum requirements for a standard build, that is, a "known state" of a system or application. These are an important part of a patch management process, as it establishes a minimum requirement, and a standard that can be reported against.

Due to the extremely important part that they play within a patch management strategy, these baselines need to be clearly documented and adhered to.

As a product traverses through its product road map, the responsible parties for the process need to determine when a new baseline of a standard build / of a component of the standard build should take place. Staff can then follow the product road map and adhere to the defined baseline configurations for new standard builds.

This sets out the road map for upgrade work. A baseline configuration standard establishes the upgrade path to move from "known state to known state".

Patch Assessment

When patches become available, they need to undergo an assessment, including assessing the risks of not applying them. The patches can then be categorized based on potential business impacts.

Not all patches are able to be deployed immediately. A patch may break a critical business application, or there may not be a change window available in the short term.

In instances such as these, I recommend that a risk register is maintained with details of the update, impacts of not patching, and reason the decision to defer was made.

There then needs to be planned activities for either mitigation, or patching at a later date once application compatibility, or end of a change freeze period has been achieved.

Consider the use of Gartner's guidelines on patching criticality to define your internal operations:

Patches can also fall into different categories when considering the scope of the change.

For example, desktop client patches for software applications like Adobe, are a far smaller change footprint than a centralised server operating system upgrade.

Items with a much larger change footprint need more rigorous testing and planning. One can expect that they will not be as frequent, and require a larger business outage.

Patch Testing

Specific tests should be defined for various patch categories and the scope of which can be quite vast. However, below are some testing guidelines that I have used in the past for various types of releases.:

Core business applications should be prioritized for testing of any operating system updates. Users' require stability in these applications first and foremost.

It will not be possible to test everything for every update, given the frequency of monthly updates, but there is still a need for testing of updates against enterprise applications and core business systems.

Implementing an effective early adopters program will smooth out issues with the monthly patch cycles before widespread deployment. This also follows the best practice for this outlined by Microsoft with their update rings:

The responsible party for any specific patching work needs to be able to identify and meet all production readiness requirements prior to business wide deployment being approved. They should be able to show evidence of testing being carried out, and passing results. They should also produce a detailed patch plan with planned activities, and milestones for implementation. This needs to include risk mitigation and roll back plans.

Patch Deployment

Before production deployment of any patches or major revisions, a change plan should be presented and approved.

For monthly patching, I recommend implementing a standardized process, back out, and change plan. Therefore items can be scheduled without the need for a detailed review. A standardized process should reduce the overhead of these changes.

For larger revision items, one should present the plans recommended in the above mentioned testing process. I suggest that for client-based items, that roll-outs are phased.

After Production deployment of a patch, application availability needs to be validated by the implementer, and perhaps some key business users who can sign of on functionality.

Upon successful production deployment, this new software version now forms the new baseline standard version for the product. Documentation on baselines should be updated to reflect this.

Reporting and Continuous Improvement

In order to ensure effectiveness and continued relevance of the patch management strategy and processes, reporting needs to be completed on deployment roll-outs.

This should cover things like failure rates, business down time, if manual intervention was required etc.

In addition, regular reporting across the fleet should be run to ensure that the baseline standards are being adhered to. Any items then not adhering to this can be rectified as well, tightening any security holes.

This activity serves as a mechanism to continually optimize, refine and update patching execution methods. Assessments of the patching processes should occur yearly at a minimum, as well as after any patch incidents and outages that were caused by the patching process.

Patch Cycles

This needs to take into account product life cycles, as well as regular security update releases. Mapping this out establishes the ongoing timing or release schedules for patching all systems, software and applications.

Planning the patch cycles is designed to achieve an appropriate balance for patching activities, given that applying patches too frequently may create unnecessary disruption to business systems, and infrequent patching or not patching at all may leave systems open to risk in regards to vulnerabilities or compliance standards.

Change freezes will need to be considered, if there will be no deployments during these times, these are then exceptions to the published cycles. However, these times can always be used for preparation and lab testing of deployments, so are not lost opportunities.

For items that are not required to be part of an emergency change workflow, the following example patch cycles form some guidelines as to what you might like to implement:

Security and Critical Patching

This should be released on a monthly basis, with the exception of change freeze periods.

Monthly Client Updates

These can follow an overlapping rolling cycle.

For example:

Monthly Server Updates

Patches are released on a monthly cycle, but may not be deployed monthly.

This depends on impacts and change window availability.

An example process:

Functional Updates

These will run on an annual or application specific release cadences. This needs be clearly defined in the establishment of each type during the laying out of the patch cycle for the software.

Major client releases

(Operating System Build Revisions / Productivity suite upgrades)

Example:

Major server / Enterprise Application revisions / version upgrades

Example:

Communications

Part of patch management is effective communication plans. If the business is prepared and engaged, then there is far less cause for concern.

As far as major client upgrades go, the users’ responsibilities need to be communicated to them as part of this process.

These need to include things like their responsibilities i.e. to store important files on network drives and to check their applications are functioning and report any issues. This is especially important in the Early Adopters program.

Implementation

The implementation of controlled patch management within an environment should be staged. This will ensure that each item brought into the process is addressed with care and that everything can be resourced in conjunction with many other competing business requirements.

Due to the scope of on boarding such a process, not everything can be achieved in a short time frame and a longer term implementation plan needs to be put in place. This is not a "quick win" item, rather an ongoing continuous improvement initiative.

As each piece of software is brought on board into the process, consider the following steps to bring it into the process:

1)     Discovery (know what is in place already and where the gaps are)

2)     Understand product life cycle

3)     Create a minimum baseline and supported versions

4)     Establish effective tools / methods for update process

5)     On-board application into Patch Management process

6)     Review and refine processes

7)     Transition to BAU

Conclusion

I hope I haven't bored you to death talking about patching, but if you have gotten this far and you have any questions or comments, please don't hesitate to reach out, and thank you for reading.