Managing Shadow IT

In early 2011, I returned to the Home Depot in the network engineering group. One of my first missions was to roll-out a NAC (or Network Access Control) solution across chain and we had less than 3 months to do it. Some times I wonder what management was thinking the day I very confidently strolled in to the directors office and told them, "no problem." It was a point of playful mocking by my fellow engineers for quite a while. It was much later I learned our vendors thought it was impossible at that speed considering the shear volumes of hosts involved.

So how did we pull it off? Team work and DevOps practices. It was a very creative blend of programming and efficient work flow patterns. The completion of this project sparked my creative energy for solving challenges at Home Depot. Since I enjoyed solving challenges and had success, I began receiving a lot of notice from directors, VPs, and managers who began coming to me to solve problems they could not get solved. This presented me with the opportunity to work in many disciplines such as network engineering, network operations, and even enterprise monitoring. I also helped out from time to time with incident management, Linux and Windows servers, web development, and even security.

One day while working in the NOC (Network Operations Center) a very interesting and fun experience finding shadow IT came about. Shadow IT often happens when internal resources become a barrier to entry because of cost or time to deploy. Internal teams often have people with enough experience to perform IT tasks, but lack the process knowledge to tie it into the enterprise. For example, marketing sets up a web server because deploying a web server inside the enterprise takes too long or costs too much. In shadow IT these systems will typically lack elements like change management processes, monitoring, capacity planning, and escalation paths for support.

The ask ... to find all servers on campus that could be shadow IT servers.

They knew using a network tool (such as SolarWinds) could help them find a lot of potential shadowy boxes, but there were a great number of them and the majority were false positives. Also network tools could only get them within 100 feet (the length of the cable). So I did a cost analysis on the physical labor required to find all possible shadow IT boxes and it worked out to be in excess of $3 million. There were simply too many possibilities. So I began contemplating on how to reduce the number of possible servers.

I figured shadow servers would exhibit a very specific behavior. So this started me thinking. "How about using behavioral analysis?" Predictive analytics could help. The typical challenge with practices like machine learning or predictive analytics is knowing which algorithm or model fits the situation the closest. But thanks to my experience with shopping carts (specifically spending lots of my own money) I knew which one would work for this case. It's called cosine similarity.

Cosine similarity is something we have all probably used, but few probably know what it is called. It is the recommender program on large shopping sights. It happens when you select an item in your cart and you get five other products like the one you choose. This is based on attributes of a product. So I surmised you could create a list of attributes that any shadow IT server would exhibit. Then run the list of potential candidates through a python program performing cosine similarity matches. This greatly reduced the number of false positives and saved the company millions of dollars in contractor labor chasing down ghosts.

In closing, it is always good to take a step back from a problem and give it some thought before blindly executing. It typically saves a great deal of time and money. It is also important to be actively learning. Even an education outside of your current area of expertise can produce remarkable insights. In this case, the idea of using the cosine similarity solution was born more from applied mathematics than computer science. And that day it paid big.