Virtualization, HyperConvergence, Containers, and Beyond...

As a Systems Administrator in a midsize organization, I always came up against resource ceilings; either the DBAs needed too much SAN storage space, or the Dev guys needed too much compute or memory. I remember telling the Dev guys to install all  related applications on a single server. However, this would then put us in hot water with our Security and Audit guys, as securely isolating the apps was next to impossible.

Then the server virtualization wave hit. Along with SAN technologies like data replication, Metro Clustering, etc. it seemed like a dream come true. Dynamic memory and overcommitment, along with hypervisor clustering provided huge benefits by increasing server density, high-availability and reliability. However, we now had to manage the hypervisor, SAN storage and the networking, each using their own plane of administration. Configuration management tools like Chef and Puppet would not make much sense in our mid-size environment.

Luckily for us, hyperconvergence products like those from Nutanix, etc, provided a simple pay-as-you-grow solution. Their management planes could cluster multiple systems together, providing storage and computing power as a single big cluster. Their management consoles allowed us to manage and monitor the whole Virtual stack. I remember Nutanix provided support even for the Hypervisor layer, by utilizing existing VMWare and Hyper-V support contracts. However, these products could not prevent us from hitting Virtualization sprawl. Imagine having hundreds of tiny, single app VMs just to provide security isolation in their own little silos. The resources used by these virtual operating systems alone sent us back to square one.      

Currently we are on the crest of the Container Virtualization wave. All the big IT Bellwethers  like Google and Facebook are heavily invested in this technology. Companies like Docker have taken linux containers and built a whole app store around it called the Docker Hub. Imagine setting up only the base operating system and kernel, and then setting up a separate file system for an app that has all the binaries and libraries required, along with its own networking stack. From the outside, each container runs and feels like a separate operating system, without the extra fat of a virtual operating system.  Since containers are able to isolate the apps, which all run on the same host operating system,we can have even more denser architectures.

Now all that's left is for the big enterprise vendors to simplify the creation and management of container technology. Google's Kubernetes and Apache Mesos are steps in the right direction, but they feel like bolt-on tools, that could probably be helpful in huge datacenters, not in midsize ones of about a few thousand containers.

What I feel is that we need a system tailor-made for just containers, and their underlying infrastructure. It would probably be proprietary kit, with its own hardware and host OS, distributed storage similar to HDFS, Docker or some other container technology and management software that brings them all together.

Containers and this management layer should be able to perform at least the following tasks using a GUI for it to become really relevant: -

• Spin up a container using a new, builtin, or Docker Hub sourced file system.
• Provide an option to make the containers persistent.
• Configure storage required for each container.
• Limit the CPU and setup dynamic memory for each container.
• Setup networking for each container.
• Setup a cluster of host machines.
• Configure Live Migration and High Availability for containers.
• Setup Metro Storage Availability and Container Live Migration for DR purposes.
• Setup snapshots and backups for containers and their storage.

Will Nutanix's Project Acropolis or Ubuntu's LXD step up to the plate? Is Microsoft's Windows Server 2016 taking containers mainstream? Or will there be a new player who can bring all the powerful open source container technologies together? Exciting times lie ahead for containers, exciting times indeed.