AWS - Infrastructure as Code

This video shows that companies are not just heavily relying on CloudFormation, but already building their own layer upon it. I would call this layer as the "automation of the automation".

This makes me ask myself a couple of questions:

Q: what happens when you have a huge number of clients's infrastructure to maintain?

A: Well, 10 clients, each with 10 EC2 (TEST, STG, PROD) + all other services (S3, VPC, Subnets, Security Groups and so forth).

Then we have 10 Cloudformation scripts to keep up to date, at least 100 AMIs IDs to associate, 200 or more security groups, subnets, VPCs and an infinite number of resources. It gets to a point where your automation is not enough anymore, therefore another layer of automation is necessary.

In the video above it is possible to see the example of Simple, a banking company which relies on AWS to run their environment and at the same time comply with very strict data privacy regulations.

They have built their own Python application, which basically interprets the Cloudformation scripts and looks for patterns on it (or the lack of), such as Security Groups and Resources naming convention, User Data scripts for the pre-baked (or not) AMIs and, finally, if all the required Cloudformation resources are there and correctly declared.

Q: Ok, I have another layer, the automation of automation. But this is for building new environments, or at least to force them to stay according to the original structure in the Cloudformation script. What about when I want to expand an existing Stack?

A: Another example demoed in the video, was a Javascript API which streamlined the task of updating Stacks in real time, with just a few command lines + a consolidated (very simple though) monitoring web application which showed the number of EC2 instances currently running. It showed whether there were enough resources, any health state alerts etc.

The most impressive is that the Javascript API was a nexus between a code versioning service (very likely GitHub) and the Stack running in the Cloud. Therefore, every change made in the Cloudformation, was first committed to the code repository to keep track of the changes between versions, and then the code would be committed and deployed into the Stack to apply all the changes.

In the video you can see that with basically 3, 4 lines of very simple Javascript coding, the guy:

1) Updates the code repository with a new version of the deployment script

2) Changes the monitoring tool HTML files to fit correctly to the screen he was using to present the web site

3) Commit the changes and really execute the code to expand the Stack and literally double the horsepower with the double of EC2 instances to keep up with the heavy traffic

In a Nutshell:

Companies are creating another layer which runs upon Cloudformation, to ensure code structure, naming conventions and that all necessary resources are being declared and also automating the deployment of this code, connecting code repositories to keep track of version changes, test builds, and finally automating the deployment of the new committed code with just a few Javascript lines.

AWS ressembles me of SharePoint. It is a fantastic platform, with a number of possibilities, but that at some point might be so strongly decoupled that people will have difficulties in understanding the relations between all its resources and services.

That's why in SharePoint we have Hoozin, Sitrion (former Newsgator), Yammer and so many other solution running upon it. To streamline it. To make it simple.

That's why we are now starting to see companies creating Management Consolidation layers upon AWS. Centralizing all the control over resources and services in the minimum virtual space possible.

The last question would therefore be, will AWS keep working on its own API and resources/services decoupling and let partners/customers be creative, developing their own automation of automation tools?

Or will (or even, ARE) them working on the next step of IaaS management consolidation?

Well, one thing I already know: That's too much philosophy for today. Bye!