A New Pipeline - Microsoft OpenHack DevOps

"YES! IT FAILED!" - Chris Niesel

>>

I want to preface this article by saying thank you to my team: Anthony Nguyen, Edwin Van Damme, Zi Jian ( Jack ) Zhang, Chris Niesel and Aruna Kumaraswamy for creating such an incredible work environment and for answering any questions I had along the way. I also want to thank Raj Nemani, Hailey Wu and Xuhong Liu for their incredible coaching and assistance throughout the event. These coaches made us feel welcome and promoted a fun environment while also consistently challenging us to think through technically demanding questions. Lastly, I want to thank Maria Pronin and Microsoft for organizing such a great learning experience for our entire team.

I also want to thank Margaret Brewster and Attilio Ravani for supporting me during this co-op term and pushing me to better myself. It's because of them and the work they do that I push myself to try new things and learn as much as I physically can. I strive to be as detail-oriented and effective as they are in the work they do.

>>

One week ago, an opportunity arose for me to participate in one of Microsoft's many OpenHack Events with other members of the Payments Innovation team as well as other peers from TD. This OpenHack topic: DevOps.

I knew little about DevOps going into this event, and even less about what I should expect from the event as a whole. This wasn't a Hackathon, we were given set challenges that we needed to walk through and complete; however, it also wasn't a workshop as the level of assistance from the coaches made it very clear that this wasn't a simple "follow the steps and get a result" type of event. You didn't get to hold any hands while trying to solve these challenges - we had to rely on our own capabilities and our ability to learn on the fly.

Each challenge you completed unlocked an increasingly vague challenge compared to the last, some with topics and concepts I've never heard before or seen in practice. The idea that we had three days to complete eight challenges at first seemed like a walk in the park. Eventually, that park slowly evolved into a fiery path that tested our patience, reminding us that we are novices using this technology - hitting us with a bombardment of obscure error messages on a framework we are only beginning to grasp.

I loved every minute of it, and I would happily do it again.

What is DevOps? What is a DevOps Engineer?

According to Microsoft, DevOps can be described as a unification of people and process in order to push our products out to clients faster and more efficiently, as outlined by Sam Guckenheimer:

DevOps is the union of people, process, and products to enable continuous delivery of value to our end users. The contraction of “Dev” and “Ops” refers to replacing siloed Development and Operations to create multidisciplinary teams that now work together with shared and efficient practices and tools. - Sam Guckenheimer, Microsoft

Based on this information, DevOps can be described as not a person or a role in a company, but a culture or idea that should be shared by teams within a corporation to facilitate fast and easy production builds. DevOps is the idea of continual learning, integration and improvement with the end goals being:

• Minimal Downtime of Existing Products
• Fast Deployment of Updates
• Quick Detection of Errors and Bugs

With this in mind, I asked myself shortly after what does a "DevOps Engineer" do? It's the job of the entire team to ensure that a deployment to production goes without a hitch, so what's the purpose of a DevOps Engineer?

I would compare the job of DevOps Engineer to that of a Scrum Master. Scrum/Agile is a concept developed in order to create a stable workflow that promotes increased quality of deliverable, iterative adaptations and easier product management around. With this in mind, the job of a Scrum Master is to ensure that a team stays true to these values, following practices set in place to promote the ideas stated above. In my mind, this is extremely similar to a DevOps Engineer - someone who ensures a team stays true to values of fast deployment with minimal bugs and minimal overall downtime to create a more efficient pipeline to produce production ready deployments. In both cases, each role may make changes to their respective pipelines to facilitate a better, more efficient environment for both the team and the end user.

This, however, is my own takeaway and my own opinion based on what I understand what I've seen around me. I would love to hear what people who work more closely with DevOps Engineers have to say about this!

Background

The background of this challenge update existing APIs previously created by a fictional company to meet any additional business requirements while also keeping the existing application functional with minimal downtime, as described below:

| Your team is the IT team of a fictitious insurance company. The company is offering their customers the ability to evaluate their driving skills. A mobile application collects data from the car and sends them to a set of APIs which evaluate the trip that has just been completed. Your customers can connect to a web application that uses the same APIs to review their trips and their driving score. Any downtime of the APIs would greatly impact your business.

The mobile application was a Kubernetes application that consisted of four APIs of different languages ( Java, NodeJs, .NET and Go) and a website that customers use. If even one part of this application went down, it would be considered downtime and we would be punished for it with a lesser score.

This challenge consisted of eight parts of increasing complexity and vagueness:

1. Establishing Our Plan
2. Implementing Continuous Integration ( CI )
3. Implementing Unit Testing
4. Implementing Continuous Deployment
5. Implementing a Basic Blue/Green Deployment Strategy
6. Implementing a Monitoring Solution for the "MyDriving" API
7. Implement Integration Testing, Code Coverage and Load Testing
8. Implementing Phased Rollout with Rollback

Challenges evolved from a multi-paragraphed description on what coaches were looking for to a vague description that equated to "Do what the title says, and don't break anything else along the way". It became very apparent very quickly that this was not a hand-holding event. We had to do our own research based on nothing more than a description of a task that needs to be carried out - experimenting new tactics and trying out new things while tip-toeing around the delicate pipelines we've already developed.

The First Tasks

The first tasks were very straightforward as I understood what they were asking and knew how to implement them with relative ease. In a previous Coop, I had plenty of experience modifying and improving upon the deployment and testing pipelines in place through Jenkins, so I have had exposure to the concepts of continuous integration, deployment and unit testing.

Continuous Integration (CI) is the concept of updating a product and having it rigorously tested and validated to ensure that no errors exist. This idea promotes automated testing of code that is merged into a release branch of the repository, such that if an error does appear the changes made to the release branch are reverted and the pipeline is stopped here.

Continuous Deployment (CD) is the concept of automatically releasing new changes to clients and customers without human intervention and minimal downtime. With this step, there is no such thing as a "release day" for products, as they are pushed automatically if all tests and stages of a build pipeline succeed.

Unit Testing is the practice of breaking down your code into pieces and individually testing each piece of code with a series of tests. The way this is done is different for each language, however, the concept stays the same. The practice of unit testing inside of a pipeline alongside CI and CD allows for continuous testing of a code-base without any human intervention.

The Final Tasks

The final tasks dealt with concepts I originally had no grasp of, nor heard of. These tasks dealt with different testing methods, monitoring and phased deployment. Knowing that we were on a time constraint, we divided up tasks to different members of the team - each task requiring sufficient research and understanding of the topic to move forward.

Blue/Green Deployment is a deployment strategy that allows for instant rollout while avoiding any versioning issues. The idea behind blue/green is that two versions are deployed simultaneously. At any one time, only one of these environments is live ( for this example, this will be green ) and released which receives all incoming traffic while the other environment is idle ( blue ). When new software is ready to be tested, it is placed inside of the idle environment ( blue ). Once all testing is done, all traffic will then be routed to the previously idle environment ( blue ), making it the live environment. The previously live environment ( green ) is now idle, and will stay idle until a new update is released. The point of following this process is that, if there is any issue in a currently live environment, you can easily switch all traffic from the live environment to the environment that has an older set of code.

Azure Monitoring Solutions allows teams that use Azure Cloud Services to create dashboards that monitor performance, downtime and overall environment health. It creates an easy visualization of a release environment to allow for quick analysis.

Integration Testing is the testing practice that tests all app components to ensure that all supporting infrastructure work as intended. This type of test is an all-encompassing functionality test.

Load Testing is the testing process of hitting an endpoint with multiple requests within an allotted time and measuring its response. Putting demand on an application like this tests an app's ability to respond to a large number of requests in a small amount of time.

Phased Rollout is the practice where software updates are incrementally implemented and released to a system. Having phased rollout alongside some sort of monitoring solution allows us to gradually switch users to a new version of an application while monitoring the overall health of the app. A practice such as this is important as it allows for a more gradual and safe deployment of a system, compared to that of a big bang deployment that deploys the entire system in one go. This could lead to problems for developers as well as end-users. If an issue pops up that slipped through QA and the deployment pipeline, it is now a more widespread issue that can affect all end-users. You can generally pair phased rollout with blue/green deployment! Instead of routing all traffic to the successfully tested idle environment instantly, a slow progression of traffic can be sent instead.

These final tasks took a lot of effort to complete. These tasks came with little help, vague descriptions that required us to do our own digging and research into these topics.

Before this event, I had never used Kubernetes or Azure DevOps tool, so this was completely new to me. I spent a good bit of time asking Anthony for Kubernetes help, watching tutorials and reading articles on Azure DevOps learning how we can implement these new features into the pipeline. My peers seemed to have no issue with me asking many dumb questions about Kubernetes as they understood that I was a novice at using this software. I was dedicated to understanding Kubernetes, from the fundamentals to applications to actually putting it into practice. We stayed up way too late, we drew out what we want as an end goal for our pipeline and we put it into practice. The last few tasks really challenged myself and other members in my team in a constructive and creative way.

The Team

In the end, our team made it through - completing all eight challenges and coming together to create a proper continuous integration and deployment pipeline ready to create production level builds automatically.

This task tested us technically, but it also tested us in a team-building sense. I full-heartedly believe that one key to our success was the team dynamic that we had built near the beginning of this challenge. The combination of communication among team members combined with delegation of different roles was extremely effective. In the early stages of the challenge, we gave everyone an API pipeline to update and complete for each task. We made a conscious effort to continuously update the rest of the team on our current progress, our findings and our failures. We discussed new tactics and methods while simultaneously testing them out ourselves, making sure we were continuously updating and integrating new changes into our pipelines.

We were a pipeline, each one of us had a job to complete in this pipeline so that we could produce a final product, updating our original product for the task at hand. We worked in parallel, telling each-other is something worked or if something didn't. In the latter case, we acted accordingly and tried again, tested, and integrated until it worked.

After the first challenges, we dispersed into groups that tackled tasks 6 to 8. We developed an understanding for our problems at hand and talked it out with people in the team, making sure that everyone understood what the challenge was, what is needed to complete it and how close we were to finishing it. If someone had a question, we dropped what we were doing and assisted. These challenges tested our ability to work individually and logically but also tested our communication and teamwork as a whole. Everyone wanted to work together, everyone was patient and everyone was on an equal ground. The environment the team created really showed how effective a friendly work environment of people with similar goals can be.

Our team all had the same mentality: we came here to learn, we came here to have fun and we came here to win. While there wasn't any "winning" as no official winner was announced, we won in our eyes. We did something no other team did at this event - completed all eight challenges with Azure DevOps tool. Shoutout to the other TD team that completed all eight challenges through Jenkins!

Closing Remarks and Takeaways

I genuinely enjoyed the overall environment of this event: figure it out! We had amazing coaches with Raj, Hailey and Xuhong, who helped us along the way when we had more in-depth technical questions; however, the general environment was testing our logic and problem solving abilities with minimal assistance along the way.

This event gave me a lot of knowledge into the DevOps world. Because of this event, I've built up even more skills revolving around DevOps alongside my own thoughts and opinions on the practice. Do I think I know everything there is to know about DevOps? Absolutely not. I'm still just a novice, there is still plenty for me to learn and experience - I definitely only got a snapshot of the DevOps world in my three days at OpenHacks. I cannot wait for an opportunity to arise in the future to allow me to continue building my skills around a team that's just as effective as the one I had the pleasure of working with here at OpenHacks. With my next co-op term around the corner starting only in May/June, I'll definitely be looking into working with DevOps and Cloud Computing!