DevOps - Lessons from the trenches
We recently celebrated the 5 year anniversary of migrating Recovery.gov to the AWS Public Cloud as one of the first US Federal Systems to do so. The Recovery Act provided incredibly short timelines for delivering complete and secure systems. New requirements and evolving user needs had to be deployed in days and weeks not months or quarters. Reflecting back to 2009 and 2010, it is interesting to see how our development and deployment practices are now collectively called DevOps. Here are some of the lessons we learned along the way.
1. Talent Density Matters!
Having smart and talented engineers in the same building working side by side is critical. The ability to seamlessly identify, resolve and execute solutions in near real-time on a diverse range of topics including development, infrastructure, network, security, and access issues was priceless. All members of team had many years of deep operational and deployment experience with a shared commitment to fast delivery. Co-location, talent parity across disciplines, and a seasoned team were key. A key part to a successful DevOps operation is the co-location of "full-stack" capabilities for rapid resolution of issues.
2. Service Owners are Foundational Blocks
The ownership of key foundational services was established early on - web & content services, data services, security services and platform engineering services with clearly defined roles & responsibilities, SLA's, and the ability to create self-contained delivery "squads" helped in driving ownership down the ranks and broke-down some of the challenges associated with traditional project teams. Traditional project teams did not quite have the concept of a "product or service" owner with complete autonomy to design, develop and deliver their services.
3. Customer Backlog versus Technical Debt?
This is a battle many DevOps organizations struggle with on a daily basis especially when software deliveries are happening on a daily or weekly basis. The rapid pace of development leads to accumulation of technical debt. Inevitably, organizations deal with prioritizing tasks and resources between addressing Customer Backlog versus Technical Debt differently. In our case, working on Customer Backlog won 99% of the time! An interesting by product of accumulating Technical Debt however was innovation! The cliche "necessity is the mother of invention" is very true. The technical team had to constantly think of new ways of solving old problems (or die trying). As an example, we accumulated a lot of technical debt in the area of services management automation within an IaaS environment. By moving to a PaaS model, some of the traditional management tasks went away thereby dramatically reducing Technical Debt.
In 2015, clearly the technology landscape has evolved tremendously -Microservices, Containers, PaaS-services e.g. AWS Lambda, Client-oriented architectures with JavaScript frameworks and focus on automating the development, testing and deployment pipeline are important foundational capabilities. As more organizations are adopting DevOps, how are you dealing with the rapid pace of delivery? Do share your war stories and describe how you are delivering innovation faster!
