Agility in DevOps – Impediments & Viability

If you are in the software industry and have not heard of the term Agile, there is a good chance you have been living under a rock. What is Agile? It isn’t stand-ups. It isn’t retros. It isn’t Jira tickets. Agile is a thought process. It is principles and values that govern your working style. They are tried and tested philosophies which cater to today’s world of fast moving requirements. You need to be ready to upscale your deliveries if you want to cater to your client’s needs and make it big in the market.  

And what is DevOps? It stands for Development and Operations. It supports agile ways of working in multiple development teams so as to be able to bring out code faster in a continuous improvement and continuous delivery fashion. CI/CD is a core part of DevOps. It would not come as a surprise if this is the one part that DevOps is known for. It is hard to imagine any style of agile practices without CI/CD. DevOps is already an integral part of Agile. 

Agile is governed by a few principles as stated in the Agile manifesto. 

1.   Satisfying customers through early and continuous delivery of valuable work.

2.   Breaking big work down into smaller tasks that can be completed quickly.

3.   Recognizing that the best work emerges from self-organized teams.

4.   Providing motivated individuals with the environment and support they need and trusting them to get the job done.

5.   K.I.S.S - Keep It Simple, Silly ! Being ruthless about cutting work that does not add value.

6.   Maintaining a sustainable pace for completed work.

7.   Welcoming changing requirements, even late in a project.

8.   Assembling the project team and business owners on a daily basis throughout the project.

9.   Having the team reflect at regular intervals on how to become more effective, then tuning and adjusting behavior accordingly.

10.Measuring progress by the amount of completed work.

11.Continually seeking technical excellence.

12.Harnessing change for a competitive advantage.

There are a few process frameworks that can help follow these. Scrum being one where list of tasks are captured in the backlog and the teams implement based on capacity and dependencies. Extreme programming being the other. Scrum ceremonies such as daily stand-up, sprint planning meetings, backlog refinement, sprint reviews, sprint retros and a lot more behave as tools that are put in place to ensure smooth functioning. These are tried and tested ways which have proven to bring out results in line with Agile principles.

As a development team, we can quite easily imagine how we can use each of these activities in our everyday work life to bring out results faster. But are these frameworks suitable for a team working on DevOps? Can we use the defined ceremonies of scrum, Kanban or XP for the functioning within a DevOps team?

In the project I worked in last year, I was a part of the DevOps team. The project was about migrating a prominent e-commerce giant to google cloud. I will not be discussing the challenges of migrating or re-architecting a live site. Conceivably that is a blog of its own. The entire project across all geographies followed scrum methodologies. 

The DevOps team started with Kanban. Kanban is when you list tasks and you keep working on them on the basis of the priority and the bandwidth. It would have been a perfect style of implementing had we been a team that was providing only support. But we were more than that. We were THE team, building the infrastructure from scratch. From nothing to everything. The foundations to the pillars to the ceilings and of course the wirings. And we had to build this along the application teams that had started their work as well.

Kanban started throwing some curve balls at us during our implementation phase. To combat that, we moved to Scrum in alignment with the other application teams hoping that we can plan our tasks better so as to help them better. But we did face some larger issues. By then now with committed deadlines, the pressures were higher. 

1.   The backlog isn’t always defined before the start of the sprint. 

• A lot of tasks that have to be catered are the support task requests that come in from the other teams. These usually come in at the time, where more often than not it is already called out as a blocker. 
• A lot of high priority stories will be added to the sprint even during the sprint. The sprint velocity is not a fixed number. 

2.   Raw velocity could not be predicted with a close accuracy

• With the team also learning newer solutions and technologies as they implement, the range of velocity between iterations was comparatively larger. 

3.   In a micro-service architecture, even a small change has to be replicated in a lot of pipelines or scripts. This is monotonous work but yet has to be accounted for in burn up charts. While the task may just be one, replicating it was the herculean task.

4.   Making changes after you have built the infrastructure is a big challenge, both in terms of development support and management approval.

• Once everything is out there and is functioning fine, calling for a downtime isn’t that easy especially if you are running a tight critical project where too many wheels are turning too fast. 

5.   Using managed services also comes with a risk of the respective services going down. 

• We used Google cloud. In a duration of 2 months near sale day, we faced 2 outages in Google. It was a worldwide phenomenon. The outage affects a lot of people using google services but the impact on the project gets streamlined via the DevOps team. Suddenly we see everyone going, ‘The site is down. Wake DevOps’
• Self-hosting or self-creating all of them would need a substantially sized team and an insane amount of research and time. That would defy the smart thought of using other services and being fast to the market. 
• There was a constant pressure on the team since there is no knowing of what would break when because of the heavy dependencies on other systems such as Google, Amazon, ElasticSearch etc…

6.   The entire infrastructure is not downright plug and play. 

• We had terraform scripts to build networks and clusters in the way we needed for our project. This meant a development time necessary for us to write the scripts and start building it from bottom up. 
• Different coding technologies are used across the project from Node, Java, Go, Python. Every design you take has to be compatible with the technologies used by the development teams. 

7.   Never a lull period. Things could go down anytime. The team has to be on alert every time. 

8.   There are no business stories defined. 

• Product owners will only have a macroscopic level of thought of what they expect from the infrastructure. Something on the lines of ‘it should be able to handle all the requests’. But converting this into the technical requirements is also a daunting task. Deep expertise in the subject is needed to understand how the business requirements can be converted into technical requirements and finally translated into ground level stories, numbers and solutioning. 

9.   Sprint showcases were perhaps the most challenging on a regular basis.

• We did not have UI’s to showcase. The initial stories that we were playing were only dozens of terraform scripts which were codes of line one after the other and did not render a 3D image of the infrastructure building up. Things like buckets, pipelines, topics, databases were words of the mind which couldn’t be translated to visuals by means other than listed bullets in the projects UI. The showcases were highly technical and the business folks who attended the showcase soon lost interest.

10.The other teams had a dependency on us to plan their stories. Kanban board in DevOps wasn’t very effective in planning the tasks within the application teams. Our development tasks were always in backlogs. 

These challenges were not all unanticipated. But some things like external systems shutting down, cyber-attacks were things that demanded us to improvise on as we went along.

In order to combat this, we started doing things slightly differently. We curated the agile practises to cater to our teams needs and the demands of the project. 

1.   We build a combination of Scrum and Kanban

• Our development tasks followed Scrum. We had an iteration goal. We had each task marked against a certain iteration based on priority and dependency. But we also left some space for adhoc/support tasks. 
• We followed Kanban thought process for all the support tasks that came from the teams i.e. we treated them on priority basis. The scrum planning helped because we could tell the teams if or not we could cater to it within the requested time and if we had to reprioritise, we did. 
• For all practical purposes, it was easier to track it in one Jira board, so we created a new epic called Adhoc support and added these support tasks into it every time we had a request. 

2.   Reprioritise on a daily basis

• Our stand up always had, ‘what is the focus of the day’. We had a larger iteration goal but since : we supported adhoc tasks ; we had a lot of dependencies on other teams which could bring our system down ; we had to close on things in order to get the wheels moving for the other teams : we had to reprioritise on a daily basis, so that the team is aware of what the priority for the day is. 
• However it is important to try not change the priority as much as possible to avoid context switching until really necessary

3.   We had sprint retros. 

• We had internal retros within the teams which helped us identify how we can work better. 
• However since a lot of our work is about the other teams, we also had external retros where we invited people from other teams to share feedback or thoughts on how things can be improved or taken forward. This ensured better collaboration. 

4.   Stand-ups were not the typical stand-up. 

• Owing to the amount of spikes and investigations we were doing as a team and the research that happened, our stand-up were more like brainstorming sessions. Going against the usual norm of a quick update, we began discussing which would take the morning discussions from 20- 45 min sometimes. We figured that way everyone knew they were on the right track for the project. 

5.   Single point of contact for support tasks

• To ensure that the task isn’t repeated, we had one person for the other teams could reach out to for adhoc tasks. Prioritisation is the key. 

6.   Good and open communication

• Transparency within the team so that everyone knows the problems the other one faces. 
• We also had constant communication with experts in partnered companies and other geographical locations to ensure we are picking the best solution. 
• We ensured that we called out the known issues in advance that will impact the development teams

7.   Keep a lookout for external outages that may impact our system. 

8.   Sprint showcase with boxes and diagrams other than the usual functionalities. 

• The business was unable to visualise the work happening on ground. To fix this we converted our work to diagrams which could better showcase the work we have done in non-technical terms so that all business owners had a chance to understand the changes happening and will be able to understand the complexity and impact. 
• For e.g.: after we built the pipelines for the entire project, we merely listed out the dozens of pipelines we cater to and what it means to build one pipeline in common man terms. This gave the audience a good understanding of what we meant by infrastructures, buckets, topics, pipelines, CI/CD and the complexity involved in bringing up a cluster. 

9.   Working without a PO.

• Since all the stories were technical in nature, we did not have a PO with the team. But we still needed all stories listed out and estimated. 
• To solve this, the PM of the team also functioned as the IM/SM who had a strong technical knowledge and was able to carefully build technical stories from the business requirements of DevOps. 

10. One the key things that helped, was putting a foot down when a downtime was mandatory. 

• When it came to upgrading systems for the betterment of the project, it became difficult to convince the other teams to pause their work for a while until the upgrade was done. 
• We collated all changes to be done, implemented as much as we could without a downtime and then assertively communicated the message about the downtime. 
• We did this keeping in mind the least impact to the other teams and kept them constantly aware of the status in order to reduce their obvious reluctance. 

The project has now stabilised at one phase but still growing to the next. The original team is now disbanded to other projects.  We had reached a long way from the start point. But even at the time when we all split, we could see there was still scope for improvement. Alongside supporting all teams in their agile ways of working, irrespective of the different projects we all have moved to, we continue to incorporate these changes within our teams and continue to deliver them faster and in high quality. After all, we are DevOps, we are Agile !!