How to model your DevOps experience?

On a Red Hat Consulting assignment; I was interacting with the client's solution architect, DevOps engineer and microservices developer. Collectively we wanted to explore how “Developer Experience” and “DevOps Experience” can be enhanced. Half an hour into the meeting; spotlight moved on to the elephant in the room - “Can this overly complex DevOps experience be simplified?”. The client had DIY Kubernetes version 1.4 cluster (yes, it’s from stone age considering Kubernetes explosive development) with Jenkins based pipeline and SVN as source code management system. Containerised applications were based on custom-built Nginx (web server for static content), Tomcat (J2EE based ISV COTS app), OpenJDK (spring-boot based apps) images.

Above sounds familiar! If you want to learn how we established the problem domain and went on to use Red Hat OpenShift to solve then go ahead with reading the series.

Do you see what I see?

It was extremely important for all to have a shared understanding of the existing DevOps practice. Typically it’s best done in a room with a developer, testers, DevOps engineer and security group members. In the virtual world; we banked on Miro frames to collaborate. We adopted a tailored down version of Event Modeling (by Adam Dymitruk) to represent the actors, systems and events. Have a look at the sample (after removing customer contextual data) below:

The brevity of the sample DevOps flow allows the readability.

Few tips that worked for me:

• Don’t rush. Always plan for buffer hours before you start.
• Instant information. Be ready with information instantly: code, config, logs etc. encourage all to share screens on demand. Participants were asked to be prepared before the workshop.
• Trust on-job people: No documentation references! The likeliness of documentation to be out-of-date is high. Moreover, the documentation doesn’t compile, run or throw errors!
• No one is an expert: No matter how silent a person participating may be; encourage them to participate as a bright suggestion may come from anywhere.
• Have a sidecar: I was accompanied by another shadowing consultant who also was the note taker, muting unwanted noise sources etc. Post-session; it’s useful to have someone to reflect on the problem domain and solutions proposed.
• Forward & Backward Replay: Client’s DevOps engineer (who contributed the least) recites flow that has been captured. This brought the unknowns on the surface with few more rounds of discussion.
• Record the session: With the client’s consent; a recorded session may lead to new learning for all participants.
• Red/Purple stickies: During the discussion; capture the constraints, issues, areas of improvement, pain points in a single colour (purple) for easy visualisation. 

Output & Outcome

1. Miro Board: An editable As-Is DevOps model that represents the latest state of the art.
2. First-cut of challenges: The Red/purple stickies capturing issues for further discussion.
3. Knowledge: Shared understanding of the process and its challenges.

Have I thought through!

DevOps practices should be shaped based on multiple criteria in the order of importance and constraints that clients may have. There are a series of questions that are brought in front of the client on different aspects of DevOps. 

As a preparation work; I use a Dart technique to trigger the conversation with clients. It’s a visual representation of DevOps facets as dart sectors with respective questionnaires on stickies. At the beginning of the conversation; the stickies are at the periphery of the dartboard.

Pick a sticky, capture the response and tag it as “MoSCoW”. Optionally; the question can also be tagged with the group who answers or is most interested in. This will help to address the most important requirement from a certain group. E.g. 

Bring in the Red/Purple stickies that were captured during the previous section “Do you see what I see” and classify.

Few tips that worked for me:

• Unanimous MoSCoW: Make sure the classification is agreed by most from the customer side.
• Encourage discussion, avoid deep dive: It’s important to have a discussion on the problem domain but should avoid solution deep dive at this point.
• Open for new questions: Be open to the constraints that the customer environment may have and requirements around that. Capture that as well.

Output & Outcome

1. Miro board: Shared understanding of different facets of DevOps.
2. Priority agreement: Clear understanding of high priority DevOps requirements.

Which one to address first!

Like in the game of Dart; you want to prioritise the “Bullseye” and “Triple” zone which; for us; translates to “MUST have” and “Should have”. Below were the top requirements.

1. Pipeline in Jenkins was created with bespoke tasks. The team wanted to move to Pipeline as code for better manageability.
2. SVN stored code and binaries (.jar/.war) in different folders. Main trunk managed by DevOps team and branches maintained by development lead. Separate SVN for production and non-production. Traceability of code and binary against feature release was cumbersome. Binary versioning was a problem.
3. Creation and management of the application base image were isolated and not part of the standard pipeline practice. No continuous patch management (CVEs).
4. Custom application base image creation has multiple steps resulting in a big image. Steps involved were RHEL 7 base image -> run hardening script -> registration with RH Satellite -> package installation -> time zone setting -> tomcat installation -> oracle jre installation -> expose ports for tomcat and prometheus (customer had DIY old prometheus stack) -> copy configuration files for jmx exporter -> copy APM agent binary and configs -> copy tomcat libraries. Needed simplification.
5. Creation of the application image was complex. Simplification was needed. None of the below added value from a business logic perspective and was changing from environment to environment. Key steps as part of customization were - a) customise catalina.sh file (for Tomcat) which includes a script to load environment-specific properties stored in a database, copy java library from SVN as the war didn’t have the dependencies (no one knows why!) etc. b) Environment specific APM configurations like context.xml etc.
6. Application image scanning for vulnerabilities was not practised. Security group (ISG) wanted to introduce it for compliance. In addition, on-demand OpenSCAP based scanning was recommended by the ISG team.
7. Application image signing was not practised. Security group wanted to introduce it for compliance.
8. The Dev to Production pipeline was fragmented with manual triggers. The team wanted to bring “time-bound-approval” and automation.
9. High availability and resource management of Jenkins was established as issues. The team wanted to leverage the containerisation of Jenkins and eventually move to TektonCD for the enterprise.

Summary

Organizations embarking on the journey of a hybrid cloud with cloud-native apps got started with DIY Kubernetes and enthusiastic teams. Scaling the model for the entire enterprise needs much more without losing the power of Open Source. Considerations like enterprise security, storage, networking, patch management, self-service, DevOps pipeline, business-focused cloud-native apps etc. Red Hat OpenShift takes a holistic approach to DevOps experience. Following the above mechanism; issues were established for the customer’s DIY Kubernetes with standalone Jenkins flow.

In the next blog; I’ll pick how source-to-image and Red Hat Builder Images from Red Hat Container Catalog helped the client. If there are other topics of interest then do comment.

Thanks.