Kubernetes Operator for Cluster Monitoring and Alert Fatigue Reduction

I am glad to share a recent project I co-developed from scratch: j-kube-watch. It is a custom Kubernetes Operator designed to streamline cluster monitoring and eliminate alert fatigue. When monitoring Kubernetes environments, repetitive warnings like a CrashLoopBackOff or a failing probe can easily flood notification channels. To solve this, we built an operator that actively watches Pod lifecycle events and routes them intelligently. Key technical aspects of the project include: ● Built with Java 21 and the Fabric8 client, utilizing Virtual Threads for lightweight, concurrent event processing. ● An intelligent deduplication engine using Caffeine cache to suppress alert storms, sending grouped summaries instead of redundant notifications. ● Fully native configuration using Custom Resource Definitions (CRDs) for routing alerts to external channels like email. ● Packaged completely with Helm to handle deployments, RBAC rules, and network policies. This project was fully co-developed from start to finish in collaboration with Mostafa Mahmoud. You can find the full source code, architecture flow, and Helm charts on GitHub here: https://lnkd.in/dTsukBRK Feedback and code reviews are always welcome. #Kubernetes #DevOps #Java #CloudNative #PlatformEngineering #Helm #Automation #ITI

We replaced a dozen scattered Python scripts with a single n8n instance. Every engineering team has them: a collection of forgotten cron jobs and small scripts holding critical processes together. They run on someone's old machine or a neglected EC2 instance. They lack monitoring, fail silently, and have API keys hardcoded from years ago. We started consolidating these into self-hosted n8n workflows. The first target was our SRE alert-to-ticket pipeline. Previously, a script would parse an incoming webhook, but it was brittle and required code changes for any logic adjustments. The new workflow is visual and resilient. A Prometheus alert triggers the workflow. n8n then enriches the payload with data from our internal asset database, creates a structured Jira ticket with the right labels, and posts a summary to a dedicated Slack channel. It's not about replacing complex code. It's about orchestrating API-driven tasks in a system that provides visibility, credential management, and error handling out of the box. The workflow itself is just JSON, so we can even check it into git. What's the most repetitive task you've automated away from your team's plate? #DevOps #Automation #n8n #SRE

Most developers know how logging works in a local Spring Boot setup. But things get interesting when your application runs inside containers on Kubernetes. Here’s a simple breakdown of how Spring Boot logs become visible in HOLMES logs in a containerized environment: 1. A Spring Boot application (using Logback/Log4j) writes logs to standard output (console). 2. When the app is containerized using Docker, these logs are captured by the container runtime. 3. In a Kubernetes cluster, the container logs are managed by the node (via containerd or Docker runtime). 4. Kubernetes exposes these logs using: * kubectl logs * Logging agents (like Fluentd/Fluent Bit) 5. These agents forward logs to centralized systems like HOLMES (or ELK/Splunk). 6. HOLMES aggregates, indexes, and makes logs searchable for debugging and monitoring. Key takeaway: If your logs are not visible in HOLMES, don’t blame the tool immediately. Check the pipeline: Application → Container → Kubernetes → Log Agent → HOLMES One break in this chain = no visibility. Understanding this flow is critical when debugging production issues, especially in distributed systems. #SpringBoot #Kubernetes #Docker #Logging #Microservices #DevOps #ProductionDebugging #LovelyOnTech

In 2016, I mass-produced microservices like a factory. By 2017, I was debugging them at 2 AM on a Saturday. Here's what 14 years taught me about microservices the hard way: We had a monolith that "needed" to be broken up. So I split it into 23 microservices in 4 months. Result? - Deployment time went from 30 min to 3 hours - Debugging a single request meant checking 7 services - Team velocity dropped 40% - Every "simple" feature needed changes in 5+ repos The problem? I created a "distributed monolith." All the pain of microservices. None of the benefits. What I learned after fixing it: 1. Start with a well-structured monolith. Split only when you MUST. 2. Each service must own its data. Shared databases = shared pain. 3. If 2 services always deploy together, they should be 1 service. 4. Invest in observability BEFORE splitting. Tracing, logging, monitoring. 5. Domain boundaries matter more than tech stack choices. We consolidated 23 services down to 8. Deployment time dropped to 15 minutes. Team happiness went through the roof. The best architecture is the one your team can actually maintain. Have you ever over-engineered a system? What happened? #systemdesign #microservices #softwarearchitecture #java #programming

CI/CD isn’t just a "best practice" — it’s the last line of defense. 🛡️ A robust automated workflow to keep production rock-solid: 🔄 Every PR Triggers: • Build Check & Lint (Spotless) • Unit Tests (JUnit/Mockito) • Quality Gate (SonarQube) ⚖️ Merge Rules: • Min. 1 Peer Approval • 100% Status Checks Pass • No Direct Push to main 🚀 Deployment: • Auto-deploy to Staging • Manual Approval for Production 🚨 3 Non-Negotiable Rules: 1. No Build, No Merge. No exceptions for "urgent" fixes. 2. Fix Tests, Don't Skip. Flaky tests are technical debt. 3. The 5-Min Rule. If it's slow, optimize it immediately. A deployment pipeline is a safety net. Don't let it have holes. What’s your pipeline’s average runtime? 👇 #CICD #GitHubActions #SpringBoot #SoftwareEngineering #Java #AWS

Stop Shipping "Heavy" Docker Images: The Power of Multi-Stage Builds & Distroless Are your Docker images bloated with unnecessary tools? If you are still shipping compilers, build logs, and shell utilities to production, you're leaving performance and security on the table. In modern DevOps, smaller is better. The Problem: Traditional Builds A standard Dockerfile often includes everything: the OS, build tools (Go, Maven, GCC), source code, and the final app. Result: A 1GB image for a 5MB application. Risk: High attack surface (shells like bash and package managers like apt can be exploited by hackers). The Solution: Multi-Stage Builds + Distroless By splitting your Dockerfile into two stages, you can drastically optimize your workflow: Stage 1 (Builder): Use a heavy image to compile your code. Stage 2 (Runtime): Use a Distroless image to run it. What is Distroless? It’s a minimalistic image that contains only your application and its runtime dependencies. No shell, no package manager, no extra bloat. Why this matters: Massive Size Reduction: Go from 800MB+ images down to <20MB. Hardened Security: By removing the shell (/bin/sh), you eliminate the most common way hackers execute malicious commands in a container. Faster Scaling: Smaller images pull from registries faster, making your CI/CD pipelines and Kubernetes deployments lightning quick. Pro-Tip: If you are building statically linked binaries (like in Go or Rust), try using FROM scratch. It's the ultimate zero-byte base image! Stop shipping your "builder" tools to production. Your infrastructure—and your security team—will thank you. #Docker #DevOps #ContainerSecurity #CloudNative #Kubernetes #SoftwareEngineering #Distroless #Microservices

Things are getting more interesting on the next stage of my lab project - CI pipeline build workloads are migrated from the Jenkins controller to ephemeral pods running on k3s cluster. In the last update we added a SonarQube integration to our Jenkins CI pipeline for a small C++ service. This time we moved pipeline execution to dynamic agents running in Kubernetes (k3s): • Jenkins controller now orchestrates pipelines only • Build workloads run on ephemeral k3s pods • Custom Docker image prepared for C++ builds (cmake, gcc, python, etc.) • Agents are provisioned on demand per pipeline run • SonarQube integration works from within k8s agents • Full pipeline (build → test → quality → package) now runs outside the controller What this changes: Instead of relying on a single VM, the system can now scale builds dynamically and keep the CI controller lightweight and stateless — which is much closer to how modern CI/CD platforms operate. Also, a lot of “invisible” details showed up during this step: agent startup behavior, Jenkins ↔ Kubernetes connectivity, container entrypoints, authentication flows, and infrastructure reproducibility. Each rebuild made the setup more deterministic and better documented. Repository (current state): 👉 https://lnkd.in/ezKib47U Next step: extend this with multi-node execution and explore autoscaling patterns for Jenkins agents 🚀 #devops #jenkins #kubernetes #cplusplus #cicd

Image Build & Promotion Pipeline with GitHub Actions In actual projects, You do not just build an image and push it to a registry. The image needs to pass through several quality and security gates first. We have a hands-on guide that walks you through a complete, production-grade Docker image build and promotion pipeline using GitHub Actions. Here is what is covered in this blog. - Automating a Java application build and Dockerization with GitHub Actions - Managing container registry credentials securely with GitHub Secrets - Promoting images through dev, stage and prod registries - Signing the final container image using Cosign - Build caching, image tagging strategy, registry architecture, and more.. 𝗗𝗲𝘁𝗮𝗶𝗹𝗲𝗱 𝗕𝗹𝗼𝗴: https://lnkd.in/gWWWEMTV Bookmark it, try the pipeline in your own repo, and let us know what you would add or change. #devops #practicaldevops #docker

It's impressive when tools can recognize complexity and offer relevant solutions, like suggesting a GitLab CI runner setup. Imagine going from simply building an app to managing a full CI/CD pipeline, including deployment, dependencies like Python and Postgres, and even choosing a web server. This goes beyond basic development, touching on maintainability and automated processes. The next step involves integrating security testing (SAST/DAST) into the pipeline to proactively catch issues. Check out more insights at https://lnkd.in/gjXm7xzG #CI/CD #DevOps #SoftwareDevelopment #CloudComputing #Automation

Spent 2 hours debugging a broken API. Turns out, I forgot to log the request body. I checked the response, traced the code, even asked a colleague. Nothing made sense. The server was throwing an error, but the logs didn’t show the actual payload. I added a log statement. The problem vanished. The request body was malformed. Log everything. Even the 'obvious' parts. A missing log line hides 80% of the problem. What’s the one thing you forgot to log that cost you time? #Debugging #DevOps #SoftwareEngineering