Chaos Testing for Kubernetes Resilience

☸️ Kubernetes v1.36.0 is out — and this is a major release with real changes you should pay attention to. This isn’t just “new features” — it’s also about what’s changing or going away. ⚠️ One important change: The long-deprecated gitRepo volume has finally been removed. 👉 If you still rely on it, workloads will break after upgrading — and you’ll need to migrate to alternatives like init containers or external sync tools. (Kubernetes) ✨ On the feature side: • Mutating Admission Policies → now stable (less reliance on webhooks) • User Namespaces → improved isolation for containers • Dynamic Resource Allocation (DRA) → continues evolving for advanced workloads These are the kinds of changes that impact: • security posture • workload isolation • cluster extensibility Kubernetes v1.36 is a good reminder: 👉 Major releases are not just about what’s new — they’re about what might break and what needs migration. At Relnx, we track these changes so you can quickly understand: ✅ breaking changes ✅ new capabilities ✅ upgrade impact 🔎 Full release breakdown: https://lnkd.in/g3PEecwm For platform teams — What’s your first step when a new Kubernetes version drops: 👉 Check features 👉 Or check breaking changes first? #Kubernetes #CloudNative #SRE #DevOps #PlatformEngineering #Relnx

Kubernetes didn’t make our system faster. It made our mistakes less dangerous. Before orchestration, a small configuration issue could mean: ->Downtime ->Manual restarts -> Panic debugging -> Emergency calls With Kubernetes, failures still happen. Containers crash. Nodes go down. Deployments misbehave. But the system doesn’t freeze. It reacts. It replaces. It reroutes. It retries. That shift changed how I build software. Now I don’t just ask: “Does this work?” I ask: “What happens when it breaks?” Because in distributed systems, things will break. The goal isn’t perfection. It’s controlled recovery. That’s what modern infrastructure taught me. #Kubernetes #CloudNative #Resilience #SoftwareEngineering #Microservices #DevOps #EngineeringMindset #ScalableSystems

One thing I’ve noticed working with Kubernetes: Most problems aren’t caused by Kubernetes… They’re caused by inconsistent usage of it. Same cluster. Same tools. #Different teams → #different standards → #unpredictable outcomes. So instead of adding more documentation, we focused on enforcing consistency. What changed when I introduced structured policies: #No more missing resource limits #No more “temporary” insecure configs reaching production #Namespaces come with quotas and network policies by default #Every workload has traceable ownership (labels enforced) And the important part: #Developers didn’t have to remember any of this. The approach was simple but intentional: #Enforce what must not break (validate) #Auto-fix what’s commonly missed (mutate) #Auto-create what should always exist (generate) You don’t scale Kubernetes by adding more control. You scale it by removing decisions from humans and putting them into the platform. That’s where governance starts to feel like enablement, not restriction. #Kubernetes #Kyverno #PlatformEngineering #DevOps #SRE

Platform Engineering 102 : Source code by itself does not make a company $$. It's only when it is deployed, and accessed by the customers, does a company start making $$. So, the whole point of a technology platform is to optimize this process, every step of the way - basically make it as seamless as possible to move code from the repo -> compute. The faster this process is, the more $$ the company is poised to make. #flowoptimization #platformengineering #devops

I revisited one of the most misunderstood (yet critical) concepts in Kubernetes: Liveness vs Readiness Probes. I realized something important — most of us use probes, but don’t fully understand how they behave over time. Here’s the clarity I gained : Everything in Kubernetes is a loop Both liveness and readiness probes run continuously, independently, and in parallel. Readiness Probe = Traffic Control - Starts checking after its own initial delay - If it fails → Pod is removed from Service endpoints - If it recovers → traffic resumes - No restart involved Liveness Probe = Self-Healing - Also runs on its own schedule - If it fails repeatedly → container is restarted - Keeps your app from staying in a broken state. Key Insight: A Pod can be: - Running but NOT ready (no traffic) - Running and ready (serving traffic) - Restarting (liveness failure) Common Misconception: “Pod removed” ≠ Pod deleted It simply means → no traffic is routed, but the container is still running and being monitored. After Restart: Same Pod, same IP Probes reset Readiness starts from scratch again Big Lesson Kubernetes isn’t just orchestrating containers — it’s continuously observing, deciding, and correcting state in loops. #Kubernetes #DevOps #CloudNative #SRE #KubernetesProbes #LivenessProbe #ReadinessProbe #Microservices #PlatformEngineering #CloudComputing #LearningInPublic

The most dangerous Kubernetes cluster is not the one that is down. It is the one that is still accepting change. Outages are obvious. Degraded clusters are expensive. This week I ran into a cluster that was still technically alive: - disk pressure on a worker - evicted pods accumulating - monitoring disruption - stale pod states everywhere It was not down. That was exactly the problem. Because the highest-cost mistakes often happen in the gap between availability and deployability. - The platform still responds. - The rollout still starts. - The incident just gets bigger. That is the trap: teams keep shipping into instability because the cluster still looks available. But “up” is not the same as: - safe - stable - trustworthy for change A lot of real damage happens in the gap between “still up” and “safe to change.” Not during the outage. Right before it. When the system still looks alive enough to invite the next bad decision. That is one of the reasons I am building a Deploy Confidence Service for Kubernetes: to help teams decide whether a cluster should accept fresh change at all. Because platform engineering should do more than expose failure. It should stop degraded environments from becoming release-driven incidents. Have you seen a platform stay “up” long enough to trick a team into making the next deployment the real outage? #Kubernetes #SRE #DevOps #CloudNative

Friday Kubernetes thoughts: Kubernetes probes are not housekeeping. They are operational policy. The takeaway is simple: a lot of reliability work starts with defining “healthy” properly. Kubernetes 101 startupProbe: are you done starting? readinessProbe: can you do useful work now? livenessProbe: are you broken badly enough to restart? The problem is that many systems answer the wrong question. A process can be alive and still be stuck. An API can return 200 and still be degraded. A dependency can be briefly unavailable, and a restart can make things worse, not better. So improving probes is not about adding more checks. It is about making them more honest. startup should protect boot readiness should protect availability liveness should protect recovery If a probe checks too little, it gives false confidence. If it checks too much, it creates restart loops. If it checks the wrong thing, it amplifies incidents. Good observability helps here. It improves stability, gives teams clearer signals, and builds trust in the services they run. Over time, that also builds trust in the company behind those services. #Kubernetes #SRE #PlatformEngineering #Observability #Reliability #DevOps

When systems grow, complexity grows with them. Observability isn’t optional anymore, it’s how modern teams stay ahead of failures, reduce downtime, and build reliable distributed systems. Because you can’t fix what you can’t see. #Observability #DistributedSystems #DevOps #SRE #SystemDesign

If you can’t see what your system is doing, you can’t support it. Lack of observability is one of the most common issues in production systems. By the time something breaks, there’s no clear way to understand why. Visibility matters more than perfection. #softwareengineering #observability #devops

Yesterday, we shared how a SaaS team was struggling with broken deployments. Here’s what we changed. Not tools. Not people. The system. We simplified the pipeline into 5 clear stages: Commit Build Test Deploy Monitor Sounds basic. But the difference was in how each stage was controlled. Every stage had a clear validation. If something failed, it stopped there. No silent failures. No surprises in production. We made environments identical. What worked in staging worked in production. We added real testing. Not just unit tests but checks that reflected actual usage. And most importantly, we added visibility during deployment. So instead of reacting to failures… the team started preventing them. The result? Deployments became predictable. Failures dropped. Confidence went up. Most pipelines don’t fail because they’re complex. They fail because they’re unclear. Clarity fixes more than complexity ever will. How is your deployment pipeline structured today? #DevOps #CICD #PlatformEngineering #CloudNative #Neoscript