Resolving Kubernetes Deployment Issues in Norway

Last month, our team lost 40+ hours fighting 7 Kubernetes issues that looked like “nothing.” Nobody talks about the actual stuff that slows teams down in production. It's not the outages. It's the invisible crap that doesn’t show up in dashboards or status checks. Here’s exactly what hit us and what finally worked: 1. “Deployment succeeded”… but nothing worked. Everything is green. Service unreachable. Turns out: readiness probes were wrong for apps with long cold starts. ✓ 𝐇𝐨𝐰 𝐰𝐞 𝐟𝐢𝐱𝐞𝐝 𝐢𝐭: Delayed probe kicks + timeout aligned with real boot time. Worked instantly. 2. CPU throttling was massive. Nobody knew. Why? Because average CPU usage looked fine. But container_cpu_cfs_throttled_seconds_total told a different story. ✓ 𝐇𝐨𝐰 𝐰𝐞 𝐟𝐢𝐱𝐞𝐝 𝐢𝐭: Alert on throttle %, not just usage. Our dashboards were lying to us. 3. One container restarted 98 times. Nobody caught it. CrashLoopBackOff silently chewing up restart credits. ✓ 𝐇𝐨𝐰 𝐰𝐞 𝐟𝐢𝐱𝐞𝐝 𝐢𝐭: Killed auto-retries. Set a hard alert on restart count > 3 in 10 mins. If it fails 3x, it’s not a blip. It’s broken. 4. Services were randomly failing to resolve each other. Classic: flaky DNS under load. ✓ 𝐇𝐨𝐰 𝐰𝐞 𝐟𝐢𝐱𝐞𝐝 𝐢𝐭: Switched to CoreDNS, added autoscaler, bumped memory limits. No drama since. 5. PVCs stuck in “Terminating” forever. Volumes wouldn’t detach. Finalizers misbehaving. ✓ 𝐇𝐨𝐰 𝐰𝐞 𝐟𝐢𝐱𝐞𝐝 𝐢𝐭: Manual patch jobs to nuke stuck finalizers. Now part of our cleanup cron. 6. Cluster Autoscaler was too efficient. Scaled down mid-job. Pods got killed with 0 warning. ✓ 𝐇𝐨𝐰 𝐰𝐞 𝐟𝐢𝐱𝐞𝐝 𝐢𝐭: Split node pools by workload type. Critical stuff stays on on-demand. Spot nodes only run retryable junk. 7. Completed jobs were hogging resources. They finished… but hung around forever. ✓ 𝐇𝐨𝐰 𝐰𝐞 𝐟𝐢𝐱𝐞𝐝 𝐢𝐭: TTL Controller + dynamic GC rules based on job type. The common theme? None of this showed up in our standard monitoring. We had to dig. Ask dumber questions. Read the logs everyone ignores. If you’ve ever lost a week to stuff like this, what tripped you up? ♻️ 𝐑𝐄𝐏𝐎𝐒𝐓 𝐒𝐨 𝐎𝐭𝐡𝐞𝐫𝐬 𝐂𝐚𝐧 𝐋𝐞𝐚𝐫𝐧.

🚀 𝗧𝗿𝗼𝘂𝗯𝗹𝗲𝘀𝗵𝗼𝗼𝘁𝗶𝗻𝗴 𝗞𝘂𝗯𝗲𝗿𝗻𝗲𝘁𝗲𝘀 𝗖𝗵𝗮𝗼𝘀: 𝗦𝗼𝗹𝘃𝗶𝗻𝗴 𝗮 𝗖𝗿𝗮𝘀𝗵𝗟𝗼𝗼𝗽𝗕𝗮𝗰𝗸𝗢𝗳𝗳 𝗡𝗶𝗴𝗵𝘁𝗺𝗮𝗿𝗲 🔥 Kubernetes is amazing—until it throws 𝗖𝗿𝗮𝘀𝗵𝗟𝗼𝗼𝗽𝗕𝗮𝗰𝗸𝗢𝗳𝗳 at you. Recently, I faced this error during a deployment that turned a simple task into a complex troubleshooting exercise. Here’s how I approached and solved it. 🛠️ 🔍 𝗧𝗵𝗲 𝗣𝗿𝗼𝗯𝗹𝗲𝗺: 𝗖𝗿𝗮𝘀𝗵𝗟𝗼𝗼𝗽𝗕𝗮𝗰𝗸𝗢𝗳𝗳 The deployment seemed fine initially, but the pods entered a restart loop with the error: 𝗖𝗿𝗮𝘀𝗵𝗟𝗼𝗼𝗽𝗕𝗮𝗰𝗸𝗢𝗳𝗳. 𝗙𝗶𝗿𝘀𝘁 𝗦𝘁𝗲𝗽𝘀: 1️⃣ Checked container logs: Minimal info—just abrupt terminations. 2️⃣ Ran kubectl describe pod: Highlighted repeated 𝗹𝗶𝘃𝗲𝗻𝗲𝘀𝘀 𝗽𝗿𝗼𝗯𝗲 𝗳𝗮𝗶𝗹𝘂𝗿𝗲𝘀 and 𝗺𝗲𝗺𝗼𝗿𝘆 𝗮𝗹𝗹𝗼𝗰𝗮𝘁𝗶𝗼𝗻 𝘄𝗮𝗿𝗻𝗶𝗻𝗴𝘀. 3️⃣ Monitored metrics with Prometheus and Grafana: Found spikes in memory and CPU usage during pod initialization. 🧩 𝗥𝗼𝗼𝘁 𝗖𝗮𝘂𝘀𝗲 𝗔𝗻𝗮𝗹𝘆𝘀𝗶𝘀 🧩 🔹 𝑴𝒊𝒔𝒄𝒐𝒏𝒇𝒊𝒈𝒖𝒓𝒆𝒅 𝑯𝒆𝒂𝒍𝒕𝒉 𝑷𝒓𝒐𝒃𝒆𝒔: The livenessProbe was too aggressive, checking health 1 second after the container started. The container actually needed 5 seconds to initialize. 🔹 𝑹𝒆𝒔𝒐𝒖𝒓𝒄𝒆 𝑨𝒍𝒍𝒐𝒄𝒂𝒕𝒊𝒐𝒏 𝑰𝒔𝒔𝒖𝒆𝒔: requests.memory was set to 256Mi, while the container required 500Mi to start. This caused the OOM (Out of Memory) killer to terminate the pods repeatedly. 💡 𝗧𝗵𝗲 𝗙𝗶𝘅 💡 After identifying the issues, I optimized the YAML manifest: 𝒂𝒑𝒊𝑽𝒆𝒓𝒔𝒊𝒐𝒏: 𝒂𝒑𝒑𝒔/𝒗1 𝒌𝒊𝒏𝒅: 𝑫𝒆𝒑𝒍𝒐𝒚𝒎𝒆𝒏𝒕 𝒎𝒆𝒕𝒂𝒅𝒂𝒕𝒂: 𝒏𝒂𝒎𝒆: 𝒎𝒊𝒄𝒓𝒐𝒔𝒆𝒓𝒗𝒊𝒄𝒆-𝒅𝒆𝒑𝒍𝒐𝒚𝒎𝒆𝒏𝒕 𝒔𝒑𝒆𝒄: 𝒓𝒆𝒑𝒍𝒊𝒄𝒂𝒔: 3 𝒔𝒆𝒍𝒆𝒄𝒕𝒐𝒓: 𝒎𝒂𝒕𝒄𝒉𝑳𝒂𝒃𝒆𝒍𝒔: 𝒂𝒑𝒑: 𝒎𝒊𝒄𝒓𝒐𝒔𝒆𝒓𝒗𝒊𝒄𝒆 𝒕𝒆𝒎𝒑𝒍𝒂𝒕𝒆: 𝒎𝒆𝒕𝒂𝒅𝒂𝒕𝒂: 𝒍𝒂𝒃𝒆𝒍𝒔: 𝒂𝒑𝒑: 𝒎𝒊𝒄𝒓𝒐𝒔𝒆𝒓𝒗𝒊𝒄𝒆 𝒔𝒑𝒆𝒄: 𝒄𝒐𝒏𝒕𝒂𝒊𝒏𝒆𝒓𝒔: - 𝒏𝒂𝒎𝒆: 𝒎𝒊𝒄𝒓𝒐𝒔𝒆𝒓𝒗𝒊𝒄𝒆-𝒄𝒐𝒏𝒕𝒂𝒊𝒏𝒆𝒓 𝒊𝒎𝒂𝒈𝒆: 𝒚𝒐𝒖𝒓-𝒓𝒆𝒑𝒐/𝒚𝒐𝒖𝒓-𝒊𝒎𝒂𝒈𝒆:𝒍𝒂𝒕𝒆𝒔𝒕 𝒑𝒐𝒓𝒕𝒔: - 𝒄𝒐𝒏𝒕𝒂𝒊𝒏𝒆𝒓𝑷𝒐𝒓𝒕: 8080 𝒍𝒊𝒗𝒆𝒏𝒆𝒔𝒔𝑷𝒓𝒐𝒃𝒆: 𝒉𝒕𝒕𝒑𝑮𝒆𝒕: 𝒑𝒂𝒕𝒉: /𝒉𝒆𝒂𝒍𝒕𝒉𝒛 𝒑𝒐𝒓𝒕: 8080 𝒊𝒏𝒊𝒕𝒊𝒂𝒍𝑫𝒆𝒍𝒂𝒚𝑺𝒆𝒄𝒐𝒏𝒅𝒔: 10 # 𝑨𝒍𝒍𝒐𝒘 𝒆𝒏𝒐𝒖𝒈𝒉 𝒕𝒊𝒎𝒆 𝒇𝒐𝒓 𝒊𝒏𝒊𝒕𝒊𝒂𝒍𝒊𝒛𝒂𝒕𝒊𝒐𝒏 𝒑𝒆𝒓𝒊𝒐𝒅𝑺𝒆𝒄𝒐𝒏𝒅𝒔: 5 𝒓𝒆𝒂𝒅𝒊𝒏𝒆𝒔𝒔𝑷𝒓𝒐𝒃𝒆: 𝒉𝒕𝒕𝒑𝑮𝒆𝒕: 𝒑𝒂𝒕𝒉: /𝒓𝒆𝒂𝒅𝒊𝒏𝒆𝒔𝒔 𝒑𝒐𝒓𝒕: 8080 𝒊𝒏𝒊𝒕𝒊𝒂𝒍𝑫𝒆𝒍𝒂𝒚𝑺𝒆𝒄𝒐𝒏𝒅𝒔: 10 𝒑𝒆𝒓𝒊𝒐𝒅𝑺𝒆𝒄𝒐𝒏𝒅𝒔: 5 𝒓𝒆𝒔𝒐𝒖𝒓𝒄𝒆𝒔: 𝒓𝒆𝒒𝒖𝒆𝒔𝒕𝒔: 𝒎𝒆𝒎𝒐𝒓𝒚: "512𝑴𝒊" # 𝑨𝒅𝒋𝒖𝒔𝒕𝒆𝒅 𝒕𝒐 𝒎𝒂𝒕𝒄𝒉 𝒂𝒄𝒕𝒖𝒂𝒍 𝒓𝒆𝒒𝒖𝒊𝒓𝒆𝒎𝒆𝒏𝒕𝒔 𝒄𝒑𝒖: "250𝒎" 𝒍𝒊𝒎𝒊𝒕𝒔: 𝒎𝒆𝒎𝒐𝒓𝒚: "1𝑮𝒊" 𝒄𝒑𝒖: "500𝒎" 📊 𝗞𝗲𝘆 𝗧𝗮𝗸𝗲𝗮𝘄𝗮𝘆𝘀: 1️⃣ Tailor Health Probes: Misconfigured probes can destabilize your deployment. 2️⃣ Monitor Metrics: Tools like Prometheus and Grafana are invaluable for identifying resource bottlenecks. Follow Aditya Jaiswal #Kubernetes #DevOps #CrashLoopBackOff #CloudNative #Microservices #Prometheus #Grafana #ContainerOrchestration #Troubleshooting #TechInsights

🚨 Struggling with Kubernetes Deployments? Let’s Break Down How to Debug Like a Pro! 🔍 Kubernetes is a game-changer for container orchestration, but let’s be honest—Kubernetes deployments don’t always go smoothly. 😬 We’ve all faced those frustrating moments when something just doesn’t work as expected. But fear not! Debugging Kubernetes can be your secret weapon to fixing deployment issues and taking your skills to the next level. 🚀 Here’s how you can debug a Kubernetes deployment like a pro and turn those headaches into solutions! Step 1: Check the Pods’ Status 1️⃣ What to Do: 🟢 Use the kubectl get pods command to check the status of your pods. 🟢 Look for pods that are stuck in a "Pending" or "CrashLoopBackOff" state. 1️⃣ Why It Matters: 🟢 This is your first indication of what’s going wrong. If your pods aren’t starting properly, there’s a deeper issue to tackle. Step 2: Inspect Pod Logs 2️⃣ What to Do: 🟡 Run kubectl logs <pod-name> to retrieve logs from a specific pod. 🟡 If your container is crashing, these logs are crucial for identifying the root cause. 2️⃣ Why It Matters: 🟡 Logs give you detailed insights into what's happening inside the pod—whether it's a misconfiguration, missing environment variables, or something else. Step 3: Describe the Deployment 3️⃣ What to Do: 🔵 Use kubectl describe deployment <deployment-name> to get a detailed breakdown of the deployment, including events, pod scheduling issues, and resource constraints. 3️⃣ Why It Matters: 🔵 This command helps you spot potential issues with node scheduling, resource limits, or even image pull errors. It’s the full story of your deployment’s health! Step 4: Check for Resource Limitations 4️⃣ What to Do: 🔴 Look for resource issues with kubectl describe node <node-name>. Check if your pods have enough memory and CPU to run properly. 4️⃣ Why It Matters: 🔴 Many deployment failures come down to insufficient resources. Scaling your resources or adjusting your pod limits might be all you need to fix the problem! Step 5: Review ConfigMaps and Secrets 5️⃣ What to Do: 🟠 Check if your deployment is correctly loading ConfigMaps and Secrets. Use kubectl get configmap and kubectl get secret to ensure they are properly mounted. 5️⃣ Why It Matters: 🟠 Misconfigured environment variables, credentials, or missing files can cause containers to fail unexpectedly. This step helps you ensure the right settings are in place. Step 6: Network Connectivity 6️⃣ What to Do: 🟣 Use kubectl exec -it <pod-name> -- /bin/bash to enter a pod’s shell and troubleshoot network connectivity with tools like curl or ping. 6️⃣ Why It Matters: 🟣 If your pods can’t communicate with each other or external services, the entire deployment can break. Ensuring connectivity is critical for debugging. 💬 Join the Discussion #Kubernetes #DevOps #CloudComputing #Containerization #K8s #Debugging #TechTips #DigitalTransformation #DilawarJavaid #DeploymentIssues #CloudSolutions #InfrastructureAsCode