GitHub CTO Apologizes for Reliability Issues and Security Breach

DNS changes used to feel like flying blind. 🌫️ You'd push a record update, wait, hope nothing broke, and only find out something went wrong when a service went down. 🔥 I built dns-sync to fix that. The idea is simple: treat DNS like infrastructure code. 🧱 The workflow: 🗂️ Edit your zone file and open a PR 🤖 The bot automatically posts a diff — exactly what gets created, updated, or deleted ✅ All checks pass → you hit merge → done No surprises. No guessing. No "oops I deleted the wrong record." 😅 One uses: line in GitHub Actions is all the setup it takes. → github.com/cl8dep/dns-sync #DevOps #DNS #GitOps #OpenSource #Infrastructure #GitHub #DeveloperTools

GitHub just archived 2 9's in a while. 89.91% uptime. 95 incidents in the last 90 days that's roughly one incident per day. For reference, "three 9's" (99.9%) means 8.7 hours of downtime per year. GitHub's at 876 hours of theoretical downtime at this rate. So what's likely going on under the hood? Actions & CI load Half the internet is using GitHub Actions as default so the scale has grown significantly now i get why some co. still prefer jenkins Lot of Dependencies Git LFS, Packages, Copilot, Codespaces, Pages, Actions these are loosely coupled but share infra. One wobble and the status page lights up. MySQL at this scale GitHub famously runs on a heavily customized MySQL stack. Keeping that healthy while the product scale grows is very difficult in itself That just shows the role of oncall engineers are very crucial one and there are still things which is hard to automate There might be other reasons into play here aswell but these are my thoughts #DevOps #SRE #GitHub #BackendEngineering #SoftwareEngineering

GitHub's secret scanning just got a serious upgrade, now pushing alerts to APIs, webhooks, and even delegated workflows. Finally, a proactive step towards shoring up those leaky secrets before they cause a headache. Good to see this evolving. ✨ #DevSecOps

> GitHub stopped updating its own status page due to terrible availability ... 90.1% uptime - This means ... issues/degradations for 2.5 hours daily ... > GitHub struggles to keep up with the increase in load from AI agents generating more code and pull requests ... Claude Code bot contributions growth in the past 3 months has been enormous ... Stream of outages ... https://lnkd.in/eYHzasTh

I honestly thought platforms of GitHub's scale would have solved the core scaling riddles by now. Turns out, even they are battling fundamental architectural demons. The article dives into GitHub's recent availability woes. We're talking cascading failures, tight coupling, and an inability to shed load effectively. Imagine your critical CI/CD pipelines suddenly grinding to a halt because an authentication database choked. That's the kind of disruption they're talking about. This isn't just another post-mortem. It's a stark reminder that rapid growth amplifies every architectural shortcut. I realized that what we often chalk up to "bad luck" in our own systems is often a predictable outcome of underlying structural choices. GitHub's candidness about "tight coupling" and "insufficient isolation" really hit home. In the digital financial services, I've seen how a seemingly minor dependency, like a third-party fraud detection API, can ripple through an entire transaction flow if not properly isolated and rate-limited. We often talk about microservices as the panacea, but if they're still tightly coupled at the data or authentication layer, you haven't solved the problem; you've just distributed it. Their challenge with effective "backpressure mechanisms" and load shedding, especially as AI-driven tooling piles on, highlights that just throwing more compute at it isn't enough. You need smart circuit breakers and adaptive traffic management, not just reactive scaling. The article also points out the disparity between official status pages and real-world developer experience, which resonates deeply with anyone who's ever debugged a "green" system. How do you proactively build resilience against these kinds of systemic failures when your system is growing at warp speed? What's your secret sauce for true architectural isolation? https://lnkd.in/e3EmWfjz #SystemArchitecture #Scalability #ReliabilityEngineering #DevOps

Last week, a packaging error at Anthropic exposed 512,000 lines of Claude Code source. By the next morning, it had triggered one of the most damaging supply chain cascades I've tracked this year. Here's what actually happened — and why it matters for every dev team. The exposed npm package got mirrored across GitHub within hours. Attackers spun up fake repos bundled with Vidar stealer. GitHub Releases — a channel most devs trust by default — became the delivery mechanism. Then it got worse. The Trivy image scanner was compromised via a GitHub Action. Soon after, axios — downloaded hundreds of millions of times weekly — shipped a version carrying a remote access trojan. Think about what lives on a developer's machine: production API keys, cloud credentials, CI/CD pipeline access, SSH keys, database passwords. One compromised laptop. Everything exposed. The technical root cause? Datadog's State of DevSecOps 2026: 71% of orgs leave GitHub Actions completely unpinned. Not pinned to a version. Not pinned to a commit SHA. Just @v2 and a prayer. This has been a known risk for years. Nobody fixes it until something breaks. The fix isn't complicated: → Pin every Action to a full commit SHA → Run secret scanning on every commit (TruffleHog, gitleaks) → Generate an SBOM on every build → Treat your CI pipeline as production infrastructure — because it is Shift-left means moving security to where the problem starts: the pipeline, not just the code it builds. This week was an expensive reminder. How does your team handle GitHub Actions pinning today? #DevSecOps #SupplyChainSecurity #CICD #GitHubActions #ShiftLeft

GitHub restructured its status page a while back- making it significantly harder to get a clear picture of platform-wide uptime. The unofficial reconstruction at The Missing GitHub Status Page (community-built, since GitHub won't publish an aggregate) reports 89.31% uptime as of this post.   When your reliability is so bad the scoreboard has to be crowd-sourced, that's not a technical problem. 𝗧𝗵𝗮𝘁'𝘀 𝗮𝗻 𝗮𝗱𝗺𝗶𝘀𝘀𝗶𝗼𝗻.  Meanwhile: * A Claude Code agent recently deleted a developer's production database and 2.5 years of volume snapshots. * Amazon is holding emergency meetings about production outages, while spending $200B on AI infrastructure.  𝗧𝗵𝗶𝘀 𝗶𝘀𝗻'𝘁 𝗮 𝗯𝗮𝗱 𝗾𝘂𝗮𝗿𝘁𝗲𝗿. 𝗧𝗵𝗶𝘀 𝗶𝘀 𝗮 𝗽𝗮𝘁𝘁𝗲𝗿𝗻.  Companies are shipping much more code without making matching investments in operational discipline.  𝗘𝗻𝘁𝗲𝗿𝗽𝗿𝗶𝘀𝗲 𝘀𝗼𝗳𝘁𝘄𝗮𝗿𝗲 𝗲𝘅𝗶𝘀𝘁𝘀 𝘁𝗼 𝘀𝗼𝗹𝘃𝗲 𝗰𝘂𝘀𝘁𝗼𝗺𝗲𝗿 𝗽𝗿𝗼𝗯𝗹𝗲𝗺𝘀. You can't do that if your service is down, your CI tooling is broken again, or an AI agent just nuked your database.  𝗥𝗲𝗹𝗶𝗮𝗯𝗶𝗹𝗶𝘁𝘆 𝗶𝘀 𝘁𝗵𝗲 𝗺𝗼𝘀𝘁 𝗶𝗺𝗽𝗼𝗿𝘁𝗮𝗻𝘁 𝗳𝗲𝗮𝘁𝘂𝗿𝗲. If you're not measuring it, making business decisions concerning it, and saying "no" based on it- you're gambling with customer trust.  𝗪𝗲'𝘃𝗲 𝘁𝗼𝘁𝗮𝗹𝗹𝘆 𝗹𝗼𝘀𝘁 𝘁𝗵𝗲 𝗽𝗹𝗼𝘁.  

Seeing this reliability issue come up more and more is very concerning. Will we look back at the heady days of the early AI era as a time when so many people forgot the hard learned lessons of running production systems?

GitHub's latest update is a welcome relief for notification fatigue! They're tweaking notification retention periods and, even better, archived repository watches won't be sending you alerts anymore. Finally, some peace for those old, dusty projects! 🙌 A small change, but it really helps keep the focus on what matters. #GitHub #DevOps

I built a lightweight version of the popular GitOps platforms entirely on top of GitHub Actions and Docker, so I never have to SSH into my home server to deploy anything. The stack is two small tools I built (github-multi-runner and docker-compose-deploy) plus the convention of keeping a real compose file in every service repo. No control plane, no Kubernetes, totally free on GitHub. Pretty fun pattern if you run your own stuff at home.