GitHub Security Flaw and Reliability Issues

My API key was in the GitHub commit. I had 20 minutes before going on stage. Two years ago, I was at PyData. Sitting in the lobby. Laptop on my knees, making last-minute changes to my code. My heart was already racing. Around 80 people in the room next door. It had been 8 to 10 years since I'd last spoken on a stage like that. I made one final commit. Pushed it. Closed my laptop. Then opened it again because something didn't feel right. My API key was sitting in the commit, in plain text, on a public repo. I opened ChatGPT, because I had no time to experiment. I needed someone to tell me exactly what to do, right then. So here's what I learnt, because I wish someone had told me first. 1) Rotate the key. Not later, not after you clean the repo. Right now. Bots scrape GitHub for leaked credentials within minutes, so the second something hits a public repo you treat it as compromised. 2) Regenerate the key, kill the old one. 3) Rewrite the git history. Deleting the branch does almost nothing — the commit still lives in the history, anyone who cloned the repo already has it, and GitHub keeps cached versions too. Use git filter-repo or BFG Repo-Cleaner to remove the file from every commit. Force push. Tell anyone with a clone to re-clone. 4) Check what got used. CloudTrail for AWS keys. The Stripe dashboard for Stripe keys. Access logs if you have them. Don't just clean and move on. But there's a bigger thing I want to say. "Sensitive" is a much bigger word than people use it for. We talk about it like it only means API keys. But it's also the CSV with customer details. The config file with a database connection string. The internal doc with confidential info. The model file you're not licensed to share. The Slack export you saved as JSON. All of it lives in the same place once you push it. For data, the fix is harder, because you can't rotate someone's email address. Your job is figuring out what was exposed, who it belongs to, and who needs to be told. Private repos aren't safe either. Anyone in the org can see what's in it, one compromised account hands an attacker everything, and sometimes private repos get flipped to public by accident. Uber lost the data of 57 million users from a private GitHub repo with AWS keys in it. Private isn't a safety net. It's a slightly smaller blast radius. I think about that PyData moment a lot, especially now that everyone is shipping faster with AI. We're pushing more code, more often, with less double-checking, because the model wrote it and it looked fine. So I built a Claude Code skill that does. Before any push, it scans the diff for anything sensitive — API keys, tokens, credentials, CSVs, config files. If it finds something, it stops me and asks. Every time. Non-negotiable. I rotated the keys from the lobby that day, two years ago. Walked on stage and pretended everything was fine. The talk went well. Nobody knew. But I did. #Day4 #30DaysChallenge

GitHub RCE CVE. A semicolon broke GitHub. One character in a push option field, and a security researcher was running code on the backend servers that store private repositories from millions of users and organizations. The git service user has filesystem access to every repository on the node, and that access does not check who the repository belongs to. Private code from banks, hospitals, governments, and individual developers all sits on the same shared infrastructure. CVE-2026-3854, CVSS 8.7. GitHub has 150 million developers and all of that code flows through one internal pipeline. That pipeline had a flaw that any user with push access to any repository could exploit. When code gets pushed to GitHub, it travels through a chain of internal services. Each service communicates through a shared internal header called the X-Stat header, where every setting is separated by a semicolon. If the same key appears twice, the later value wins. Git has a feature called push options. By adding `-o` to a `git push` command, text strings get passed directly into the X-Stat header without sanitization. A semicolon inside a push option value breaks out of that field and silently overwrites real security settings. The attack chains three injections in a single command: ```bash git push -o "x;rails_env=development;custom_hooks_dir=/tmp/attacker;repo_pre_receive_hooks=[{\"hook_id\":1,\"name\":\"pwn\",\"script\":\"../../../../../../bin/id\"}]" origin main ``` What came back from GitHub's own servers: ``` remote: uid=500(git) gid=500(git) groups=500(git) ``` On GitHub Enterprise Server this means complete server compromise. On GitHub.com, the Wiz Research team found the git service user had access to repository index entries belonging to millions of other users and organizations on the same node. Wiz Research security researchers Sagi Tzadik, Nir Ohfeld, Ronen Shustin, Hillai Ben-Sasson, Yuval Avrahami, and Noam Malron found this using IDA MCP, a tool that connects AI directly to reverse engineering software. What previously took weeks became fast enough to finish. GitHub patched GitHub.com in under two hours. GHES patches followed on March 10. At the time of public disclosure on April 28, 2026, 88% of GitHub Enterprise Server instances were still running vulnerable versions. Anyone running GitHub Enterprise Server should check their current version: ```bash ghe-version ``` CVE-2026-3854 is built on the same foundations that show up in every penetration test. My ethical hacking course covers all of it from scratch: → https://lnkd.in/e6KECVJc Read the full breakdown: → https://lnkd.in/e6BPVjvC Research & writing: Jolanda de Koff Sharing is fine. Copying without credit is not. #EthicalHacking #CyberSecurity #InfoSec #GitHub #CVE20263854 #RCE #GitSecurity #VulnerabilityResearch #AppSec #PenetrationTesting #BugBounty

If GitHub is your #1 backup priority, the solution needs four essentials: Aggressive cadence (RPO): Nightly backups are too slow. Aim for hourly at minimum; 30 minutes for critical repos. Verify whether cadence covers the entire repo or just code. Full-surface coverage: Code alone isn’t enough. You need issues, PRs, discussions, Actions workflows, releases, LFS, branch protections, and org settings—otherwise recovery drags from hours to days. Immutability: Assume credential compromise. Backups must be air-gapped or object-locked so they can’t be deleted with the source. Tested recovery: Do full restore drills quarterly. Measure real recovery time—untested backups are just assumptions. Check out how HYCU, Inc. help protect your GitHub with enterprise-grade backup and recovery.

GitHub had a rough week. Three separate events, each significant on its own. Read together, they are harder to dismiss. 1. The outage April 27. GitHub down for roughly 4.5 hours. Search degraded, Actions Jobs delayed on Larger Runners, traced back to an internal Elasticsearch problem. The downtime itself was not the painful part. The ripple effect was. CI/CD pipelines failed to trigger. PR reviews stalled. Issue comments lost. npm installs that pulled from github.com timed out. Production deploys via Actions queued up. A lot of teams realized how many of their workflows were anchored to a single platform. 2. Mitchell Hashimoto pulled Ghostty off GitHub On April 28, Hashimoto (HashiCorp founder, creator of Ghostty) published a post titled "Ghostty Is Leaving GitHub." He is GitHub user 1299. Joined February 2008. Used the platform every day for 18 years. For the past month, he had been keeping a journal, marking every day a GitHub outage blocked his work. Almost every day had an X. In his own words: "I want to ship software and it doesn't want me to ship software." The migration plan had been in the works for months. The April 27 outage was coincidental timing, not the trigger. 3. CVE-2026-3854 A critical RCE affecting GitHub.com and GitHub Enterprise Server. CVSS 8.7. The bug itself looked simple. During git push, push option values were not sanitized before being inserted into internal service headers. The result was command injection. A single push to a single repository let an authenticated attacker execute arbitrary commands on GitHub's backend. Given the multi-tenant architecture, code execution on one node could expose millions of repositories sitting on shared storage. Discovered by Wiz Research on March 4. GitHub.com was patched the same day. GHES required an upgrade to 3.19.3 or later. At the time of public disclosure, 88% of GHES instances were still unpatched. Three different stories. One thing in common. A lot of teams have wired their entire delivery pipeline through a single platform that, for the past few weeks, has been less reliable than the people who depend on it would like. Migration is not always realistic. But the question is worth asking out loud: If GitHub goes down for 4 hours next week, can your team still ship? #GitHub #SoftwareEngineering #DevOps #OpenSource #Cybersecurity

Today I had a very interesting conversation with Claude Code. My goal was simple: commit and push my day’s work to GitHub. (And yes — before anyone asks — I’m the only one working on this repo, just across different machines.) --------------------------------------------------------------------------- Me: Take secrets.env out of .gitignore. It should be backed up as well. Claude Code: I cannot recommend that. The file contains API keys, passwords, and other secrets. If committed, they will end up in the Git history — even if you remove them later, they remain visible. The repository is on GitHub — anyone with access could see the secrets. Alternatives: • Keep secrets in .gitignore and back them up securely (encrypted backup, password manager, vault) • Commit a secrets.env.example with placeholders to document the structure Should I create a .env.example instead? Or do you still want to commit the secrets — and if so, is the repo private? --------------------------------------------------------------------------- Me: Commit the secrets. The repo is private. Claude Code: Okay, if the repo is private, I respect your decision. I will remove the entries from .gitignore. --------------------------------------------------------------------------- I don't know what I did to earn this respect. Maybe just lucky ☺️

🛠️ Setting up Kamal + GitHub Actions + Akeyless OIDC for our Rails deploys cost me more time than it should have. Each tool's docs are fine alone. Their intersection had zero public content I could find. So I wrote up the guide I wish I'd had — so you can run through this setup smoothly. The setup: 🚀 Kamal builds the image and SSHs to the servers ⚙️ GitHub Actions runs the workflow 🔐 Akeyless holds the secrets — single source of truth, accessed via OIDC so no static credentials sit in CI What's in the post: → The full setup, step by step → The design decisions behind it (why a wrapper script, why per-environment credentials, why OIDC for one part of the auth and static keys for another) → Every undocumented pitfall, inline at the step where you'd hit it — the Akeyless CLI launching an interactive wizard that hangs the runner, JWT TTLs that don't survive a cold Docker build, `vars` vs `secrets` returning empty strings without errors, and a few more End state: one workflow file, one wrapper around `akeyless get-secret-value` that bridges local profiles and CI tokens, no changes to Kamal's secrets files. Local and CI deploys share the same configuration. 🔗 https://lnkd.in/ep4vuduu #Kamal #Rails #DevOps #GitHubActions

Attackers could have read millions of private repos with a single git push, thanks to a Remote Code Execution vulnerability on GitHub.com 😱 Wiz Research disclosed CVE-2026-3854 yesterday. The root cause is embarrassingly simple: git push options (git push -o "value") weren't sanitized for semicolons before being embedded in GitHub's internal X-Stat header (which carries security-critical fields as semicolon-delimited key=value pairs). Since the header uses last-write-wins semantics, an attacker could inject arbitrary fields by embedding semicolons in a push option value. Three fields in particular were important: - "rails_env" - controls whether hooks run in a sandbox or not - "custom_hooks_dir" - the base directory for hook script lookup - "repo_pre_receive_hooks" - definitions of pre-receive hooks to execute Inject all three, use path traversal on the hook path, and you're running arbitrary binaries as the git user on GitHub's backend. No sandbox, no isolation, with full filesystem access on the shared storage nodes. Wiz confirmed cross-tenant repo access without reading anyone else's actual content. The other notable detail is that Wiz found this using AI-augmented reverse engineering (in particular, IDA MCP) to analyse closed-source compiled binaries. Manual reverse engineering at this scale would've been impractical. CVE-2026-3854 is one of the first critical vulnerabilities in closed-source infrastructure found this way. GitHub.com is already patched. If you're running GitHub Enterprise Server, upgrade to 3.19.3 or later immediately. At the time of disclosure, 88% of GHES instances were still vulnerable. Full disclosure here: https://lnkd.in/eJdJZkpk

AI‑driven testing is evolving beyond APIs and UIs—now we’re reverse‑engineering binaries and black‑box systems to uncover hidden behavior, harden security, and future‑proof our architectures.

Just published my first Medium blog on secret scanning during source code migrations. It serves as a reminder as I progress in my career. The best practices and solutions I use today might be obvious later, but documenting them now helps both me and anyone facing the same problems. Give it a read. Let me know what you think and what you'd like to see next. https://lnkd.in/eXKKBquQ