eBPF Deployment Safety: GitHub's 8 Key Insights for Breaking Circular Dependencies

When you run one of the world's largest developer platforms on its own code, you create a unique challenge: any outage can prevent you from fixing it. GitHub faces this exact circular dependency, where deploying a fix relies on the very system that's broken. While mirrors and rollback assets help, other hidden dependencies lurk inside deployment scripts. That's where eBPF (extended Berkeley Packet Filter) comes in—a powerful kernel technology that lets GitHub selectively monitor and block network calls during deployments. In this listicle, we break down the eight critical lessons GitHub learned about using eBPF to make deployments safer and avoid self-inflicted outages.

1. Self-Hosting Creates an Unexpected Vulnerability

GitHub hosts all its own source code on github.com—a point of pride that also introduces a circular dependency. If github.com goes down, engineers lose access to the very repositories needed to deploy a fix. This forces GitHub to maintain a separate mirror of code and pre-built assets for emergency rollbacks. While that handles the obvious case, it doesn't cover subtler dependencies that can creep into deployment scripts. eBPF provides a way to enforce boundaries that were previously impossible to guarantee.

eBPF Deployment Safety: GitHub's 8 Key Insights for Breaking Circular Dependencies — Source: github.blog

2. The Doom Loop: An Outage During an Outage

Imagine a MySQL outage that prevents GitHub from serving release data. To recover, you need to run a deploy script on the affected MySQL nodes. But if that script tries to pull a binary from GitHub (which is down), the script fails—creating a deadly loop. This scenario illustrates how direct dependencies can stall recovery. eBPF allows GitHub to block specific outgoing calls from deployment scripts, preventing such self-defeating behavior. The kernel-level inspection catches attempts before they happen, breaking the cycle.

3. Direct Dependencies Are the Most Obvious Trap

In the MySQL outage example, the deploy script attempts to download the latest release of an open source tool from GitHub. Since GitHub can't serve the release data due to the outage, the script hangs indefinitely. This is a classic direct dependency—the script literally depends on the service it's trying to fix. With eBPF, GitHub can write programs that intercept network calls from the script to known problematic endpoints (like github.com) and either block them or redirect them to a local cache. This ensures the script completes without leaving the recovery path.

4. Hidden Dependencies Lurk in Off-the-Shelf Tools

Even when a tool is already installed on disk, it may secretly phone home for updates. For instance, a servicing tool used by the MySQL deploy script might check GitHub for a newer version before proceeding. If GitHub is unreachable, the tool may fail or hang—a hidden dependency. Manual code reviews often miss these behaviors. eBPF can monitor system calls and block any network request that touches GitHub's domains, regardless of the tool's source. This catches hidden dependencies that would otherwise slip through.

5. Transient Dependencies Propagate Failures

Sometimes a deploy script calls an internal API (like a migrations service), which in turn tries to fetch a binary from GitHub. The failure cascades back to the original script—this is a transient dependency. These are particularly hard to trace because the dependency isn't in the script itself but in a chain of services. eBPF's ability to trace system calls across processes allows GitHub to see the entire chain and block the downstream request at the kernel level, stopping the cascade before it starts.

6. Previous Mitigations Were Fragile

Before eBPF, GitHub relied on mirrors and pre-built assets to break the primary circular dependency. But these only cover the scenario where the deploy script explicitly pulls from GitHub. They do nothing for tools that check for updates or internal services that reach out. Teams had to manually review scripts and hope no hidden call existed. This approach was labor-intensive and error-prone, especially as deployment scripts evolved. eBPF offers a systematic, automated layer of protection that works regardless of how the dependency arises.

7. Manual Reviews Shift the Burden to Teams

Historically, each team owning stateful hosts had to review their deployment scripts and identify every possible circular dependency. This was a heavy burden—teams had to understand every tool and service their script touched. In practice, many dependencies aren't obvious until an outage occurs. eBPF eliminates this burden by providing a centralized enforcement point. A small team can define deployment safety policies that apply across all hosts, freeing service teams from manual audits.

8. eBPF Offers Fine-Grained Control Over Network Calls

eBPF allows GitHub to attach programs to kernel events—including system calls for network access—without modifying application code. For deployment safety, they can write eBPF programs that inspect every outgoing connection from a deploy script. If the destination matches a blacklisted service (like github.com), the program can block the call or terminate the process. This selective monitoring ensures that only the critical recovery paths are allowed, while preventing accidental circular dependencies. eBPF's low overhead makes it suitable for production use.

In conclusion, GitHub's journey with eBPF shows that even the most well-intentioned deployment scripts can harbor hidden circular dependencies. By moving safety checks from manual reviews into the kernel, GitHub has built a robust layer of protection that scales across its entire fleet. eBPF's ability to monitor and block specific network calls at runtime gives teams confidence that recovery scripts will never accidentally create a doom loop. For any organization running critical services, adopting eBPF for deployment safety is a practical step toward more resilient infrastructure.

Tags: