GitHub, like many tech giants, faces a unique challenge: it hosts its own source code on github.com. While this dogfooding approach ensures internal testing before user-facing releases, it creates a critical circular dependency. If github.com goes down, engineers can't access the code needed to fix it. To mitigate this, GitHub maintains a mirror of its code for recovery. But even with that safety net, deployment scripts themselves can introduce new circular dependencies, pulling from internal services or downloading binaries from GitHub. This is where eBPF (extended Berkeley Packet Filter) steps in—a powerful kernel technology that lets GitHub selectively monitor and block calls within deployment processes. In this article, we’ll explore five key things you need to know about how GitHub uses eBPF to enhance deployment safety and break those tricky dependency loops.
1. The Core Challenge: Circular Dependencies in Deployments
At first glance, the solution to a circular dependency seems straightforward: keep a backup copy of your code. GitHub already does this by mirroring repositories and storing built assets for rollbacks. However, deployment scripts are often more complex. They may need to fetch tools, check for updates, or communicate with internal services. During an outage—say, a MySQL failure that makes github.com unreachable—these scripts can inadvertently create new circular dependencies. A script that tries to pull a binary from GitHub while GitHub is down will fail, compounding the incident. GitHub’s real challenge was not just fixing the immediate dependency but preventing all possible circular references that deployment code could introduce.
2. Three Types of Circular Dependencies: Direct, Hidden, Transient
To tackle the problem, GitHub categorized circular dependencies into three types. Direct dependencies are obvious: a script attempts to download a tool from the very service it's trying to fix. Hidden dependencies are subtler—a tool already on disk might check for updates at runtime, causing a hang or failure if it can't reach GitHub. Transient dependencies occur when a script calls an internal API (like a migrations service), which in turn tries to fetch a binary from GitHub; the failure cascades back to the deploy script. Recognizing these categories helped GitHub design a more comprehensive solution that goes beyond simple code mirroring.
3. Traditional Mitigation: Manual Reviews and Their Limits
Before adopting eBPF, GitHub relied on individual teams to review their deployment scripts and identify circular dependencies manually. Each team owning stateful hosts had to audit every command and external call. But this approach had major flaws: dependencies are often indirect or hidden (like a tool's auto-update check), and reviewing thousands of lines of script is error-prone. Moreover, as the platform evolved, new scripts could introduce fresh dependencies without detection. The manual process couldn't scale, and the risk of an undetected circular dependency causing an outage remained high. GitHub needed a more automated, runtime-based method to enforce safe deployment behavior.
4. How eBPF Provides a Runtime Safety Net
eBPF (extended Berkeley Packet Filter) is a Linux kernel technology that allows running sandboxed programs in response to system events. GitHub leveraged eBPF to monitor system calls (syscalls) made by deployment scripts. By attaching eBPF programs to syscall tracepoints, they could inspect every network connection, file access, and process execution in real-time. If a script attempted to access github.com (or any blacklisted endpoint), the eBPF program could block the call and log the violation. This worked at the kernel level, so even if a binary within the script tried to phone home, it couldn't bypass the filter. The eBPF programs were lightweight and could be dynamically loaded without restarting services, making them ideal for deployment safety.
5. Practical Implementation: Writing eBPF Programs for Deploy Safety
GitHub developed eBPF programs that specifically targeted syscalls like connect(), open(), and execve(). For instance, to block direct network connections to github.com, they used a kprobe on the connect syscall, checking the destination IP against a list of prohibited addresses. They also monitored execve to detect when a script tried to download a file via curl or wget. To handle hidden dependencies, they traced library calls using uprobes. The eBPF programs were packaged as small CO-RE (Compile Once – Run Everywhere) binaries for portability across kernel versions. Testing involved simulating outages in a staging environment, ensuring that legitimate operations (e.g., local log writes) were not impacted. The result: a robust runtime enforcement layer that catches circular dependencies automatically during deployment.
By adopting eBPF, GitHub moved from a reactive, manual review process to a proactive, kernel-enforced safety mechanism. This not only reduced the risk of deployment failures during incidents but also freed up engineering teams from tedious audits. As GitHub continues to evolve, eBPF offers a flexible foundation for further safety controls—like monitoring for unexpected DNS queries or blocking certain file writes. For any organization managing large-scale infrastructure, eBPF provides a powerful way to break dependency loops without modifying application code. The lesson is clear: sometimes the best safety net lies deep within the operating system.