At GitHub, we host our own source code on github.com—a classic dogfooding approach. But this creates a dangerous circular dependency: if GitHub goes down, we can't access the code needed to fix it. While we maintain a mirror and built assets for rollbacks, hidden dependencies persist in deployment scripts. To solve this, we turned to eBPF, a powerful kernel technology. This listicle explores the circular dependency problem and how eBPF helps us monitor and block risky calls during deployments, ensuring safer, more resilient operations.
1. The Circular Dependency Challenge
Imagine a MySQL outage takes GitHub offline. To fix it, you need to deploy a configuration change—but the deployment script itself relies on GitHub to pull tools or binaries. That's a direct circular dependency. Even with code mirrors, these loops can stall fixes. GitHub's architecture amplifies this risk because every deploy is internal. We needed a way to enforce dependency boundaries without breaking automation. eBPF became our answer, providing kernel-level visibility and control.
2. Why eBPF?
eBPF (extended Berkeley Packet Filter) allows running sandboxed programs in the Linux kernel without modifying kernel code. For deployment safety, it can intercept system calls, network requests, and file operations. Unlike traditional monitoring, eBPF is lightweight and dynamic—perfect for detecting unintended dependencies at runtime. We chose it to selectively block calls that would create circular dependencies (e.g., fetching from github.com during an outage). It's like a safety net that doesn't slow down normal operations.
3. Direct Dependencies: The Obvious Loop
A direct dependency occurs when a deployment script explicitly tries to pull a release or binary from GitHub—the very service that's down. For example, a MySQL fix script might run curl https://github.com/.../binary. Without safeguards, it fails. eBPF can monitor outgoing HTTP requests to github.com and block them during critical deployments, forcing the script to use local mirrors or cached assets. This breaks the loop instantly.
4. Hidden Dependencies: Sneaky Backdoors
Even if a script doesn't directly contact GitHub, a tool it calls might. A servicing tool on disk may check for updates online. It github.com is unreachable, the tool might hang or error out, stalling the entire deploy. These hidden dependencies are hard to catch in code reviews. eBPF traces all system calls—including those made by child processes—so you can see every network or file access. We can then block or fail-fast on any contact with external services that shouldn't be there.
5. Transient Dependencies: Chain Reactions
A deployment script might call an internal migration service via API. That service, in turn, tries to fetch a new binary from GitHub (transient dependency). The failure cascades back. eBPF's ability to track inter-process communication and system calls across different processes helps us map these chains. By instrumenting both the script and the services it invokes, we can set policies that forbid any unauthorized network activity during incident response.
6. The Role of eBPF Monitoring
Before blocking anything, we needed to understand what dependencies actually exist. eBPF monitoring gives us a real-time view of all system calls made during a deployment—DNS lookups, socket connections, file reads. We recorded traces from hundreds of deployments to build a baseline of allowed behaviors. This data revealed unexpected dependencies that went unnoticed for years, like a log shipper that checked GitHub for updates on every run.
7. Selective Blocking with eBPF
eBPF programs can not only observe but also act. Using eBPF's helper functions, we can drop packets, return errors to system calls, or kill processes that violate policies. For deployment safety, we block any outbound network connection to github.com unless explicitly allowed. We also block DNS queries for known external hosts during incidents. This selective blocking ensures normal operations continue while dangerous calls are prevented—no more deployment scripts creating circular loops.
8. Deployment to Production: What We Learned
Rolling out eBPF-based protection required careful testing. We started by deploying eBPF agents in simulation mode—monitoring only, no blocking. After identifying false positives (e.g., a legitimate cache update), we fine-tuned the rules. Eventually, we enabled blocking on a small set of production hosts. The results were promising: zero incidents caused by circular dependencies during the test period. Teams gained confidence that their scripts would not inadvertently rely on GitHub during outages.
9. Writing Your Own eBPF Programs
You can get started with eBPF using tools like bcc or libbpf. For deployment safety, focus on tracing network syscalls: connect, sendto, recvfrom. Write a simple program that checks the destination IP against a blocklist (e.g., GitHub's CIDR ranges). Attach it to the sys_enter_connect tracepoint. When a match occurs, return -EPERM to block the call. We'll share detailed examples in our documentation (available on our mirror, of course).
10. The Future of Deployment Safety at GitHub
eBPF is just one piece of a broader safety strategy. We're now exploring eBPF for runtime security policies, such as preventing deployments from accessing secrets they shouldn't. Combined with our existing code mirrors and rollback procedures, eBPF adds a critical layer of defense against circular dependencies. Our goal is to make deployment scripts self-contained and resilient—so that even if github.com is down, our engineers can fix it. eBPF helps us achieve that.
In conclusion, circular dependencies are a silent threat in any system that eats its own dogfood. GitHub's use of eBPF to monitor and block these loops during deployments demonstrates how kernel-level technology can enhance safety without sacrificing speed. By understanding direct, hidden, and transient dependencies, and by applying eBPF selectively, we've made our deployment process more robust. The lessons apply to any organization with complex automation—start tracing, identify patterns, and let eBPF be your safety net.