The Internet Has a Security Blind Spot at Boot Time

Most security people focus on the steady state. Is the WAF rule tuned correctly? Is the SIEM catching the right events? Is the policy up to date? What they often miss is the window between “the process started” and “the process is ready.” In Kubernetes, that window can be a meaningful attack surface — and in the case of Fastly’s Next-Gen WAF (formerly Signal Sciences) NGINX Ingress integration, it was.

I found this. I reported it. Fastly fixed it. It shipped as generally available in March 2024. This is the story of how that happened, and why you should care if you’re running a WAF in a containerized environment.

What Fastly’s Next-Gen WAF Actually Is

Before getting into the vulnerability, some context.

Fastly’s Next-Gen WAF (originally Signal Sciences, acquired by Fastly in 2020) is a web application firewall designed around a module-plus-agent architecture. The module sits inside your web server (NGINX, Apache, IIS, etc.) and intercepts requests. The agent runs alongside it, holds your rules and configuration, and makes the actual allow/block decisions. The module and agent communicate over a Unix socket.

In Kubernetes, this often means a sidecar deployment: your NGINX ingress controller pod with the Signal Sciences module baked in, plus a sigsci-agent sidecar container sharing the socket path via a volume mount. The agent fetches your workspace rules from Fastly’s cloud, loads them, and begins inspecting traffic routed through the socket.

Fastly’s customer roster running this kind of configuration is not small. Their customer list includes GitHub, Stripe, Twitter, BuzzFeed, and Hearst — and those are the public names. They hold roughly 5% of global CDN market share, which sounds modest until you recognize that the CDN market processes trillions of requests daily. This is infrastructure that sits in front of real production traffic for real organizations.

The Problem: Traffic Moves Faster Than Rules Load

Here is the sequence of events when a pod starts in a normal Fastly NGWAF Kubernetes deployment, before the fix:

  1. Pod starts. Both containers — the NGINX ingress controller and the sigsci-agent — launch.
  2. The NGINX module loads. It begins accepting connections immediately because NGINX is ready.
  3. The sigsci-agent starts its initialization process: authenticate to the cloud, fetch defined workspace rules, load rule set into memory. This takes time — network round trips, rule parsing, configuration application.
  4. During step 3, the NGINX module is running but the agent hasn’t finished loading rules. Requests arrive, get passed to the agent socket, and the agent (not yet ready) passes them through without inspection.

That last sentence is the problem. The default behavior was fail-open. Traffic passed through a WAF that had not yet loaded the rules it was supposed to enforce. You can easily see where this could be DDOS’d and exploited: throw tons of traffic, forace a controller scale-up, and eventually some of your attacks get through.

This isn’t a theoretical edge case. Kubernetes pods restart. Deployments roll. HPA scales out new replicas under load. Every one of those events creates this window and the window was silent. No log entries. No alerts. No indication to operators that the WAF was running in passthrough mode. From the outside, the pod was healthy. NGINX was serving traffic. The agent process was running. Everything looked fine.

This is also a lesson in being curious about how things work, and looking at requests. I don’t recall, now a couple years later, what I was looking at when I found this, but I know how my mind works and trying to understand the “how” underbeath what most understand as “black box magic”.

Why Fail-Open Is the Wrong Default for a WAF

There’s a legitimate design argument for fail-open in some contexts. If your CDN or load balancer fails, you probably want traffic to keep flowing to your origin rather than dropping all connections. Uptime over security, in those cases, is a defensible choice.

A WAF is not that context.

A WAF exists specifically to inspect and enforce policy on traffic. Its entire value proposition is that it sits in the request path and makes decisions. If the default posture on startup is “let everything through,” then the WAF provides zero protection during every scale-up event, every rolling restart, and every pod eviction, which in a busy Kubernetes cluster can be continuous. An attacker who knows this can time exploitation attempts to coincide with deployment windows. Or they don’t even need to know, they just benefit from it accidentally.

The problem compounds in ingress controller deployments, where the WAF is typically the outermost security control. There may be nothing else between the internet and your application during that window.

What I Did With It

I didn’t file this publicly. I didn’t write a tweet thread. I worked with Fastly’s support team directly.

My concern was the same concern I’d have about any infrastructure I run on behalf of clients: I use this product, and I needed to understand whether the behavior I was seeing was intentional or a gap. The answer mattered for my own deployment decisions before it mattered for anything else. Ultimately, I wanted the security gap closed and my organization made better and safer because of it.

Fastly’s team was responsive. To this day, I will say they are some of the most responsive and competant front-line support I’ve ever worked with and truly set the bar. The communication was direct and professional. This is exactly the kind of behavior you want from a security vendor when someone raises an issue: no defensiveness, no dismissal, engagement with the substance of the problem.

The Fix: Startup Probes

The fix shipped in two parts. Agent 4.51.0 (February 2024) introduced the mechanism in beta. Agent 4.52.0 (March 14, 2024) marked general availability of the startup probe configuration options startup-probe-listener and startup-probe-filepath.

The mechanism integrates with Kubernetes’ native startup probe system. Two options:

HTTP probe: Set SIGSCI_STARTUP_PROBE_LISTENER to an address, and the agent exposes an HTTP endpoint. The agent endpoint will send an HTTP 503 response until the agent has loaded rules. After loading rules, the agent endpoint will send an HTTP 200 response. Kubernetes won’t mark the container ready until it gets a 200.

File probe: Set SIGSCI_STARTUP_PROBE_FILEPATH to a path, and the agent creates a file at that path once rules are loaded. A Kubernetes exec probe checks for the file’s existence.

In both cases, the effect is the same: Kubernetes holds the pod out of the load balancer rotation until the WAF agent is rule-loaded and ready to actually inspect traffic. The window goes from “unknown duration, silent” to “zero, enforced by the orchestrator.”

The configuration diff for the HTTP probe looks like this:

containers:
  - name: sigsci-agent
    image: signalsciences/sigsci-agent:latest
    ports:
      - name: "startup-port"
        containerPort: 2024
    env:
      - name: SIGSCI_STARTUP_PROBE_LISTENER
        value: "0.0.0.0:2024"
    startupProbe:
      httpGet:
        path: /startup
        port: startup-port
      failureThreshold: 90
      periodSeconds: 2

The failureThreshold: 90 with periodSeconds: 2 gives the agent up to 3 minutes to become ready before Kubernetes marks the container as failed — reasonable headroom for network latency in rule fetching.

One Important Caveat

This is opt-in. Fastly did not change the default behavior. If you are running the NGWAF agent in Kubernetes and have not added startup probe configuration, your pods are still starting in the same fail-open mode they always did. This is also a lesson in reading release notes, though I do think Fastly ought to have done a better job socializing this change to customers, albeit even internally that “This is an important change you should consider”.

Check your deployment manifests. Look for SIGSCI_STARTUP_PROBE_LISTENER or SIGSCI_STARTUP_PROBE_FILEPATH in your agent container env block. If they’re not there, you have the pre-fix behavior: that the WAF passes all traffic during startup, and Kubernetes has no way to know when it’s actually ready to enforce policy.

The fix exists. It’s documented. It’s been GA for over a year. It requires a few lines of YAML.

The Broader Pattern

This is not a Fastly-specific problem. It’s a Kubernetes-specific problem that surfaces in any security sidecar that needs to load configuration before it’s ready to enforce policy. The same failure mode can exist in:

  • Service mesh proxies (Envoy, Linkerd) that load remote policy
  • OPA sidecars that need to fetch policy bundles
  • Any WAF or IDS agent that authenticates to a cloud backend before becoming operational

The Kubernetes startupProbe primitive exists exactly for this use case: distinguishing “the process is running” from “the process is ready to do its job.” The gap occurs when security tools don’t hook into it, and platform teams don’t notice that those are two different things.

If you run any kind of policy-enforcing sidecar in Kubernetes, go look at your probe configuration right now. If your startup probe is just checking that the process is alive rather than that the policy is loaded, you probably have the same window.