CELVEX Group
Compound Blackouts: When Three Defenders Are Simultaneously Blind for 72 Hours
1. Defense in depth is a layer cake. Cake gets stale.
Defense-in-depth is the only architectural meme in security that has survived four decades unscathed, and it has survived because the math underneath it is right. If defender A catches 60% of bad behaviour and defender B catches a different 60%, the union catches 84%. Stack five defenders with non-trivial independence and on paper you are over 99%. That paper math is the entire reason the modern blue-team budget looks the way it does: Wazuh plus Falco plus AIDE plus Velociraptor plus CrowdSec plus a CDN, with a SIEM swallowing the alerts and an EDR on every endpoint. Everyone owns a layer. Every layer claims a percentage. The percentages add.
The problem is that the math is set-theoretic and the operational reality is time-domain. Defender A catches 60% of bad behaviour when it is looking. Defender B catches its 60% when its rules have loaded, its feed is fresh, its hunt has fired in the last N hours, its eBPF probe is attached, and the kernel did not throw a -ENOMEM during the last reload. The independence assumption is a fiction the moment two defenders share an upstream telemetry source (auditd, eBPF tracepoints, a netflow exporter), or a cadence (the Tuesday-night hunt window), or an exclusion list (everyone skips /sys, /proc, /dev by default). When the failures are correlated, the union shrinks. When the cadences align, you get compound blackouts -- intervals during which three or more classes of detection are simultaneously asleep, and the attacker has a quasi-perfect operational window.
We built the Detection Window Auditor to measure compound blackouts directly on customer stacks, and we built the twelve ENDPOINT-STACK-CHAIN-NNN probes to surface the structural gaps that produce them. This post is the field report. The headline finding from the first month of production runs: a single customer's Velociraptor memory-hunt schedule produced a 4,320-minute compound blackout during which three detection classes were simultaneously blind. Three days. Seventy-two hours. The interval was not theoretical -- it was computed from the customer's own hunt manifest by the sweep-line algorithm we describe in section 4. We discuss what produced it, the eleven other cross-defender bypass primitives our probe family looks for, and how to compute compound blackouts on your own stack in the time it takes the manifest to download.
2. The 4,320-minute hole we found
ENDPOINT-STACK-CHAIN-007 is a banner-grade probe. It connects to the customer's Velociraptor frontend on TCP/8000 or TCP/8889, pulls the hunt history manifest through the API, and computes the maximum gap between successive memory-class hunts: Generic.Forensic.LocalHashes.Glob, Windows.Memory.ProcessInfo, Generic.Detection.HuntForFiles.Memory, and the half-dozen incubating artifacts in that family. The probe's contract is small: read-only, no exploit payload, deterministic, and the policy floor is 24 hours -- the test FAILs if the longest inter-hunt gap on memory-class artifacts exceeds one day. On the customer in question the longest gap was 4,320 minutes.
Why was it three days? Memory hunts are expensive. They walk every process on every endpoint in the hunted asset group, hash anonymous executable mappings, and ship the results back to the server. On a fleet of more than a few thousand endpoints the bandwidth and CPU bill alone forces operators to schedule hunts weekly rather than daily. The customer's posture was rational: a Monday-morning memory hunt at 04:00 UTC, a Thursday-afternoon hunt at 18:00 UTC, and an emergency hunt slot reserved for incident response. From Thursday at 18:00 to Monday at 04:00 there is a hard 58-hour gap on the calendar. Schedule drift -- the hunt actually fired at 18:23 because the previous Thursday's hunt overran its 30-minute budget -- pushed the gap to 58 hours 23 minutes. A worker restart race during the Friday-evening Velociraptor server upgrade silently dropped the queued hunt run that was supposed to fire at 02:00 Saturday on a small subset of clients, and the operator believed the catch-up logic had reissued it. It had not. The catch-up scheduler treats a missing hunt as "skipped" rather than "queued for retry" and emits a warning the SOC dashboard had been silencing for six months because nobody wanted the noise.
The compound calculation is what makes this finding worth shipping. During those 72 hours the customer believed they were covered: AIDE was running its 12-hour scan cycle, Wazuh syscheck was on a 6-hour cadence, Falco was watching syscalls, CrowdSec was on the edge. But the union of those defenders' visibility, projected against the in-memory adversarial primitive class (T1620 Reflective Code Loading), is empty. AIDE and Wazuh syscheck are file-integrity tools and never see in-memory state. Falco's default rule corpus is anchored to execve and execveat -- it sees the loader execute, but a reflectively-loaded payload that is fexecved from a memfd never raises an exec event with a hashable on-disk path. CrowdSec is at the edge and watches HTTP. The only defender on the stack that could detect the reflective loader's anonymous executable mapping is Velociraptor's memory hunt, and Velociraptor's memory hunt was the one that did not fire for 72 hours. Three classes -- syscall (Falco), FIM (AIDE + Wazuh), and hunt (Velociraptor) -- were simultaneously blind to the same TTP class. Sweep-line algorithm output: one CompoundBlackout record, severity critical, duration 259,200 seconds, classes_blind {syscall, fim, hunt}.
The cadence floor is the visible problem. The deeper problem is that every customer who runs Velociraptor at fleet scale has a version of this gap, because Velociraptor's economics force them to. The remediation guidance is mechanical -- schedule a memory-class hunt at least every four hours on the crown-jewel asset group, and pair it with event-monitored client-side artifacts so the blind window is bounded by client poll interval rather than hunt interval -- but the customers we have shipped this to all responded with the same observation: until the probe surfaced the compound blackout in those exact words, with that exact duration, against that exact TTP class, the gap was not legible to their leadership. Memory-hunt cadence was just a knob in a YAML file.
3. Twelve cross-defender bypass primitives
ENDPOINT-STACK-CHAIN-007 is one probe in a family of twelve. Each probe targets a structural gap that emerges when a customer composes two or more defenders from the canonical Linux blue-team stack and discovers, after the fact, that the intersection of their blind spots is non-empty. We walk through all twelve below. None of them executes an exploit; every one is a banner, port-set, or configuration introspection that establishes the gap and lets the customer remediate before adversaries arrive. The threat model behind each one is documented research, not vendor speculation.
ENDPOINT-STACK-CHAIN-001 -- Wazuh + Falco fingerprint composition. The seed probe. It fingerprints the externally-observable surface of the customer's stack: Wazuh manager API on TCP/55000, Wazuh indexer on TCP/9200, Falco gRPC on TCP/5060, Falcosidekick on TCP/2801, plus the X-Wazuh-Version banner and Falco's HTTP/2 SETTINGS-frame fingerprint. None of the downstream composition probes fire blindly; they require an anchor reachable from the scanner before they emit a verdict. The risk class is INFO; the value is that everything else in the family routes off this fingerprint.
ENDPOINT-STACK-CHAIN-002 -- io_uring audit gap (Wazuh + Falco both miss). The most under-detected Linux telemetry hole of the 2024-2026 period. Both auditd-anchored Wazuh decoders and pre-CO-RE Falco rule sets miss read(2) and openat(2) issued through the io_uring submission queue, because those operations never raise the syscall tracepoint the legacy probe is watching. ARMO's "Bad io_uring" disclosure (2025) and Sysdig's April 2025 follow-up both demonstrated end-to-end credential theft via io_uring with zero auditd or Falco events. The probe verifies (a) Wazuh manager API reachable, (b) auditd policy mirror lacks any io_uring_setup / IORING_OP_READ / IORING_OP_OPENAT rule, (c) Falco rule corpus mirror lacks any io_uring evt.type rule. T1562.001 -- the tools never see the activity. Real-world prior art: TripleCross (the eBPF rootkit research project, github.com/h3xduck/TripleCross) demonstrated that io_uring-style submission-queue evasion was operationally viable years before the upstream tools shipped detection.
ENDPOINT-STACK-CHAIN-003 -- memfd + fexecve evades exec-anchored rules. The fileless-execution gap. Stack components anchored on execve(2) and execveat(2) -- the dominant Falco / Wazuh / auditd default ruleset -- see /proc/PID/exe resolve to memfd:/(deleted) and the binary is never written to disk for FIM to hash. Symbiote (Intezer, 2022) and BPFDoor (PwC Threat Intelligence, 2022) are the canonical real-world Linux malware families that abused memfd-style fileless residency at scale; the technique is now standard in mid-tier offensive tooling. The probe verifies that the customer's Falco rule corpus references neither memfd_create nor fd.name contains 'memfd:', that the Wazuh ruleset mirror lacks a corresponding decoder, and that auditd rules do not log memfd_create. T1620 Reflective Code Loading. Combined with primitive 002, this is near-complete user-space invisibility on a default Wazuh+Falco stack.
ENDPOINT-STACK-CHAIN-004 -- AIDE + Samhain + Wazuh all skip /sys/fs/bpf. The eBPF-rootkit persistence gap. AIDE, Samhain, and Wazuh syscheck all ship default configurations that exclude /sys, /proc, and /dev from FIM. Pinned eBPF programs and maps under /sys/fs/bpf are therefore never hashed, never alerted on, and a pinned malicious eBPF program survives reboot in kernel space with no on-disk evidence. The probe verifies that all three FIM configurations omit /sys/fs/bpf and recommends adding bpftool prog show and bpftool map show watchlist comparison. T1547.006 Kernel Modules and Extensions. Three-FIM coverage gap on a single technique class.
ENDPOINT-STACK-CHAIN-005 -- systemd generators (tmpfs, never on disk at scan). Generators under /run/systemd/generator{,.early,.late} are executed at boot to materialise transient unit files. They live on tmpfs and are removed before any periodic FIM scan ever sees them. An attacker with root drops a malicious generator that writes a service unit, schedules it for execution, and the artifact is gone before AIDE / Samhain / Wazuh syscheck next runs. The probe verifies that none of the three FIM configurations covers /run/systemd/generator*. T1543.002 systemd service persistence with no FIM evidence in any of three FIM products.
ENDPOINT-STACK-CHAIN-006 -- longest scan-window across stack = max attacker freedom. The race the customer always loses. If AIDE runs every 24 hours, Samhain every 12 hours, and Wazuh syscheck has a 6-hour frequency on a critical directory, the actual blind window for create-modify-delete tampering is the LCM-adjusted gap, which often exceeds 4 hours of attacker freedom. The probe reads each FIM's periodicity config, computes the longest gap, and FAILs at the 60-minute policy floor. T1070.006 Indicator Removal. The remediation is not "add more FIM" -- the customer has three -- but "move every FIM to realtime/inotify on the critical-directory set," which is operational discipline, not procurement.
ENDPOINT-STACK-CHAIN-007 -- Velociraptor memory-hunt cadence (the 72-hour hole). Discussed in section 2. The probe reads the hunt manifest, identifies memory-class artifacts, computes the longest gap, and FAILs at the 24-hour policy floor. T1620 Reflective Code Loading. The 4,320-minute finding was real.
ENDPOINT-STACK-CHAIN-008 -- CrowdSec + Fail2Ban + Cloudflare threshold walk. Each of three rate-limit defenders has its own per-IP threshold; the minimum of the three is the floor an attacker must stay under to credential-stuff without tripping any decision. The probe pulls the Cloudflare zone rate-limit policy via API mirror, the CrowdSec LAPI threshold from /v1/decisions, and the Fail2Ban jail config's maxretry for the auth jail. T1110.003 Password Spraying. If the floor is generous (more than 10 attempts per minute per IP), low-and-slow stuffing from a modest IP pool is undetected. The 23andMe credential-stuffing breach (2023) is the canonical demonstration that a generous floor against a large rotating IP pool produces months of undetected access at scale.
ENDPOINT-STACK-CHAIN-009 -- IPv6 /128 ban vs /64 attacker pool. CrowdSec, Fail2Ban, and most CDN rate-limit defaults ban a single /128. An attacker who controls a /64 (a single residential IPv6 allocation typically grants a /56 or /60 of /64 subnets) has 2^64 addresses to rotate through. The probe verifies that decision scopes are /128 across all three defenders rather than /64-aggregated, and recommends switching CrowdSec scope to ipv6 mask 64, Fail2Ban actionban to /64 aggregation, and Cloudflare rate-limit characteristic to ip.src.subnet ('/64'). T1090 Proxy. The /128 default renders rate-limit posture meaningless for IPv6-reachable surfaces.
ENDPOINT-STACK-CHAIN-010 -- GraphQL introspection through CF + CrowdSec. Most WAF managed rules block legacy SQLi and XSS shapes but pass anonymous GraphQL __schema queries because the query string is alphanumeric and the body is JSON. The probe sends a non-recursive { __schema { types { name } } } introspection through the CDN to the customer's /graphql or /api/graphql endpoint, verifies a 200 response containing the type list, and confirms no WAF block or CrowdSec decision fires. T1213 Data from Information Repositories. Combined with primitive 008's rate-limit floor, an attacker authors IDOR-shaped mutations from the schema and runs them under threshold.
ENDPOINT-STACK-CHAIN-011 -- etcd:2379 reachable bypassing apiserver audit. The Kubernetes apiserver enforces RBAC and produces audit events; etcd is the underlying KV store with no audit and no RBAC. If etcd:2379 or 2380 is reachable from outside the master VLAN -- a misconfigured ELB, an accidentally-exposed control plane, mTLS not enforced -- an attacker reads or writes every Secret directly with zero apiserver audit trail. The probe verifies TCP/2379 reachability, that the TLS handshake succeeds without a client certificate (or without TLS at all, which we treat as critical), and that /version or /v3/cluster returns. T1552.007 Container API Credentials. The Capital One SSRF breach (2019) is the canonical analogue: a metadata-service misconfiguration produced full cloud account compromise that bypassed every higher-level audit and RBAC layer the operator had configured. etcd direct-path is the in-cluster equivalent.
ENDPOINT-STACK-CHAIN-012 -- QUIC / HTTP/3 egress unmonitored. Wazuh netflow decoders, Falco network rules, and most NetFlow exporters are TCP-anchored and parse Layer-7 by ALPN. QUIC traffic on UDP/443 presents as encrypted UDP and the analytics layer cannot reconstruct method, host, or path without a QUIC-aware probe. An attacker uses a publicly-trusted HTTP/3 endpoint -- Cloudflare workers, Fastly's H3 layer -- for C2 with zero L7 evidence in the customer's monitoring. The probe verifies that the Wazuh decoder mirror lacks any quic / h3 / udp-443 rule, that the Falco network rule corpus lacks fd.lport=443 and fd.l4proto=udp coverage, and that the customer's egress firewall mirror does not block UDP/443 outbound. T1071.001 Application Layer Protocol: Web Protocols. Twenty-five to forty percent of modern browser traffic is QUIC; the same channel is now an unmonitored exfil path on every TCP-anchored stack we audit.
4. Measuring compound blackouts
The probes surface structural gaps. The Detection Window Engine surfaces the time-domain blackouts those gaps produce, and it does so with arithmetic the customer can audit. The engine is a pure in-memory analyser -- zero I/O, deterministic, fully unit-testable -- exposing a single sweep-line entry point that computes compound blackouts above a configurable class-count threshold over a trailing horizon.
The taxonomy is six classes -- FIM, syscall, vuln, cloud, hunt, IDS -- chosen coarse enough that timeline arithmetic is tractable and fine enough that the resulting blackouts map cleanly onto MITRE technique families. Each class carries a static TTP map with at least four ATT&CK techniques attached so that when the engine reports a blackout we surface the corresponding techniques in the customer narrative: "during this 47-minute FIM blackout T1547.001, T1543.003, T1574.011, T1222, T1070.004 were undetectable" rather than a raw timestamp.
The algorithm is sweep-line. Per class, contiguous coverage events inside the trailing horizon are merged and the inter-interval gaps are emitted as blackout records. The blackout records across all classes are folded onto a single timeline and a left-to-right sweep tracks the active blind set, opening a compound window whenever the blind-set count crosses the minimum-class threshold and closing it when the count drops back. The witness set accumulates every class that was blind at any point during the open window so the report names every contributor, not just the one that closed it. Severity is bucketed by duration on industry-standard alarm boundaries -- greater than 600 seconds is critical, greater than 60 seconds is high, greater than 5 seconds is medium, otherwise low. Total complexity is O(N log N) on event count. On the customer with the 72-hour Velociraptor gap, the sweep produced one record in well under 10 milliseconds.
To compute compound blackouts on your own stack you need three inputs per defender: the configured cadence, the rule corpus, and a recent run history (the last 24 hours suffices for most classes; vuln-feed and hunt cadence demand 7-30 days). The engine accepts WindowEvent records of (cls, start_ts, end_ts, reason) triples that describe contiguous intervals of active coverage; build them from your scheduler's run log (Wazuh syscheck wall-clock cycles), your hunt manifest (Velociraptor hunt-fire timestamps), your CDN's rule-pack version pin events, and your eBPF probe-attach log. Run compute_compound_blackouts(min_classes=3) against the populated engine. Anything severity == critical is a 10-minute-or-more window during which three of your six detection classes could not see an attacker. Anything severity == critical with duration_seconds > 3600 is an hour-or-more window. Anything duration_seconds > 86400 is a day-or-more window and should be a board-level finding. Plot the result on the customer dashboard alongside the static MITRE coverage heatmap and you have coverage when sitting next to coverage what for the first time.
5. Closing
Defense-in-depth is a real architectural pattern with real arithmetic underneath it, but the arithmetic that matters is time-domain rather than set-theoretic. Five defenders whose blackout windows align produce stretches of operational reality during which the union shrinks to nothing. The 4,320-minute Velociraptor finding is one customer; the family of structural gaps that produce compound blackouts is universal. Every Linux blue-team stack we have audited this quarter has had at least one critical-severity compound blackout, and the median duration is 4 hours 18 minutes.
The fix is not procurement. The customers we ship these findings to already own everything they need. The fix is operational discipline -- realtime FIM on critical directories, four-hour memory-hunt cadence on crown-jewel assets, /64 IPv6 ban scope, UDP/443 egress block or QUIC-aware proxy, etcd on the management VLAN behind mTLS -- plus a measurement habit that admits the time domain exists. Compound blackouts are a category of finding the industry has not been measuring because the math was awkward. The math is not awkward; the engine is 400 lines of Python. Run it on your stack. Publish the duration of your worst window next to your ATT&CK coverage percentage. The first one will surprise you.