Multi-Layer Packet Processing Pipeline for DDoS Mitigation

Part of the Mirai Guard open research series on multi-layer DDoS defense architecture.

The fundamental principle behind effective DDoS mitigation is deceptively simple: drop malicious traffic as early as possible, as cheaply as possible. Every processing cycle spent on a packet that will ultimately be discarded is wasted CPU time — and under a sustained flood, wasted CPU time is how systems collapse.

This document describes the packet processing pipeline used in this mitigation architecture, from the moment a packet arrives at the network interface to the moment it either reaches a backend service or is dropped. Each stage in the pipeline has a specific role, a specific cost, and a specific set of threats it addresses. Understanding why the pipeline is structured this way matters as much as understanding what each layer does.

The Core Design Principle: Fail Early, Fail Cheap

The cost of processing a packet increases dramatically as it moves deeper into the network stack. An XDP drop at the driver level costs on the order of 100 nanoseconds and consumes no kernel memory. A drop after TCP connection establishment has already allocated kernel buffers, a conntrack entry, and potentially triggered application code — orders of magnitude more expensive.

This asymmetry defines the entire architecture. The pipeline is designed to front-load the cheapest, coarsest filters at the earliest possible point, progressively applying more sophisticated (and more expensive) analysis to the traffic that survives each stage. By the time traffic reaches the application layer, it has already been validated at multiple lower levels.

Architecture Overview

Internet
    │
    ▼
┌─────────────────────────────────┐
│   Upstream Volumetric Scrubbing │  ← OVH / Transit provider
│   (L3/L4 flood absorption)      │
└─────────────────┬───────────────┘
                  │ Filtered transit
                  ▼
┌─────────────────────────────────┐
│   NIC / Network Driver          │
│   (Hardware RX queues, RSS)     │
└─────────────────┬───────────────┘
                  │
                  ▼
┌─────────────────────────────────┐
│   XDP Program (eBPF)            │  ← Pre-stack, kernel bypass
│   IP filtering, PPS limiting,   │
│   early drop                    │
└─────────────────┬───────────────┘
                  │ XDP_PASS
                  ▼
┌─────────────────────────────────┐
│   Linux Kernel Networking Stack │
│   TCP/IP, SYN handling,         │
│   socket buffers, conntrack     │
└─────────────────┬───────────────┘
                  │
                  ▼
┌─────────────────────────────────┐
│   Netfilter / nftables          │  ← Stateful firewall
│   Rate limiting, IP sets,       │
│   protocol validation           │
└─────────────────┬───────────────┘
                  │
                  ▼
┌─────────────────────────────────┐
│   Reverse Proxy (HAProxy/Nginx) │  ← L7 analysis
│   HTTP inspection, connection   │
│   limits, behavioral filtering  │
└─────────────────┬───────────────┘
                  │ Clean traffic only
                  ▼
┌─────────────────────────────────┐
│   Backend Origin Services       │
└─────────────────────────────────┘

Stage 1: Upstream Volumetric Scrubbing

Before a packet reaches the mitigation server, it has already passed through upstream infrastructure — in this architecture, OVH’s transit network, which provides volumetric DDoS scrubbing at the network edge.

This upstream layer is designed to absorb attacks that would saturate the mitigation server’s physical network interface before any software-level filtering could run. A 100 Gbps UDP flood is not a software problem — no amount of XDP programming will save a server whose NIC is receiving more traffic than its bus can transfer to RAM. Upstream scrubbing handles this category of attack.

What upstream scrubbing addresses:

• Volumetric UDP floods exceeding interface capacity
• ICMP floods
• IP spoofing at scale
• Basic L3/L4 amplification attacks (NTP, DNS, memcached reflection)

What it does not address:

• Sophisticated L7 attacks using valid HTTP requests
• Low-and-slow attacks that don’t trigger volumetric thresholds
• Attacks that deliberately stay below scrubbing detection limits
• Application-specific abuse patterns

The upstream layer is a prerequisite, not a solution. Traffic arriving at the mitigation server has survived volumetric filtering — which means it looks legitimate enough at the network layer to pass. The remaining stages handle what the upstream layer cannot.

Stage 2: NIC Reception and Hardware Offload

Packets that pass upstream filtering arrive at the mitigation server’s Network Interface Card. At this point the packet exists only as bytes in NIC memory — it has not touched the CPU, has not been processed by the kernel, and has consumed no software resources.

Modern NICs provide several hardware features that are critical for high-PPS operation:

Receive Side Scaling (RSS) distributes incoming packets across multiple hardware receive queues based on a hash of the packet’s flow tuple (source IP, destination IP, source port, destination port). Each queue is assigned to a different CPU core, enabling true parallel packet processing. Without RSS, all packets hit a single CPU core regardless of how many cores the system has — a bottleneck that becomes catastrophic under high PPS.

Hardware interrupt coalescing batches multiple packets into a single interrupt notification rather than triggering an interrupt per packet. Under a flood, per-packet interrupts would consume the entire CPU budget on interrupt handling before any actual filtering runs.

Large Receive Offload (LRO) / Generic Receive Offload (GRO) coalesces multiple packets belonging to the same TCP flow into a single larger buffer, reducing the number of individual packets the kernel must process. This is beneficial for legitimate high-throughput traffic but can interfere with per-packet analysis in XDP programs — the tradeoff must be evaluated based on the specific workload.

The NIC forwards received packets to the driver’s receive ring buffer. This is the attachment point for XDP programs.

Stage 3: XDP Processing — The Primary Defense Layer

XDP (eXpress Data Path) is the most critical stage in this pipeline for high-PPS attack scenarios. An XDP program is a verified eBPF program that executes directly in the network driver’s receive path — after the NIC delivers a packet but before the kernel allocates an sk_buff (socket buffer) structure for it.

This timing is everything.

Allocating an sk_buff is not free. It requires memory allocation, field initialization, and engagement of the kernel's networking subsystem. Under a 10 Mpps flood, the overhead of allocating and freeing sk_buff structures for packets that will ultimately be dropped can consume significant CPU resources. XDP drops packets before this allocation occurs — the packet is examined and discarded directly from the DMA memory region where the NIC placed it.

XDP Return Codes

An XDP program returns one of the following verdicts for each packet:

Return CodeEffectXDP_DROPDrop the packet immediately. Zero kernel involvement.XDP_PASSPass to normal kernel networking stack.XDP_TXTransmit back out the same interface (useful for reflection defense).XDP_REDIRECTRedirect to another interface or CPU queue.XDP_ABORTEDDrop with error trace (debugging only).

What XDP Handles in This Architecture

IP-based filtering — The XDP program maintains a BPF map (hash map in kernel memory) containing known malicious IP addresses and CIDR ranges. For each incoming packet, the source IP is looked up in this map. Hits result in immediate XDP_DROP. Map updates are performed from userspace without reloading the XDP program, enabling real-time blocklist updates with zero downtime.

Protocol validation — Basic sanity checks on packet structure: minimum length validation, IP header checksum verification, protocol field checks. Malformed packets are dropped before they can trigger parsing bugs in upper layers.

PPS rate limiting per source IP — The XDP program maintains a per-IP packet counter in a BPF map with a sliding window. Sources exceeding a configurable PPS threshold are dropped. This handles single-source floods where the source IP is real (not spoofed) — common in botnet attacks where each bot has a legitimate IP.

TCP flag filtering — Packets with invalid TCP flag combinations (e.g., SYN+FIN, SYN+RST) are dropped immediately. These are either crafted attack packets or scanning activity and have no legitimate use.

SYN flood mitigation — For pure SYN floods, the XDP program can implement SYN cookie validation at the driver level, bypassing the kernel’s own SYN handling entirely for suspicious sources.

XDP Program Attachment Modes

Native XDP runs directly in the NIC driver, providing maximum performance. This requires driver support — not all drivers implement the XDP hooks. When available, native XDP is always preferred.

Generic XDP (also called SKB-mode XDP) runs after sk_buff allocation, losing the primary performance benefit of XDP. It provides compatibility with drivers that don't support native XDP but should be avoided on production mitigation nodes.

Offloaded XDP executes the eBPF program on the NIC’s SmartNIC processor, offloading packet processing from the host CPU entirely. This provides the highest possible performance but requires SmartNIC hardware support (Netronome, Mellanox ConnectX-6 Dx and later).

Stage 4: Linux Kernel Networking Stack

Packets that return XDP_PASS enter the Linux kernel's networking stack. This is where the bulk of protocol processing occurs: sk_buff allocation, TCP/IP parsing, connection tracking, and socket delivery.

Get Lenny Alexandra’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

At high PPS, this stage can become a bottleneck even after XDP filtering — either because the XDP filter doesn’t catch everything, or because legitimate traffic volume itself is high. The kernel networking parameters described in detail in the companion document (Kernel Networking Tuning for High-PPS DDoS Mitigation) are essential here:

• net.core.netdev_max_backlog must be large enough to absorb burst traffic before the softirq handler processes it
• net.ipv4.tcp_max_syn_backlog and somaxconn must be sized to handle connection bursts without dropping legitimate SYN packets
• Conntrack table limits must be configured to prevent table exhaustion from causing blanket connection drops

The key interaction between XDP and the kernel stack is that XDP should have already dropped the highest-volume attack traffic before it reaches this stage. The kernel stack should be handling traffic that XDP could not classify as definitively malicious — which means more sophisticated analysis is required.

Stage 5: Netfilter / nftables

Packets that pass through the kernel stack reach the Netfilter framework, where stateful firewall rules are applied. This architecture uses nftables for its rule management, which provides several advantages over legacy iptables for high-PPS scenarios:

• Atomic rule replacement (no ruleset flush-and-reload that causes traffic drops during updates)
• Native set support with hash and interval types optimized for large IP blocklists
• Combined address family handling (inet tables cover both IPv4 and IPv6)
• Better performance with fewer kernel calls for complex rulesets

What nftables Handles That XDP Cannot

XDP programs have a constrained execution environment — they cannot maintain complex state across packets, cannot perform DNS lookups, and have limited access to connection-level information. Netfilter handles the stateful analysis that XDP cannot:

Connection rate limiting — Using nft's meter statement, connection rates can be limited per source IP across a sliding time window. This handles distributed attacks where each source IP individually stays below the XDP per-IP PPS limit, but collectively generate excessive connection attempts.

Protocol-specific validation — Netfilter’s ct (connection tracking) module enables rules based on connection state (NEW, ESTABLISHED, RELATED). Rules can restrict which protocol states are accepted from which sources, blocking attacks that exploit connection state transitions.

IP set management — Large blocklists (tens of thousands of CIDR ranges) are managed via nftables sets with hash type, enabling O(1) lookup performance regardless of blocklist size.

Geo-based filtering (optional) — When origin traffic analysis indicates attacks originating from specific ASNs or geographic regions not relevant to the protected service, coarse geographic filtering can be applied at this stage.

nftables Rule Structure

The general rule priority in this architecture follows a principle of cheapest checks first:

table inet filter {
    chain input {
        type filter hook input priority 0; policy drop;
        
        # 1. Allow established/related (cheap conntrack lookup)
        ct state established,related accept
        
        # 2. Drop invalid state
        ct state invalid drop
        
        # 3. Blocklist check (hash set, O(1))
        ip saddr @blocklist drop
        
        # 4. Rate limiting (new connections per source)
        ct state new meter conn_rate { ip saddr limit rate 100/second } accept
        
        # 5. Protocol-specific rules
        tcp dport { 80, 443 } accept
        
        # Default: drop (set in policy)
    }
}

Stage 6: Reverse Proxy — Layer 7 Analysis

Traffic that passes all previous stages reaches userspace via a reverse proxy — HAProxy or Nginx in this architecture. This is the most expensive filtering stage, as it involves full TCP connection establishment, HTTP parsing, and application-level logic. It is also the only stage capable of analyzing HTTP semantics.

Why a Reverse Proxy for L7 Filtering

The kernel cannot inspect HTTP request bodies, evaluate User-Agent strings, or analyze URL patterns — these operations require a full HTTP parser running in userspace. The reverse proxy provides this capability while also serving as the TLS termination point.

HAProxy is preferred for this role in high-connection-rate scenarios due to its event-driven architecture and ability to handle tens of thousands of concurrent connections with low per-connection overhead. Key L7 filtering capabilities:

Connection limiting — maxconn per frontend and per backend controls prevent any single source from exhausting connection capacity. Combined with stick-table entries that track per-IP connection counts and rates, this enables dynamic rate limiting based on observed behavior.

HTTP request validation — Requests with malformed HTTP structures, missing mandatory headers, or suspicious header combinations can be rejected at the proxy level before reaching the application.

Slow request detection — HAProxy’s timeout configuration (timeout http-request, timeout client) detects and terminates Slowloris-style attacks that attempt to hold connections open indefinitely by sending headers slowly.

Backend health checking — The proxy continuously monitors backend health and removes unhealthy backends from rotation, preventing a stressed backend from receiving additional traffic that could cause cascading failure.

Stage 7: Backend Origin Services

Traffic that survives all six preceding stages reaches the backend origin servers. At this point, the traffic population has been filtered through:

1. Volumetric scrubbing at the network edge
2. IP-based and PPS-based filtering at the XDP layer
3. Protocol validation and connection rate limiting at Netfilter
4. HTTP-level behavioral analysis at the reverse proxy

The backend should only be receiving traffic that looks legitimate across all of these dimensions simultaneously. This doesn’t mean the traffic is guaranteed to be legitimate — sophisticated attacks can survive all filtering stages — but it means the attack surface exposed to the origin has been dramatically reduced.

Performance Characteristics

The layered pipeline has different cost profiles at each stage:

StageApproximate cost per packetState maintainedUpstream scrubbingExternal, zero local costNone on serverNIC reception~50 ns (hardware)RX ring bufferXDP drop~100–200 nsBPF mapsKernel stack drop~500 ns — 2 µssk_buff, conntrackNetfilter drop~1–5 µsconntrack, setsProxy rejection~50–500 µsFull TCP connection

This table illustrates why front-loading drops at XDP is critical. A packet dropped at XDP costs roughly 100 nanoseconds. The same packet dropped after reaching the reverse proxy costs three to four orders of magnitude more. Under a sustained 5 Mpps flood, the difference between XDP drops and proxy-level drops can be the difference between a functioning system and a crashed one.

Limitations and Attack Vectors This Pipeline Does Not Address

Documenting limitations is as important as documenting capabilities.

Distributed L7 attacks from legitimate IPs — If an attacker controls a botnet of real residential IPs, each sending requests at a rate that looks legitimate individually, the XDP and Netfilter rate limiters will not trigger. Detection at the proxy layer via behavioral analysis (request pattern similarity, URL targeting) is the primary defense, but sophisticated attackers can randomize these signals.

TLS handshake floods — TLS negotiation is computationally expensive. A flood of TLS ClientHello messages that never complete the handshake consumes significant CPU before reaching any application-level filtering. Hardware TLS offload or dedicated TLS termination infrastructure is required to address this at scale.

Spoofed source IPs — XDP IP blocklists are ineffective against attacks using spoofed source addresses. SYN cookie validation at the XDP level provides some protection for TCP-based attacks, but UDP-based attacks with spoofed sources require upstream scrubbing for effective mitigation.

Slow L7 attacks targeting specific endpoints — Attacks that target computationally expensive application endpoints (heavy database queries, search functionality) at low request rates can cause application exhaustion without triggering any rate limits. These require application-level caching, query optimization, and endpoint-specific rate limiting.

Relationship to the Full Architecture

This packet processing pipeline is one component of a broader mitigation architecture. The full system design, including the threat model that informed these filtering choices and the XDP program design patterns, is documented in the companion files in this repository:

• architecture.md — System architecture and component relationships
• kernel-tuning.md — Kernel parameter tuning for high-PPS workloads
• threat-model.md — Attack taxonomy and mitigation mapping

Alexandra Corbero is the founder of Mirai Guard, a free DDoS consultation and victim assistance platform. The full mitigation architecture is documented at github.com/alexandra-idc/ddos-mitigation-architecture.