Packet Flow: NIC, DMA, Rings, Kernel, XDP, and DPDK

If you peer at a NIC datasheet (say, the Intel 82599 for 10 GbE, or the Mellanox ConnectX-5 for 100 GbE), you find that almost all of the “NIC” is just a programmable controller exposing two things to the host:

1. A bank of memory-mapped registers (the MMIO region, also called the BAR space).
2. A description of what the NIC will do if you, the driver, configure those registers a certain way.

PCI/PCIe devices advertise their MMIO regions via Base Address Registers (BARs). At enumeration time, the kernel reads each device’s PCI configuration space and learns: “this device wants 128 KiB of MMIO mapped somewhere”. The kernel picks a physical address range, programs the BARs, and then maps that range into kernel virtual address space with ioremap(). From then on, when the driver writes to a particular kernel virtual address, the write actually goes out over PCIe as a Memory Write Transaction Layer Packet (TLP) addressed to the NIC.

You can see this in any driver. The ixgbe probe path does roughly:

/* drivers/net/ethernet/intel/ixgbe/ixgbe_main.c, simplified */
err = pci_request_mem_regions(pdev, ixgbe_driver_name);
hw->hw_addr = ioremap(pci_resource_start(pdev, 0),
                      pci_resource_len(pdev, 0));

After this, hw->hw_addr is a pointer into kernel virtual memory. Writes through it land in the NIC’s register file. The NIC sees those writes the same way a CPU “sees” a normal memory write: a transaction shows up on its internal fabric and updates a register.

Two things to internalize:

• MMIO is host-to-device. When the driver wants to tell the NIC something — “here are buffers for you”, “please start”, “process the queue at index N” — it writes to an MMIO register.
• DMA is device-to-host. When the NIC wants to write packet data into RAM, it issues a PCIe write directly to a host physical address. The CPU is not involved per byte.

That asymmetry is the spine of the whole story. Everything that follows is the choreography to make MMIO and DMA cooperate.

A NIC is, by itself, completely ignorant about the host’s memory layout. From its point of view, RAM is a 64-bit address space, and it can write anywhere it has been given an address for. The driver must explicitly hand it addresses. Without that, the NIC has no business poking at host memory and the IOMMU (or the lack of one) will catch it if it tries.

There are three relevant pieces:

1. Buffers. Plain chunks of physical memory the kernel has allocated, big enough to hold a packet.
2. Descriptors. Small fixed-size structures, also in host memory, that contain a DMA address pointing at a buffer plus a few control/status bits. One descriptor per buffer.
3. A ring of descriptors. A contiguous array of descriptors, with head/tail pointers, treated as a circular queue.

The driver’s bargain with the NIC is:

“Here is an array of N descriptors at physical address D. Each descriptor points to a buffer of size B. When you receive a frame, write it into the buffer pointed to by the next descriptor I haven’t reclaimed yet, then mark that descriptor done.”

The descriptor itself is a tiny on-the-wire data structure that both parties agree on. For Intel’s e1000 family, the legacy RX descriptor is in drivers/net/ethernet/intel/e1000/e1000_hw.h:

/* simplified */
struct e1000_rx_desc {
    __le64 buffer_addr;     /* DMA address of buffer */
    __le16 length;          /* set by NIC */
    __le16 csum;            /* set by NIC */
    u8     status;          /* DD, EOP, VP, etc. — set by NIC */
    u8     errors;          /* set by NIC */
    __le16 special;
};

Sixteen bytes. The driver fills in buffer_addr (and writes zeros to the rest). The NIC fills in everything else when a packet lands.

For the 82599 (ixgbe), an “advanced” receive descriptor is also 16 bytes, but the layout is reused for two purposes: before the NIC writes to it, it’s a read format (the driver wrote a DMA address); after, it’s a write-back format (the NIC wrote length, RSS hash, checksum, vlan tag, status):

/* drivers/net/ethernet/intel/ixgbe/ixgbe_type.h */
union ixgbe_adv_rx_desc {
    struct {
        __le64 pkt_addr;       /* read format: buffer DMA addr */
        __le64 hdr_addr;       /* optional header buffer       */
    } read;
    struct {
        struct {
            union {
                __le32 data;
                struct { __le16 pkt_info, hdr_info; } hs_rss;
            } lo_dword;
            union {
                __le32 rss;
                struct { __le16 ip_id; __sum16 csum; } csum_ip;
            } hi_dword;
        } lower;
        struct {
            __le32 status_error;
            __le16 length;
            __le16 vlan;
        } upper;
    } wb;  /* write-back format */
};

The two layouts overlap in the same 16 bytes. It is a tagged union by convention: the status_error field, specifically the Descriptor Done (DD) bit, tells you which view is current. If status_error & RXD_STAT_DD, the NIC has written this descriptor and the write-back layout is valid.

The NIC’s descriptor-eating loop is, in pseudocode:

ring head H, ring tail T (in NIC's internal state)
loop:
    if H == T: drop or buffer the frame (no descriptors available)
    desc = RAM[ring_base + H * desc_size]
    write packet bytes to RAM[desc.buffer_addr]
    fill in length, status, checksum, RSS hash at desc
    set desc.status_error |= DD
    H = (H + 1) mod N
    maybe raise interrupt

The driver’s loop is the mirror image:

loop:
    desc = RAM[ring_base + driver_next * desc_size]
    if not (desc.status_error & DD): nothing new yet, return
    consume packet at desc.buffer_addr (length = desc.length)
    allocate a fresh buffer, write its DMA address into desc
    clear status fields (so the NIC can use it again)
    driver_next = (driver_next + 1) mod N
    tell the NIC: tail is now driver_next (via MMIO write to RDT)

Notice the symmetry. The descriptor ring is the interface. Both sides advance pointers in it. Neither side is allowed to touch a descriptor the other side currently owns.

If you’ve ever read about lock-free single-producer/single-consumer ring buffers, this should look extremely familiar. That’s exactly what it is, except the producer is silicon and the consumer is a kernel thread (or vice versa for TX). The ring is the contract that allows two asynchronous parties to share work without locks.

Let’s now actually trace, in source, how a freshly-loaded driver sets all this up. We’ll use ixgbe because it’s representative and the code reads cleanly.

When you ip link set eth0 up, the kernel calls the driver’s ndo_open function, which for ixgbe is ixgbe_open() in drivers/net/ethernet/intel/ixgbe/ixgbe_main.c. Inside that:

ixgbe_setup_all_rx_resources(adapter);   /* allocate descriptor rings + buffers */
ixgbe_configure(adapter);                /* program the hardware */
ixgbe_request_irq(adapter);              /* hook up interrupt handlers */
ixgbe_up_complete(adapter);              /* enable RX/TX */

Two function calls are doing the interesting work.

/* simplified */
ring->size = ring->count * sizeof(union ixgbe_adv_rx_desc);
ring->desc = dma_alloc_coherent(dev, ring->size, &ring->dma, GFP_KERNEL);

/* drivers/net/ethernet/intel/ixgbe/ixgbe_main.c, condensed */
rdba = ring->dma;                               /* DMA base of descriptor ring */
IXGBE_WRITE_REG(hw, IXGBE_RDBAL(reg_idx),       /* base low  */
                rdba & 0xFFFFFFFF);
IXGBE_WRITE_REG(hw, IXGBE_RDBAH(reg_idx),       /* base high */
                rdba >> 32);
IXGBE_WRITE_REG(hw, IXGBE_RDLEN(reg_idx),       /* length in bytes */
                ring->count * sizeof(union ixgbe_adv_rx_desc));
IXGBE_WRITE_REG(hw, IXGBE_RDH(reg_idx), 0);     /* head = 0 */
IXGBE_WRITE_REG(hw, IXGBE_RDT(reg_idx), 0);     /* tail = 0 */

ixgbe_alloc_rx_buffers(ring, ring->count - 1);
/* inside that: */
writel(tail, ring->tail);    /* doorbell — the only sync handshake */

Now we have buffers in the ring, the NIC is armed, and frames are arriving from the wire. The NIC DMAs each into the next descriptor’s buffer, marks the descriptor with DD, and … then what? How does the driver actually learn that a packet arrived?

There are two options. Both are used.

Interrupts. The NIC raises an interrupt — historically via INTx pin, modernly via MSI-X — which the CPU dispatches to a registered ISR. The ISR runs in the driver, sees a bit set in the NIC’s interrupt cause register, and goes off to process the ring.

Polling. Software periodically reads the ring’s descriptors and looks at their DD bits. No interrupts.

A pure interrupt model is what early Ethernet drivers did, and it works fine at 10 Mb/s. It catches fire at 10 Gb/s. At 14 Mpps you would take 14 million interrupts per second, each of which costs maybe 1–5 µs of CPU time for the context switch, the interrupt handler, the EOI, and the cache pollution from jumping into the kernel. That alone would consume more CPU than you have. Worse, under sustained load the CPU spends so much time servicing interrupts that user processes never run — a phenomenon called livelock, famously described in Mogul & Ramakrishnan’s 1996 paper “Eliminating Receive Livelock in an Interrupt-Driven Kernel”. The system is busy, but doing no work.

Pure polling is the opposite. You burn a CPU on a tight loop reading descriptors. Cheap at high load, ridiculous at low load. A mostly-idle server should not pin a core.

Linux’s solution, in place since 2003 or so, is NAPI (“New API”). The idea is hybrid: interrupt-driven on entry, polling-driven on the body, with adaptive switching.

The mechanism, codified in include/linux/netdevice.h and net/core/dev.c:

struct napi_struct {
    struct list_head poll_list;
    unsigned long    state;
    int              weight;
    int              (*poll)(struct napi_struct *, int budget);
    ...
};

Every driver registers one napi_struct per RX queue. Its poll callback knows how to drain that queue.

Here is the actual flow:

1. A frame arrives. The NIC raises an MSI-X interrupt for that queue.
2. The driver’s hardirq handler runs. It does almost nothing: it masks further interrupts for this queue (so it doesn’t keep getting re-interrupted while it works), and calls napi_schedule().
3. napi_schedule() adds this queue’s napi_struct to the per-CPU softnet_data.poll_list and raises the NET_RX_SOFTIRQ.
4. The softirq runs (either tail-of-interrupt or in ksoftirqd if backed up). It walks poll_list and calls each napi_struct’s poll(napi, budget).
5. The driver’s poll drains up to budget packets from the ring (typically 64) and returns how many it processed.
6. If poll processed < budget, the queue is drained; the driver re-enables interrupts and removes the napi_struct from the list with napi_complete_done().
7. If poll processed == budget, the queue is still full; the softirq does not re-enable interrupts. It loops back and polls again next time around, or schedules ksoftirqd to keep going.

You can see this exact shape in any driver. ixgbe_clean_rx_irq() is the meat of ixgbe’s poll routine. The skeleton:

/* drivers/net/ethernet/intel/ixgbe/ixgbe_main.c, condensed */
static int ixgbe_clean_rx_irq(struct ixgbe_q_vector *q_vector,
                              struct ixgbe_ring *rx_ring,
                              const int budget)
{
    unsigned int total_rx_packets = 0;

    while (likely(total_rx_packets < budget)) {
        union ixgbe_adv_rx_desc *rx_desc;
        struct sk_buff *skb;

        rx_desc = IXGBE_RX_DESC(rx_ring, rx_ring->next_to_clean);
        if (!ixgbe_test_staterr(rx_desc, IXGBE_RXD_STAT_DD))
            break;

        dma_rmb();  /* ensure we read DD before the rest of the descriptor */

        skb = ixgbe_run_xdp(adapter, rx_ring, rx_desc);
        if (!skb) { /* XDP consumed it */ ... continue; }

        ixgbe_process_skb_fields(rx_ring, rx_desc, skb);
        ixgbe_rx_skb(q_vector, skb);

        total_rx_packets++;
    }

    ixgbe_alloc_rx_buffers(rx_ring, cleaned_count); /* replenish */
    return total_rx_packets;
}

Three things to note:

• The dma_rmb() is a real-deal memory barrier. The NIC writes the body of the descriptor before setting DD. The driver must read DD first; if it sees DD, the dma_rmb() ensures subsequent reads of the descriptor body see the NIC’s writes and not stale values. Drop this barrier and you will, on some platforms, occasionally process garbage.
• The poll replenishes buffers. Every consumed descriptor must be given a fresh buffer and its tail updated, or the NIC will run out and drop frames. The doorbell write at the end of ixgbe_alloc_rx_buffers tells the NIC the tail has advanced.
• The poll budget is a fairness mechanism. Without it, a single queue could starve all others. With a budget of 64, one poll call processes at most 64 frames before yielding.

That’s it. That’s the whole interrupt-vs-polling story. NAPI is interrupt-driven at low rates (rare interrupts, no wasted polling cycles) and polling-driven at high rates (interrupts off, the softirq just keeps draining). Real systems run somewhere in between, and modern hardware adds interrupt moderation — registers like EITR on ixgbe that let the NIC delay interrupts by a configurable number of microseconds — to further coalesce work.

Before the freshly-built sk_buff even reaches netif_receive_skb, the driver typically hands it to napi_gro_receive(). GRO — Generic Receive Offload — is a software trick that merges adjacent TCP segments into one large sk_buff before the rest of the stack sees them.

Here is why it matters. The kernel’s per-packet overhead is non-trivial: traversing the protocol layers, looking up the socket, accounting, locking. If you’re receiving a TCP bulk transfer at 10 Gb/s, each MTU-sized segment going through the full stack individually is expensive. GRO notices that many consecutive segments belong to the same TCP flow with consecutive sequence numbers, identical options, and identical flags, and coalesces them into a single 64 KiB super-segment with skb_shinfo->frags[] holding pointers to the underlying pages.

The implementation, dev_gro_receive() in net/core/gro.c, tries each registered protocol-specific GRO callback (tcp4_gro_receive, udp_gro_receive, etc.). A match adds the new segment to an existing in-progress super-segment; a mismatch flushes the current super-segment up the stack.

Net effect: at 10 Gb/s, GRO routinely cuts the kernel’s per-packet cost by an order of magnitude because the stack sees one logical packet per ~40 wire packets.

There is a TX-side mirror called TSO/GSO that does the opposite for outbound traffic. They are the two halves of the same trick: keep the kernel doing the math, but let it do the math once per segment train instead of once per segment.

It helps to attach order-of-magnitude latency numbers to each step, on a modern server with a 25 GbE NIC. These are rough but useful:

So a single packet, kernel path, blocking recv(), on a healthy system, lands in the application about 5–15 microseconds after it arrived at the NIC. NAPI batching hides this by amortizing the interrupt and softirq cost over many packets — at line rate, those one-time costs disappear into per-packet costs of a few hundred nanoseconds.

The biggest variable is the wakeup. If the process is already polling (epoll, busy poll, io_uring), the wakeup cost is essentially zero. If it has to be woken from sleep, possibly migrated to a different core, the cost can balloon. This is why latency-sensitive applications (HFT, RPC infrastructure) often pin processes to cores and busy-poll instead of sleeping.

For throughput-oriented workloads, none of this matters: GRO + NAPI + page_pool can saturate 100 Gb/s on a handful of cores. For latency, every microsecond matters, and that’s where XDP and DPDK earn their keep.

If XDP can redirect a packet to “an AF_XDP socket”, what is that?

AF_XDP is a socket family that connects a userspace process to an XDP program via four shared-memory rings:

UMEM (shared between kernel and user, a huge contiguous region):
    a pool of equally-sized frames

FILL ring:  user -> kernel  ("here are free frames you can fill")
RX ring:    kernel -> user  ("here are descriptors of received frames")
TX ring:    user -> kernel  ("here are descriptors of frames to send")
COMP ring:  kernel -> user  ("here are frames I'm done transmitting")

This is exactly the descriptor-ring pattern from earlier, but between the kernel driver and a userspace process instead of between the driver and the NIC. The kernel acts as the intermediary: XDP redirects into the UMEM, the driver writes a descriptor into the RX ring, userspace reads from the RX ring with recvmsg or busy-polling, processes the packet, and either returns the frame to the FILL ring or queues it on the TX ring.

In zero-copy mode (XDP_FLAGS_SKB_MODE off, with driver support), the NIC DMAs directly into UMEM frames. The userspace process sees the packet bytes in shared memory with no copy. The cost compared to a normal socket: no sk_buff, no protocol stack, no recvmsg copy. The cost compared to DPDK: you still pay the driver’s NAPI poll and the XDP execution, you still go through the kernel for interrupts and configuration.

The kernel docs in Documentation/networking/af_xdp.rst are the canonical reference. The implementation is in net/xdp/.

The mental shift: AF_XDP is a socket flavor that takes the descriptor ring all the way to userspace. Once you see that, the API stops feeling exotic. It is just rings, all the way down.

Put it all together and you get something like:

                                  +-------------------------------+
                                  |   Application                 |
                                  |   recv() / epoll              |
                                  +---------------^---------------+
                                                  |
                                       wake + copy to user
                                                  |
+----------------------+         +----------------+----------------+
| Kernel stack         |         |     Socket receive queue        |
| ip_rcv, tcp_v4_rcv   |-------->|     (per-socket)                |
| (per-CPU softirq)    |         +----------------+----------------+
+----------+-----------+                          ^
           ^                                      |
           |                                      |
+----------+-----------+    XDP_PASS              |
|  napi_gro_receive    |--------------+           |
|  (GRO coalescing)    |              |           |
+----------+-----------+              |           |
           ^                          v           |
           |                +---------+--------+  |
           |                |  XDP program     |  |
           |                |  DROP/PASS/TX/   |  |
           |                |  REDIRECT        |  |
           |                +---------+--------+  |
           |                          |           |
           |                          v           |
+----------+-----------+        +-----+------+    |
| Driver NAPI poll     |        |  AF_XDP    |    |
| reads RX descriptors |        |  UMEM/rings|----+--> userspace fastpath
| build_skb / page_pool|        +------------+
+----------+-----------+
           ^
           | DMA write of packet + status
           |
+----------+-----------+
|        NIC           |
|   RX ring (RDBAL/RDH/|        +---- doorbell (MMIO write) <---- driver
|    RDT/RDLEN)        |
|   RSS hashing -> Q   |
|   DMA engine         |
+----------+-----------+
           ^
        wire bytes
           |
       [Ethernet]

Every arrow is either a memory transaction or a control-plane handshake. Every box is either a producer or a consumer in some ring. The kernel path is the long left road; XDP is a fork off it; AF_XDP and DPDK are progressively more aggressive shortcuts into userspace.

A NIC is a PCIe peripheral with a register file and a DMA engine. The driver hands it descriptor rings — circular arrays of structs containing DMA addresses — through MMIO writes. The NIC eats descriptors from one end (writing packets into the buffers they point to and marking them done), the driver eats them from the other end (reading completed descriptors and replenishing them with fresh buffers). Doorbell registers are the only synchronous handshake; everything else is RAM-mediated ownership transfer. The kernel path turns those buffers into sk_buffs, runs them through GRO and the protocol stack, and finally copies them into a userspace buffer when an application asks. XDP runs an eBPF program at the earliest moment in the driver so you can make decisions without paying skb overhead. AF_XDP extends the descriptor-ring pattern all the way into userspace. DPDK skips the kernel entirely by running the driver in userspace. Performance is about removing unnecessary work, batching necessary work, keeping memory hot in cache, avoiding unnecessary wakeups, and remembering that bytes you don’t have to touch are the fastest bytes of all.

The packet doesn’t “arrive in the application.” It is handed up through a sequence of producers and consumers, each of which exists to spare the next one from work it can’t afford to do. Now when you read a driver, an XDP program, or a DPDK loop, you’ll see the same shapes everywhere: rings, descriptors, doorbells, polls. That’s the language. Everything else is dialect.