11. Input/Output Systems
A computer that cannot talk to the outside world computes nothing anyone can use, and the devices it must talk to differ from one another more than any other component of a system — a keyboard delivers bytes at human speed, an NVMe drive at gigabytes per second, a network card at unpredictable intervals decided by someone else's machine. This chapter covers how the hardware exposes those devices, how the kernel layers driver code beneath a uniform interface so applications never see the differences, and what the I/O subsystem does with the requests once it has them. It matters because I/O is where almost every real system spends its waiting time, and because the techniques here — interrupts, DMA, buffering, asynchrony — are the ones that decide whether that waiting overlaps with useful work or simply stops it.
<a id="111-io-hardware"></a>
11.1 I/O Hardware
This section covers what the machine physically offers: how a device is addressed and controlled, the buses that connect it, and the three escalating strategies — polling, interrupts, and DMA — for getting data across without wasting the processor.
Device Controllers and Ports
Theory
A CPU cannot drive a disk motor or modulate a radio. What it can do is read and write locations, so every device is presented to it as a small set of locations that happen to have side effects. The hardware that provides those locations is the device controller — a small special-purpose computer sitting between the bus and the physical device, translating "write 0x20 to this register" into whatever the device actually requires.
The analogy worth holding is a hotel's front desk. Guests do not walk into the kitchen; they leave a request at the desk, and the desk staff know how to turn "breakfast at seven" into actions in a part of the building the guest never sees. The controller is that desk, and its registers are the forms you fill in.
Nearly every controller exposes the same four kinds of register, whatever the device:
| Register | Direction | Purpose |
|---|---|---|
| Data-in | Host reads | Bytes arriving from the device |
| Data-out | Host writes | Bytes going to the device |
| Status | Host reads | Busy, ready, error, data-available flags |
| Control | Host writes | Start a command, set a mode, enable interrupts |
Two mechanisms exist for reaching those registers, and the distinction still appears in driver code:
Port-mapped I/O (PMIO). The registers live in a separate address space reached only by special instructions — IN and OUT on x86. A device occupies a range of port numbers (the legacy serial port at 0x3F8, the PS/2 keyboard at 0x60). The address space is small (65,536 ports on x86) and the instructions are privileged, which gives protection for free: a user program simply cannot execute OUT.
Memory-mapped I/O (MMIO). The registers appear at ordinary physical memory addresses. The driver reads and writes them with ordinary load and store instructions, and protection comes from page tables — the kernel maps the region and user space does not. This is how essentially every modern device works, because it needs no special instructions, supports 64-bit address space, and allows large regions (a GPU exposes hundreds of megabytes of framebuffer this way).
MMIO carries one hazard that catches people: these are not memory. Reading a status register twice may give different answers, and writing a control register does something. So the CPU's normal freedoms — caching values, reordering accesses, merging writes — are all wrong here. Device regions must be mapped uncacheable, and drivers must use the platform's ordering primitives (readl/writel on Linux, which embed the necessary barriers) rather than raw pointer dereferences. A driver that uses a plain volatile pointer and no barriers works on one machine and fails on another.
Finding the registers is the remaining problem, and it is solved by enumeration. PCI Express devices expose a configuration space describing their vendor, device class, and the Base Address Registers (BARs) that declare how much space they need; firmware or the kernel assigns real addresses and writes them back. This is why modern systems have no fixed port numbers to memorise — the bus tells the kernel what is present and where.
Example
WHERE THE CPU MEETS THE DEVICE
CPU ----- system bus -----+------------------+------------------+
| | |
+-------v-------+ +-------v-------+ +-------v-------+
| disk | | network | | USB |
| CONTROLLER | | CONTROLLER | | CONTROLLER |
+---------------+ +---------------+ +---------------+
| data-in reg | | |
| data-out reg | | |
| status reg | [ ethernet ] [ keyboard ]
| control reg |
+-------+-------+
|
[ actual disk ]
TWO WAYS TO ADDRESS THOSE REGISTERS
PORT-MAPPED MEMORY-MAPPED
+------------------+ physical address space
| memory 0..N | +-------------------------+
+------------------+ | RAM 0x0000_0000 |
| ports 0x0..0xFFFF| <- separate | ... |
+------------------+ space | RAM 0x7FFF_FFFF |
+-------------------------+
outb(0x20, 0x3F8); | NIC regs 0xF000_0000 | <- device
val = inb(0x3FD); | GPU BAR 0xE000_0000 |
+-------------------------+
special instructions, writel(0x20, nic->ctrl);
privileged by the ISA val = readl(nic->status);
protected by PAGE TABLES
WHY MMIO REGIONS MUST NOT BE CACHED
status = *reg; // cached: CPU returns the OLD value forever
while (status & BUSY) // loops until the heat death of the universe
status = *reg; // because nothing invalidates the cache line
*ctrl = START; // write-combined: may be merged, reordered,
*ctrl = ABORT; // or dropped. The device sees the wrong sequence.
// A minimal PCIe driver's register access, the way Linux actually does it.
#include <linux/pci.h>
#include <linux/io.h>
#define REG_STATUS 0x08 // offsets into the device's BAR
#define REG_CONTROL 0x0C
#define REG_DATA 0x10
#define STATUS_READY (1u << 0)
#define CTRL_START (1u << 1)
struct mydev {
void __iomem *regs; // __iomem: NOT an ordinary pointer
};
static int mydev_probe(struct pci_dev *pdev, const struct pci_device_id *id)
{
struct mydev *dev = kzalloc(sizeof(*dev), GFP_KERNEL);
pci_enable_device(pdev); // power it up, enable decoding
pci_request_regions(pdev, "mydev"); // claim the BARs exclusively
// Map BAR 0 uncached. The address came from the BUS, not from a constant.
dev->regs = pci_iomap(pdev, 0, 0);
if (!dev->regs)
return -ENOMEM;
pci_set_master(pdev); // allow the device to do DMA
pci_set_drvdata(pdev, dev);
return 0;
}
static int mydev_send(struct mydev *dev, u32 value)
{
// readl/writel embed the compiler and CPU barriers this needs.
// A plain *(volatile u32 *) would compile, and would be wrong.
unsigned long deadline = jiffies + msecs_to_jiffies(100);
while (!(readl(dev->regs + REG_STATUS) & STATUS_READY)) {
if (time_after(jiffies, deadline))
return -ETIMEDOUT; // ALWAYS bound a hardware wait
cpu_relax(); // hint to the CPU: spin loop
}
writel(value, dev->regs + REG_DATA); // load the data register first
wmb(); // ordering: data BEFORE command
writel(CTRL_START, dev->regs + REG_CONTROL); // now tell it to go
return 0;
}
# What the bus says is present, and where its registers landed.
lspci -v | head -20
# 00:1f.2 SATA controller: Intel Corporation ... (rev 05)
# Flags: bus master, 66MHz, medium devsel, latency 0, IRQ 27
# Memory at f7f00000 (32-bit, non-prefetchable) [size=8K] <- BAR: MMIO
# I/O ports at f070 [size=8] <- BAR: PMIO
# Capabilities: [80] MSI: Enable+ Count=1/1
lspci -vvv -s 00:1f.2 | grep -A3 'Region' # every BAR of one device
sudo lspci -xxx -s 00:1f.2 | head -4 # raw PCI configuration space
# The two address spaces, as the kernel sees them.
sudo cat /proc/iomem | head -20 # memory-mapped regions
# f7f00000-f7f01fff : 0000:00:1f.2
# ^ the SATA controller's registers
sudo cat /proc/ioports | head -20 # port-mapped ranges
# 0060-0060 : keyboard
# 03f8-03ff : serial
# Which driver claimed which device.
lspci -k | head -12
ls -l /sys/bus/pci/devices/0000:00:1f.2/driver
Key takeaways
- A device controller is the hardware intermediary that turns register accesses into physical device actions; the CPU never touches the device itself.
- Almost every controller exposes the same four register types: data-in, data-out, status, and control.
- Port-mapped I/O uses a separate address space and privileged instructions; memory-mapped I/O uses ordinary addresses protected by page tables and is what modern devices use.
- MMIO regions are not memory: they must be mapped uncacheable, and drivers must use ordering-aware accessors (
readl/writel) rather than raw pointers. - PCIe devices declare their register requirements through Base Address Registers, and the kernel or firmware assigns the actual addresses at enumeration.
- Every wait on a hardware status bit must be bounded by a timeout, because a wedged device otherwise hangs the driver forever.
🧪 Practice
- List the four standard controller registers and, for a "read one sector" operation, state the order in which a driver would touch each.
- Run
lspci -vandcat /proc/iomemon your machine, and match one device's BAR to its entry in the memory map.- Interview-style: "A driver polls a status register in a loop and the loop never exits, though the device is working. What is the most likely cause?" Hint: ask what the CPU is allowed to assume about a value it has already loaded, and what property the mapping needs to forbid that assumption.
Buses and Interfaces
Theory
Controllers must be connected to the CPU and to memory, and the wiring that does it is the bus. The bus is not a neutral pipe: its topology and protocol decide how many devices can be attached, how much bandwidth each gets, and whether one slow device can hold up everybody else.
The historical arc is from shared to switched, and it mirrors what happened to Ethernet. Early buses (ISA, PCI, the original SCSI) were genuinely shared wires: every device saw every transaction, only one could transmit at a time, and arbitration decided who. That is simple and cheap, but total bandwidth is fixed no matter how many devices you attach, and electrical limits cap both the clock rate and the number of connections.
PCI Express replaced this with point-to-point serial lanes joined by switches. Each device gets its own link to a switch, so transfers to different devices proceed simultaneously; a link is negotiated as x1, x4, x8, or x16 lanes, so bandwidth is allocated per device rather than shared. Because each lane is a differential serial pair rather than a wide parallel bus, it clocks far higher — parallel buses died precisely because keeping 64 wires in skew at gigahertz rates is harder than running one pair very fast.
Bandwidth roughly doubles per PCIe generation:
| Generation | Per lane (each direction) | x4 link | x16 link |
|---|---|---|---|
| PCIe 3.0 | ~1 GB/s | ~4 GB/s | ~16 GB/s |
| PCIe 4.0 | ~2 GB/s | ~8 GB/s | ~32 GB/s |
| PCIe 5.0 | ~4 GB/s | ~16 GB/s | ~64 GB/s |
Above the bus sit interfaces — the command protocols devices speak. Here the important lesson is that a protocol designed for one device's physics can strangle another's. AHCI/SATA was designed for rotating disks: one command queue, 32 outstanding commands, an interrupt per completion. That is ample for a device doing 200 operations per second and absurd for flash capable of a million. NVMe was designed for the latter: up to 65,535 queues of 65,536 commands each, one queue per CPU core so there is no shared lock, doorbell registers instead of legacy handshakes, and a command set with a fraction of the overhead.
| Interface | Typical use | Queues | Queue depth | Latency overhead |
|---|---|---|---|---|
| SATA/AHCI | HDDs, budget SSDs | 1 | 32 | ~6 µs |
| SAS | Enterprise disks, expanders | 1 | 256 | ~5 µs |
| NVMe | Modern SSDs | up to 65,535 | up to 65,536 | ~2 µs |
| USB | Peripherals, external storage | polled schedule | varies | ~100 µs+ |
| Thunderbolt | External PCIe, displays | PCIe tunnelled | — | ~2 µs |
USB deserves separate mention because it is the opposite design point. It is a polled, host-scheduled, tiered-star bus: the host controller asks each device in turn whether it has data, on a fixed frame schedule. This makes devices cheap and hot-pluggable and gives predictable bandwidth allocation for isochronous transfers (audio, video), at the cost of latency that no amount of bandwidth fixes. It is also why a USB device cannot initiate a transfer — and why USB storage feels slower than its throughput number suggests.
Two practical consequences follow. First, lane count matters as much as generation: an NVMe drive in an x1 slot is capped regardless of its rating. Second, some slots are wired to the chipset rather than directly to the CPU, and everything behind the chipset shares one uplink — which is how two fast drives can each benchmark well alone and disappoint together.
Example
SHARED BUS vs SWITCHED FABRIC
SHARED (PCI, ISA) SWITCHED (PCI Express)
CPU --+---+---+---+ CPU
| | | | | x16
[A] [B] [C] [D] +---+----+
| ROOT |
one transaction at a time; | COMPLEX|
total bandwidth FIXED and +-+--+--++
divided among all devices | | | x4 each
[A][B][ SWITCH ]
| |
[C] [D]
A->CPU and B->CPU proceed SIMULTANEOUSLY.
Adding devices adds bandwidth (until an uplink saturates).
THE UPLINK TRAP
CPU ==x16== [GPU] direct: full bandwidth
CPU ==x4=== [CHIPSET] --x4-- [NVMe 1] NVMe 1 + NVMe 2 + USB + NIC
--x4-- [NVMe 2] ALL SHARE the single x4 uplink.
--x1-- [NIC] Each benchmarks fine alone;
------ [USB] together they contend.
WHY NVMe BEAT AHCI (the same SSD behind each)
AHCI NVMe
+-------------------+ core0 -> [SQ0][CQ0] -+
| ONE queue, 32 deep| core1 -> [SQ1][CQ1] -+--> device
| shared lock | core2 -> [SQ2][CQ2] -+
| all cores contend | core3 -> [SQ3][CQ3] -+
+-------------------+ no shared lock, no cross-core traffic
~100k IOPS ceiling millions of IOPS
uncached register handshake doorbell write per batch
DEVICE SPEED SPANS SEVEN ORDERS OF MAGNITUDE
keyboard 10 bytes/s <- human-limited
USB 2.0 mouse ~1 KB/s
gigabit NIC 125 MB/s
SATA SSD 550 MB/s
PCIe 4 NVMe 7 GB/s
DDR5 memory 50 GB/s
One interface cannot serve all of these well. Hence many buses.
# The PCIe topology of this machine, as a tree.
lspci -tv
# -[0000:00]-+-00.0 Host bridge
# +-01.0-[01]----00.0 NVIDIA GPU
# +-1c.0-[02]----00.0 NVMe controller
# \-1f.2 SATA controller
# Negotiated link speed and width -- the numbers that actually apply.
sudo lspci -vv -s 02:00.0 | grep -E 'LnkCap|LnkSta'
# LnkCap: Speed 16GT/s, Width x4 <- what the device can do
# LnkSta: Speed 8GT/s, Width x4 <- what it NEGOTIATED
# A mismatch here is the single most common "why is my drive slow" cause.
# Per-device theoretical bandwidth.
for d in $(lspci | awk '{print $1}'); do
cap=$(sudo lspci -vv -s $d 2>/dev/null | grep -m1 LnkSta)
[ -n "$cap" ] && echo "$d $cap"
done
# Storage interfaces present.
lsblk -o NAME,TRAN,ROTA,SIZE,MODEL
# NAME TRAN ROTA SIZE MODEL
# sda sata 1 1.8T ST2000DM008
# nvme0n1 nvme 0 931G Samsung SSD 980 PRO
# NVMe queue structure: the reason it scales.
sudo nvme id-ctrl /dev/nvme0 | grep -iE 'sqes|cqes|mdts'
ls /sys/block/nvme0n1/mq/ # one directory per hardware queue
# 0 1 2 3 4 5 6 7 <- matches the core count
cat /sys/block/sda/mq/0/nr_tags # AHCI: a single queue, 32 tags
# USB's tiered-star topology and its speed tiers.
lsusb -t
# /: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 5000M
# |__ Port 2: Dev 3, If 0, Class=Mass Storage, Driver=usb-storage, 5000M
# Is the link or the device the bottleneck? Compute before benchmarking.
PCIE_PER_LANE_GBPS = {3: 0.985, 4: 1.969, 5: 3.938} # GB/s per lane per direction
def link_bandwidth(gen, lanes):
return PCIE_PER_LANE_GBPS[gen] * lanes
devices = [
# name, gen, lanes, device's own rated GB/s
("NVMe Gen4 x4", 4, 4, 7.0),
("NVMe Gen4 drive in x1", 4, 1, 7.0), # the classic misconfiguration
("NVMe Gen3 x4", 3, 4, 3.5),
("100 GbE NIC on x8 Gen3", 3, 8, 12.5),
("100 GbE NIC on x4 Gen3", 3, 4, 12.5),
]
print(f"{'device':<26}{'link GB/s':>11}{'device GB/s':>13}{'limited by':>13}")
for name, gen, lanes, rated in devices:
link = link_bandwidth(gen, lanes)
limiter = "LINK" if link < rated else "device"
print(f"{name:<26}{link:>11.1f}{rated:>13.1f}{limiter:>13}")
# Shared uplink contention: what N devices behind one x4 link actually get.
UPLINK = link_bandwidth(3, 4) # chipset uplink, ~3.9 GB/s
for n in (1, 2, 4):
print(f"\n{n} device(s) behind a Gen3 x4 chipset uplink:")
print(f" each gets at most {UPLINK/n:.2f} GB/s "
f"({'no contention' if n == 1 else 'contended'})")
Key takeaways
- Buses evolved from shared parallel wires to switched point-to-point serial links, so bandwidth now scales with device count instead of being divided among devices.
- PCIe bandwidth is per-lane and roughly doubles each generation; both generation and negotiated lane width must be checked, since either can throttle a device.
- Devices behind a chipset share one uplink, which is why two fast drives can each benchmark well alone and contend when used together.
- Interface protocols encode assumptions about device physics: AHCI's single 32-deep queue suited disks and strangled flash.
- NVMe gives each core its own submission and completion queue, removing shared-lock contention and enabling millions of IOPS.
- USB is a host-polled tiered star: cheap, hot-pluggable, and predictable for isochronous traffic, but with latency that bandwidth cannot fix.
🧪 Practice
- Compute the theoretical bandwidth of PCIe 4.0 x4 and PCIe 3.0 x16, and say which is faster and by how much.
- Run
lspci -vvon a storage or network device and compareLnkCapwithLnkSta. Explain any difference you find.- Interview-style: "An NVMe drive rated for 7 GB/s benchmarks at 1.6 GB/s in one slot and 6.8 GB/s in another, on the same machine. Explain." Hint: two independent properties of a PCIe link are negotiated at boot, and a motherboard's slots are not all wired to the same place.
Polling
Theory
Once a driver has issued a command, it must find out when the device is done. The simplest possible answer is to keep asking. Polling — also called busy-waiting or programmed I/O — means the CPU reads the status register in a loop until the ready bit appears.
The handshake is a four-step protocol, and it is worth memorising because every device interaction is a variation of it:
- The host reads the status register repeatedly until busy is clear.
- The host sets the command-ready bit and writes the command and data.
- The device notices the command, sets busy, and performs the operation.
- The device clears busy and sets the ready or error bit; the host, still polling, observes this.
Polling's virtues are real. It is trivially simple, it has the lowest possible latency — the CPU learns of completion in the time it takes to execute one load instruction, with no interrupt delivery, no context switch, and no cache pollution — and it is completely deterministic, which matters in real-time systems where an interrupt's variable delivery time is itself the problem.
Its vice is equally clear: while polling, the CPU does nothing else. If a device takes 8 milliseconds, a 3 GHz processor burns roughly 24 million cycles' worth of potential work discovering that the device is not finished yet. For slow devices this is indefensible, and it is why interrupts were invented.
But the arithmetic reverses when devices get fast. An interrupt costs something — very roughly 1 to 5 microseconds once you count delivery, the handler, the context switch, and the cache and TLB damage. If the device completes in 100 microseconds, that overhead is 1-5% and interrupts are obviously right. If the device completes in 2 microseconds, the interrupt costs more than the operation. Polling wins.
This is why polling has returned in modern high-performance systems, not as a legacy technique but as the preferred one:
- NAPI in the Linux network stack: an interrupt signals the first packet, then interrupts are disabled and the driver polls until the queue drains. Under load this amortises one interrupt over hundreds of packets; when idle, it falls back to interrupts so nothing spins.
- DPDK and SPDK take it to the limit: user-space drivers that dedicate entire cores to polling, bypassing the kernel entirely. A core is consumed permanently in exchange for single-digit-microsecond latency.
- io_uring with
IORING_SETUP_IOPOLL, and the block layer's hybrid polling, which sleeps for an estimated fraction of the expected completion time and then polls — capturing most of polling's latency benefit while returning most of the CPU.
The general rule: poll when the wait is shorter than the cost of being interrupted, interrupt otherwise — and hybrid approaches exist precisely because real workloads straddle the line.
Example
THE POLLING HANDSHAKE
HOST DEVICE
read status ---------------------> busy?
| busy=1, loop |
| busy=1, loop |
| busy=0 |
write data to data-out reg --------> |
set command-ready in control ------> | sees command
| sets busy=1
read status (busy=1) <-------------+ ... performing the operation ...
read status (busy=1) <-------------+
read status (busy=0, ready=1) <----+ done
read data-in register -------------->
WHERE THE CPU TIME GOES
POLLING an 8 ms disk on a 3 GHz CPU
|<--------------------- 8 ms = ~24,000,000 cycles ------------------->|
[read status][read status][read status] ... 100% CPU, 0 useful work
POLLING a 2 us NVMe completion
|<-- 2 us = ~6,000 cycles -->|
[poll][poll][poll] vs INTERRUPT: ~1-5 us of OVERHEAD ALONE
-> polling is FASTER *and* cheaper here
THE CROSSOVER
device latency | interrupt overhead as % | better choice
----------------+-------------------------+---------------
8 ms (HDD) | 0.03% | interrupt
100 us (SATA SSD)| 2% | interrupt
10 us (NVMe) | 20% | hybrid
2 us (Optane) | 150% | POLL
NAPI: THE HYBRID THAT RUNS EVERY LINUX MACHINE
idle -> interrupts ENABLED (no CPU burned while nothing arrives)
1st packet-> interrupt fires, driver DISABLES further interrupts
busy -> poll the ring until empty or the budget is spent
drained -> re-enable interrupts, go back to idle
One interrupt can cover hundreds of packets under load.
// Polling in a driver: correct, and with the two details people forget.
static int poll_wait_ready(struct mydev *dev, unsigned int timeout_us)
{
ktime_t deadline = ktime_add_us(ktime_get(), timeout_us);
u32 status;
for (;;) {
status = readl(dev->regs + REG_STATUS); // uncached read: hits the device
if (status & STATUS_ERROR)
return -EIO;
if (status & STATUS_READY)
return 0;
// (1) ALWAYS bound the wait. A wedged device must not hang the kernel.
if (ktime_after(ktime_get(), deadline))
return -ETIMEDOUT;
// (2) Tell the CPU this is a spin loop: on x86 this is PAUSE, which
// saves power and avoids a memory-order violation penalty on exit.
cpu_relax();
}
}
// Hybrid: sleep for most of the expected wait, then poll for the tail.
static int hybrid_wait(struct mydev *dev, unsigned int expected_us)
{
// Sleeping gives the CPU back; the final poll avoids the wake-up latency.
usleep_range(expected_us / 2, expected_us * 3 / 4);
return poll_wait_ready(dev, expected_us * 4);
}
# When does polling beat interrupts? The arithmetic, and the CPU cost of each.
INTERRUPT_OVERHEAD_US = 3.0 # delivery + handler + context switch + cache damage
POLL_COST_PER_US = 1.0 # polling burns one CPU-microsecond per microsecond
def compare(device_latency_us, ops_per_sec):
# Interrupt-driven: CPU is free during the wait, but pays per completion.
interrupt_cpu = ops_per_sec * INTERRUPT_OVERHEAD_US / 1e6 # CPU fraction
interrupt_latency = device_latency_us + INTERRUPT_OVERHEAD_US
# Polling: CPU is busy for the whole wait, but completion is seen instantly.
poll_cpu = ops_per_sec * device_latency_us * POLL_COST_PER_US / 1e6
poll_latency = device_latency_us
return interrupt_cpu, interrupt_latency, poll_cpu, poll_latency
devices = [("HDD", 8000.0, 150),
("SATA SSD", 100.0, 80_000),
("NVMe", 10.0, 500_000),
("Optane/CXL", 2.0, 900_000)]
print(f"{'device':<12}{'IRQ cpu':>9}{'IRQ lat':>10}{'poll cpu':>10}{'poll lat':>10} winner")
for name, lat, iops in devices:
ic, il, pc, pl = compare(lat, iops)
winner = "poll" if (pl < il and pc <= 1.0) else "interrupt"
print(f"{name:<12}{ic:>8.1%}{il:>9.1f}us{min(pc,1.0):>9.1%}{pl:>9.1f}us {winner}")
# Note the poll-CPU column saturates at 100%: that is a whole core consumed.
# DPDK and SPDK accept exactly that trade deliberately.
# Polling in production, on the block layer.
cat /sys/block/nvme0n1/queue/io_poll # 1 = polling available
cat /sys/block/nvme0n1/queue/io_poll_delay
# -1 = classic (pure) polling
# 0 = hybrid polling with an adaptive sleep
# >0 = sleep this many nanoseconds, then poll
echo 0 | sudo tee /sys/block/nvme0n1/queue/io_poll_delay # enable hybrid
fio --name=poll --ioengine=io_uring --hipri=1 --rw=randread \
--bs=4k --iodepth=1 --filename=/dev/nvme0n1 --runtime=10 --time_based
# --hipri uses the polled path. Compare p50/p99 latency against the default.
# Network polling: NAPI budget and how often it is exhausted.
sysctl net.core.netdev_budget # packets per poll cycle (default 300)
sysctl net.core.netdev_budget_usecs # time bound per poll cycle
awk '{print $1, $2, $3}' /proc/net/softnet_stat | head -4
# column 3 counts "time squeeze": budget exhausted with work remaining.
# Persistently non-zero means raising the budget may help.
# Interrupt coalescing: the hardware's own version of the same trade.
sudo ethtool -c eth0
# rx-usecs: 50 <- wait up to 50 us to batch completions
# rx-frames: 64 <- or until 64 frames have arrived
sudo ethtool -C eth0 rx-usecs 0 # lowest latency, highest interrupt rate
sudo ethtool -C eth0 rx-usecs 100 # highest throughput, worst latency
Key takeaways
- Polling repeatedly reads a status register until the device signals completion, following a four-step busy/command-ready handshake.
- It has the lowest possible completion latency and is fully deterministic, which matters for real-time work.
- It consumes the CPU for the entire wait, which is indefensible for slow devices — the original motivation for interrupts.
- The trade reverses for very fast devices: when a completion takes less time than an interrupt costs to deliver, polling is both faster and cheaper.
- NAPI, DPDK, SPDK, and
io_uring's polled mode are all modern returns to polling, usually hybridised so idle systems do not spin. - Every polling loop needs a timeout and a
cpu_relax()-style pause hint; omitting either produces hangs or wasted power.
🧪 Practice
- A device completes in 40 µs and an interrupt costs 3 µs. Compute the overhead percentage, then repeat for a device completing in 3 µs, and state which method each favours.
- Write the four-step polling handshake as pseudocode with a timeout, and mark the exact point at which a missing timeout would hang the system.
- Interview-style: "Why does a high-performance packet processing framework deliberately burn a whole CPU core spinning?" Hint: compare the cost of an interrupt against the time to process one packet at 10 million packets per second, and consider what else that core would have been doing.
Interrupt-Driven I/O
Theory
Polling wastes the CPU because the CPU is asking. Interrupts invert the relationship: the device tells the CPU when it is ready, and until then the CPU runs other work. This single inversion is what makes multiprogramming possible — without it, every I/O would idle the machine.
The mechanism, in the order it happens:
- The device raises an interrupt on its assigned line or sends an interrupt message.
- The CPU finishes the current instruction, then saves enough state to resume.
- It reads the interrupt number and indexes the interrupt vector table (the IDT on x86) to find the handler address.
- It switches to kernel mode and, typically, to a dedicated interrupt stack, then jumps to the handler.
- The handler services the device, acknowledges the interrupt at the controller, and returns.
- State is restored and the interrupted work resumes — or the scheduler runs, if the handler woke something more important.
Two properties of that sequence drive everything else. First, an interrupt handler runs in interrupt context: there is no process to block, so it may not sleep, may not take a mutex that could sleep, and may not do anything slow. Second, interrupts have priority levels, so a high-priority device can preempt a lower-priority handler, and critical sections can mask lower levels while remaining responsive to higher ones. Non-maskable interrupts (NMI) sit above everything for hardware failures and watchdogs.
The "handlers must be fast" constraint creates the standard structure of every real driver: the top half / bottom half split. The top half is the actual interrupt handler — it does the minimum required to make the device stop shouting (acknowledge the interrupt, grab the status, copy the data out of a small buffer) and then schedules the rest. The bottom half — a softirq, tasklet, or workqueue on Linux; a DPC on Windows — runs later with interrupts enabled and, for workqueues, in a context that may sleep. Protocol processing, waking the waiting thread, and starting the next request all belong there.
Delivery has evolved along with buses. Legacy pin-based interrupts used physical lines (IRQ 0-15 originally), which were scarce and had to be shared — several devices on one line, so each handler had to check "was it me?" and return IRQ_NONE if not. MSI and MSI-X replaced this: the device signals an interrupt by writing to a special memory address, which needs no dedicated wire, allows thousands of distinct vectors, cannot be confused with another device's, and — crucially — eliminates a race in which a completion arrived before its interrupt did. MSI-X also lets one device have many vectors, which is what allows a NIC or NVMe drive to direct each queue's completions to a different CPU core.
Two pathologies are worth knowing by name. An interrupt storm occurs when a device raises interrupts faster than the system can retire them, so the machine spends all its time in handlers and makes no forward progress; the defences are coalescing (batch completions in hardware), NAPI-style interrupt disabling, and the kernel's own storm detection that disables a misbehaving line. And interrupt affinity matters on multi-core machines: if every completion is delivered to core 0, that core saturates while others idle, so the kernel (or irqbalance, or explicit pinning) spreads vectors across cores — ideally to the core that issued the request, so the data is already in its cache.
Example
THE INTERRUPT PATH
CPU running user code
|
| <=== device asserts interrupt / sends MSI write
v
finish current instruction
save PC, flags, minimal registers
read vector number --> IDT[vector] --> handler address
switch to kernel mode + interrupt stack
|
v
+--------------- TOP HALF (interrupt context) ---------------+
| - acknowledge the device (stop it re-raising) |
| - read status, copy data out of the hardware buffer |
| - schedule the bottom half |
| - MAY NOT SLEEP, MAY NOT TAKE A SLEEPING LOCK |
+------------------------------------------------------------+
| return from interrupt
v
... later, with interrupts enabled ...
+------------- BOTTOM HALF (softirq / workqueue) ------------+
| - protocol processing, checksum, wake the waiting thread |
| - start the next queued request |
| - workqueues MAY sleep; softirqs may not |
+------------------------------------------------------------+
SHARED PIN-BASED vs MSI-X
PIN-BASED (legacy) MSI-X
[dev A] --+ [dev A] -- writes 0xFEE00000, data=42
[dev B] --+--- IRQ 11 --> CPU [dev B] -- writes 0xFEE00000, data=57
[dev C] --+ [NIC q0] - vector 60 -> core 0
[NIC q1] - vector 61 -> core 1
every handler must ask "was it [NIC q2] - vector 62 -> core 2
me?"; one broken device can no wires, no sharing, thousands of
wedge the whole line vectors, per-queue CPU steering
INTERRUPT STORM
normal: [work][irq][work][irq][work] forward progress
storm: [irq][irq][irq][irq][irq][irq] 100% in handlers, 0 progress
defences: coalescing (rx-usecs), NAPI (disable + poll), kernel's
"nobody cared" auto-disable after 100,000 spurious interrupts
AFFINITY: WHERE COMPLETIONS LAND
all vectors -> core 0 : core 0 saturates, 15 cores idle
one vector per core : parallel, and the completion lands where the
request was issued, so its data is cache-hot
// The canonical two-half driver structure.
#include <linux/interrupt.h>
#include <linux/workqueue.h>
struct mydev {
void __iomem *regs;
struct work_struct slow_work; // bottom half that may sleep
struct tasklet_struct fast_work; // bottom half that may not
struct completion done;
u32 last_status;
};
// ---------- TOP HALF: runs in interrupt context, must be tiny ----------
static irqreturn_t mydev_isr(int irq, void *data)
{
struct mydev *dev = data;
u32 status = readl(dev->regs + REG_STATUS);
// On a SHARED line this check is mandatory: the interrupt may be
// another device's, and claiming it would break that device.
if (!(status & STATUS_INTERRUPT_PENDING))
return IRQ_NONE;
writel(status, dev->regs + REG_INT_ACK); // acknowledge FIRST, or it re-fires
dev->last_status = status;
if (status & STATUS_DATA_READY)
tasklet_schedule(&dev->fast_work); // defer: no processing here
if (status & STATUS_ERROR)
schedule_work(&dev->slow_work); // error recovery may need to sleep
return IRQ_HANDLED;
}
// ---------- BOTTOM HALF: interrupts enabled, real work happens ----------
static void mydev_tasklet(unsigned long data)
{
struct mydev *dev = (struct mydev *)data;
// Process the buffer, hand it upstream, start the next request.
complete(&dev->done); // wake whoever was waiting
}
static void mydev_error_work(struct work_struct *w)
{
struct mydev *dev = container_of(w, struct mydev, slow_work);
// A workqueue runs in process context: msleep() and mutexes are legal here.
msleep(10);
mydev_reset(dev);
}
static int mydev_setup_irq(struct mydev *dev, struct pci_dev *pdev)
{
int nvec, i;
// Prefer MSI-X: no sharing, one vector per queue, steerable per CPU.
nvec = pci_alloc_irq_vectors(pdev, 1, num_online_cpus(),
PCI_IRQ_MSIX | PCI_IRQ_MSI | PCI_IRQ_LEGACY);
if (nvec < 0)
return nvec;
for (i = 0; i < nvec; i++) {
int irq = pci_irq_vector(pdev, i);
int err = request_irq(irq, mydev_isr,
(nvec == 1) ? IRQF_SHARED : 0, // sharing only on legacy
"mydev", dev);
if (err)
return err;
// Steer each vector to its own CPU so completions land cache-hot.
irq_set_affinity_hint(irq, cpumask_of(i % num_online_cpus()));
}
return 0;
}
# Which device uses which vector, and where its interrupts are landing.
cat /proc/interrupts | head -15
# CPU0 CPU1 CPU2 CPU3
# 27: 1240912 0 0 0 PCI-MSI nvme0q0
# 28: 0 8823401 0 0 PCI-MSI nvme0q1
# ^ one queue per core, each on its own CPU: correct.
# 19: 40122 39887 40003 39914 IO-APIC ehci_hcd:usb1
# ^ legacy shared line, spread by irqbalance
# Interrupt rate per second -- the number that reveals a storm.
watch -n1 "grep nvme /proc/interrupts"
sudo perf stat -e irq:irq_handler_entry -a sleep 5
# Affinity: read it, then pin it.
cat /proc/irq/28/smp_affinity_list # 1 -> this vector goes to CPU 1
echo 2 | sudo tee /proc/irq/28/smp_affinity_list
systemctl status irqbalance # the daemon that usually does this
# Reducing interrupt rate at the source: coalescing.
sudo ethtool -c eth0
sudo ethtool -C eth0 rx-usecs 50 rx-frames 64 # batch: fewer interrupts
sudo ethtool -C eth0 adaptive-rx on # let the NIC decide
# Time spent in interrupt and softirq context -- the health check.
mpstat -P ALL 1
# CPU %usr %sys %irq %soft %idle
# 0 5.0 12.0 0.5 38.0 44.5 <- 38% softirq: network bottom halves
vmstat 1 # 'in' = interrupts/s, 'cs' = switches/s
cat /proc/softirqs | head -8 # NET_RX, NET_TX, BLOCK, TIMER counts
dmesg | grep -i 'nobody cared' # the kernel disabling a storming line
Key takeaways
- Interrupts invert control: the device signals completion, so the CPU can do other work while waiting — the mechanism that makes multiprogramming possible.
- Delivery indexes an interrupt vector table to find the handler, switches to kernel mode, and runs in interrupt context where sleeping is forbidden.
- Drivers split into a top half (acknowledge, grab status, defer) and a bottom half (softirq, tasklet, or workqueue) so handlers stay short.
- Legacy pin-based interrupts are shared, requiring every handler to check whether the interrupt was its own; MSI/MSI-X are message writes with thousands of distinct vectors and no sharing.
- MSI-X lets one device use many vectors, enabling per-queue steering so completions land on the core that issued the request.
- Interrupt storms are defended against with hardware coalescing, NAPI-style disabling, and the kernel's spurious-interrupt detection; affinity settings decide whether one core saturates or the load spreads.
🧪 Practice
- List the steps from a device asserting an interrupt to the handler's first instruction, and mark the point at which the mode switch occurs.
- Examine
/proc/interruptson your machine, identify one MSI device and one legacy line, and explain what the per-CPU distribution tells you.- Interview-style: "A driver's interrupt handler calls
kmalloc(GFP_KERNEL)and the machine occasionally deadlocks. Why?" Hint: consider what that allocation flag permits the allocator to do, and what is impossible in the context the handler runs in.
DMA Transfers
Theory
Interrupts fixed the waiting problem but not the moving problem. With programmed I/O, the CPU still copies every byte between the device's data register and memory, one word per instruction. Transferring 4 KB means about 1,000 loads and 1,000 stores; transferring a gigabyte means the CPU does nothing else for a long time. Direct Memory Access removes the CPU from the data path entirely.
The idea: give the device (or a dedicated DMA engine) the ability to read and write main memory itself. The CPU's role shrinks to setting up and being told when it is over:
- The driver allocates a buffer and obtains its bus address (below).
- It programs the device with that address, the length, and the direction.
- It issues the command and goes off to do something else entirely.
- The device transfers the data directly to or from memory, over many bus cycles.
- The device raises one interrupt — for the whole transfer, not per byte.
The saving is enormous: one interrupt and a few register writes instead of thousands of instructions. This is why every storage device, network card, and GPU is a bus master — capable of initiating its own memory transactions.
Three complications follow, and they are the substance of real DMA code.
Addresses. The device does not use virtual addresses, and on many systems it does not use physical addresses either. It uses bus addresses, which an IOMMU may translate. This is why drivers must call the DMA mapping API (dma_map_single, dma_alloc_coherent) rather than passing a pointer: the mapping call returns the address the device should use, establishes any IOMMU entry required, and bounces the buffer through an intermediate region if the device cannot address it (a 32-bit device on a machine with 64 GB of RAM). It also means DMA buffers must be physically contiguous, or described as a scatter-gather list of separate segments — which is what modern devices prefer, since it avoids the need for large contiguous allocations.
Coherence. If the CPU has cached part of the buffer and the device writes memory underneath it, the CPU may read stale data; if the CPU has dirty cache lines and the device reads memory, it may read stale data. On x86 the hardware keeps DMA coherent automatically; on many ARM systems it does not. The API handles both: dma_alloc_coherent returns memory that is consistent by construction (uncached or hardware-coherent), while dma_map_single plus dma_sync_single_for_cpu/_for_device explicitly flushes or invalidates around the transfer. Getting this wrong produces corruption that appears only on some architectures and only under load.
Protection. A device that can write any physical address can overwrite the kernel. This is not hypothetical — DMA attacks over Thunderbolt and FireWire are a real class of exploit. The IOMMU (Intel VT-d, AMD-Vi, ARM SMMU) is the answer: it sits between devices and memory as an MMU for I/O, so each device gets its own address space and can touch only the pages explicitly mapped for it. This is also what makes device passthrough to virtual machines safe.
Two refinements complete the picture. Scatter-gather lets one command describe many discrete memory segments, so a read into a user buffer spanning fifteen non-contiguous pages is still one operation. And cycle stealing describes how the DMA engine shares the memory bus with the CPU — taking cycles when available rather than seizing the bus, so the CPU is slowed somewhat but never stopped.
Example
PROGRAMMED I/O vs DMA (4 KB transfer)
PROGRAMMED I/O
CPU: [ld][st][ld][st][ld][st] ... x1024 ... + 1 interrupt
~2,048 instructions of pure copying; CPU unavailable throughout
DMA
CPU: [setup: 4 register writes] ---- goes and does other work ----+
|
DEVICE: reads/writes memory directly over the bus |
... 4096 bytes moved without the CPU ... |
<---------------- one interrupt: "done" --------------------+
~10 instructions of CPU involvement
THE THREE ADDRESS SPACES A BUFFER LIVES IN
driver's pointer : 0xffff888012340000 (kernel VIRTUAL)
physical address : 0x0000000012340000 (what the MMU produces)
bus/DMA address : 0x00000000fffe0000 (what the DEVICE must use)
^ produced by dma_map_*(); may differ because of
an IOMMU, a bounce buffer, or a bus offset.
Passing a kernel pointer to a device is a bug, even when it happens to work.
SCATTER-GATHER: ONE COMMAND, MANY SEGMENTS
user buffer spans 4 non-contiguous pages:
[page A][page B][page C][page D] (physically scattered)
SG list handed to the device:
+--------------------+--------+
| bus addr 0x1000000 | 4096 |
| bus addr 0x8A2F000 | 4096 |
| bus addr 0x0031000 | 4096 |
| bus addr 0x77C1000 | 4096 |
+--------------------+--------+
ONE transfer, no contiguous allocation, no copying.
THE IOMMU: AN MMU FOR DEVICES
without IOMMU with IOMMU
device --> ANY physical address device --> [IOMMU] --> only MAPPED pages
(can overwrite the |
kernel; Thunderbolt +-- per-device page tables
DMA attacks work) +-- unmapped access = fault
COHERENCE, AND WHERE IT BITES
x86: hardware-coherent. DMA snoops the caches. Nothing to do.
ARM: not necessarily. Sequence for a DEVICE->CPU read:
dma_map(..., DMA_FROM_DEVICE) invalidate the cache lines
... device writes memory ...
dma_sync_for_cpu(...) invalidate again before reading
Skip these and you read stale cache lines -- intermittently.
// The three DMA patterns a driver actually uses.
#include <linux/dma-mapping.h>
// ---- 1. COHERENT (consistent) mapping: for long-lived shared structures ----
// descriptor rings, doorbell areas -- things both sides touch constantly.
static int setup_ring(struct mydev *dev, struct device *d)
{
dev->ring = dma_alloc_coherent(d, RING_BYTES, &dev->ring_dma, GFP_KERNEL);
if (!dev->ring)
return -ENOMEM;
// dev->ring : CPU pointer
// dev->ring_dma : the address to program into the DEVICE
writel(lower_32_bits(dev->ring_dma), dev->regs + REG_RING_LO);
writel(upper_32_bits(dev->ring_dma), dev->regs + REG_RING_HI);
return 0;
}
// ---- 2. STREAMING mapping: for one transfer, then unmapped ----
static int do_transfer(struct mydev *dev, struct device *d,
void *buf, size_t len)
{
dma_addr_t dma;
// Map for the device. This may flush caches and/or allocate a bounce buffer.
dma = dma_map_single(d, buf, len, DMA_TO_DEVICE);
if (dma_mapping_error(d, dma))
return -EIO; // ALWAYS check: mapping can fail
writel(lower_32_bits(dma), dev->regs + REG_ADDR_LO);
writel(upper_32_bits(dma), dev->regs + REG_ADDR_HI);
writel(len, dev->regs + REG_LEN);
wmb(); // descriptors visible before the doorbell
writel(CTRL_START, dev->regs + REG_CONTROL);
wait_for_completion(&dev->done); // CPU is free for other work meanwhile
// Unmap: on non-coherent systems this performs the cache maintenance.
// Touching `buf` before this point is undefined behaviour.
dma_unmap_single(d, dma, len, DMA_TO_DEVICE);
return 0;
}
// ---- 3. SCATTER-GATHER: one command over many discrete pages ----
static int do_sg_transfer(struct mydev *dev, struct device *d,
struct scatterlist *sgl, int nents)
{
struct scatterlist *sg;
int i, mapped;
mapped = dma_map_sg(d, sgl, nents, DMA_FROM_DEVICE);
if (!mapped)
return -EIO;
// The IOMMU may COALESCE segments, so 'mapped' can be < nents.
for_each_sg(sgl, sg, mapped, i) {
dev->desc[i].addr = cpu_to_le64(sg_dma_address(sg)); // bus address
dev->desc[i].len = cpu_to_le32(sg_dma_len(sg));
}
wmb();
writel(mapped, dev->regs + REG_DESC_COUNT);
writel(CTRL_START, dev->regs + REG_CONTROL);
wait_for_completion(&dev->done);
dma_unmap_sg(d, sgl, nents, DMA_FROM_DEVICE);
return 0;
}
// Declare what the device can address, BEFORE any mapping call.
// A 32-bit device on a 64 GB machine needs bounce buffers; saying so lets
// the kernel arrange them instead of silently corrupting memory.
static int set_mask(struct device *d)
{
if (dma_set_mask_and_coherent(d, DMA_BIT_MASK(64)) == 0)
return 0;
return dma_set_mask_and_coherent(d, DMA_BIT_MASK(32));
}
# Is the IOMMU on? (It should be, for both security and VM passthrough.)
dmesg | grep -iE 'DMAR|IOMMU|AMD-Vi'
# DMAR: IOMMU enabled
# DMAR: Intel(R) Virtualization Technology for Directed I/O
ls /sys/kernel/iommu_groups/ | wc -l # 0 means no IOMMU
ls /sys/kernel/iommu_groups/12/devices/ # which devices share a group
# Enable it if absent (GRUB):
# intel_iommu=on iommu=pt (pt = passthrough for host devices: faster)
# amd_iommu=on iommu=pt
# Bounce buffers: the swiotlb pool, used when a device cannot reach a buffer.
dmesg | grep -i swiotlb
# software IO TLB: mapped [mem 0x...] (64MB)
grep -i swiotlb /proc/vmstat
# Heavy swiotlb traffic means a device's DMA mask is too small: it is copying
# every transfer twice, and throughput will be poor.
# Which devices are bus masters (allowed to initiate DMA).
sudo lspci -vv | grep -B8 'BusMaster+' | grep -E '^[0-9a-f]{2}:'
# DMA-attack surface: Thunderbolt/PCIe hotplug without an IOMMU is an open door.
cat /sys/bus/thunderbolt/devices/*/authorized 2>/dev/null
boltctl list 2>/dev/null
# Watch DMA happen: zero CPU copying during a large read.
dd if=/dev/nvme0n1 of=/dev/null bs=1M count=4000 iflag=direct &
mpstat -P ALL 1 5
# %sys stays low and %idle high while gigabytes move: that is DMA.
# With programmed I/O, one core would be pinned at 100%.
Key takeaways
- DMA lets a device read and write main memory itself, reducing the CPU's role to setup plus one completion interrupt per transfer instead of per word.
- Devices use bus addresses, not kernel pointers: the DMA mapping API returns the correct address and sets up any IOMMU entry or bounce buffer needed.
- Cache coherence must be managed explicitly on architectures that lack hardware coherence, via coherent allocations or
dma_sync_*around the transfer. - Scatter-gather lists let one command cover many physically discrete segments, avoiding large contiguous allocations and extra copies.
- An IOMMU gives each device its own address space, containing buggy devices and defending against DMA attacks over hot-pluggable buses.
- A device's DMA mask must be declared accurately; an understated address range forces bounce-buffer copying and an overstated one corrupts memory.
🧪 Practice
- Compare the CPU instruction count for transferring 1 MB by programmed I/O (one 32-bit word per load/store pair) versus by DMA. State the ratio.
- Explain why a driver may not pass a
kmallocpointer directly to a device, and name the three address spaces the buffer exists in.- Interview-style: "A driver works on x86 but corrupts data intermittently on ARM. What DMA-related mistake should you look for first?" Hint: one architecture keeps DMA coherent with the CPU caches in hardware and the other does not, so a missing API call is invisible on the first and fatal on the second.
<a id="112-io-software-layers"></a>
11.2 I/O Software Layers
Hardware gives every device a different interface; applications need one. This section covers the software that closes that gap: the drivers that know each device's specifics, the two abstractions the kernel presents them through, the three ways a program can wait for I/O, and the buffering that sits between them all.
Device Drivers
Theory
There are tens of thousands of device models and one kernel. The only way that works is if device-specific knowledge is confined to replaceable modules and everything above them sees a uniform interface. That module is the device driver, and it is the largest category of code in any modern kernel — the majority of Linux's source is drivers.
A driver's contract runs in two directions. Downward, it knows the device's registers, its command encoding, its quirks, and its errata. Upward, it implements a standard set of operations the kernel defined — open, read, write, ioctl, release for a character device; queue_rq for a block device; ndo_start_xmit for a network device. The kernel calls those operations through a table of function pointers, exactly as the VFS does for file systems (Section 10.2). The polymorphism is the point: read() works on a mouse, a disk, and a temperature sensor because each driver filled in the same slot with different code.
A driver's life cycle is worth understanding because it explains most driver bugs:
- Registration. The module declares which devices it supports — a table of PCI vendor/device IDs, a USB ID table, or a device-tree
compatiblestring on embedded systems. - Probe. When a matching device appears (at boot or on hot-plug), the bus calls the driver's
probe. This is where the driver maps registers, allocates buffers, requests interrupts, and registers the device with the appropriate subsystem. - Operation. The kernel invokes the driver's operations on behalf of user requests.
- Remove. On unplug or module unload, everything acquired in probe must be released, in reverse order. Failing to do so leaks memory, leaves interrupt handlers pointing at freed structures, or wedges the device.
Two structural ideas save enormous effort. First, the subsystem layer: a driver rarely implements a raw character device. It registers with a subsystem — the network stack, the block layer, V4L2 for video, ALSA for audio, input for keyboards — and gets protocol handling, queueing, and the user-facing interface for free. A new NIC driver implements transmit, receive, and configuration; it does not implement TCP.
Second, bus-level generic drivers. USB mass storage, HID, and CDC-ACM are class specifications, so one driver serves every conforming device. This is why a new USB flash drive needs no driver at all: it declares itself as mass storage and the existing driver handles it.
Where the driver runs is a genuine design axis. Kernel-space drivers are fast — no boundary crossings — but a bug takes the machine down and the security surface is the whole kernel. User-space drivers (FUSE for file systems, UIO and VFIO for devices, libusb, DPDK and SPDK for networking and storage) are isolated: a crash kills a process. Microkernels put nearly all drivers in user space by design, accepting message-passing overhead for robustness. The modern compromise is that mainstream kernels keep performance-critical drivers inside and increasingly host the rest outside.
Example
WHERE A DRIVER SITS
application: read(fd, buf, 4096)
|
+----v-------------------------------------------------+
| system call interface |
+----+--------------------------------------------------+
| SUBSYSTEM: block layer / network stack / input / ALSA |
| provides: queueing, protocol, user-facing interface |
+----+--------------------------------------------------+
| DRIVER: knows THIS device's registers and quirks |
| implements the subsystem's operation table |
+----+--------------------------------------------------+
| bus layer (PCI / USB / I2C / platform) |
+----+--------------------------------------------------+
v registers, interrupts, DMA
DEVICE
ONE INTERFACE, MANY IMPLEMENTATIONS
read(fd, ...)
|
+---------+---------+---------+----------+
| | | | |
disk drv mouse drv tty drv sensor drv net drv
| | | | |
NVMe USB HID UART I2C PCIe
DRIVER LIFE CYCLE (and the symmetry that must hold)
probe(): remove():
pci_enable_device() free_irq()
pci_request_regions() pci_iounmap()
pci_iomap() <------> pci_release_regions()
request_irq() pci_disable_device()
register_with_subsystem() unregister_from_subsystem()
Anything acquired on the left MUST be released on the right, in
reverse order. Most driver crashes are an asymmetry here.
KERNEL-SPACE vs USER-SPACE DRIVERS
kernel-space user-space (VFIO/UIO/FUSE)
speed no crossings syscall or mmap per batch
a bug kernel panic process dies, restart it
security surface entire kernel one process
debugging printk and crash dumps gdb, valgrind, normal tools
examples nvme, e1000, ext4 DPDK, SPDK, FUSE, libusb
// A complete (minimal) character driver: the operation table is the contract.
#include <linux/module.h>
#include <linux/fs.h>
#include <linux/cdev.h>
#include <linux/uaccess.h>
#define DEVBUF 4096
static char *buffer;
static size_t data_len;
static dev_t devno;
static struct cdev my_cdev;
static DEFINE_MUTEX(lock); // process context: a sleeping lock is fine
static int my_open(struct inode *inode, struct file *f)
{
return 0; // per-open state would be attached here
}
static ssize_t my_read(struct file *f, char __user *ubuf,
size_t count, loff_t *ppos)
{
ssize_t ret;
mutex_lock(&lock);
if (*ppos >= data_len) { ret = 0; goto out; } // 0 = EOF
count = min(count, data_len - (size_t)*ppos);
// NEVER dereference a user pointer directly: it may be invalid, swapped
// out, or another process's. copy_to_user handles faults safely.
if (copy_to_user(ubuf, buffer + *ppos, count)) { ret = -EFAULT; goto out; }
*ppos += count;
ret = count; // return BYTES TRANSFERRED, not success
out:
mutex_unlock(&lock);
return ret;
}
static ssize_t my_write(struct file *f, const char __user *ubuf,
size_t count, loff_t *ppos)
{
ssize_t ret;
if (count > DEVBUF) count = DEVBUF;
mutex_lock(&lock);
if (copy_from_user(buffer, ubuf, count)) { ret = -EFAULT; goto out; }
data_len = count;
ret = count;
out:
mutex_unlock(&lock);
return ret;
}
// ioctl: the escape hatch for operations read/write cannot express.
#define MY_IOC_RESET _IO('M', 1)
static long my_ioctl(struct file *f, unsigned int cmd, unsigned long arg)
{
switch (cmd) {
case MY_IOC_RESET:
mutex_lock(&lock); data_len = 0; mutex_unlock(&lock);
return 0;
default:
return -ENOTTY; // the conventional "unknown ioctl" errno
}
}
// THIS TABLE IS THE DRIVER'S ENTIRE UPWARD INTERFACE.
static const struct file_operations my_fops = {
.owner = THIS_MODULE,
.open = my_open,
.read = my_read,
.write = my_write,
.unlocked_ioctl = my_ioctl,
};
static int __init my_init(void)
{
int err;
buffer = kzalloc(DEVBUF, GFP_KERNEL);
if (!buffer) return -ENOMEM;
err = alloc_chrdev_region(&devno, 0, 1, "mydev"); // ask for a major number
if (err) goto free_buf;
cdev_init(&my_cdev, &my_fops);
err = cdev_add(&my_cdev, devno, 1); // device becomes live HERE
if (err) goto unregister;
pr_info("mydev: major=%d minor=%d\n", MAJOR(devno), MINOR(devno));
return 0;
unregister: // unwind in exact reverse order
unregister_chrdev_region(devno, 1);
free_buf:
kfree(buffer);
return err;
}
static void __exit my_exit(void)
{
cdev_del(&my_cdev);
unregister_chrdev_region(devno, 1);
kfree(buffer);
}
module_init(my_init);
module_exit(my_exit);
MODULE_LICENSE("GPL");
# Which driver is bound to which device.
lspci -k | head -12
# 00:1f.2 SATA controller: Intel ...
# Kernel driver in use: ahci
# Kernel modules: ahci
ls -l /sys/bus/pci/devices/0000:00:1f.2/driver
ls /sys/bus/usb/drivers/ # every USB driver registered
# Module management.
lsmod | head # loaded modules and their users
modinfo nvme # author, params, supported aliases
sudo modprobe -r nvme # unload (fails if in use)
sudo modprobe nvme poll_queues=4 # load with a parameter
cat /sys/module/nvme/parameters/poll_queues # read a live parameter
# Rebinding a device to a different driver -- how VFIO passthrough is set up.
echo 0000:02:00.0 | sudo tee /sys/bus/pci/devices/0000:02:00.0/driver/unbind
echo 8086 1533 | sudo tee /sys/bus/pci/drivers/vfio-pci/new_id
# The device now belongs to a user-space driver (a VM, or DPDK).
# Driver messages and failures.
sudo dmesg -w | grep -i 'probe\|driver'
journalctl -k -b | grep -iE 'firmware|failed to probe'
# Hot-plug events, live.
sudo udevadm monitor --kernel --property
udevadm info -a -n /dev/sda | head -20 # attributes udev rules can match
Key takeaways
- A driver confines device-specific knowledge to one module and exposes a standard operation table upward, so the kernel's generic code works across every device.
- The life cycle is register, probe, operate, remove — and everything acquired in probe must be released in reverse order in remove.
- Drivers normally register with a subsystem (block, network, input, ALSA) and inherit its queueing, protocol handling, and user interface rather than implementing them.
- Class drivers (USB mass storage, HID, CDC) serve every conforming device, which is why most peripherals need no vendor driver.
- Kernel-space drivers are fastest but a bug panics the machine; user-space drivers (VFIO, UIO, FUSE, DPDK, SPDK) trade some overhead for isolation and ordinary debugging tools.
- Driver code must never dereference user pointers directly —
copy_to_user/copy_from_userexist because those addresses can be invalid or unmapped.
🧪 Practice
- List the operations a character driver must supply and state which system call invokes each.
- Use
lspci -kandlsmodto identify the driver behind your network device, then read its parameters under/sys/module/.- Interview-style: "Why does a driver call
copy_to_userinstead ofmemcpywhen returning data to a process?" Hint: think about what guarantees the kernel has about a pointer that arrived from user space, and what must happen if that address is not currently valid.
Block vs Character Devices
Theory
The kernel does not treat every device identically. It sorts them into two families whose access models differ so fundamentally that they get separate subsystems, separate caching strategies, and separate driver interfaces.
A block device is addressable storage: a numbered array of fixed-size blocks that can be read or written in any order. Disks, SSDs, USB sticks, RAM disks, and loop devices are block devices. Because any block can be reached at any time, the kernel can do interesting things on the path: cache blocks in the page cache, read ahead, merge adjacent requests, reorder the queue, and postpone writes. The block layer is where all the machinery of Chapter 9 and Chapter 10 attaches.
A character device is a byte stream: bytes arrive or depart in sequence, and there is generally no meaningful "seek to byte 5,000". Terminals, serial ports, mice, sound cards, /dev/random, and /dev/null are character devices. Because the stream is sequential and often generated on demand, most of the block layer's tricks are meaningless — you cannot read ahead on a keyboard — so character drivers talk almost directly to user space.
The distinction shows up concretely:
| Property | Block device | Character device |
|---|---|---|
| Addressing | Random, by block number | Sequential byte stream |
| Granularity | Fixed blocks (512 B / 4 KB) | Single bytes |
| Kernel caching | Page cache, read-ahead, writeback | Usually none (small internal buffers) |
| Request reordering | Yes — the I/O scheduler | No |
seek meaningful | Yes | Usually not |
mmap typical | Yes | Only for special cases (framebuffers) |
ls -l type | b | c |
| Examples | /dev/sda, /dev/nvme0n1 | /dev/tty, /dev/null, /dev/random |
Both are named in the file system by a major and minor number. The major identifies the driver; the minor identifies which device that driver is responsible for. /dev/sda and /dev/sdb share a major and differ in minor. This pair, not the filename, is what the kernel dispatches on — which is why mknod can create a working device node under any name, and why a device node on a mounted removable medium is a security problem (hence the nodev mount option from Section 10.1).
A third family exists in practice: pseudo-devices that have no hardware at all. /dev/null discards writes and returns EOF on read; /dev/zero produces zeros forever; /dev/random and /dev/urandom produce cryptographic randomness; /dev/loop0 presents a file as a block device. These are pure kernel functions wearing a device interface, and they exist because the interface is useful even when there is nothing physical behind it.
Some devices can be reached both ways. Classic UNIX exposed a "raw" character interface to disks so databases could bypass the buffer cache; Linux dropped raw devices in favour of the O_DIRECT flag, which gets the same effect through the block interface. The underlying need — "let me manage my own caching" — is permanent even when the mechanism changes.
Example
THE TWO PATHS THROUGH THE KERNEL
BLOCK DEVICE CHARACTER DEVICE
read(fd, buf, 4096) read(fd, buf, 1)
| |
+----v-----------------+ +----v-----------------+
| page cache | HIT -> done | (little or no cache) |
+----v-----------------+ +----v-----------------+
| I/O scheduler | merge/sort | driver |
+----v-----------------+ +----v-----------------+
| driver | v
+----v-----------------+ keyboard / serial line
v
disk
Block: caching, read-ahead, merging, reordering all apply.
Char : none of them make sense on a byte stream from a human.
MAJOR AND MINOR: WHAT THE KERNEL ACTUALLY DISPATCHES ON
$ ls -l /dev/sda /dev/sda1 /dev/null /dev/tty0
brw-rw---- 8, 0 /dev/sda <- b = block, major 8, minor 0
brw-rw---- 8, 1 /dev/sda1 <- same driver (8), different minor
crw-rw-rw- 1, 3 /dev/null <- c = char, major 1, minor 3
crw--w---- 4, 0 /dev/tty0
major -> WHICH DRIVER minor -> WHICH DEVICE of that driver
The NAME is irrelevant to the kernel:
mknod /tmp/mydisk b 8 0 creates a second, working door to /dev/sda
which is exactly why `nodev` exists as a mount option.
PSEUDO-DEVICES: THE INTERFACE WITHOUT THE HARDWARE
/dev/null writes vanish, reads return EOF
/dev/zero an infinite supply of 0x00
/dev/full writes fail with ENOSPC (for testing error paths)
/dev/random cryptographic randomness
/dev/loop0 a FILE presented as a block device
# Identify the two families and their numbers.
ls -l /dev/sda /dev/nvme0n1 /dev/null /dev/tty /dev/random
# brw-rw---- 8, 0 /dev/sda
# brw-rw---- 259, 0 /dev/nvme0n1
# crw-rw-rw- 1, 3 /dev/null
stat -c '%n %F major=%t minor=%T' /dev/sda /dev/null # %t/%T are hex
cat /proc/devices # every registered major, by family
# Character devices:
# 1 mem
# 4 tty
# Block devices:
# 8 sd
# 259 blkext
# Block-device behaviour: caching and read-ahead exist.
sudo blockdev --getra /dev/sda # read-ahead, in 512-byte sectors
sudo blockdev --getbsz --getsize64 /dev/sda
sudo hdparm -t --direct /dev/sda # --direct bypasses the page cache
# Character-device behaviour: no caching, byte at a time.
head -c 16 /dev/urandom | xxd
echo "test" > /dev/null # discarded
dd if=/dev/zero of=/dev/null bs=1M count=1000 # pure kernel, no hardware
# Proving major/minor is what matters, not the name.
sudo mknod /tmp/fakenull c 1 3 && echo "gone" > /tmp/fakenull # works
sudo mknod /tmp/fakedisk b 8 0 && sudo head -c 64 /tmp/fakedisk | xxd | head -2
# A file made into a block device: the loop driver.
dd if=/dev/zero of=disk.img bs=1M count=100 status=none
sudo losetup -fP --show disk.img # /dev/loop0
sudo mkfs.ext4 -q /dev/loop0 && sudo mount /dev/loop0 /mnt/loop
lsblk /dev/loop0
sudo umount /mnt/loop && sudo losetup -d /dev/loop0
# Who creates the nodes: udev, from kernel uevents.
sudo udevadm monitor --udev &
sudo modprobe loop
cat /etc/udev/rules.d/* 2>/dev/null | head
import os, stat, time
def classify(path):
st = os.stat(path)
if stat.S_ISBLK(st.st_mode):
kind = "block"
elif stat.S_ISCHR(st.st_mode):
kind = "char"
else:
kind = "not a device"
return (f"{path:<16} {kind:<12} "
f"major={os.major(st.st_rdev):>3} minor={os.minor(st.st_rdev):>3}")
for p in ["/dev/sda", "/dev/null", "/dev/zero", "/dev/tty", "/dev/urandom"]:
try:
print(classify(p))
except FileNotFoundError:
pass
# Seeking: meaningful on one family, not the other.
fd = os.open("/dev/zero", os.O_RDONLY)
print("\n/dev/zero seek to 1MB:", os.lseek(fd, 1_000_000, os.SEEK_SET)) # accepted,
os.close(fd) # but meaningless
try:
fd = os.open("/dev/tty", os.O_RDONLY)
os.lseek(fd, 100, os.SEEK_SET) # ESPIPE: a terminal cannot seek
except OSError as e:
print("/dev/tty seek :", e.strerror)
finally:
try: os.close(fd)
except Exception: pass
# The caching difference, measured on a block device.
os.system("sync; echo 3 | sudo tee /proc/sys/vm/drop_caches > /dev/null")
t0 = time.perf_counter()
with open("/dev/sda", "rb") as f: f.read(1 << 20) # cold: reaches the device
cold = time.perf_counter() - t0
t0 = time.perf_counter()
with open("/dev/sda", "rb") as f: f.read(1 << 20) # warm: page cache
warm = time.perf_counter() - t0
print(f"\nblock device 1 MB read: cold {cold*1000:.1f} ms, warm {warm*1000:.1f} ms")
# A character device shows no such gap: there is nothing to cache.
Key takeaways
- Block devices are randomly addressable fixed-size-block storage; character devices are sequential byte streams.
- The block layer adds page caching, read-ahead, request merging, and scheduling — all of which are meaningless on a byte stream, so character drivers talk almost directly to user space.
- Every device node carries a major number (which driver) and a minor number (which device); the filename is irrelevant to the kernel.
- Because any name can be given to any major/minor pair, device nodes on untrusted media are a privilege-escalation risk, which is what
nodevprevents. - Pseudo-devices like
/dev/null,/dev/zero, and/dev/loop0use the device interface with no hardware behind them. - The "let me bypass the cache" need that raw character devices once served is met today by
O_DIRECTon the block interface.
🧪 Practice
- Run
ls -lon four device nodes and identify the family, major, and minor of each. Explain what two nodes sharing a major have in common.- Create a loop device from a file, format and mount it, then explain which operations in that sequence only work because it is a block device.
- Interview-style: "Why does read-ahead make sense for
/dev/sdabut not for/dev/ttyS0?" Hint: read-ahead is a bet about what will be requested next; ask whether that bet is even well-defined when the data does not exist yet.
Blocking and Non-Blocking I/O
Theory
When a program calls read() and the data is not there yet, something has to give. The three possible answers define the three I/O styles, and choosing among them is one of the most consequential decisions in any networked or interactive program.
Blocking I/O is the default and the simplest. The calling thread is put to sleep — moved out of the run queue entirely — until the data arrives, at which point the kernel wakes it and the call returns. The code reads exactly like the problem statement: request, then use the result. It costs the thread nothing while waiting (a blocked thread consumes no CPU), and it costs the programmer nothing in complexity.
Its limitation is that one thread can wait for exactly one thing. Serving 10,000 concurrent connections means 10,000 threads, each with its own stack (8 MB of reserved address space by default on Linux, typically 8-64 KB actually touched) and each contributing to scheduler and context-switch overhead. That is the "C10K problem", and it is why blocking I/O plus thread-per-connection stopped scaling in the 1990s.
Non-blocking I/O (O_NONBLOCK) changes the contract: if the operation cannot proceed immediately, it returns EAGAIN/EWOULDBLOCK instead of sleeping. The thread keeps control. Used naively — retrying in a loop — this is simply polling with all of polling's waste. Used correctly, it is half of the answer, and the other half is readiness notification.
I/O multiplexing provides that other half. select, poll, epoll, and kqueue let one thread ask "which of these hundreds of descriptors are ready?" and block until at least one is. Combined with non-blocking descriptors, a single thread can service thousands of connections: wait for readiness, then perform operations that are guaranteed not to block. This is the event loop, and it is the architecture of nginx, Redis, Node.js, and essentially every modern high-concurrency server.
The multiplexing calls are not equivalent, and the difference is asymptotic:
| Call | Interest set | Cost per call | Descriptor limit |
|---|---|---|---|
select | Rebuilt and copied every call | O(n) scan | FD_SETSIZE, typically 1024 |
poll | Rebuilt and copied every call | O(n) scan | None |
epoll (Linux) | Registered once in the kernel | O(ready), not O(n) | None |
kqueue (BSD/macOS) | Registered once | O(ready) | None |
epoll and kqueue win because the kernel keeps the interest set and returns only what is ready. With 10,000 idle connections and 5 active, poll copies and scans 10,000 entries per iteration; epoll_wait returns 5.
epoll additionally offers two trigger modes. Level-triggered reports readiness whenever data is available — forgiving, and the default. Edge-triggered reports only transitions, so the application must drain the descriptor until EAGAIN or lose the notification. Edge-triggered is faster and is where subtle bugs live.
The crucial caveat: readiness is not the same as completion. epoll tells you a read() will not block; you still perform the read, and the copy still costs CPU. For regular files it is worse — a disk file is always "ready", so multiplexing gives no benefit at all and the read blocks anyway. That gap is exactly what asynchronous I/O, the next topic, closes.
Example
THE THREE STYLES, SERVING THREE CONNECTIONS
BLOCKING, THREAD PER CONNECTION
thread1: read(c1) [=== blocked 40ms ===] process |
thread2: read(c2) [=== blocked 12ms ===] process | concurrent, but
thread3: read(c3) [=== blocked 90ms ===] process | 3 stacks, 3 contexts
10,000 connections -> 10,000 threads -> scheduler and memory pressure
NON-BLOCKING WITHOUT MULTIPLEXING (the naive trap)
loop: read(c1) EAGAIN, read(c2) EAGAIN, read(c3) EAGAIN, repeat...
ONE thread, but 100% CPU discovering that nothing has happened.
NON-BLOCKING + epoll (the event loop)
epoll_wait() [== sleeps, 0% CPU ==] -> returns {c2 ready}
read(c2) -> returns immediately, guaranteed
process c2
epoll_wait() [== sleeps ==] -> returns {c1, c3 ready}
ONE thread, near-zero idle cost, thousands of connections.
WHY epoll SCALES AND poll DOES NOT
10,000 connections, 5 active
poll : build 10,000-entry array -> copy to kernel -> kernel scans
all 10,000 -> copy back -> app scans all 10,000 for revents
=> O(n) per iteration, EVERY iteration
epoll: interest set already in the kernel; epoll_wait returns 5 entries
=> O(ready) per iteration
LEVEL-TRIGGERED vs EDGE-TRIGGERED
1000 bytes arrive; the app reads 400.
LEVEL-triggered: next epoll_wait reports ready again (600 remain). Safe.
EDGE-triggered : next epoll_wait reports NOTHING -- no new transition.
The remaining 600 bytes are stranded until more data
arrives. You MUST loop until EAGAIN.
THE LIMIT OF READINESS
socket : epoll says "ready" -> read() will not block. Works.
FILE : epoll ALWAYS says "ready" -> read() blocks on disk anyway.
Readiness models do not cover disk I/O. Use AIO / io_uring.
import socket, selectors, os, errno
# ---------- 1. BLOCKING: simple, but one thread waits for one thing ----------
def blocking_server(port=9001):
s = socket.socket(); s.bind(("", port)); s.listen()
while True:
conn, _ = s.accept() # BLOCKS here: no other work possible
data = conn.recv(4096) # BLOCKS again, for this client only
conn.sendall(data.upper())
conn.close()
# Serving N clients concurrently requires N threads.
# ---------- 2. NON-BLOCKING, naive: correct but burns a core ----------
def spinning_server(port=9002):
s = socket.socket(); s.bind(("", port)); s.listen()
s.setblocking(False)
while True:
try:
conn, _ = s.accept()
except BlockingIOError: # EAGAIN: nothing pending
continue # 100% CPU asking "anything yet?"
conn.setblocking(False)
# ---------- 3. NON-BLOCKING + MULTIPLEXING: the event loop ----------
def event_loop_server(port=9003):
sel = selectors.DefaultSelector() # epoll on Linux, kqueue on BSD
s = socket.socket()
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind(("", port)); s.listen(); s.setblocking(False)
sel.register(s, selectors.EVENT_READ, data=None) # registered ONCE
while True:
# Sleeps at 0% CPU; returns ONLY the descriptors that are ready.
for key, _ in sel.select(timeout=None):
if key.data is None: # the listening socket
conn, addr = key.fileobj.accept() # guaranteed not to block
conn.setblocking(False)
sel.register(conn, selectors.EVENT_READ, data=bytearray())
else: # a client socket
conn = key.fileobj
try:
chunk = conn.recv(4096) # guaranteed not to block
except (BlockingIOError, InterruptedError):
continue
if not chunk: # orderly close
sel.unregister(conn); conn.close(); continue
conn.sendall(chunk.upper())
# ONE thread, thousands of connections, near-zero cost while idle.
# ---------- edge-triggered: the drain-to-EAGAIN requirement ----------
def drain(conn):
"""With EPOLLET you MUST read until EAGAIN or lose the remaining data."""
out = bytearray()
while True:
try:
chunk = conn.recv(4096)
except BlockingIOError: # EAGAIN: NOW the socket is truly empty
return out # only now is it safe to go back to wait
if not chunk:
return out # peer closed
out += chunk
// epoll directly, showing what the Python selector wraps.
#include <sys/epoll.h>
#include <fcntl.h>
static void set_nonblocking(int fd) {
int flags = fcntl(fd, F_GETFL, 0);
fcntl(fd, F_SETFL, flags | O_NONBLOCK); // read/write now return EAGAIN
}
int run_loop(int listen_fd)
{
struct epoll_event ev, events[64];
int epfd = epoll_create1(0);
set_nonblocking(listen_fd);
ev.events = EPOLLIN; // level-triggered (no EPOLLET)
ev.data.fd = listen_fd;
epoll_ctl(epfd, EPOLL_CTL_ADD, listen_fd, &ev); // registered ONCE
for (;;) {
// Returns only READY descriptors: cost is O(ready), not O(registered).
int n = epoll_wait(epfd, events, 64, -1);
for (int i = 0; i < n; i++) {
int fd = events[i].data.fd;
if (fd == listen_fd) {
int c = accept4(listen_fd, NULL, NULL, SOCK_NONBLOCK);
ev.events = EPOLLIN | EPOLLET; // edge-triggered client
ev.data.fd = c;
epoll_ctl(epfd, EPOLL_CTL_ADD, c, &ev);
} else {
char buf[4096];
ssize_t r;
// EPOLLET: drain fully, or the rest is never reported again.
while ((r = read(fd, buf, sizeof buf)) > 0)
write(fd, buf, r);
if (r == 0 || (r < 0 && errno != EAGAIN)) {
epoll_ctl(epfd, EPOLL_CTL_DEL, fd, NULL);
close(fd);
}
}
}
}
}
# The descriptor limits an event-loop server runs into first.
ulimit -n # per-process soft limit (often 1024)
ulimit -Hn # hard limit
cat /proc/sys/fs/file-max # system-wide
sudo sysctl -w fs.file-max=2097152
# /etc/security/limits.conf:
# * soft nofile 1048576
# * hard nofile 1048576
# Watch a server's multiplexing calls.
strace -c -p $(pgrep -n nginx)
# % time seconds calls syscall
# 45.2 0.008 2104 epoll_wait <- one call covering many sockets
# 30.1 0.005 8021 read
ls -l /proc/$(pgrep -n nginx)/fd | head # epoll instances appear here
# How many connections a single-threaded event loop is actually holding.
ss -s
ss -tn state established | wc -l
# Blocking vs event-driven, measured.
wrk -t4 -c10000 -d30s http://localhost:9003/
# Compare RSS and context-switch counts against a thread-per-connection server:
pidstat -w -p $(pgrep -n nginx) 1 # cswch/s should stay low
Key takeaways
- Blocking I/O sleeps the calling thread until data arrives: simplest to write, and one thread can wait for only one thing.
- Thread-per-connection scales poorly because each thread costs stack memory and scheduler work — the C10K problem.
- Non-blocking descriptors return
EAGAINinstead of sleeping, but retrying in a loop is just polling and wastes a core. - Multiplexing (
select,poll,epoll,kqueue) lets one thread wait on many descriptors and is what makes the event-loop architecture possible. epollandkqueuekeep the interest set in the kernel and cost O(ready) rather than O(registered), which is why they replacedselectandpollat scale.- Edge-triggered mode reports only transitions, so the application must drain to
EAGAINor strand data; level-triggered is the safer default. - Readiness is not completion, and regular files are always "ready" — so multiplexing does not solve disk I/O at all.
🧪 Practice
- Write an echo server twice, once blocking with a thread per client and once with a
selectorsevent loop, and compare memory use at 500 connections.- Explain what
EAGAINmeans and give two situations in which a correct program receives it.- Interview-style: "Why does
epolloutperformpollat 10,000 connections but not at 10?" Hint: compare what each call transfers and scans per invocation, and consider what fraction of descriptors are typically ready.
Asynchronous I/O
Theory
Multiplexing solved "which descriptor is ready", but readiness has two limits: the application still performs the transfer itself, and regular files are always reported ready so disk I/O still blocks. Asynchronous I/O closes both gaps by changing what is submitted. Instead of asking "can I read?", the program says "perform this read and tell me when it is done".
The distinction is worth stating precisely, because the terms get used loosely:
- Synchronous blocking:
read()returns when the data is in your buffer; your thread slept meanwhile. - Synchronous non-blocking:
read()returns immediately, with data orEAGAIN; you retry. - Readiness-based (
epoll): the kernel tells you when a subsequentread()will not block; you still do the read. - Completion-based (true async): you submit the operation with its buffer; the kernel performs it and notifies you when the bytes are already in place.
Completion-based I/O is what Windows has offered since NT via I/O Completion Ports (IOCP), and it is generally considered the cleaner model: the transfer overlaps with your computation, not just the waiting.
UNIX took longer to get there. POSIX AIO (aio_read, aio_write) exists but is widely regarded as a disappointment: on Linux, glibc implements it with a hidden thread pool, so it is not truly asynchronous, and the kernel's own io_submit interface only works reliably with O_DIRECT, silently falling back to blocking otherwise. The result was that for two decades, high-performance Linux applications used thread pools for disk I/O and epoll for sockets — two separate mechanisms for one problem.
io_uring (Linux 5.1, 2019) is the modern answer and worth understanding in some detail, because it changes the shape of the interface rather than just its speed. Two ring buffers are shared between the application and the kernel via mmap: a submission queue (SQ) the application writes into, and a completion queue (CQ) the kernel writes into. To perform I/O, the application fills in a submission queue entry and advances a tail pointer — no system call at all. A single io_uring_enter can submit hundreds of operations, and in polled mode (SQPOLL, where a kernel thread watches the ring) even that disappears.
What this buys:
- Batching. Hundreds of operations per crossing instead of one per call.
- Uniformity. Files, sockets, timers,
accept,fsync,openat,statx— all through one interface, so an application needs one event mechanism rather than two. - Zero-copy paths with registered buffers and registered files, which skip per-operation reference counting and mapping.
- Chaining.
IOSQE_IO_LINKmakes one operation depend on the previous one's success, so a read-then-write sequence needs no round trip through user space.
The costs are honest ones: the programming model is harder (completions arrive out of order and must be correlated by user_data), buffer lifetime rules are strict (the kernel owns the buffer until the completion arrives), and its rapid evolution has produced a real security-bug history, so some hardened environments disable it.
The general principle survives whichever mechanism is used: asynchrony converts waiting into overlap. Whether it is worth the complexity depends on whether there is other work to overlap with.
Example
READINESS vs COMPLETION
READINESS (epoll) COMPLETION (io_uring, IOCP)
app: "tell me when I can read" app: "read 4 KB into THIS buffer"
... ...
kernel: "fd 7 is readable" kernel: "done; the bytes are there"
app: read(7, buf, 4096) <- the app: use buf directly
copy happens HERE, in your (the transfer already happened)
thread, on your time
Two crossings per operation One submission, one completion,
and the copy is yours. and both can be batched.
io_uring: TWO SHARED RING BUFFERS, MMAPPED
user space kernel
+---------------------------+
| SUBMISSION QUEUE (SQ) | app writes entries, advances TAIL
| [op][op][op][op][ ][ ] |-------------------------> kernel reads HEAD
+---------------------------+ |
| performs I/O
+---------------------------+ |
| COMPLETION QUEUE (CQ) | app reads entries, advances HEAD
| [res][res][ ][ ][ ] |<------------------------- kernel writes TAIL
+---------------------------+
SYSTEM CALLS PER 1,000 READS
read() : 1,000 crossings
epoll + read : ~1,000 reads + ~N epoll_wait calls
io_uring : 1 io_uring_enter (batched)
io_uring+SQPOLL: 0 (a kernel thread polls the ring)
ORDERING: COMPLETIONS ARE NOT IN SUBMISSION ORDER
submit: [A: read block 900][B: read block 2][C: read block 450]
complete: B (2) C (450) A (900)
Correlate with user_data. Assuming order is the classic async bug.
CHAINING (IOSQE_IO_LINK): no user-space round trip between steps
[openat] -> [read] -> [write] -> [close]
submitted as one linked chain; a failure breaks the rest of the chain.
# The conceptual model first: submit, do other work, collect completions.
import asyncio, time
async def fetch(name, delay):
"""Stands in for any awaitable I/O: the wait does not hold the thread."""
await asyncio.sleep(delay) # yields control; the loop runs others
return f"{name} done after {delay}s"
async def sequential():
t0 = time.perf_counter()
for name, d in [("a", 0.3), ("b", 0.3), ("c", 0.3)]:
await fetch(name, d) # each wait is serialised
return time.perf_counter() - t0
async def concurrent():
t0 = time.perf_counter()
# Submit all three, then collect: the waits OVERLAP.
await asyncio.gather(fetch("a", 0.3), fetch("b", 0.3), fetch("c", 0.3))
return time.perf_counter() - t0
async def main():
print(f"sequential : {await sequential():.2f}s") # ~0.90s: sum of waits
print(f"concurrent : {await concurrent():.2f}s") # ~0.30s: max of waits
asyncio.run(main())
# The CPU work is identical. Only the overlap changed -- which is the entire
# point of asynchrony, and also its limit: it helps only when you WAIT.
// io_uring with liburing: submit a batch, collect completions out of order.
#include <liburing.h>
#include <fcntl.h>
#define QD 64
int read_many(const char *path, off_t *offsets, int n)
{
struct io_uring ring;
struct io_uring_sqe *sqe;
struct io_uring_cqe *cqe;
char *bufs[QD];
int fd, i;
io_uring_queue_init(QD, &ring, 0); // create the SQ/CQ rings
fd = open(path, O_RDONLY | O_DIRECT);
// ---- SUBMIT: n operations, ZERO system calls so far ----
for (i = 0; i < n && i < QD; i++) {
posix_memalign((void **)&bufs[i], 4096, 4096); // O_DIRECT: aligned
sqe = io_uring_get_sqe(&ring); // a slot in the ring
io_uring_prep_read(sqe, fd, bufs[i], 4096, offsets[i]);
io_uring_sqe_set_data(sqe, (void *)(intptr_t)i); // CORRELATION TAG:
// completions are
// unordered
}
io_uring_submit(&ring); // ONE crossing for all n operations
// (with SQPOLL enabled, not even this)
// ---- COLLECT: completions arrive in DEVICE order, not submission order ----
for (i = 0; i < n; i++) {
io_uring_wait_cqe(&ring, &cqe);
int idx = (int)(intptr_t)io_uring_cqe_get_data(cqe); // which request?
if (cqe->res < 0)
fprintf(stderr, "request %d failed: %s\n", idx, strerror(-cqe->res));
else
process(bufs[idx], cqe->res); // bytes are ALREADY in place
io_uring_cqe_seen(&ring, cqe); // release the CQ slot
}
for (i = 0; i < n; i++) free(bufs[i]);
close(fd);
io_uring_queue_exit(&ring);
return 0;
}
// Chaining: openat -> read -> close with no user-space round trips.
void linked_chain(struct io_uring *ring, const char *path, char *buf)
{
struct io_uring_sqe *sqe;
sqe = io_uring_get_sqe(ring);
io_uring_prep_openat(sqe, AT_FDCWD, path, O_RDONLY, 0);
sqe->flags |= IOSQE_IO_LINK; // the NEXT entry waits for this one
sqe = io_uring_get_sqe(ring);
io_uring_prep_read(sqe, 0, buf, 4096, 0); // fd comes from the linked open
sqe->flags |= IOSQE_IO_LINK;
sqe = io_uring_get_sqe(ring);
io_uring_prep_close(sqe, 0);
io_uring_submit(ring); // three operations, one crossing
// If the open fails, the linked read and close are cancelled automatically.
}
# Is io_uring available and permitted here?
uname -r # 5.1+ for basic support, 5.10+ realistically
sysctl kernel.io_uring_disabled # 0=on, 1=privileged only, 2=off entirely
# Some hardened distributions ship 2 because of io_uring's CVE history.
# Measure the three paths against each other on the same device.
fio --name=sync --ioengine=sync --rw=randread --bs=4k --iodepth=1 \
--direct=1 --filename=/dev/nvme0n1 --runtime=10 --time_based
fio --name=libaio --ioengine=libaio --rw=randread --bs=4k --iodepth=64 \
--direct=1 --filename=/dev/nvme0n1 --runtime=10 --time_based
fio --name=uring --ioengine=io_uring --rw=randread --bs=4k --iodepth=64 \
--direct=1 --filename=/dev/nvme0n1 --runtime=10 --time_based
# Compare IOPS and the syscall count:
strace -c -f fio --name=x --ioengine=io_uring ... 2>&1 | tail -5
# Even fewer crossings: kernel-side submission polling.
fio --ioengine=io_uring --sqthread_poll=1 --hipri=1 ... # SQPOLL + IOPOLL
# What POSIX AIO actually does on Linux (spoiler: threads).
strace -f -e trace=clone,io_submit ./posix_aio_program
# clone(...) clone(...) <- glibc's hidden thread pool, not kernel async
# Applications already using it.
sudo ls -l /proc/*/fd 2>/dev/null | grep -c 'io_uring'
Key takeaways
- Asynchronous I/O is completion-based: you submit an operation with its buffer and are notified when the data is already in place, rather than when a transfer could begin.
- Readiness models (
epoll) still perform the copy in your thread and do not work for regular files, which are always reported ready. - POSIX AIO on Linux is largely a disappointment — glibc emulates it with threads, and the kernel path effectively requires
O_DIRECT. io_uringuses two mmapped ring buffers so submissions need no system call, batches hundreds of operations per crossing, and covers files, sockets, timers, and metadata operations uniformly.- Registered buffers and files cut per-operation overhead further, and
IOSQE_IO_LINKchains dependent operations without returning to user space. - Completions arrive out of order and must be correlated by a user tag; buffers belong to the kernel until the completion arrives.
- Asynchrony converts waiting into overlap, so it helps exactly as much as there is other work to overlap with.
🧪 Practice
- Explain the difference between readiness notification and completion notification, and give one case where only the latter helps.
- Run the three
fioengines above on the same device at the same queue depth and report IOPS and system-call counts for each.- Interview-style: "Why can
epollnot solve the problem of a server that reads many small files from disk?" Hint: ask whatepollreports for a regular file descriptor, and what the subsequentread()then does.
Buffering and Caching
Theory
Buffers and caches both hold data in memory on the way to or from a device, and they are constantly conflated. The distinction is about authority:
- A buffer is a staging area. The data is in transit; the buffer exists because producer and consumer cannot be perfectly synchronised.
- A cache is a copy. The authoritative data lives elsewhere; the cache exists to avoid fetching it again.
Buffering solves three distinct problems, and it is worth separating them because they justify different buffer designs.
Speed mismatch. A producer and consumer rarely run at the same rate. A program generating log lines at bursty intervals and a disk that prefers large sequential writes need something between them to absorb the difference — a bucket under a tap that runs unevenly. The buffer smooths bursts and lets each side work at its own natural granularity.
Size mismatch. Devices have a preferred transfer size — 4 KB sectors, 1,500-byte network frames — and applications have their own. Buffering accumulates or fragments so each side sees its natural unit. A program writing one byte at a time would otherwise trigger a 4 KB read-modify-write per byte.
Semantic requirements. Some operations need the data to stay still. A DMA transfer must not have its source buffer modified mid-transfer, so the kernel copies into a stable buffer first. Network retransmission needs the original bytes available after send() returns. The copy is the price of the guarantee.
The classic arrangements:
| Scheme | Behaviour | Trade-off |
|---|---|---|
| Single buffer | Producer waits while the consumer drains | Simple, no overlap |
| Double buffer | Fill one while draining the other | Overlap; twice the memory |
| Circular buffer | N slots, producer and consumer chase each other | Smooths bursts; the standard design |
Double buffering is the classic overlap technique — the same idea as a graphics back buffer. Circular (ring) buffers are what real systems use, because they handle variable rates and, in the single-producer/single-consumer case, can be implemented lock-free with just two indices and a memory barrier.
Caching is the other half. The page cache (Section 10.4) makes most reads memory-speed, and the same principle appears at every level: the device's own DRAM cache, the block layer, the file system, the application, and — for network services — a separate cache tier. The recurring danger is multiple layers caching the same data, which wastes memory and, worse, produces inconsistency when one layer is invalidated and another is not. Databases typically use O_DIRECT precisely to eliminate the kernel's copy, because they manage a better-informed cache of their own.
Buffering is not free, and the cost is copies. A naive file-to-socket path copies four times: device to page cache, page cache to user buffer, user buffer to socket buffer, socket buffer to NIC. Each copy costs memory bandwidth and cache pollution. The zero-copy interfaces exist to remove them: sendfile() keeps the data in the kernel, splice() moves it between descriptors via a pipe, mmap removes the user-space copy, and MSG_ZEROCOPY and io_uring's registered buffers remove more. On a server pushing 10 GB/s, removing two copies of 10 GB/s is 20 GB/s of memory bandwidth returned.
Example
BUFFER vs CACHE
BUFFER (data in transit) CACHE (a copy of data that lives elsewhere)
producer -> [buffer] -> consumer device --> [cache] --> reader
the data is HERE and nowhere the AUTHORITATIVE copy is on the device;
else until consumed the cache can be dropped at any time
SINGLE / DOUBLE / CIRCULAR
SINGLE producer: [fill......] [fill......]
consumer: [drain] [drain]
-> strictly alternating, no overlap
DOUBLE producer: [fill A][fill B][fill A][fill B]
consumer: [drain A][drain B][drain A]
-> full overlap; costs 2x memory
CIRCULAR (ring)
+---+---+---+---+---+---+---+---+
| D | D | D | | | | D | D |
+---+---+---+---+---+---+---+---+
^tail ^head
producer advances head, consumer advances tail
absorbs BURSTS in both directions; the standard design
THE COPY COUNT ON A FILE->SOCKET PATH
NAIVE read()+write() : 4 copies, 2 context switches
disk --DMA--> page cache
page cache --CPU--> user buffer <- copy 1 (and a syscall)
user buffer --CPU--> socket buffer <- copy 2 (and a syscall)
socket buffer --DMA--> NIC
sendfile() : 2 copies (or 1 with scatter-gather DMA)
disk --DMA--> page cache
page cache --DMA--> NIC <- never enters user space
At 10 GB/s, removing two CPU copies returns ~20 GB/s of memory bandwidth
and stops evicting the application's working set from L3.
import os, time, socket
# ---------- buffering: the same bytes, different call counts ----------
N = 200_000
data = b"log line\n"
t0 = time.perf_counter()
fd = os.open("/tmp/unbuf.log", os.O_CREAT | os.O_WRONLY | os.O_TRUNC, 0o644)
for _ in range(N):
os.write(fd, data) # ONE system call per line
os.close(fd)
unbuffered = time.perf_counter() - t0
t0 = time.perf_counter()
with open("/tmp/buf.log", "wb", buffering=64 * 1024) as f:
for _ in range(N):
f.write(data) # accumulates; ~28 system calls total
buffered = time.perf_counter() - t0
print(f"unbuffered {unbuffered:.3f}s buffered {buffered:.3f}s "
f"{unbuffered/buffered:.1f}x")
# ---------- a lock-free SPSC ring buffer: the standard design ----------
class RingBuffer:
"""Single producer, single consumer. No lock needed: each index has
exactly one writer, and one slot is left empty to distinguish
'full' from 'empty'."""
def __init__(self, capacity):
self.buf = [None] * capacity
self.cap = capacity
self.head = 0 # written ONLY by the producer
self.tail = 0 # written ONLY by the consumer
def push(self, item):
nxt = (self.head + 1) % self.cap
if nxt == self.tail:
return False # full: apply backpressure here
self.buf[self.head] = item
self.head = nxt # publish AFTER the store
return True
def pop(self):
if self.tail == self.head:
return None # empty
item = self.buf[self.tail]
self.tail = (self.tail + 1) % self.cap
return item
rb = RingBuffer(8)
print("\npushed:", [rb.push(i) for i in range(9)]) # last one fails: full
print("popped:", [rb.pop() for _ in range(4)])
# ---------- zero-copy: the same transfer, without the user-space round trip ----------
def naive_send(path, sock):
with open(path, "rb") as f:
while chunk := f.read(65536): # copy 1: kernel -> user
sock.sendall(chunk) # copy 2: user -> kernel
def zerocopy_send(path, sock):
with open(path, "rb") as f:
offset, size = 0, os.fstat(f.fileno()).st_size
while offset < size:
# The bytes never enter user space at all.
sent = os.sendfile(sock.fileno(), f.fileno(), offset, size - offset)
if sent == 0:
break
offset += sent
# Buffer memory, per subsystem.
free -h
# total used free shared buff/cache available
# Mem: 31Gi 8Gi 2Gi 1Gi 21Gi 22Gi
# ^ "buff/cache" is RECLAIMABLE
grep -E 'Buffers|Cached|Dirty' /proc/meminfo
# Buffers: block-device metadata buffers
# Cached: the page cache (file data)
# Network socket buffers: where bursts are absorbed.
sysctl net.core.rmem_max net.core.wmem_max
sysctl net.ipv4.tcp_rmem net.ipv4.tcp_wmem # min / default / max
sudo sysctl -w net.core.rmem_max=134217728 # for long fat pipes
ss -tmi | head -6 # per-socket buffer usage
# NIC ring buffers: where packet bursts are absorbed before the driver runs.
sudo ethtool -g eth0
# Pre-set maximums: RX: 4096 TX: 4096
# Current settings: RX: 512 TX: 512
sudo ethtool -G eth0 rx 2048 tx 2048 # raise if drops are occurring
ip -s link show eth0 | grep -A2 RX # 'dropped' rising = ring too small
# Zero-copy in practice.
strace -c -e trace=sendfile,splice,read,write nginx -g 'daemon off;' &
# nginx with sendfile on: few read/write pairs, many sendfile calls.
grep -r 'sendfile' /etc/nginx/nginx.conf
# Avoiding double caching: let the application own its cache.
# PostgreSQL: shared_buffers + the OS page cache (deliberately both)
# Oracle/MySQL InnoDB with O_DIRECT: skip the OS cache entirely
mysql -e "SHOW VARIABLES LIKE 'innodb_flush_method';" # O_DIRECT
Key takeaways
- A buffer stages data in transit; a cache holds a copy whose authoritative version lives elsewhere. The difference is which copy is authoritative.
- Buffering exists to bridge speed mismatches, size mismatches, and semantic requirements such as keeping DMA source data stable.
- Double buffering overlaps production and consumption; circular buffers handle variable rates and can be lock-free for one producer and one consumer.
- Multiple layers caching the same data wastes memory and risks inconsistency, which is why databases often use
O_DIRECTand manage their own cache. - Every buffer boundary is a potential copy, and copies cost memory bandwidth and cache residency, not just time.
- Zero-copy paths (
sendfile,splice,mmap,MSG_ZEROCOPY, registered buffers) remove those copies, which matters enormously at high throughput.
🧪 Practice
- Write the same 100,000 log lines with and without library buffering, and use
strace -cto explain the timing difference by system-call count.- Implement a circular buffer with
pushandpop, and explain why leaving one slot empty distinguishes full from empty.- Interview-style: "A file server is CPU-bound at 40% of line rate, with most cycles in
memcpy. What do you change and why?" Hint: count the copies on the read-then-write path, then name the call that eliminates the ones that pass through user space.
Spooling
Theory
Some devices cannot be shared by interleaving. If two processes write to a disk file alternately, the file system keeps their data separate. If two processes write to a printer alternately, you get one ruined document with lines from both jobs. The device has no notion of concurrent streams, and the output is physical and irreversible.
Spooling — Simultaneous Peripheral Operation On-Line — is the answer: instead of letting processes write to the device, give each one a file, and have a single dedicated daemon feed those files to the device one complete job at a time. The spool directory becomes a queue, and the device sees exactly one client.
This buys four things at once:
- Serialisation without blocking. Ten users can print simultaneously from their own point of view; each writes a file and continues immediately. The jobs are serialised at the device, not at the application.
- Speed decoupling. A program generating a 200-page document at memory speed does not run at printer speed. It writes the spool file and exits.
- Queue management. Because jobs are now files with metadata, they can be listed, prioritised, held, reordered, cancelled, and retried — none of which is possible with a direct device write.
- Fault tolerance. If the printer jams or the machine reboots, the spool files persist and the jobs resume. Nothing is lost mid-stream.
Spooling was historically the answer to slow card readers and line printers — in batch systems it was the entire reason a job's output did not hold the CPU hostage. But the pattern long outlived printers, and recognising it in modern form is the useful part:
- Mail queues.
sendmailand Postfix spool outbound mail to disk, then retry delivery over hours or days. Accepting a message and delivering it are deliberately separated. - Print systems. CUPS still does exactly the classic thing, in
/var/spool/cups. - Batch and cron systems.
atandcronspool jobs in/var/spool/. - Message queues. RabbitMQ, Kafka, SQS, and every task queue are spooling generalised: a durable intermediary so producers and consumers need not be simultaneously available or equally fast.
- Log shipping and CI pipelines. Write locally, ship asynchronously, retry on failure.
The general pattern is worth naming because it is a design tool: when a resource cannot be shared concurrently, do not serialise the clients — serialise the work by putting a durable queue in front of it. The costs are the ones any queue has: disk space for the backlog, added latency before the work actually happens, a queue that can grow without bound if the consumer is slower than the producer, and the need for a policy about what to do when the queue fills or a job repeatedly fails.
Example
WITHOUT SPOOLING: THE DEVICE CANNOT MULTIPLEX
process A ---> [ PRINTER ] <--- process B
output: "Dear Sir,Invoice #42Yours faithfully,Total: $99"
^ two documents interleaved, paper wasted, both unusable
The alternative -- an exclusive lock -- blocks B for A's entire job.
WITH SPOOLING: ONE WRITER, MANY PRODUCERS
process A --write--> /var/spool/cups/d00042-001 --+
process B --write--> /var/spool/cups/d00043-001 --+--> [ SPOOLER ]
process C --write--> /var/spool/cups/d00044-001 --+ |
(each returns IMMEDIATELY) v
[ PRINTER ]
one COMPLETE job at a time
THE SPOOL DIRECTORY IS A QUEUE, SO IT CAN BE MANAGED
$ lpq
Rank Owner Job Files Total Size
active ana 42 report.pdf 842000 bytes
1st ben 43 invoice.ps 12000 bytes
2nd cara 44 thesis.pdf 4200000 bytes
operations a direct device write could never support:
lprm 43 cancel
lp -i 44 -H hold hold
lp -q 90 ... prioritise
(and after a crash, the files are still there)
THE SAME PATTERN, MODERN NAMES
printing : app -> spool dir -> cupsd -> printer
mail : app -> /var/spool/postfix -> SMTP -> remote server
batch : at -> /var/spool/cron/atjobs -> atd
messaging : producer -> Kafka topic -> consumer group
CI : job -> queue -> runner pool
logs : app -> local buffer file -> shipper -> aggregator
THE COSTS ANY QUEUE HAS
- disk space for the backlog (a stuck printer fills /var)
- latency: the work happens later, not now
- unbounded growth if consumers are slower than producers
- a policy is REQUIRED for: queue full, job poisoned, retries exhausted
# The classic case, still working exactly as described.
lpstat -p -d # printers and the default
lp -d office-printer report.pdf # returns IMMEDIATELY
# request id is office-printer-42 (1 file(s))
lpq # the queue
sudo ls -l /var/spool/cups/ # the actual spool files
# -rw------- 1 root lp 842000 d00042-001 <- the document data
# -rw-r----- 1 root lp 512 c00042 <- the control file (metadata)
lprm 42 # cancel a queued job
lp -i 43 -H hold # hold it
cancel -a office-printer # drain the queue
# Mail: the same pattern, with retry over days.
mailq
postqueue -p
# -Queue ID- --Size-- ----Arrival Time---- -Sender/Recipient-------
# 3F2A1B4C21 2451 Thu Sep 11 09:14:02 sender@example.com
# (connect to mx.example.net: Connection timed out)
sudo postqueue -f # flush: retry everything now
sudo postsuper -d 3F2A1B4C21 # delete one message
ls -l /var/spool/postfix/deferred/ # jobs waiting for a retry
# Batch jobs.
echo "backup.sh" | at 02:00
atq
ls -l /var/spool/cron/atjobs/
# What happens when the consumer stops: the queue is the failure mode.
df -h /var # a stuck spooler fills this
du -sh /var/spool/*
find /var/spool -type f -mtime +7 # jobs nobody is retrying successfully
# The modern generalisation.
kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
--describe --group workers
# TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG
# tasks 0 88213 91007 2794
# ^ the backlog, in messages
redis-cli LLEN task_queue # the same number, for a Redis queue
import os, json, time, uuid, fcntl
# A spooler in miniature: producers write files, one consumer drains them.
SPOOL = "/tmp/myspool"
os.makedirs(SPOOL, exist_ok=True)
def submit(payload, priority=50):
"""Producer side: durable, atomic, and returns immediately."""
job_id = f"{priority:03d}-{int(time.time())}-{uuid.uuid4().hex[:8]}"
tmp = os.path.join(SPOOL, f".{job_id}.tmp")
final = os.path.join(SPOOL, f"{job_id}.job")
with open(tmp, "w") as f:
json.dump({"id": job_id, "payload": payload,
"submitted": time.time(), "attempts": 0}, f)
f.flush()
os.fsync(f.fileno()) # durable BEFORE it becomes visible
os.rename(tmp, final) # atomic publish: no partial job is ever seen
return job_id
def drain(process_fn, max_attempts=3):
"""Consumer side: ONE worker touches the device, jobs run to completion."""
# Sorting by name sorts by priority prefix, then by submission time.
for name in sorted(os.listdir(SPOOL)):
if not name.endswith(".job"):
continue
path = os.path.join(SPOOL, name)
# Claim the job exclusively, so two spoolers cannot double-print it.
try:
fd = os.open(path, os.O_RDWR)
fcntl.flock(fd, fcntl.LOCK_EX | fcntl.LOCK_NB)
except (FileNotFoundError, BlockingIOError):
continue # another worker has it, or it is gone
try:
job = json.load(os.fdopen(os.dup(fd), "r"))
process_fn(job["payload"]) # the device sees ONE complete job
os.unlink(path) # success: remove from the queue
except Exception as e:
job["attempts"] += 1
if job["attempts"] >= max_attempts:
os.rename(path, path + ".failed") # dead-letter: do NOT retry
else: # retry policy is REQUIRED
os.lseek(fd, 0, os.SEEK_SET); os.ftruncate(fd, 0)
os.write(fd, json.dumps(job).encode())
finally:
os.close(fd)
for doc, pri in [("report.pdf", 50), ("urgent.pdf", 10), ("thesis.pdf", 90)]:
print("queued:", submit(doc, pri))
drain(lambda p: print("printing:", p))
# Output order: urgent.pdf (010), report.pdf (050), thesis.pdf (090).
Key takeaways
- Spooling puts a durable queue in front of a device that cannot be shared by interleaving, so one daemon feeds the device complete jobs at a time.
- Producers write a file and return immediately, which decouples their speed from the device's and avoids blocking them for the whole job.
- Turning jobs into files makes listing, prioritising, holding, cancelling, and retrying possible — none of which a direct device write supports.
- Spool files persist across crashes, so a jam or reboot does not lose work mid-stream.
- The pattern generalises far beyond printers: mail queues,
at/cron, message brokers, CI pipelines, and log shipping are all spooling. - Every spool needs a policy for backlog growth, retry limits, and poisoned jobs, and a stuck consumer will eventually fill the disk.
🧪 Practice
- Explain what physically happens if two processes write to a printer device simultaneously, and how spooling prevents it without blocking either process.
- Inspect
/var/spool/on your machine and identify three subsystems using it. For one, find the metadata that accompanies each queued item.- Interview-style: "A task queue's backlog grows steadily during business hours and drains overnight. Is this a problem?" Hint: consider whether the queue is absorbing bursts or hiding a capacity deficit, and what the latency of the oldest item implies in each case.
<a id="113-kernel-io-subsystem"></a>
11.3 Kernel I/O Subsystem
Above the drivers sits the kernel code that serves every device alike: deciding what order requests go out in, coping when they fail, keeping one process's I/O away from another's, granting exclusive use where it is needed, and giving administrators the levers to tune all of it.
I/O Scheduling
Theory
Requests arrive at the block layer faster than a device can serve them, so a queue forms — and once there is a queue, the order in which it drains is a policy decision. I/O scheduling is that policy, and its correct answer changed completely when storage stopped having moving parts.
The classical objective was minimising mechanical motion. Chapter 9 covered the algorithms — FCFS, SSTF, SCAN, C-SCAN, LOOK, C-LOOK — all of which exist to keep a disk arm from thrashing back and forth. On a device where positioning costs 8 ms and transfer costs 20 µs, reordering a queue can improve throughput threefold, and it is worth considerable effort.
Two device-independent optimisations sit alongside that, and they survive the transition to flash:
- Merging. Two requests for adjacent blocks become one larger request. This reduces per-request overhead (each request costs a structure, a queue entry, and a completion) and is worth doing even when positioning is free.
- Plugging. The queue is deliberately held closed for a brief moment when it was empty, so that requests arriving together can be merged and sorted rather than dispatched one at a time.
Once the arm disappeared, the objectives changed. NVMe devices have internal parallelism across many flash dies, deep queues, and no positioning cost — so sorting by block number buys nothing and costs CPU and latency. Linux's answer was a new architecture, blk-mq (multi-queue block layer): per-CPU software queues feeding multiple hardware queues, so submission needs no shared lock and each core's completions come back to that core.
The schedulers that matter today:
| Scheduler | Designed for | Behaviour |
|---|---|---|
none | NVMe, fast SSDs | No reordering; merge only. Lowest overhead. |
mq-deadline | HDDs, SATA SSDs | Sorted queue plus per-request deadlines (500 ms reads, 5 s writes) so nothing starves |
bfq | Desktops, interactive | Proportional-share fairness per process; best perceived responsiveness under load |
kyber | Fast multi-queue devices | Targets latency directly by throttling queue depth to hit a configured p99 |
Two ideas deserve emphasis because they recur.
Reads and writes are not symmetric. A read usually has a thread blocked on it; a write usually went into the page cache and nobody is waiting. So deadline schedulers give reads much tighter deadlines than writes. Getting this backwards produces a system that benchmarks well and feels terrible.
Fairness needs enforcement. Without it, one process doing sequential streaming I/O can starve another doing small random reads, because the streaming requests keep arriving and keep merging. Deadlines bound the damage; cgroup I/O controllers (io.max, io.weight, io.latency) go further, capping IOPS and bandwidth per control group — which is how containers and multi-tenant systems avoid a noisy neighbour consuming the whole device.
The practical guidance is short: none for NVMe, mq-deadline for SATA SSDs and HDDs in servers, bfq for interactive desktops — and measure, because the wrong choice can cost a factor of two in either direction.
Example
WHAT A SCHEDULER DOES BETWEEN SUBMISSION AND THE DEVICE
processes: A(blk 500) B(blk 12) C(blk 501) D(blk 13) E(blk 502)
|
+----v--------------------------------------------------+
| per-CPU SOFTWARE queues (blk-mq): no shared lock |
+----v--------------------------------------------------+
| SCHEDULER: merge adjacent | sort | enforce deadlines |
| merged: {500,501,502} -> ONE request |
| merged: {12,13} -> ONE request |
+----v--------------------------------------------------+
| HARDWARE queues (one per core on NVMe) |
+----v--------------------------------------------------+
v
device
5 requests became 2. That saving applies to EVERY device type.
REORDERING: WORTH IT ON A DISK, HARMFUL ON FLASH
HDD, head at 53, queue {98,183,37,122,14,124,65,67}
FCFS : 53->98->183->37->122->14->124->65->67 = 640 cylinders
C-LOOK : 53->65->67->98->122->124->183->14->37 = 322 cylinders
~2x less arm motion -> ~2x throughput. REORDERING PAYS.
NVMe, same queue of LBAs
no arm, internal parallelism across dies, 2 us service time
sorting costs CPU and ADDS latency; the device reorders better
than the kernel can. USE 'none'.
READ vs WRITE ASYMMETRY (why deadline schedulers separate them)
read : a thread is BLOCKED right now, waiting -> deadline 500 ms
write : already in the page cache, nobody waits -> deadline 5000 ms
Treating them equally makes an interactive system feel broken while
its throughput numbers look fine.
CGROUP THROTTLING: THE NOISY-NEIGHBOUR FIX
without: [backup job] ############################ device saturated
[database ] # starved
with io.max on the backup cgroup:
[backup job] ######## capped at 50 MB/s
[database ] #################### responsive
# Which scheduler is active, and what is available.
cat /sys/block/nvme0n1/queue/scheduler
# [none] mq-deadline kyber bfq <- brackets mark the active one
cat /sys/block/sda/queue/scheduler
# none [mq-deadline] kyber bfq
# Change it (immediately effective, no remount).
echo none | sudo tee /sys/block/nvme0n1/queue/scheduler
echo mq-deadline | sudo tee /sys/block/sda/queue/scheduler
echo bfq | sudo tee /sys/block/sda/queue/scheduler # desktop
# Persist with a udev rule, because sysfs resets at boot.
cat <<'EOF' | sudo tee /etc/udev/rules.d/60-ioschedulers.rules
ACTION=="add|change", KERNEL=="nvme[0-9]n[0-9]", ATTR{queue/scheduler}="none"
ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/rotational}=="1", ATTR{queue/scheduler}="bfq"
ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/rotational}=="0", ATTR{queue/scheduler}="mq-deadline"
EOF
# Deadline tuning: the read/write asymmetry, in milliseconds.
cat /sys/block/sda/queue/iosched/read_expire # 500
cat /sys/block/sda/queue/iosched/write_expire # 5000
cat /sys/block/sda/queue/iosched/writes_starved # 2 reads batches per write batch
# The queue's own parameters.
cat /sys/block/nvme0n1/queue/nr_requests # queue depth
cat /sys/block/nvme0n1/queue/rotational # 0 = flash (affects defaults)
cat /sys/block/nvme0n1/queue/nomerges # 0 = merge, 2 = never merge
ls /sys/block/nvme0n1/mq/ # one dir per hardware queue
# Compare schedulers on YOUR workload -- the only way to choose correctly.
for s in none mq-deadline kyber bfq; do
echo $s | sudo tee /sys/block/sda/queue/scheduler > /dev/null
echo "=== $s ==="
fio --name=t --filename=/dev/sda --rw=randread --bs=4k --iodepth=32 \
--numjobs=4 --direct=1 --runtime=20 --time_based --group_reporting \
--output-format=terse | awk -F';' '{print " IOPS="$8" p99us="$28}'
done
# Per-cgroup limits: the multi-tenant answer.
sudo mkdir -p /sys/fs/cgroup/backup
echo "259:0 rbps=52428800 wbps=52428800" | \
sudo tee /sys/fs/cgroup/backup/io.max # 50 MB/s each way
echo "259:0 target=10000" | sudo tee /sys/fs/cgroup/db/io.latency # protect 10ms
echo $$ | sudo tee /sys/fs/cgroup/backup/cgroup.procs
cat /sys/fs/cgroup/backup/io.stat
# Observing the queue in flight.
iostat -x 1
# Device r/s w/s rareq-sz aqu-sz r_await w_await %util
# nvme0n1 8420 120 4.0 12.3 1.42 0.31 94.2
# ^ aqu-sz = average queue depth; r_await = read latency (ms)
sudo biolatency-bpfcc 10 1 # latency histogram
sudo biotop-bpfcc # which processes are generating the I/O
# Why merging matters regardless of device type, and why sorting does not.
PER_REQUEST_OVERHEAD_US = 2.0 # request struct, queue entry, completion
TRANSFER_US_PER_KB = 0.5
def cost(requests_kb):
"""requests_kb: list of request sizes."""
return sum(PER_REQUEST_OVERHEAD_US + TRANSFER_US_PER_KB * kb
for kb in requests_kb)
unmerged = [4] * 256 # 256 separate 4 KB requests
merged = [1024] # one 1 MB request, same bytes
print(f"256 x 4 KB unmerged : {cost(unmerged):8.1f} us")
print(f"1 x 1 MB merged : {cost(merged):8.1f} us")
print(f"overhead removed : {cost(unmerged)-cost(merged):8.1f} us\n")
# Reordering: valuable on a disk, pure cost on flash.
queue = [98, 183, 37, 122, 14, 124, 65, 67]
head = 53
def seek_total(order, start):
pos, total = start, 0
for c in order:
total += abs(c - pos); pos = c
return total
clook = sorted([c for c in queue if c >= head]) + sorted([c for c in queue if c < head])
print(f"HDD FCFS : {seek_total(queue, head):>4} cylinders of arm motion")
print(f"HDD C-LOOK: {seek_total(clook, head):>4} cylinders "
f"({seek_total(queue,head)/seek_total(clook,head):.1f}x better)")
SORT_CPU_US, NVME_SERVICE_US = 0.8, 2.0
print(f"\nNVMe: service {NVME_SERVICE_US} us, sorting costs {SORT_CPU_US} us/request")
print(f" -> sorting adds {SORT_CPU_US/NVME_SERVICE_US:.0%} latency and saves nothing.")
print(" -> scheduler 'none': merge, do not sort.")
Key takeaways
- Once requests queue, their order is a policy decision; the right policy depends entirely on whether the device has moving parts.
- Merging adjacent requests and plugging an empty queue briefly are worthwhile on every device, because per-request overhead is real even without positioning cost.
- Sorting by block number pays on disks (up to several times less arm motion) and actively hurts on NVMe, where it adds CPU and latency for no benefit.
- blk-mq gives each CPU its own software queue feeding multiple hardware queues, removing the shared-lock bottleneck of the old single-queue design.
- Reads have a thread waiting and writes usually do not, so deadline schedulers give reads far tighter deadlines — reversing this ruins interactivity.
- Use
nonefor NVMe,mq-deadlinefor SATA and HDD servers,bfqfor interactive desktops, and cgroupio.max/io.latencyto contain noisy neighbours.
🧪 Practice
- Given a head at cylinder 40 and the queue {10, 180, 45, 120, 22}, compute the total arm movement under FCFS and under C-LOOK.
- Check the active scheduler for every block device on your machine and say whether each choice matches the device type.
- Interview-style: "Setting the scheduler to
noneon an NVMe drive improved p99 latency. Why would removing scheduling make things better?" Hint: consider what the scheduler was spending time doing, and whether the device's own controller already does it more effectively.
Error Handling
Theory
I/O is where things go wrong. Memory rarely fails silently in a way software must handle; devices do it constantly — cables come loose, sectors go bad, networks partition, drives overheat, firmware has bugs. The kernel's job is to distinguish what it can fix from what it cannot, and to report the latter honestly rather than pretending.
The first distinction is transient versus permanent. A transient error may succeed if simply tried again: a bus timeout under load, a read that failed once because of vibration, a network packet that was dropped. A permanent error will never succeed: a sector whose magnetic domain is dead, a removed device, a write to read-only media. Retrying a permanent error wastes time and, on a failing disk, actively accelerates its death. Retrying a transient error is usually free. Getting the classification wrong in either direction is the main failure mode of error-handling code.
Layers handle errors at different levels, and the higher you go, the more context there is:
- The device corrects internally. Every disk sector carries ECC; a single-bit flip is fixed and never reported. SSDs relocate wearing blocks silently.
- The driver retries a small number of times and may reset the device.
- The block layer or protocol stack retries, fails over to another path (multipath I/O), or reconstructs from redundancy (RAID).
- The file system may remount read-only, replay its journal, or repair from a checksum-verified copy.
- The application gets an error code and must decide — and this is where most software is weakest.
The errors a program actually sees are a small vocabulary worth knowing:
| Code | Meaning | Usual response |
|---|---|---|
EIO | Low-level I/O failure | Often permanent; do not blindly retry |
EAGAIN | Would block | Retry when ready (not an error) |
EINTR | Interrupted by a signal | Retry the call |
ENOSPC | No space | Free space; retrying without doing so is pointless |
EDQUOT | Quota exceeded | Same, but per-user |
ENODEV / ENXIO | Device gone | Fail; do not retry |
EROFS | Read-only file system | Often the aftermath of an earlier error |
ETIMEDOUT | Operation timed out | Usually transient |
Three application-level mistakes recur so often they are worth naming.
Unchecked writes. write() returns the number of bytes written, which may be less than requested, and close() can fail — returning the error from a deferred writeback. A program that ignores both can lose data with no indication. The correct pattern is to loop on partial writes and check close() as well as fsync().
Retrying without backoff. Immediate retries on a struggling device or network add load precisely when the system is least able to take it, converting a transient problem into a sustained outage. Exponential backoff with jitter is the standard remedy; the jitter matters because synchronised retries from many clients produce a thundering herd.
Silent corruption. The hardest class, because nothing reports an error at all. A drive returns wrong bytes, a cable flips a bit, firmware misdirects a write. Only end-to-end checksums (Section 10.4) detect this — which is the strongest argument for ZFS, Btrfs, or application-level verification.
Finally, errors must be visible. A system that retries silently forever looks healthy until it collapses. SMART attributes, dmesg, kernel error counters, and per-device I/O error metrics should be collected and alerted on, because rising reallocated-sector counts and occasional bus resets are warnings that arrive well before the failure does.
Example
WHERE AN ERROR IS HANDLED, AND WHAT EACH LAYER CAN DO
application "EIO on read of report.pdf" <- most context, least info
^ about the cause
file system remount read-only; repair from checksum+redundancy
^
block/RAID retry; failover to another path; reconstruct from parity
^
driver retry N times; reset the device
^
device ECC corrects the bit; remap the sector; report if it cannot
<- least context, most information about the cause
Most errors never reach the top. The ones that do have already
defeated every layer below.
TRANSIENT vs PERMANENT: THE CLASSIFICATION THAT MATTERS
transient (retry helps) permanent (retry harms)
bus timeout under load medium error on a dead sector
dropped network packet device unplugged
temporary queue-full write to read-only media
ETIMEDOUT, EAGAIN, EINTR ENODEV, EROFS, repeated EIO
Retrying a permanent error on a failing disk makes it fail FASTER
and delays the failover that would have saved the service.
EXPONENTIAL BACKOFF WITH JITTER
no backoff : ||||||||||||||||||||||| retries pile onto a sick device
backoff : | | | | load falls as failures persist
+ jitter : | | | | clients de-synchronise
attempt: 1 2 3 4 5
delay : 0.1s 0.2s 0.4s 0.8s 1.6s (x random 0.5-1.5)
THE ERROR NOBODY REPORTS
write(fd, buf, 4096) -> 4096 "success"
close(fd) -> -1 EIO <- the writeback failed; the data is GONE
Programs that ignore close() lose data with NO indication whatsoever.
import os, time, random, errno
# ---------- 1. A correct write: partial writes and close() both checked ----------
def write_all(fd, data: bytes):
"""write() may transfer FEWER bytes than requested. Looping is mandatory."""
view, total = memoryview(data), 0
while total < len(data):
try:
n = os.write(fd, view[total:])
except InterruptedError: # EINTR: a signal arrived
continue # not an error; just retry
except BlockingIOError: # EAGAIN on a non-blocking fd
time.sleep(0.001); continue
if n == 0:
raise IOError("write returned 0")
total += n
return total
def durable_write(path, data: bytes):
fd = os.open(path, os.O_CREAT | os.O_WRONLY | os.O_TRUNC, 0o644)
try:
write_all(fd, data)
os.fsync(fd) # errors surface HERE...
finally:
os.close(fd) # ...or HERE. Both must be checked;
# a bare close() in a finally can
# swallow the error, so in real code
# close explicitly and check it.
# ---------- 2. Retry: only for transient errors, always with backoff ----------
TRANSIENT = {errno.EAGAIN, errno.EINTR, errno.ETIMEDOUT, errno.EBUSY, errno.ENOBUFS}
PERMANENT = {errno.ENODEV, errno.ENXIO, errno.EROFS, errno.ENOSPC, errno.EACCES}
def with_retry(op, attempts=5, base=0.1, cap=10.0):
for attempt in range(attempts):
try:
return op()
except OSError as e:
if e.errno in PERMANENT:
raise # retrying CANNOT help
if e.errno not in TRANSIENT and e.errno != errno.EIO:
raise
if attempt == attempts - 1:
raise
# Exponential backoff with full jitter: de-synchronises clients.
delay = min(cap, base * (2 ** attempt)) * random.uniform(0.5, 1.5)
print(f" attempt {attempt+1} failed ({e.strerror}); "
f"retrying in {delay:.2f}s")
time.sleep(delay)
# ---------- 3. Detecting what no layer reports: silent corruption ----------
import hashlib
def write_verified(path, data: bytes):
durable_write(path, data)
digest = hashlib.blake2b(data, digest_size=16).hexdigest()
durable_write(path + ".sum", digest.encode())
return digest
def read_verified(path):
with open(path, "rb") as f:
data = f.read()
with open(path + ".sum") as f:
expected = f.read().strip()
actual = hashlib.blake2b(data, digest_size=16).hexdigest()
if actual != expected:
# No errno was ever returned. Only the checksum knows.
raise IOError(f"SILENT CORRUPTION in {path}: {expected} != {actual}")
return data
# The kernel's own record of I/O trouble -- check this first, always.
dmesg -T | grep -iE 'i/o error|medium error|reset|ata[0-9]|nvme.*error|failed command'
# [Thu Sep 11 09:14:02] blk_update_request: I/O error, dev sda, sector 1234567
# [Thu Sep 11 09:14:02] ata1.00: failed command: READ FPDMA QUEUED
journalctl -k -p err -b
# Per-device error counters.
cat /sys/block/sda/device/ioerr_cnt
sudo nvme error-log /dev/nvme0 | head -20
sudo nvme smart-log /dev/nvme0 | grep -iE 'critical|media_errors|num_err'
# The pre-failure warning nobody reads until it is too late.
sudo smartctl -a /dev/sda | grep -iE 'Reallocated|Pending|Uncorrectable|Offline'
# 5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail - 128
# 197 Current_Pending_Sector 0x0012 100 100 000 Old_age - 8
# ^ pending sectors are failing reads RIGHT NOW
sudo smartctl -t short /dev/sda && sleep 120 && sudo smartctl -l selftest /dev/sda
sudo systemctl enable --now smartd # continuous monitoring + alerts
# Multipath: the failover that makes a path error survivable.
multipath -ll
# mpatha (3600508b4000...) dm-0
# |-+- policy='service-time 0' prio=50 status=active
# | `- 2:0:0:1 sdb 8:16 active ready running
# `-+- policy='service-time 0' prio=10 status=enabled
# `- 3:0:0:1 sdc 8:32 active ghost running <- standby path
# Timeouts: how long before the kernel gives up on a request.
cat /sys/block/sda/device/timeout # 30 seconds by default
echo 10 | sudo tee /sys/block/sda/device/timeout # fail over sooner
# Testing error paths deliberately, rather than discovering them in production.
sudo dmsetup create flaky --table "0 $(blockdev --getsz /dev/sdb) \
flakey /dev/sdb 0 8 4" # 8s up, 4s returning errors
dd if=/dev/zero of=/dev/full bs=1M count=1 # a device that always gives ENOSPC
Key takeaways
- The essential classification is transient versus permanent: retrying the first is nearly free, retrying the second wastes time and can accelerate a failing device's death.
- Errors are handled at every layer — device ECC, driver retries, RAID or multipath recovery, file system repair — and only what defeats all of them reaches the application.
write()can transfer fewer bytes than requested andclose()can report a deferred writeback failure; ignoring either loses data silently.- Retries need exponential backoff with jitter, or synchronised clients turn a transient fault into a sustained outage.
- Silent corruption produces no error code at all and is detectable only with end-to-end checksums.
- Error visibility matters as much as error handling: SMART attributes, kernel logs, and per-device counters warn well before a failure becomes an outage.
🧪 Practice
- Classify each of
EAGAIN,EIO,ENOSPC,EINTR, andENODEVas transient or permanent, and state the correct response to each.- Write a retry wrapper with exponential backoff and jitter, and explain what breaks in a fleet of 1,000 clients if the jitter is omitted.
- Interview-style: "A backup script reports success but the restored archive is truncated. Where would you look in the code?" Hint: two system calls in the write path return values that programs habitually discard, and one of them reports failures that happened long after the data was handed over.
I/O Protection
Theory
Every device is a capability. A program that can write raw blocks to /dev/sda can rewrite the file system, including the password file, regardless of file permissions. A program that can put a network interface in promiscuous mode can read other people's traffic. A program that can program a DMA-capable device can write anywhere in physical memory. So I/O is a protection problem before it is a performance problem, and the defences are layered.
Layer 1: privileged instructions. The I/O instructions themselves (IN, OUT on x86) are available only in kernel mode, and MMIO regions are simply not mapped into user address spaces. A user program cannot touch a device register, so all device access must go through system calls — which is the whole reason dual-mode operation exists.
Layer 2: device file permissions. Once access is mediated by the file system, ordinary permissions apply. /dev/sda is brw-rw---- root disk, so only root and members of disk can open it — and membership of disk is effectively equivalent to root, a fact that surprises people. /dev/null is world-writable because it is harmless; /dev/mem is root-only and usually disabled entirely (CONFIG_STRICT_DEVMEM) because it would otherwise expose all of physical memory.
Layer 3: capabilities. Coarse root-or-not is too blunt. Linux splits root's powers into capabilities so a program can be given exactly one: CAP_NET_RAW for raw sockets (what ping needs), CAP_NET_ADMIN for interface configuration, CAP_SYS_RAWIO for raw device access, CAP_SYS_ADMIN for a regrettably large grab-bag. Granting CAP_NET_RAW to a binary is vastly safer than making it setuid-root.
Layer 4: mount options. Section 10.1's nodev, nosuid, and noexec prevent a device node on removable or user-writable media from being a path to privilege. Without nodev, a user who can mount a USB stick containing a /dev/sda node owns the machine.
Layer 5: the IOMMU. All of the above are useless against a device that does DMA to arbitrary physical addresses. The IOMMU confines each device to explicitly mapped pages, which is what makes Thunderbolt hot-plug and VM device passthrough safe.
Layer 6: sandboxing. seccomp filters which system calls a process may make at all — a process that cannot call ioctl cannot misuse a device even if it holds a descriptor. Namespaces give containers their own /dev with only the entries they need. SELinux and AppArmor add mandatory policy on top of discretionary permissions.
Two mechanisms deserve special mention because they sit at the boundary. ioctl is a hole in the type system: it takes an arbitrary command number and an arbitrary pointer, so every driver's ioctl handler is attack surface, and historically a rich source of privilege-escalation bugs. Drivers must validate command numbers, validate sizes, and copy user data properly. And device passthrough to a VM hands a guest direct control of real hardware; it is safe only with an IOMMU, and only at the granularity of an IOMMU group, since devices sharing a group can reach each other.
Example
THE LAYERS, AND WHAT EACH STOPS
user process
| (1) cannot execute IN/OUT; device registers are not mapped
v
system call boundary
| (2) /dev node permissions: who may open this device at all
| (3) capabilities: CAP_NET_RAW, CAP_SYS_RAWIO -- root, subdivided
| (6) seccomp: may this process even CALL ioctl?
v
kernel driver
| (4) mount options: nodev stops device nodes on foreign media
v
device
| (5) IOMMU: the device may touch ONLY its mapped pages
v
physical memory
WHY /dev/sda PERMISSIONS ARE A PRIVILEGE BOUNDARY
$ ls -l /dev/sda
brw-rw---- 1 root disk 8, 0 /dev/sda
^^^^ membership of 'disk' == the ability to rewrite
/etc/shadow directly, bypassing every file
permission on the system. It IS root.
THE nodev ESCALATION, IN THREE STEPS
1. attacker creates a USB image containing: /dev/sda (b 8 0), mode 0666
2. mounts it WITHOUT nodev
3. writes to /media/usb/dev/sda -> raw disk write as an unprivileged user
Mount options exist to close exactly this.
ioctl: WHY IT IS THE CLASSIC ATTACK SURFACE
ioctl(fd, SOME_CMD, user_pointer)
^ ^ ^
| | +-- arbitrary, driver-interpreted, often a struct
| +------------- arbitrary number the driver switches on
+-------------------- any device the process can open
The kernel cannot type-check this. Every driver must validate it itself.
CAPABILITIES vs setuid-root
setuid root binary : holds ALL of root's powers for its whole lifetime
CAP_NET_RAW only : may open raw sockets; may NOT read /etc/shadow,
load modules, or write raw devices
# Layer 2: device permissions, and what they actually grant.
ls -l /dev/sda /dev/null /dev/mem /dev/kmsg
# brw-rw---- root disk /dev/sda <- 'disk' group == root, effectively
# crw-rw-rw- root root /dev/null <- harmless, so world-writable
# crw-r----- root kmem /dev/mem <- all physical memory; usually disabled
grep CONFIG_STRICT_DEVMEM /boot/config-$(uname -r) # =y means /dev/mem is limited
getent group disk kmem video # audit who is in these
# Layer 3: capabilities instead of setuid.
getcap /usr/bin/ping
# /usr/bin/ping cap_net_raw=ep <- NOT setuid root any more
sudo setcap cap_net_bind_service=+ep /usr/local/bin/myserver # bind port 80 only
getpcaps $$ # this shell's capabilities
capsh --print | head -5
find / -perm -4000 -type f 2>/dev/null # audit remaining setuid binaries
# Layer 4: mount options on anything a user can supply.
mount | grep -E 'nodev|nosuid|noexec'
# /etc/fstab for removable media:
# /dev/sdb1 /media/usb auto rw,nosuid,nodev,noexec,noauto,user 0 0
findmnt -o TARGET,OPTIONS /tmp /home /media
# Layer 5: the IOMMU.
dmesg | grep -iE 'DMAR: IOMMU enabled|AMD-Vi: .*initialized'
ls /sys/kernel/iommu_groups/ | wc -l
# Passthrough is only as isolated as the GROUP:
for g in /sys/kernel/iommu_groups/*/devices/*; do
echo "group ${g#*groups/} -> $(lspci -s $(basename $g) | cut -c1-60)"
done | sort | head
# Two devices in one group cannot be separated: pass through both, or neither.
# Layer 6: sandboxing.
systemd-analyze security sshd # per-unit exposure score
# Hardening directives in a unit file:
# PrivateDevices=yes -> a minimal private /dev
# DeviceAllow=/dev/null rw
# RestrictAddressFamilies=AF_INET AF_INET6
# SystemCallFilter=@system-service
# NoNewPrivileges=yes
# Containers get their own /dev by default.
docker run --rm alpine ls /dev # a short, safe list
docker run --rm --privileged alpine ls /dev | wc -l # everything: avoid
docker run --rm --device=/dev/snd alpine ls /dev/snd # grant ONE device
# udev decides ownership and mode as devices appear.
cat <<'EOF' | sudo tee /etc/udev/rules.d/99-mydev.rules
SUBSYSTEM=="usb", ATTR{idVendor}=="1234", MODE="0660", GROUP="plugdev"
EOF
sudo udevadm control --reload && sudo udevadm trigger
// Validating an ioctl: what every driver must do, and what attackers rely on
// drivers forgetting.
static long my_ioctl(struct file *f, unsigned int cmd, unsigned long arg)
{
struct my_request req;
// (1) Validate the command's magic number: reject anything not ours.
if (_IOC_TYPE(cmd) != MY_IOC_MAGIC)
return -ENOTTY;
if (_IOC_NR(cmd) > MY_IOC_MAXNR)
return -ENOTTY;
// (2) Validate the size encoded in the command against our struct.
if (_IOC_SIZE(cmd) != sizeof(req))
return -EINVAL;
// (3) Check privilege PER OPERATION, not once at open().
// An fd can be passed to another process over a UNIX socket.
if (cmd == MY_IOC_RAW_ACCESS && !capable(CAP_SYS_RAWIO))
return -EPERM;
// (4) Copy in safely -- never dereference the user pointer directly.
if (copy_from_user(&req, (void __user *)arg, sizeof(req)))
return -EFAULT;
// (5) Validate EVERY field AFTER copying. Checking before the copy is
// a time-of-check-to-time-of-use bug: user space can change it.
if (req.offset > DEVICE_SIZE || req.length > DEVICE_SIZE - req.offset)
return -EINVAL; // this check stops the classic
// integer-overflow out-of-bounds
return do_operation(&req);
}
Key takeaways
- Device access is a privilege boundary: raw block access bypasses every file permission, and DMA-capable devices can reach all of physical memory.
- Privileged I/O instructions and unmapped MMIO regions mean user code can only reach devices through system calls.
- Device node permissions matter enormously — membership of the
diskgroup is effectively root — andnodevstops device nodes on user-supplied media becoming an escalation path. - Capabilities subdivide root's powers so a program can hold exactly
CAP_NET_RAWorCAP_SYS_RAWIOrather than everything. - The IOMMU is the only defence against a malicious or buggy device doing arbitrary DMA, and it is what makes passthrough and hot-plug safe.
ioctlis untyped by construction, so every driver must validate the command number, the size, the caller's privilege, and every field after copying it in.
🧪 Practice
- Explain why adding a user to the
diskgroup is equivalent to giving them root, with a concrete sequence of commands they could run.- Compare a setuid-root binary with one holding only
CAP_NET_RAW, listing two things the former can do that the latter cannot.- Interview-style: "Why is device passthrough to a VM unsafe without an IOMMU?" Hint: think about what address space the device uses for DMA and who controls the values the guest programs into it.
Device Reservation
Theory
Most devices multiplex happily: the kernel interleaves requests from many processes and each sees a coherent view. Some cannot. A tape drive has one physical position, so two processes writing alternately produce a useless tape. A CD burner in the middle of a session cannot serve anybody else. A serial port tied to an instrument needs one owner for the duration of the conversation. For these, the kernel or the application must arrange exclusive access.
The simplest mechanism is exclusive open. open(O_EXCL) on a block device fails if the device is already open by anybody, which is why you cannot mkfs a mounted file system. Windows has the same idea in CreateFile's share mode. This is enforced by the kernel and cannot be ignored, unlike the advisory file locks of Section 10.1.
Advisory mechanisms cover the rest. Lock files in /var/lock are the traditional UNIX convention for serial ports and similar devices, with the format LCK..ttyS0 containing the owner's PID. They only work if every program participates — which, by convention, they mostly do. flock on the device node is the modern equivalent and is automatically released when the holder dies, which avoids the stale-lock problem that plagues PID files.
Once processes wait for exclusive devices, deadlock becomes possible, and this is Chapter 6 applied to hardware. A process holding the tape drive and requesting the scanner, while another holds the scanner and requests the tape drive, is the textbook circular wait — with the added difficulty that preemption is often impossible (you cannot take a tape drive away mid-write) and rollback is meaningless (you cannot un-burn a CD). The practical defences are the familiar ones: impose a global ordering on device acquisition, acquire everything at once or nothing, and always use timeouts so a wedged holder does not block the system forever.
Three practical issues complete the picture.
Stale locks. A process holding an exclusive device that dies without cleaning up leaves the device unusable. Kernel-enforced mechanisms (O_EXCL, flock) release automatically on process exit; PID files do not, so any PID-file scheme needs a liveness check — and a PID can be reused, so the check must be more than "does this PID exist".
Timeouts. Every acquisition should be bounded. A batch job that blocks forever on a device somebody forgot to release is an outage, not a wait.
Virtualisation. In containers and VMs the question becomes which guest gets the device. Passthrough (VFIO) gives one guest exclusive control at IOMMU-group granularity; SR-IOV lets a capable device present multiple virtual functions so several guests each get a real, isolated slice; and mediated devices (vGPU) let a driver time-slice one physical device among several guests. These are the modern forms of the same reservation problem.
Example
WHY SOME DEVICES CANNOT BE INTERLEAVED
SHAREABLE (disk) NON-SHAREABLE (tape)
P1: write block 100 P1: write record -> tape advances
P2: write block 500 P2: write record -> tape advances
P1: write block 101 P1: write record -> tape advances
the file system keeps them one physical position; the records
separate; both are correct interleave and BOTH streams are ruined
THE MECHANISMS, FROM STRONGEST TO WEAKEST
O_EXCL on a block device : KERNEL-ENFORCED. Cannot be bypassed.
Released automatically on process exit.
flock on the device node : advisory, but kernel-tracked and
auto-released on exit. The sane default.
/var/lock/LCK..ttyS0 : pure convention. Works only if everyone
participates. Goes STALE if the holder dies.
DEADLOCK WITH DEVICES (Chapter 6, in hardware)
P1: holds TAPE waits for SCANNER
P2: holds SCANNER waits for TAPE
+--------+ +----------+
| P1 |--- wants ----->| SCANNER |
+--------+ +----------+
^ |
holds holds
| v
+----------+ +--------+
| TAPE |<-- wants ---| P2 |
+----------+ +--------+
Worse than software deadlock: you cannot PREEMPT a tape drive
mid-write, and you cannot ROLL BACK a burned disc.
Defences: global acquisition ORDER, all-or-nothing acquisition, TIMEOUTS.
THE STALE-LOCK PROBLEM
PID file : holder is SIGKILLed -> file remains -> device unusable
forever, and the PID may be REUSED by an unrelated process
flock/O_EXCL: holder dies -> kernel releases the lock -> next waiter
proceeds. This is why kernel-tracked locks are preferred.
VIRTUALISATION: THE SAME QUESTION, NEW FORMS
passthrough (VFIO) : one guest owns the device; granularity is the
entire IOMMU GROUP, not one function
SR-IOV : the device presents N virtual functions; each
guest gets a real, isolated slice
mediated (vGPU) : a host driver time-slices one device among guests
import os, fcntl, errno, time, contextlib
# ---------- 1. Kernel-enforced exclusivity on a block device ----------
def open_block_exclusive(path):
"""O_EXCL on a block device fails if ANYONE has it open -- including
the kernel, if it is mounted. This is why mkfs refuses."""
try:
return os.open(path, os.O_RDWR | os.O_EXCL)
except OSError as e:
if e.errno == errno.EBUSY:
raise RuntimeError(f"{path} is mounted or already held") from e
raise
# ---------- 2. flock: advisory, but auto-released on process death ----------
@contextlib.contextmanager
def device_lock(path, timeout=10.0):
f = open(path, "r+b")
deadline = time.monotonic() + timeout
while True:
try:
fcntl.flock(f, fcntl.LOCK_EX | fcntl.LOCK_NB)
break
except BlockingIOError:
if time.monotonic() > deadline: # ALWAYS bound the wait
f.close()
raise TimeoutError(f"{path} held by another process")
time.sleep(0.05)
try:
yield f
finally:
fcntl.flock(f, fcntl.LOCK_UN)
f.close()
# If this process is SIGKILLed mid-block, the kernel releases the lock.
# A PID file would have been left behind, stale, forever.
# ---------- 3. Multiple devices: a global order prevents deadlock ----------
DEVICE_ORDER = {"/dev/st0": 1, "/dev/sg1": 2, "/dev/ttyUSB0": 3}
@contextlib.contextmanager
def acquire_all(paths, timeout=10.0):
"""Acquire in a FIXED GLOBAL ORDER, so a cycle cannot form."""
ordered = sorted(paths, key=lambda p: DEVICE_ORDER.get(p, 999))
held = []
try:
for p in ordered:
cm = device_lock(p, timeout)
held.append((p, cm, cm.__enter__()))
yield [h[2] for h in held]
finally:
for _, cm, _ in reversed(held): # release in reverse order
with contextlib.suppress(Exception):
cm.__exit__(None, None, None)
# P1 asking for {tape, scanner} and P2 asking for {scanner, tape} now BOTH
# acquire tape first. One waits; neither deadlocks.
# ---------- 4. Why PID files need a liveness check, and why it is not enough ----
def stale_lock_check(lockfile):
try:
with open(lockfile) as f:
pid = int(f.read().strip())
except (FileNotFoundError, ValueError):
return "no lock"
try:
os.kill(pid, 0) # signal 0: existence test only
except ProcessLookupError:
return "STALE: holder is gone (safe to remove)"
except PermissionError:
return "held by another user's process"
return "held (but the PID may have been REUSED by an unrelated process)"
# Kernel-enforced exclusivity, demonstrated.
sudo mount /dev/sdb1 /mnt/x
sudo mkfs.ext4 /dev/sdb1
# /dev/sdb1 is mounted; will not make a filesystem here!
# (mkfs opens with O_EXCL, and the mount holds it)
sudo umount /mnt/sdb1 && sudo mkfs.ext4 /dev/sdb1 # now it succeeds
# Who is holding a device open.
sudo fuser -v /dev/sdb1
sudo lsof /dev/ttyUSB0
# COMMAND PID USER FD TYPE DEVICE NAME
# minicom 4821 ana 3u CHR 188,0 /dev/ttyUSB0
# The traditional serial-port convention, still in use.
ls -l /var/lock/
# -rw-r--r-- 1 ana dialout 11 Sep 11 09:14 LCK..ttyUSB0
cat /var/lock/LCK..ttyUSB0 # the holder's PID, in ASCII
# Stale after a crash: nothing removes it. This is the scheme's weakness.
# The modern equivalent: kernel-tracked, self-releasing.
flock -x /dev/ttyUSB0 -c 'stty -F /dev/ttyUSB0 115200; cat /dev/ttyUSB0'
flock -n /dev/st0 -c 'tar cf /dev/st0 /data' || echo "tape drive busy"
cat /proc/locks | grep FLOCK
# Tape drives: the canonical non-shareable device.
mt -f /dev/st0 status
mt -f /dev/st0 rewind
tar cf /dev/nst0 /data # 'n' = no-rewind: position is STATE
# Virtualisation: reserving a device for one guest.
lspci -nn | grep -i nvidia
echo 0000:01:00.0 | sudo tee /sys/bus/pci/devices/0000:01:00.0/driver/unbind
echo 10de 1b80 | sudo tee /sys/bus/pci/drivers/vfio-pci/new_id
ls /sys/kernel/iommu_groups/13/devices/ # ALL of these move together
# SR-IOV: one card, several isolated slices.
cat /sys/class/net/eth0/device/sriov_totalvfs
echo 4 | sudo tee /sys/class/net/eth0/device/sriov_numvfs
lspci | grep 'Virtual Function'
Key takeaways
- Some devices cannot be interleaved because they carry physical state — tape position, burn session, instrument conversation — so they need exclusive access.
O_EXCLon a block device is kernel-enforced and cannot be bypassed, which is whymkfsrefuses to format a mounted volume.flockon a device node is advisory but kernel-tracked and released automatically on process death, making it far safer than a PID lock file.- Traditional
/var/lock/LCK..*files work only by convention and go stale when the holder is killed; any PID scheme needs a liveness check, and even that is imperfect because PIDs are reused. - Waiting for exclusive devices reintroduces deadlock, made worse because devices usually cannot be preempted or rolled back — so use a global acquisition order and always time out.
- In virtualised environments the same problem appears as passthrough (whole IOMMU group), SR-IOV (hardware-isolated slices), or mediated devices (time-sliced).
🧪 Practice
- Explain why
mkfsrefuses to run on a mounted device and name the exact mechanism that prevents it.- Two programs need both a tape drive and a scanner. Show the acquisition order that makes deadlock impossible, and explain why it works.
- Interview-style: "A serial-port application leaves
/var/lock/LCK..ttyUSB0behind after a crash, and the device stays unusable. Fix it properly." Hint: the problem is that nothing is tracking liveness; ask which locking mechanism the kernel cleans up on its own when a process dies.
STREAMS
Theory
STREAMS is a framework from UNIX System V for building character-device and network I/O paths out of stackable, reusable modules. Its practical importance today is limited — Linux never adopted it and Solaris was its main home — but it is worth understanding because the idea it embodies is everywhere, and because it makes a clean case study in a design trade-off.
The problem it addressed: a terminal driver, a network protocol stack, and a pseudo-terminal all need similar processing steps — line editing, character translation, flow control, protocol framing — and in a monolithic driver each one reimplements them. STREAMS proposed making each step a module with a standard interface, so a path could be assembled by pushing modules onto a stack at runtime.
A stream has three parts:
- The stream head, at the top, interfacing with user system calls.
- Zero or more modules in the middle, each with a read queue and a write queue, doing one transformation.
- The driver end, at the bottom, talking to the device.
Modules communicate by passing messages up and down the queues, never by calling each other directly, and each queue has high and low water marks for flow control. Because the interface is uniform, ioctl(fd, I_PUSH, "ldterm") can insert a line-discipline module onto a live stream, and I_POP removes it. The composability is genuine: a TCP connection over a tunnel over a compression module is just three modules stacked.
The costs are what kept it from dominating. Every message crosses every module, so each layer adds allocation, queueing, and scheduling overhead on a path that is extremely hot. The message-based flow-control model is subtle and hard to reason about. And the flexibility is rarely needed at runtime — most stacks are known at compile time, so paying a runtime-composition cost forever to enable reconfiguration that almost nobody performs is a poor trade. Linux chose specialised, tightly-coupled paths (the tty layer with its line disciplines, netfilter hooks in the network stack) and got better performance with less generality.
The idea, however, won everywhere that performance is not the binding constraint. Linux line disciplines are STREAMS modules in all but name — N_TTY for terminals, N_PPP, N_SLIP — pushed onto a tty to change how bytes are processed. Netfilter hooks let modules inspect and transform packets at defined points. eBPF attaches programs at hook points throughout the kernel. Outside the kernel, the same shape appears in Unix pipelines, servlet filters, HTTP middleware chains, audio and video plugin graphs, and every service mesh sidecar.
The transferable lesson: stackable modules with a uniform message interface buy composability and cost per-layer overhead. That trade is right when paths change or vary and wrong on a path where every nanosecond is counted.
Example
A STREAM: HEAD, MODULES, DRIVER
user process
| read() / write() / ioctl()
+----v--------------------+
| STREAM HEAD | the system-call interface
+----+--------------------+
| messages down ^ messages up
+----v-----------------+--+
| MODULE: ldterm | line discipline: echo, erase, line buffering
| [write q] [read q] |
+----+-----------------+--+
| ^
+----v-----------------+--+
| MODULE: compress | pushed at RUNTIME with I_PUSH
| [write q] [read q] |
+----+-----------------+--+
| ^
+----v-----------------+--+
| DRIVER END | the actual hardware
+----+--------------------+
v
serial port
Each module has TWO queues and does ONE transformation.
Modules never call each other: they pass MESSAGES.
RUNTIME COMPOSITION -- the feature that justified the design
ioctl(fd, I_PUSH, "ldterm"); insert line editing
ioctl(fd, I_PUSH, "compress"); insert compression underneath it
ioctl(fd, I_POP, 0); remove the topmost module
ioctl(fd, I_LIST, &list); what is currently stacked
THE COST
monolithic tty driver : one function call chain, one buffer
STREAMS : message alloc + enqueue + scheduling PER MODULE
on a path handling millions of bytes per second
Generality is not free, and this path is extremely hot.
THE SAME IDEA, WHERE IT ACTUALLY WON
Linux line disciplines : N_TTY, N_PPP, N_SLIP pushed onto a tty
netfilter : PREROUTING -> FORWARD -> POSTROUTING hooks
eBPF : programs attached at defined kernel hook points
shell pipelines : cmd | filter | filter | sink
HTTP middleware : auth -> log -> compress -> handler
audio/video graphs : GStreamer, JACK, Web Audio nodes
// STREAMS as it was actually used (System V / Solaris).
#include <stropts.h>
int build_stream(const char *path)
{
int fd = open(path, O_RDWR);
struct str_list list;
char names[8][FMNAMESZ + 1];
int i;
// Push modules onto the stream. Order matters: the LAST pushed is
// the TOPMOST, i.e. closest to the user process.
ioctl(fd, I_PUSH, "ldterm"); // line discipline: erase, kill, echo
ioctl(fd, I_PUSH, "ttcompat"); // BSD tty compatibility translation
// Inspect the current stack.
list.sl_nmods = 8;
list.sl_modlist = (struct str_mlist *)names;
if (ioctl(fd, I_LIST, &list) >= 0)
for (i = 0; i < list.sl_nmods; i++)
printf("module %d: %s\n", i, names[i]); // ttcompat, ldterm, driver
ioctl(fd, I_POP, 0); // remove ttcompat, leaving ldterm
// Messages, not byte streams: control and data are separate bands.
struct strbuf ctl = { .len = sizeof(cmd), .buf = (char *)&cmd };
struct strbuf data = { .len = payload_len, .buf = payload };
putmsg(fd, &ctl, &data, 0); // send a control + data message down
int flags = 0;
getmsg(fd, &ctl, &data, &flags); // receive one from above
return fd;
}
# Linux's line disciplines: the surviving form of the same idea.
cat /proc/tty/ldiscs
# n_tty 0 <- the default: cooked mode, line editing, signals
# n_ppp 3
# n_hdlc 13
cat /proc/tty/drivers
# Pushing a different discipline onto a live tty -- STREAMS' I_PUSH, in effect.
sudo ldattach PPP /dev/ttyS0 # attach the PPP discipline
sudo ldattach -d GPS /dev/ttyUSB0 # NMEA GPS discipline
# The line discipline's transformations, switched off and on.
stty -a -F /dev/pts/0
# speed 38400 baud; line = 0;
# -icanon -echo <- raw: no line editing, no echo (what vim sets)
# icanon echo <- cooked: the discipline does erase/kill/echo for you
stty raw -echo; stty sane # toggle, then restore
# Netfilter: modules at defined hook points in the packet path.
sudo nft list ruleset | head -20
sudo iptables -L -v -n --line-numbers
# PREROUTING -> [modules] -> FORWARD -> [modules] -> POSTROUTING
# eBPF: the modern, verified, JIT-compiled version of "push a module".
sudo bpftool prog list | head
sudo bpftool net list # programs attached at network hooks
sudo tc filter add dev eth0 ingress bpf obj filter.o sec ingress
# The userspace expression of exactly the same pattern.
cat access.log | grep ' 500 ' | awk '{print $1}' | sort | uniq -c | sort -rn
# ^ source ^ filter ^ transform ^ aggregate ^ sink
# Each stage does one thing, communicates only through the stream between
# them, and can be reordered or replaced independently.
Key takeaways
- STREAMS builds an I/O path from stackable modules, each with a read and a write queue, communicating only by passing messages.
- A stream is a head (system-call interface), zero or more modules, and a driver end; modules are pushed and popped at runtime with
I_PUSHandI_POP. - The benefit is genuine composability and reuse — one line-discipline module serves every terminal-like device.
- The cost is per-module message allocation, queueing, and scheduling on a very hot path, which is why Linux chose specialised paths instead.
- The idea survives as line disciplines, netfilter hooks, and eBPF attachment points in the kernel, and as pipelines, middleware chains, and media graphs outside it.
- The transferable lesson is the trade itself: uniform stackable interfaces buy flexibility and cost per-layer overhead, which is right when paths vary and wrong when nanoseconds are counted.
🧪 Practice
- Name the three parts of a stream and state what each is responsible for.
- Run
cat /proc/tty/ldiscsandstty -a, then explain which transformations the current line discipline is applying to your terminal.- Interview-style: "Why did Linux reject STREAMS but adopt netfilter hooks, which look like the same idea?" Hint: compare how often each path is traversed, and how much processing happens per traversal, then ask where a fixed per-module overhead actually hurts.
I/O Performance Tuning
Theory
I/O is where most systems spend their waiting time, so tuning it has outsized effect — and it is also where guessing does the most damage, because the layers interact and a change that helps one workload devastates another. The discipline is therefore: measure, identify the actual bottleneck, change one thing, measure again.
The first question is always which resource is saturated. There are only a few candidates, and each has a distinctive signature:
| Bottleneck | Signature | Where to look |
|---|---|---|
| Device throughput | %util near 100, queue depth high, await rising | iostat -x |
| Device latency | await high, %util moderate, queue shallow | iostat, biolatency |
| CPU in the I/O path | High %sys or %soft, low %util | mpstat, perf top |
| Interrupt handling | One core at 100% %soft, others idle | /proc/interrupts, mpstat -P ALL |
| Memory / cache | High page-cache miss rate, heavy swap | /proc/meminfo, vmstat |
| Application | Device idle, low CPU, poor throughput | strace -c, flame graphs |
Two derived numbers are worth internalising. Little's Law — concurrency = throughput × latency — tells you how many operations must be in flight to reach a target rate: 100,000 IOPS at 100 µs latency requires 10 operations outstanding at all times. If the application submits one at a time, the queue depth is 1 and no amount of device speed helps. This single relation explains most "the hardware is fast but the application is slow" situations.
And the latency curve is not linear. Utilisation of 90% is not "10% of headroom left"; queueing theory puts average wait at service_time × u/(1-u), so going from 80% to 90% utilisation doubles the wait, and 95% doubles it again. Systems should be sized so that normal load sits well below the knee.
The levers, roughly in order of typical payoff:
- Fix the application's access pattern. Sequential instead of random, batched instead of per-item, larger requests, and enough concurrency to fill the device. This usually dwarfs every kernel tunable.
- Match the request size. Too small wastes per-request overhead; too large adds latency. Align to the device's preferred size, and align partitions and file system blocks to physical sectors.
- Increase queue depth if the device is idle and the application is serialised — via threads, async I/O, or
io_uring. - Choose the right scheduler:
nonefor NVMe,mq-deadlineorbfqotherwise. - Tune caching and writeback:
dirty_ratio,dirty_background_ratio, read-ahead size, andvm.swappiness. - Spread interrupts across cores and enable coalescing where latency permits.
- Remove copies with
sendfile,splice, or registered buffers if the profile showsmemcpydominating. - Consider
O_DIRECTonly when the application has a better-informed cache of its own.
Three warnings. Benchmark the real workload — synthetic sequential reads predict nothing about a random-read database. Change one variable at a time, or the result is uninterpretable. And know your target metric: throughput and latency tuning pull in opposite directions, and the settings that maximise one (large batches, deep queues, aggressive coalescing) systematically harm the other.
Example
THE DIAGNOSTIC LADDER -- work top to bottom, do not skip
1. Is the DEVICE busy? iostat -x 1 -> %util
no -> the bottleneck is above it (app, CPU, locks, serialisation)
yes -> continue
2. Is it LATENCY or THROUGHPUT? r_await / w_await vs r/s + rMB/s
latency high, queue shallow -> per-request cost; check alignment/size
queue deep, bandwidth at spec -> genuinely saturated; add devices
3. Is CPU the limit? mpstat -P ALL 1
one core 100% %soft -> interrupt affinity / coalescing
high %sys everywhere -> copies, syscall rate; check strace -c
4. Is MEMORY the limit? free -h ; vmstat 1
cache too small, or swapping
LITTLE'S LAW: THE MOST USEFUL LINE OF ARITHMETIC IN I/O
concurrency = throughput x latency
target 100,000 IOPS, device latency 100 us
=> need 100,000 x 0.0001 = 10 requests IN FLIGHT at all times
an application doing synchronous one-at-a-time I/O has depth 1
=> achieves 1 / 0.0001 = 10,000 IOPS, and NO device upgrade helps
THE UTILISATION KNEE (wait = service x u/(1-u))
u | relative wait 100% | *
------+--------------- | *
0.50 | 1.0x | *
0.80 | 4.0x wait | *
0.90 | 9.0x | *
0.95 | 19.0x | *
0.99 | 99.0x | * *
0% +------------------------- utilisation
0% 100%
"90% utilised" is not "10% of headroom": it is already 9x the wait.
THROUGHPUT AND LATENCY PULL IN OPPOSITE DIRECTIONS
for THROUGHPUT for LATENCY
deep queues shallow queues
large requests small requests
interrupt coalescing coalescing off (rx-usecs 0)
aggressive read-ahead read-ahead off for random access
batched commits fsync per operation
Decide WHICH ONE you are optimising before changing anything.
# ---------- STEP 1: where is the time going? ----------
iostat -x 1 5
# Device r/s w/s rMB/s rareq-sz aqu-sz r_await w_await %util
# nvme0n1 8420 120 32.9 4.0 12.3 1.42 0.31 94.2
# ^IOPS ^BW ^avg size ^depth ^latency(ms) ^busy
# %util ~100 + deep queue + rising await = the device is the bottleneck.
# %util low + poor throughput = look UP the stack, not down.
mpstat -P ALL 1 5 # %soft pinned on one core = interrupt affinity issue
vmstat 1 5 # 'b' = blocked on I/O; 'wa' = iowait; 'si/so' = swap
free -h # is the page cache being squeezed?
# ---------- STEP 2: what is the latency distribution, not the average? ----------
sudo biolatency-bpfcc -m 10 1 # histogram: averages hide the p99 tail
sudo biosnoop-bpfcc | head -20 # per-I/O: process, sector, size, latency
sudo biotop-bpfcc # which process is generating the load
sudo funclatency-bpfcc vfs_read
# ---------- STEP 3: is the application the problem? ----------
strace -c -p $(pgrep -n myapp)
# % time calls syscall
# 62.1 482911 read <- 482,911 reads: what size were they?
strace -e trace=read -p $(pgrep -n myapp) 2>&1 | head -5
# read(7, "...", 512) = 512 <- 512-byte reads on a 4 KB device:
# 8x the necessary request count
sudo perf top -p $(pgrep -n myapp)
sudo perf record -g -p $(pgrep -n myapp) -- sleep 10 && sudo perf report
# ---------- STEP 4: the tunables, one at a time ----------
# scheduler
echo none | sudo tee /sys/block/nvme0n1/queue/scheduler
# read-ahead: raise for sequential, drop to 0 for random
sudo blockdev --setra 512 /dev/sda # 256 KB
sudo blockdev --setra 0 /dev/nvme0n1 # random workload: prefetch is waste
# writeback: keep the dirty pool small so flushes never stall the system
sudo sysctl -w vm.dirty_background_bytes=134217728 # 128 MB
sudo sysctl -w vm.dirty_bytes=536870912 # 512 MB
sudo sysctl -w vm.swappiness=10 # protect the page cache
# queue depth
cat /sys/block/nvme0n1/queue/nr_requests
echo 1024 | sudo tee /sys/block/nvme0n1/queue/nr_requests
# interrupts: spread them, and coalesce if latency permits
systemctl status irqbalance
sudo ethtool -C eth0 rx-usecs 50 rx-frames 64 # throughput
sudo ethtool -C eth0 rx-usecs 0 # latency
sudo ethtool -G eth0 rx 2048 # bigger NIC ring
# ---------- STEP 5: measure the REAL workload, not a synthetic one ----------
fio --name=realistic --filename=/dev/nvme0n1 --direct=1 \
--rw=randrw --rwmixread=70 --bs=8k --iodepth=32 --numjobs=4 \
--runtime=60 --time_based --group_reporting --percentile_list=50:95:99:99.9
# read: IOPS=284k, BW=2.2GiB/s
# clat percentiles (usec): 50.00th=[95], 99.00th=[412], 99.90th=[1729]
# ^ the number users feel
# Little's Law and the utilisation knee: the two calculations to do FIRST.
def required_concurrency(target_iops, latency_us):
return target_iops * (latency_us / 1e6)
def achievable_iops(queue_depth, latency_us):
return queue_depth / (latency_us / 1e6)
print("TARGET 100,000 IOPS on a device with 100 us latency")
print(f" requests that must be in flight: "
f"{required_concurrency(100_000, 100):.0f}\n")
print(f"{'queue depth':>12}{'achievable IOPS':>18}")
for qd in (1, 4, 8, 16, 32, 64):
print(f"{qd:>12}{achievable_iops(qd, 100):>18,.0f}")
print(" -> a synchronous, one-at-a-time application is capped at 10,000 IOPS")
print(" regardless of how fast the device is.\n")
# The queueing knee: why 90% utilisation is not '10% of headroom left'.
def wait_multiplier(u):
return u / (1 - u)
print(f"{'utilisation':>12}{'wait vs service time':>24}")
for u in (0.5, 0.7, 0.8, 0.9, 0.95, 0.99):
print(f"{u:>11.0%}{wait_multiplier(u):>22.1f}x")
# Request size: the trade between per-request overhead and per-request latency.
OVERHEAD_US, US_PER_KB = 5.0, 0.4
print(f"\n{'req size':>10}{'reqs for 1 MB':>15}{'total us':>11}{'per-req lat':>13}")
for kb in (4, 16, 64, 256, 1024):
reqs = 1024 // kb
per = OVERHEAD_US + US_PER_KB * kb
print(f"{kb:>8} KB{reqs:>15}{reqs*per:>11.0f}{per:>12.1f}us")
# Bigger requests: better throughput, worse per-request latency. Pick one.
Key takeaways
- Tuning starts with identifying which resource is saturated; each bottleneck has a distinctive signature across
iostat,mpstat,vmstat, and profiling tools. - Little's Law (concurrency = throughput × latency) shows why a synchronous application cannot reach a fast device's rated IOPS regardless of hardware.
- Wait time grows as
u/(1-u), so 90% utilisation already means nine times the service time in queueing — size systems well below the knee. - The application's access pattern and request size usually matter far more than any kernel tunable.
- Throughput and latency tuning pull in opposite directions: deep queues, large requests, and coalescing help the first and harm the second.
- Benchmark the real workload, change one variable at a time, and look at percentile latency rather than averages, because averages hide the tail users actually experience.
🧪 Practice
- A device has 80 µs average latency. Compute the queue depth needed to sustain 50,000 IOPS, and the IOPS achievable at queue depth 1.
- Run
iostat -x 1during a heavy copy and interpret%util,aqu-sz, andr_await. State which resource is the bottleneck and how you concluded it.- Interview-style: "A service's average I/O latency is 2 ms and users complain constantly about slowness. The dashboards look fine. What is happening?" Hint: ask which statistic a user actually experiences during a request, and what an average conceals about the shape of a distribution.
<a id="12-protection-and-security"></a>