13. Virtualization
Every chapter so far has described an operating system multiplexing hardware among processes; this one describes software multiplexing hardware among entire operating systems. Virtualization takes the interposition trick that makes a process think it owns the CPU and memory and applies it one level down, so a kernel can be given the same convincing illusion — and then examines the lighter-weight alternative, containers, which reuses the kernel's own isolation primitives instead of simulating hardware. It matters because essentially all server computing now runs on one of these two substrates, and the differences between them determine performance, density, and, critically, where the security boundary actually lies.
One distinction is worth fixing before anything else. A virtual machine duplicates the hardware interface, so the thing running inside it is a complete guest kernel with its own drivers, scheduler, and page tables. A container duplicates the system-call interface, so the thing running inside it is a set of processes sharing the host kernel. Both produce something that "looks like its own machine" from the inside; they differ in what is being simulated, and therefore in what an escape requires.
<a id="131-virtualization-concepts"></a>
13.1 Virtualization Concepts
This section builds the conceptual apparatus: what a virtual machine formally is, the two places a hypervisor can sit, and the three strategies — trap-and-emulate, cooperative guests, and translation — for making guest code run correctly on hardware it does not exclusively own.
Virtual Machine Abstraction
Theory
Start with the problem virtualization was invented to solve. In the 1960s an IBM mainframe was enormously expensive and could run one operating system at a time, while different departments wanted different operating systems. Buying more mainframes was absurd; so IBM's CP-67 instead created the illusion of many mainframes on one, each running its own OS. Sixty years later the economics are the same in a different currency: a data-centre server has 128 cores and a terabyte of RAM, no single tenant's workload uses that, and you cannot safely put several tenants' workloads in one operating system.
A virtual machine is an efficient, isolated duplicate of a real machine. The software that creates it is the virtual machine monitor (VMM) or hypervisor. The essential move is one you have already seen: the OS gives a process the illusion of owning the CPU by intercepting its attempts to do anything privileged; the hypervisor gives a whole kernel the same illusion by the same mechanism. The guest kernel runs deprivileged — it believes it is in kernel mode, but it is not — and every instruction it executes that would actually touch hardware traps into the hypervisor, which emulates the effect against virtual state.
This is trap-and-emulate, and Popek and Goldberg formalised it in 1974 with three requirements a VMM must satisfy:
| Requirement | Meaning | Why it is hard |
|---|---|---|
| Fidelity (equivalence) | A program runs in the VM exactly as it would on bare metal, except for timing and resource availability | The guest must not be able to tell, except through timing |
| Performance (efficiency) | The overwhelming majority of guest instructions execute directly on the real CPU, unmediated | Interpreting every instruction would be correct but useless |
| Safety (resource control) | The VMM has complete control of resources; the guest can never escape its allocation | One guest must not be able to affect another, or the host |
From these they derived the classic theorem. Divide the instruction set into privileged instructions (those that trap when executed in user mode) and sensitive instructions (those that either change the machine's configuration — control-sensitive — or whose behaviour depends on it — behaviour-sensitive). Then:
A VMM using trap-and-emulate can be constructed if the set of sensitive instructions is a subset of the set of privileged instructions.
The intuition is simple: the hypervisor can only intervene where it gets control, and it only gets control on a trap. Any instruction that matters but does not trap is an instruction the guest executes behind the hypervisor's back, producing either wrong results or an escape.
Classic x86 famously failed this test. It had 17 sensitive-but-unprivileged instructions — the notorious example being POPF, which pops flags from the stack. In kernel mode it updates the interrupt-enable flag IF; in user mode it silently ignores that part rather than faulting. A deprivileged guest kernel executing cli-then-popf sequences would think it had disabled interrupts while nothing had happened, and no trap would ever tell the hypervisor to fix it. This single architectural detail is why x86 virtualization took the detour through binary translation (covered later in this section) before hardware support arrived.
Terminology worth keeping straight: the hypervisor's own environment is the host; each virtual machine is a guest; and the guest's illusion is maintained in three dimensions — CPU (vCPUs time-shared onto physical cores), memory (guest-physical addresses translated to host-physical), and I/O (virtual devices backed by real ones, or by files).
Example
THE SAME TRICK, APPLIED ONE LEVEL DOWN
Ordinary OS Hypervisor
------------------------------ ------------------------------
[ proc ][ proc ][ proc ] [ guest OS ][ guest OS ]
| syscall / fault | sensitive instruction
v v
[ kernel ] [ hypervisor ]
[ hardware ] [ hardware ]
The kernel gives a PROCESS the illusion of owning the machine.
The hypervisor gives a KERNEL the same illusion, by the same means:
intercept anything privileged, emulate it against virtual state.
TRAP-AND-EMULATE, ONE INSTRUCTION AT A TIME
guest kernel (running deprivileged, in user mode)
|
| mov eax, [mem] <- ordinary instruction: runs DIRECTLY on
| add eax, 1 the real CPU at full speed. No trap.
|
| mov cr3, eax <- SENSITIVE + PRIVILEGED: reloading the page
| table would hijack the real MMU
+--> TRAP ------------------> hypervisor
decode the instruction
update the VM's VIRTUAL cr3
rebuild the real mapping
advance the guest's PC
<-- resume guest
Fidelity : the guest observes exactly what real hardware would.
Performance: only the rare privileged instruction pays the trap cost.
Safety : the real cr3 was never written by the guest.
WHY CLASSIC x86 BROKE THE THEOREM
sensitive instructions privileged instructions
+----------------------+ +----------------------+
| popf, pushf, sgdt, | | mov cr3, hlt, |
| sidt, sldt, smsw, | | invlpg, lgdt, ... |
| lar, lsl, verr, ... | | |
+----------------------+ +----------------------+
| ^
| these 17 are sensitive but NOT privileged
+---- they do NOT trap in user mode -----------X
popf in kernel mode : updates the interrupt-enable flag IF
popf in user mode : silently ignores the IF bit, does NOT fault
A deprivileged guest "disables interrupts", nothing happens, and the
hypervisor is never told. The illusion is broken silently -- the worst
possible failure mode. Requirement: sensitive (subset of) privileged.
# A miniature trap-and-emulate VMM. The "CPU" executes guest instructions
# directly unless they are sensitive, in which case control transfers to
# the monitor, which emulates them against per-VM virtual state.
class VirtualCPU:
"""Per-VM state the guest THINKS is real hardware."""
def __init__(self):
self.registers = {"eax": 0}
self.cr3 = 0x0 # guest's idea of the page-table base
self.interrupts_on = True
SENSITIVE = {"mov_cr3", "cli", "sti", "hlt", "in", "out"}
class Hypervisor:
def __init__(self):
self.vms = {}
self.exits = 0 # every exit is measurable overhead
self.direct = 0 # instructions run natively: the fast path
def run(self, vm_id, instruction, operand=None):
vcpu = self.vms.setdefault(vm_id, VirtualCPU())
if instruction not in SENSITIVE:
self.direct += 1 # executes on the real CPU
return "native"
self.exits += 1 # TRAP: control leaves the guest
return self._emulate(vcpu, instruction, operand)
def _emulate(self, vcpu, instruction, operand):
if instruction == "mov_cr3":
vcpu.cr3 = operand # virtual cr3 only; the REAL
return f"virtual cr3 <- {operand:#x}" # cr3 is never touched
if instruction == "cli":
vcpu.interrupts_on = False # guest believes interrupts are
return "guest IF cleared (host unaffected)" # off; the host's are not
if instruction == "sti":
vcpu.interrupts_on = True
return "guest IF set"
if instruction == "hlt":
return "vCPU descheduled (host keeps running)" # NOT a real halt
if instruction in ("in", "out"):
return f"device emulation for port {operand:#x}"
raise NotImplementedError(instruction)
hv = Hypervisor()
program = [("mov_eax", None), ("add", None), ("cli", None),
("mov_cr3", 0x7F000), ("add", None), ("sti", None), ("hlt", None)]
for instr, arg in program:
result = hv.run("vm1", instr, arg)
print(f"{instr:<10} -> {result}")
total = hv.direct + hv.exits
print(f"\ndirect: {hv.direct}/{total} exits: {hv.exits}/{total}")
# Real workloads sit near 99.9% direct execution. Performance is entirely
# a question of how often you leave the guest, not how fast emulation is.
Key takeaways
- A virtual machine is an efficient, isolated duplicate of a real machine; the hypervisor creates it by applying to a kernel the same interposition the kernel applies to a process.
- Popek and Goldberg require fidelity, performance, and safety — and performance specifically demands that most guest instructions run natively rather than being interpreted.
- Trap-and-emulate is constructible when every sensitive instruction is also privileged, because the hypervisor can only act where a trap gives it control.
- Sensitive means control-sensitive (changes machine configuration) or behaviour-sensitive (behaves differently depending on it); privileged means it faults in user mode.
- Classic x86 had 17 sensitive-but-unprivileged instructions,
POPFchief among them, which silently did the wrong thing instead of trapping — making pure trap-and-emulate impossible. - The guest illusion has three dimensions — CPU, memory, and I/O — and each needs its own mechanism, covered in section 13.2.
- Virtualization overhead is dominated by the frequency of exits from the guest, not by the cost of any single emulation.
🧪 Practice
- Classify each of
add,hlt,mov cr3,pushf, andrdtscas sensitive, privileged, both, or neither, and say which of them breaks pure trap-and-emulate.- A guest measures elapsed time with
rdtscand notices gaps where the hypervisor descheduled its vCPU. Which Popek–Goldberg requirement does this bear on, and why is it considered acceptable in practice?- Extend the
Hypervisorclass above so thatin/outon port0x3F8(the serial port) appends a byte to a per-VM output buffer, and print the buffer at the end. Note how many exits a one-line message costs.- Interview-style: "Why can't you virtualize a CPU by simply running the guest kernel in kernel mode and hoping for the best?" Hint: think about what the guest's
mov cr3would do to the host's page tables, and what a guest'shltwould do to the other tenants.
Type 1 and Type 2 Hypervisors
Theory
Given that a hypervisor must interpose on privileged operations, there are exactly two places to put it: underneath every operating system, or inside one. That choice is the Type 1 / Type 2 distinction, and almost every practical difference between virtualization products follows from it.
A Type 1 hypervisor — also called bare-metal or native — runs directly on the hardware. It is the lowest-level software on the machine; there is no host operating system beneath it. It owns the physical CPUs, memory, and devices, and schedules guests onto them. VMware ESXi, Xen, Microsoft Hyper-V, and Nutanix AHV are Type 1. Because nothing sits between it and the hardware, it has the shortest path to devices, the smallest trusted computing base, and the most control over scheduling — which is why every serious cloud provider runs one.
A Type 2 hypervisor — hosted — runs as an application on a conventional operating system. It uses the host OS for device drivers, memory allocation, and process scheduling, and its virtual machines appear to the host as ordinary processes. VMware Workstation, VirtualBox, and Parallels are Type 2. The convenience is enormous: you install it like any other program, it inherits driver support for every piece of laptop hardware, and you can run a VM alongside your browser. The cost is a second scheduler and a second memory manager between the guest and the hardware.
The clean two-way split is somewhat dated, because the most important modern hypervisors are deliberately hybrid:
- KVM (Linux) turns the Linux kernel itself into a hypervisor via a loadable module. Linux keeps doing everything it did — scheduling, memory management, drivers — and a VM becomes a Linux process whose threads are vCPUs. Argue whichever way you like: it has a full OS beneath it (Type 2) but that OS is executing as the hypervisor with direct hardware access (Type 1). The practical answer is that KVM gets Type 1 performance with Type 2 driver support, which is why it underpins most of the public cloud.
- Xen uses a privileged guest, dom0, which runs a normal Linux kernel and owns the physical device drivers on behalf of the tiny Type 1 hypervisor beneath it. The hypervisor is minimal; the driver mass lives in a VM.
- Hyper-V similarly demotes the "host" Windows install into a privileged root partition once the role is enabled.
| Dimension | Type 1 (bare-metal) | Type 2 (hosted) |
|---|---|---|
| Runs on | Hardware directly | A host operating system |
| Guest I/O path | Hypervisor to device | Guest to hypervisor to host OS to device |
| Scheduling | Hypervisor schedules vCPUs onto cores | Host OS schedules the VMM's threads |
| Trusted computing base | Small (thin hypervisor) | Large (entire host OS plus its applications) |
| Device driver support | Limited to a certified list | Everything the host OS supports |
| Typical overhead | Low (single-digit percent) | Higher, and less predictable |
| Boot requirement | Dedicated machine | Installs like an application |
| Examples | ESXi, Xen, Hyper-V | VMware Workstation, VirtualBox, Parallels |
| Best suited to | Servers, cloud, consolidation | Desktops, development, testing |
The security consequence deserves emphasis. The attack surface of a hypervisor is everything a guest can reach: the instruction emulator, the virtual devices, the hypercall interface. For a Type 1 hypervisor that is a few hundred thousand lines. For a Type 2 hypervisor it is that plus the entire host kernel, because the VMM makes host system calls on the guest's behalf — and plus every other program running on the host, any of which might be compromised and then attack the VMM's memory. This is why "run the malware sample in VirtualBox on my laptop" is a meaningfully weaker containment story than "run it on a dedicated ESXi host".
Example
WHERE THE HYPERVISOR SITS
TYPE 1 (bare-metal) TYPE 2 (hosted)
+----------+ +----------+ +----------+ +---------+ +--------+
| Guest OS | | Guest OS | | Guest OS | | Guest OS| | Browser|
+----------+ +----------+ +----------+ +---------+ | etc. |
+------------------------+ +------------------------+ +-------+
| HYPERVISOR | | Hypervisor (an app) | |
+------------------------+ +---------------------------------+
| HARDWARE | | HOST OS |
+------------------------+ +---------------------------------+
| HARDWARE |
+---------------------------------+
Note what the guest's I/O has to cross in each case.
KVM: THE HYBRID THAT WON
+----------+ +----------+ +---------+ +---------+
| Guest OS | | Guest OS | | sshd | | nginx | ordinary
+----------+ +----------+ +---------+ +---------+ processes
| QEMU | | QEMU | <- userspace: device models, VM lifecycle
+----------+--+----------+---+---------+--+---------+
| LINUX KERNEL + kvm.ko (this IS the hypervisor) |
+-----------------------------------------------------------------+
| HARDWARE |
+-----------------------------------------------------------------+
A VM is a Linux PROCESS. Each vCPU is a Linux THREAD.
Consequences that follow directly from that one design decision:
- `top` shows VMs; `nice` and cgroups work on them
- the CFS/EEVDF scheduler schedules vCPUs
- the OOM killer can kill a virtual machine
- every Linux driver is available to the host
XEN: A THIN HYPERVISOR PLUS A DRIVER VM
+--------+ +--------+ +--------+
| dom0 | | domU | | domU | dom0 is privileged: it owns
| Linux | | guest | | guest | the real drivers and the
| DRIVERS| | | | | control tools (xl, toolstack)
+--------+ +--------+ +--------+
+--------------------------------+
| XEN HYPERVISOR (~150k LOC) | <- small, so it is auditable
+--------------------------------+
| HARDWARE |
+--------------------------------+
The trusted computing base is small; the driver mass is in a VM that
can crash and be restarted without taking the hypervisor down.
THE I/O PATH, COUNTED IN BOUNDARY CROSSINGS
Type 1: guest driver -> VM exit -> hypervisor -> device (2)
Type 2: guest driver -> VM exit -> VMM -> host syscall
-> host driver -> device (4+)
Each crossing is a context switch and a cache disturbance. This is
the entire performance story, and it is why Type 1 dominates servers.
# ---------- What is actually running on this machine? ----------
# Am I inside a VM, and if so, under what?
systemd-detect-virt # kvm | vmware | xen | oracle | microsoft | none
lscpu | grep -i hypervisor # "Hypervisor vendor: KVM"
# The CPUID hypervisor-present bit: set by any well-behaved hypervisor.
grep -o hypervisor /proc/cpuinfo | head -1
# Does this hardware support acceleration at all?
grep -Eoc '(vmx|svm)' /proc/cpuinfo # vmx = Intel VT-x, svm = AMD-V; 0 = none
lsmod | grep -E '^kvm' # kvm_intel / kvm_amd loaded?
# ---------- KVM: a VM really is just a process ----------
# Start a guest (Type 2-looking command line, Type 1-class performance):
# qemu-system-x86_64 -enable-kvm -m 4G -smp 4 -drive file=disk.qcow2
#
# Then observe it with ordinary process tools:
ps -eo pid,comm,nlwp | grep qemu # nlwp = threads; one per vCPU plus I/O
# PID COMMAND NLWP
# 4821 qemu-system-x86 7 <- 4 vCPU threads + iothreads + main
# Which thread is which vCPU, and where is it running?
ps -T -p 4821 -o spid,comm,psr
# SPID COMMAND PSR
# 4823 CPU 0/KVM 2 <- vCPU 0, currently on physical core 2
# 4824 CPU 1/KVM 7
# Because it is a process, every Linux resource control applies to it:
systemd-run --scope -p CPUQuota=200% -p MemoryMax=8G qemu-system-x86_64 ...
taskset -cp 8-15 4823 # pin vCPU 0 to a NUMA-local core set
# Count VM exits -- the single most useful virtualization metric:
sudo perf kvm stat live # live histogram of exit reasons
sudo perf stat -e kvm:kvm_exit -p 4821 sleep 5
# Frequent EPT_VIOLATION -> memory mapping churn
# Frequent IO_INSTRUCTION -> emulated device; switch it to virtio
# Frequent HLT -> idle guest; usually harmless
Key takeaways
- A hypervisor sits either directly on hardware (Type 1, bare-metal) or as an application on a host OS (Type 2, hosted); everything else follows from that placement.
- Type 1 gives a shorter I/O path, direct control of scheduling, and a small trusted computing base — the reasons it runs servers and clouds.
- Type 2 inherits the host OS's entire driver ecosystem and installs like any program, at the cost of an extra scheduler, an extra memory manager, and a far larger attack surface.
- KVM blurs the categories by making Linux itself the hypervisor: a VM is a process and each vCPU is a thread, so all existing Linux tooling applies to virtual machines.
- Xen keeps the hypervisor tiny and puts the physical drivers in a privileged guest (dom0), trading an extra hop for auditability and driver-crash isolation.
- The security boundary of a Type 2 setup includes the entire host OS and every other process on it, which makes it a weaker containment story than a dedicated Type 1 host.
systemd-detect-virtand the CPUID hypervisor bit answer "am I virtualized?";perf kvm statanswers "why is it slow?" by showing exit reasons.
🧪 Practice
- Run
systemd-detect-virtandgrep -Eoc '(vmx|svm)' /proc/cpuinfoon your machine and explain what each result tells you about where your code is executing.- Draw the boundary crossings for a single 4 KiB disk read issued by a guest under (a) ESXi and (b) VirtualBox on Windows, and explain which of the two has more variable latency.
- Argue both sides of "KVM is a Type 2 hypervisor", then state which properties of Type 1 and Type 2 it actually exhibits.
- Interview-style: "A security team wants to detonate untrusted binaries. They propose VirtualBox on an analyst's laptop. What is your objection, and what would you propose instead?" Hint: enumerate the code an escaping guest would then be adjacent to, and count how much of it is privileged.
Full Virtualization
Theory
Full virtualization means the guest operating system runs completely unmodified — the same binary kernel that boots on bare metal boots in the VM, with no awareness that it is virtualized. This is the property that made virtualization commercially viable: you could take a running Windows NT server, image it, and boot it in a VM without source code, recompilation, or a vendor's cooperation.
Achieving it on an architecture that satisfies Popek–Goldberg is straightforward trap-and-emulate. Achieving it on classic x86, which did not, required an ingenious workaround that VMware shipped in 1999 and which is worth understanding even though hardware has since made it unnecessary.
The technique is binary translation, applied selectively. The insight is that the problem is confined to kernel code: user-mode guest applications contain no sensitive instructions that matter, so they can run directly on the CPU at full native speed. Only the guest kernel needs treatment. So the VMM:
- Runs all guest user code directly on the CPU, unmodified. This is the overwhelming majority of cycles in a typical workload, and it costs nothing.
- Intercepts guest kernel code and translates it, one basic block at a time, before execution. The translation is x86-to-x86 and is nearly the identity function — most instructions are copied verbatim.
- Replaces each problematic instruction in that block with a call into the VMM.
popfbecomes a sequence that updates the virtual interrupt flag.cli/stibecome updates to virtual state. Control-flow at the end of the block is patched to chain to the next translated block. - Caches translated blocks in a translation cache, so a loop is translated once and executed thousands of times. This is what makes the approach viable rather than merely correct.
Guest kernel code, therefore, never executes as written — but it also never notices, because the translation preserves semantics exactly with respect to the virtual machine state.
Two more pieces complete the illusion. Memory is handled with shadow page tables (section 13.2): the guest builds page tables mapping guest-virtual to guest-physical, and the VMM maintains a shadow set mapping guest-virtual directly to host-physical, which is what the real MMU walks. Devices are handled by emulating well-known legacy hardware — an Intel E1000 network card, an LSI SCSI controller, a PS/2 keyboard — precisely because every OS already ships drivers for them. The guest loads its ordinary E1000 driver, writes to what it believes are the card's registers, and each write traps to the VMM, which translates it into real I/O.
That last point is also where full virtualization is slowest, and it explains the natural next step. Emulating an E1000 faithfully costs several VM exits per packet. If you are willing to give up the "completely unmodified guest" property just for the device drivers, you can do much better — which is paravirtualization, the next topic. In practice, modern systems are hybrids: hardware-assisted full virtualization for the CPU and MMU, paravirtual drivers for I/O.
Example
FULL VIRTUALIZATION: WHO RUNS WHAT
guest user code ------------------> runs DIRECTLY on the CPU
(~99% of cycles) full native speed, no work
guest kernel code ---> BINARY TRANSLATOR ---> translation cache
rewrite the 17 bad |
instructions; copy v
everything else execute translated block
(subsequent hits: no
translation, just run)
guest devices ---> emulate an E1000 / LSI / PS/2, because every
OS already has a driver for those
BINARY TRANSLATION, ONE BASIC BLOCK
guest kernel source block translated block in the cache
------------------------- -----------------------------
mov eax, [ebx] mov eax, [ebx] <- copied
add eax, 4 add eax, 4 <- copied
cli mov [VCPU_IF], 0 <- REWRITTEN
mov cr3, eax call VMM_SET_CR3 <- REWRITTEN
sti mov [VCPU_IF], 1 <- REWRITTEN
jmp target call VMM_CHAIN(target) <- patched,
then linked
directly on
the 2nd hit
Nearly the identity translation. Only the instructions that would
touch real machine state are replaced -- with code that touches
VIRTUAL machine state instead.
WHY CACHING IS THE WHOLE GAME
a kernel loop executed 10,000 times:
without a cache : 10,000 translations + 10,000 executions (hopeless)
with a cache : 1 translation + 10,000 executions (viable)
steady state: the translation cache hit rate exceeds 99.9%, so the
translator itself stops appearing in profiles entirely.
FULL vs PARAVIRTUAL: WHAT THE GUEST KNOWS
full virtualization paravirtualization
guest kernel unmodified modified / PV-aware
guest knows? no yes
privileged ops trapped or translated explicit hypercalls
device model emulated legacy HW ring buffers (virtio)
can run Windows? yes, off the shelf only with PV drivers
installed
Modern reality is the diagonal: HARDWARE-ASSISTED full virtualization
for CPU and MMU + PARAVIRTUAL drivers for disk and network.
# A binary translator's core loop, reduced to its essentials: translate a
# basic block once, cache it, and execute the cached form thereafter.
SENSITIVE_REWRITES = {
# instruction -> replacement that manipulates VIRTUAL state instead
"cli": "mov [VCPU.if], 0",
"sti": "mov [VCPU.if], 1",
"popf": "call VMM_EMULATE_POPF", # the instruction that broke x86
"mov cr3": "call VMM_SET_CR3",
"hlt": "call VMM_YIELD_VCPU",
"in": "call VMM_PORT_IN",
"out": "call VMM_PORT_OUT",
}
class BinaryTranslator:
def __init__(self):
self.cache = {} # guest PC -> translated block
self.translations = 0
self.cache_hits = 0
def translate_block(self, block):
out = []
for instr in block:
key = next((k for k in SENSITIVE_REWRITES if instr.startswith(k)), None)
# The identity case dominates: most instructions are simply copied.
out.append(SENSITIVE_REWRITES[key] if key else instr)
return out
def execute(self, pc, block):
if pc in self.cache:
self.cache_hits += 1 # the steady-state path
else:
self.translations += 1
self.cache[pc] = self.translate_block(block)
return self.cache[pc]
kernel_loop = ["mov eax, [ebx]", "add eax, 4", "cli",
"mov cr3, eax", "sti", "jmp 0x1000"]
bt = BinaryTranslator()
translated = bt.execute(0x1000, kernel_loop)
for src, dst in zip(kernel_loop, translated):
mark = " <- rewritten" if src != dst else ""
print(f"{src:<18} => {dst}{mark}")
for _ in range(9999): # the loop runs again and again
bt.execute(0x1000, kernel_loop)
total = bt.translations + bt.cache_hits
print(f"\ntranslations: {bt.translations} cache hits: {bt.cache_hits}")
print(f"hit rate: {bt.cache_hits / total:.4%}")
# Translation cost amortises to nothing. Only the rewritten instructions
# still cost anything, and only when they actually execute.
Key takeaways
- Full virtualization runs a completely unmodified guest kernel, which is what allowed existing OS installations to be moved into VMs without vendor cooperation.
- On architectures satisfying Popek–Goldberg this is plain trap-and-emulate; on classic x86 it required binary translation of guest kernel code.
- Binary translation is applied only to kernel code — guest user code runs directly on the CPU at native speed, which is where nearly all cycles are.
- The translation is x86-to-x86 and almost the identity function: only sensitive instructions are rewritten into calls that update virtual state.
- Caching translated blocks is what makes the technique practical; steady-state hit rates above 99.9% make the translator invisible in profiles.
- Devices are emulated as well-known legacy hardware (E1000, LSI, PS/2) precisely because every guest OS already ships those drivers.
- Emulated device I/O is the main remaining cost, which is exactly the gap paravirtual drivers were created to close.
🧪 Practice
- Explain why binary translation is applied to guest kernel code but not guest user code, and what would change if a user program executed
cli.- Compute the amortised cost per execution of a 20-instruction kernel block that takes 2000 ns to translate and 40 ns to execute, when the block runs 1, 100, and 100,000 times.
- Add a second sensitive instruction and a second basic block to the
BinaryTranslatorexample, and report the cache hit rate after a mixed execution trace.- Interview-style: "Hardware virtualization support has existed since 2006. Why did VMware keep its binary translator for years afterwards?" Hint: early VT-x made every privileged operation a full VM exit; compare the cost of one exit against a rewritten instruction that never leaves the guest.
Paravirtualization
Theory
Full virtualization is defined by what the guest does not know. Paravirtualization takes the opposite position: tell the guest it is virtualized, modify it accordingly, and collect the performance that honesty buys.
The motivation is the cost of maintaining an illusion nobody actually needs. A guest kernel disabling interrupts around a short critical section does not truly require interrupts to be disabled on a physical core — it requires that its own virtual CPU not be interrupted. A guest's network driver does not truly need to poke an E1000's registers — it needs to hand a packet to the outside world. Emulating the hardware faithfully is an expensive way to achieve an outcome that could be requested directly.
So in a paravirtualized system, the guest kernel is ported: privileged operations are replaced with explicit hypercalls to the hypervisor. A hypercall is precisely analogous to a system call, one level down — a deliberate, structured request across a known interface, rather than a trap the hypervisor has to decode and interpret. The differences compound:
- One hypercall can batch many operations. Xen's
mmu_updateapplies a whole list of page-table changes in a single crossing, where trap-and-emulate would cost one exit per write. - No instruction decoding is required. The hypervisor is handed structured arguments rather than having to fetch, decode, and emulate an instruction.
- The guest can be given useful truths that no real hardware exposes: which of its vCPUs are currently scheduled, how much time was stolen from it, that a spinlock's holder is not running and spinning is pointless.
The historically important implementation is Xen, which around 2003 delivered near-native performance on pre-VT-x x86 by requiring ported guests. The cost was exactly the property full virtualization was prized for: a paravirtualized guest kernel must be modified, so Linux and the BSDs could participate and Windows could not.
The compromise that actually won is the hybrid, and it is what essentially everything runs today. Use hardware-assisted full virtualization for the CPU and MMU, so the guest kernel is unmodified and any OS can boot; then load paravirtual drivers inside that unmodified guest for the parts where emulation hurts most — disk, network, console, memory ballooning, clock. The guest is fully virtualized where that is cheap and paravirtualized where that pays.
virtio is the standard interface for this, and its design is the reason it is ubiquitous. Rather than modelling a device, virtio defines a transport: shared-memory virtqueues, in which the guest places descriptors pointing at its own buffers and then "kicks" the host, and the host places completions back. Notifications can be suppressed in both directions when a queue is busy, so a saturated link approaches zero exits per packet — the guest and host simply poll shared memory. There is no register-level device to emulate at all.
| Aspect | Full virtualization | Paravirtualization |
|---|---|---|
| Guest kernel | Unmodified | Ported, or PV drivers installed |
| Privileged operations | Trapped and emulated, or translated | Explicit hypercalls |
| Batching | One exit per operation | Many operations per hypercall |
| Device model | Emulated legacy hardware | Shared-memory ring buffers |
| Exits per network packet | Several | Approaching zero under load |
| Runs closed-source guests | Yes | Only with vendor-provided drivers |
| Guest can be told the truth | No | Yes (steal time, vCPU state) |
The "useful truths" point is worth its own emphasis, because it fixes a genuine pathology. A guest spinlock assumes the lock holder is running and will release shortly — an assumption that is simply false when the hypervisor has descheduled the holder's vCPU. The waiter then spins uselessly through its whole timeslice, a phenomenon called lock-holder preemption, and it can cost an order of magnitude in a spinlock-heavy workload. Paravirtual spinlocks fix it: the guest asks the hypervisor to block the waiting vCPU and wake it when the lock is released. That fix is impossible without the guest knowing it is a guest.
Example
THE SAME OPERATION, THREE WAYS
Guest wants to update 64 page-table entries.
(a) TRAP-AND-EMULATE (full virtualization)
write PTE -> trap -> emulate -> resume ] x 64
cost: 64 VM exits, ~1500 cycles each = ~96,000 cycles
(b) HYPERCALL (paravirtualization)
build a 64-entry list in guest memory
hypercall(MMU_UPDATE, list, 64) -> one crossing
cost: 1 crossing + 64 cheap in-hypervisor updates = ~3,000 cycles
(c) NESTED PAGING (hardware, section 13.2)
write PTE directly; the hardware walks both levels
cost: 0 exits -- but every TLB miss walks two page tables
VIRTIO: A DEVICE WITH NO REGISTERS TO EMULATE
GUEST | HOST
|
driver fills a buffer |
| |
v |
+-----------------------------+ |
| AVAILABLE ring (guest -> host) | shared memory: BOTH sides
| [desc 3][desc 7][desc 9] | | map the same physical pages.
+-----------------------------+ | No copying, no emulation.
| DESCRIPTOR table | |
| 3 -> {addr, len, flags} |<--+--- host reads the guest's buffer
+-----------------------------+ | in place
| USED ring (host -> guest) |
| [desc 3 done, 1514 bytes] | |
+-----------------------------+ |
^ |
| "kick" (one exit) -- and SUPPRESSED while the queue is busy
+----------------------------+
idle link : 1 exit per packet (a kick is needed to wake the host)
busy link : ~0 exits per packet (host is already polling; guest sets
NO_NOTIFY and just writes into the ring)
Throughput therefore IMPROVES the busier it gets -- the opposite of
emulated devices, where every packet costs the same several exits.
LOCK-HOLDER PREEMPTION: THE BUG ONLY AN HONEST GUEST CAN FIX
bare metal : holder runs, releases in ~100 ns, waiter spins 100 ns
naive virtualization: holder's vCPU is DESCHEDULED mid-critical-section
waiter spins for a FULL TIMESLICE (~ms) for nothing
...and may itself be holding another lock
paravirtual spinlock:
waiter spins briefly, then hypercall(VCPU_BLOCK)
releaser does hypercall(VCPU_KICK, waiter)
-> the wasted timeslice becomes useful work for another guest
/* Paravirtual interfaces as they appear in a real guest kernel.
* The pattern is always: replace an instruction that would be trapped
* and emulated with a structured request the hypervisor can act on
* directly -- and, where possible, batch. */
/* --- 1. A hypercall: a syscall to the layer below --- */
struct mmu_update {
uint64_t ptr; /* machine address of the PTE to update */
uint64_t val; /* the new entry */
};
struct mmu_update updates[64];
for (int i = 0; i < 64; i++) {
updates[i].ptr = pte_machine_addr(i);
updates[i].val = make_pte(frames[i], PTE_PRESENT | PTE_RW);
}
/* ONE crossing applies all 64 changes. Trap-and-emulate would cost 64. */
HYPERVISOR_mmu_update(updates, 64, NULL, DOMID_SELF);
/* --- 2. Information no real hardware can provide --- */
uint64_t steal = paravirt_steal_clock(cpu);
/* Nanoseconds this vCPU was RUNNABLE but not scheduled by the hypervisor.
* This is why a guest can show 100% "idle" while running slowly: the time
* was taken by another tenant. Visible as %st in top / vmstat. */
/* --- 3. Paravirtual spinlock: stop spinning against a descheduled holder --- */
static void pv_wait_for_lock(struct spinlock *lock, int want)
{
for (int spins = 0; spins < SPIN_THRESHOLD; spins++) {
if (try_acquire(lock)) /* fast path: holder is running */
return;
cpu_relax();
}
/* Slow path: the holder is probably not on a physical CPU at all.
* Give the core back instead of burning a timeslice. */
HYPERVISOR_vcpu_op(VCPUOP_poll, smp_processor_id(), &lock->waitq);
}
static void pv_unlock(struct spinlock *lock)
{
store_release(&lock->locked, 0);
if (lock->has_waiters) /* explicitly wake the blocked vCPU */
HYPERVISOR_vcpu_op(VCPUOP_send_nmi, lock->waiter_cpu, NULL);
}
/* --- 4. virtio transmit: notification suppression is the point --- */
static int virtio_xmit(struct virtqueue *vq, struct sk_buff *skb)
{
add_buf(vq, skb->data, skb->len); /* write a descriptor into shared memory */
if (vq->used->flags & VRING_USED_F_NO_NOTIFY)
return 0; /* host is polling: NO exit at all */
virtqueue_kick(vq); /* otherwise one exit to wake the host */
return 0;
}
# Is this guest using paravirtual devices, and is it being starved of CPU?
lspci | grep -i virtio
# 00:03.0 Ethernet controller: Red Hat, Inc. Virtio network device
# 00:05.0 SCSI storage controller: Red Hat, Inc. Virtio block device
# If you see "Intel 82540EM" instead, you are on an EMULATED E1000 --
# typically a several-fold throughput loss. Switch the device model.
# Which paravirtual features did the hypervisor negotiate?
dmesg | grep -iE 'kvm|xen|paravirt'
# kvm-clock: using sched offset
# kvm-guest: KVM setup pv remote TLB flush
# kvm-guest: setup PV sched yield
# kvm-guest: KVM setup pv IPIs
# Steal time: the truth an honest guest is told.
vmstat 1 5
# procs -----------cpu------------
# r b us sy id wa st
# 8 0 45 8 2 0 45 <- 45% st: nearly half your CPU is
# going to other tenants. No amount
# of guest-side tuning fixes this.
cat /proc/stat | head -1 # the 8th field is steal, in USER_HZ
Key takeaways
- Paravirtualization tells the guest it is virtualized and modifies it to cooperate, trading the unmodified-guest property for performance.
- Hypercalls are system calls one level down: structured, batchable requests that avoid instruction decoding and collapse many operations into one crossing.
- Xen proved the approach on pre-VT-x x86 but could not support guests whose source it could not modify, notably Windows.
- The arrangement that actually won is hybrid: hardware-assisted full virtualization for CPU and MMU, plus paravirtual drivers for disk, network, clock, and ballooning.
- virtio is not a device model but a shared-memory transport; notification suppression means a busy queue approaches zero exits per packet, so throughput improves under load.
- Paravirtualization uniquely allows the hypervisor to tell the guest things no real hardware could — steal time, vCPU scheduling state — which is what makes fixes like paravirtual spinlocks possible.
- Lock-holder preemption, where a guest spins against a descheduled lock holder, is a real and severe pathology that only an honest guest can avoid.
- Seeing an emulated
Intel 82540EMrather than aVirtio network deviceinlspciis a concrete, fixable performance bug.
🧪 Practice
- Explain the analogy "a hypercall is to a hypervisor what a system call is to a kernel", and name one thing a hypercall can do that a trapped instruction cannot.
- A guest shows 40% steal time in
vmstat. Explain what that means, and say whether adding vCPUs to the guest would help.- Run
lspci | grep -i virtioinside any VM you have access to and report which devices are paravirtual and which are emulated.- Interview-style: "A database in a VM performs far worse than the same database on bare metal with identical CPU counts, and profiling shows enormous time in spinlocks. What is happening?" Hint: ask what the hypervisor might have done to the vCPU holding the lock, and what the waiter has no way of knowing.
Emulation and Binary Translation
Theory
Everything so far has assumed the guest and host share an instruction set — that guest instructions can run directly on the physical CPU, and the hypervisor's job is only to intercept the dangerous ones. Emulation abandons that assumption. It implements a target machine entirely in software, so an ARM binary can run on an x86 host, or a 6502 program from 1982 can run on anything.
The naive approach is an interpreter: a loop that fetches the next guest instruction, decodes it, and executes a host function implementing its semantics. It is simple, portable, and trivially correct — and roughly 10 to 100 times slower than native, because each guest instruction costs dozens of host instructions of dispatch overhead before any useful work happens.
Dynamic binary translation (DBT) eliminates that overhead in the same way the previous topic's translation cache did, but working across architectures. Instead of interpreting instruction by instruction, the emulator:
- Identifies a basic block — a straight-line run of guest instructions ending in a branch.
- Translates the whole block into host machine code, once.
- Caches the result, keyed by guest address.
- Executes the cached host code directly on subsequent visits, and chains blocks so that a hot loop stops returning to the dispatcher entirely.
Because real programs spend nearly all their time in a small fraction of their code, hit rates are extremely high and the cost of translation amortises away. Good DBT lands within 1.5 to 4 times native speed instead of 10 to 100. QEMU's TCG (Tiny Code Generator) works this way, translating guest instructions into an intermediate representation and then into host code, which is how one codebase emulates a dozen architectures on a dozen others.
Real systems add more layers. Tiered translation applies a fast, low-quality translator first and re-optimises only the blocks that prove hot, exactly as JavaScript engines do. Apple's Rosetta 2 goes further with ahead-of-time translation: it translates an entire x86-64 binary to ARM64 at install time, so most code never needs runtime translation at all, and falls back to DBT only for JIT-generated code. Its remarkable performance also owes something to hardware — Apple Silicon implements x86's stricter total store order memory model in hardware, removing what is otherwise DBT's single worst cost.
That memory-model point generalises into the hardest problem in emulation. Correct translation must preserve not only instruction semantics but the target's memory ordering and its exact exception behaviour. Emulating a strongly-ordered guest (x86) on a weakly-ordered host (ARM) requires inserting memory barriers that can dominate the cost; getting the precise fault state right at every instruction that might trap forces the translator to be conservative about what it can optimise. And self-modifying code invalidates cached translations, so the emulator must write-protect translated guest pages to detect it.
| Technique | Guest ISA | Speed vs native | Example |
|---|---|---|---|
| Interpretation | Any | 10–100x slower | QEMU without TCG; classic console emulators |
| Dynamic binary translation | Any | 1.5–4x slower | QEMU TCG, Rosetta 2 runtime path |
| Ahead-of-time translation | Any | 1.1–1.5x slower | Rosetta 2 install-time path |
| Same-ISA binary translation | Same | ~1.05x slower | Classic VMware, pre-VT-x |
| Hardware-assisted virtualization | Same | ~1.01x slower | KVM, VT-x/AMD-V |
The decision rule is clean: if guest and host share an ISA, use hardware virtualization, because nothing else comes close. Use emulation only when you genuinely need a different machine — cross-architecture development, running legacy or foreign binaries, firmware and embedded development before silicon exists, or malware analysis where full instruction-level introspection is worth an order of magnitude in speed.
Example
THE THREE EXECUTION STRATEGIES, SIDE BY SIDE
INTERPRETATION per guest instruction:
fetch -> decode -> switch -> execute -> update PC
~50-200 host instructions to perform ONE guest instruction
cost is paid EVERY time, even in a loop's ten-thousandth iteration
DYNAMIC BINARY TRANSLATION
first visit to a block: translate whole block -> host code -> cache
every later visit: jump straight into cached host code
hot loops: blocks are CHAINED, so the dispatcher is never re-entered
HARDWARE VIRTUALIZATION (same ISA only)
guest instructions ARE host instructions. Nothing to translate.
QEMU TCG: ONE IR, MANY FRONT ENDS AND BACK ENDS
ARM code --\ /-- x86-64 host
RISC-V ---\ +----------------+ +-----+ /--- ARM64 host
MIPS ----+->| TCG frontend |-->| IR |-+---- RISC-V host
PowerPC ---/ +----------------+ +-----+ \--- PowerPC host
x86 --/ \-- ...
N guests x M hosts needs N + M pieces of code, not N x M.
That factorisation is why QEMU emulates everything on everything.
TRANSLATING ONE BLOCK: ARM GUEST -> x86-64 HOST
guest (ARM64) IR (TCG ops) host (x86-64)
----------------- ------------------- ------------------
ldr x0, [x1] ld_i64 t0, env_x1, 0 mov rax, [rbx]
add x0, x0, #4 addi_i64 t0, t0, 4 add rax, 4
str x0, [x2] st_i64 t0, env_x2, 0 mov [rcx], rax
b loop goto_tb 0 jmp <chained block>
^
block chaining: after the
first execution this becomes
a DIRECT jump, so the loop
never returns to QEMU
THE COSTS THAT DO NOT GO AWAY
memory ordering
x86 guest (TSO, strong) on ARM host (weak):
every guest store may need a `dmb ish` barrier
-> can be the single largest source of slowdown
Apple Silicon implements TSO IN HARDWARE and Rosetta 2 enables it
per-process -- a large part of why Rosetta 2 is so fast
self-modifying code
guest writes to a page that has cached translations
-> emulator write-protects translated pages, catches the fault,
invalidates the affected blocks, re-translates
JITs (a browser inside an emulated guest) hit this constantly
precise exceptions
if guest instruction 7 of a 20-instruction block faults, the guest
must observe EXACTLY the state after instruction 6
-> restricts reordering; the translator must be able to reconstruct
intermediate state at any potentially-faulting instruction
# Interpretation versus dynamic binary translation, with the cost model
# made explicit. The structure mirrors QEMU's: decode a basic block,
# translate it once, cache it, chain to the next block.
class Interpreter:
"""One decode-dispatch cycle per guest instruction, every time."""
def __init__(self):
self.host_ops = 0
def run(self, program, iterations):
for _ in range(iterations):
for instr in program:
self.host_ops += 1 # fetch
self.host_ops += 3 # decode: opcode, operands, width
self.host_ops += 8 # dispatch + execute semantics
self.host_ops += 2 # update guest PC and flags
return self.host_ops
class DBT:
"""Translate a basic block once; execute the cached host code after."""
def __init__(self):
self.cache = {}
self.host_ops = 0
self.translations = 0
self.chained = set()
def translate(self, addr, block):
self.translations += 1
self.host_ops += 200 * len(block) # translation is EXPENSIVE, once
# Roughly 1.3 host instructions per guest instruction once compiled.
self.cache[addr] = int(1.3 * len(block))
return self.cache[addr]
def run(self, addr, program, iterations):
for i in range(iterations):
cost = self.cache.get(addr) or self.translate(addr, program)
self.host_ops += cost
if addr in self.chained:
pass # direct jump: no dispatcher cost
else:
self.host_ops += 20 # dispatcher lookup, first time only
self.chained.add(addr)
return self.host_ops
block = ["ldr", "add", "str", "cmp", "b.ne"] # a 5-instruction hot loop
for iterations in (1, 100, 1_000_000):
interp = Interpreter().run(block, iterations)
dbt = DBT().run(0x1000, block, iterations)
native = int(1.0 * len(block) * iterations)
print(f"{iterations:>9} iters | interp {interp/native:6.1f}x | "
f"dbt {dbt/native:5.2f}x (native = 1.0x)")
# 1 iters | interp 14.0x | dbt 208.00x <- translation dominates
# 100 iters | interp 14.0x | dbt 3.36x
# 1000000 iters | interp 14.0x | dbt 1.30x <- amortised to nothing
#
# The crossover is early. This is exactly why every serious emulator,
# JIT, and JavaScript engine translates rather than interprets -- and why
# they all interpret briefly first, before a block proves hot.
# ---------- Emulation in practice ----------
# Full-system emulation of a DIFFERENT architecture (no KVM possible):
qemu-system-aarch64 -M virt -cpu cortex-a72 -m 2G \
-kernel Image -append "console=ttyAMA0" -nographic
# Every guest instruction is translated by TCG. Expect 2-5x slowdown.
# Same architecture: use hardware virtualization instead. Never emulate
# what the CPU can execute directly.
qemu-system-x86_64 -enable-kvm -cpu host -m 4G -smp 4
# ^^^^^^^^^^^ the difference between ~1.01x and ~3x
# User-mode emulation: run a single foreign binary on this kernel.
qemu-aarch64 -L /usr/aarch64-linux-gnu ./arm-binary
# Only the instructions are translated; system calls are passed to the
# host kernel directly. Much faster than full-system emulation.
# Transparent foreign binaries via binfmt_misc (how multi-arch Docker works):
docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
docker run --rm --platform linux/arm64 alpine uname -m # -> aarch64
# The kernel sees an ARM64 ELF header and silently invokes qemu-aarch64.
# macOS: check whether a process is being translated by Rosetta 2.
# sysctl -n sysctl.proc_translated # 1 = running under Rosetta 2
# arch -x86_64 ./binary # force the translated path
Key takeaways
- Emulation implements a foreign machine in software, so the guest ISA need not match the host's — the one thing virtualization cannot do.
- Interpretation is simple and correct but 10–100x slower than native, because decode-and-dispatch overhead is paid on every execution of every instruction.
- Dynamic binary translation translates a basic block once, caches the host code, and chains blocks so hot loops never re-enter the dispatcher, reaching 1.5–4x native.
- QEMU's TCG factors the problem through an intermediate representation, so N guest architectures and M host architectures cost N + M implementations rather than N × M.
- Ahead-of-time translation (Rosetta 2) moves the work to install time and gets closer still, falling back to runtime translation only for JIT-generated code.
- The persistent costs are memory-ordering mismatches (strong guest on weak host needs barriers), self-modifying code (forces translation invalidation), and precise exceptions (limits reordering).
- Apple Silicon's hardware TSO mode is a direct hardware answer to the memory-ordering cost, and a large part of Rosetta 2's speed.
- If the ISAs match, use hardware virtualization; emulate only when you actually need a different machine.
🧪 Practice
- Explain why an emulator interprets a block the first few times before translating it, rather than translating everything immediately.
- Using the cost model in the Python example, find the iteration count at which DBT overtakes interpretation, and explain what that number represents physically.
- Run
docker run --rm --platform linux/arm64 alpine uname -mon an x86 machine and time a small compile inside it against the native platform; explain the ratio you observe.- Interview-style: "Your x86 emulator running on ARM produces correct results single-threaded but intermittently wrong results with multiple threads. What did you get wrong?" Hint: consider what x86 guarantees about the order in which one core's stores become visible to another, and whether ARM makes the same promise.
<a id="132-virtualization-implementation"></a>
13.2 Virtualization Implementation
Having established what a hypervisor must achieve, this section covers how it actually achieves it: the CPU extensions that made x86 virtualizable, the two-level address translation that virtualizes memory, how vCPUs and guest RAM are managed under contention, the spectrum of I/O models from full emulation to direct hardware assignment, and the mechanism that lets a running VM move between machines.
Hardware Virtualization Support
Theory
Recall the x86 defect: 17 sensitive instructions that did not trap, making pure trap-and-emulate impossible and forcing the binary-translation detour. Intel VT-x (2005) and AMD-V (2006) fixed it, and the way they fixed it is more interesting than "they made those instructions trap".
The mechanism is a new orthogonal execution mode. Rather than adding a ring below ring 0, the CPU gained two operating modes, each containing all four rings:
- VMX root operation — where the hypervisor runs. Full privilege, behaves like a normal CPU.
- VMX non-root operation — where guests run. Ring 0 exists here and the guest kernel genuinely runs in it, so the guest is not deprivileged and
popfbehaves exactly as the guest expects. But configurable classes of operation cause a VM exit into root mode instead of executing.
This is why the informal name "ring -1" is misleading but conveys the right idea: the hypervisor is not in a lower ring, it is in a different mode that contains the guest's entire ring hierarchy.
The state that makes this work is the VMCS (Virtual Machine Control Structure; AMD calls it the VMCB). It is a per-vCPU region of memory holding four things:
- Guest state — every register the CPU must load on entry and save on exit: control registers, segment registers, RIP, RSP, RFLAGS, the interrupt state.
- Host state — what the CPU restores when exiting back to the hypervisor, so the transition needs no software save/restore.
- Execution controls — the configuration that decides which operations exit. Should
CPUIDexit?RDTSC? Writes toCR3? Which exceptions? Which I/O ports? This is where a hypervisor tunes its overhead. - Exit information — filled in by the CPU on every exit: the reason, the qualification, the faulting address, the instruction length.
Two instructions drive the loop. VMLAUNCH/VMRESUME performs a VM entry: the CPU atomically saves host state into the VMCS, loads guest state from it, and begins executing guest code. Any configured trigger causes a VM exit: the CPU saves guest state, loads host state, records the exit reason, and resumes the hypervisor at a fixed entry point. The hypervisor handles the event and re-enters.
The performance story reduces to one number: exits per second. An exit costs roughly 1,000–3,000 cycles of pure architectural overhead — the state swap itself — plus indirect costs in TLB and cache disturbance, plus however long the handler runs. First-generation VT-x was often slower than binary translation precisely because it made every privileged operation a full exit, whereas a translated instruction never left the guest at all. Everything since has been an effort to reduce exit frequency: nested paging removed the page-fault exits, posted interrupts removed the interrupt-injection exits, APICv removed the virtual-APIC exits, and virtio removed most device exits.
Beyond the core, the hardware family includes EPT/NPT for memory (next topic), VPID/ASID tags so the TLB need not be flushed on every transition, the IOMMU (VT-d / AMD-Vi) for safe device assignment, SR-IOV for hardware device multiplexing, and nested virtualization support so a guest can itself run a hypervisor — which is what makes cloud-hosted development VMs and Windows' virtualization-based security possible.
Example
VMX: A SECOND DIMENSION, NOT A LOWER RING
BEFORE (rings only) AFTER (modes x rings)
VMX ROOT (hypervisor)
ring 3 applications +---------------------------+
ring 2 (unused) | ring 0 hypervisor |
ring 1 (unused) +---------------------------+
ring 0 kernel ^ |
VM exit | | VM entry
Nowhere left to put a | v
hypervisor without moving VMX NON-ROOT (guests)
the guest kernel out of +---------------------------+
ring 0 -- which is what | ring 3 guest apps |
broke popf. | ring 0 GUEST KERNEL |
+---------------------------+
^ genuinely ring 0, so every
instruction behaves correctly
THE VMCS AND THE ENTRY/EXIT CYCLE
+-------------------- VMCS (one per vCPU) --------------------+
| GUEST STATE : CR0 CR3 CR4, RIP RSP RFLAGS, segments, ... |
| HOST STATE : where to resume the hypervisor |
| CONTROLS : which operations cause an exit <-- TUNABLE |
| (CPUID? RDTSC? CR3 writes? port 0x60?) |
| EXIT INFO : reason, qualification, guest-linear address |
+-------------------------------------------------------------+
hypervisor guest
| VMLAUNCH / VMRESUME
|------------------ VM ENTRY ---------> [ guest runs natively,
| (CPU saves host, loads guest) for as long as it can ]
| |
|<----------------- VM EXIT -------------------+
| (CPU saves guest, loads host,
| writes the exit reason)
| handle it, advance guest RIP, re-enter
EXIT REASONS AND WHAT THEY COST
reason typical cause typical fix
------------------ ------------------------- ----------------------
EPT_VIOLATION guest touched unmapped pre-fault, huge pages
memory
IO_INSTRUCTION emulated device register switch to virtio
EXTERNAL_INTERRUPT host interrupt arrived posted interrupts
CPUID guest feature discovery unavoidable, but rare
HLT guest idle halt-polling
MSR_WRITE timer / APIC programming APICv, kvm-clock
CR_ACCESS guest page-table switch nested paging (EPT)
COST MODEL
one VM exit ~ 1,000-3,000 cycles of architectural overhead
+ handler time
+ TLB/cache disturbance
10,000 exits/s -> negligible
100,000 exits/s -> a few percent
1,000,000 exits/s -> you have a serious problem; find the reason
Why VT-x 1.0 lost to binary translation:
translated `cli` -> a store to virtual state, ~5 cycles, NO exit
trapped `cli` -> a full VM exit, ~1500 cycles
Making an instruction trap is not automatically an improvement.
/* The hypervisor's inner loop, as it appears in every VT-x hypervisor.
* Note the shape: configure once, then enter/exit repeatedly, doing as
* little as possible per exit -- and, ideally, exiting as rarely as possible. */
void configure_vmcs(struct vcpu *vcpu)
{
/* Which operations leave the guest? Every bit set here is a source of
* overhead; every bit cleared is a source of risk if handled wrongly. */
vmcs_write(CPU_BASED_VM_EXEC_CONTROL,
CPU_BASED_HLT_EXITING | /* idle guest -> yield core */
CPU_BASED_USE_IO_BITMAPS | /* only listed ports exit */
CPU_BASED_USE_MSR_BITMAPS | /* only listed MSRs exit */
CPU_BASED_ACTIVATE_SECONDARY_CONTROLS);
vmcs_write(SECONDARY_VM_EXEC_CONTROL,
SECONDARY_EXEC_ENABLE_EPT | /* nested paging: removes
* nearly all MMU exits */
SECONDARY_EXEC_ENABLE_VPID | /* tag the TLB, so entries
* survive entry/exit */
SECONDARY_EXEC_APIC_REGISTER_VIRT); /* APICv: no exit for most
* interrupt controller ops */
/* Which exceptions are the hypervisor's business? Usually almost none:
* let the guest handle its own #PF, #GP, #UD. A bitmap of 0 means the
* guest's own IDT is used, with no exit at all. */
vmcs_write(EXCEPTION_BITMAP, 0);
vmcs_write(HOST_RIP, (unsigned long)vm_exit_handler); /* fixed resume point */
}
void vcpu_run(struct vcpu *vcpu)
{
for (;;) {
/* VM ENTRY. The CPU swaps the entire architectural state atomically;
* there is no software save/restore path to get wrong. */
if (vcpu->launched)
asm volatile("vmresume");
else
asm volatile("vmlaunch"), vcpu->launched = 1;
/* Control arrives here on the next VM EXIT. */
u32 reason = vmcs_read(VM_EXIT_REASON) & 0xffff;
vcpu->stats.exits[reason]++; /* what perf kvm stat reports */
switch (reason) {
case EXIT_REASON_EPT_VIOLATION:
handle_ept_fault(vcpu, vmcs_read(GUEST_PHYSICAL_ADDRESS));
continue; /* no RIP advance: retry the insn */
case EXIT_REASON_CPUID:
emulate_cpuid(vcpu); /* mask features the guest must
* not see; this is how a VM is
* given a stable CPU model */
break;
case EXIT_REASON_HLT:
/* The guest is idle. Poll briefly -- a wakeup often arrives within
* microseconds, and re-entering is cheaper than a full schedule. */
if (!halt_poll(vcpu))
schedule();
break;
case EXIT_REASON_IO_INSTRUCTION:
emulate_pio(vcpu); /* the expensive path: this is
* what virtio exists to avoid */
break;
default:
handle_other(vcpu, reason);
}
/* Most handlers must step over the instruction that caused the exit. */
vmcs_write(GUEST_RIP, vmcs_read(GUEST_RIP) + vmcs_read(VM_EXIT_INSTRUCTION_LEN));
}
}
# ---------- Is hardware support present and enabled? ----------
grep -Eo '(vmx|svm)' /proc/cpuinfo | sort -u # vmx=Intel VT-x, svm=AMD-V
# Empty output with a modern CPU almost always means it is disabled in
# firmware, not absent. Enable "Intel VT-x" / "AMD SVM" in the BIOS.
# Which virtualization features does this CPU actually offer?
grep -Eo 'ept|vpid|npt|vmcb_clean|tsc_scale|apicv|flexpriority' /proc/cpuinfo | sort -u
# ept nested paging (Intel)
# vpid tagged TLB: no flush on VM entry/exit
# flexpriority hardware TPR virtualization
lsmod | grep kvm # kvm_intel / kvm_amd
cat /sys/module/kvm_intel/parameters/nested # Y = guests may run hypervisors
# ---------- The metric that matters: exits per second ----------
sudo perf kvm stat live
# VM-EXIT Samples Samples% Time% Avg time
# EPT_VIOLATION 8421 41.2% 22.1% 2.31us <- memory mapping churn
# MSR_WRITE 5102 25.0% 11.4% 1.02us <- timer programming
# HLT 3011 14.7% 61.2% 41.20us <- idle; time% is sleep
# IO_INSTRUCTION 2955 14.5% 4.9% 0.77us <- EMULATED DEVICE: fix this
# Per-VM counters, straight from the kernel:
sudo cat /sys/kernel/debug/kvm/*/vcpu0/exits 2>/dev/null
sudo perf stat -e 'kvm:kvm_exit' -a sleep 5
# ---------- Reducing exits ----------
# Halt-polling: spin briefly before descheduling an idle vCPU.
cat /sys/module/kvm/parameters/halt_poll_ns # default 200000 (200us)
# Raising it trades host CPU for lower guest wakeup latency.
# Give the guest the real CPU model so it can use native features and
# does not have to ask (and exit) about masked ones:
# qemu-system-x86_64 -cpu host -enable-kvm ...
Key takeaways
- VT-x and AMD-V add an orthogonal mode dimension — VMX root for the hypervisor, non-root for guests — rather than a ring below ring 0.
- Because the guest kernel genuinely runs in ring 0 of non-root mode, sensitive instructions behave correctly and the classic x86 defect disappears.
- The VMCS holds guest state, host state, execution controls, and exit information, letting the CPU swap entire architectural contexts atomically.
- Execution controls decide which operations exit; tuning them is how a hypervisor trades safety against overhead.
- A VM exit costs roughly 1,000–3,000 cycles plus handler time plus cache and TLB disturbance, so exits per second is the metric that governs virtualization performance.
- First-generation VT-x often lost to binary translation because trapping an instruction is more expensive than rewriting it — trapping is not automatically better.
- Every subsequent hardware feature (EPT, VPID, APICv, posted interrupts, SR-IOV) exists to remove a category of exit.
perf kvm stat livenames the dominant exit reason, which usually points directly at the fix — most often replacing an emulated device with virtio.
🧪 Practice
- Explain why VMX non-root mode contains its own ring 0, and what would break if the guest kernel were simply run in ring 1 instead.
- A VM shows 800,000
IO_INSTRUCTIONexits per second at 2,000 cycles each on a 3 GHz core. What fraction of one core is lost to exit overhead alone, and what is the likely fix?- Run
perf kvm stat liveagainst a VM (or read/sys/kernel/debug/kvm/) and identify the top exit reason; propose a change that would reduce it.- Interview-style: "Hardware virtualization shipped in 2005 and was sometimes slower than the software approach it replaced. How is that possible?" Hint: compare the cost of a full architectural state swap against the cost of a rewritten instruction that never leaves the guest.
Shadow Page Tables and Nested Paging
Theory
Memory virtualization is where the elegant trap-and-emulate story gets genuinely difficult, and the reason is that there is now one level of translation too many.
On bare metal there are two address spaces: virtual and physical. The guest's page tables map virtual to physical and the MMU walks them. Under virtualization there are three:
- Guest virtual (GVA) — what a process inside the guest uses.
- Guest physical (GPA) — what the guest kernel believes is physical RAM, starting at 0 and contiguous.
- Host physical (HPA) — actual machine memory, scattered wherever the hypervisor allocated it.
The guest's page tables map GVA to GPA. But the hardware MMU has exactly one page-table base register, CR3, and it walks whatever CR3 points to and emits a real physical address. If the hypervisor lets the guest load its own page tables into CR3, the MMU will emit guest-physical addresses as though they were machine addresses, and the guest will read another VM's memory. That cannot be allowed.
Shadow page tables were the software answer. The hypervisor maintains its own set of page tables — the shadows — which map GVA directly to HPA by composing the guest's GVA-to-GPA mapping with the hypervisor's GPA-to-HPA mapping. CR3 points at the shadow tables, so the MMU walks a correct, composed mapping at full hardware speed, and the guest's own tables are never used by hardware at all.
Keeping the shadows correct is the problem. The guest edits its page tables whenever it maps a file, forks, or handles a fault, and the hypervisor must notice. The technique is to write-protect every guest page-table page: a guest write traps, the hypervisor emulates the write and updates the corresponding shadow entry. The consequences are severe:
- Every guest page-table modification costs a VM exit.
fork()and process exit touch hundreds. - Each guest process needs its own shadow table, so the hypervisor caches many shadow hierarchies and consumes significant host memory.
- Guest
CR3writes — every context switch — exit so the shadow root can be swapped. - A page fault must be classified: is it real (the guest's own mapping is absent, so inject it into the guest) or hidden (the guest mapping exists but the shadow is stale, so fix it silently)? Getting this wrong is a correctness bug.
Memory-intensive workloads could lose 10–30% to shadow paging, and the complexity was a notorious source of hypervisor bugs.
Nested paging — Intel EPT (Extended Page Tables), AMD NPT (Nested Page Tables), ARM Stage-2 translation — puts the second level in hardware. The MMU now holds two pointers: the guest's CR3 for GVA-to-GPA, and an EPT pointer, controlled only by the hypervisor, for GPA-to-HPA. On a TLB miss the hardware walks both, composing them itself.
The guest may now write CR3 and edit its page tables freely, with no exits at all, because the EPT layer cannot be bypassed regardless of what the guest does. Exits occur only on genuine EPT violations — the guest touched a guest-physical page the hypervisor has not yet backed — which is roughly once per page rather than once per page-table edit.
The cost moves rather than disappearing. A TLB miss now requires a two-dimensional page walk: each of the 4 levels of the guest walk produces a GPA that must itself be translated through 4 levels of EPT, so the worst case is (4+1) × (4+1) − 1 = 24 memory accesses instead of 4. TLB hits are unaffected, so the penalty falls entirely on TLB misses — which is why huge pages matter so much more under virtualization than on bare metal. 2 MiB pages shorten both dimensions of the walk and cover 512 times more memory per TLB entry; enabling them on both the guest and host side is the single highest-value memory tuning for a VM. VPID/ASID tagging complements this by letting TLB entries survive VM entry and exit instead of being flushed.
| Shadow page tables | Nested paging (EPT/NPT) | |
|---|---|---|
| Where GVA→HPA composition happens | Software (hypervisor) | Hardware (MMU) |
| Guest page-table write | VM exit, emulate, update shadow | No exit |
Guest CR3 write (context switch) | VM exit, swap shadow root | No exit |
| Exits caused by memory | Very many | Only real EPT violations |
| TLB hit cost | Native | Native |
| TLB miss cost | Native (1-D walk) | Up to 24 accesses (2-D walk) |
| Host memory overhead | High (a shadow per guest process) | Low (one EPT per VM) |
| Best for | Nothing, today | Everything, especially with huge pages |
Example
THREE ADDRESS SPACES, TWO TRANSLATIONS
process in guest guest kernel's view real machine
+-------------+ +---------------+ +---------------+
| GVA | --1-> | GPA | --2--> | HPA |
| 0x7fff1000 | | 0x00201000 | | 0x8a3f1000 |
+-------------+ +---------------+ +---------------+
guest page tables hypervisor's map
(guest controls) (guest must NEVER control)
The MMU has one CR3 and emits one physical address. Something has to
compose the two mappings. Software did it first; hardware does it now.
SHADOW PAGE TABLES: COMPOSE IN SOFTWARE, WRITE-PROTECT TO STAY CORRECT
guest page tables SHADOW page tables CR3 --> shadow
(write-protected!) (hypervisor-built)
GVA 0x7fff1000 GVA 0x7fff1000
-> GPA 0x201000 +----> -> HPA 0x8a3f1000 <- what the MMU
| actually walks
hypervisor's GPA map |
GPA 0x201000 ----------+
-> HPA 0x8a3f1000
guest writes a PTE
|
v page is WRITE-PROTECTED -> VM EXIT
hypervisor: emulate the write, update the shadow entry, resume
-> correct, but one exit PER PAGE-TABLE WRITE.
fork() of a large process: hundreds of exits.
context switch (CR3 write): an exit, plus a shadow root swap.
NESTED PAGING: COMPOSE IN HARDWARE
+-----------------------------------------------+
| MMU |
| guest CR3 ----> GVA to GPA (guest owns) |
| EPT pointer --> GPA to HPA (hypervisor) |
+-----------------------------------------------+
The guest edits its page tables and reloads CR3 freely: NO EXITS.
The EPT layer is not bypassable, so safety no longer needs traps.
THE PRICE: A TWO-DIMENSIONAL PAGE WALK
bare metal, 4-level paging: 4 memory accesses per TLB miss
nested, 4-level x 4-level:
each of the guest's 4 levels yields a GPA, which must itself be
translated through 4 EPT levels, plus the final translation
guest L4 -> [EPT walk: 4] -> guest L3 -> [EPT walk: 4] ->
guest L2 -> [EPT walk: 4] -> guest L1 -> [EPT walk: 4] ->
GPA -> [EPT walk: 4]
worst case: 24 memory accesses per TLB miss (~6x)
TLB HITS ARE UNAFFECTED. The entire penalty falls on misses --
so the fix is to have fewer misses:
4 KiB pages : 4-level guest walk x 4-level EPT = up to 24 accesses
2 MiB pages : 3-level x 3-level = up to 15 accesses
AND 512x more memory per TLB entry
1 GiB pages : 2-level x 2-level = up to 8 accesses
Huge pages must be enabled on BOTH sides to get the full effect.
This is the single highest-value memory tuning for a VM.
VPID / ASID: STOP FLUSHING THE TLB
without VPID: every VM entry and exit flushes the whole TLB
-> the guest restarts cold thousands of times a second
with VPID : each VM's entries are TAGGED and coexist in the TLB
-> entries survive the round trip
# Shadow page tables versus nested paging, counting the two costs that
# actually differ: VM exits (shadow's problem) and page-walk memory
# accesses on a TLB miss (nested paging's problem).
class ShadowPaging:
def __init__(self):
self.exits = 0
self.walk_accesses = 0
def guest_writes_pte(self):
self.exits += 1 # write-protected page -> trap, emulate,
# update the shadow entry
def guest_context_switch(self):
self.exits += 1 # CR3 write -> swap the shadow root
def tlb_miss(self):
self.walk_accesses += 4 # the MMU walks the SHADOW table: 1-D,
# so a miss costs exactly native
class NestedPaging:
def __init__(self, guest_levels=4, ept_levels=4):
self.exits = 0
self.walk_accesses = 0
self.g = guest_levels
self.e = ept_levels
def guest_writes_pte(self):
pass # no exit: the EPT layer cannot be bypassed
def guest_context_switch(self):
pass # no exit: guest CR3 is the guest's own
def tlb_miss(self):
# Each guest level emits a GPA needing a full EPT walk, plus the
# final GPA translation.
self.walk_accesses += (self.g + 1) * (self.e + 1) - 1
def ept_violation(self):
self.exits += 1 # only for genuinely unbacked guest pages
def workload(mmu, forks=1000, ptes_per_fork=200,
switches=50_000, tlb_misses=2_000_000):
for _ in range(forks):
for _ in range(ptes_per_fork):
mmu.guest_writes_pte()
for _ in range(switches):
mmu.guest_context_switch()
for _ in range(tlb_misses):
mmu.tlb_miss()
return mmu
shadow = workload(ShadowPaging())
nested = workload(NestedPaging())
nested_2m = workload(NestedPaging(guest_levels=3, ept_levels=3)) # 2 MiB pages
EXIT_CYCLES, ACCESS_CYCLES = 2000, 100 # a walk access often misses cache
for name, mmu in (("shadow (4K)", shadow),
("nested (4K)", nested),
("nested (2M)", nested_2m)):
cycles = mmu.exits * EXIT_CYCLES + mmu.walk_accesses * ACCESS_CYCLES
print(f"{name:<12} exits {mmu.exits:>9,} walk accesses "
f"{mmu.walk_accesses:>12,} total {cycles/1e9:7.2f} Gcycles")
# shadow (4K) exits 250,000 walk accesses 8,000,000 total 1.30 Gcycles
# nested (4K) exits 0 walk accesses 48,000,000 total 4.80 Gcycles
# nested (2M) exits 0 walk accesses 30,000,000 total 3.00 Gcycles
#
# Note what this shows: nested paging is NOT unconditionally faster. With a
# TLB-thrashing workload and 4 KiB pages it can lose. What makes it win in
# practice is that real TLB hit rates are ~99%+, and that huge pages both
# shorten the walk AND eliminate most misses. Measure; do not assume.
# ---------- Confirm nested paging is on ----------
grep -Eo 'ept|npt' /proc/cpuinfo | sort -u
cat /sys/module/kvm_intel/parameters/ept # Y
cat /sys/module/kvm_amd/parameters/npt # Y
# ---------- Huge pages: the highest-value VM memory tuning ----------
# Transparent huge pages, host side:
cat /sys/kernel/mm/transparent_hugepage/enabled
# [always] madvise never
grep -i AnonHugePages /proc/meminfo # how much is actually backed
# Explicit 1 GiB pages for a large, latency-sensitive VM:
# kernel cmdline: default_hugepagesz=1G hugepagesz=1G hugepages=32
grep Huge /proc/meminfo
# HugePages_Total: 32
# Hugepagesize: 1048576 kB
# Back a VM with them (QEMU):
# -object memory-backend-file,id=mem,size=32G,mem-path=/dev/hugepages,share=on \
# -numa node,memdev=mem -mem-prealloc
# Inside the guest, enable THP as well -- both dimensions of the walk matter.
# ---------- Measure the actual cost ----------
sudo perf stat -e dtlb_load_misses.walk_active,dtlb_load_misses.walk_completed \
-p $(pgrep -f qemu | head -1) sleep 10
# Rising walk_active under virtualization is the two-dimensional walk showing up.
sudo perf kvm stat live | grep EPT_VIOLATION
# Steady-state EPT violations should be near zero once the guest is warm.
# Continuous violations mean memory is being reclaimed or ballooned away.
Key takeaways
- Virtualization introduces a third address space, so GVA must be translated to GPA and then to HPA while the MMU offers only one page-table base register.
- Shadow page tables compose the two mappings in software and point
CR3at the result, keeping the composed table correct by write-protecting the guest's page tables. - That correctness mechanism makes every guest page-table write and every guest context switch a VM exit, costing 10–30% on memory-intensive workloads.
- Nested paging (EPT/NPT/Stage-2) gives the MMU a second, hypervisor-owned pointer so hardware composes both mappings, eliminating those exits entirely.
- The cost becomes a two-dimensional page walk of up to 24 memory accesses per TLB miss instead of 4; TLB hits are unaffected.
- Huge pages shorten both dimensions of the walk and drastically reduce miss frequency, making them far more valuable under virtualization than on bare metal.
- VPID/ASID tags let TLB entries survive VM entry and exit, removing what would otherwise be a full flush thousands of times per second.
- Nested paging is not unconditionally faster than shadow paging for every workload — it wins because real TLB hit rates are high and huge pages amplify that.
🧪 Practice
- Given a 4-level guest page table and 4-level EPT, derive the 24-access worst case for a single TLB miss, and recompute it for 1 GiB pages on both sides.
- Explain why shadow paging must write-protect guest page tables, and describe what goes wrong if one write is missed.
- Check whether transparent huge pages are enabled on a host you control, run a memory-intensive benchmark in a VM with THP on and off, and report the difference.
- Interview-style: "A VM's memory-heavy workload runs 25% slower than bare metal, but
perf kvm statshows almost no VM exits. Where is the time going?" Hint: exits are not the only virtualization cost; think about what a TLB miss has to do now that it did not have to do before.
CPU and Memory Virtualization
Theory
The previous topics covered the mechanisms; this one covers the resource management built on top of them, and its defining feature is overcommitment — selling more virtual resources than physically exist, because not everyone uses their allocation simultaneously. It is the same bet an airline makes with seats, and it is where virtualization's economics actually come from.
CPU virtualization. Each vCPU is a schedulable entity — under KVM, literally a host thread. The hypervisor schedules vCPUs onto physical cores exactly as an OS schedules threads, which produces the phenomenon of double scheduling: the guest kernel schedules its processes onto vCPUs, unaware that the hypervisor is independently scheduling those vCPUs onto cores. Two schedulers make decisions with incomplete information about each other, and the pathologies follow:
- Lock-holder preemption — a vCPU is descheduled while holding a guest spinlock, and other vCPUs spin uselessly (fixed by paravirtual spinlocks).
- vCPU stacking — the guest assumes its vCPUs run simultaneously, but they are time-sliced, so barrier and IPI-based synchronisation stalls.
- Steal time — time a vCPU was runnable but not scheduled. It appears as
%stin the guest and explains workloads that are slow while apparently idle.
Gang scheduling (VMware calls it co-scheduling) mitigates this by trying to run a VM's vCPUs simultaneously, at the cost of scheduling flexibility. The far simpler mitigation is the one that most often matters in practice: do not give a VM more vCPUs than its workload uses. An 8-vCPU VM running a single-threaded workload is harder to schedule, suffers more stacking, and performs worse than a 2-vCPU VM.
Memory virtualization faces a harder version of the same problem, because memory is not time-shared — a page is either resident or it is not. The hypervisor's difficulty is that a guest OS is designed to use all available memory, mostly as page cache, and never voluntarily gives it back. From the hypervisor's outside view, a guest with 16 GiB allocated looks fully occupied even when 12 GiB is discardable cache. Several techniques address this:
- Ballooning. A driver inside the guest allocates memory and pins it, then tells the hypervisor those pages are free to reclaim. The balloon "inflates", and the guest OS — which understands its own working set far better than the hypervisor ever could — decides what to evict. This is paravirtualization applied to memory, and it is the only technique that uses the guest's own knowledge.
- Page sharing / deduplication. Identical pages across VMs — the same kernel text, the same libc — are mapped copy-on-write to one host page. Linux implements this as KSM. It can save substantially on homogeneous fleets, but it enables cross-VM side channels (a write to a shared page takes measurably longer), which is why cloud providers generally disable it between tenants.
- Host swapping. The hypervisor pages guest memory to disk. It is the last resort and the worst option, because the hypervisor cannot see which guest pages matter and will happily swap out a page the guest is using as its own page cache — producing double paging, where the guest evicts a page the host has already swapped, forcing a read from disk purely to write it back out.
- Free page reporting — a modern virtio feature where the guest proactively tells the host which pages are free, achieving ballooning's benefit without the balloon.
The ordering of those techniques is the operational rule: share and balloon first, swap only under duress. And the hard rule above all: never overcommit memory for latency-sensitive workloads. CPU overcommit degrades gracefully; memory overcommit falls off a cliff.
Example
DOUBLE SCHEDULING
guest scheduler: [p1][p2][p3] -> vCPU0 [p4][p5] -> vCPU1
| |
v v
host scheduler: vCPU0 --+ +-- vCPU1
| |
core 3 (also running another VM's vCPU)
Neither scheduler knows what the other is doing.
guest believes : "p1 is running right now on vCPU0"
reality : vCPU0 has not been on a core for 8 ms
Consequences:
- spinlocks spin against a holder that is not running
- barriers and IPIs stall waiting for stacked vCPUs
- guest timekeeping drifts (hence kvm-clock)
- the guest's own scheduling decisions are based on a fiction
WHY MORE vCPUs CAN BE SLOWER
8-vCPU VM, single-threaded workload, contended host:
the host must find 8 cores to co-schedule (under gang scheduling)
-> the VM waits longer to be scheduled at all
-> 7 vCPUs idle-spin, and every guest IPI must reach all 8
2-vCPU VM, same workload:
scheduled almost immediately, every time.
RULE: size vCPUs to the workload, not to the invoice.
MEMORY OVERCOMMIT: THE TECHNIQUES, IN ORDER OF PREFERENCE
host has 64 GiB. Four VMs are each promised 24 GiB (96 GiB total).
1. PAGE SHARING (KSM) identical pages -> one host page, COW
VM1 libc ---\
VM2 libc ----+--> one host frame (read-only, COW on write)
VM3 libc ---/
saves the most on homogeneous fleets; leaks a side channel
2. BALLOONING the guest chooses what to give back
hypervisor: "inflate to 8 GiB"
balloon driver in guest: allocate 8 GiB, pin it, report the PFNs
guest OS: evicts its own COLDEST pages to satisfy the allocation
^^^^^^^^^^ the guest knows; the hypervisor does not
hypervisor: reclaims those host frames
3. FREE PAGE REPORTING guest volunteers free pages continuously
no balloon, no allocation pressure; modern virtio-balloon feature
4. HOST SWAPPING last resort, and genuinely bad
the host picks victims BLINDLY:
DOUBLE PAGING
host swaps out guest page X (it looked cold from outside)
guest decides to evict X from its own page cache
-> the guest must READ X back from host swap
just to WRITE it to guest swap
-> two disk I/Os to discard a page that was already free
THE ASYMMETRY THAT GOVERNS CAPACITY PLANNING
CPU overcommit : degrades LINEARLY. 2x overcommit ~ 2x slower.
4:1 is routine for general-purpose workloads.
MEMORY overcommit: degrades off a CLIFF. Fits in RAM -> full speed.
Does not fit -> swapping -> 1000x slower.
For latency-sensitive workloads: do not overcommit.
# ---------- CPU: is this guest actually getting a CPU? ----------
vmstat 1 5
# r b us sy id wa st
# 12 0 38 6 4 0 52 <- 52% STEAL. Half the CPU you are paying
# for is going elsewhere. No guest-side
# tuning fixes this; you need fewer
# neighbours or a dedicated host.
# Host side: where are the vCPU threads and how much are they waiting?
ps -T -p $(pgrep -f 'qemu.*vm1') -o spid,comm,psr,pcpu
grep -E 'se.statistics.wait_sum' /proc/<vcpu-tid>/sched # accumulated runqueue wait
# Pin vCPUs to dedicated cores and keep memory NUMA-local:
virsh vcpupin vm1 0 8 # vCPU 0 -> physical CPU 8
virsh numatune vm1 --nodeset 1 --mode strict
numactl --hardware # confirm the node layout first
# Isolate cores from the host scheduler entirely, for latency-sensitive VMs:
# kernel cmdline: isolcpus=8-15 nohz_full=8-15 rcu_nocbs=8-15
# ---------- Memory: what is real and what is promised? ----------
free -h # inside a guest this shows the ILLUSION
cat /proc/meminfo | grep -i commit
# CommitLimit: 67108864 kB
# Committed_AS: 98566144 kB <- promised more than exists: overcommitted
# Ballooning:
virsh dommemstat vm1
# actual 24117248 <- currently assigned
# balloon 16777216 <- balloon has reclaimed 8 GiB back to the host
# unused 2097152
virsh setmem vm1 12G --live # inflate the balloon; the GUEST picks victims
# Page deduplication (KSM):
cat /sys/kernel/mm/ksm/pages_sharing # pages saved by dedup
cat /sys/kernel/mm/ksm/run # 1 = on
echo 0 | sudo tee /sys/kernel/mm/ksm/run # DISABLE for multi-tenant hosts:
# a write to a shared page is measurably slower, which is a cross-VM channel.
# Is the host swapping guest memory? This is the number that matters.
vmstat 1 | awk '{print $7, $8}' # si/so columns on the HOST
# Non-zero si/so with VMs running means guest memory is on disk. Fix by
# reducing overcommit, not by tuning the guest.
# Why memory overcommit is a cliff and CPU overcommit is a slope.
def cpu_overcommit(vcpus_total, physical_cores):
ratio = vcpus_total / physical_cores
# Time-sharing degrades proportionally: each vCPU gets 1/ratio of a core.
return max(1.0, ratio) # slowdown factor
def memory_overcommit(promised_gb, physical_gb, working_set_gb):
if working_set_gb <= physical_gb:
return 1.0 # fits: full speed, regardless of what
# was PROMISED. Overcommit is free here.
# Does not fit: the excess is served from disk, ~100,000x slower per access.
excess_fraction = (working_set_gb - physical_gb) / working_set_gb
return 1 + excess_fraction * 100_000
print("CPU overcommit (32 physical cores):")
for vcpus in (32, 64, 128, 256):
print(f" {vcpus:>3} vCPUs ({vcpus/32:.0f}:1) -> {cpu_overcommit(vcpus, 32):6.1f}x slower")
print("\nMemory overcommit (64 GiB physical, 96 GiB promised):")
for ws in (40, 60, 64, 68, 80):
print(f" working set {ws:>2} GiB -> {memory_overcommit(96, 64, ws):12,.1f}x slower")
# CPU overcommit (32 physical cores):
# 32 vCPUs (1:1) -> 1.0x slower
# 256 vCPUs (8:1) -> 8.0x slower <- painful but LINEAR
#
# Memory overcommit (64 GiB physical, 96 GiB promised):
# working set 64 GiB -> 1.0x slower
# working set 68 GiB -> 5,883.4x slower <- 4 GiB over: a CLIFF
#
# Overcommitting memory is safe right up until the moment it is catastrophic,
# and the transition is a step function. That is why the two resources are
# planned with completely different risk tolerances.
Key takeaways
- Overcommitment — selling more virtual resources than physically exist — is where virtualization's economics come from, but CPU and memory overcommit behave completely differently.
- Each vCPU is a schedulable entity, so guest and hypervisor schedulers make decisions independently: this double scheduling causes lock-holder preemption, vCPU stacking, and steal time.
- Steal time (
%st) is time a vCPU was runnable but not scheduled, and it is the first thing to check when a guest is slow but apparently idle. - Over-provisioning vCPUs makes a VM harder to schedule and often slower; size vCPUs to the workload's actual parallelism.
- Guest operating systems consume all available memory as page cache and never return it, so the hypervisor needs explicit mechanisms to reclaim.
- Ballooning works because the guest, not the hypervisor, decides which pages to evict — it is the only reclamation technique that uses the guest's own knowledge of its working set.
- Page deduplication (KSM) saves memory on homogeneous fleets but creates a measurable cross-VM timing channel, so it is normally disabled between tenants.
- Host swapping is the worst option and causes double paging, where a page is read back from host swap purely so the guest can discard it.
- CPU overcommit degrades linearly and 4:1 is routine; memory overcommit degrades as a step function, so latency-sensitive workloads should never be memory-overcommitted.
🧪 Practice
- Explain why a guest OS "using all its memory" is normal and not a reason to allocate more, and why that makes the hypervisor's reclamation problem hard.
- Run
vmstat 1inside any VM you can access and interpret thestcolumn; state what action you would take if it read 30%.- Describe double paging step by step and count the disk I/Os involved in discarding one page under it.
- Interview-style: "A tenant complains their 16-vCPU VM is slower than the 4-vCPU VM they replaced, on an otherwise unchanged host. What do you investigate?" Hint: consider what the host scheduler has to arrange before this VM can run at all, and what the guest's synchronisation primitives assume about simultaneity.
I/O Virtualization
Theory
CPU and memory virtualization reached near-native performance through hardware assistance. I/O took longer, and the reason is structural: a CPU instruction executes and is done, but an I/O operation involves a device that runs asynchronously, performs DMA into memory, and raises interrupts. Each of those is a boundary crossing, and each crossing is expensive.
There are four models, forming a clear spectrum from maximum compatibility to maximum performance.
1. Full device emulation. The hypervisor implements a real, well-known device in software — an Intel E1000 NIC, an LSI SCSI controller, an IDE disk. The guest loads its stock driver and writes to what it believes are hardware registers; each access traps to the hypervisor, which updates emulated device state and eventually performs real I/O. It works with absolutely any guest, including ancient and closed ones, and it is slow: several VM exits per operation, plus interrupt injection, giving perhaps 10–20% of native throughput. Its irreplaceable role is booting — a guest needs an emulated IDE controller or NVMe device to load the kernel that contains its paravirtual drivers.
2. Paravirtualized devices (virtio). Covered in 13.1: shared-memory virtqueues, batching, and notification suppression, reaching 80–95% of native. This is the default for essentially every cloud VM. vhost improves it further by moving the datapath into the host kernel (vhost-net) or a dedicated userspace process (vhost-user, used by DPDK), so the guest's ring operations are serviced without waking the hypervisor's userspace at all.
3. Direct device assignment (passthrough). The physical device is removed from the host and assigned exclusively to one guest, which drives it with its native driver. Performance is genuinely native. Two problems had to be solved first:
- DMA safety. A device DMAs to physical addresses. A guest programming a device directly would supply guest-physical addresses, which the device would treat as machine addresses — writing into the host or another VM. The IOMMU (Intel VT-d, AMD-Vi) solves this: it is an MMU for devices, translating device-issued addresses through hypervisor-controlled page tables. Passthrough without an IOMMU is not merely slow, it is a complete loss of isolation.
- Interrupts. The device's interrupts must reach the right guest; interrupt remapping handles this, and posted interrupts deliver them directly to a running vCPU with no VM exit at all.
The costs: the device serves exactly one VM, and live migration becomes very difficult because the guest holds opaque device state the hypervisor cannot capture.
4. SR-IOV (Single Root I/O Virtualization). Hardware multiplexing: the device itself presents one physical function (PF) and many lightweight virtual functions (VFs), each with its own registers, queues, and DMA context. Each VF is passed through to a different guest. This gives passthrough performance with the sharing that passthrough lacks — one 100 GbE NIC serving 64 VMs at near-native speed. The limits are a fixed VF count set by the hardware, the same migration difficulty, and the need for guest drivers for that specific device.
| Model | Throughput | Guest driver | Shareable | Migratable | Use for |
|---|---|---|---|---|---|
| Emulation | 10–20% | Stock legacy | Yes | Yes | Booting, ancient guests |
| virtio (PV) | 80–95% | virtio | Yes | Yes | The default for everything |
| virtio + vhost | 90–98% | virtio | Yes | Yes | High-throughput networking |
| Passthrough | ~100% | Native device | No | Hard | GPUs, specialised hardware |
| SR-IOV | ~100% | Native VF | Yes (fixed N) | Hard | NFV, HPC, low-latency trading |
The decision rule: use virtio unless you have measured a reason not to, because it costs a few percent and preserves migration, snapshots, and hardware independence. Reach for SR-IOV or passthrough when the workload genuinely needs native I/O and you can accept pinning the VM to that host.
Example
THE I/O SPECTRUM: COMPATIBILITY <-> PERFORMANCE
EMULATION VIRTIO SR-IOV / PASSTHROUGH
any guest any modern guest needs a specific driver
10-20% native 80-95% native ~100% native
migrates freely migrates freely migration is hard
+-----------------------------------------------------------+
compatibility ------------------------------> performance
(1) EMULATED E1000: COUNTING THE CROSSINGS FOR ONE PACKET
guest driver writes TDT (transmit tail) register
| VM EXIT #1
v
hypervisor: decode the MMIO write, update emulated register state,
read the descriptor ring from guest memory,
copy the packet, send it on the real NIC
|
completion: update emulated status, INJECT an interrupt
| VM EXIT #2 (interrupt delivery)
v
guest ISR reads ICR (interrupt cause register)
| VM EXIT #3
v
...several more for descriptor cleanup
~5 exits x ~2000 cycles = ~10,000 cycles PER PACKET.
At 1.5 Mpps that is more than five full cores of pure overhead.
(2) VIRTIO + VHOST: THE DATAPATH LEAVES USERSPACE
without vhost with vhost-net
guest -> KVM -> QEMU -> host guest -> KVM -> vhost thread -> host
(userspace hop) (stays in the kernel)
and with notification suppression under load, the "-> KVM" step
disappears too: the vhost thread simply polls the shared ring.
(3) PASSTHROUGH: WHY THE IOMMU IS NOT OPTIONAL
WITHOUT an IOMMU
guest programs the NIC: "DMA this packet to 0x1000"
guest means GUEST-physical 0x1000
device writes MACHINE physical 0x1000 <- the host's memory.
Total loss of isolation. Not a performance issue: a security hole.
WITH an IOMMU (VT-d / AMD-Vi)
+--------+ DMA to 0x1000 +-------+ translate +----------+
| device |-------------------->| IOMMU |--------------->| HPA |
+--------+ +-------+ via hypervisor| 0x8a3f000|
page tables +----------+
A device can reach EXACTLY the pages the hypervisor mapped for it,
and nothing else. The IOMMU is to devices what the MMU is to processes.
(4) SR-IOV: THE DEVICE MULTIPLEXES ITSELF
+---------------------------------------------------+
| PHYSICAL NIC (one card) |
| +------+ +------+ +------+ +------+ |
| | PF | | VF0 | | VF1 | | VF2 | ... |
| +------+ +------+ +------+ +------+ |
| host VM A VM B VM C |
| Each VF has its OWN registers, queues, MAC and |
| DMA context. An embedded switch sorts the traffic.|
+---------------------------------------------------+
Passthrough performance WITH sharing. The trade remains migration:
the guest holds device state the hypervisor cannot serialise.
# ---------- Which model is this VM using? ----------
lspci -nn | grep -iE 'ethernet|scsi|nvme'
# Red Hat, Inc. Virtio network device <- paravirtual: good
# Intel 82540EM Gigabit Ethernet <- EMULATED: fix this
# Intel X710 Virtual Function <- SR-IOV VF: near-native
ethtool -i eth0 | head -3
# driver: virtio_net (or: e1000 / iavf / ixgbevf)
# ---------- Enable and use SR-IOV ----------
# 1. IOMMU on, in firmware and on the kernel command line:
# intel_iommu=on iommu=pt (or amd_iommu=on)
dmesg | grep -iE 'DMAR|IOMMU' | head
ls /sys/kernel/iommu_groups/ | wc -l # non-zero => IOMMU active
# 2. Create virtual functions on the PF:
cat /sys/class/net/enp3s0f0/device/sriov_totalvfs # hardware maximum
echo 8 | sudo tee /sys/class/net/enp3s0f0/device/sriov_numvfs
lspci | grep 'Virtual Function'
# 3. Assign one to a guest:
sudo modprobe vfio-pci
echo 0000:03:10.0 | sudo tee /sys/bus/pci/devices/0000:03:10.0/driver/unbind
echo 8086 154c | sudo tee /sys/bus/pci/drivers/vfio-pci/new_id
# qemu-system-x86_64 -device vfio-pci,host=03:10.0 ...
# IOMMU GROUPS: the unit of assignment. Everything in a group moves together.
for g in /sys/kernel/iommu_groups/*/devices/*; do
echo "group ${g%%/devices*}: $(lspci -nns ${g##*/})"
done | sort -V | head
# A GPU sharing a group with the host's SATA controller cannot be passed
# through without also handing over the disks -- a common, frustrating blocker.
# ---------- vhost: keep the datapath in the kernel ----------
ps aux | grep '\[vhost' # one kernel thread per virtio queue
# -netdev tap,id=n0,vhost=on (QEMU)
# ---------- Measure ----------
sudo perf kvm stat live | grep -E 'IO_INSTRUCTION|EPT_MISCONFIG|MSR_WRITE'
# High IO_INSTRUCTION = emulated device. Switch to virtio before tuning
# anything else; no amount of queue tuning fixes an emulated E1000.
Key takeaways
- I/O virtualization is harder than CPU or memory virtualization because devices operate asynchronously, perform DMA, and raise interrupts — three separate boundary crossings.
- Full device emulation reproduces well-known legacy hardware so any guest works unmodified, at roughly 10–20% of native throughput and several VM exits per operation.
- Emulated devices remain necessary for booting, since the guest must load a kernel before it has paravirtual drivers.
- virtio reaches 80–95% of native via shared-memory rings and notification suppression, and is the correct default for essentially every workload.
- vhost moves the virtio datapath into the host kernel or a dedicated polling process, removing the userspace hop.
- Direct assignment gives native performance but dedicates the device to one guest and makes live migration very difficult.
- The IOMMU is what makes passthrough safe: without it, a guest programming a device with guest-physical addresses can DMA anywhere in host memory.
- IOMMU groups are the unit of assignment, and a device sharing a group with host-critical hardware cannot be passed through independently.
- SR-IOV lets the device multiplex itself into virtual functions, delivering passthrough performance with sharing, bounded by a fixed VF count and the same migration difficulty.
🧪 Practice
- Count the VM exits for transmitting one packet through an emulated E1000 versus virtio with notification suppression active, and express the difference in cycles at 2,000 cycles per exit.
- Explain precisely what a guest could do to host memory if passthrough were enabled without an IOMMU.
- Run
lspci -nnin a VM you can access, identify each device's virtualization model, and say whether any should be changed.- Interview-style: "A low-latency trading application needs sub-10-microsecond network latency in a VM. What do you configure, and what capability do you give up?" Hint: think about which I/O model removes the hypervisor from the datapath entirely, and what that makes impossible to do with the VM afterwards.
Live Migration
Theory
Live migration moves a running virtual machine from one physical host to another without stopping it — the guest's processes keep executing, its TCP connections stay open, and users notice, at most, a brief pause. It is the capability that turns physical servers into a fungible pool: you can drain a host for maintenance, rebalance load, or evacuate failing hardware without a maintenance window.
The problem is that a VM's state is large and changing. CPU registers and device state are small and can be copied in microseconds. Memory is gigabytes, and the guest keeps writing to it while you copy. Copy memory from the start and by the time you reach the end, the beginning is stale.
Pre-copy migration is the standard solution, and it is an iterative convergence:
- Setup. Reserve resources on the destination.
- Warm-up round 1. Copy all guest memory while the VM keeps running. Enable dirty-page tracking — the hypervisor write-protects guest pages (or uses hardware dirty bits in the EPT) so it learns which pages the guest modifies during the copy.
- Warm-up rounds 2..n. Copy only the pages dirtied during the previous round. Each round transfers less than the last, provided the guest dirties pages more slowly than the network transfers them.
- Stop-and-copy. When the remaining dirty set is small enough — or a round limit is hit — pause the VM, transfer the last pages plus CPU and device state, and resume on the destination. This pause is the downtime, typically 50–300 ms.
- Cleanup. Redirect the network (a gratuitous ARP moves the MAC), release the source.
Two metrics matter and they trade against each other. Downtime is the pause the guest experiences. Total migration time is the whole operation. Stopping earlier gives shorter total time but longer downtime.
Convergence is the failure mode. If the guest dirties memory faster than the link can carry it — a large in-memory database under write load on a 1 GbE link — the dirty set never shrinks and migration never completes. Mitigations: auto-convergence (deliberately throttle the guest's vCPUs to slow the dirty rate), XBZRLE (send deltas against a cached previous version of each page), compression, and multi-threaded or multi-fd transfer.
Post-copy migration inverts the approach: pause briefly, transfer only CPU and device state, and resume immediately on the destination. Memory pages are then fetched on demand when the guest faults on them, while remaining pages are pushed in the background. Downtime is minimal and bounded, and total time is deterministic because every page is sent exactly once. The serious drawback is fault tolerance: after the switch, the VM's memory is split across two hosts, so losing either destroys the VM. Pre-copy can always abort and continue on the source. Production systems often use a hybrid — one pre-copy round, then switch to post-copy — combining bounded downtime with a much reduced failure window.
| Pre-copy | Post-copy | |
|---|---|---|
| Order | Memory first, then switch | Switch first, then memory |
| Downtime | Variable; can be large if convergence fails | Small and bounded |
| Total time | Unbounded if the guest dirties fast | Deterministic; each page sent once |
| Pages transferred | Hot pages sent repeatedly | Each page exactly once |
| Failure during migration | Safe — abort and keep running on the source | Fatal — state is split across two hosts |
| Post-switch performance | Native immediately | Degraded until the working set arrives |
Requirements worth stating explicitly: shared storage or storage migration, compatible CPU features on both hosts (hence "CPU models" that expose a lowest common denominator across a cluster), a shared layer-2 network or an overlay, and no passed-through devices — which is precisely the cost of SR-IOV and passthrough noted in the previous topic.
Example
PRE-COPY: ITERATIVE CONVERGENCE
VM has 16 GiB, link carries 1 GiB/s, guest dirties 200 MiB/s
round 0: copy all 16 GiB 16.0 s [VM RUNNING]
dirtied meanwhile: 3.2 GiB
round 1: copy 3.2 GiB 3.2 s [VM RUNNING]
dirtied meanwhile: 640 MiB
round 2: copy 640 MiB 0.64 s [VM RUNNING]
dirtied meanwhile: 128 MiB
round 3: copy 128 MiB 0.13 s [VM RUNNING]
dirtied meanwhile: 26 MiB
round 4: STOP. copy 26 MiB + CPU + device state
0.026 s [VM PAUSED] <- DOWNTIME
resume on destination; gratuitous ARP redirects the network
total ~20 s, downtime ~26 ms. Converges because each round's dirty
set is 20% of the last (dirty rate / bandwidth = 0.2).
WHEN IT DOES NOT CONVERGE
same VM, but a write-heavy database dirties 1.2 GiB/s on a 1 GiB/s link
round 0: copy 16 GiB -> dirtied 19.2 GiB <- MORE than was copied
round 1: copy 19.2 GiB -> dirtied 23.0 GiB
round 2: copy 23.0 GiB -> dirtied 27.6 GiB
...the dirty set GROWS. Migration never finishes.
ratio = dirty_rate / bandwidth
< 1 -> converges (geometrically)
>= 1 -> never converges
Fixes, in order of preference:
faster link (10/25 GbE) -- lowers the ratio directly
XBZRLE delta compression -- fewer bytes per dirty page
multifd (parallel streams) -- more effective bandwidth
auto-convergence (throttle vCPU) -- lowers the numerator by
deliberately slowing the guest
switch to post-copy -- sidestep convergence entirely
POST-COPY: SWITCH FIRST, FETCH ON DEMAND
t0 pause VM, send CPU + device state only (a few MiB) -- ~10 ms
t1 RESUME on destination immediately
t2 guest touches a page that has not arrived
-> page fault -> request from source -> stall ~1 ms -> continue
meanwhile, background push sends the rest
t3 all pages present; native performance
downtime : ~10 ms, BOUNDED regardless of dirty rate
total time: deterministic -- every page crosses exactly once
RISK : between t1 and t3 the VM's memory lives on TWO hosts.
Lose either one and the VM is unrecoverable.
Pre-copy can always abort and keep running on the source.
HYBRID (what production systems actually do)
one pre-copy round (moves the cold bulk, safely)
then post-copy (bounded downtime, no convergence problem)
-> the dangerous window shrinks to the residual working set
# Will this migration converge, and what will it cost? The model below is
# the one to reason with before starting a migration in production.
def simulate_precopy(vm_gib, bandwidth_gibps, dirty_rate_gibps,
max_rounds=30, downtime_target_ms=300):
remaining = vm_gib
total_time = 0.0
rounds = []
for r in range(max_rounds):
transfer_time = remaining / bandwidth_gibps
total_time += transfer_time
# Pages dirtied WHILE that round was in flight become the next round.
newly_dirty = dirty_rate_gibps * transfer_time
rounds.append((r, remaining, transfer_time, newly_dirty))
# Stop when the final pause would meet the downtime target.
if (newly_dirty / bandwidth_gibps) * 1000 <= downtime_target_ms:
downtime_ms = (newly_dirty / bandwidth_gibps) * 1000
return {"converged": True, "rounds": rounds,
"total_s": total_time + newly_dirty / bandwidth_gibps,
"downtime_ms": downtime_ms}
if newly_dirty >= remaining: # the dirty set is not shrinking
return {"converged": False, "rounds": rounds,
"reason": "dirty rate >= bandwidth"}
remaining = newly_dirty
return {"converged": False, "rounds": rounds, "reason": "round limit"}
scenarios = [
("idle web server", 16, 1.0, 0.05),
("busy app server", 16, 1.0, 0.20),
("write-heavy DB", 16, 1.0, 1.20), # dirties faster than the link
("same DB, 25 GbE", 16, 3.0, 1.20), # the fix: raise the bandwidth
]
for name, size, bw, dirty in scenarios:
r = simulate_precopy(size, bw, dirty)
ratio = dirty / bw
if r["converged"]:
print(f"{name:<18} ratio {ratio:4.2f} CONVERGED in {len(r['rounds'])} rounds, "
f"{r['total_s']:5.1f}s total, {r['downtime_ms']:6.1f} ms downtime")
else:
print(f"{name:<18} ratio {ratio:4.2f} FAILED: {r['reason']}")
# idle web server ratio 0.05 CONVERGED in 2 rounds, 16.8s total, 40.0 ms downtime
# busy app server ratio 0.20 CONVERGED in 4 rounds, 20.1s total, 128.0 ms downtime
# write-heavy DB ratio 1.20 FAILED: dirty rate >= bandwidth
# same DB, 25 GbE ratio 0.40 CONVERGED in 5 rounds, 9.0s total, 145.0 ms downtime
#
# The single governing number is dirty_rate / bandwidth. Below 1 it converges
# geometrically; at or above 1 no amount of patience helps.
# ---------- Perform and observe a live migration ----------
virsh migrate --live --verbose vm1 qemu+ssh://host2/system
virsh domjobinfo vm1 # progress, while it runs
# Job type: Unbounded
# Data processed: 4.210 GiB
# Data remaining: 1.021 GiB
# Memory iteration: 3 <- which pre-copy round
# Dirty rate: 182 MiB/s <- compare against your link!
# Expected downtime: 84 ms
# Set the acceptable pause explicitly:
virsh migrate-setmaxdowntime vm1 200 # milliseconds
# ---------- When it will not converge ----------
# Throttle the guest so it dirties more slowly (costs guest performance):
virsh migrate --live --auto-converge --auto-converge-initial 20 \
--auto-converge-increment 10 vm1 qemu+ssh://host2/system
# Send deltas instead of whole pages for repeatedly-dirtied pages:
virsh migrate --live --comp-xbzrle vm1 qemu+ssh://host2/system
# Parallel streams: use the whole link.
virsh migrate --live --parallel --parallel-connections 4 vm1 ...
# Post-copy: bounded downtime, at the cost of a split-state window.
virsh migrate --live --postcopy --postcopy-after-precopy vm1 ...
# ^ hybrid: one pre-copy round first, then switch
# ---------- Why a migration may be refused ----------
virsh capabilities | grep -A5 '<cpu>' # destination must support the CPU model
# Use a cluster-wide baseline model so any host can accept any VM:
# <cpu mode='custom'><model>Haswell-noTSX</model></cpu>
# NOT <cpu mode='host-passthrough'/>, which pins the VM to identical hardware.
virsh dumpxml vm1 | grep hostdev # any passed-through device blocks it
# SR-IOV/passthrough VMs cannot migrate: the guest holds opaque device state.
Key takeaways
- Live migration moves a running VM between hosts with only a brief pause, which is what makes physical servers a fungible pool for maintenance, rebalancing, and evacuation.
- The difficulty is that memory is large and the guest keeps writing to it during the copy.
- Pre-copy transfers memory in rounds, copying only what was dirtied during the previous round, then pauses briefly for a final stop-and-copy.
- Convergence is governed by dirty rate divided by bandwidth: below 1 it converges geometrically, at or above 1 it never completes.
- Non-convergence is fixed by raising bandwidth, delta compression (XBZRLE), parallel streams, or auto-convergence, which deliberately throttles the guest.
- Post-copy switches first and fetches pages on demand, giving bounded downtime and deterministic total time, with each page sent exactly once.
- Post-copy's risk is that memory is split across two hosts until it completes, so a failure destroys the VM; pre-copy can always abort safely on the source.
- Hybrid migration runs one pre-copy round then switches to post-copy, minimising both downtime and the dangerous window.
- Migration requires compatible CPU models across the cluster and no passed-through devices — which is the real cost of choosing SR-IOV or passthrough.
🧪 Practice
- A 32 GiB VM dirties 400 MiB/s and the migration link carries 2 GiB/s. Compute the number of pre-copy rounds and the resulting downtime.
- Explain why
host-passthroughCPU mode improves guest performance but can prevent migration, and what you would configure instead in a mixed cluster.- Describe a workload for which post-copy is clearly the right choice, and one for which its risk is unacceptable.
- Interview-style: "A migration has been running for 40 minutes on a 16 GiB VM and shows no sign of finishing. What is happening and what are your options?" Hint: look at the reported dirty rate next to the link bandwidth, and consider both raising one and lowering the other.
<a id="133-containers-and-lightweight-isolation"></a>
13.3 Containers and Lightweight Isolation
Virtual machines simulate hardware; containers do something cheaper and quite different — they reuse the kernel's own isolation primitives to give a group of processes a private view of the system. This section covers the two kernel features that make that possible, the runtime stack built on them, an honest comparison with VMs, and the radical alternative at the other end of the spectrum.
Namespaces
Theory
A virtual machine isolates by duplicating hardware. A container isolates by something far cheaper: it changes what a process can see.
Consider what actually makes two processes feel like they are on the same machine. They see the same process IDs, the same mount table, the same network interfaces, the same hostname, the same user IDs. Take all of those away — give a process its own PID numbering starting at 1, its own filesystem tree, its own eth0, its own hostname, its own view of who is root — and it is, for every purpose it can detect, alone on its own machine. Nothing was virtualized. The kernel simply stopped showing it everything.
That is a namespace: a kernel mechanism that wraps a global resource so processes inside the namespace see their own instance of it. Linux provides eight:
| Namespace | Isolates | Consequence inside |
|---|---|---|
| PID | Process IDs | Its first process is PID 1; it cannot see or signal outside processes |
Mount (mnt) | Filesystem mount table | Its own root filesystem and mounts |
Network (net) | Interfaces, routes, firewall rules, ports | Its own eth0, its own port 80, its own iptables |
| UTS | Hostname, domain name | hostname returns the container's name |
| IPC | System V IPC, POSIX message queues | Cannot reach another container's shared memory |
| User | UID/GID mappings | Root inside can map to an unprivileged UID outside |
| Cgroup | Cgroup root directory | Cannot see the host's resource hierarchy |
| Time | Boot time and monotonic clock offsets | Its own uptime (useful for checkpoint/restore) |
Three system calls manage them. clone() creates a process in new namespaces (CLONE_NEWPID, CLONE_NEWNET, …). unshare() moves the calling process into new namespaces. setns() joins an existing namespace — which is exactly what docker exec and nsenter do.
The user namespace deserves special attention, because it is what makes unprivileged containers possible. It maps UIDs: UID 0 inside the namespace can correspond to UID 100000 outside. A process can be fully root inside its container — able to mount filesystems, configure its own network, install packages — while the host kernel treats it as an ordinary unprivileged user. If it escapes, it escapes as nobody in particular. This is the single most valuable container hardening measure, and the reason rootless Podman exists.
Namespaces are also independent, which is a genuinely important design property. A container is not an object the kernel knows about; it is a combination chosen by userspace. You can share some namespaces and not others, and real systems do:
- A Kubernetes pod shares network and IPC namespaces between its containers, so they reach each other on
localhost, but keeps mount namespaces separate. docker run --network hostgives a container its own PID and mount namespaces but the host's network stack.- A debug sidecar shares the target's PID namespace so it can see and inspect its processes.
There is no struct container in the Linux kernel. "Container" is a userspace convention for a process running with a particular set of namespaces, cgroups, capabilities, and a seccomp filter. Understanding that is what makes container security reasoning possible: you are not asking "is the container secure", you are asking "which of these mechanisms is applied, and what does each one actually cover".
Example
NAMESPACES: A PRIVATE VIEW, NOT A PRIVATE MACHINE
HOST VIEW CONTAINER'S VIEW (PID ns)
PID 1 systemd PID 1 nginx <- same process!
PID 842 sshd PID 7 worker
PID 1503 dockerd (nothing else exists)
PID 1876 nginx <---------------------+
PID 1882 worker <---------------------+
One process, two PIDs. The kernel is running it exactly once; the
NUMBER it reports depends on who is asking. Nothing was simulated.
THE EIGHT NAMESPACES, AND WHAT EACH HIDES
+-- PID -> its own process tree, its own PID 1
+-- mount -> its own root filesystem and mount table
+-- network -> its own interfaces, routes, ports, firewall rules
+-- UTS -> its own hostname
+-- IPC -> its own shared memory and message queues
+-- user -> its own UID map: root inside != root outside
+-- cgroup -> its own view of the resource hierarchy
+-- time -> its own boot time and uptime
A "container" is just a process with a chosen COMBINATION of these.
The kernel has no container object at all.
USER NAMESPACES: ROOT THAT IS NOT ROOT
inside the container host kernel's view
uid 0 (root) <----> uid 100000 (unprivileged)
uid 1 (daemon) <----> uid 100001
uid 1000 (app) <----> uid 101000
The process can apt-get install, mount, and bind port 80 inside.
If it escapes, it is uid 100000 on the host: it owns nothing,
can signal nothing, can read no other user's files.
This is the highest-value single hardening measure for containers.
NAMESPACES ARE INDEPENDENT -- SHARE THEM SELECTIVELY
Kubernetes POD:
+------------------------------------------+
| shared: network ns, IPC ns |
| +----------------+ +----------------+ |
| | container A | | container B | |
| | own mount ns | | own mount ns | |
| | own PID ns | | own PID ns | |
| +----------------+ +----------------+ |
| A reaches B at 127.0.0.1 -- same netns |
+------------------------------------------+
docker run --network host : own PID+mount, HOST network
docker run --pid container:X : share X's process view (debug sidecars)
nsenter -t <pid> -n -m : join an existing container's ns from outside
# ---------- Build a container by hand, one namespace at a time ----------
# This is what a runtime does; doing it manually shows there is no magic.
# 1. UTS namespace: an isolated hostname.
sudo unshare --uts sh -c 'hostname isolated; hostname; ls /'
hostname # unchanged on the host
# 2. PID namespace: a fresh process tree. --fork and --mount-proc are
# required because PID 1 semantics apply to the CHILD, and /proc must
# be remounted to reflect the new numbering.
sudo unshare --pid --fork --mount-proc sh -c 'ps aux'
# USER PID COMMAND
# root 1 sh <- our shell is PID 1
# root 5 ps
# The host's hundreds of processes are simply not visible.
# 3. Network namespace: its own, empty network stack.
sudo unshare --net sh -c 'ip link show'
# 1: lo: <LOOPBACK> state DOWN <- only loopback, and it is down
# 4. Mount namespace: mounts here do not affect the host.
sudo unshare --mount sh -c 'mount -t tmpfs none /mnt && df -h /mnt'
df -h /mnt # host: unaffected
# 5. User namespace: root inside, nobody outside. Note: NO sudo.
unshare --user --map-root-user sh -c 'id; cat /proc/self/uid_map'
# uid=0(root) gid=0(root)
# 0 1000 1 <- inside 0 == outside 1000
# 6. All at once -- this is essentially `docker run`:
sudo unshare --pid --fork --mount-proc --uts --ipc --net --mount \
sh -c 'hostname mybox; ps aux; ip link'
# ---------- Inspect the namespaces of a running container ----------
CPID=$(docker inspect -f '{{.State.Pid}}' mycontainer)
sudo ls -l /proc/$CPID/ns/
# net -> 'net:[4026532281]' <- the inode IS the namespace identity
# pid -> 'pid:[4026532283]'
# uts -> 'uts:[4026532279]'
ls -l /proc/1/ns/net # different inode => different namespace
# Matching inodes mean the namespace is SHARED (e.g. within a pod).
lsns # every namespace on the system, with owners
lsns -t net # just network namespaces
# ---------- Enter one from outside (this is `docker exec`) ----------
sudo nsenter -t $CPID -n ip addr # join only the network namespace
sudo nsenter -t $CPID -a /bin/sh # join all of them
# ---------- Rootless containers: user namespaces in production ----------
podman run --rm -it alpine id # no daemon, no root on the host
cat /etc/subuid # the UID ranges delegated to you
# youruser:100000:65536
/* What a container runtime actually calls. The whole mechanism is one
* clone() with the right flags -- everything else is setup. */
#define _GNU_SOURCE
#include <sched.h>
#include <sys/mount.h>
#include <sys/wait.h>
#include <unistd.h>
#include <stdio.h>
static char child_stack[1024 * 1024];
static int child_main(void *arg)
{
/* We are PID 1 inside the new PID namespace. */
sethostname("container", 9); /* UTS ns: ours alone */
/* Mount ns: give this process its own root. pivot_root is what a real
* runtime uses; chroot is shown here for brevity and is weaker. */
mount("none", "/", NULL, MS_REC | MS_PRIVATE, NULL); /* do not leak
* mounts back */
chroot("/var/lib/containers/rootfs");
chdir("/");
mount("proc", "/proc", "proc", 0, NULL); /* /proc must reflect the
* NEW pid namespace */
printf("inside: pid=%d\n", getpid()); /* prints 1 */
execlp("/bin/sh", "sh", NULL);
return 1;
}
int main(void)
{
int flags = CLONE_NEWPID /* own process tree */
| CLONE_NEWNS /* own mount table */
| CLONE_NEWNET /* own network stack */
| CLONE_NEWUTS /* own hostname */
| CLONE_NEWIPC /* own SysV IPC */
| CLONE_NEWUSER /* own UID map: root != root */
| CLONE_NEWCGROUP /* own cgroup root */
| SIGCHLD;
pid_t pid = clone(child_main, child_stack + sizeof(child_stack), flags, NULL);
printf("outside: child is pid=%d\n", pid); /* a normal host PID */
/* With CLONE_NEWUSER the parent must write the UID/GID maps before the
* child can use its privileges -- this is what makes root-inside safe. */
/* write "0 100000 65536" to /proc/<pid>/uid_map and gid_map */
waitpid(pid, NULL, 0);
return 0;
}
Key takeaways
- A namespace isolates by changing what a process can see, not by duplicating hardware, which is why containers cost almost nothing to create.
- Linux provides eight namespaces — PID, mount, network, UTS, IPC, user, cgroup, and time — each wrapping one global resource.
clone()creates a process in new namespaces,unshare()moves the caller into new ones, andsetns()joins existing ones (whatdocker execandnsenterdo).- The user namespace maps UIDs so a process can be root inside while being unprivileged on the host; it is the highest-value container hardening measure.
- Namespaces are independent and can be shared selectively, which is how a Kubernetes pod gives its containers a shared network but separate filesystems.
- There is no container object in the Linux kernel — a container is a userspace convention combining namespaces, cgroups, capabilities, and seccomp.
- Because the combination is chosen, container security must be assessed mechanism by mechanism rather than as a single property.
lsnsand/proc/<pid>/ns/expose namespace membership directly, and identical inode numbers mean two processes share a namespace.
🧪 Practice
- Run
sudo unshare --pid --fork --mount-proc sh -c 'ps aux'and explain why both--forkand--mount-procare required.- Compare
ls -l /proc/1/ns/with the same listing for a container's PID and explain what matching or differing inode numbers indicate.- Create two network namespaces with
ip netns add, connect them with a veth pair, and demonstrate that they can ping each other but the host cannot see their addresses.- Interview-style: "A container runs as root. Is that dangerous?" Hint: the answer depends on one namespace in particular; ask what UID the host kernel actually sees, and what changes if the answer is 0.
Control Groups
Theory
Namespaces control visibility; they say nothing about consumption. A process in a perfectly isolated set of namespaces can still allocate all the host's memory, spin every core, and saturate the disk — taking down every other container on the machine. This is the noisy neighbour problem, and control groups (cgroups) are the answer.
A cgroup is a hierarchical grouping of processes with resource controllers attached. Each controller accounts for and limits one resource type:
| Controller | Limits | Key knobs (cgroup v2) |
|---|---|---|
cpu | CPU time | cpu.max (quota/period), cpu.weight (relative share) |
memory | RAM and swap | memory.max (hard), memory.high (throttle), memory.low (protect) |
io | Block I/O | io.max (IOPS and bandwidth caps), io.weight |
pids | Process count | pids.max — the fork-bomb defence |
cpuset | Which cores and NUMA nodes | cpuset.cpus, cpuset.mems |
hugetlb | Huge page usage | hugetlb.*.max |
Two distinctions matter more than the individual knobs.
Limits versus weights. A limit (cpu.max = 50000 100000, i.e. 0.5 CPU) is absolute: the cgroup is throttled at that ceiling even if the machine is idle. A weight (cpu.weight = 100) is proportional: it only takes effect under contention, and an idle machine lets the cgroup use everything. Weights give better utilisation; limits give predictability and are what you need to prevent one tenant affecting another's latency. Kubernetes exposes exactly this pair as requests (weights and scheduling) and limits (hard ceilings).
CPU versus memory enforcement. This asymmetry catches people constantly. Exceeding a CPU limit means being throttled — the cgroup is descheduled until its next quota period, so it just runs slower. Exceeding a memory limit means the OOM killer terminates a process in the cgroup, because memory cannot be time-sliced. A container that exceeds CPU is slow; a container that exceeds memory is dead. This is why a Kubernetes pod shows OOMKilled with exit code 137 (128 + SIGKILL) rather than simply slowing down.
CPU throttling has a notorious subtlety. The quota is enforced per period (100 ms by default), and a multi-threaded application can exhaust its entire quota in the first few milliseconds of a period and then sit idle for the remaining 90+ ms. The result is that an application shows severe latency spikes while reporting average CPU usage well below its limit. Diagnosing this means reading nr_throttled and throttled_time in cpu.stat, not average utilisation.
cgroup v2 replaced v1's separate per-controller hierarchies with a single unified tree, which fixed genuine coordination problems — v1 could not sensibly account for page cache written by one cgroup on behalf of another, because memory and I/O lived in different hierarchies. v2 also adds pressure stall information (PSI), which reports the fraction of time tasks were stalled waiting for CPU, memory, or I/O. PSI is a far better saturation signal than utilisation: 100% CPU utilisation with zero pressure means the work fits, while 70% utilisation with high pressure means tasks are waiting.
Example
NAMESPACES AND CGROUPS ARE ORTHOGONAL, AND YOU NEED BOTH
namespaces only : perfect isolation of VIEW,
zero isolation of CONSUMPTION
-> one container can starve every other
cgroups only : resource limits, but everything is visible
-> processes can see and signal each other
namespaces + cgroups : a container
"What can I see?" -> namespaces
"How much can I use?" -> cgroups
CGROUP v2: ONE UNIFIED TREE
/sys/fs/cgroup/ cpu.max = max (unlimited)
|
+-- system.slice/ cpu.weight = 100
| +-- sshd.service/
| +-- docker.service/
|
+-- kubepods.slice/ memory.max = 48G
+-- burstable/
| +-- pod-a/ cpu.max = 200000 100000 (2 CPUs)
| +-- container-1/ memory.max = 2G
+-- guaranteed/
+-- pod-b/ cpuset.cpus = 8-11 (pinned)
Limits are HIERARCHICAL: a child can never exceed its ancestors'.
LIMIT vs WEIGHT -- the distinction that governs utilisation
cpu.max = 50000 100000 (a LIMIT: 50 ms of every 100 ms = 0.5 CPU)
idle machine -> still capped at 0.5 CPU. Predictable; wasteful.
busy machine -> capped at 0.5 CPU. Neighbours are protected.
cpu.weight = 100 (a WEIGHT: a relative share)
idle machine -> use ALL the CPUs. Nothing is wasted.
busy machine -> proportional share against other weights.
Kubernetes: requests -> weights; limits -> ceilings.
THE ENFORCEMENT ASYMMETRY (the one that surprises people)
exceed CPU limit -> THROTTLED. The container runs slower. It lives.
exceed memory limit -> OOM KILLED. A process is terminated. Exit 137.
Memory cannot be time-sliced: you either have the page or you do not.
CPU THROTTLING: SLOW DESPITE LOW AVERAGE UTILISATION
cpu.max = 100000 100000 (1 CPU), application has 8 threads
period 1 (100 ms):
|XXXXXXXXXX|.....................................................|
|<- 12.5ms ->|<------------ 87.5 ms THROTTLED, doing nothing ---->|
8 threads burn the 100 ms quota in 12.5 ms of wall time
average utilisation over the period : 12.5% <- looks fine!
p99 latency : catastrophic
cpu.stat: nr_throttled rising <- THE signal to look at
Fixes: raise the quota, reduce thread count to match the quota,
or shorten cpu.cfs_period_us for finer-grained scheduling.
# ---------- Which cgroup version is this system on? ----------
stat -fc %T /sys/fs/cgroup/ # cgroup2fs = v2 (unified), tmpfs = v1
cat /proc/self/cgroup # "0::/path" means v2
# ---------- Create a cgroup and constrain something ----------
sudo mkdir /sys/fs/cgroup/demo
# CPU: half a core. Format is "QUOTA PERIOD", both in microseconds.
echo "50000 100000" | sudo tee /sys/fs/cgroup/demo/cpu.max
# Memory: a hard ceiling (OOM kill) and a soft one (reclaim/throttle first).
echo "512M" | sudo tee /sys/fs/cgroup/demo/memory.max
echo "400M" | sudo tee /sys/fs/cgroup/demo/memory.high # throttle BEFORE killing
# Fork-bomb defence -- cheap, and almost always worth setting.
echo "100" | sudo tee /sys/fs/cgroup/demo/pids.max
# I/O: cap a specific device. 8:0 is the major:minor from lsblk.
echo "8:0 rbps=10485760 wiops=500" | sudo tee /sys/fs/cgroup/demo/io.max
# Move a shell into it; everything it spawns inherits the limits.
echo $$ | sudo tee /sys/fs/cgroup/demo/cgroup.procs
stress-ng --cpu 4 --timeout 10s # 4 threads share 0.5 CPU
# ---------- The two files worth reading ----------
cat /sys/fs/cgroup/demo/cpu.stat
# usage_usec 5012843
# nr_periods 100
# nr_throttled 87 <- throttled in 87 of 100 periods
# throttled_usec 4310000 <- THIS is why p99 latency is bad, even
# though average utilisation looks low
cat /sys/fs/cgroup/demo/memory.current # actual usage right now
cat /sys/fs/cgroup/demo/memory.events
# high 1420 <- hit memory.high: reclaim pressure applied
# max 3 <- hit memory.max
# oom_kill 1 <- something was killed. Exit code 137 in the container.
# ---------- PSI: saturation, not utilisation ----------
cat /sys/fs/cgroup/demo/cpu.pressure
# some avg10=41.23 avg60=38.10 avg300=22.4 total=98234211
# ^ 41% of the last 10s, at least one task was STALLED waiting for CPU.
# 100% utilisation with 0% pressure = healthy and fully used.
# 70% utilisation with 40% pressure = saturated; tasks are queuing.
cat /sys/fs/cgroup/demo/memory.pressure
cat /sys/fs/cgroup/demo/io.pressure
# ---------- The same thing through higher-level tools ----------
docker run --cpus=0.5 --memory=512m --pids-limit=100 nginx
systemd-run --scope -p CPUQuota=50% -p MemoryMax=512M -p TasksMax=100 ./app
systemd-cgtop # live per-cgroup resource usage
# Kubernetes exposes exactly the weight/limit distinction, and the choice
# determines the pod's Quality of Service class.
apiVersion: v1
kind: Pod
metadata:
name: web
spec:
containers:
- name: app
image: nginx
resources:
requests: # -> cpu.weight and scheduling placement.
cpu: "500m" # The scheduler reserves this; under
memory: "256Mi" # contention this is the proportional share.
limits: # -> cpu.max and memory.max: hard ceilings.
cpu: "1000m" # Exceeding CPU -> THROTTLED (slow).
memory: "512Mi" # Exceeding RAM -> OOMKilled (exit 137).
# QoS classes, derived from the above:
# Guaranteed requests == limits for every resource -> evicted last
# Burstable requests < limits -> evicted second
# BestEffort neither set -> evicted first
#
# A common production pattern: set memory requests == limits (so the pod is
# never OOM-killed by overcommit) but leave the CPU limit OFF, so the pod can
# burst into idle capacity instead of being throttled at an arbitrary ceiling.
Key takeaways
- Namespaces isolate what a process can see; cgroups limit what it can consume, and a container needs both.
- Controllers exist for CPU, memory, I/O, process count, CPU/NUMA placement, and huge pages, arranged in a hierarchy where a child can never exceed its ancestors.
- Limits are absolute ceilings enforced even on an idle machine; weights are proportional shares that apply only under contention, trading predictability against utilisation.
- Exceeding a CPU limit throttles the container; exceeding a memory limit invokes the OOM killer, because memory cannot be time-sliced.
- CPU quota is enforced per 100 ms period, so a multi-threaded process can exhaust it in milliseconds and stall for the rest — producing latency spikes at low average utilisation.
cpu.stat'snr_throttledandthrottled_usecare the diagnostic for that pathology; average utilisation will not reveal it.- cgroup v2's unified hierarchy fixed v1's inability to coordinate memory and I/O accounting for the same pages.
- Pressure stall information (PSI) reports stall time rather than utilisation, making it a much better saturation signal.
pids.maxis a cheap and effective fork-bomb defence that is frequently left unset.
🧪 Practice
- Set
cpu.maxto50000 100000for a cgroup, run a 4-thread CPU load in it, and explain thenr_throttledvalue you observe.- Explain why a container exceeding its CPU limit survives while one exceeding its memory limit is killed, and what exit code 137 signifies.
- Compare
cpu.pressureand average CPU utilisation for a workload of your choice, and explain which one better predicts user-visible latency.- Interview-style: "A service is reported as slow, but its CPU utilisation averages 30% against a 1-core limit and it has eight worker threads. What do you check?" Hint: average utilisation is computed over a window much longer than the quota period; ask what happens within a single 100 ms slice.
Container Runtimes
Theory
Namespaces and cgroups are kernel mechanisms; a container runtime is the software that assembles them into something usable, together with the other half of the story — the filesystem image.
The stack is layered, and the layering is the result of a deliberate standardisation effort:
- Low-level runtime (
runc,crun,youki). Takes a directory containing a root filesystem plus a JSONconfig.json, and creates the container:clone()with namespace flags, set up cgroups, apply seccomp and capabilities,pivot_root, andexecthe entrypoint. It does this once and exits. This behaviour is specified by the OCI Runtime Specification, which is whyruncandcrunare interchangeable. - High-level runtime (
containerd,CRI-O). Manages image pulls, storage, networking, and container lifecycle, then calls a low-level runtime to do the actual creation. Kubernetes talks to this layer through the CRI (Container Runtime Interface). - Engine / platform (Docker, Podman, Kubernetes). User-facing tooling: builds, registries, compose files, orchestration.
The image side is equally important and is where containers get their most valuable property. An OCI image is a stack of read-only layers, each a tarball of filesystem changes, plus a manifest and a config. A union filesystem — normally overlayfs — presents the stack as a single tree, with a thin writable layer on top for the running container. Writes to an existing file trigger copy-up: the file is copied into the writable layer and modified there, leaving the underlying layer untouched.
This design produces the properties that made containers spread:
- Sharing. A hundred containers from the same image share one copy of its layers on disk and in page cache. Starting the hundredth costs only its writable layer.
- Content addressing. Layers are identified by SHA-256 digest, so pulls transfer only missing layers and identity is verifiable.
- Immutability and reproducibility. An image digest names exactly one bit-identical filesystem, which is what makes "the same artifact in dev and prod" a real guarantee rather than an aspiration.
The layering also explains a security rule that is otherwise mysterious: deleting a secret in a later layer does not remove it. A RUN rm /tmp/credentials creates a whiteout entry in a new layer; the file remains intact in the earlier layer and anyone who pulls the image can extract it. Secrets must never enter an image layer in the first place.
Runtime architecture has moved in two directions worth knowing. Daemonless runtimes (Podman) drop the central root daemon — containers become direct children of the invoking user's shell, managed by systemd, and combined with user namespaces this gives fully rootless containers. And sandboxed runtimes replace runc under the same OCI interface to strengthen the boundary: gVisor interposes a userspace kernel that implements syscalls itself, and Kata Containers runs each container inside a lightweight VM. Both plug in where runc was, so the orchestration layer above is unchanged.
Example
THE STACK, AND WHERE THE STANDARDS SIT
kubectl / docker CLI / podman CLI
|
+----v-------------------------------+
| Kubernetes / Docker Engine | orchestration, build, registry
+----|-------------------------------+
| CRI (a standard interface)
+----v-------------------------------+
| containerd / CRI-O | images, storage, networking,
+----|-------------------------------+ lifecycle
| OCI Runtime Spec (config.json + rootfs)
+----v-------------------------------+
| runc / crun / gVisor / Kata | actually creates the container
+----|-------------------------------+
| clone(), cgroups, seccomp, capabilities, pivot_root
+----v-------------------------------+
| LINUX KERNEL |
+------------------------------------+
Because the OCI interface is standard, swapping runc for gVisor or Kata
changes the ISOLATION STRENGTH without changing anything above it.
OVERLAYFS: WHY THE 100th CONTAINER IS FREE
container's view (merged)
+---------------------------------------+
| /app/config.yaml (modified) |
| /app/server |
| /usr/lib/libc.so |
+---------------------------------------+
^ overlayfs merges top-down
+---------------------------------------+
| UPPER (writable, per container) | <- only this is per-container
| /app/config.yaml [copied up] | typically a few MiB
+---------------------------------------+
| layer 3: COPY app/ /app | \
+---------------------------------------+ | read-only, SHARED by
| layer 2: RUN apt-get install ... | | every container from
+---------------------------------------+ | this image -- on disk
| layer 1: FROM debian:12 | / AND in page cache
+---------------------------------------+
100 containers from a 1 GiB image: ~1 GiB total, not 100 GiB.
Start time is milliseconds because there is nothing to copy.
COPY-UP: the first write to a lower-layer file copies the WHOLE file
into the upper layer. Writing one byte of a 2 GiB file copies 2 GiB --
which is why databases use volumes, not the container filesystem.
THE LAYER SECURITY TRAP
Dockerfile:
RUN curl -H "Authorization: Bearer $TOKEN" ... && rm -f /root/.netrc
^^^^^^^^^^^^^^^^^
If the rm is in a LATER layer, the file is still in the earlier one:
layer 4: whiteout entry for /root/.netrc <- "deleted"
layer 3: /root/.netrc (the actual secret) <- STILL THERE, extractable
docker save img | tar -x && find . -name '*.tar' -exec tar -tf {} \;
A secret that ever existed in a layer is IN THE IMAGE, permanently.
Use BuildKit secret mounts (--mount=type=secret), which are never
committed to a layer.
THE THREE RUNTIME STRENGTHS, SAME OCI INTERFACE
runc : namespaces + cgroups + seccomp. Shared kernel.
syscall -> host kernel directly. fastest, weakest
gVisor : a userspace kernel (Sentry) implements ~200 syscalls itself.
syscall -> Sentry -> (a few) host syscalls
smaller kernel attack surface; ~10-30% syscall overhead
Kata : each container in a microVM with its OWN kernel.
syscall -> guest kernel -> (rarely) hypervisor
hardware-enforced boundary; ~100 ms start, ~50 MiB overhead
# ---------- The layers are just tarballs; look inside one ----------
docker history nginx:latest --no-trunc --format '{{.Size}}\t{{.CreatedBy}}'
docker save nginx:latest -o /tmp/nginx.tar && mkdir -p /tmp/x && tar -xf /tmp/nginx.tar -C /tmp/x
cat /tmp/x/manifest.json | jq '.[0].Layers' # each layer, by digest
# Every layer is extractable by anyone who can pull the image. There is no
# such thing as a "deleted" file in an earlier layer.
# Where overlayfs mounted them for a running container:
docker inspect nginx | jq '.[0].GraphDriver.Data'
# LowerDir: /var/lib/docker/overlay2/<a>/diff:/var/lib/docker/overlay2/<b>/diff
# UpperDir: /var/lib/docker/overlay2/<c>/diff <- the only per-container part
# MergedDir: /var/lib/docker/overlay2/<c>/merged <- what the container sees
mount | grep overlay
# ---------- Drive the low-level runtime directly ----------
mkdir -p /tmp/ctr/rootfs && cd /tmp/ctr
docker export $(docker create alpine) | tar -C rootfs -xf -
runc spec # generates an OCI config.json
jq '.process.args = ["/bin/sh","-c","hostname; id; ps aux"]' config.json > c.json
mv c.json config.json
sudo runc run demo # creates the container and exits
sudo runc list # OCI state, no daemon involved
# The config.json IS the container definition -- namespaces, cgroups,
# capabilities and seccomp, all declared:
jq '{namespaces: .linux.namespaces,
capabilities: .process.capabilities.bounding,
seccomp: .linux.seccomp.defaultAction}' config.json
# ---------- Swap the runtime to change the isolation strength ----------
docker run --runtime=runsc alpine uname -a # gVisor
docker run --runtime=kata-runtime alpine uname -r # Kata: a DIFFERENT kernel
# Compare with the host's `uname -r`. Under runc they match, because the
# kernel IS the host's. Under Kata they differ -- that is the whole point.
# ---------- Hardening a container with runc's own mechanisms ----------
docker run \
--user 10000:10000 \ # never root, even namespaced
--read-only \ # immutable root filesystem
--tmpfs /tmp:rw,noexec,nosuid \
--cap-drop=ALL \ # drop every capability...
--cap-add=NET_BIND_SERVICE \ # ...then add back only what is needed
--security-opt=no-new-privileges \ # setuid binaries cannot escalate
--security-opt seccomp=profile.json \
--pids-limit=100 \
myimage
# Each flag closes one escape route. None of them is implied by "container".
# ---------- Keep secrets out of layers ----------
# BuildKit: the secret is mounted for one RUN and never committed.
# # syntax=docker/dockerfile:1
# RUN --mount=type=secret,id=token \
# curl -H "Authorization: Bearer $(cat /run/secrets/token)" ...
# DOCKER_BUILDKIT=1 docker build --secret id=token,src=./token .
Key takeaways
- The runtime stack is layered: an engine or orchestrator calls a high-level runtime (containerd, CRI-O), which calls a low-level runtime (runc, crun) that performs the actual kernel calls.
- The OCI Runtime Specification standardises the low-level interface — a rootfs plus
config.json— which is why runtimes are interchangeable. - An OCI image is a stack of content-addressed read-only layers presented as one tree by a union filesystem, usually overlayfs, with a thin per-container writable layer on top.
- Layer sharing is why a hundred containers from one image cost roughly one image on disk and in page cache, and why container start time is measured in milliseconds.
- Copy-up means the first write to a file from a lower layer copies the entire file, which is why write-heavy data belongs in volumes rather than the container filesystem.
- Deleting a file in a later layer only adds a whiteout; the original remains extractable, so secrets must never enter a layer at all.
- Daemonless runtimes such as Podman make containers ordinary child processes and, with user namespaces, allow fully rootless operation.
- gVisor and Kata plug into the same OCI interface to provide a userspace kernel or a per-container microVM, strengthening isolation without changing the layers above.
- Container hardening is a set of explicit choices — non-root user, read-only root, dropped capabilities, seccomp,
no-new-privileges, pids limit — none of which is implied by using a container.
🧪 Practice
- Run
docker historyon any image and explain which instructions created layers and why their order affects rebuild speed.- Extract an image with
docker saveand locate a file that a later layer deleted, then state the correct way to handle build-time secrets.- Use
runc specto generate aconfig.json, modify the entrypoint, and run the container without any daemon; identify which fields correspond to namespaces and which to cgroups.- Interview-style: "A database container's write throughput is far worse than the same database on the host, though CPU and memory limits are generous. What do you suspect?" Hint: think about what happens the first time the container writes to a file that came from an image layer, and how large that file is.
Containers vs Virtual Machines
Theory
The two technologies are routinely presented as competitors, but they answer different questions. A VM asks "what if this machine were several machines?"; a container asks "what if these processes had the system to themselves?". Choosing between them is mostly a question of where the isolation boundary needs to be.
Resource cost favours containers unambiguously. A VM carries an entire guest OS: its own kernel, init system, drivers, page cache, and background daemons — typically 512 MiB to 2 GiB of memory and 10–60 seconds to boot, before the application starts. A container carries only its process tree; the kernel is already running. Memory overhead is a few megabytes and start time is milliseconds. On identical hardware you can run perhaps 10–50 VMs or 100–1,000 containers.
Isolation strength favours VMs just as unambiguously, and the reason is structural. Every container on a host shares one kernel. That kernel exposes roughly 350 system calls, plus ioctls, procfs, sysfs, and netlink — an enormous interface reachable directly from inside every container. A single kernel privilege-escalation vulnerability compromises every container on the machine simultaneously. A VM guest's interface to the hypervisor is far narrower: a handful of virtual devices and a hypercall interface, amounting to a few hundred entry points rather than thousands, and the boundary is enforced by hardware (VT-x, EPT, the IOMMU) rather than by kernel code paths.
This is the central trade, and it has a practical corollary: containers are not a sufficient security boundary between mutually untrusting tenants. They are an excellent boundary between your own workloads. Every public cloud that runs untrusted customer containers puts a VM underneath — AWS Fargate on Firecracker, Google Cloud Run on gVisor, Azure Container Instances on Hyper-V.
| Dimension | Virtual machine | Container |
|---|---|---|
| Isolates | Hardware | Processes (namespaces + cgroups) |
| Kernel | Its own, per VM | Shared with the host |
| Memory overhead | 512 MiB – 2 GiB | A few MiB |
| Start time | 10–60 s (Firecracker: ~125 ms) | 10–200 ms |
| Density per host | 10s | 100s–1000s |
| Guest OS choice | Any, including Windows on Linux | Same kernel only |
| Attack surface from inside | Narrow: virtual devices, hypercalls | Wide: ~350 syscalls, ioctls, /proc, /sys |
| Boundary enforced by | Hardware (VT-x, EPT, IOMMU) | Kernel code |
| Escape blast radius | One VM | Every container on the host |
| Live migration | Mature | Checkpoint/restore (CRIU), limited |
| Image size | GiB | MiB |
| Best for | Multi-tenant isolation, mixed OSes, legacy | Microservices, CI, dense same-OS workloads |
The genuinely interesting development is the middle ground, which has largely dissolved the dichotomy:
- Firecracker — a minimal VMM written for exactly this purpose, offering only five emulated devices, booting a microVM in about 125 ms with roughly 5 MiB of overhead. It powers AWS Lambda and Fargate: VM-strength isolation at container-like cost.
- Kata Containers — OCI-compatible, so Kubernetes schedules them normally, but each pod runs in its own microVM with its own kernel.
- gVisor — no VM at all; a userspace kernel intercepts container syscalls and implements most of them itself, so the host kernel sees a small, constrained set. Weaker than a VM, much stronger than plain
runc, with a syscall-overhead cost.
The practical decision rule is short: use containers for your own workloads, put a VM boundary under anything untrusted, and reach for a microVM runtime when you need both. And note that in real deployments this is rarely either/or — the overwhelmingly common production shape is containers running inside VMs, with the VM providing the tenant boundary and the container providing packaging and density.
Example
WHERE THE BOUNDARY IS
VIRTUAL MACHINES CONTAINERS
+---------+ +---------+ +-------+ +-------+ +-------+
| app | | app | | app | | app | | app |
| libs | | libs | | libs | | libs | | libs |
| GUEST | | GUEST | +-------+ +-------+ +-------+
| KERNEL | | KERNEL | | namespaces + cgroups |
+---------+ +---------+ +-----------------------------+
| HYPERVISOR | | SHARED HOST KERNEL |
+---------------------+ +-----------------------------+
| HARDWARE (VT-x) | | HARDWARE |
+---------------------+ +-----------------------------+
The boundary is HARDWARE-enforced The boundary is KERNEL CODE.
and narrow. One kernel bug crosses it,
for EVERY container at once.
THE ATTACK SURFACE, COUNTED
container -> host kernel
~350 system calls, each with many argument combinations
+ ioctl (thousands of device-specific commands)
+ /proc, /sys, netlink, eBPF, io_uring
-> millions of lines of privileged C, reachable directly
VM guest -> hypervisor
a handful of virtual devices (virtio queues)
+ a small hypercall interface
+ instructions that cause VM exits
-> Firecracker exposes FIVE devices, ~50k lines of Rust
This ratio is the entire argument. Seccomp narrows the container
number substantially (a good profile blocks ~250 of the 350), but
does not change the shape of the problem.
THE COST, COUNTED
VM container Firecracker microVM
memory overhead ~1 GiB ~5 MiB ~5 MiB
start time ~30 s ~50 ms ~125 ms
per host ~30 ~800 ~500
own kernel yes NO yes
hardware boundary yes no yes
Firecracker sits where the table says it cannot: VM isolation at
container cost, achieved by deleting almost everything a general
purpose VMM does (no BIOS, no PCI, no USB, no graphics, 5 devices).
WHAT PRODUCTION ACTUALLY LOOKS LIKE
+--------------------------------------------------------+
| physical host |
| +--------------------------+ +---------------------+ |
| | VM (tenant A) | | VM (tenant B) | |
| | +------+ +------+ | | +------+ +------+ | |
| | | ctr | | ctr | ... | | | ctr | | ctr | | |
| | +------+ +------+ | | +------+ +------+ | |
| +--------------------------+ +---------------------+ |
+--------------------------------------------------------+
VM boundary -> between TENANTS (hardware-enforced)
container boundary-> between SERVICES of one tenant (packaging,
density, deployment)
This is a Kubernetes node in every managed cloud offering.
DECISION RULE
untrusted code / multi-tenant? -> VM or microVM. Not negotiable.
different kernel or OS needed? -> VM. Containers cannot.
your own microservices, dense? -> containers.
need both? -> Kata / Firecracker / gVisor.
compliance requires hardware isolation? -> VM.
# ---------- Measure the difference yourself ----------
# Start time:
time docker run --rm alpine true
# real 0m0.412s <- most of it is the daemon, not the container
time (virsh start testvm && until virsh qemu-agent-command testvm \
'{"execute":"guest-ping"}' 2>/dev/null; do sleep 1; done)
# real 0m24.8s
# Memory overhead for an idle instance:
docker stats --no-stream alpine-sleeper
# MEM USAGE: 1.2MiB
virsh dommemstat testvm | grep rss
# rss 1048576 <- ~1 GiB for an idle VM
# Whose kernel am I using? The clearest single demonstration.
uname -r # host: 6.8.0-45-generic
docker run --rm alpine uname -r # container: 6.8.0-45-generic <- SAME
docker run --rm --runtime=kata alpine uname -r # Kata: 5.19.2 <- different
virsh console testvm # VM: whatever it booted
# ---------- The shared-kernel consequence, made concrete ----------
docker run --rm alpine sysctl -w vm.max_map_count=1000000
# sysctl: error: permission denied on key 'vm.max_map_count'
# Because that sysctl is GLOBAL: there is one kernel, and changing it
# would change it for the host and every other container.
docker run --rm --privileged alpine sysctl -w vm.max_map_count=1000000
# vm.max_map_count = 1000000
# --privileged just changed the HOST's kernel parameter from inside a
# "container". This is why --privileged is not a container at all.
# ---------- Narrow the container's surface ----------
docker run --rm --security-opt seccomp=/path/default.json alpine true
# Docker's default profile blocks ~44 syscalls outright; a tight
# application-specific profile blocks far more. Generate one by tracing:
strace -c -f -e trace=all ./myapp 2>&1 | awk 'NR>3{print $NF}' | sort -u
# ---------- microVMs: both properties at once ----------
# Firecracker: no BIOS, no PCI, 5 devices, boots in ~125 ms.
firecracker --api-sock /tmp/fc.sock &
curl --unix-socket /tmp/fc.sock -X PUT 'http://localhost/boot-source' \
-d '{"kernel_image_path":"vmlinux","boot_args":"console=ttyS0 reboot=k panic=1 pci=off"}'
# ^^^^^^
# no PCI bus at all: nothing to enumerate
Key takeaways
- VMs virtualize hardware and carry their own kernel; containers virtualize the system-call interface and share the host's.
- Containers cost a few megabytes and start in milliseconds; VMs cost hundreds of megabytes to gigabytes and start in tens of seconds.
- The isolation argument is about attack surface: a container faces roughly 350 system calls plus ioctls and pseudo-filesystems, while a VM guest faces a handful of virtual devices and hypercalls enforced by hardware.
- One kernel vulnerability compromises every container on a host simultaneously; a hypervisor escape is a much rarer and harder class of bug.
- Containers are an appropriate boundary between your own workloads and an inappropriate one between mutually untrusting tenants.
- Every major cloud that runs untrusted customer containers places a VM or sandboxed runtime underneath.
- Firecracker collapses the trade-off by deleting almost everything a general-purpose VMM does, giving VM-strength isolation at roughly 5 MiB and 125 ms.
- gVisor offers an intermediate point with a userspace kernel and no VM, at the cost of syscall overhead.
--privilegedremoves the container boundary entirely, which is why it should be treated as "run this as root on the host".- The dominant production shape is containers inside VMs: the VM is the tenant boundary, the container is the packaging and density mechanism.
🧪 Practice
- Run
uname -ron the host and inside a container, explain why they match, and name one consequence for what the container can configure.- Measure start time and idle memory for a container and a VM on the same machine, and express the density difference for a host with 64 GiB of RAM.
- Explain why a CI system running untrusted pull-request code should not use plain containers, and describe two acceptable alternatives.
- Interview-style: "A team proposes replacing their VM-per-customer architecture with one container per customer to cut costs. What is your response?" Hint: separate the packaging argument from the isolation argument, and ask what a single kernel CVE would mean under each design.
Unikernels
Theory
Every layer this chapter has examined adds isolation and generality at some cost. Unikernels ask what happens if you push in the opposite direction: what if an application did not run on an operating system, but was one?
The observation motivating them is that a VM running one application carries an enormous amount of machinery it will never use. A general-purpose Linux kernel supports thousands of devices, multiple filesystems, many users, arbitrary process creation, and a full network stack with every protocol — and a containerised microservice uses a tiny fraction of that. All of it must still be loaded, maintained, patched, and is reachable as attack surface.
A unikernel is a specialised, single-address-space machine image built by compiling an application together with just the OS functionality it actually calls, using a library operating system. Filesystem, network stack, and scheduler are ordinary libraries linked into the binary. The result boots directly on a hypervisor with no separate kernel.
The consequences are dramatic in both directions.
What is gained:
- Size. Images of 1–10 MiB rather than hundreds. A minimal web server can be under 1 MiB.
- Boot time. Tens of milliseconds — sometimes under 20 ms — because there is no device probing, no init system, and no dynamic linking. Fast enough to boot per request.
- Attack surface. If the application never uses a filesystem, no filesystem code is linked in and there is no filesystem to attack. There is no shell to drop into, no
/bin, no package manager, and typically no way to execute new code at all. - Performance. A single address space with no user/kernel boundary means no system-call transitions: the "syscall" is a function call. No process abstraction means no context switches.
What is given up, and it is a great deal:
- No process model. No
fork(), noexec(), often a single address space with cooperative or lightweight threading only. - No debugging as you know it. No shell, no
ps, nostrace, no ability to attach to a running instance. Observability must be designed in from the start. - Language and library constraints. MirageOS requires OCaml; other frameworks support specific runtimes. Anything calling an unsupported POSIX interface will not build.
- Weak internal isolation. A single address space means a memory-safety bug can reach anything in the image. The argument is that the external surface is tiny and the image is often written in a memory-safe language, but there is no internal boundary.
- Operational immaturity. Patching means rebuilding and redeploying the whole image, and the tooling ecosystem is nothing like Linux's.
Notable implementations: MirageOS (OCaml, the most rigorous realisation of the idea), IncludeOS (C++), OSv (runs unmodified Linux binaries in a unikernel-like single address space), Nanos with the Ops tooling, and Unikraft, the most practical current effort — a modular library OS with POSIX compatibility layers that substantially reduces the porting cost.
Unikernels have not displaced containers, and the reason is instructive: the operational cost of losing the entire Linux tooling ecosystem outweighs the gains for most workloads. Their real influence has been indirect — they demonstrated how much of a general-purpose OS a single application does not need, and that argument shaped Firecracker's minimal device model, distroless and scratch container images, and WebAssembly-based runtimes like WASI, which pursue similar goals with far lower migration cost.
Example
THE SPECIALISATION SPECTRUM
traditional VM container microVM UNIKERNEL
+-----------+ +---------+ +---------+ +-----------+
| app | | app | | app | | app |
| libs | | libs | | libs | | +only the |
| full OS | | (shared | | minimal | | OS parts |
| + daemons | | kernel)| | kernel | | it CALLS |
| + drivers | +---------+ +---------+ +-----------+
+-----------+
~2 GiB ~50 MiB ~50 MiB ~2 MiB
~30 s ~50 ms ~125 ms ~20 ms
full generality no generality
ADDRESS SPACE: THE STRUCTURAL DIFFERENCE
TRADITIONAL UNIKERNEL
+---------------------------+ +---------------------------+
| user space (ring 3) | | |
| application | | application |
| libc | | network stack (a lib) |
+---------------------------+ | filesystem (a lib) |
| <<< SYSCALL BOUNDARY >>> | | scheduler (a lib) |
+---------------------------+ | |
| kernel space (ring 0) | | ONE address space, |
| scheduler, VFS, TCP/IP, | | ONE privilege level |
| drivers, 350 syscalls | +---------------------------+
+---------------------------+ | HYPERVISOR |
| HARDWARE | +---------------------------+
+---------------------------+
read() in a unikernel is a FUNCTION CALL, not a mode transition:
~1 ns, not ~200 ns. And there is no boundary to attack.
ATTACK SURFACE BY SUBTRACTION
what a typical unikernel image DOES NOT CONTAIN:
no shell -> nothing to spawn after an RCE
no /bin, no busybox -> no tools for an attacker to use
no package manager -> nothing to install
no fork/exec -> a "command injection" has no command to run
no other users -> no privilege escalation target
no unused drivers -> no vulnerable code path
no filesystem (if unused) -> no path traversal to exploit
The classic exploit chain (RCE -> spawn a shell -> pivot) has no
second step. That is the security argument, and it is a strong one.
THE HONEST COST
production incident, Linux VM production incident, unikernel
ssh in (no ssh)
ps aux, top, strace -p (none of these exist)
tcpdump, lsof, /proc (none of these exist)
attach gdb, take a core (must be designed in)
patch one package, restart rebuild the ENTIRE image
---------------------------------- ----------------------------
You have a live system to inspect. You have logs and metrics you
thought to emit in advance.
This, not performance, is why unikernels remain niche.
(* MirageOS: the entire "operating system" for an HTTP service.
What is NOT here is the point -- no process model, no filesystem,
no users, no shell. Only what this service calls gets linked in. *)
open Lwt.Infix
module Main (S : Cohttp_mirage.Server.S) = struct
let handler _conn request _body =
(* The network stack is a LIBRARY in this same address space.
Reading from the socket is a function call, not a syscall:
no mode transition, no ring change, no TLB implications. *)
let path = Uri.path (Cohttp.Request.uri request) in
match path with
| "/" ->
S.respond_string ~status:`OK ~body:"hello from a unikernel\n" ()
| "/health" ->
S.respond_string ~status:`OK ~body:"ok\n" ()
| _ ->
S.respond_string ~status:`Not_found ~body:"not found\n" ()
(* This IS the whole system. There is no init, no PID 1, no other
process -- because there is no process abstraction at all. *)
let start http =
http (`TCP 8080) (S.make ~callback:handler ())
end
# ---------- Build one and look at what you get ----------
# MirageOS: configure for a hypervisor target, then compile.
mirage configure -t hvt # hvt = a minimal KVM/Solo5 target
make depend && make
ls -lh dist/httpserver.hvt
# -rwxr-xr-x 1.8M httpserver.hvt
# 1.8 MiB. The equivalent Linux VM image is 300 MiB to 2 GiB.
solo5-hvt --net:service=tap0 dist/httpserver.hvt
# Solo5: Bindings version v0.9.0
# ...
# boot time: 0.019s <- 19 ms, cold, from nothing
# ---------- Compare, honestly, across the spectrum ----------
# image boot idle RAM has a shell?
# Ubuntu VM 2.3 GiB 28 s 512 MiB yes
# Alpine VM 130 MiB 6 s 48 MiB yes
# Docker (alpine) 7 MiB 50 ms 2 MiB yes
# Docker (scratch) 4 MiB 40 ms 1 MiB NO
# Firecracker microVM 50 MiB 125 ms 5 MiB yes
# MirageOS unikernel 1.8 MiB 19 ms 8 MiB NO
# ---------- Unikraft: the practical modern attempt ----------
kraft pkg update
kraft build --target nginx/qemu-x86_64 # POSIX compat layer means
kraft run nginx # real apps can be ported
# Unikraft's contribution is reducing the porting cost, which was the
# single biggest barrier to adoption.
# ---------- The ideas that actually shipped everywhere ----------
# 1. "scratch" and distroless images: no shell, no package manager.
cat <<'EOF' > Dockerfile
FROM golang:1.22 AS build
WORKDIR /src
COPY . .
RUN CGO_ENABLED=0 go build -o /app .
FROM scratch # <- NOTHING. Not even libc.
COPY --from=build /app /app
ENTRYPOINT ["/app"]
EOF
# ~4 MiB image, no shell to exec into after an RCE. The unikernel
# security argument, achieved with ordinary container tooling.
# 2. WebAssembly + WASI: a capability-based sandbox, similar goals,
# far lower migration cost -- arguably where the idea actually landed.
# wasmtime run --dir=/data app.wasm
Key takeaways
- A unikernel compiles an application together with only the OS functionality it actually calls, using a library operating system, producing a single-address-space image that boots directly on a hypervisor.
- Images are 1–10 MiB and boot in tens of milliseconds because there is no device probing, no init system, and no dynamic linking.
- With one address space and one privilege level there is no user/kernel boundary, so a system call becomes an ordinary function call.
- The security argument is subtraction: no shell, no
fork/exec, no package manager, and no unused drivers means a remote-code-execution exploit has no second stage. - The costs are severe: no process model, no conventional debugging, language and library restrictions, and no internal isolation within the single address space.
- Patching requires rebuilding and redeploying the entire image rather than updating a package.
- MirageOS is the most rigorous implementation, while Unikraft is the most practical, reducing porting cost through POSIX compatibility layers.
- Unikernels remain niche because losing the Linux tooling ecosystem outweighs the gains for most workloads — an operational argument, not a technical one.
- Their ideas shipped elsewhere: Firecracker's minimal device model,
scratchand distroless images, and WebAssembly/WASI all pursue the same specialisation with far lower migration cost.
🧪 Practice
- List four components of a general-purpose Linux kernel that a stateless HTTP microservice never uses, and explain what each contributes to attack surface.
- Build a Go service into a
FROM scratchimage, compare its size with thealpine-based equivalent, and explain which unikernel property you have reproduced and which you have not.- Explain why a system call in a unikernel costs roughly a function call, and what safety property is given up to achieve that.
- Interview-style: "Unikernels boot in 20 ms, are a hundred times smaller, and have almost no attack surface. Why is essentially nobody running them in production?" Hint: imagine being paged at 3 a.m. for a latency regression, and list the commands you would normally run first.
<a id="14-distributed-and-specialized-systems"></a>