6. Routing
Chapter 5 established that a router forwards a packet by consulting a table; this chapter is about where that table comes from. It covers the algorithms that compute paths, the protocols that run those algorithms inside an organisation, the very different protocol that stitches 75,000 independent networks into one internet, and the forwarding techniques that go beyond destination-based lookup. Routing is where the internet's decentralisation becomes concrete: no authority computes the paths, yet packets arrive.
<a id="61-routing-fundamentals"></a>
6.1 Routing Fundamentals
This subchapter covers the choices that exist before any protocol is selected: whether routes are typed in or learned, how a router handles destinations it knows nothing about, how it chooses between conflicting information, and how it uses several paths at once.
Static vs Dynamic Routing
Theory
A routing table can be populated two ways: someone types the entries in, or the router learns them by talking to its neighbours.
Static routing means an administrator configures each route by hand: "to reach 10.2.0.0/16, send to 192.168.1.2." The router stores it and uses it until someone changes it. There is no protocol, no negotiation, and no adaptation.
Static routing's virtues are real and often underrated. It consumes no CPU, no memory, and no bandwidth — there are no protocol packets at all. It is completely predictable: the path is whatever you wrote, and it cannot be influenced by a neighbour's misconfiguration or a protocol bug. It is secure in the sense that there is no protocol to attack, no adjacency to spoof, no malicious advertisement to accept. And it is trivially auditable — the configuration is the routing policy.
Its defect is equally structural: it does not react to anything. If the configured next hop goes down, the router keeps sending packets into the void. The route remains in the table, pointing at a path that no longer works, because nothing is monitoring it. Recovery requires a human.
Dynamic routing means routers run a protocol — RIP, OSPF, IS-IS, EIGRP, BGP — that discovers neighbours, exchanges reachability information, and recomputes paths automatically. When a link fails, the protocol notices and the network converges on a new set of paths without human involvement.
The costs are the mirror image: CPU and memory for protocol state, bandwidth for protocol messages, a convergence period during which the network is temporarily inconsistent, configuration complexity, and an attack surface (a neighbour can advertise routes to you).
The choice is not ideological; it follows from the topology:
| Situation | Choice | Why |
|---|---|---|
| Stub network, one exit | Static default route | There is only one path; a protocol would compute what you already know |
| Small network, stable, few links | Static | Manual effort is low, predictability is high |
| Any network with redundant paths | Dynamic | The value of redundancy is automatic failover, which static cannot provide |
| Large or frequently changing network | Dynamic | Manual configuration does not scale past a few dozen routes |
| Between organisations | Dynamic (BGP) | Policy and reachability change constantly and are not yours to configure |
The decisive question is simple: is there more than one path, and do you want it used automatically? If yes, you need a protocol. If a network has exactly one way out, a static default route is not a compromise — it is the correct engineering answer, and running a protocol would be pure overhead.
Real networks mix the two. A common pattern is dynamic routing in the core with static routes at the edges, plus static floating routes — a backup route configured with a deliberately worse administrative distance so it activates only when the dynamic route disappears.
One refinement makes static routing far more usable: route tracking (or object tracking). The router periodically probes the next hop, and withdraws the static route if the probe fails. This adds the one thing static routing lacks — failure detection — without a full protocol.
Examples
Why static routing fails on a redundant topology
STATIC:
[A] ----primary---- [B] ---- [dest]
\ /
\---backup---[C]-/
ip route 10.2.0.0/16 via B <- configured once, by hand
B fails:
A keeps sending to B. Forever.
The backup path through C exists and is healthy.
Nothing notices. Traffic is black-holed until a human intervenes.
DYNAMIC:
B fails -> A's protocol detects the lost adjacency
-> A recomputes, installs the path via C
-> traffic flows again in seconds, unattended.
When static is CORRECT: a stub network
[branch office] ---- single link ---- [ISP] ---- internet
|
ip route 0.0.0.0/0 via ISP
There is exactly ONE path. A routing protocol would exchange
messages, consume CPU, and arrive at the same single answer.
Static is not a compromise here; it is the right choice.
class StaticRouter:
"""No protocol. No monitoring. Absolute predictability, zero adaptation."""
def __init__(self, name):
self.name = name
self.routes = {} # prefix -> next_hop
self.link_state = {} # next_hop -> up/down (reality, which the
# router does NOT consult)
def add_static(self, prefix, next_hop):
self.routes[prefix] = next_hop
def link_fails(self, next_hop):
self.link_state[next_hop] = "down"
# NOTE: the routing table is deliberately untouched. This is the
# entire problem with static routing.
def forward(self, prefix):
nh = self.routes.get(prefix)
if nh is None:
return "DROP (no route)"
if self.link_state.get(nh) == "down":
return f"BLACK HOLE: still sending to {nh}, which is down"
return f"forward via {nh}"
r = StaticRouter("A")
r.add_static("10.2.0.0/16", "B")
print(r.forward("10.2.0.0/16")) # forward via B
r.link_fails("B")
print(r.forward("10.2.0.0/16")) # BLACK HOLE: still sending to B, which is down
class DynamicRouter(StaticRouter):
"""A protocol watches adjacencies and recomputes when they change."""
def __init__(self, name):
super().__init__(name)
self.candidates = {} # prefix -> [(next_hop, cost), ...]
def learn(self, prefix, next_hop, cost):
self.candidates.setdefault(prefix, []).append((next_hop, cost))
self.recompute(prefix)
def recompute(self, prefix):
# Only paths whose next hop is currently up are eligible.
alive = [(nh, c) for nh, c in self.candidates.get(prefix, [])
if self.link_state.get(nh) != "down"]
if alive:
self.routes[prefix] = min(alive, key=lambda x: x[1])[0]
else:
self.routes.pop(prefix, None)
def link_fails(self, next_hop):
self.link_state[next_hop] = "down"
for prefix in list(self.candidates): # THE difference: react
self.recompute(prefix)
d = DynamicRouter("A")
d.learn("10.2.0.0/16", "B", cost=10) # primary
d.learn("10.2.0.0/16", "C", cost=20) # backup, worse metric
print(d.forward("10.2.0.0/16")) # forward via B
d.link_fails("B")
print(d.forward("10.2.0.0/16")) # forward via C <- automatic failover
# Static routing's cost is HUMAN effort, and it grows quadratically.
def config_burden(routers, prefixes_each):
"""Full-mesh static configuration: every router needs a route to every prefix."""
total_prefixes = routers * prefixes_each
entries = routers * (total_prefixes - prefixes_each) # excluding its own
return total_prefixes, entries
print()
for n in (3, 10, 50):
prefixes, entries = config_burden(n, prefixes_each=2)
print(f"{n:>3} routers, 2 prefixes each: {prefixes:>4} prefixes, "
f"{entries:>6,} static entries to type and maintain")
# 3 routers, 2 prefixes each: 6 prefixes, 12 static entries to type and maintain
# 10 routers, 2 prefixes each: 20 prefixes, 180 static entries to type and maintain
# 50 routers, 2 prefixes each: 100 prefixes, 4,900 static entries to type and maintain
# A dynamic protocol requires the SAME configuration regardless of size.
# Static routes: explicit, permanent, unmonitored.
$ sudo ip route add 10.2.0.0/16 via 192.168.1.2 dev eth0
$ ip route show
# 10.2.0.0/16 via 192.168.1.2 dev eth0 <- no 'proto' = statically set
# default via 192.168.1.1 dev eth0 proto dhcp
# Cisco equivalent, with a floating backup:
# ip route 10.2.0.0 255.255.0.0 192.168.1.2 <- AD 1 (default), primary
# ip route 10.2.0.0 255.255.0.0 192.168.1.3 200 <- AD 200, FLOATING backup
# ^^^ only installed if the primary
# route disappears entirely
# Route tracking: give a static route the one thing it lacks — failure detection.
# ip sla 1
# icmp-echo 192.168.1.2 source-interface GigabitEthernet0/0
# frequency 5
# ip sla schedule 1 life forever start-time now
# track 10 ip sla 1 reachability
# ip route 10.2.0.0 255.255.0.0 192.168.1.2 track 10
# ^^^^^^^^ withdraw the route if
# the probe stops answering
# Dynamic routes carry their source protocol:
$ ip route show proto ospf
# 10.2.0.0/16 via 192.168.1.2 dev eth0 proto ospf metric 20
# ^^^^^^^^^^ learned, not configured
$ vtysh -c "show ip route" | head -5
# Codes: K - kernel, C - connected, S - static, R - RIP, O - OSPF, B - BGP
# S>* 10.2.0.0/16 [1/0] via 192.168.1.2, eth0
# O>* 10.3.0.0/16 [110/20] via 192.168.1.5, eth0
# ^ the code letter tells you instantly whether a human or a protocol put it there
Key takeaways
- Static routing is manually configured: no CPU, no bandwidth, fully predictable and auditable, and no reaction to failure.
- Dynamic routing runs a protocol that discovers neighbours and recomputes paths automatically, at the cost of resources, convergence delay, complexity, and an attack surface.
- The deciding question is whether redundant paths exist — redundancy is worthless if nothing switches to it automatically.
- A stub network with one exit is correctly served by a static default route; a protocol there is pure overhead.
- Static routes black-hole traffic when the next hop dies, because nothing monitors the path.
- Floating static routes (deliberately high administrative distance) provide backup paths that activate only when the primary disappears.
- Route tracking / IP SLA adds failure detection to static routes without running a full protocol.
- Real networks mix both: dynamic in the core, static at stub edges.
🧪 Practice
- A branch office has one link to headquarters and one to the internet. Decide static or dynamic for each, and justify both answers.
- Explain precisely what happens to traffic when a statically configured next hop fails, and how a floating static route changes the outcome.
- Compute the number of static entries needed for 20 routers each originating 3 prefixes in a fully reachable network, and state what a protocol would require instead.
- Interview-style: When is static routing the better engineering choice, not just the lazier one? (Hint: count the number of usable paths, then ask what a protocol could possibly compute that you do not already know.)
Default Routes and Gateways
Theory
No router can hold a route to every destination on the internet — and most routers should not try. A home router has a few kilobytes of routing memory and needs to reach a billion destinations. The default route resolves this: a single entry meaning "anything I do not have a specific route for, send here."
The default route is written 0.0.0.0/0 in IPv4 and ::/0 in IPv6. Its elegance is that it requires no special-case logic in the forwarding path. Recall longest prefix match from Chapter 5: a /0 prefix fixes zero bits, so it matches every address and loses to every other entry. "Use this only if nothing else matched" falls directly out of the ordinary lookup rule. There is no "default route" code path — there is just a very short prefix.
The default gateway is the next hop that a default route points to. On a host, it is the router to send off-subnet traffic to. The host's decision procedure for every packet is exactly two steps:
- Compute
destination AND my_mask. Does it equal my own network address? - Same subnet → resolve the destination's MAC with ARP/NDP and send directly. Different subnet → resolve the gateway's MAC and send the frame there, with the destination IP unchanged.
That last detail matters and is frequently misunderstood: the frame's destination MAC is the gateway's, but the packet's destination IP remains the final destination. Layer 2 addressing changes at every hop; layer 3 addressing does not. This is the clearest practical illustration of what encapsulation means.
Default routes create a natural hierarchy of knowledge that mirrors the internet's structure:
| Position | Routing knowledge |
|---|---|
| Host | Its own subnet, plus a default route |
| Access/branch router | Its local networks, plus a default toward the ISP |
| ISP edge | Its customers' prefixes, plus a default toward the core |
| Tier 1 core (DFZ) | The full table (~1M prefixes). No default route exists |
The top of that hierarchy is the Default-Free Zone. A Tier 1 router has nowhere to send a default route to — there is no "up" from the top — so it must know every prefix on the internet. This is precisely why the growth of the global routing table is an industry-wide concern rather than an individual operator's problem.
Two failure modes are worth recognising on sight:
The default route black hole. A default route always matches, so a router with one can never report "no route to host". Packets for genuinely unreachable destinations are forwarded to the gateway, which forwards them onward, until TTL expires. Failures that would have been fast and explicit become slow timeouts.
Dual default routes. Two default routes with equal cost cause traffic to be load-balanced across both, which is usually not what a multihomed network intends and interacts badly with stateful firewalls and NAT — a flow's outbound and return paths may differ, and the stateful device on the return path has no matching state.
Examples
The default route is just a very short prefix — no special case needed
Routing table:
10.1.0.0/16 -> R2 (16 bits fixed)
192.168.1.0/24 -> R3 (24 bits fixed)
0.0.0.0/0 -> R1 ( 0 bits fixed — matches EVERYTHING)
dst 10.1.5.9 -> matches /16 and /0 -> LONGER wins -> R2
dst 192.168.1.7 -> matches /24 and /0 -> LONGER wins -> R3
dst 93.184.216.34-> matches only /0 -> R1 (the default)
Nothing in the lookup code knows what a "default route" is.
Same subnet or not: the two-step decision a host makes per packet
host 192.168.1.20/24, gateway 192.168.1.1
dst 192.168.1.50: dst 8.8.8.8:
192.168.1.50 AND /24 = 192.168.1.0 8.8.8.8 AND /24 = 8.8.8.0
== my network -> LOCAL != my network -> REMOTE
ARP for 192.168.1.50 ARP for 192.168.1.1 (the GATEWAY)
The frame sent for 8.8.8.8:
+---------------------------+-----------------------------+
| dst MAC = gateway's MAC | dst IP = 8.8.8.8 |
| src MAC = my MAC | src IP = 192.168.1.20 |
+---------------------------+-----------------------------+
^ layer 2 changes every hop ^ layer 3 does not change at all
The hierarchy of routing knowledge
host [ my subnet | default ] 2 entries
|
branch router [ local nets | default ] ~10 entries
|
ISP edge [ customer prefixes | default ] ~1,000 entries
|
Tier 1 core [ EVERY prefix on the internet ] ~1,000,000 entries
^ the DEFAULT-FREE ZONE: no "up" exists to default toward
import ipaddress
class Host:
def __init__(self, addr_cidr, gateway):
self.iface = ipaddress.ip_interface(addr_cidr)
self.gateway = gateway
def next_hop_for(self, dst):
d = ipaddress.ip_address(dst)
if d in self.iface.network:
# Same subnet: ARP for the destination itself.
return f"DIRECT — ARP for {dst}, frame goes straight to it"
# Different subnet: ARP for the GATEWAY, but dst IP stays unchanged.
return (f"VIA GATEWAY {self.gateway} — ARP for the gateway; "
f"frame dst MAC = gateway, packet dst IP = {dst}")
h = Host("192.168.1.20/24", gateway="192.168.1.1")
for dst in ["192.168.1.50", "192.168.1.254", "8.8.8.8", "10.0.0.1"]:
print(f"{dst:<16} {h.next_hop_for(dst)}")
# 192.168.1.50 DIRECT — ARP for 192.168.1.50, frame goes straight to it
# 192.168.1.254 DIRECT — ARP for 192.168.1.254, frame goes straight to it
# 8.8.8.8 VIA GATEWAY 192.168.1.1 — ARP for the gateway; frame dst MAC = gateway, packet dst IP = 8.8.8.8
# 10.0.0.1 VIA GATEWAY 192.168.1.1 — ARP for the gateway; frame dst MAC = gateway, packet dst IP = 10.0.0.1
# The default route needs no special handling — LPM already does the right thing.
FIB = [("10.1.0.0/16", "R2"), ("192.168.1.0/24", "R3"), ("0.0.0.0/0", "R1")]
def lookup(dst):
d = ipaddress.ip_address(dst)
best = None
for cidr, nh in FIB:
net = ipaddress.ip_network(cidr)
if d in net and (best is None or net.prefixlen > best[0].prefixlen):
best = (net, nh)
return best
print()
for dst in ["10.1.5.9", "192.168.1.7", "93.184.216.34"]:
net, nh = lookup(dst)
kind = "DEFAULT ROUTE" if net.prefixlen == 0 else f"/{net.prefixlen} specific"
print(f"{dst:<16} -> {nh} ({kind})")
# 10.1.5.9 -> R2 (/16 specific)
# 192.168.1.7 -> R3 (/24 specific)
# 93.184.216.34 -> R1 (DEFAULT ROUTE)
# Why a default route converts fast failures into slow timeouts.
def failure_mode(has_default, dst_reachable):
if dst_reachable:
return "delivered"
if has_default:
return ("forwarded to the gateway, then onward, until TTL expires — "
"the user sees a SLOW TIMEOUT")
return "ICMP Destination Unreachable immediately — the user sees a FAST, clear error"
print()
for has_def in (True, False):
print(f"default route present={has_def!s:<5} -> {failure_mode(has_def, False)}")
# default route present=True -> forwarded to the gateway, then onward, until TTL expires — the user sees a SLOW TIMEOUT
# default route present=False -> ICMP Destination Unreachable immediately — the user sees a FAST, clear error
# Table size by position in the hierarchy.
HIERARCHY = [("host", 2), ("branch router", 10), ("enterprise core", 500),
("ISP edge", 5_000), ("Tier 1 / DFZ", 1_000_000)]
print()
for role, entries in HIERARCHY:
default = "no default route possible" if entries >= 1_000_000 else "uses a default route"
print(f"{role:<18}{entries:>10,} entries {default}")
# host 2 entries uses a default route
# branch router 10 entries uses a default route
# enterprise core 500 entries uses a default route
# ISP edge 5,000 entries uses a default route
# Tier 1 / DFZ 1,000,000 entries no default route possible
# The default route on a host.
$ ip route show default
# default via 192.168.1.1 dev eth0 proto dhcp src 192.168.1.20 metric 100
# ^^^ this is 0.0.0.0/0; 'ip route' prints it as the word "default"
$ ip route get 8.8.8.8
# 8.8.8.8 via 192.168.1.1 dev eth0 src 192.168.1.20 uid 1000
# ^^^ matched the default route
$ ip route get 192.168.1.50
# 192.168.1.50 dev eth0 src 192.168.1.20 uid 1000
# ^^^ no "via": same subnet, delivered directly
# Proof that only the MAC changes, not the destination IP:
$ sudo tcpdump -i eth0 -e -n host 8.8.8.8 -c 1
# aa:bb:cc:dd:ee:ff > 00:1a:2b:3c:4d:5e, ethertype IPv4, 192.168.1.20 > 8.8.8.8
# ^^^^^^^^^^^^^^^^^ the GATEWAY's MAC
# ^^^^^^^ the
# FINAL destination IP, unchanged
# IPv6: the gateway is a link-local address, learned from the RA.
$ ip -6 route show default
# default via fe80::1 dev eth0 proto ra metric 1024 expires 1789sec
# Originating a default route INTO a protocol, for downstream routers:
# router ospf 1
# default-information originate always
# ^^^^^^^^^^^^^^^^ advertise 0.0.0.0/0 to all neighbours
# router bgp 64500
# neighbor 10.0.0.2 default-originate
# A Default-Free Zone router has no default — verify by its absence:
$ vtysh -c "show ip route 0.0.0.0/0"
# % Network not in table
# ^ correct for a DFZ router: it must know every prefix explicitly
Key takeaways
- The default route (
0.0.0.0/0,::/0) means "use this if nothing more specific matched". - It needs no special-case code: with zero bits fixed it matches everything and loses to everything under longest prefix match.
- The default gateway is the next hop for off-subnet traffic; a host decides local-vs-remote with one
ANDagainst its mask. - When sending via a gateway, the frame's destination MAC is the gateway's while the packet's destination IP is unchanged — layer 2 changes per hop, layer 3 does not.
- Default routes create a hierarchy of knowledge: hosts know almost nothing, the Default-Free Zone knows everything.
- DFZ routers have no default route because there is nowhere above them to point it, which is why global table growth is an industry-wide concern.
- A default route turns "no route to host" into a slow timeout, because it always matches.
- Two equal-cost default routes load-balance traffic, which often breaks stateful firewalls and NAT through asymmetric return paths.
🧪 Practice
- A host is
10.5.3.40/22with gateway10.5.0.1. Determine whether it sends directly or via the gateway for10.5.2.9,10.5.4.200, and10.6.0.1.- Explain why
0.0.0.0/0requires no special logic in the forwarding path, referring to longest prefix match.- Explain why a Tier 1 backbone router cannot use a default route, and what the consequence is for its memory requirements.
- Interview-style: A user reports that unreachable hostnames take 30 seconds to fail instead of failing instantly. How might the routing configuration explain this? (Hint: ask which table entry matches a destination that does not exist, and what the router does with the packet as a result.)
Administrative Distance and Route Metrics
Theory
A router frequently learns the same prefix from several sources: a static route says one thing, OSPF says another, BGP says a third. All three cannot be installed. Two distinct comparisons resolve this, and they operate at different levels — confusing them is one of the most common conceptual errors in routing.
Administrative distance (AD) compares across protocols. It answers: which source do I trust more? It is a locally significant number from 0 to 255, never transmitted to any other router, and lower means more trusted.
| Source | AD | Rationale |
|---|---|---|
| Connected interface | 0 | Directly observed; cannot be wrong |
| Static route | 1 | A human explicitly said so |
| eBGP | 20 | External, but deliberately chosen |
| EIGRP (internal) | 90 | Sophisticated metric, fast convergence |
| OSPF | 110 | Link-state, full topology knowledge |
| IS-IS | 115 | Link-state, similar to OSPF |
| RIP | 120 | Hop count only — crude |
| External EIGRP | 170 | Redistributed, less trustworthy |
| iBGP | 200 | Internal BGP, prefer IGP paths |
| Unreachable | 255 | Never installed |
The ordering encodes a judgement about information quality. A connected route reflects direct observation. A static route reflects human intent. A link-state protocol has the full topology. RIP knows only hop counts, so it is trusted least among IGPs. eBGP's low value of 20 is deliberate: routes from outside your organisation are real reachability information that should beat your internal protocols' guesses about external destinations.
Metrics compare within one protocol. They answer: among several paths my protocol knows, which is best? Each protocol defines its own, and the values are meaningless across protocols — OSPF cost 10 and RIP metric 10 are not comparable quantities.
| Protocol | Metric | Basis |
|---|---|---|
| RIP | Hop count | Number of routers, max 15 |
| OSPF | Cost | reference_bandwidth / interface_bandwidth |
| IS-IS | Cost | Configured, default 10 per interface |
| EIGRP | Composite | Bandwidth and delay (by default), K-value weighted |
| BGP | Path attributes | Policy, not a single number |
The order of evaluation is where people go wrong, so state it precisely:
- Longest prefix match first. A /24 always beats a /16 for an address inside it, no matter which protocol supplied either.
- Administrative distance next, among routes to the identical prefix.
- Metric last, among routes from the same protocol to the identical prefix.
So a RIP-learned /24 beats an OSPF-learned /16 covering the same address — specificity is resolved before trust. This is the same rule that makes BGP hijacking by more-specific announcement work (Chapter 5).
Two practical consequences follow.
AD is locally significant. You can change it on one router and no other router knows. This makes it a blunt but effective tool: raising a static route's AD to 200 creates a floating static route that only installs when the dynamic route vanishes.
AD manipulation causes routing loops when used carelessly, particularly during redistribution between protocols. Router A redistributes OSPF into RIP, router B redistributes RIP back into OSPF, and if the AD values let the round-trip route win, traffic circulates. This is why redistribution requires route tags and filtering rather than AD tweaking alone.
OSPF's metric deserves one specific warning. Cost is reference_bandwidth / interface_bandwidth, with a default reference of 100 Mb/s. Any interface at or above 100 Mb/s therefore gets cost 1 — a 100 Mb/s link, a 10 Gb/s link, and a 100 Gb/s link are all indistinguishable. On any modern network the reference bandwidth must be raised (commonly to 100,000, i.e. 100 Gb/s) and raised identically on every router, or the protocol will happily choose a 100 Mb/s path over a 100 Gb/s one.
Examples
Two comparisons at two different levels
PREFIX 10.1.0.0/16 learned three times:
source AD metric
static 1 0 <-- WINS: lowest AD
OSPF 110 20
RIP 120 3 <-- lowest metric, but IRRELEVANT here:
metrics are never compared across protocols
AD decides which PROTOCOL's answer to take.
Metric decides only among that protocol's own candidates.
Order of evaluation — the part people get wrong
Routing table:
10.1.5.0/24 via RIP (AD 120, metric 8)
10.1.0.0/16 via OSPF (AD 110, metric 2)
Destination 10.1.5.42:
STEP 1 longest prefix match -> /24 beats /16. DECIDED HERE.
STEP 2 administrative distance -> never consulted: only one /24 exists.
STEP 3 metric -> never consulted.
Result: the RIP route wins, despite RIP being the LEAST trusted protocol.
Specificity is resolved before trust.
OSPF's reference bandwidth trap
default reference = 100 Mb/s:
FastEthernet 100 Mb/s -> cost 100/100 = 1
GigabitEth 1 Gb/s -> cost 100/1000 = 0.1 -> rounded UP to 1
TenGig 10 Gb/s -> cost 100/10000 = 0.01 -> rounded UP to 1
^^^^^^^^^^^^^
ALL THE SAME. OSPF cannot tell a 100 Mb/s link from a 10 Gb/s link.
reference raised to 100,000 (100 Gb/s):
FastEthernet -> 1000
GigabitEth -> 100
TenGig -> 10
HundredGig -> 1 <- now the metric means something
ADMIN_DISTANCE = {
"connected": 0, "static": 1, "ebgp": 20, "eigrp": 90, "ospf": 110,
"isis": 115, "rip": 120, "eigrp-external": 170, "ibgp": 200, "unreachable": 255,
}
def install_best(candidates):
"""AD first, then metric — but ONLY among routes to the same prefix."""
best = None
for c in candidates:
ad = ADMIN_DISTANCE[c["source"]]
if ad == 255:
continue # never installed
key = (ad, c["metric"])
if best is None or key < (ADMIN_DISTANCE[best["source"]], best["metric"]):
best = c
return best
candidates = [
{"prefix": "10.1.0.0/16", "source": "rip", "metric": 3, "nh": "10.0.0.3"},
{"prefix": "10.1.0.0/16", "source": "ospf", "metric": 20, "nh": "10.0.0.2"},
{"prefix": "10.1.0.0/16", "source": "static", "metric": 0, "nh": "10.0.0.4"},
]
win = install_best(candidates)
print(f"installed: {win['source']} via {win['nh']} "
f"(AD {ADMIN_DISTANCE[win['source']]}, metric {win['metric']})")
print("note: RIP had the LOWEST metric (3) and still lost — AD decides first.")
# installed: static via 10.0.0.4 (AD 1, metric 0)
# note: RIP had the LOWEST metric (3) and still lost — AD decides first.
# The full evaluation order, including prefix length.
import ipaddress
def select_route(table, dst):
d = ipaddress.ip_address(dst)
matching = [r for r in table if d in ipaddress.ip_network(r["prefix"])]
if not matching:
return None
# STEP 1: longest prefix wins outright.
longest = max(ipaddress.ip_network(r["prefix"]).prefixlen for r in matching)
finalists = [r for r in matching
if ipaddress.ip_network(r["prefix"]).prefixlen == longest]
# STEP 2 and 3: AD, then metric — only among the equally-specific finalists.
return min(finalists, key=lambda r: (ADMIN_DISTANCE[r["source"]], r["metric"]))
table = [
{"prefix": "10.1.5.0/24", "source": "rip", "metric": 8, "nh": "10.0.0.3"},
{"prefix": "10.1.0.0/16", "source": "ospf", "metric": 2, "nh": "10.0.0.2"},
]
r = select_route(table, "10.1.5.42")
print(f"\n10.1.5.42 -> {r['prefix']} via {r['source']} "
f"(AD {ADMIN_DISTANCE[r['source']]})")
print("the LEAST trusted protocol won, because its route was MORE SPECIFIC.")
# 10.1.5.42 -> 10.1.5.0/24 via rip (AD 120)
# the LEAST trusted protocol won, because its route was MORE SPECIFIC.
# OSPF cost, and the reference-bandwidth trap.
def ospf_cost(bw_mbps, reference_mbps=100):
return max(1, round(reference_mbps / bw_mbps)) # never below 1
LINKS = [("FastEthernet", 100), ("GigabitEthernet", 1000),
("TenGig", 10_000), ("HundredGig", 100_000)]
print(f"\n{'interface':<18}{'bandwidth':>12}{'cost @100':>11}{'cost @100000':>14}")
for name, bw in LINKS:
print(f"{name:<18}{bw:>9} Mb/s{ospf_cost(bw):>11}{ospf_cost(bw, 100_000):>14}")
# interface bandwidth cost @100 cost @100000
# FastEthernet 100 Mb/s 1 1000
# GigabitEthernet 1000 Mb/s 1 100
# TenGig 10000 Mb/s 1 10
# HundredGig 100000 Mb/s 1 1
# With the default reference, every modern link has cost 1: OSPF would pick
# a 100 Mb/s path over a 100 Gb/s path if it had one fewer hop.
# A floating static route: AD used deliberately as a tie-breaker.
print()
for ad, label in [(1, "normal static (AD 1)"), (200, "floating static (AD 200)")]:
ospf_ad = ADMIN_DISTANCE["ospf"]
winner = "STATIC installed" if ad < ospf_ad else "OSPF installed; static waits in reserve"
print(f"{label:<28} vs OSPF (AD {ospf_ad}) -> {winner}")
# normal static (AD 1) vs OSPF (AD 110) -> STATIC installed
# floating static (AD 200) vs OSPF (AD 110) -> OSPF installed; static waits in reserve
# Seeing AD and metric together — the bracketed pair is [AD/metric].
$ vtysh -c "show ip route 10.1.0.0/16"
# Routing entry for 10.1.0.0/16
# Known via "static", distance 1, metric 0, best
# ^^^^^^^^^^ ^^^^^^^^ ^^^^ installed in the FIB
# 10.0.0.4, via eth0
# Known via "ospf", distance 110, metric 20
# 10.0.0.2, via eth1
# ^ OSPF's route is KNOWN but not installed — it lost on AD.
$ ip route show | head -3
# 10.1.0.0/16 via 10.0.0.4 dev eth0 proto static metric 0
# 10.3.0.0/16 via 10.0.0.2 dev eth1 proto ospf metric 20
# Cisco notation: O = OSPF, the [110/20] is [AD/metric]
# O 10.3.0.0/16 [110/20] via 10.0.0.2, GigabitEthernet0/1
# S 10.1.0.0/16 [1/0] via 10.0.0.4
# B 172.16.0.0/16 [20/0] via 203.0.113.1
# Raising the OSPF reference bandwidth — MUST be identical on every router:
# router ospf 1
# auto-cost reference-bandwidth 100000 <- 100 Gb/s, in Mb/s units
# ! A mismatch here produces asymmetric routing that is very hard to diagnose.
$ vtysh -c "show ip ospf interface eth0" | grep -i cost
# Router ID 10.0.0.1, Network Type BROADCAST, Cost: 10
# ^^^^^^^ derived from bandwidth
# Changing AD to create a floating static backup:
# ip route 10.2.0.0 255.255.0.0 192.168.1.3 200
# ^^^ above OSPF's 110: dormant
# until the OSPF route is gone
Key takeaways
- Administrative distance compares across protocols — "which source do I trust?" — from 0 to 255, lower is better, and it is never advertised to other routers.
- Key values: connected 0, static 1, eBGP 20, EIGRP 90, OSPF 110, IS-IS 115, RIP 120, iBGP 200, unreachable 255.
- Metrics compare within one protocol and are not comparable across protocols.
- Evaluation order is longest prefix match → administrative distance → metric; specificity is resolved before trust.
- Consequently a RIP-learned /24 beats an OSPF-learned /16 for addresses inside the /24.
- AD is locally significant, which makes floating static routes (AD 200) a simple backup mechanism.
- Careless AD manipulation during redistribution causes routing loops; use route tags and filters instead.
- OSPF cost is
reference_bandwidth / interface_bandwidth; the default 100 Mb/s reference makes all modern links cost 1, so it must be raised identically on every router.
🧪 Practice
- A prefix is learned via eBGP (metric 0), OSPF (metric 5), and a static route (metric 0). State which is installed and why the metrics are irrelevant.
- Compute OSPF costs for 10 Mb/s, 1 Gb/s, and 40 Gb/s interfaces under reference bandwidths of 100 and 100,000, and explain why the default is dangerous.
- A router has
172.16.0.0/16via OSPF and172.16.8.0/24via RIP. Determine the path taken to172.16.8.9and to172.16.9.9.- Interview-style: Why does eBGP have an administrative distance of 20, lower than every IGP, while iBGP sits at 200? (Hint: consider who has authoritative knowledge of an external destination, and who has better knowledge of paths inside your own network.)
Routing Table Construction
Theory
The routing table is not a single structure that protocols write into directly. It is the output of a pipeline, and understanding the stages explains most "why isn't this route installed?" questions.
The pipeline has four stages:
1. Sources produce candidates. Connected networks appear automatically when an interface comes up with an address. Static routes come from configuration. Each dynamic protocol maintains its own database and selects its own best path within that protocol — OSPF runs Dijkstra over its link-state database, BGP runs its own selection process, RIP compares hop counts.
2. The RIB collects everything. The Routing Information Base holds every candidate from every source, including the losers. This is what show ip route displays.
3. Selection installs winners into the FIB. For each prefix, the best route is chosen by administrative distance, then metric, and pushed into the Forwarding Information Base — the hardware structure the data plane actually uses. Chapter 5 covered the RIB/FIB split; this is the process that bridges them.
4. Recursive resolution completes the entry. This stage is easy to overlook and causes real failures. A route may specify a next hop that is not directly connected — BGP routes typically do. The router must look up that next hop to find a directly connected interface and link-layer address. If the next hop is unresolvable, the route is not installed, no matter how good its AD and metric. A BGP route whose next hop has no IGP path is simply invalid and disappears from the table.
Several behaviours follow from this structure.
Connected routes appear and vanish with the interface. Bring an interface down and its connected route leaves the RIB immediately — and every route that recursively resolved through it becomes invalid too. A single interface failure can therefore invalidate many routes at once.
Route validity is checked continuously, not just at install time. This is what makes dynamic routing react at all.
The RIB is usually much larger than the FIB, because it retains all the alternatives. On a BGP router receiving full tables from three peers, the RIB may hold three million paths while the FIB holds one million prefixes.
Backup paths are already present. Because losing candidates stay in the RIB, failover after a link failure does not require relearning anything — the alternative is re-evaluated and installed. This is why convergence can be fast, and it is the basis of pre-computed schemes like Loop-Free Alternates (LFA) and BGP PIC, which pre-install the backup in the FIB so failover happens in hardware without waiting for the control plane.
Examples
The pipeline from protocol to hardware
SOURCES (each selects its own best first)
connected ---+
static ---+
OSPF (Dijkstra over its LSDB) ---+
BGP (its own 13-step selection) --+
RIP (lowest hop count) ---+
|
v
+--------------------------------------------------+
| RIB — every candidate, winners AND losers | 'show ip route'
| 10.1.0.0/16 static AD1 m0 <- best |
| 10.1.0.0/16 ospf AD110 m20 |
| 10.1.0.0/16 rip AD120 m3 |
+----------------------+---------------------------+
| AD, then metric
v
+--------------------------------------------------+
| RECURSIVE RESOLUTION |
| next hop directly connected? -> install |
| not connected? -> look IT up; unresolvable |
| -> route NOT installed |
+----------------------+---------------------------+
v
+--------------------------------------------------+
| FIB — one winner per prefix, in hardware | 'show ip cef'
| 10.1.0.0/16 -> 10.0.0.4, eth0, MAC aa:bb:.. |
+--------------------------------------------------+
Recursive resolution, two levels deep
BGP says: 192.0.2.0/24 via 10.255.0.7 (a loopback, NOT connected)
|
IGP says: 10.255.0.7/32 via 10.0.0.2 (still not connected)
|
connected: 10.0.0.0/30 on eth1 <-- RESOLVED
|
FINAL FIB entry: 192.0.2.0/24 -> out eth1, next-hop MAC of 10.0.0.2
If the IGP loses 10.255.0.7/32, the BGP route becomes UNRESOLVABLE
and is withdrawn — even though BGP itself never changed.
import ipaddress
ADMIN_DISTANCE = {"connected": 0, "static": 1, "ebgp": 20, "eigrp": 90,
"ospf": 110, "isis": 115, "rip": 120, "ibgp": 200}
class RoutingTable:
def __init__(self):
self.rib = [] # every candidate from every source
self.connected = {} # prefix -> interface
self.fib = {} # prefix -> resolved forwarding entry
def add_connected(self, prefix, interface):
self.connected[prefix] = interface
self.rib.append({"prefix": prefix, "source": "connected",
"metric": 0, "nh": "direct", "iface": interface})
self.rebuild()
def add_route(self, prefix, source, metric, nh):
self.rib.append({"prefix": prefix, "source": source,
"metric": metric, "nh": nh, "iface": None})
self.rebuild()
def interface_down(self, interface):
self.connected = {p: i for p, i in self.connected.items() if i != interface}
self.rib = [r for r in self.rib if r.get("iface") != interface]
self.rebuild() # routes resolving through it may now be invalid
def resolve(self, nh, depth=0):
"""Walk the next hop down to a connected interface. Recursive."""
if depth > 8:
return None # loop guard
if nh == "direct":
return "direct"
addr = ipaddress.ip_address(nh)
for prefix, iface in self.connected.items():
if addr in ipaddress.ip_network(prefix):
return iface # resolved
# Not directly connected: find a RIB route TO the next hop and recurse.
for r in sorted(self.rib, key=lambda r: ADMIN_DISTANCE[r["source"]]):
if addr in ipaddress.ip_network(r["prefix"]) and r["nh"] != nh:
return self.resolve(r["nh"], depth + 1)
return None # UNRESOLVABLE
def rebuild(self):
self.fib = {}
by_prefix = {}
for r in self.rib:
by_prefix.setdefault(r["prefix"], []).append(r)
for prefix, cands in by_prefix.items():
# AD, then metric.
best = min(cands, key=lambda r: (ADMIN_DISTANCE[r["source"]], r["metric"]))
iface = self.resolve(best["nh"])
if iface is None:
continue # UNRESOLVABLE -> not installed, whatever its AD
self.fib[prefix] = {"via": best["nh"], "iface": iface,
"source": best["source"]}
def show(self, label):
print(f"\n--- {label} ---")
print(f"RIB: {len(self.rib)} candidates FIB: {len(self.fib)} installed")
for prefix, e in sorted(self.fib.items()):
print(f" {prefix:<16} via {e['via']:<12} {e['iface']:<6} ({e['source']})")
unresolved = {r['prefix'] for r in self.rib} - set(self.fib)
for p in sorted(unresolved):
print(f" {p:<16} NOT INSTALLED — next hop unresolvable")
rt = RoutingTable()
rt.add_connected("10.0.0.0/30", "eth1")
rt.add_route("10.255.0.7/32", "ospf", 20, "10.0.0.2") # IGP path to a loopback
rt.add_route("192.0.2.0/24", "ebgp", 0, "10.255.0.7") # BGP next hop = loopback
rt.show("healthy: two-level recursion succeeds")
# --- healthy: two-level recursion succeeds ---
# RIB: 3 candidates FIB: 3 installed
# 10.0.0.0/30 via direct eth1 (connected)
# 10.255.0.7/32 via 10.0.0.2 eth1 (ospf)
# 192.0.2.0/24 via 10.255.0.7 eth1 (ebgp)
# ^^^^ resolved through TWO lookups
rt.interface_down("eth1")
rt.show("eth1 down: one interface invalidates everything above it")
# --- eth1 down: one interface invalidates everything above it ---
# RIB: 2 candidates FIB: 0 installed
# 10.255.0.7/32 NOT INSTALLED — next hop unresolvable
# 192.0.2.0/24 NOT INSTALLED — next hop unresolvable
# A single interface failure invalidated the IGP route AND the BGP route
# that recursed through it. BGP itself never sent a withdrawal.
# Why the RIB dwarfs the FIB on a BGP router.
def rib_vs_fib(prefixes, peers):
return prefixes * peers, prefixes
for peers in (1, 3, 5):
rib, fib = rib_vs_fib(1_000_000, peers)
print(f"\n{peers} full-table peers: RIB {rib:>9,} paths FIB {fib:>9,} prefixes"
f" ratio {rib//fib}:1")
# 1 full-table peers: RIB 1,000,000 paths FIB 1,000,000 prefixes ratio 1:1
# 3 full-table peers: RIB 3,000,000 paths FIB 1,000,000 prefixes ratio 3:1
# 5 full-table peers: RIB 5,000,000 paths FIB 1,000,000 prefixes ratio 5:1
# The losers are retained precisely so failover needs no relearning.
# The RIB — everything known, including routes that lost:
$ vtysh -c "show ip route 192.0.2.0/24"
# Routing entry for 192.0.2.0/24
# Known via "bgp", distance 20, metric 0, best
# Last update 00:04:12 ago
# 10.255.0.7 (recursive)
# ^^^^^^^^^^^^^^^^^^^ the next hop is NOT directly connected
# via 10.0.0.2, eth1
# ^^^^^^^^^^^^^^^^ what it recursively resolved to
# The FIB — what the hardware will actually do, fully resolved:
$ show ip cef 192.0.2.0/24 detail
# 192.0.2.0/24, epoch 2, flags [rib defined all labels]
# recursive via 10.255.0.7
# attached to GigabitEthernet0/1
# next hop 10.0.0.2, GigabitEthernet0/1
# ^ the recursion is FLATTENED here: the data plane does one lookup
# The classic failure: a BGP route present but not installed.
$ show ip bgp 192.0.2.0/24
# 10.255.0.7 (inaccessible) from 10.255.0.7 (10.255.0.7)
# ^^^^^^^^^^^^^^ the IGP has no path to the next hop
# Origin IGP, localpref 100, valid, external
# ^ "valid" as a BGP path, but NOT marked "best" and NOT in the FIB.
# FIX: ensure the IGP advertises the loopback, or use 'next-hop-self'.
# Watching the RIB react to an interface going down:
$ sudo ip link set eth1 down
$ ip route show | grep -c . # connected route and everything recursing
# through it vanish together
# Pre-installed backups: failover in hardware, no control plane involved.
# router ospf 1
# fast-reroute per-prefix enable prefix-priority high
# ^^^^^^^^^^^^^^^^^ Loop-Free Alternates precomputed into the FIB
$ show ip cef 10.3.0.0/16 detail | grep -A1 repair
# repair: attached-nexthop 10.0.0.9 GigabitEthernet0/2
# ^^^^^^^^^^^^^^^^ the backup is ALREADY in hardware
Key takeaways
- The table is built by a pipeline: sources → RIB → selection (AD, then metric) → recursive resolution → FIB.
- Each protocol runs its own best-path algorithm first, and submits only its own winner to the RIB.
- The RIB holds every candidate including losers; the FIB holds one resolved winner per prefix.
- Recursive resolution walks a non-connected next hop down to a connected interface; an unresolvable next hop means the route is not installed, regardless of AD or metric.
- A BGP route whose next hop has no IGP path shows as
(inaccessible)and never enters the FIB — commonly fixed withnext-hop-self. - Bringing an interface down removes its connected route and invalidates every route that recursed through it.
- On a multi-peer BGP router the RIB can be several times the FIB, because alternatives are retained.
- Retained losers make failover fast, and schemes like LFA and BGP PIC pre-install backups directly in hardware.
🧪 Practice
- List the four pipeline stages in order and state what can cause a route with the best AD and metric to still not appear in the FIB.
- Trace the recursive resolution of a BGP route with next hop
10.255.0.9, given an OSPF route10.255.0.9/32 via 10.0.0.6and a connected10.0.0.4/30oneth2.- Explain why one interface going down can invalidate dozens of BGP routes even though no BGP message was exchanged.
- Interview-style: A BGP route appears in
show ip bgpas valid but traffic does not use it, and it is absent from the FIB. What is the most likely cause? (Hint: BGP next hops are frequently loopbacks — ask which protocol is responsible for reaching them.)
Equal-Cost Multi-Path Routing
Theory
When several paths to the same destination have identical cost, a router need not choose one and waste the others. ECMP (Equal-Cost Multi-Path) installs all of them and distributes traffic across them.
The benefits are straightforward: aggregate bandwidth (four 10 Gb/s paths behave like 40 Gb/s), load distribution, and instant failover — if one path dies, the others are already installed and traffic simply redistributes. ECMP is the foundation of modern data centre design, where leaf-spine topologies deliberately create many equal-cost paths between any two servers.
The critical design question is how to distribute packets across the paths, and the naive answer is wrong.
Per-packet load balancing — alternate paths packet by packet — gives perfect distribution and is almost never used. Different paths have different latencies, so packets of the same TCP flow arrive out of order. TCP interprets out-of-order arrival as loss, triggers duplicate ACKs and fast retransmit, and throughput collapses. Perfect load balancing produces terrible performance.
Per-flow load balancing is what real routers do. The router hashes a set of header fields — typically the 5-tuple (source IP, destination IP, source port, destination port, protocol) — and uses the hash to select a path. Since every packet of a flow hashes identically, every packet of a flow takes the same path, preserving order. Different flows land on different paths, spreading the load statistically.
The trade-offs of hashing are worth knowing:
Elephant flows. Hashing balances flows, not bytes. One enormous flow occupies exactly one path, so a single 10 Gb/s backup stream can saturate one link while parallel links sit idle. Flowlet switching and adaptive schemes (CONGA, and various load-aware fabrics) exist to address this.
Polarisation. If every router in a multi-stage topology uses the same hash function with the same inputs, they all make correlated decisions and traffic concentrates on a subset of paths. The fix is to seed each device's hash differently — most implementations now include the device ID or a random seed by default.
Small flow counts distribute poorly. With 8 paths and 16 flows, the expected distribution is uneven, and some paths may receive nothing. Hashing works well only when flows are numerous.
Encapsulation hides the tuple. A tunnel presents the same outer 5-tuple for all inner traffic, so everything hashes to one path. This is exactly why IPv6's Flow Label and VXLAN's practice of deriving the outer UDP source port from the inner flow's hash exist — they restore entropy that encapsulation removed.
ECMP requires exactly equal cost. Paths that differ by any amount are not eligible. EIGRP is the notable exception: its variance command enables unequal-cost load balancing, distributing traffic in proportion to metric across paths within a multiplier of the best. No other common IGP offers this.
Examples
Leaf-spine: equal-cost paths by design
[spine1] [spine2] [spine3] [spine4]
| | | | | | | | | | | |
+----+-+-+----+-+-+----+-+-+----+-+-+
| | | | |
[leaf1] [leaf2] [leaf3] [leaf4] ...
| |
servers servers
leaf1 -> leaf3: FOUR equal-cost paths (one per spine).
ECMP uses all four: 4x bandwidth, and a spine failure costs 25%
of capacity rather than causing an outage.
Per-packet vs per-flow — why the "better" option is worse
PER-PACKET (never do this):
flow X packets: 1 -> path A (10 ms)
2 -> path B (14 ms)
3 -> path A (10 ms)
arrival order at the receiver: 1, 3, 2
^^^^ OUT OF ORDER
TCP sees duplicate ACKs -> fast retransmit -> cwnd collapses.
Perfect balance, terrible throughput.
PER-FLOW (what routers actually do):
hash(5-tuple) -> path index
flow X: ALL packets -> path A order preserved
flow Y: ALL packets -> path B order preserved
flow Z: ALL packets -> path A
Statistically balanced, order intact.
The elephant flow problem
path A: [========= 9.5 Gb/s backup stream =========] SATURATED
path B: [= 0.2 =] idle
path C: [] idle
path D: [= 0.1 =] idle
Hashing balances FLOWS, not BYTES. One flow = one path, always.
import hashlib
def ecmp_hash(src_ip, dst_ip, src_port, dst_port, proto, num_paths, seed=0):
"""Hash the 5-tuple; the same flow always maps to the same path."""
key = f"{seed}|{src_ip}|{dst_ip}|{src_port}|{dst_port}|{proto}"
digest = hashlib.md5(key.encode()).digest()
return int.from_bytes(digest[:4], "big") % num_paths
# Every packet of one flow lands on the same path — this is the whole point.
flow = ("192.168.1.20", "10.0.5.9", 51234, 443, "tcp")
paths = [ecmp_hash(*flow, num_paths=4) for _ in range(5)]
print(f"same flow, 5 packets -> paths {paths} (all identical: order preserved)")
# same flow, 5 packets -> paths [3, 3, 3, 3, 3] (all identical: order preserved)
# Different flows spread across the paths.
print()
for sport in (51234, 51235, 51236, 51237, 51238):
p = ecmp_hash("192.168.1.20", "10.0.5.9", sport, 443, "tcp", num_paths=4)
print(f" src port {sport} -> path {p}")
# src port 51234 -> path 3
# src port 51235 -> path 1
# src port 51236 -> path 0
# src port 51237 -> path 2
# src port 51238 -> path 3
# Distribution quality depends heavily on flow COUNT.
from collections import Counter
def distribution(num_flows, num_paths):
c = Counter(ecmp_hash("10.0.0.1", "10.0.9.9", 20000 + i, 443, "tcp", num_paths)
for i in range(num_flows))
counts = [c.get(p, 0) for p in range(num_paths)]
ideal = num_flows / num_paths
worst_dev = max(abs(x - ideal) for x in counts) / ideal * 100
return counts, round(worst_dev, 1)
print()
for n in (8, 64, 1000, 100_000):
counts, dev = distribution(n, 4)
print(f"{n:>7,} flows over 4 paths: {str(counts):<34} worst deviation {dev:>5.1f}%")
# 8 flows over 4 paths: [2, 2, 1, 3] worst deviation 50.0%
# 64 flows over 4 paths: [15, 15, 17, 17] worst deviation 6.2%
# 1,000 flows over 4 paths: [252, 253, 242, 253] worst deviation 3.2%
# 100,000 flows over 4 paths: [25041, 25173, 24943, 24843] worst deviation 0.7%
# ECMP is a statistical mechanism: it needs MANY flows to balance well.
# Elephant flows: bytes are not balanced, only flows are.
def byte_distribution(flows, num_paths):
load = [0.0] * num_paths
for name, mbps, tup in flows:
load[ecmp_hash(*tup, num_paths=num_paths)] += mbps
return load
flows = [("backup", 9500, ("10.0.0.5", "10.0.9.9", 40001, 873, "tcp")),
("web1", 120, ("10.0.0.6", "10.0.9.9", 40002, 443, "tcp")),
("web2", 80, ("10.0.0.7", "10.0.9.9", 40003, 443, "tcp")),
("db", 200, ("10.0.0.8", "10.0.9.9", 40004, 5432, "tcp"))]
load = byte_distribution(flows, 4)
print()
for i, mbps in enumerate(load):
bar = "#" * int(mbps / 250)
print(f"path {i}: {mbps:>7.0f} Mb/s {bar}")
print("one elephant flow pins an entire path; the rest sit nearly idle.")
# path 0: 0 Mb/s
# path 1: 80 Mb/s
# path 2: 9700 Mb/s ######################################
# path 3: 120 Mb/s
# one elephant flow pins an entire path; the rest sit nearly idle.
# Polarisation: identical hashes at every stage concentrate traffic.
print()
for seeded in (False, True):
stage1 = [ecmp_hash("10.0.0.1", "10.0.9.9", 20000+i, 443, "tcp", 4, seed=0)
for i in range(2000)]
stage2 = [ecmp_hash("10.0.0.1", "10.0.9.9", 20000+i, 443, "tcp", 4,
seed=7 if seeded else 0) for i in range(2000)]
correlated = sum(1 for a, b in zip(stage1, stage2) if a == b)
label = "different seed per device" if seeded else "SAME hash everywhere"
print(f"{label:<28} stage-2 decisions matching stage 1: {correlated/2000:.0%}")
# SAME hash everywhere stage-2 decisions matching stage 1: 100%
# different seed per device stage-2 decisions matching stage 1: 25%
# 100% correlation IS polarisation: the second stage adds no spreading at all.
# ECMP in the routing table: one prefix, several next hops.
$ ip route show 10.0.5.0/24
# 10.0.5.0/24 proto ospf metric 20
# nexthop via 10.1.1.2 dev eth1 weight 1
# nexthop via 10.1.2.2 dev eth2 weight 1
# nexthop via 10.1.3.2 dev eth3 weight 1
# nexthop via 10.1.4.2 dev eth4 weight 1
# ^ four equal-cost next hops, all installed and all in use
# Which path will a specific flow take? Ask the kernel.
$ ip route get 10.0.5.9 from 192.168.1.20 ipproto tcp sport 51234 dport 443
# 10.0.5.9 from 192.168.1.20 via 10.1.2.2 dev eth2
# ^^^^^^^^ deterministic for this exact 5-tuple
# Choose the hash inputs (L4 gives far better entropy than L3 alone):
$ sysctl net.ipv4.fib_multipath_hash_policy
# net.ipv4.fib_multipath_hash_policy = 1 # 0 = L3 only, 1 = L3+L4 (5-tuple)
# Maximum paths — protocols cap this, and the default is often just 4:
# router ospf 1
# maximum-paths 8
# router bgp 64500
# maximum-paths 4 <- eBGP multipath, off by default
# EIGRP's unique capability: UNEQUAL-cost load balancing.
# router eigrp 1
# variance 2
# ^ use any path whose metric is within 2x the best, proportionally.
# No other common IGP can do this.
# Verifying the distribution is real, not theoretical:
$ for i in $(seq 1 4); do
> ethtool -S eth$i | grep -w rx_packets
> done
# rx_packets: 184221934 # eth1
# rx_packets: 179882201 # eth2
# rx_packets: 181044887 # eth3
# rx_packets: 183991002 # eth4
# Within ~2% — healthy. A wildly skewed column suggests an elephant flow
# or hash polarisation upstream.
Key takeaways
- ECMP installs all equal-cost paths and distributes traffic across them, giving aggregate bandwidth and instant failover from already-installed alternatives.
- Per-packet balancing causes out-of-order delivery, which TCP reads as loss — perfect balance, collapsed throughput.
- Per-flow hashing on the 5-tuple keeps each flow on one path, preserving order while spreading load statistically.
- ECMP balances flows, not bytes: a single elephant flow saturates one path while others idle.
- Polarisation occurs when every device hashes identically; per-device seeds fix it.
- Distribution is poor with few flows and excellent with many — it is a statistical mechanism.
- Encapsulation hides the inner tuple, collapsing entropy; the IPv6 Flow Label and VXLAN's hashed outer source port restore it.
- Costs must be exactly equal; EIGRP's
varianceis the only common IGP mechanism for unequal-cost load balancing. - Leaf-spine data centre topologies are built specifically to create many equal-cost paths.
🧪 Practice
- Explain why per-packet load balancing degrades TCP throughput despite distributing traffic perfectly evenly.
- A leaf-spine fabric has 6 spines. State the number of equal-cost paths between two leaves and the capacity lost when one spine fails.
- A 10 Gb/s backup runs over 4 ECMP paths, yet one link is saturated while three are idle. Explain the cause and name one mitigation.
- Interview-style: Why does traffic inside a VXLAN tunnel often fail to load-balance across an ECMP fabric, and how is this solved? (Hint: ask which 5-tuple the transit routers can actually see, and where the encapsulator could inject entropy.)
<a id="62-routing-algorithms"></a>
6.2 Routing Algorithms
This subchapter covers the mathematics underneath every routing protocol: how a network becomes a graph, the two algorithm families that compute shortest paths over it, and the pathologies that arise when the computation is distributed across machines that disagree.
Graph Abstraction of Networks
Theory
Before any algorithm can run, the network must be turned into something an algorithm can operate on. That representation is a graph: routers become vertices (nodes), links become edges, and each edge carries a weight (cost).
The abstraction deliberately discards almost everything. It does not represent cable types, frame formats, vendors, or physical geography. It keeps exactly two facts: what connects to what, and what each connection costs. Every routing protocol, from RIP to BGP, is ultimately computing over this structure.
The edge weight is where policy enters, and it is worth dwelling on because it is the main tuning knob a network operator has. "Shortest path" does not mean shortest in distance — it means lowest total weight, and the weight means whatever the operator decides:
| Weight basis | Result | Used by |
|---|---|---|
| All edges = 1 | Fewest hops | RIP |
reference_bw / bandwidth | Prefer faster links | OSPF, IS-IS |
| Measured latency | Prefer low-delay paths | Some SDN controllers |
| Monetary cost | Prefer cheap transit | Policy routing, BGP-adjacent |
| Manual values | Prefer whatever you want | Traffic engineering |
Since the algorithm optimises the numbers you give it, choosing the weights is choosing the paths. A common failure in practice is leaving OSPF's default reference bandwidth in place (Chapter 6.1), which makes every modern link weigh the same and quietly reduces the shortest-path computation to hop counting.
Several graph properties map directly to network realities.
Directed vs undirected. Most textbook examples treat links as undirected with one weight. Real links can have asymmetric costs — different weights in each direction — which produces asymmetric routing, where traffic takes one path out and a different path back. This is completely normal on the internet and is why traceroute shows only half the story.
Connectivity. A connected graph has a path between every pair. A partitioned graph does not, and that is exactly a network split.
Cut vertices and bridges. A vertex whose removal disconnects the graph is a single point of failure — an articulation point. An edge with the same property is a bridge. Identifying them is a formal way of asking "what breaks the network if it fails?"
Degree. A node's edge count is its redundancy. Degree 1 means a stub with no alternative.
Two representations are used in practice, and the choice matters at scale:
- An adjacency matrix is an N×N array — O(N²) memory, O(1) edge lookup. Fine for dense graphs.
- An adjacency list stores each node's neighbours — O(N+E) memory. Real networks are sparse (a router has tens of neighbours, not thousands), so this is what implementations use.
One final property drives algorithm design: a shortest path never revisits a node. If it did, the loop could be removed and the path would be shorter or equal. Therefore shortest paths are simple paths, and the set of shortest paths from one source forms a tree — the shortest-path tree. Both Dijkstra and Bellman-Ford exploit this.
Examples
A network, and the graph it becomes
PHYSICAL GRAPH
(A)
[RouterA]---1Gb---[RouterB] / | \
| | 1 / | \ 5
100Mb 10Gb / |4 \
| | (B) | (C)
[RouterC]---1Gb---[RouterD] \ | /
2 \ | / 3
\ | /
(D)
Vertices: routers. Edges: links. Weights: administrative cost.
Discarded: cable type, vendor, geography, frame format.
The same topology, three weightings, three different "shortest" paths
A --10-- B --10-- D (2 hops, weight 20)
A --------1------- D (1 hop, weight 1, but a slow 10 Mb/s link)
weights = hop count (RIP): A-D direct wins (1 hop)
weights = 100/bandwidth (OSPF): A-B-D wins if the direct link is slow
(10 Mb/s -> cost 10 vs two 1 Gb/s -> 1+1)
The ALGORITHM is identical. Only the numbers changed.
Single points of failure, formally
(A)---(B)---(C)---(D)
|
(E)
Removing C disconnects {D,E} from {A,B} -> C is a CUT VERTEX
Removing edge B-C does the same -> B-C is a BRIDGE
deg(A)=1 -> A is a stub with no redundancy
deg(C)=3 -> the most connected, and the most critical
from collections import defaultdict, deque
class NetworkGraph:
def __init__(self):
self.adj = defaultdict(dict) # adjacency LIST: sparse-friendly
def add_link(self, u, v, cost, reverse_cost=None):
"""Undirected by default; pass reverse_cost for an asymmetric link."""
self.adj[u][v] = cost
self.adj[v][u] = cost if reverse_cost is None else reverse_cost
def neighbors(self, n):
return self.adj[n]
def degree(self, n):
return len(self.adj[n])
def connected(self):
"""BFS from an arbitrary node; did we reach everything?"""
if not self.adj:
return True
start = next(iter(self.adj))
seen, q = {start}, deque([start])
while q:
for nb in self.adj[q.popleft()]:
if nb not in seen:
seen.add(nb)
q.append(nb)
return len(seen) == len(self.adj)
def cut_vertices(self):
"""Nodes whose removal partitions the graph — single points of failure."""
cuts = []
for node in list(self.adj):
saved = {n: dict(nbrs) for n, nbrs in self.adj.items()}
for nb in list(self.adj[node]): # remove the node
del self.adj[nb][node]
del self.adj[node]
if self.adj and not self.connected():
cuts.append(node)
self.adj = defaultdict(dict, saved) # restore
return cuts
g = NetworkGraph()
for u, v, c in [("A","B",1), ("B","C",1), ("C","D",1), ("C","E",1)]:
g.add_link(u, v, c)
print("connected: ", g.connected())
print("degrees: ", {n: g.degree(n) for n in sorted(g.adj)})
print("cut vertices: ", g.cut_vertices())
# connected: True
# degrees: {'A': 1, 'B': 2, 'C': 3, 'D': 1, 'E': 1}
# cut vertices: ['B', 'C']
# B and C are single points of failure. A, D, E are degree-1 stubs.
# Asymmetric costs produce asymmetric routing — normal on the internet.
a = NetworkGraph()
a.add_link("X", "Y", cost=10, reverse_cost=50) # cheap out, expensive back
print(f"\nX -> Y costs {a.adj['X']['Y']}")
print(f"Y -> X costs {a.adj['Y']['X']} <- traceroute shows only ONE direction")
# X -> Y costs 10
# Y -> X costs 50 <- traceroute shows only ONE direction
# Why real implementations use adjacency LISTS.
def memory_model(nodes, avg_degree):
matrix = nodes * nodes # O(N^2)
lst = nodes + nodes * avg_degree # O(N + E)
return matrix, lst, matrix / lst
print()
for n in (10, 100, 10_000):
m, l, ratio = memory_model(n, avg_degree=4)
print(f"{n:>6,} nodes (avg degree 4): matrix {m:>12,} list {l:>8,} "
f"matrix is {ratio:>7,.0f}x larger")
# 10 nodes (avg degree 4): matrix 100 list 50 matrix is 2x larger
# 100 nodes (avg degree 4): matrix 10,000 list 500 matrix is 20x larger
# 10,000 nodes (avg degree 4): matrix 100,000,000 list 50,000 matrix is 2,000x larger
# Real networks are SPARSE, so the list representation wins decisively.
# The weights ARE the policy: same graph, different answers.
def total_cost(graph, path, weight_fn):
return sum(weight_fn(graph, path[i], path[i+1]) for i in range(len(path)-1))
t = NetworkGraph()
t.add_link("A", "D", 10) # direct but slow: 10 Mb/s
t.add_link("A", "B", 1) # 1 Gb/s
t.add_link("B", "D", 1) # 1 Gb/s
hop = lambda g, u, v: 1
ospf = lambda g, u, v: g.adj[u][v]
print()
for name, fn in [("hop count (RIP)", hop), ("bandwidth cost (OSPF)", ospf)]:
direct = total_cost(t, ["A","D"], fn)
via_b = total_cost(t, ["A","B","D"], fn)
winner = "A-D direct" if direct <= via_b else "A-B-D"
print(f"{name:<24} A-D={direct:<3} A-B-D={via_b:<3} -> {winner}")
# hop count (RIP) A-D=1 A-B-D=2 -> A-D direct
# bandwidth cost (OSPF) A-D=10 A-B-D=2 -> A-B-D
# Identical topology, opposite conclusions, purely from the weighting.
# The graph a real protocol holds — OSPF's link-state database IS the graph.
$ vtysh -c "show ip ospf database router"
# Router Link States (Area 0.0.0.0)
# LS age: 421
# Link State ID: 10.0.0.1 <- a VERTEX
# Number of Links: 3
# Link connected to: another Router (point-to-point)
# (Link ID) Neighboring Router ID: 10.0.0.2 <- an EDGE
# TOS 0 Metrics: 10 <- its WEIGHT
# Link connected to: another Router (point-to-point)
# (Link ID) Neighboring Router ID: 10.0.0.3
# TOS 0 Metrics: 100
# Every router in an area holds an IDENTICAL copy of this graph:
$ vtysh -c "show ip ospf database" | grep -c "Link State ID"
# 47 # 47 vertices; run this on any router in the area
# and you get the same 47. That is the key invariant.
# The computed shortest-path tree:
$ vtysh -c "show ip ospf route"
# ============ OSPF network routing table ============
# N 10.1.0.0/24 [10] area: 0.0.0.0
# directly attached to eth0
# N 10.3.0.0/24 [20] area: 0.0.0.0
# via 10.0.0.2, eth1
# ^^^^^^^^^^^^ the next hop on the shortest-path tree
# Asymmetric routing, visible as a traceroute that does not match the reverse:
$ traceroute -n 10.9.0.1 # forward path: 4 hops via R2
$ ssh 10.9.0.1 traceroute -n $(hostname -I | cut -d' ' -f1)
# # return path: 6 hops via R7 — perfectly normal
Key takeaways
- A network becomes a graph: routers are vertices, links are edges, costs are edge weights; everything else is discarded.
- "Shortest path" means lowest total weight, not fewest hops or least distance — choosing the weights chooses the paths.
- Weight bases include hop count (RIP),
reference_bw / bandwidth(OSPF, IS-IS), latency, and manual traffic-engineering values. - Asymmetric edge costs produce asymmetric routing, which is normal and is why traceroute shows only the forward path.
- Cut vertices and bridges are the formal names for single points of failure; degree measures a node's redundancy.
- Real networks are sparse, so implementations use adjacency lists (O(N+E)) rather than matrices (O(N²)).
- A shortest path never revisits a node, so shortest paths are simple and form a shortest-path tree from any source.
- OSPF's link-state database is literally this graph, and every router in an area holds an identical copy.
🧪 Practice
- Draw the graph for four routers in a square with one diagonal, assign weights, and identify every cut vertex and bridge.
- Given a direct 10 Mb/s link and a two-hop path of 1 Gb/s links, compute the winner under hop-count weighting and under OSPF cost weighting.
- Compare adjacency-matrix and adjacency-list memory for 5,000 routers averaging 6 neighbours, and state which a protocol implementation uses.
- Interview-style: Two routers disagree about the best path to the same destination, yet both are running correctly. Give a graph-level explanation. (Hint: they are not computing from the same source vertex — and the edge weights may not be the same in both directions.)
Dijkstra's Link-State Algorithm
Theory
Dijkstra's algorithm computes the shortest path from one source to every other node in a graph with non-negative weights. It is the algorithm inside OSPF and IS-IS, which is why both are called link-state protocols.
The prerequisite is the defining feature of the link-state approach: every router must know the entire topology. Routers flood descriptions of their own links to every other router, so each builds an identical link-state database (LSDB) — the complete graph. Then each router independently runs Dijkstra with itself as the source.
This is worth pausing on, because it is what separates link-state from distance-vector. Every router has the same map and computes different routes from it, because each computes from its own position. Nobody is told what the paths are; everybody is told what the network looks like and works out its own paths. Since all routers reason from identical data, they reach consistent conclusions — which is why link-state protocols do not suffer the count-to-infinity problem covered later.
The algorithm maintains a tentative distance to each node and a set of settled nodes whose shortest distance is final:
- Set the source's distance to 0 and every other node's to infinity.
- Pick the unsettled node with the smallest tentative distance and settle it.
- For each of its neighbours, check whether going through the settled node is cheaper than the neighbour's current tentative distance. If so, relax the edge: update the distance and record the predecessor.
- Repeat until every node is settled.
The greedy choice is the crux, and it rests on a specific argument: when you select the unsettled node with the smallest tentative distance, no shorter path to it can exist. Any alternative route would have to pass through some other unsettled node, which by construction is already at least as far — and since all weights are non-negative, going through it cannot reduce the total. The greedy choice is therefore provably correct.
That argument collapses if weights can be negative, which is exactly why Dijkstra requires non-negative weights. Network costs are never negative, so this constraint is free in practice.
Complexity depends on the priority queue:
| Implementation | Complexity | Notes |
|---|---|---|
| Array scan | O(V²) | Fine for dense graphs |
| Binary heap | O((V+E) log V) | What real implementations use |
| Fibonacci heap | O(E + V log V) | Better asymptotically, worse constants |
For a 1,000-router area with 4,000 links, a heap implementation runs in single-digit milliseconds — which is why OSPF can recompute on every topology change.
The practical properties that follow:
Fast, loop-free convergence. All routers compute from the same map, so their results are mutually consistent and transient loops are rare and short.
Memory and CPU scale with topology size, not with the number of destinations. This is why OSPF uses areas — they partition the LSDB so no single router must hold the whole network's graph.
Every topology change triggers a recomputation, so an unstable link ("flapping") causes repeated SPF runs. Real implementations use SPF throttling (exponential backoff) and incremental SPF to avoid burning CPU on instability.
Examples
Dijkstra from A, step by step
Graph: (B)---2---(D)
/ | |
1/ |5 |1
/ | |
(A)---4---(C)---3---(E)
step settled tentative distances action
---- ------- ---------------------------- ---------------------------
0 {} A=0 B=inf C=inf D=inf E=inf start
1 {A} A=0 B=1 C=4 D=inf E=inf settle A (min=0), relax A-B, A-C
2 {A,B} A=0 B=1 C=4 D=3 E=inf settle B (min=1), relax B-D (1+2=3)
B-C would be 1+5=6 > 4, no change
3 {A,B,D} A=0 B=1 C=4 D=3 E=4 settle D (min=3), relax D-E (3+1=4)
4 {A,B,D,C} A=0 B=1 C=4 D=3 E=4 settle C (min=4), C-E = 4+3=7 > 4
5 all FINAL: A=0 B=1 C=4 D=3 E=4
Shortest-path TREE from A: Routing table at A:
A dest cost next hop
/ \ B 1 B
B C C 4 C
| D 3 B <- via B, not direct
D E 4 B <- via B, then D
|
E
Why the greedy choice is safe
About to settle D with tentative distance 3.
Could a shorter path to D exist? It would have to route through some
other UNSETTLED node — but every unsettled node has distance >= 3,
and all edge weights are >= 0, so any such path costs >= 3.
Therefore 3 is final. This argument FAILS with negative weights.
import heapq
def dijkstra(graph, source):
"""Shortest paths from source to all nodes. graph: {node: {neighbor: cost}}"""
dist = {n: float("inf") for n in graph}
prev = {n: None for n in graph}
dist[source] = 0
settled = set()
pq = [(0, source)] # (tentative distance, node)
while pq:
d, u = heapq.heappop(pq)
if u in settled: # a stale queue entry
continue
settled.add(u) # GREEDY: this distance is now final
for v, w in graph[u].items():
if v in settled:
continue
nd = d + w # cost of reaching v through u
if nd < dist[v]: # RELAX the edge
dist[v] = nd
prev[v] = u
heapq.heappush(pq, (nd, v))
return dist, prev
def path_to(prev, dest):
p, cur = [], dest
while cur is not None:
p.append(cur)
cur = prev[cur]
return list(reversed(p))
def next_hop(prev, source, dest):
"""The FIB needs only the first hop, not the whole path."""
p = path_to(prev, dest)
return p[1] if len(p) > 1 else "direct"
G = {
"A": {"B": 1, "C": 4},
"B": {"A": 1, "C": 5, "D": 2},
"C": {"A": 4, "B": 5, "E": 3},
"D": {"B": 2, "E": 1},
"E": {"C": 3, "D": 1},
}
dist, prev = dijkstra(G, "A")
print(f"{'dest':<6}{'cost':>6}{'next hop':>10} full path")
for n in sorted(G):
if n == "A":
continue
print(f"{n:<6}{dist[n]:>6}{next_hop(prev,'A',n):>10} {' -> '.join(path_to(prev,n))}")
# dest cost next hop full path
# B 1 B A -> B
# C 4 C A -> C
# D 3 B A -> B -> D
# E 4 B A -> B -> D -> E
# Note E: cost 4 via B-D, NOT via the seemingly direct A-C-E (which costs 7).
# Every router computes from the SAME graph but a DIFFERENT source.
print()
for src in sorted(G):
d, p = dijkstra(G, src)
table = {n: next_hop(p, src, n) for n in sorted(G) if n != src}
print(f"router {src} computes: {table}")
# router A computes: {'B': 'B', 'C': 'C', 'D': 'B', 'E': 'B'}
# router B computes: {'A': 'A', 'C': 'C', 'D': 'D', 'E': 'D'}
# router C computes: {'A': 'A', 'B': 'B', 'D': 'E', 'E': 'E'}
# router D computes: {'A': 'B', 'B': 'B', 'C': 'E', 'E': 'E'}
# router E computes: {'A': 'D', 'B': 'D', 'C': 'C', 'D': 'D'}
# Same map, different vantage points -> consistent, loop-free routing.
# Verify consistency: following next hops must reach the destination
# without looping. This is the property link-state protocols guarantee.
def trace(graph, src, dst):
hops, cur, seen = [src], src, {src}
while cur != dst:
_, p = dijkstra(graph, cur)
cur = next_hop(p, cur, dst)
if cur in seen:
return hops + [cur], "LOOP DETECTED"
seen.add(cur)
hops.append(cur)
return hops, "delivered"
print()
for dst in ["C", "E"]:
hops, status = trace(G, "A", dst)
print(f"A -> {dst}: {' -> '.join(hops)} ({status})")
# A -> C: A -> C (delivered)
# A -> E: A -> B -> D -> E (delivered)
# Complexity: why OSPF can afford to recompute on every change.
import math
def spf_cost(v, e):
return v * v, (v + e) * math.log2(v)
print()
for v, e in [(50, 150), (1000, 4000), (10_000, 40_000)]:
arr, heap = spf_cost(v, e)
print(f"{v:>6,} nodes, {e:>6,} edges: array O(V^2)={arr:>12,.0f} "
f"heap O((V+E)logV)={heap:>10,.0f} speedup {arr/heap:>6,.0f}x")
# 50 nodes, 150 edges: array O(V^2)= 2,500 heap O((V+E)logV)= 1,129 speedup 2x
# 1,000 nodes, 4,000 edges: array O(V^2)= 1,000,000 heap O((V+E)logV)= 49,829 speedup 20x
# 10,000 nodes, 40,000 edges: array O(V^2)= 100,000,000 heap O((V+E)logV)= 664,386 speedup 151x
# The LSDB is the graph; SPF is Dijkstra over it.
$ vtysh -c "show ip ospf database" | head -6
# OSPF Router with ID (10.0.0.1)
# Router Link States (Area 0.0.0.0)
# Link ID ADV Router Age Seq# CkSum Link count
# 10.0.0.1 10.0.0.1 332 0x80000009 0x9a2b 3
# 10.0.0.2 10.0.0.2 341 0x80000007 0x3c11 3
# 10.0.0.3 10.0.0.3 298 0x80000005 0x7d44 2
# Verify every router in the area has an IDENTICAL database — the invariant
# that makes independent computation produce consistent results:
$ for r in 10.0.0.1 10.0.0.2 10.0.0.3; do
> ssh $r "vtysh -c 'show ip ospf database' | md5sum"
> done
# 4f2a... -
# 4f2a... - <- identical
# 4f2a... - <- identical
# How often SPF actually runs, and how long it takes:
$ vtysh -c "show ip ospf" | grep -i spf
# SPF algorithm executed 47 times
# SPF algorithm last executed 2m18s ago
# Last SPF duration 3.421 msec
# ^^^^^^^^^^ milliseconds for a real area — cheap enough
# to rerun on every topology change
# SPF throttling: exponential backoff protects the CPU from a flapping link.
# router ospf 1
# timers throttle spf 50 200 5000
# ^^ ^^^ ^^^^ initial delay, hold, max hold (ms)
# ! 1st change: wait 50 ms. Rapid successive changes: 200, 400, 800 ...
# ! capped at 5000 ms. Stability returns -> reset to 50 ms.
$ vtysh -c "show ip ospf route" | head -4
# ============ OSPF network routing table ============
# N 10.0.12.0/24 [10] area: 0.0.0.0
# directly attached to eth1
# N 10.0.34.0/24 [30] area: 0.0.0.0
# via 10.0.12.2, eth1 <- Dijkstra's output
Key takeaways
- Dijkstra computes shortest paths from one source to all nodes, and is the algorithm inside OSPF and IS-IS.
- Link-state protocols flood link descriptions so every router builds an identical LSDB — the complete graph — then each runs Dijkstra from itself.
- Same map, different source, different routes: consistent results without any router being told what the paths are.
- The algorithm repeatedly settles the nearest unsettled node and relaxes its edges.
- The greedy choice is provably correct only because weights are non-negative — any alternative path must pass through a node already at least as far.
- Complexity with a binary heap is O((V+E) log V); a 1,000-router area recomputes in single-digit milliseconds.
- Resource use scales with topology size, not destination count, which is why OSPF uses areas to partition the LSDB.
- Every change triggers SPF, so SPF throttling (exponential backoff) and incremental SPF protect the CPU from flapping links.
🧪 Practice
- Run Dijkstra by hand from node A on: A-B=2, A-C=1, B-D=3, C-D=5, C-E=2, D-E=1. Give the settle order, final costs, and the next hop for each destination.
- Explain why Dijkstra requires non-negative weights, describing precisely which step of the correctness argument fails otherwise.
- Explain how five routers with the same LSDB produce five different routing tables, and why those tables cannot contain a loop.
- Interview-style: A link flaps every two seconds and router CPU spikes. Which mechanism prevents this from overwhelming the control plane, and what is the trade-off? (Hint: the protection delays recomputation — ask what that costs during a genuine, single failure.)
Bellman-Ford Distance-Vector Algorithm
Theory
Dijkstra requires the whole map. Bellman-Ford does not — and that difference defines the other family of routing protocols.
A distance-vector router knows only two things: the cost to each directly connected neighbour, and what each neighbour claims about its own distances to destinations. It never sees the topology. It reasons by hearsay:
"My neighbour B says it can reach network Z at cost 3. The link to B costs 1. Therefore I can reach Z at cost 4 — via B."
Formally, this is the Bellman-Ford equation:
D(x, y) = min over all neighbours v of [ c(x,v) + D(v, y) ]
where c(x,v) is the link cost to neighbour v and D(v,y) is v's advertised distance to y. Each router periodically sends its distance vector — its table of destinations and costs — to its neighbours, applies the equation, and advertises the result onward.
The distinction from link-state is fundamental and worth stating sharply:
| Link-state (Dijkstra) | Distance-vector (Bellman-Ford) | |
|---|---|---|
| Knowledge | The full topology | Only neighbours' claims |
| Shares | Link descriptions, flooded to all | Its whole table, to neighbours only |
| Computation | Each router computes the full graph | Each router adds a link cost to received totals |
| Analogy | Everyone has a map | Everyone asks the person next to them |
| Memory | O(topology) | O(destinations × neighbours) |
| Convergence | Fast, consistent | Slow, prone to transient inconsistency |
| Protocols | OSPF, IS-IS | RIP, (EIGRP, with enhancements) |
The route-by-hearsay analogy captures the weakness precisely. If you ask a stranger for directions and they answer based on what another stranger told them, you can be confidently misdirected — and neither of you can detect it, because neither has a map.
That is exactly the count-to-infinity problem, covered in the next topic. A router cannot tell whether a neighbour's advertised route goes through the router itself. When a destination becomes unreachable, routers can mutually reinforce each other's stale beliefs, incrementing costs one step at a time.
Bellman-Ford's genuine advantages are real, though:
Simplicity. The entire algorithm is one equation. RIP implementations are tiny compared with OSPF.
No global knowledge required, so memory does not grow with topology complexity.
It tolerates negative edge weights (unlike Dijkstra), which is irrelevant for networks but is why the algorithm matters in general graph theory.
It is naturally distributed. No router needs a synchronised global view, which is precisely why BGP — a path-vector protocol, a Bellman-Ford descendant — can work across 75,000 independently administered networks where no shared map could ever exist.
The convergence time is bounded by the network diameter: information propagates one hop per update cycle. With RIP's 30-second update timer, a 10-hop network can take minutes to converge — one of the main reasons RIP was superseded.
Examples
Distance-vector reasoning: hearsay, one hop at a time
A ---1--- B ---1--- C ---1--- Z
round 0 (only directly connected networks known):
A: {A:0} B: {B:0} C: {C:0, Z:1}
round 1 (each sends its vector to its neighbours):
B hears C: "Z at 1" -> B: Z = 1 + c(B,C)=1 -> 2
A hears B: "Z at inf" (B did not know yet) -> no change
A: {A:0, B:1} B: {B:0, A:1, C:1, Z:2} C: {C:0, Z:1, B:1}
round 2:
A hears B: "Z at 2" -> A: Z = 2 + c(A,B)=1 -> 3
A: {A:0, B:1, C:2, Z:3}
Information travels ONE HOP PER ROUND. Convergence time is bounded
by the network DIAMETER — the core scaling limit of distance-vector.
The Bellman-Ford equation, applied at one router
Router A, neighbours B (cost 1) and C (cost 4).
B advertises: Z at 3. C advertises: Z at 1.
D(A,Z) = min[ c(A,B) + D(B,Z), c(A,C) + D(C,Z) ]
= min[ 1 + 3 , 4 + 1 ]
= min[ 4 , 5 ]
= 4, via B
Note A has NO IDEA what B's path to Z looks like. It might even
pass back through A. A cannot tell. That is the whole problem.
INF = float("inf")
class DistanceVectorRouter:
def __init__(self, name, links):
self.name = name
self.links = links # {neighbor: cost} — the ONLY topology known
self.table = {name: (0, "direct")} # dest -> (cost, next_hop)
self.received = {} # neighbor -> their advertised vector
def advertise(self):
"""Send my whole table of costs to my neighbours."""
return {dest: cost for dest, (cost, _) in self.table.items()}
def receive(self, from_neighbor, vector):
self.received[from_neighbor] = vector
def recompute(self):
"""Apply the Bellman-Ford equation. Returns True if anything changed."""
new = {self.name: (0, "direct")}
for nb, link_cost in self.links.items():
new[nb] = min(new.get(nb, (INF, None)), (link_cost, nb))
for nb, vector in self.received.items():
link_cost = self.links[nb]
for dest, their_cost in vector.items():
if dest == self.name:
continue
total = link_cost + their_cost # c(x,v) + D(v,y)
if total < new.get(dest, (INF, None))[0]:
new[dest] = (total, nb)
changed = new != self.table
self.table = new
return changed
def simulate(routers, rounds=5):
for r in range(1, rounds + 1):
vectors = {name: rt.advertise() for name, rt in routers.items()}
for name, rt in routers.items():
for nb in rt.links:
rt.receive(nb, vectors[nb])
changed = any(rt.recompute() for rt in routers.values())
print(f"\n--- round {r} ---")
for name in sorted(routers):
t = routers[name].table
shown = {d: f"{c}/{nh}" for d, (c, nh) in sorted(t.items())}
print(f" {name}: {shown}")
if not changed:
print(" (converged: no router changed its table)")
break
routers = {
"A": DistanceVectorRouter("A", {"B": 1}),
"B": DistanceVectorRouter("B", {"A": 1, "C": 1}),
"C": DistanceVectorRouter("C", {"B": 1, "Z": 1}),
"Z": DistanceVectorRouter("Z", {"C": 1}),
}
simulate(routers, rounds=5)
# --- round 1 ---
# A: {'A': '0/direct', 'B': '1/B'}
# B: {'A': '1/A', 'B': '0/direct', 'C': '1/C'}
# C: {'B': '1/B', 'C': '0/direct', 'Z': '1/Z'}
# Z: {'C': '1/C', 'Z': '0/direct'}
#
# --- round 2 ---
# A: {'A': '0/direct', 'B': '1/B', 'C': '2/B'}
# B: {'A': '1/A', 'B': '0/direct', 'C': '1/C', 'Z': '2/C'}
# C: {'A': '2/B', 'B': '1/B', 'C': '0/direct', 'Z': '1/Z'}
# Z: {'B': '2/C', 'C': '1/C', 'Z': '0/direct'}
#
# --- round 3 ---
# A: {'A': '0/direct', 'B': '1/B', 'C': '2/B', 'Z': '3/B'}
# ...
# Z reached A after 3 rounds — one hop of knowledge per round.
# Convergence time is set by DIAMETER, not by node count.
def convergence_time(diameter, update_interval_s):
return diameter * update_interval_s
print()
for diameter in (3, 10, 15):
for interval, proto in [(30, "RIP (30 s timer)"), (0.05, "triggered updates")]:
t = convergence_time(diameter, interval)
print(f"diameter {diameter:>2}, {proto:<22} -> {t:>7.2f} s to fully converge")
# diameter 3, RIP (30 s timer) -> 90.00 s to fully converge
# diameter 3, triggered updates -> 0.15 s to fully converge
# diameter 10, RIP (30 s timer) -> 300.00 s to fully converge
# diameter 10, triggered updates -> 0.50 s to fully converge
# diameter 15, RIP (30 s timer) -> 450.00 s to fully converge
# diameter 15, triggered updates -> 0.75 s to fully converge
# 7.5 minutes for a 15-hop RIP network is why RIP was superseded.
# The blindness at the heart of distance-vector.
def what_a_router_knows(style):
if style == "link-state":
return ["every router", "every link", "every cost",
"the COMPLETE PATH to every destination"]
return ["my direct links and their costs",
"what each neighbour CLAIMS its distances are",
"NOT whether a neighbour's path passes back through me"]
print()
for style in ("link-state", "distance-vector"):
print(f"{style}:")
for fact in what_a_router_knows(style):
print(f" - {fact}")
# RIP: the distance vector on the wire — destinations and metrics, no topology.
$ sudo tcpdump -i eth0 -n -v udp port 520
# IP 10.0.0.2.520 > 224.0.0.9.520: RIPv2, Response, length 24
# ^^^^^^^^^ the RIPv2 multicast group
# AFI IPv4, 10.3.0.0/16, tag 0x0000, metric 2, next-hop: self
# ^^^^^^^^ "I can reach it in 2 hops"
# AFI IPv4, 10.4.0.0/16, tag 0x0000, metric 3, next-hop: self
# ^ NOTE: no link descriptions, no topology — only destination/cost pairs.
# Compare with OSPF, which floods the links themselves.
$ vtysh -c "show ip rip"
# Codes: R - RIP, C - connected, S - Static, O - OSPF, B - BGP
# Network Next Hop Metric From Tag Time
# R(n) 10.3.0.0/16 10.0.0.2 2 10.0.0.2 0 02:41
# ^ my cost = their metric + 1
# C(i) 10.0.0.0/30 0.0.0.0 1 self 0
# The timers that make convergence slow:
$ vtysh -c "show ip protocols"
# Routing Protocol is "rip"
# Sending updates every 30 seconds with +/-50%, next due in 12 seconds
# ^^^^^^^^^^^^^^^ one hop of progress per 30 s
# Timeout after 180 seconds, garbage collect after 120 seconds
# Default redistribution metric is 1
# Maximum path: 4
# Triggered updates cut convergence dramatically without changing the algorithm:
# router rip
# flush_timer 60
# timers basic 10 60 60 <- update, timeout, garbage-collect
# Compare the information each family transmits:
$ vtysh -c "show ip ospf database" | wc -l # link-state: the TOPOLOGY
# 48
$ vtysh -c "show ip rip" | wc -l # distance-vector: only TOTALS
# 12
Key takeaways
- Distance-vector routers know only their link costs and their neighbours' advertised distances — never the topology.
- The Bellman-Ford equation is
D(x,y) = min over neighbours v of [ c(x,v) + D(v,y) ]. - Routers periodically send their whole table to neighbours only, in contrast to link-state flooding of link descriptions to everyone.
- The analogy is routing by hearsay: you trust a neighbour's total without knowing how they achieve it.
- A router cannot tell whether a neighbour's path passes back through itself, which is the root of count-to-infinity.
- Information spreads one hop per update cycle, so convergence is bounded by network diameter — RIP's 30 s timer makes a 15-hop network take minutes.
- Advantages: simplicity, no global knowledge, tolerance of negative weights, and natural distribution.
- That last property is why BGP, a path-vector descendant, scales across 75,000 independently run networks where no shared map could exist.
🧪 Practice
- Apply the Bellman-Ford equation at router X with neighbours P (cost 2, advertises Z at 4) and Q (cost 5, advertises Z at 1). Give the chosen cost and next hop.
- For a linear topology A-B-C-D-E, show which routers know about E's network after each of the first three update rounds.
- Compute RIP's worst-case convergence time for a network of diameter 12, and contrast it with triggered updates at 50 ms.
- Interview-style: Why does BGP use a distance-vector-style algorithm rather than a link-state one, given that link-state converges faster? (Hint: ask whether 75,000 competing organisations would be willing — or able — to share a single consistent map of their internal topologies.)
Count-to-Infinity and Split Horizon
Theory
The count-to-infinity problem is distance-vector's signature pathology, and it follows directly from the blindness described above: a router cannot tell whether a neighbour's advertised route goes back through itself.
Consider the simplest possible case. Routers A and B are connected, and A is attached to network Z.
- A advertises "Z at cost 1".
- B learns "Z at cost 2, via A", and advertises that back.
- A ignores B's advertisement because its own route is better.
Now A's link to Z fails. A marks Z unreachable. But before A can tell B, B's periodic update arrives: "I can reach Z at cost 2." A has no way to know that B's path is A's own failed path. So A accepts it: "Z at cost 3, via B."
A then advertises Z at 3. B recomputes: "A says 3, so I am at 4." A hears 4 and says 5. The cost climbs, one increment per update cycle, while both routers cheerfully forward Z's traffic back and forth in a routing loop. Left alone, it would continue indefinitely — hence "counting to infinity".
The mitigations form a layered defence, each with a specific scope.
Defining infinity. RIP declares 16 = unreachable. The count cannot exceed 16, so the loop terminates in bounded time. This caps the damage but does not prevent the loop — and it permanently limits RIP to networks of 15 hops or fewer, which is a steep price.
Split horizon. The key rule: never advertise a route back out the interface you learned it on. If B learned about Z from A, B does not tell A about Z. The reasoning is airtight for this case — A is the source of that information, so telling A is at best useless and at worst a lie. Split horizon completely eliminates two-router loops.
Split horizon with poison reverse goes further: instead of staying silent, advertise the route back with an infinite metric. Rather than "I won't mention Z", it says "Z is unreachable through me." The explicit statement is more robust than silence, because silence could also mean a lost packet.
The crucial limitation: split horizon does not prevent loops involving three or more routers. In a triangle A-B-C, each pair obeys split horizon, yet information can still circulate around the ring and produce a count-to-infinity. This is why the timer-based mechanisms in the next topic exist, and ultimately why link-state protocols — which have no such problem at all, because every router has the map — displaced distance-vector in large networks.
Examples
Count-to-infinity, step by step (no split horizon)
Z ---- [A] ---- [B]
initial: A: Z cost 1 (direct) B: Z cost 2 via A
A's link to Z FAILS:
t0 A: Z unreachable B: Z cost 2 via A (not yet told)
t1 B's update arrives: "Z at 2"
A believes it: Z cost 3 via B <-- B's path IS A's dead path.
A cannot know this.
t2 A advertises 3 -> B: Z cost 4 via A
t3 B advertises 4 -> A: Z cost 5 via B
t4 ... 6 ... 7 ... 8 ...
t15 A: 16 = INFINITY -> finally unreachable
Throughout, packets for Z bounce A->B->A->B until TTL kills them.
15 update cycles at 30 s each = 7.5 MINUTES of looping.
Split horizon fixes the two-router case completely
B learned Z from A -> B NEVER advertises Z back to A.
t0 A: Z unreachable. B stays SILENT about Z toward A.
t1 A has no false information to believe. Z stays unreachable.
t2 A tells B "Z unreachable". B removes it.
CONVERGED — no loop, no counting.
...but it does NOT fix three-router loops
[A]
/ \
[B]---[C]
\ /
[Z] (attached to A)
A loses Z. A tells B and C (correctly).
But B learned Z from A, and C learned Z from A.
B does not tell A (split horizon) — but B CAN tell C.
C does not tell A — but C CAN tell B.
Information about the dead route circulates B -> C -> B,
and can leak back to A from a router A did not learn it from.
Counting resumes. Split horizon is NECESSARY, not SUFFICIENT.
INF = 16 # RIP's definition of infinity
class DVRouter:
def __init__(self, name, split_horizon=False, poison_reverse=False):
self.name = name
self.table = {} # dest -> (cost, learned_from)
self.links = {} # neighbor -> cost
self.split_horizon = split_horizon
self.poison_reverse = poison_reverse
def advertise_to(self, neighbor):
"""What I tell this specific neighbour — the heart of split horizon."""
out = {}
for dest, (cost, learned_from) in self.table.items():
if learned_from == neighbor:
if self.poison_reverse:
out[dest] = INF # "unreachable THROUGH ME" — explicit
elif self.split_horizon:
continue # say nothing at all
else:
out[dest] = cost # naive: the bug that starts the count
else:
out[dest] = cost
return out
def receive(self, neighbor, vector):
changed = False
for dest, their_cost in vector.items():
total = min(INF, their_cost + self.links[neighbor])
current, learned_from = self.table.get(dest, (INF, None))
# Accept if better, OR if it is an update from my current next hop.
if total < current or learned_from == neighbor:
if (total, neighbor) != (current, learned_from):
self.table[dest] = (total, neighbor)
changed = True
return changed
def run(split_horizon, poison_reverse, label, max_rounds=20):
A = DVRouter("A", split_horizon, poison_reverse)
B = DVRouter("B", split_horizon, poison_reverse)
A.links, B.links = {"B": 1}, {"A": 1}
A.table["Z"] = (1, "direct") # A is attached to Z
B.receive("A", A.advertise_to("B")) # B learns Z at cost 2
print(f"\n--- {label} ---")
print(f"before failure: A={A.table['Z']} B={B.table['Z']}")
A.table["Z"] = (INF, None) # A's link to Z FAILS
history = []
for r in range(1, max_rounds + 1):
A.receive("B", B.advertise_to("A"))
B.receive("A", A.advertise_to("B"))
a, b = A.table["Z"][0], B.table["Z"][0]
history.append((r, a, b))
if a >= INF and b >= INF:
print(f"converged after {r} rounds: both = unreachable")
break
for r, a, b in history[:6]:
note = " <- counting up, packets looping" if a < INF and r > 1 else ""
print(f" round {r:>2}: A cost={a:<4} B cost={b:<4}{note}")
if len(history) > 6:
print(f" ... ({len(history)} rounds total)")
run(split_horizon=False, poison_reverse=False, label="NAIVE: count-to-infinity")
run(split_horizon=True, poison_reverse=False, label="SPLIT HORIZON")
run(split_horizon=True, poison_reverse=True, label="SPLIT HORIZON + POISON REVERSE")
#
# --- NAIVE: count-to-infinity ---
# before failure: A=(1, 'direct') B=(2, 'A')
# converged after 8 rounds: both = unreachable
# round 1: A cost=3 B cost=4
# round 2: A cost=5 B cost=6 <- counting up, packets looping
# round 3: A cost=7 B cost=8 <- counting up, packets looping
# round 4: A cost=9 B cost=10 <- counting up, packets looping
# round 5: A cost=11 B cost=12 <- counting up, packets looping
# round 6: A cost=13 B cost=14 <- counting up, packets looping
# ... (8 rounds total)
#
# --- SPLIT HORIZON ---
# before failure: A=(1, 'direct') B=(2, 'A')
# converged after 1 rounds: both = unreachable
# round 1: A cost=16 B cost=16
#
# --- SPLIT HORIZON + POISON REVERSE ---
# before failure: A=(1, 'direct') B=(2, 'A')
# converged after 1 rounds: both = unreachable
# round 1: A cost=16 B cost=16
#
# 8 rounds of looping becomes 1 round of correct withdrawal.
# Why RIP is capped at 15 hops — the cap IS the loop terminator.
print()
for infinity in (16, 100, 256):
cycles = infinity - 2
print(f"infinity={infinity:<4} -> loop lasts {cycles:>3} update cycles "
f"= {cycles*30/60:>5.1f} min at RIP's 30 s timer "
f"(max usable network: {infinity-1} hops)")
# infinity=16 -> loop lasts 14 update cycles = 7.0 min at RIP's 30 s timer (max usable network: 15 hops)
# infinity=100 -> loop lasts 98 update cycles = 49.0 min at RIP's 30 s timer (max usable network: 99 hops)
# infinity=256 -> loop lasts 254 update cycles = 127.0 min at RIP's 30 s timer (max usable network: 255 hops)
# A larger "infinity" permits bigger networks but LONGER loops.
# 16 is a deliberate trade, not an arbitrary constant.
# Split horizon is on by default — and you can see it being applied.
$ vtysh -c "show ip rip status"
# Routing Protocol is "rip"
# Sending updates every 30 seconds with +/-50%
# Default version control: send version 2, receive version 2
# Interface Send Recv Key-chain
# eth0 2 2
# Routing for Networks:
# eth0
# Distance: (default is 120)
# Split horizon per interface (Cisco):
# interface GigabitEthernet0/1
# ip split-horizon <- default ON for most media
# no ip split-horizon <- required on hub-and-spoke Frame Relay,
# where spokes learn from each other
# through the hub's single interface
# Poison reverse on the wire — metric 16 means "unreachable through me":
$ sudo tcpdump -i eth0 -n -v udp port 520
# IP 10.0.0.2.520 > 224.0.0.9.520: RIPv2, Response, length 44
# AFI IPv4, 10.3.0.0/16, tag 0x0000, metric 16, next-hop: self
# ^^^^^^^^^ POISONED: explicitly
# unreachable, rather than merely absent
# Watching a route being poisoned when a link fails:
$ vtysh -c "show ip rip" | grep 10.5
# R(n) 10.5.0.0/16 10.0.0.2 16 10.0.0.2 0 00:04
# ^^ metric 16 = infinity = withdrawn
# Link-state protocols have no equivalent problem to configure:
$ vtysh -c "show ip ospf" | grep -ic "split.horizon"
# 0
# There is nothing to configure, because every router has the map and
# can see for itself whether a path passes back through it.
Key takeaways
- Count-to-infinity occurs because a distance-vector router cannot tell whether a neighbour's route passes back through itself.
- When a destination dies, routers mutually reinforce stale information, incrementing the cost one step per update cycle while traffic loops.
- RIP defines 16 as infinity, bounding the count — and thereby limiting RIP to networks of 15 hops or fewer.
- Split horizon: never advertise a route back out the interface it was learned on. This completely eliminates two-router loops.
- Poison reverse advertises the route back with an infinite metric instead of staying silent — an explicit statement is more robust than absence.
- Split horizon does not prevent loops of three or more routers; information can still circulate around a ring.
- A larger "infinity" allows bigger networks but longer loops — 16 is a deliberate trade.
- Link-state protocols have no equivalent problem, because every router can see for itself whether a path returns through it.
🧪 Practice
- Trace the count-to-infinity sequence for two routers over six update cycles, showing each router's cost at every step.
- Explain why split horizon eliminates two-router loops but not three-router loops, using a triangle topology.
- State the difference between split horizon and poison reverse, and give a reason the explicit version is preferred.
- Interview-style: Why is RIP limited to 15 hops? What would break if the limit were 100? (Hint: the limit is not about path length — ask what the number 16 is actually being used for, and how long a loop would last if it were larger.)
Poison Reverse and Hold-Down Timers
Theory
Split horizon handles the two-router case. Larger loops need mechanisms that work even when a false route arrives from a router you did not learn it from. RIP layers several timers on top, and understanding them is understanding why distance-vector convergence is measured in minutes.
Route poisoning is the active withdrawal mechanism. When a route fails, instead of quietly removing it and waiting for neighbours to time it out, the router immediately advertises it with metric 16 — explicitly unreachable. This propagates the bad news actively rather than passively. Silence is ambiguous (was the update lost?); metric 16 is unambiguous.
Triggered updates (also called flash updates) make poisoning fast. Normally RIP waits for its 30-second timer. A triggered update sends the change immediately, so bad news propagates in milliseconds rather than tens of seconds. The classic principle is "bad news travels fast" — good news can wait for the next periodic update, but withdrawals should not.
Hold-down timers are the most aggressive and the most controversial. After a route is marked unreachable, the router refuses any new information about that route for a hold-down period (RIP default: 180 seconds), unless it comes from the original source or has a strictly better metric.
The logic: after a failure, the network is temporarily inconsistent, and some routers still hold stale information. Hold-down deliberately ignores everything for long enough that the stale data ages out everywhere. It is a blunt instrument — it trades convergence speed for correctness, and during hold-down a genuinely valid alternate path is also rejected. A route that comes back within the hold-down window stays unusable until the timer expires.
RIP's complete timer set:
| Timer | Default | Purpose |
|---|---|---|
| Update | 30 s | Send the full table to neighbours |
| Invalid | 180 s | No update heard → mark the route unreachable (metric 16) |
| Hold-down | 180 s | Refuse new information about it |
| Flush | 240 s | Remove the entry from the table entirely |
Note the relationships: invalid is 6 update periods, so several updates must be missed before declaring failure; flush exceeds hold-down so the entry survives long enough to be poisoned to neighbours.
The worst-case cost is severe. A route can take 180 seconds to be declared dead, then 180 more in hold-down before an alternative is accepted — up to six minutes to fail over on a network where a healthy alternate path existed the entire time.
This is the central indictment of distance-vector for large networks. Link-state protocols detect failure in milliseconds (via Hello timers or BFD) and converge in seconds, because there is no counting problem to defend against. The timers exist solely to compensate for the algorithm's blindness. EIGRP escapes this with DUAL, which maintains loop-free backup paths (feasible successors) verified by an explicit condition — no timers needed.
Examples
The RIP timer cascade for one route
t=0 last valid update received
|
t=30 expected update... (missed)
t=60 expected update... (missed)
...
t=180 INVALID TIMER EXPIRES
-> mark metric 16, start HOLD-DOWN, poison to neighbours
|
t=180 .. 360 HOLD-DOWN: refuse all new info about this route,
| even a genuinely good alternate path
|
t=240 FLUSH TIMER: remove the entry entirely
|
t=360 hold-down ends; alternatives may finally be accepted
WORST CASE: ~6 minutes to fail over, with a healthy backup path
available the entire time.
Triggered updates: bad news travels fast
WITHOUT: link fails at t=0
... wait for the periodic timer ...
t=30 neighbour finally hears about it
WITH: link fails at t=0
t=0.001 poisoned route sent IMMEDIATELY
neighbour knows in milliseconds
Good news (a new/better route) can wait for the periodic update.
Bad news (a withdrawal) must not.
Hold-down's trade-off, made concrete
R1 --X-- [dead primary]
\
`---- [valid backup via R2, cost 5]
t=180 R1 declares the primary dead, enters hold-down
t=185 R2 advertises the valid backup at cost 5
R1 REFUSES IT — hold-down is active
t=360 hold-down expires, R1 finally accepts the backup
175 seconds of blackout, with a working path available throughout.
This is the price paid for preventing stale information from
re-establishing a loop.
class RIPRoute:
UPDATE, INVALID, HOLDDOWN, FLUSH = 30, 180, 180, 240
def __init__(self, dest, cost, next_hop, now=0):
self.dest, self.cost, self.next_hop = dest, cost, next_hop
self.cost_before_failure = cost # what "better" is measured against
self.last_update = now
self.holddown_until = None
self.state = "valid"
def tick(self, now):
if self.state == "valid" and now - self.last_update >= self.INVALID:
self.state = "invalid"
self.cost_before_failure = self.cost # remember it first
self.cost = 16 # POISON
self.holddown_until = now + self.HOLDDOWN # start hold-down
return "INVALID: poisoned to metric 16, hold-down begins"
if self.state == "invalid" and now - self.last_update >= self.FLUSH:
self.state = "flushed"
return "FLUSHED: entry removed from the table"
if self.holddown_until and now >= self.holddown_until:
self.holddown_until = None
return "hold-down expired: new information will now be accepted"
return None
def offer(self, cost, next_hop, now):
"""A neighbour offers a route. Hold-down may refuse it."""
if self.holddown_until and now < self.holddown_until:
# Exceptions: the original source, or a genuinely better metric than
# the route had BEFORE it failed. Comparing against the poisoned 16
# would let every route through and defeat hold-down entirely.
if next_hop == self.next_hop or cost < self.cost_before_failure:
pass
else:
remaining = self.holddown_until - now
return f"REFUSED (hold-down active, {remaining}s remaining)"
self.cost, self.next_hop = cost, next_hop
self.last_update, self.state = now, "valid"
self.holddown_until = None
return f"ACCEPTED: cost {cost} via {next_hop}"
r = RIPRoute("10.5.0.0/16", cost=3, next_hop="R2", now=0)
print("timeline for one route after its source goes silent:\n")
for now in (30, 90, 180, 185, 240, 360):
event = r.tick(now)
if event:
print(f" t={now:>3}s {event}")
if now == 185:
print(f" t={now:>3}s backup offered via R7 -> {r.offer(5, 'R7', now)}")
if now == 360:
print(f" t={now:>3}s backup offered via R7 -> {r.offer(5, 'R7', now)}")
# t=180s INVALID: poisoned to metric 16, hold-down begins
# t=185s backup offered via R7 -> REFUSED (hold-down active, 175s remaining)
# t=240s FLUSHED: entry removed from the table
# t=360s hold-down expired: new information will now be accepted
# t=360s backup offered via R7 -> ACCEPTED: cost 5 via R7
# Worst-case failover, compared across protocol families.
def failover_time(protocol):
return {
"RIP (invalid + hold-down)": 180 + 180,
"RIP (triggered updates)": 180 + 1,
"OSPF (default hellos)": 40 + 5, # dead interval + SPF
"OSPF (fast hellos)": 1 + 1,
"OSPF + BFD": 0.15,
"EIGRP (feasible successor)": 0.05, # pre-verified backup
}[protocol]
print()
for proto in ["RIP (invalid + hold-down)", "RIP (triggered updates)",
"OSPF (default hellos)", "OSPF (fast hellos)",
"OSPF + BFD", "EIGRP (feasible successor)"]:
t = failover_time(proto)
print(f"{proto:<30}{t:>8.2f} s {'#' * min(40, int(t/9)+1)}")
# RIP (invalid + hold-down) 360.00 s ########################################
# RIP (triggered updates) 181.00 s #####################
# OSPF (default hellos) 45.00 s ######
# OSPF (fast hellos) 2.00 s #
# OSPF + BFD 0.15 s #
# EIGRP (feasible successor) 0.05 s #
# Timers exist to compensate for the ALGORITHM's blindness. Remove the
# blindness (link-state, or DUAL's feasibility condition) and they vanish.
# Why the timer values relate to each other as they do.
print()
rel = {
"invalid / update": (180, 30, "6 missed updates before declaring failure"),
"flush / invalid": (240, 180, "entry survives 60 s past invalid, so it can be poisoned"),
"holddown / invalid":(180, 180, "ignore new info for as long as it took to declare death"),
}
for name, (a, b, why) in rel.items():
print(f"{name:<20} {a}/{b} = {a/b:>4.1f} {why}")
# invalid / update 180/30 = 6.0 6 missed updates before declaring failure
# flush / invalid 240/180 = 1.3 entry survives 60 s past invalid, so it can be poisoned
# holddown / invalid 180/180 = 1.0 ignore new info for as long as it took to declare death
# RIP's timers, and how to make them less painful:
$ vtysh -c "show ip protocols"
# Routing Protocol is "rip"
# Sending updates every 30 seconds with +/-50%, next due in 8 seconds
# Timeout after 180 seconds, garbage collect after 120 seconds
# ^^^^^^^^^^^^^^^^^^^^^^^^^ invalid timer ^^^^^^^^^^^^^^^^^^^^ flush
# router rip
# timers basic 10 60 60
# ^^ ^^ ^^ update, invalid/timeout, flush
# ! Faster convergence, more bandwidth. MUST match on all routers.
# Watching a poisoned route propagate:
$ sudo tcpdump -i eth0 -n -v udp port 520 | grep "metric 16"
# IP 10.0.0.2.520 > 224.0.0.9.520: RIPv2, Response
# AFI IPv4, 10.5.0.0/16, tag 0x0000, metric 16, next-hop: self
# ^^^^^^^^^ an active withdrawal,
# sent immediately (triggered update)
$ vtysh -c "show ip rip" | grep 10.5
# R(n) 10.5.0.0/16 10.0.0.2 16 10.0.0.2 0 00:02
# ^^ poisoned; will flush shortly
# The modern equivalents — none of which need hold-down at all:
# OSPF fast hellos (sub-second failure detection):
# interface GigabitEthernet0/1
# ip ospf dead-interval minimal hello-multiplier 4 <- 250 ms hellos
# BFD: a dedicated sub-second liveness protocol, protocol-independent:
# interface GigabitEthernet0/1
# bfd interval 50 min_rx 50 multiplier 3 <- 150 ms detection
# router ospf 1
# bfd all-interfaces
$ show bfd neighbors
# NeighAddr LD/RD RH/RS State Int
# 10.0.0.2 1/1 Up Up Gi0/1
# ^^ liveness in 150 ms instead of 180 seconds
Key takeaways
- Route poisoning actively advertises a failed route with metric 16 instead of silently removing it — explicit beats ambiguous.
- Triggered updates send changes immediately rather than waiting 30 seconds: bad news travels fast, good news can wait.
- Hold-down timers refuse new information about a failed route for 180 s, letting stale data age out everywhere — trading convergence speed for correctness.
- During hold-down, a genuinely valid alternate path is also rejected.
- RIP's timers: update 30 s, invalid 180 s, hold-down 180 s, flush 240 s; invalid is 6 update periods, and flush exceeds invalid so the route can still be poisoned.
- Worst-case RIP failover is roughly 6 minutes, even with a healthy backup available throughout.
- These timers exist only to compensate for distance-vector's blindness.
- Link-state with fast hellos or BFD detects failure in milliseconds; EIGRP's DUAL uses a pre-verified feasible successor and needs no hold-down at all.
🧪 Practice
- List RIP's four timers with their defaults and state what each one triggers.
- Explain why hold-down can delay recovery even when a perfectly good alternate path exists, and give the worst-case duration.
- Contrast triggered updates with periodic updates, and explain why withdrawals are treated differently from new routes.
- Interview-style: Why do OSPF and EIGRP not need hold-down timers? (Hint: hold-down defends against believing stale information — ask what each protocol knows that makes such belief impossible in the first place.)
Convergence and Routing Loops
Theory
Convergence is the state in which every router's forwarding table is consistent with the current topology and with every other router's table. Convergence time is how long it takes to get there after a change. It is the single most important operational metric for a routing protocol, because during convergence the network is actively wrong: packets are dropped, looped, or sent down dead paths.
Convergence has four distinct phases, and knowing which one dominates tells you what to tune:
1. Detection. Noticing that something changed. This is usually the largest and most tunable component. Physical link-down is immediate (milliseconds). Waiting for Hello timers to expire is slow (OSPF's default dead interval is 40 s). BFD provides sub-second detection independent of the routing protocol, typically 150 ms.
2. Propagation. Telling the other routers. Link-state floods LSAs in milliseconds; distance-vector propagates one hop per update cycle.
3. Computation. Recalculating paths. Dijkstra on a large area takes single-digit milliseconds, and is rarely the bottleneck.
4. Installation. Programming the new entries into hardware. On a router with a million BGP prefixes, this can take seconds and is often the dominant term at internet scale.
Routing loops occur during the window when routers disagree. Router A believes the path to Z is via B; B believes it is via A. Packets bounce until TTL expires.
Loops are almost always transient — a symptom of inconsistent state during convergence rather than a permanent misconfiguration. TTL (Chapter 5) is what makes them survivable: without it, a single loop would accumulate packets until the links saturated permanently.
| Loop cause | Family affected | Prevention |
|---|---|---|
| Inconsistent state mid-convergence | All | Faster convergence; ordered FIB updates |
| Count-to-infinity | Distance-vector | Split horizon, poison reverse, hold-down |
| Redistribution between protocols | All | Route tags, filtering, careful AD |
| Misconfigured static routes | All | Review; route tracking |
| iBGP without full mesh | BGP | Full mesh, route reflectors, or confederations |
The techniques that make convergence fast, roughly in order of impact:
BFD (Bidirectional Forwarding Detection) — a lightweight protocol whose only job is to detect liveness between two systems in milliseconds. It is protocol-independent, so OSPF, IS-IS, and BGP can all use the same session. This attacks the detection phase, which is usually the biggest win available.
Fast hellos — sub-second Hello intervals. Effective, but heavier on CPU than BFD.
LFA / Remote LFA (Loop-Free Alternates) — pre-compute a backup next hop that provably does not loop, and install it in hardware alongside the primary. Failover becomes a local hardware switch in tens of milliseconds, with no control-plane involvement. TI-LFA with segment routing extends this to guarantee a backup for every destination.
BGP PIC (Prefix Independent Convergence) — a hierarchical FIB where many prefixes share a next-hop pointer. Changing one pointer redirects all of them at once, instead of rewriting a million entries individually.
Graceful Restart / NSF — keep forwarding with the existing FIB while the control plane restarts, as covered in Chapter 5.
Faster convergence has a cost: stability. Aggressive timers make a network react to brief transients that would have resolved themselves, causing route flapping. This is why dampening (penalising repeatedly flapping routes) and SPF throttling exist. The engineering target is to converge quickly on real failures and ignore noise — and those two goals pull in opposite directions.
Examples
The four phases, and which one actually dominates
event: link fails at t=0
|<-- DETECTION -->|<-PROP->|<-COMP->|<-- INSTALLATION -->|
| | | | |
0 variable ~ms ~ms ms to seconds
^ ^
USUALLY THE BIGGEST TERM DOMINATES AT INTERNET SCALE
(40 s with default OSPF hellos, (1M prefixes to program)
0.15 s with BFD)
Enterprise OSPF, defaults: 40 s + 0.01 + 0.005 + 0.1 = ~40 s
Enterprise OSPF, with BFD: 0.15 + 0.01 + 0.005 + 0.1 = ~0.27 s
Internet-scale BGP: 0.15 + 1.2 + 0.5 + 8.0 = ~9.9 s
^^^^^^^^^^^^ installation wins
A transient loop during convergence
Before: A --> B --> Z (both agree)
Link B-Z fails. B knows instantly. A does not yet.
t=0.00 B: "no route to Z"; A: "Z via B" (STALE)
t=0.00 B recomputes: its best remaining path to Z is... via A!
(A still advertises reachability, because A has not updated)
t=0.01 packet for Z: A --> B --> A --> B --> ...
^^^^^^^^^^^^^^^^^^^^^^^ TRANSIENT LOOP
t=0.15 A receives the update, removes its route
t=0.16 loop resolved
It lasted 160 ms. TTL bounded the damage. This is NORMAL and expected;
the goal is to make the window short, not to eliminate it entirely.
Loop-Free Alternate: failover without the control plane
primary: A --10--> B --> Z
backup: A --15--> C --> Z (pre-verified: C does NOT route via A)
Both installed in hardware BEFORE any failure.
Link A-B fails -> hardware switches to the backup in ~20 ms.
The control plane converges afterwards, at its leisure.
def convergence_breakdown(scenario):
"""Where the time actually goes, per phase, in seconds."""
return {
"OSPF default hellos": {"detect": 40.0, "propagate": 0.01,
"compute": 0.005, "install": 0.10},
"OSPF fast hellos": {"detect": 1.0, "propagate": 0.01,
"compute": 0.005, "install": 0.10},
"OSPF + BFD": {"detect": 0.15, "propagate": 0.01,
"compute": 0.005, "install": 0.10},
"OSPF + BFD + LFA": {"detect": 0.15, "propagate": 0.0,
"compute": 0.0, "install": 0.02},
"BGP internet scale": {"detect": 0.15, "propagate": 1.20,
"compute": 0.50, "install": 8.00},
"BGP + PIC": {"detect": 0.15, "propagate": 0.0,
"compute": 0.0, "install": 0.05},
}[scenario]
print(f"{'scenario':<24}{'detect':>8}{'prop':>8}{'comp':>8}{'install':>9}{'TOTAL':>9}")
for s in ["OSPF default hellos", "OSPF fast hellos", "OSPF + BFD",
"OSPF + BFD + LFA", "BGP internet scale", "BGP + PIC"]:
b = convergence_breakdown(s)
total = sum(b.values())
print(f"{s:<24}{b['detect']:>8.2f}{b['propagate']:>8.2f}"
f"{b['compute']:>8.3f}{b['install']:>9.2f}{total:>9.2f}")
# scenario detect prop comp install TOTAL
# OSPF default hellos 40.00 0.01 0.005 0.10 40.12
# OSPF fast hellos 1.00 0.01 0.005 0.10 1.11
# OSPF + BFD 0.15 0.01 0.005 0.10 0.27
# OSPF + BFD + LFA 0.15 0.00 0.000 0.02 0.17
# BGP internet scale 0.15 1.20 0.500 8.00 9.85
# BGP + PIC 0.15 0.00 0.000 0.05 0.20
# Note where the win comes from: LFA and PIC remove ENTIRE PHASES by
# pre-computing and pre-installing the backup.
# What a convergence window costs in lost traffic.
def packets_lost(convergence_s, pps=1_000_000, loss_fraction=1.0):
lost = convergence_s * pps * loss_fraction
return lost, lost * 1500 / 1e9 # GB at 1500-byte packets
print()
for s in ["OSPF default hellos", "OSPF + BFD", "OSPF + BFD + LFA"]:
total = sum(convergence_breakdown(s).values())
pkts, gb = packets_lost(total)
print(f"{s:<22}{total:>7.2f} s -> {pkts:>14,.0f} packets {gb:>8.2f} GB lost")
# OSPF default hellos 40.12 s -> 40,115,000 packets 60.17 GB lost
# OSPF + BFD 0.27 s -> 265,000 packets 0.40 GB lost
# OSPF + BFD + LFA 0.17 s -> 170,000 packets 0.25 GB lost
# The transient loop: why it forms and why TTL saves you.
def transient_loop(a_updated, b_updated, ttl=64):
if not a_updated and b_updated:
hops = ttl # bounces until the budget is gone
return (f"LOOP: A still points at B (stale), B fell back to A. "
f"Packets bounce {hops} times, then TTL expires. Bounded.")
if a_updated and b_updated:
return "converged: consistent tables, no loop"
return "partial: some traffic on a stale path, no loop yet"
print()
for a, b in [(False, True), (True, True)]:
print(f"A updated={a!s:<5} B updated={b!s:<5} -> {transient_loop(a, b)}")
# A updated=False B updated=True -> LOOP: A still points at B (stale), B fell back to A. Packets bounce 64 times, then TTL expires. Bounded.
# A updated=True B updated=True -> converged: consistent tables, no loop
# Stability vs speed: aggressive timers react to noise as well as to failures.
def flap_response(hold_ms, transient_ms):
if transient_ms < hold_ms:
return f"ignored (transient {transient_ms} ms < hold {hold_ms} ms) — STABLE"
return f"reconverged on a {transient_ms} ms blip — CHURN"
print()
for hold in (50, 1000):
for transient in (30, 200):
print(f"hold={hold:>5} ms, blip={transient:>4} ms -> {flap_response(hold, transient)}")
# hold= 50 ms, blip= 30 ms -> ignored (transient 30 ms < hold 50 ms) — STABLE
# hold= 50 ms, blip= 200 ms -> reconverged on a 200 ms blip — CHURN
# hold= 1000 ms, blip= 30 ms -> ignored (transient 30 ms < hold 1000 ms) — STABLE
# hold= 1000 ms, blip= 200 ms -> ignored (transient 200 ms < hold 1000 ms) — STABLE
# Fast detection and stability are in direct tension. Dampening and
# throttling are how the trade is managed.
# Measure real convergence: fail a link and watch how long traffic stops.
$ ping -i 0.01 -D 10.9.0.1 > /tmp/conv.log &
$ sudo ip link set eth1 down
$ sleep 5 && sudo ip link set eth1 up
$ awk -F'[][]' '/icmp_seq/{if(last && $2-last > 0.1) print "gap:", $2-last, "s"; last=$2}' /tmp/conv.log
# gap: 0.271 s
# ^^^^^ the actual convergence window, measured end to end
# BFD — the single highest-impact convergence improvement:
# interface GigabitEthernet0/1
# bfd interval 50 min_rx 50 multiplier 3 <- detect in 150 ms
# router ospf 1
# bfd all-interfaces
# router bgp 64500
# neighbor 10.0.0.2 fall-over bfd <- BGP uses the SAME session
$ show bfd neighbors details | grep -E "State|Detect"
# Session state is UP and not using echo function.
# Detect time: 150ms <- vs OSPF's 40,000 ms default dead interval
# Loop-Free Alternates: pre-installed backups in hardware.
# router ospf 1
# fast-reroute per-prefix enable prefix-priority high
$ show ip cef 10.9.0.0/16 detail | grep -A2 repair
# repair: attached-nexthop 10.0.2.2 GigabitEthernet0/2
# ^^^^^^^^^^^^^^^^ already programmed; failover is a hardware event
# BGP PIC — one pointer change redirects a million prefixes:
# router bgp 64500
# address-family ipv4
# bgp additional-paths install
$ show ip bgp 192.0.2.0/24 | grep -i backup
# Backup path from 10.0.0.9, installed
# ^ no need to rewrite 1M FIB entries one by one
# Dampening: stop reacting to a link that cannot stay up.
# router bgp 64500
# bgp dampening 15 750 2000 60
# ^^ ^^^ ^^^^ ^^ half-life, reuse, suppress, max-suppress (min)
$ show ip bgp dampening flap-statistics
# Network From Flaps Duration Reuse Path
# *d 203.0.113.0/24 10.0.0.2 27 02:14:31 00:38:12 64500
# ^^ suppressed: flapped 27 times; will be ignored for 38 more minutes
Key takeaways
- Convergence is the state where all forwarding tables are mutually consistent; during it, the network is actively wrong.
- Four phases: detection, propagation, computation, installation.
- Detection usually dominates in enterprise networks (OSPF's 40 s default dead interval); installation dominates at internet scale with a million prefixes.
- Routing loops during convergence are transient and normal; TTL is what bounds the damage.
- BFD gives protocol-independent sub-second detection (~150 ms) and is typically the highest-impact single change.
- LFA/TI-LFA and BGP PIC remove entire phases by pre-computing and pre-installing backups, making failover a hardware event in tens of milliseconds.
- Graceful Restart / NSF keeps forwarding on the existing FIB while the control plane restarts.
- Fast convergence and stability are in tension: aggressive timers react to noise, which is why dampening and SPF throttling exist.
🧪 Practice
- Name the four convergence phases and state which dominates for a default-timer OSPF network and for a full-table BGP router.
- Compute the packets lost during a 40-second versus a 0.27-second convergence at 500,000 packets per second.
- Explain how a transient loop forms between two routers mid-convergence and what stops it from lasting forever.
- Interview-style: Your network converges in 40 seconds and you must reduce it to under one second. What do you change first, and what risk does it introduce? (Hint: find the phase consuming 39.9 of those 40 seconds — then ask what happens when a link flaps briefly under the new setting.)
<a id="63-interior-gateway-protocols"></a>
6.3 Interior Gateway Protocols
This subchapter covers the protocols that run inside a single organisation, where all routers are under one administration and the goal is simply to find good paths quickly.
RIP Versions and Operation
Theory
RIP (Routing Information Protocol) is the oldest routing protocol still encountered, and it is worth studying precisely because it is simple enough to understand completely — and because its limitations motivated everything that followed.
RIP is distance-vector with hop count as its only metric. Every router periodically broadcasts its entire routing table to its neighbours; each neighbour adds 1 to every received metric and keeps the lowest. That is the whole protocol.
RIPv1 (RFC 1058, 1988) was classful. Its updates carried no subnet mask, so receivers had to assume the classful default (Chapter 5). This made VLSM impossible — a network could not use both /24 and /27 subnets of the same class B — and made discontiguous subnets silently unroutable. Updates were broadcast to 255.255.255.255, interrupting every host on the segment, and there was no authentication.
RIPv2 (RFC 2453, 1998) fixed the protocol-level problems without changing the algorithm:
| Feature | RIPv1 | RIPv2 |
|---|---|---|
| Subnet mask in updates | No (classful) | Yes (classless, VLSM works) |
| Destination | Broadcast 255.255.255.255 | Multicast 224.0.0.9 |
| Authentication | None | Plaintext or MD5 |
| Route tags | No | Yes (for redistribution marking) |
| Next-hop field | No | Yes (avoids an extra hop) |
RIPng (RFC 2080) is the IPv6 version, using multicast ff02::9 and UDP port 521 instead of 520.
Everything else stayed the same — and that is the problem, because the limits are in the algorithm, not the encoding:
Maximum 15 hops. 16 means unreachable (6.2), so any network with a diameter above 15 is simply unroutable. This alone disqualifies RIP from large networks.
Hop count ignores bandwidth. A path over three 10 Gb/s links (3 hops) loses to a path over one 64 kb/s link (1 hop). RIP will confidently choose the dial-up line.
Slow convergence. 30-second updates plus 180-second invalid and hold-down timers mean minutes to converge, as computed in 6.2.
Periodic full-table updates waste bandwidth: the whole table every 30 seconds, whether anything changed or not.
No hierarchy. There are no areas, so every router carries every route.
RIP survives in three niches: very small networks (a handful of routers), teaching, and as a lowest-common-denominator protocol where heterogeneous equipment must interoperate. For anything else, OSPF, IS-IS, or EIGRP is the right answer, and the reasons are all in the table above.
Examples
RIP's metric ignores everything that matters
+--10 Gb/s--[R2]--10 Gb/s--[R3]--10 Gb/s--+
| |
[R1] [R5]
| |
+-------------64 kb/s----------------------+
RIP metric: via R2/R3/R4 = 3 hops
direct = 1 hop <-- RIP CHOOSES THIS
Actual: 30 Gb/s of capacity ignored in favour of a 64 kb/s line.
OSPF cost (ref 100 Mb/s... or better, 100 Gb/s):
via R2/R3 = 10 + 10 + 10 = 30
direct = 100000/0.064 = huge
-> OSPF chooses the fast path. The METRIC is the difference.
RIPv1 vs RIPv2 on the wire
RIPv1: dst 255.255.255.255 (BROADCAST — every host is interrupted)
entry: 10.1.0.0 metric 2
^^^^^^^^ no mask! receiver ASSUMES /8 (class A default)
-> a /24 subnet of 10.x is unrepresentable
RIPv2: dst 224.0.0.9 (MULTICAST — only RIP speakers listen)
entry: 10.1.5.0/24 metric 2 tag 0 next-hop 0.0.0.0
^^^ mask present -> VLSM works
+ optional MD5 authentication entry
The 15-hop ceiling
R1-R2-R3-R4-R5-R6-R7-R8-R9-R10-R11-R12-R13-R14-R15-R16
|<------------------ 15 hops ------------------>| reachable
|<--->| hop 16 = INFINITY
UNREACHABLE
The network is physically fine. RIP simply cannot express the distance.
RIP_INFINITY = 16
def rip_reachable(hops):
return hops < RIP_INFINITY
def rip_vs_bandwidth(paths):
"""RIP picks fewest hops; a bandwidth-aware metric picks the fast path."""
rip_best = min(paths, key=lambda p: p["hops"])
bw_best = min(paths, key=lambda p: sum(100_000 / b for b in p["bandwidths_mbps"]))
return rip_best, bw_best
paths = [
{"name": "via R2-R3-R4", "hops": 3, "bandwidths_mbps": [10_000, 10_000, 10_000]},
{"name": "direct link", "hops": 1, "bandwidths_mbps": [0.064]},
]
rip_best, bw_best = rip_vs_bandwidth(paths)
print(f"RIP chooses: {rip_best['name']} ({rip_best['hops']} hops)")
print(f"bandwidth-aware chooses:{bw_best['name']} ({bw_best['hops']} hops)")
# RIP chooses: direct link (1 hop)
# bandwidth-aware chooses: via R2-R3-R4 (3 hops)
# RIP selects a 64 kb/s line over 30 Gb/s of capacity. The algorithm is
# fine; the METRIC is the defect.
# Reachability ceiling.
print()
for hops in (1, 15, 16, 20):
print(f"{hops:>3} hops -> {'reachable' if rip_reachable(hops) else 'UNREACHABLE (>= 16 = infinity)'}")
# 1 hops -> reachable
# 15 hops -> reachable
# 16 hops -> UNREACHABLE (>= 16 = infinity)
# 20 hops -> UNREACHABLE (>= 16 = infinity)
# RIPv1's classful blindness, demonstrated.
import ipaddress
def classful_assumption(addr):
first = int(addr.split(".")[0])
if first < 128: return 8
if first < 192: return 16
return 24
def ripv1_receives(advertised_prefix):
net = ipaddress.ip_network(advertised_prefix)
assumed = classful_assumption(str(net.network_address))
assumed_net = ipaddress.ip_network(f"{net.network_address}/{assumed}",
strict=False)
ok = assumed == net.prefixlen
return assumed_net, ok
print()
for prefix in ["10.1.5.0/24", "172.16.8.0/24", "192.168.1.0/24", "10.0.0.0/8"]:
assumed, ok = ripv1_receives(prefix)
verdict = "correct" if ok else f"WRONG — heard as {assumed}, real subnet LOST"
print(f"advertised {prefix:<16} RIPv1 assumes /{assumed.prefixlen:<3} {verdict}")
# advertised 10.1.5.0/24 RIPv1 assumes /8 WRONG — heard as 10.0.0.0/8, real subnet LOST
# advertised 172.16.8.0/24 RIPv1 assumes /16 WRONG — heard as 172.16.0.0/16, real subnet LOST
# advertised 192.168.1.0/24 RIPv1 assumes /24 correct
# advertised 10.0.0.0/8 RIPv1 assumes /8 correct
# This is exactly why VLSM is impossible with RIPv1.
# Bandwidth spent on periodic full-table updates.
def rip_overhead(routes, neighbors, interval=30):
entry_bytes, header = 20, 24 # RIPv2 sizes
per_update = header + routes * entry_bytes
packets = -(-routes // 25) # max 25 entries per packet
bytes_per_s = (per_update * neighbors) / interval
return per_update, packets, bytes_per_s * 8 / 1000 # kb/s
print()
for routes in (10, 100, 500):
size, pkts, kbps = rip_overhead(routes, neighbors=4)
print(f"{routes:>4} routes, 4 neighbours: {size:>6,} B/update, {pkts:>2} packets, "
f"{kbps:>7.2f} kb/s of constant background traffic")
# 10 routes, 4 neighbours: 224 B/update, 1 packets, 0.24 kb/s of constant background traffic
# 100 routes, 4 neighbours: 2,024 B/update, 4 packets, 2.16 kb/s of constant background traffic
# 500 routes, 4 neighbours: 10,024 B/update, 20 packets, 10.69 kb/s of constant background traffic
# Sent every 30 s whether or not anything changed. OSPF sends only CHANGES.
# Configuring RIPv2 — always force version 2; v1 is never the right choice.
# router rip
# version 2
# network 10.0.0.0
# no auto-summary <- CRITICAL: without this, RIPv2 still summarises
# to classful boundaries at major network edges
# passive-interface default
# no passive-interface GigabitEthernet0/1
$ vtysh -c "show ip rip"
# Codes: R - RIP, C - connected, S - Static, O - OSPF, B - BGP
# Sub-codes: (n) - normal, (s) - static, (d) - default, (r) - redistribute
# Network Next Hop Metric From Tag Time
# R(n) 10.3.0.0/24 10.0.0.2 2 10.0.0.2 0 02:41
# R(n) 10.4.0.0/24 10.0.0.2 3 10.0.0.2 0 02:41
# ^ hop count, nothing more
# The v1/v2 difference, visible in a capture:
$ sudo tcpdump -i eth0 -n -v udp port 520
# RIPv1: IP 10.0.0.2.520 > 255.255.255.255.520: RIPv1, Response
# 10.0.0.0, metric 1 ^^^^^^^^^^^^^^^ broadcast
# ^^^^^^^^ no mask shown — there is none in the packet
# RIPv2: IP 10.0.0.2.520 > 224.0.0.9.520: RIPv2, Response
# AFI IPv4, 10.3.0.0/24, tag 0x0000, metric 2, next-hop: self
# ^^^ mask present
# Authentication (RIPv2 only):
# key chain RIPKEY
# key 1
# key-string SecretValue
# interface GigabitEthernet0/1
# ip rip authentication mode md5
# ip rip authentication key-chain RIPKEY
# Proving the 15-hop limit is real:
$ vtysh -c "show ip rip" | awk '$4>=16 {print "UNREACHABLE:", $2}'
# UNREACHABLE: 10.99.0.0/24
# The network exists and is healthy. RIP cannot express its distance.
# RIPng for IPv6 — same algorithm, different transport:
# router ipv6 rip
# network eth0
$ sudo tcpdump -i eth0 -n udp port 521 # note: 521, not 520
# IP6 fe80::1.521 > ff02::9.521: RIPng, Response, length 44
# ^^^^^^^ the IPv6 RIP multicast group
Key takeaways
- RIP is distance-vector with hop count as its only metric; routers broadcast their whole table every 30 seconds.
- RIPv1 was classful — no mask in updates — so VLSM was impossible; it broadcast to
255.255.255.255with no authentication. - RIPv2 added masks (classless), multicast
224.0.0.9, MD5 authentication, route tags, and a next-hop field. - RIPng is the IPv6 version: multicast
ff02::9, UDP port 521. - The algorithm's limits remain: 15-hop maximum, hop count that ignores bandwidth, minutes-long convergence, periodic full-table updates, and no hierarchy.
- RIP will choose a 64 kb/s one-hop path over a 30 Gb/s three-hop path — the metric is the defect, not the algorithm.
- Always configure
version 2andno auto-summary; RIPv2 still summarises at classful boundaries otherwise. - RIP is appropriate only for very small or heterogeneous networks; OSPF, IS-IS, or EIGRP is otherwise the correct choice.
🧪 Practice
- State three things RIPv2 added over RIPv1 and name the one that makes VLSM possible.
- Given a 3-hop path of gigabit links and a 1-hop 128 kb/s link, determine RIP's choice and explain why this is a metric problem rather than an algorithm problem.
- Compute RIPv2's periodic update bandwidth for 200 routes and 3 neighbours, and contrast it with a protocol that sends only changes.
- Interview-style: Why is RIP limited to 15 hops, and why did RIPv2 not raise the limit? (Hint: the number is doing double duty — recall what value signals "unreachable" and what would happen to loop duration if it were raised.)
OSPF Areas and LSAs
Theory
OSPF (Open Shortest Path First, RFC 2328 for v2, RFC 5340 for v3/IPv6) is the dominant enterprise IGP. It is link-state: every router floods descriptions of its own links, every router builds an identical LSDB, and every router runs Dijkstra from itself (6.2).
That design has one scaling problem. The LSDB holds the entire topology, SPF runs over all of it, and every topology change anywhere triggers a recomputation everywhere. In a 2,000-router network, a single flapping link in a distant branch costs every router a full SPF run.
Areas are OSPF's answer: partition the network so that detailed topology stays local. An area is a set of routers sharing an identical LSDB for that area only. Topology detail does not leave an area — only summaries of reachable prefixes do.
The structure is a strict two-level hierarchy with one rigid rule: every area must connect to area 0, the backbone. All inter-area traffic transits the backbone, which prevents inter-area routing loops by construction — there is no way to form a cycle between areas if all paths pass through a single hub. When physical topology cannot satisfy this, a virtual link tunnels a disconnected area through an intermediate area, though this is a repair rather than a design.
Router roles follow from the structure:
| Role | Position |
|---|---|
| Internal router | All interfaces in one area |
| ABR (Area Border Router) | Interfaces in area 0 and another area; generates summaries |
| Backbone router | At least one interface in area 0 |
| ASBR (AS Boundary Router) | Redistributes external routes into OSPF |
LSAs (Link-State Advertisements) are the units flooded. The main types:
| Type | Name | Originated by | Scope |
|---|---|---|---|
| 1 | Router LSA | Every router | Its own area |
| 2 | Network LSA | The DR on a multi-access segment | Its own area |
| 3 | Summary LSA | ABR | Other areas — inter-area prefixes |
| 4 | ASBR Summary | ABR | How to reach an ASBR |
| 5 | AS-External | ASBR | Flooded through the whole AS |
| 7 | NSSA External | ASBR inside an NSSA | That area only; the ABR converts it to type 5 |
The crucial distinction: types 1 and 2 describe topology and never leave their area. Type 3 carries only prefix-and-cost — a distance-vector-like summary. OSPF is therefore link-state within an area and distance-vector-ish between areas.
Special area types further reduce LSDB size by filtering external routes:
| Area type | Blocks | Substitutes |
|---|---|---|
| Stub | Type 5 (externals) | A default route from the ABR |
| Totally stubby | Types 3, 4, 5 | A default route only (Cisco-specific) |
| NSSA | Type 5, but allows local externals as type 7 | Type 7 → 5 conversion at the ABR |
A branch office with one exit gains nothing from carrying 500,000 external routes; a stub area gives it a default route instead, and its routers hold a fraction of the state.
The design guidance that follows: keep areas to roughly 50–100 routers, keep area 0 contiguous and well-connected, place ABRs on capable hardware, and summarise at ABRs — summarisation is what actually contains the flooding of individual prefix changes.
Examples
OSPF's two-level hierarchy — every area touches area 0
+-------------------+
| AREA 0 |
| (backbone) |
| R1 ---- R2 |
+--/-----------\----+
ABR1 ABR2
| |
+-------+----+ +----+--------+
| AREA 1 | | AREA 2 |
| R3 -- R4 | | R5 -- R6 |
+------------+ +-------------+
R3 knows area 1's FULL TOPOLOGY (types 1, 2).
R3 knows area 2 only as PREFIXES AND COSTS (type 3).
A link flap inside area 2 does NOT cause SPF in area 1.
All inter-area traffic crosses area 0 -> no inter-area loops possible.
What crosses an area boundary, and what does not
inside area 1: at ABR1: inside area 0:
+------------------+ +------------------+
| type 1: R3 has | ---FILTERED OUT---> X | |
| links to R4 | | |
| type 2: DR says | ---FILTERED OUT---> X | |
| segment X | | |
+------------------+ | |
| prefix 10.1.0.0/16 cost 30 -> type 3 ------->| 10.1.0.0/16 c 30 |
+------------------+ +------------------+
^ prefix + cost only:
no topology at all
Stub areas: trading detail for scale
NORMAL area: 50 internal + 3 summaries + 500,000 externals
-> huge LSDB, big SPF, lots of memory
STUB area: 50 internal + 3 summaries + 1 default route
-> tiny LSDB
TOTALLY STUBBY: 50 internal + 1 default route
-> smallest possible
Correct whenever the area has a single exit: knowing 500,000 external
routes cannot change the decision, because there is only one way out.
LSA_TYPES = {
1: ("Router LSA", "every router", "own area", "the router's own links (TOPOLOGY)"),
2: ("Network LSA", "the DR", "own area", "routers on a multi-access segment (TOPOLOGY)"),
3: ("Summary LSA", "ABR", "other areas","inter-area PREFIX + COST (no topology)"),
4: ("ASBR Summary", "ABR", "other areas","how to reach an ASBR"),
5: ("AS-External", "ASBR", "whole AS", "routes redistributed from outside OSPF"),
7: ("NSSA External", "ASBR in NSSA", "NSSA only", "external route; ABR converts it to type 5"),
}
print(f"{'type':<5}{'name':<16}{'originator':<15}{'scope':<13}contents")
for t, (name, orig, scope, contents) in sorted(LSA_TYPES.items()):
print(f"{t:<5}{name:<16}{orig:<15}{scope:<13}{contents}")
# What each area type actually stores — the scaling argument, quantified.
def lsdb_size(area_type, internal_routers, other_areas, external_routes):
type1 = internal_routers
type2 = max(1, internal_routers // 4)
if area_type == "normal":
type3, type5 = other_areas * 10, external_routes
elif area_type == "stub":
type3, type5 = other_areas * 10, 1 # default route only
elif area_type == "totally stubby":
type3, type5 = 1, 0 # default route only
elif area_type == "nssa":
type3, type5 = other_areas * 10, 5 # local externals as type 7
return {"type1": type1, "type2": type2, "type3": type3, "type5": type5,
"total": type1 + type2 + type3 + type5}
print()
print(f"{'area type':<18}{'t1':>6}{'t2':>6}{'t3':>7}{'t5':>9}{'TOTAL':>10}")
for at in ("normal", "stub", "nssa", "totally stubby"):
s = lsdb_size(at, internal_routers=50, other_areas=3, external_routes=500_000)
print(f"{at:<18}{s['type1']:>6}{s['type2']:>6}{s['type3']:>7}"
f"{s['type5']:>9,}{s['total']:>10,}")
# area type t1 t2 t3 t5 TOTAL
# normal 50 12 30 500,000 500,092
# stub 50 12 30 1 93
# nssa 50 12 30 5 97
# totally stubby 50 12 1 0 63
# 500,092 LSAs vs 63. For an area with one exit, the extra 500,029 cannot
# change a single forwarding decision.
# Why areas contain the blast radius of a topology change.
def spf_runs_per_change(total_routers, areas, change_in_area=True):
per_area = total_routers // areas
return per_area if change_in_area else total_routers
print()
for areas in (1, 4, 10):
affected = spf_runs_per_change(2000, areas)
print(f"2000 routers in {areas:>2} area(s): one intra-area link flap causes "
f"{affected:>5,} SPF runs ({100*affected/2000:>5.1f}% of the network)")
# 2000 routers in 1 area(s): one intra-area link flap causes 2,000 SPF runs (100.0% of the network)
# 2000 routers in 4 area(s): one intra-area link flap causes 500 SPF runs ( 25.0% of the network)
# 2000 routers in 10 area(s): one intra-area link flap causes 200 SPF runs ( 10.0% of the network)
# Summarisation at the ABR: many prefixes become one advertisement.
import ipaddress
def summarise(prefixes):
nets = [ipaddress.ip_network(p) for p in prefixes]
return list(ipaddress.collapse_addresses(nets))
area1 = [f"10.1.{i}.0/24" for i in range(8)]
summary = summarise(area1)
print(f"\narea 1 originates {len(area1)} prefixes: {area1[0]} ... {area1[-1]}")
print(f"ABR advertises {len(summary)}: {[str(s) for s in summary]}")
print("a flap of any ONE /24 inside area 1 changes NOTHING outside it,")
print("because the summary's contents are invisible from the other side.")
# area 1 originates 8 prefixes: 10.1.0.0/24 ... 10.1.7.0/24
# ABR advertises 1: ['10.1.0.0/21']
# a flap of any ONE /24 inside area 1 changes NOTHING outside it,
# because the summary's contents are invisible from the other side.
# Area membership and router roles.
$ vtysh -c "show ip ospf"
# OSPF Routing Process, Router ID: 10.0.0.1
# This router is an ABR, Number of areas attached to this router: 2
# ^^^ interfaces in area 0 AND another area
# Area ID: 0.0.0.0 (Backbone)
# Number of interfaces in this area: 2
# SPF algorithm executed 12 times
# Area ID: 0.0.0.1 (Stub)
# ^^^^ no type 5 LSAs will be flooded here
# The LSDB, by type:
$ vtysh -c "show ip ospf database" | grep -A3 "Summary Link States"
# Summary Link States (Area 0.0.0.1)
# Link ID ADV Router Age Seq# CkSum Route
# 10.2.0.0 10.0.0.1 421 0x80000003 0x2a4b 10.2.0.0/16
# ^^^^^^^^ the ABR originated this type 3: PREFIX + COST only.
# Area 1's routers cannot see area 2's topology.
$ vtysh -c "show ip ospf database" | awk '/States/{print}'
# Router Link States (Area 0.0.0.0) <- type 1, topology
# Net Link States (Area 0.0.0.0) <- type 2, topology
# Summary Link States (Area 0.0.0.1) <- type 3, prefixes
# AS External Link States <- type 5, whole-AS scope
# Configuring a stub area (must match on EVERY router in the area):
# router ospf 1
# area 1 stub <- blocks type 5; ABR injects a default
# area 1 stub no-summary <- TOTALLY STUBBY: also blocks type 3
# area 1 nssa <- allows local externals as type 7
# Summarisation at the ABR — the single most effective scaling control:
# router ospf 1
# area 1 range 10.1.0.0 255.255.248.0
# ^^^^^^^^^^^^^^ advertise one /21 instead of eight /24s
$ vtysh -c "show ip ospf database summary" | grep 10.1
# 10.1.0.0 10.0.0.1 312 0x80000002 0x7c19 10.1.0.0/21
# ^^^^^^^^^^^ one entry
# A virtual link: repairing an area that cannot physically touch area 0.
# router ospf 1
# area 1 virtual-link 10.0.0.9
# ! tunnels area 2's connection to the backbone THROUGH area 1.
# ! A repair, not a design — fix the physical topology if you can.
$ vtysh -c "show ip ospf border-routers"
# ============ OSPF router routing table =============
# R 10.0.0.5 [10] area: 0.0.0.0, ABR
# R 10.0.0.9 [20] area: 0.0.0.0, ASBR
# ^^^^ redistributes externals
Key takeaways
- OSPF is link-state: identical LSDB per area, Dijkstra run independently by each router.
- Areas partition the LSDB so topology detail stays local and a change triggers SPF only within its own area.
- Every area must connect to area 0 (the backbone), which makes inter-area loops structurally impossible; virtual links are a repair for topologies that cannot.
- Roles: internal, ABR (area 0 plus another area), backbone, ASBR (redistributes externals).
- Types 1 and 2 carry topology and never leave their area; type 3 carries only prefix and cost, so OSPF is link-state within an area and distance-vector-like between them.
- Type 5 (AS-external) floods through the entire AS; type 7 exists so an NSSA can originate externals locally.
- Stub, totally stubby, and NSSA areas block external LSAs and substitute a default route, shrinking the LSDB by orders of magnitude.
- Summarisation at ABRs hides individual prefix changes, which is the main practical scaling control; target roughly 50–100 routers per area.
🧪 Practice
- Name the LSA types carrying topology versus those carrying only prefixes, and state which cross an area boundary.
- A branch area has one ABR and needs no external routes. Choose the area type and compute the LSDB reduction from 200,000 externals.
- Explain why every OSPF area must touch area 0, and what property this guarantees.
- Interview-style: A link inside area 3 flaps constantly, yet routers in area 1 show no SPF activity. Explain why, and name what would change if the areas were merged. (Hint: ask which LSA types cross the ABR, and whether any of them describe the flapping link.)
OSPF Adjacency Formation and DR/BDR
Theory
Before two OSPF routers exchange any routing information, they must form an adjacency — a verified, synchronised relationship. The process is deliberately careful, because a link-state protocol's correctness depends entirely on every router having an identical database.
Adjacency progresses through defined states:
| State | Meaning |
|---|---|
| Down | No Hello received |
| Init | A Hello arrived, but I do not see myself in its neighbour list |
| 2-Way | I see myself in their Hello — bidirectional communication confirmed. DR/BDR is elected here |
| ExStart | Negotiating master/slave and the initial sequence number |
| Exchange | Trading Database Description (DBD) packets — summaries of held LSAs |
| Loading | Requesting the LSAs found to be missing |
| Full | Databases are synchronised; the adjacency is operational |
The 2-Way state is the important checkpoint. Seeing your own router ID in a neighbour's Hello proves the link works in both directions — a one-way link would otherwise produce a database that can never converge.
Hello packets also enforce agreement. Several parameters must match or the adjacency never forms, and a mismatch is one of the most common OSPF faults:
- Area ID
- Hello and Dead intervals
- Authentication type and key
- Subnet and mask (the interfaces must be on the same subnet)
- Stub area flags
- MTU (checked during Exchange; a mismatch famously hangs the adjacency in ExStart/Exchange)
The DR/BDR mechanism solves a separate problem specific to multi-access segments like Ethernet. If n routers share a segment and every pair formed a full adjacency, the count would be n(n-1)/2 — for 10 routers, 45 adjacencies, each flooding to the others. The flooding overhead is quadratic and wasteful, since every router on the segment can already hear every other directly.
OSPF elects a Designated Router (DR) and a Backup DR (BDR). All other routers (DROTHERs) form full adjacencies only with the DR and BDR, reducing adjacency count from n(n-1)/2 to 2n - 3. The DR represents the segment: it originates the type 2 Network LSA describing which routers are attached, and relays flooding.
Multicast addresses divide the work: routers send to 224.0.0.6 (AllDRouters) to reach the DR and BDR; the DR floods to 224.0.0.5 (AllSPFRouters) to reach everyone.
Election rules have two properties that surprise people:
Highest priority wins; the highest router ID breaks ties. Priority 0 means "never be DR".
The election is non-preemptive. A new router with a higher priority does not take over from an existing DR. This deliberately avoids re-elections and re-synchronisation every time a router reboots — but it means the DR is often whichever router happened to boot first, not the one you intended. To choose the DR deliberately, set priorities and force a re-election.
On point-to-point links there are only two routers, so no DR is elected — the optimisation would serve no purpose.
Examples
Adjacency state machine
Down --Hello received--> Init --I see myself in their Hello--> 2-Way
|
(DR/BDR elected here) |
|
DROTHER-to-DROTHER pairs STOP at 2-Way <-----------------+
|
adjacency needed? ----------------------->
v
ExStart --negotiate master/slave--> Exchange --DBD packets--> Loading
|
--request missing--> Full
^
databases synchronised -+
Why DR/BDR exists: adjacency count on a shared segment
WITHOUT DR, 6 routers on one Ethernet:
R1---R2 R1---R3 R1---R4 R1---R5 R1---R6
R2---R3 R2---R4 R2---R5 R2---R6
R3---R4 R3---R5 R3---R6
R4---R5 R4---R6
R5---R6 = 15 adjacencies, all flooding to each other
WITH DR/BDR:
[DR]---[BDR]
/ | \ / | \
R3 R4 R5 R6 each DROTHER adjacent ONLY to DR and BDR
= 2(6) - 3 = 9 adjacencies
Multicast roles on the segment
DROTHER --> 224.0.0.6 (AllDRouters) "here is my update, DR and BDR only"
DR --> 224.0.0.5 (AllSPFRouters) "flooding to everyone on the segment"
The BDR listens silently, ready to take over. It does NOT flood
while the DR is alive.
def adjacencies(n, use_dr=False):
if not use_dr:
return n * (n - 1) // 2 # full mesh
return max(0, 2 * n - 3) # each DROTHER to DR and BDR
print(f"{'routers':>8}{'full mesh':>12}{'with DR/BDR':>14}{'reduction':>12}")
for n in (2, 4, 6, 10, 20, 50):
full, dr = adjacencies(n), adjacencies(n, use_dr=True)
red = f"{100*(1-dr/full):.0f}%" if full else "n/a"
print(f"{n:>8}{full:>12,}{dr:>14,}{red:>12}")
# routers full mesh with DR/BDR reduction
# 2 1 1 0%
# 4 6 5 17%
# 6 15 9 40%
# 10 45 17 62%
# 20 190 37 81%
# 50 1,225 97 92%
# The saving grows with segment size — which is exactly where it is needed.
# DR/BDR election: highest priority, then highest router ID, NON-PREEMPTIVE.
def elect(routers, existing_dr=None, existing_bdr=None):
"""routers: [(router_id, priority)]. Priority 0 = never DR."""
eligible = [(rid, pri) for rid, pri in routers if pri > 0]
if existing_dr: # NON-PREEMPTIVE: incumbent stays
return existing_dr, existing_bdr
ranked = sorted(eligible, key=lambda r: (r[1], [int(o) for o in r[0].split(".")]),
reverse=True)
dr = ranked[0][0] if ranked else None
bdr = ranked[1][0] if len(ranked) > 1 else None
return dr, bdr
segment = [("10.0.0.1", 1), ("10.0.0.2", 1), ("10.0.0.3", 1), ("10.0.0.4", 0)]
dr, bdr = elect(segment)
print(f"\nfresh election: DR={dr} BDR={bdr}")
print("10.0.0.4 has priority 0 -> never eligible, whatever its ID")
# fresh election: DR=10.0.0.3 BDR=10.0.0.2
# 10.0.0.4 has priority 0 -> never eligible, whatever its ID
# Now a higher-priority router joins — and does NOT take over.
segment.append(("10.0.0.9", 255))
dr2, bdr2 = elect(segment, existing_dr=dr, existing_bdr=bdr)
print(f"\n10.0.0.9 joins with priority 255 (the maximum):")
print(f" DR={dr2} BDR={bdr2} <- UNCHANGED: the election is non-preemptive")
print(" To make it DR you must force a re-election (clear the OSPF process).")
# 10.0.0.9 joins with priority 255 (the maximum):
# DR=10.0.0.3 BDR=10.0.0.2 <- UNCHANGED: the election is non-preemptive
# To make it DR you must force a re-election (clear the OSPF process).
# The parameters that MUST match, and the symptom when they do not.
MUST_MATCH = {
"area ID": "adjacency never leaves Init/Down",
"hello interval": "adjacency never forms; Hellos are ignored",
"dead interval": "adjacency never forms",
"authentication": "adjacency never forms; auth mismatch logged",
"subnet/mask": "Hellos ignored — not on a common subnet",
"stub flags": "adjacency never forms",
"MTU": "STUCK in ExStart/Exchange — the classic symptom",
}
print()
for param, symptom in MUST_MATCH.items():
print(f"{param:<18} mismatch -> {symptom}")
# area ID mismatch -> adjacency never leaves Init/Down
# hello interval mismatch -> adjacency never forms; Hellos are ignored
# dead interval mismatch -> adjacency never forms
# authentication mismatch -> adjacency never forms; auth mismatch logged
# subnet/mask mismatch -> Hellos ignored — not on a common subnet
# stub flags mismatch -> adjacency never forms
# MTU mismatch -> STUCK in ExStart/Exchange — the classic symptom
# Which pairs actually reach Full on a multi-access segment?
def expected_state(role_a, role_b):
if "DR" in (role_a, role_b) or "BDR" in (role_a, role_b):
return "FULL"
return "2-WAY (normal! not a fault)"
print()
for a, b in [("DROTHER", "DR"), ("DROTHER", "BDR"), ("DROTHER", "DROTHER"), ("DR", "BDR")]:
print(f"{a:<8} <-> {b:<8} -> {expected_state(a, b)}")
# DROTHER <-> DR -> FULL
# DROTHER <-> BDR -> FULL
# DROTHER <-> DROTHER -> 2-WAY (normal! not a fault)
# DR <-> BDR -> FULL
# Seeing 2-WAY between two DROTHERs is CORRECT. It is not a problem
# to troubleshoot — it is the optimisation working as designed.
# Neighbour states — note that 2-WAY between DROTHERs is expected.
$ vtysh -c "show ip ospf neighbor"
# Neighbor ID Pri State Dead Time Address Interface
# 10.0.0.3 1 Full/DR 00:00:34 10.0.12.3 eth0:10.0.12.1
# 10.0.0.2 1 Full/BDR 00:00:32 10.0.12.2 eth0:10.0.12.1
# 10.0.0.4 1 2-Way/DROTHER 00:00:31 10.0.12.4 eth0:10.0.12.1
# ^^^^^^^^^^^^^ CORRECT — both are DROTHERs.
# Do not "fix" this.
# The classic MTU-mismatch hang:
$ vtysh -c "show ip ospf neighbor"
# 10.0.0.7 1 ExStart/DROTHER 00:00:38 10.0.13.7 eth1:10.0.13.1
# ^^^^^^^ stuck here = almost always an MTU mismatch
$ ip link show eth1 | grep -o "mtu [0-9]*"
# mtu 1500
$ ssh 10.0.13.7 "ip link show eth0 | grep -o 'mtu [0-9]*'"
# mtu 9000 <- there it is
# Workaround if MTUs genuinely must differ:
# interface GigabitEthernet0/1
# ip ospf mtu-ignore
# Influencing the election — priority plus a forced re-election:
# interface GigabitEthernet0/1
# ip ospf priority 255 <- strongly prefer DR
# ip ospf priority 0 <- never become DR (typical for edge routers)
$ vtysh -c "clear ip ospf process" # forces a re-election; NON-preemptive
# means this is the only way to take over
# Verify the DR and the type 2 LSA it originates:
$ vtysh -c "show ip ospf interface eth0" | grep -E "State|Designated"
# State DROther, Priority 1, Designated Router (ID) 10.0.0.3
# Backup Designated Router (ID) 10.0.0.2
$ vtysh -c "show ip ospf database network"
# Net Link States (Area 0.0.0.0)
# Link ID ADV Router Age Seq# CkSum
# 10.0.12.3 10.0.0.3 284 0x80000004 0x1f2a
# ^^^^^^^^ the DR, describing the segment on everyone's behalf
# Watch the multicast roles in action:
$ sudo tcpdump -i eth0 -n proto ospf
# IP 10.0.12.1 > 224.0.0.6: OSPFv2, LS-Update <- DROTHER to DR/BDR
# IP 10.0.12.3 > 224.0.0.5: OSPFv2, LS-Update <- DR floods to everyone
Key takeaways
- Adjacency states: Down → Init → 2-Way → ExStart → Exchange → Loading → Full.
- 2-Way confirms bidirectional communication (you see your own router ID in their Hello) and is where DR/BDR is elected.
- Hello parameters that must match: area ID, hello/dead intervals, authentication, subnet/mask, and stub flags; an MTU mismatch hangs the adjacency in ExStart/Exchange.
- On multi-access segments, a full mesh would need
n(n-1)/2adjacencies; DR/BDR reduces this to2n - 3. - DROTHERs stay at 2-Way with each other — this is correct, not a fault.
- The DR originates the type 2 Network LSA representing the segment.
- Multicast split: DROTHERs send to
224.0.0.6(AllDRouters); the DR floods to224.0.0.5(AllSPFRouters). - Election is by highest priority, then highest router ID, priority 0 means never, and it is non-preemptive — a better router will not take over without a forced re-election.
- Point-to-point links elect no DR, since only two routers exist.
🧪 Practice
- List the adjacency states in order and state what is proven at 2-Way.
- Compute adjacency counts with and without DR/BDR for 12 routers on one segment, and give the percentage reduction.
- Two routers are stuck in ExStart. Name the most likely cause and two ways to resolve it.
- Interview-style: You configure priority 255 on a new router expecting it to become DR, but it does not. Why, and what would you do? (Hint: consider whether OSPF prefers stability or optimality when a DR already exists and is healthy.)
IS-IS Overview
Theory
IS-IS (Intermediate System to Intermediate System) is the other major link-state IGP. Algorithmically it is OSPF's twin — flood link state, build an identical database, run Dijkstra — but it comes from a different lineage, and that lineage gives it properties that made it the preferred protocol among large service providers.
IS-IS was designed by ISO for the OSI protocol suite, not for IP. "Intermediate System" is OSI's word for a router. It was later extended to carry IP routing information (Integrated IS-IS, RFC 1195), and the key architectural consequence is that IS-IS does not run over IP. It runs directly over layer 2, as its own protocol with its own frames.
That single fact produces several practical advantages:
Protocol independence. Since IS-IS does not depend on IP, carrying IPv6 required only a new TLV — no new protocol version. Compare OSPF, where IPv6 support required OSPFv3, effectively a separate protocol with a separate process and separate adjacencies. An IS-IS network can be dual-stacked with one protocol instance and one set of adjacencies.
Immunity to IP-level attacks. You cannot reach the IS-IS control plane by sending IP packets to a router, because IS-IS does not listen on IP at all. For a service provider backbone, this is a meaningful security property.
Extensibility via TLVs. IS-IS encodes everything as Type-Length-Value structures, so new capabilities are new TLVs that older routers ignore. This is why IS-IS absorbed traffic engineering, segment routing, and multi-topology extensions smoothly, while OSPF needed opaque LSAs and, in places, new protocol versions.
IS-IS addressing uses NSAP/NET rather than IP. A NET looks like 49.0001.1921.6800.1001.00, decomposing into an area ID (49.0001), a system ID (six bytes, usually derived from an IP address for readability), and an NSEL of 00. Each router has one NET, so — unlike OSPF — a router belongs to exactly one area.
That difference reshapes the hierarchy:
| OSPF | IS-IS | |
|---|---|---|
| Hierarchy | Area 0 plus other areas | Level 2 (backbone) and Level 1 (area) |
| Area boundary | On the router (ABRs have interfaces in two areas) | On the link between routers |
| Router membership | Can be in multiple areas | Exactly one area |
| Levels | — | L1 (intra-area), L2 (inter-area), L1/L2 (both) |
| Runs over | IP (protocol 89) | Layer 2 directly |
| IPv6 | Requires OSPFv3 | A new TLV in the same protocol |
| Units flooded | LSAs (several types) | LSPs (one type, TLV-structured) |
| Terminology | Router, network | IS, ES, NET, PDU |
Because the boundary is on the link, the backbone is not a designated area but a contiguous chain of L2-capable routers. Providers find this more flexible than OSPF's rule that every area must touch area 0.
Two smaller differences matter operationally. IS-IS elects a DIS (Designated Intermediate System) on broadcast segments, analogous to OSPF's DR, but with two differences: there is no backup (the DIS is re-elected quickly if it fails), and the election is preemptive. And IS-IS metrics were originally 6 bits (max 63), which the wide metrics extension raised to 32 bits — essential for modern networks and standard practice today.
The practical summary: both are excellent, and the choice is usually organisational. Enterprises pick OSPF (more familiar, better documented, more vendor-neutral tooling). Large providers pick IS-IS (cleaner IPv6 story, TLV extensibility, layer-2 isolation, and demonstrated scaling to thousands of routers in a single level).
Examples
Area boundaries: on the ROUTER (OSPF) vs on the LINK (IS-IS)
OSPF — the ABR sits IN both areas:
area 1 area 0
[R1]---[R2]---[ABR]---[R3]---[R4]
^^^^^ this router has interfaces in BOTH areas
IS-IS — the boundary is the LINK; each router is in exactly one area:
area 49.0001 area 49.0002
[R1]---[R2]==========[R3]---[R4]
L1 L1/L2 ^ L1/L2 L1
|
the L2 adjacency IS the boundary.
R2 is in area 0001. R3 is in area 0002. Neither is in both.
The L2 backbone = the contiguous chain of L2-capable routers,
not a numbered area that everything must touch.
A NET address, decomposed
49.0001.1921.6800.1001.00
\_/ \__/ \________________/ \/
| | | |
| | | NSEL (always 00 for a router)
| | system ID (6 bytes; here 192.168.1.1 written
| | as 1921.6800.1001 for readability)
| area ID
AFI (49 = private/local, like RFC 1918 for OSI)
IPv6 support: one TLV vs a whole new protocol
OSPF: OSPFv2 process (IPv4) + OSPFv3 process (IPv6)
two processes, two LSDBs, two sets of adjacencies,
two configurations that can silently diverge
IS-IS: ONE process, ONE set of adjacencies
TLV 128/130 -> IPv4 reachability
TLV 236 -> IPv6 reachability
Dual-stack for the cost of one more TLV.
def parse_net(net):
"""Decompose an IS-IS NET address."""
parts = net.split(".")
afi, area = parts[0], ".".join(parts[1:-4])
system_id = ".".join(parts[-4:-1])
nsel = parts[-1]
# System IDs are conventionally an IP address padded to 12 hex digits.
raw = system_id.replace(".", "")
ip_like = ".".join(str(int(raw[i:i+3])) for i in range(0, 12, 3))
return {"AFI": afi, "area_id": f"{afi}.{area}", "system_id": system_id,
"looks_like_ip": ip_like, "NSEL": nsel}
for net in ["49.0001.1921.6800.1001.00", "49.0002.1921.6800.2002.00"]:
d = parse_net(net)
print(f"{net}")
for k, v in d.items():
print(f" {k:<14} {v}")
# 49.0001.1921.6800.1001.00
# AFI 49
# area_id 49.0001
# system_id 1921.6800.1001
# looks_like_ip 192.168.1.1
# NSEL 00
# Level roles determine what a router floods and what it knows.
LEVELS = {
"L1": ("intra-area only", "area topology + a default toward the nearest L1/L2"),
"L2": ("backbone only", "inter-area routes; the L2 chain IS the backbone"),
"L1/L2": ("both", "full area topology AND backbone — like an OSPF ABR"),
}
print()
for lvl, (scope, knows) in LEVELS.items():
print(f"{lvl:<6} {scope:<18} knows: {knows}")
# L1 intra-area only knows: area topology + a default toward the nearest L1/L2
# L2 backbone only knows: inter-area routes; the L2 chain IS the backbone
# L1/L2 both knows: full area topology AND backbone — like an OSPF ABR
# Metric ranges: why wide metrics are mandatory on modern networks.
def max_path_cost(metric_bits, hops):
return (2 ** metric_bits - 1) * hops
print()
for bits, name in [(6, "narrow (original)"), (32, "wide (RFC 5305)")]:
per_link = 2 ** bits - 1
print(f"{name:<20} max per-link metric {per_link:>12,} "
f"10-hop path max {max_path_cost(bits, 10):>14,}")
# narrow (original) max per-link metric 63 10-hop path max 630
# wide (RFC 5305) max per-link metric 4,294,967,295 10-hop path max 42,949,672,950
# With only 63 values, a 10 Gb/s and a 100 Gb/s link cannot be distinguished
# meaningfully — the same problem as OSPF's reference bandwidth.
# The dual-stack configuration burden, compared.
def dual_stack_cost(protocol):
if protocol == "OSPF":
return {"processes": 2, "adjacency_sets": 2, "databases": 2,
"note": "OSPFv2 + OSPFv3 are separate protocols"}
return {"processes": 1, "adjacency_sets": 1, "databases": 1,
"note": "one protocol; IPv6 reachability is TLV 236"}
print()
for p in ("OSPF", "IS-IS"):
c = dual_stack_cost(p)
print(f"{p:<7} processes={c['processes']} adjacencies={c['adjacency_sets']} "
f"databases={c['databases']} ({c['note']})")
# OSPF processes=2 adjacencies=2 databases=2 (OSPFv2 + OSPFv3 are separate protocols)
# IS-IS processes=1 adjacencies=1 databases=1 (one protocol; IPv6 reachability is TLV 236)
# DIS vs DR — similar idea, different behaviour.
print()
comparison = [("backup elected?", "OSPF: yes (BDR)", "IS-IS: no backup"),
("preemptive?", "OSPF: NO", "IS-IS: YES"),
("tie-break", "OSPF: highest router ID", "IS-IS: highest MAC")]
for aspect, ospf, isis in comparison:
print(f"{aspect:<18}{ospf:<26}{isis}")
# backup elected? OSPF: yes (BDR) IS-IS: no backup
# preemptive? OSPF: NO IS-IS: YES
# tie-break OSPF: highest router ID IS-IS: highest MAC
# IS-IS runs over layer 2 — there is no IP socket to attack or filter.
$ sudo tcpdump -i eth0 -e isis -c 1
# 00:1a:2b:3c:4d:5e > 01:80:c2:00:00:14, 802.3, length 1497: OSI, IS-IS,
# ^^^^^^^^^^^^^^^^^ the AllL1ISs multicast MAC
# length 1476, L1 LSP, hlen 27, v1, pdu-v1, sys-id 1921.6800.1001.00-00
# NOTE: no IP header anywhere in this frame.
$ sudo tcpdump -i eth0 "ip proto 89" -c 1 # OSPF's protocol number
# (nothing — IS-IS does not appear here, because it is not carried in IP)
# Basic configuration:
# router isis CORE
# net 49.0001.1921.6800.1001.00
# is-type level-2-only <- backbone router
# metric-style wide <- ESSENTIAL on modern networks
# address-family ipv6 unicast <- same process, just another TLV
# interface GigabitEthernet0/1
# ip router isis CORE
# ipv6 router isis CORE <- ONE adjacency carries both
$ vtysh -c "show isis neighbor"
# Area CORE:
# System Id Interface L State Holdtime SNPA
# 1921.6800.2002 eth0 2 Up 27 2020.2020.2020
# ^ level 2 adjacency
$ vtysh -c "show isis database"
# Area CORE:
# IS-IS Level-2 link-state database:
# LSP ID PduLen SeqNumber Chksum Holdtime ATT/P/OL
# 1921.6800.1001.00-00 578 0x00000012 0x8a21 1140 0/0/0
# ^^^ attached/partition/overload
# One protocol, both address families — inspect the TLVs:
$ vtysh -c "show isis database detail" | grep -E "IPv4|IPv6"
# IPv4 Reachability: 10.1.0.0/24 (Metric: 10)
# IPv6 Reachability: 2001:db8:1::/64 (Metric: 10)
# ^ same LSP, same adjacency, both families. No second process.
# The overload bit — gracefully remove a router from transit paths,
# something OSPF emulates only with max-metric:
# router isis CORE
# set-overload-bit on-startup 300
# ! do not use me for transit for 5 minutes after boot, until BGP converges
Key takeaways
- IS-IS is link-state like OSPF, but descends from OSI and runs directly over layer 2, not over IP.
- Running below IP gives protocol independence, immunity to IP-based control-plane attacks, and clean extensibility.
- Everything is encoded as TLVs, so IPv6 needed only a new TLV — while OSPF required a separate protocol, OSPFv3.
- Dual-stack IS-IS uses one process, one adjacency set, one database; dual-stack OSPF needs two of each.
- Addressing uses NET/NSAP (
49.0001.1921.6800.1001.00= AFI + area + system ID + NSEL); a router belongs to exactly one area. - Hierarchy is Level 1 (intra-area) and Level 2 (backbone), with L1/L2 routers bridging; the area boundary is on the link, not on the router.
- The L2 backbone is a contiguous chain of L2 routers rather than a numbered area everything must touch.
- The DIS parallels OSPF's DR but has no backup and is preemptive.
- Wide metrics (32-bit) are mandatory on modern networks; the original 6-bit metric allowed only 63 values.
- Choice is organisational: enterprises favour OSPF, large providers favour IS-IS.
🧪 Practice
- Decompose the NET
49.0005.1921.6800.3003.00into AFI, area ID, system ID, and NSEL.- Explain why adding IPv6 to IS-IS required no new protocol version while OSPF needed OSPFv3.
- Contrast OSPF's area boundary with IS-IS's, and state how many areas a router can belong to in each.
- Interview-style: Why do large service providers tend to choose IS-IS over OSPF for backbone networks? (Hint: consider the operational cost of running two parallel protocols for dual-stack, and what it means that the control plane is unreachable by IP.)
EIGRP and DUAL
Theory
EIGRP (Enhanced Interior Gateway Routing Protocol) occupies a category of its own. It is distance-vector at heart — routers exchange distances, not topologies — but it eliminates distance-vector's fatal weakness through a genuinely different mechanism. It is therefore usually called an advanced distance-vector or hybrid protocol. Originally Cisco-proprietary, it was opened as an informational RFC (7868) in 2016.
The core is DUAL (Diffusing Update Algorithm), and its central insight is a way to identify backup paths that are provably loop-free without knowing the topology.
Recall the count-to-infinity problem (6.2): a router cannot tell whether a neighbour's advertised path passes back through itself. DUAL answers this with the feasibility condition:
A neighbour is a feasible successor if its reported distance (its own cost to the destination) is strictly less than my current feasible distance (my best known cost to that destination).
The reasoning is elegant. If a neighbour's cost to the destination is already lower than my best cost, then its path cannot run through me — because if it did, its cost would have to include the cost of reaching me plus my cost onward, which would necessarily be greater than my cost. A strictly lower reported distance is therefore proof of loop-freedom, obtained with no topology knowledge whatsoever.
This yields EIGRP's defining terminology:
| Term | Meaning |
|---|---|
| Successor | The current best next hop (installed in the routing table) |
| Feasible distance (FD) | My best-known total cost to the destination |
| Reported distance (RD) | A neighbour's own cost to the destination, as advertised |
| Feasible successor (FS) | A backup whose RD < FD — guaranteed loop-free |
| Topology table | All known paths, including all feasible successors |
The operational payoff is dramatic. When a successor fails:
- If a feasible successor exists, EIGRP switches to it immediately — typically in tens of milliseconds — with no computation and no queries, because loop-freedom was already proven.
- If none exists, the route goes Active and EIGRP sends queries to its neighbours asking whether they have a path. Convergence now waits for replies.
That second case has a failure mode worth knowing: Stuck-In-Active (SIA). If a query goes unanswered — a slow, overloaded, or unreachable neighbour — the route stays Active until a timer (default 3 minutes) expires, and the neighbour relationship may be torn down. The query scope is therefore what limits EIGRP's scale, and the defences are summarisation and stub routers, both of which stop queries from propagating.
EIGRP's composite metric uses bandwidth and delay by default, scaled by K-values. The classic formula:
metric = 256 * ( K1*bandwidth + (K2*bandwidth)/(256-load) + K3*delay ) * ( K5/(reliability+K4) )
with defaults K1=K3=1 and K2=K4=K5=0, which simplifies to 256 * (bandwidth + delay). Bandwidth here is 10^7 / minimum_bandwidth_kbps along the path, and delay is the sum of interface delays in tens of microseconds. K-values must match between neighbours or the adjacency will not form. Including load and reliability (K2, K4, K5) is strongly discouraged, because metrics then change with traffic and the network oscillates.
EIGRP also supports unequal-cost load balancing via variance (6.1) — the only common IGP that does.
Its remaining drawback is deployment reality: despite RFC 7868, multi-vendor support is thin, so EIGRP is effectively a Cisco-environment protocol. In mixed environments, OSPF or IS-IS is the safer choice.
Examples
The feasibility condition, and why it proves loop-freedom
Destination Z. Router A's best cost (FD) = 10.
Neighbour B reports RD = 6.
Is 6 < 10? YES -> B is a FEASIBLE SUCCESSOR.
WHY IT IS SAFE: if B's path ran through A, B's cost would be
(B->A cost) + (A's cost to Z) = something + 10 > 10.
It reported 6. Therefore B's path CANNOT pass through A. Proven.
Neighbour C reports RD = 12.
Is 12 < 10? NO -> C is NOT a feasible successor.
C's path MIGHT loop back through A. Cannot use it without asking.
(It might still be a perfectly good path — DUAL is conservative.)
Failover with and without a feasible successor
WITH FS:
successor fails at t=0
t=0.00 DUAL promotes the feasible successor — already proven safe
t=0.02 done. ~20 ms, NO queries, NO computation.
WITHOUT FS:
successor fails at t=0
t=0.00 route goes ACTIVE
t=0.00 QUERY sent to all neighbours: "do you have a path to Z?"
t=0.05 replies arrive -> recompute -> install
t=0.05 done. Slower, and it consumed neighbours' CPU.
If ANY neighbour does not reply:
t=180 STUCK-IN-ACTIVE timer fires -> neighbour relationship torn down
^^^^^^^^^^^^^^^^ the failure mode that limits EIGRP's scale
Containing queries — the real scaling control
WITHOUT summarisation: a query floods outward through the whole network
[A]--[B]--[C]--[D]--[E]--[F] ... every router must answer
WITH summarisation at B:
[A]--[B]| summary boundary |--[C]--[D]...
^ B answers the query itself from its summary.
The query STOPS HERE. Scale is preserved.
class EIGRPTopologyEntry:
def __init__(self, dest):
self.dest = dest
self.paths = {} # neighbor -> (reported_distance, link_cost)
self.fd = float("inf") # feasible distance: my best known total cost
self.successor = None
def add_path(self, neighbor, reported_distance, link_cost):
self.paths[neighbor] = (reported_distance, link_cost)
total = reported_distance + link_cost
if total < self.fd:
self.fd = total
self.successor = neighbor
def feasible_successors(self):
"""FEASIBILITY CONDITION: reported distance strictly less than FD."""
return {nb: (rd, rd + lc) for nb, (rd, lc) in self.paths.items()
if nb != self.successor and rd < self.fd}
def successor_fails(self):
fs = self.feasible_successors()
if fs:
best = min(fs.items(), key=lambda kv: kv[1][1])
self.successor, self.fd = best[0], best[1][1]
return f"INSTANT failover to {best[0]} (total {best[1][1]}) — no queries"
return "no feasible successor -> route goes ACTIVE, queries sent"
e = EIGRPTopologyEntry("10.9.0.0/16")
e.add_path("B", reported_distance=6, link_cost=4) # total 10 -> successor
e.add_path("C", reported_distance=12, link_cost=3) # total 15
e.add_path("D", reported_distance=8, link_cost=9) # total 17
print(f"successor: {e.successor}, feasible distance (FD) = {e.fd}")
print(f"{'neighbor':<10}{'RD':>5}{'total':>8} feasible successor?")
for nb, (rd, lc) in e.paths.items():
if nb == e.successor:
print(f"{nb:<10}{rd:>5}{rd+lc:>8} (this IS the successor)")
else:
ok = "YES — RD < FD, loop-free by proof" if rd < e.fd else \
f"no — RD {rd} >= FD {e.fd}, might loop back through me"
print(f"{nb:<10}{rd:>5}{rd+lc:>8} {ok}")
# successor: B, feasible distance (FD) = 10
# neighbor RD total feasible successor?
# B 6 10 (this IS the successor)
# C 12 15 no — RD 12 >= FD 10, might loop back through me
# D 8 17 YES — RD < FD, loop-free by proof
# NOTE: D's TOTAL cost (17) is worse than C's (15), yet D is usable
# immediately and C is not. Feasibility is about SAFETY, not cost.
print()
print(e.successor_fails())
# INSTANT failover to D (total 17) — no queries
# The case with no feasible successor.
e2 = EIGRPTopologyEntry("10.8.0.0/16")
e2.add_path("B", reported_distance=5, link_cost=5) # total 10, FD = 10
e2.add_path("C", reported_distance=11, link_cost=2) # RD 11 >= 10: not feasible
print(e2.successor_fails())
# no feasible successor -> route goes ACTIVE, queries sent
# The composite metric.
def eigrp_metric(min_bandwidth_kbps, total_delay_tens_us, K1=1, K3=1):
bw = (10 ** 7) // min_bandwidth_kbps # scaled minimum bandwidth
return 256 * (K1 * bw + K3 * total_delay_tens_us)
print()
print(f"{'link':<22}{'min bw (kbps)':>14}{'delay':>8}{'metric':>12}")
for name, bw, delay in [("GigabitEthernet", 1_000_000, 10),
("FastEthernet", 100_000, 100),
("T1 serial", 1_544, 2000),
("64k serial", 64, 2000)]:
print(f"{name:<22}{bw:>14,}{delay:>8}{eigrp_metric(bw, delay):>12,}")
# link min bw (kbps) delay metric
# GigabitEthernet 1,000,000 10 5,120
# FastEthernet 100,000 100 51,200
# T1 serial 1,544 2000 2,169,856
# 64k serial 64 2000 40,512,000
# Unlike RIP's hop count, the metric reflects real link capacity.
# Why K-values beyond the defaults are discouraged.
print()
for use_load in (False, True):
label = "K2 (load) enabled" if use_load else "defaults (K1=K3=1)"
if use_load:
note = "metric changes as traffic changes -> routes flap -> OSCILLATION"
else:
note = "metric is stable: bandwidth and delay do not vary with load"
print(f"{label:<22} {note}")
# defaults (K1=K3=1) metric is stable: bandwidth and delay do not vary with load
# K2 (load) enabled metric changes as traffic changes -> routes flap -> OSCILLATION
# The topology table — this is where feasible successors live.
$ show ip eigrp topology
# P 10.9.0.0/16, 1 successors, FD is 2560
# ^ Passive = stable. "A" here would mean ACTIVE = querying.
# via 10.0.0.2 (2560/1280), GigabitEthernet0/1
# ^^^^ ^^^^ (my total / neighbour's reported distance)
# RD 1280 < FD 2560 -> feasible
# via 10.0.0.6 (3072/1536), GigabitEthernet0/2
# RD 1536 < FD 2560 -> FEASIBLE SUCCESSOR
# via 10.0.0.9 (5120/3584), GigabitEthernet0/3
# RD 3584 >= FD 2560 -> NOT feasible
# Only successors appear in the routing table; feasible successors wait:
$ show ip route eigrp | grep 10.9
# D 10.9.0.0/16 [90/2560] via 10.0.0.2, GigabitEthernet0/1
# ^ AD 90 ^ the successor only
# A route that is stuck active — the failure mode to recognise:
$ show ip eigrp topology active
# A 10.8.0.0/16, 0 successors, FD is Inf, Q
# ^ ACTIVE, querying 1 replies, active 00:02:41, query-origin: Local origin
# Remaining replies:
# via 10.0.0.9, r, GigabitEthernet0/3
# ^ 'r' = reply still outstanding. If this persists,
# SIA fires at 3 minutes and the adjacency drops.
# The two defences that bound query scope:
# router eigrp 1
# eigrp stub connected summary <- stub routers are never queried
# interface GigabitEthernet0/1
# ip summary-address eigrp 1 10.1.0.0 255.255.0.0
# ^^^^^^^^ a query for 10.1.x.x STOPS here
# K-values must match, or no adjacency forms:
$ show ip protocols | grep -i "K1"
# EIGRP metric weight K1=1, K2=0, K3=1, K4=0, K5=0
# ^^^^^^^^^^^^^^^^^^^^^^^^^^^ identical on both ends,
# or the neighbour never comes up
# Unequal-cost load balancing — unique among common IGPs:
# router eigrp 1
# variance 2 <- also use paths up to 2x the best metric,
# proportionally to their cost
$ show ip route 10.9.0.0
# Routing Descriptor Blocks:
# * 10.0.0.2, from 10.0.0.2 traffic share count 3
# 10.0.0.6, from 10.0.0.6 traffic share count 1
# ^^^^^^^^^^^^^^^^^^^ 3:1 split by metric
Key takeaways
- EIGRP is advanced distance-vector: it exchanges distances, not topology, but avoids count-to-infinity through DUAL.
- The feasibility condition — a neighbour's reported distance < my feasible distance — proves its path cannot loop back through me, with no topology knowledge.
- Terms: successor (installed best path), feasible distance (FD), reported distance (RD), feasible successor (FS), topology table.
- With a feasible successor, failover is immediate (tens of milliseconds) — no queries, no computation.
- Without one, the route goes Active and EIGRP queries neighbours; an unanswered query leads to Stuck-In-Active and a torn-down adjacency.
- Query scope limits EIGRP's scale; summarisation and stub routers are the defences that stop query propagation.
- A feasible successor may have a worse total cost than a non-feasible path — feasibility is about provable safety, not optimality.
- The composite metric uses bandwidth and delay by default (
K1=K3=1); K-values must match between neighbours, and enabling load/reliability causes oscillation. - EIGRP is the only common IGP offering unequal-cost load balancing (
variance). - Despite RFC 7868, multi-vendor support is limited, so it is effectively a Cisco-environment protocol.
🧪 Practice
- A router's FD to a prefix is 25. Neighbours report RD values of 12, 25, and 30. Identify the feasible successors and explain each verdict.
- Explain, in terms of costs, why
RD < FDguarantees a neighbour's path does not pass back through the local router.- Describe what happens when a successor fails with no feasible successor available, and name the failure mode if a neighbour never replies.
- Interview-style: EIGRP is distance-vector, yet it converges faster than OSPF in many topologies. How? (Hint: OSPF must recompute after a failure — ask what EIGRP has already done and verified before the failure occurs.)
Route Redistribution and Summarization
Theory
Real networks rarely run a single routing protocol. Mergers, acquisitions, legacy equipment, partner connections, and migrations all produce boundaries where one protocol must hand routes to another. Redistribution is that handoff — and it is the single most common source of serious routing incidents, because it removes the protections each protocol builds for itself.
The fundamental problem: metrics are not comparable across protocols. RIP has hop counts (1–15), OSPF has bandwidth-derived costs, EIGRP has a composite in the millions, and BGP has path attributes rather than a metric at all. When redistributing, the router must invent a metric for the receiving protocol, because no meaningful translation exists. That invented value is called the seed metric, and forgetting to set it is a classic failure — some protocols default to infinity, so the routes are advertised as unreachable and silently discarded.
| Into | Default seed metric | Result if unset |
|---|---|---|
| RIP | Infinity | Routes are not advertised at all |
| OSPF | 20 (E2) | Works, but every route has the same cost |
| EIGRP | Infinity | Routes are not advertised at all |
| BGP | Uses IGP metric as MED | Usually acceptable |
The deeper danger is routing loops from mutual redistribution. With two or more boundary routers between the same pair of protocols, a route can leave protocol A, enter protocol B, travel around, and re-enter protocol A — now appearing to originate from the wrong direction. Since the receiving protocol's own loop prevention (split horizon, SPF, AS-PATH) operates only within that protocol, it cannot detect a loop that passed through a foreign protocol. Administrative distance interacts badly here: a re-injected route may have a better AD than the original, so the router prefers the looped path.
The defences, in order of reliability:
Route tagging is the primary control. Mark routes as they are redistributed, and filter on that tag at every other boundary. A route tagged "came from OSPF" is never allowed back into OSPF. This works regardless of topology and is the mechanism to reach for first.
Filtering with prefix lists or distribute lists explicitly controls which routes cross.
One-way redistribution wherever possible: redistribute into the core and use a default route outward. If routes only flow one way, no loop can form.
AD manipulation can help but is fragile and should not be the primary defence.
Summarisation is the complementary technique and is mostly beneficial. Advertising 10.1.0.0/16 instead of 256 individual /24s:
- Shrinks routing tables — less memory, faster lookups.
- Hides instability, which is the most valuable property: a flapping /24 inside the summary causes no advertisement outside it, so remote routers never recompute.
- Bounds query scope in EIGRP and LSA flooding in OSPF.
Its costs are real but usually acceptable: suboptimal routing is possible, since remote routers cannot distinguish members of the summary and may send traffic to the wrong exit when the block is split across locations. And a summary is advertised as long as at least one component exists, so traffic for missing components is attracted and dropped — the standard mitigation is a discard route (a route to Null0 for the summary) so those packets are dropped locally rather than looping.
The practical rule: summarise wherever addressing is hierarchical, and redistribute as little as possible. Where you must redistribute, tag and filter at every boundary without exception.
Examples
Mutual redistribution — how the loop forms
OSPF domain R1, R2 both redistribute RIP domain
+---------------+ +---------------+
| 10.1.0.0/16 |------[R1]--------------------- | |
| (origin) | | |
| |------[R2]--------------------- | |
+---------------+ +---------------+
1. 10.1.0.0/16 originates in OSPF.
2. R1 redistributes it into RIP. RIP: "10.1.0.0/16, 1 hop"
3. It propagates through RIP to R2.
4. R2 redistributes it BACK into OSPF as an EXTERNAL route.
5. Routers near R2 now see 10.1.0.0/16 as external via R2,
possibly preferring it over the real internal route.
6. Traffic goes to R2 -> into RIP -> to R1 -> back into OSPF -> LOOP.
OSPF's SPF cannot detect this: the route left OSPF's world entirely
and came back wearing a disguise.
THE FIX — tag at every boundary:
R1: redistribute ospf into rip, set tag 100
R2: deny any route with tag 100 from entering ospf
(and symmetrically for the other direction)
Summarisation hides instability — the property that matters most
WITHOUT summary: 10.1.5.0/24 flaps
-> LSA/update floods the entire network
-> every router recomputes, repeatedly
WITH summary at the ABR (10.1.0.0/16):
10.1.5.0/24 flaps inside the area
-> the SUMMARY does not change (other /24s still exist)
-> NOTHING is advertised outward
-> zero remote recomputation
The summary is a firewall against churn.
The summary black hole, and the discard route
ABR advertises 10.1.0.0/16 (covering .0 to .255)
But only 10.1.1.0/24 and 10.1.2.0/24 actually exist.
Packet for 10.1.9.9 -> attracted by the summary -> arrives at the ABR
-> ABR has no specific route -> may follow its OWN
default route back out -> LOOP
FIX: ip route 10.1.0.0 255.255.0.0 Null0
A discard route: the packet is dropped locally, at the ABR.
SEED_METRIC_DEFAULTS = {
"rip": (None, "INFINITY — routes silently NOT advertised"),
"ospf": (20, "E2 type, cost 20 — all routes get the SAME cost"),
"eigrp": (None, "INFINITY — routes silently NOT advertised"),
"bgp": ("igp","IGP metric used as MED — usually fine"),
}
print(f"{'redistribute into':<20}{'default seed':<10}consequence")
for proto, (seed, note) in SEED_METRIC_DEFAULTS.items():
print(f"{proto:<20}{str(seed):<10}{note}")
# redistribute into default seed consequence
# rip None INFINITY — routes silently NOT advertised
# ospf 20 E2 type, cost 20 — all routes get the SAME cost
# eigrp None INFINITY — routes silently NOT advertised
# bgp igp IGP metric used as MED — usually fine
# Redistribution with tag-based loop prevention.
class Boundary:
def __init__(self, name, from_proto, into_proto, tag, deny_tags=()):
self.name, self.from_proto, self.into_proto = name, from_proto, into_proto
self.tag, self.deny_tags = tag, set(deny_tags)
def redistribute(self, route):
if route.get("tag") in self.deny_tags:
return None, (f"{self.name}: BLOCKED — tag {route['tag']} came from "
f"{self.into_proto}; refusing to send it back")
if route["proto"] != self.from_proto:
return None, f"{self.name}: not a {self.from_proto} route, ignored"
out = dict(route, proto=self.into_proto, tag=self.tag, external=True)
return out, f"{self.name}: redistributed with tag {self.tag}"
# Two boundary routers, each tagging outbound and denying the other's tag.
R1 = Boundary("R1", "ospf", "rip", tag=100, deny_tags={200})
R2 = Boundary("R2", "rip", "ospf", tag=200, deny_tags={100})
route = {"prefix": "10.1.0.0/16", "proto": "ospf", "tag": None}
print(f"\norigin: {route['prefix']} in OSPF")
in_rip, msg = R1.redistribute(route)
print(f" {msg}")
back_to_ospf, msg = R2.redistribute(in_rip)
print(f" {msg}")
print(f" result: {'LOOP PREVENTED' if back_to_ospf is None else 'LOOP CREATED'}")
# origin: 10.1.0.0/16 in OSPF
# R1: redistributed with tag 100
# R2: BLOCKED — tag 100 came from ospf; refusing to send it back
# result: LOOP PREVENTED
# Without tags, the same sequence creates the loop.
R1_untagged = Boundary("R1", "ospf", "rip", tag=None)
R2_untagged = Boundary("R2", "rip", "ospf", tag=None)
in_rip, _ = R1_untagged.redistribute(route)
back, _ = R2_untagged.redistribute(in_rip)
print(f"\nwithout tags: route returns to OSPF as external = "
f"{back is not None} -> LOOP RISK")
# without tags: route returns to OSPF as external = True -> LOOP RISK
# Summarisation: table reduction and, more importantly, churn suppression.
import ipaddress
def summarise(prefixes):
return list(ipaddress.collapse_addresses(
[ipaddress.ip_network(p) for p in prefixes]))
subnets = [f"10.1.{i}.0/24" for i in range(256)]
summary = summarise(subnets)
print(f"\n{len(subnets)} prefixes -> {len(summary)}: {[str(s) for s in summary]}")
print(f"table reduction: {100*(1-len(summary)/len(subnets)):.1f}%")
# 256 prefixes -> 1: ['10.1.0.0/16']
# table reduction: 99.6%
def churn(flaps_per_hour, summarised):
"""Advertisements leaving the area per hour."""
return 0 if summarised else flaps_per_hour
print()
for summarised in (False, True):
adverts = churn(120, summarised)
label = "summarised" if summarised else "not summarised"
print(f"{label:<16} one /24 flapping 120x/hour -> {adverts:>3} external "
f"advertisements/hour")
# not summarised one /24 flapping 120x/hour -> 120 external advertisements/hour
# summarised one /24 flapping 120x/hour -> 0 external advertisements/hour
# THIS is summarisation's most valuable property, not the memory saving.
# The black hole a summary can create.
def summary_blackhole(summary, existing, dst):
s = ipaddress.ip_network(summary)
d = ipaddress.ip_address(dst)
if d not in s:
return "not covered by the summary — routed normally elsewhere"
if any(d in ipaddress.ip_network(e) for e in existing):
return "delivered to a real subnet"
return ("ATTRACTED by the summary but no specific route exists -> "
"needs a Null0 discard route, or it may loop")
print()
for dst in ["10.1.1.5", "10.1.9.9", "10.2.0.1"]:
print(f"{dst:<12} {summary_blackhole('10.1.0.0/16', ['10.1.1.0/24','10.1.2.0/24'], dst)}")
# 10.1.1.5 delivered to a real subnet
# 10.1.9.9 ATTRACTED by the summary but no specific route exists -> needs a Null0 discard route, or it may loop
# 10.2.0.1 not covered by the summary — routed normally elsewhere
# Redistribution done properly: seed metric, tag, and filter — all three.
# route-map OSPF_TO_RIP permit 10
# match ip address prefix-list ALLOWED
# set metric 5 <- SEED METRIC: without it, infinity
# set tag 100 <- mark the origin
# route-map RIP_TO_OSPF deny 10
# match tag 100 <- never let our own routes back in
# route-map RIP_TO_OSPF permit 20
# set metric 50
# set metric-type type-1
# set tag 200
#
# router rip
# redistribute ospf 1 route-map OSPF_TO_RIP
# router ospf 1
# redistribute rip route-map RIP_TO_OSPF subnets
# ^^^^^^^ without 'subnets', OSPF
# redistributes ONLY classful routes
# Finding redistributed routes and their tags:
$ vtysh -c "show ip route ospf" | head -3
# O E2 10.5.0.0/16 [110/20] via 10.0.0.2, eth0
# ^^^^ E2 = external type 2: cost does NOT increase with distance
# E1 would add internal cost to the external cost
$ show ip ospf database external | grep -A4 10.5.0.0
# Link State ID: 10.5.0.0 (External Network Number)
# Metric Type: 2 (Larger than any link state path)
# Metric: 20
# External Route Tag: 100 <- the tag, visible and filterable
# Summarisation, per protocol:
# OSPF at an ABR: area 1 range 10.1.0.0 255.255.0.0
# OSPF at an ASBR: summary-address 10.5.0.0 255.255.0.0
# EIGRP per interface: ip summary-address eigrp 1 10.1.0.0 255.255.0.0
# BGP: aggregate-address 10.1.0.0 255.255.0.0 summary-only
# The discard route is created automatically by some platforms — verify it:
$ show ip route 10.1.0.0 255.255.0.0
# O 10.1.0.0/16 is a summary, 00:12:41, Null0
# ^^^^^ discard route: packets for
# non-existent components die HERE
# instead of following a default route
# Detecting a redistribution loop — look for routes that should be internal
# appearing as external:
$ vtysh -c "show ip route 10.1.0.0/16"
# Routing entry for 10.1.0.0/16
# Known via "ospf", distance 110, metric 20, type extern 2
# ^^^^^^^^^^^^ EXTERNAL, but this
# prefix ORIGINATES inside our OSPF domain. It went out and came back.
# -> mutual redistribution without tag filtering. Fix the boundaries.
Key takeaways
- Redistribution moves routes between protocols and is the most common source of serious routing incidents.
- Metrics are not comparable across protocols, so a seed metric must be invented; RIP and EIGRP default to infinity, silently dropping the routes.
- Mutual redistribution at two or more boundaries can loop: a route leaves protocol A, traverses B, and re-enters A disguised as external, where A's own loop prevention cannot see it.
- Route tagging is the primary defence — mark on the way out, filter that tag on the way back in, at every boundary.
- Supporting defences: prefix filtering, one-way redistribution with a default route, and (fragile) AD manipulation.
- OSPF requires the
subnetskeyword, or only classful routes are redistributed. - Summarisation shrinks tables, but its most valuable property is hiding instability — a flapping component inside a summary causes zero external advertisements.
- Summarisation costs: possible suboptimal routing, and a black hole for non-existent components — mitigated with a Null0 discard route.
- Practical rule: summarise wherever addressing is hierarchical; redistribute as little as possible, and always with tags and filters.
🧪 Practice
- Explain why redistributing into RIP or EIGRP without a seed metric results in no routes being advertised.
- Trace how a route loops through mutual redistribution between OSPF and RIP at two boundary routers, and show where tags break the cycle.
- Summarise
172.20.0.0/24through172.20.15.0/24into one prefix and state the number of external advertisements caused by one component flapping.- Interview-style: After a merger you must connect an OSPF network to an EIGRP network at two points. What is your primary loop-prevention mechanism and why not administrative distance? (Hint: AD is a local preference value — ask whether it travels with the route to the other boundary router.)
<a id="64-exterior-gateway-routing"></a>
6.4 Exterior Gateway Routing
This subchapter covers BGP, the protocol that joins tens of thousands of independently operated networks into one internet — where the objective is not the shortest path but the path that policy permits.
Autonomous Systems and AS Numbers
Theory
The IGPs covered so far all assume a single administration: one organisation, one set of goals, and routers that trust each other completely. That assumption fails at the scale of the internet, where competing companies must exchange traffic without trusting each other and without revealing their internal topology.
An Autonomous System (AS) is the unit of that division: a network, or set of networks, under one administrative authority with a single, coherent routing policy. "Autonomous" is the operative word — an AS decides for itself which routes to accept, which to advertise, and to whom, and no external party can override those decisions.
This produces a two-level routing architecture that mirrors the trust structure:
| Interior (IGP) | Exterior (EGP) | |
|---|---|---|
| Scope | Within one AS | Between autonomous systems |
| Goal | Find the shortest path | Enforce policy |
| Trust | Full | None |
| Topology visible | Yes | Deliberately hidden |
| Protocols | OSPF, IS-IS, EIGRP, RIP | BGP |
| Scale | Thousands of routers | ~75,000 ASes, ~1M prefixes |
The shift from "shortest" to "policy" is the conceptual heart of this subchapter. Inside an AS, a shorter path is simply better. Between ASes, the best path is frequently not the shortest — because a longer path may be cheaper, contractually required, or the only one permitted. BGP is built to express those constraints, which is why it carries attributes rather than a metric.
AS numbers (ASNs) identify each AS. Originally 16-bit (1–65535), the space was exhausted, and 32-bit ASNs (RFC 6793) extended it to about 4.3 billion. Private ranges exist for internal use, analogous to RFC 1918:
| Range | Type |
|---|---|
| 1 – 64495 | Public 16-bit |
| 64512 – 65534 | Private 16-bit |
| 65536 – 4199999999 | Public 32-bit |
| 4200000000 – 4294967294 | Private 32-bit |
| 23456 | AS_TRANS, for 16-bit routers meeting 32-bit ASNs |
ASNs are allocated by RIRs (ARIN, RIPE NCC, APNIC, LACNIC, AFRINIC) and are a matter of public record — anyone can look up who owns one, which is fundamental to routing security.
The relationships between ASes determine how traffic and money flow, and they explain nearly all internet routing policy:
Transit (customer–provider). The customer pays the provider for reachability to the entire internet. The provider advertises the customer's routes to everyone.
Peering (settlement-free). Two ASes exchange traffic between their own customers only, usually without payment, because it is mutually beneficial. Crucially, a peer does not provide transit — you cannot reach the rest of the internet through a peer.
Sibling. ASes under common ownership, exchanging everything.
From these comes the valley-free (Gao-Rexford) rule that structures the entire internet's routing:
Never provide free transit. Routes learned from a peer or provider are advertised only to customers — never to other peers or providers.
The economic logic is exact: advertising a peer's routes to another peer would carry their traffic for free, paying transit costs for someone else's benefit. Every operator applies this rule, which is why internet paths have a characteristic shape — up to a provider, across a peering link, then down to a customer, never up again after coming down.
Examples
The two-level architecture, and where trust ends
+--- AS 64500 ---+ +--- AS 64501 ---+
| OSPF inside | | IS-IS inside |
| R1--R2--R3 | | R7--R8--R9 |
| [R4]--+--BGP---+--[R6] |
+----------------+ +----------------+
^^^^^
policy boundary: R4 shows R6 only what it CHOOSES to.
R6 never learns AS 64500's internal topology, link costs,
or router count. That opacity is the entire point.
AS relationships and the valley-free rule
[Tier 1 A] ===peer=== [Tier 1 B]
| |
provider provider
| |
[ISP C] ======peer====== [ISP D]
| |
provider provider
| |
[Customer E] [Customer F]
ALLOWED path: E -> C -> D -> F
(up to provider, across peer, down to customer)
FORBIDDEN: A -> B via C
C would carry Tier 1 traffic for free between two
networks it PAYS. C advertises peer routes to its
CUSTOMERS only, never to its other peers/providers.
The shape: a path goes UP, across at most one peering link, then DOWN.
Never up again after coming down. Hence "valley-free".
Why "shortest" stops being the goal
AS 64500 can reach AS 64999 two ways:
via provider X: AS-PATH length 2 ($$$ per megabit)
via peer Y: AS-PATH length 4 (free)
BGP will be configured to prefer the LONGER path, because it is free.
No IGP would ever do this. Policy, not distance, decides.
ASN_RANGES = [
(1, 64495, "public 16-bit"),
(64496, 64511, "documentation (RFC 5398)"),
(64512, 65534, "PRIVATE 16-bit"),
(65535, 65535, "reserved"),
(65536, 4199999999, "public 32-bit"),
(4200000000, 4294967294, "PRIVATE 32-bit"),
]
def classify_asn(asn):
if asn == 23456:
return "AS_TRANS (32-bit ASN seen by a 16-bit-only router)"
for lo, hi, name in ASN_RANGES:
if lo <= asn <= hi:
return name
return "reserved/invalid"
for asn in (1, 15169, 23456, 64512, 65534, 131072, 4200000001):
print(f"AS{asn:<12} {classify_asn(asn)}")
# AS1 public 16-bit
# AS15169 public 16-bit
# AS23456 AS_TRANS (32-bit ASN seen by a 16-bit-only router)
# AS64512 PRIVATE 16-bit
# AS65534 PRIVATE 16-bit
# AS131072 public 32-bit
# AS4200000001 PRIVATE 32-bit
print(f"\n16-bit ASN space: {65535:,}")
print(f"32-bit ASN space: {4294967295:,} ({4294967295/65535:,.0f}x larger)")
# 16-bit ASN space: 65,535
# 32-bit ASN space: 4,294,967,295 (65,537x larger)
# The valley-free rule, implemented. This single function explains most
# internet routing policy.
def should_advertise(route_learned_from, advertise_to):
"""Gao-Rexford: NEVER provide free transit."""
if route_learned_from == "customer":
return True, "customer routes go to EVERYONE (they pay us for reachability)"
if advertise_to == "customer":
return True, "customers may reach everything (that is what they pay for)"
return False, (f"BLOCKED: {route_learned_from} route -> {advertise_to} "
f"would be FREE TRANSIT")
roles = ["customer", "peer", "provider"]
print(f"\n{'learned from':<12}{'advertise to':<12}{'allowed':<9}reason")
for src in roles:
for dst in roles:
ok, why = should_advertise(src, dst)
print(f"{src:<12}{dst:<12}{str(ok):<9}{why}")
# learned from advertise to allowed reason
# customer customer True customer routes go to EVERYONE (they pay us for reachability)
# customer peer True customer routes go to EVERYONE (they pay us for reachability)
# customer provider True customer routes go to EVERYONE (they pay us for reachability)
# peer customer True customers may reach everything (that is what they pay for)
# peer peer False BLOCKED: peer route -> peer would be FREE TRANSIT
# peer provider False BLOCKED: peer route -> provider would be FREE TRANSIT
# provider customer True customers may reach everything (that is what they pay for)
# provider peer False BLOCKED: provider route -> peer would be FREE TRANSIT
# provider provider False BLOCKED: provider route -> provider would be FREE TRANSIT
# Validating that a path is valley-free.
def valley_free(path_relationships):
"""A valid path: zero or more 'up', at most one 'peer', then zero or more 'down'."""
phase = "up"
for rel in path_relationships:
if phase == "up":
if rel == "up": continue
if rel == "peer": phase = "down"; continue
if rel == "down": phase = "down"; continue
elif phase == "down":
if rel == "down": continue
return False, f"INVALID: '{rel}' after descending — this is a valley"
return True, "valid valley-free path"
print()
for path in [["up", "peer", "down"], ["up", "up", "down", "down"],
["up", "down", "up"], ["peer", "peer"]]:
ok, why = valley_free(path)
print(f"{str(path):<38}{why}")
# ['up', 'peer', 'down'] valid valley-free path
# ['up', 'up', 'down', 'down'] valid valley-free path
# ['up', 'down', 'up'] INVALID: 'up' after descending — this is a valley
# ['peer', 'peer'] INVALID: 'peer' after descending — this is a valley
# Why an AS might deliberately prefer the LONGER path.
def choose_path(options):
# Real policy: cost first, AS-PATH length only as a tie-break.
return min(options, key=lambda o: (o["cost_per_mbit"], o["as_path_len"]))
options = [
{"via": "provider X", "as_path_len": 2, "cost_per_mbit": 1.50},
{"via": "peer Y", "as_path_len": 4, "cost_per_mbit": 0.00},
]
best = choose_path(options)
print(f"\nchosen: {best['via']} (AS-PATH {best['as_path_len']}, "
f"${best['cost_per_mbit']}/Mbit)")
print("the LONGER path won. No IGP would ever make this choice.")
# chosen: peer Y (AS-PATH 4, $0.0/Mbit)
# the LONGER path won. No IGP would ever make this choice.
# Who owns an AS number? Public record, and the basis of routing security.
$ whois -h whois.radb.net AS15169 | head -4
# aut-num: AS15169
# as-name: GOOGLE
# descr: Google LLC
# admin-c: GOGL-ARIN
# Which AS does an address belong to?
$ whois -h whois.cymru.com " -v 8.8.8.8"
# AS | IP | BGP Prefix | CC | Registry | Allocated | AS Name
# 15169 | 8.8.8.8 | 8.8.8.0/24 | US | arin | 1992-12-01 | GOOGLE, US
# See the AS-level path, not just the router-level path:
$ traceroute -A -n 8.8.8.8 | head -5
# 1 192.168.1.1 [*] 1.1 ms
# 2 10.20.0.1 [AS64500] 9.8 ms <- your ISP
# 3 62.115.120.1 [AS1299] 11.4 ms <- a Tier 1 transit provider
# 4 142.250.166.1 [AS15169] 12.1 ms <- Google's own network
# ^^^^^^^^^^ each [ASnnnn] is a separate administrative domain
# Our own AS and its configured neighbours:
$ vtysh -c "show ip bgp summary"
# BGP router identifier 10.0.0.1, local AS number 64500 vrf-id 0
# ^^^^^^^^^^^^^^^^^^^^^
# Neighbor V AS MsgRcvd MsgSent Up/Down State/PfxRcd
# 203.0.113.1 4 1299 184211 9112 3w2d 974213
# ^^^^ eBGP: a DIFFERENT AS -> a policy boundary
# 10.0.0.2 4 64500 14021 13998 8w1d 1204
# ^^^^^ iBGP: the SAME AS -> inside our administration
# Enforcing valley-free policy with communities — how it is really done:
# route-map FROM_PEER permit 10
# set community 64500:200 <- tag: "learned from a peer"
# route-map TO_PEER deny 10
# match community PEER_OR_PROVIDER <- never re-advertise to another peer
# route-map TO_PEER permit 20
# match community CUSTOMER_ROUTES <- customer routes only
$ show ip bgp community 64500:200 | wc -l
# 184291 # routes learned from peers; none of these go to other peers
Key takeaways
- An Autonomous System is a network under one administrative authority with a single routing policy; it decides independently what to accept and advertise.
- Routing splits two ways: IGPs optimise for the shortest path inside an AS; BGP enforces policy between ASes.
- BGP deliberately hides internal topology — a neighbouring AS learns only what you choose to tell it.
- ASNs are 16-bit (1–65535, private 64512–65534) or 32-bit (private 4200000000–4294967294); AS23456 is
AS_TRANSfor backward compatibility. - ASNs are allocated by RIRs and are public record, which underpins routing security.
- Relationships: transit (customer pays provider), peering (settlement-free, customers' traffic only), and sibling.
- A peer does not provide transit — you cannot reach the whole internet through a peer.
- The valley-free / Gao-Rexford rule: routes from a peer or provider are advertised only to customers, because anything else is free transit.
- Consequently the best BGP path is often not the shortest — a longer free path beats a shorter paid one.
🧪 Practice
- Classify AS64512, AS23456, AS15169, and AS4200000005, naming the range each falls in.
- Build the advertisement matrix for routes learned from customer, peer, and provider, and state which four combinations are forbidden and why.
- Explain why an AS would prefer a 5-hop AS-PATH over a 2-hop one, and why no IGP would do the same.
- Interview-style: Your network peers with another ISP. They complain they cannot reach the rest of the internet through you. Is this a fault? (Hint: consider what distinguishes a peering relationship from a transit one, and who would be paying for that traffic.)
BGP Session Establishment and Messages
Theory
BGP's operation differs from every IGP covered so far, and the differences all follow from one design decision: BGP runs over TCP, on port 179.
No IGP does this. OSPF rides directly in IP (protocol 89), IS-IS rides in layer 2, RIP uses UDP. BGP chose TCP because it needs guaranteed, ordered, reliable delivery of very large amounts of data — a full internet routing table is roughly a million prefixes and hundreds of megabytes of updates. Re-implementing reliability inside the routing protocol would duplicate TCP badly.
The consequences are pervasive:
- Peers must be explicitly configured. There is no discovery mechanism, no Hello multicast — you must know your neighbour's IP address and AS number. This is appropriate: you do not want to automatically peer with whoever appears on a link between competing organisations.
- Peers need IP reachability first, normally provided by an IGP or a static route.
- BGP inherits TCP's properties, including its vulnerability to session resets — hence TCP-AO/MD5 authentication and GTSM (Chapter 5).
- Updates are incremental. After the initial full table exchange, BGP sends only changes, and keepalives maintain the session.
The session progresses through a defined state machine:
| State | Meaning |
|---|---|
| Idle | Nothing happening; refusing connections |
| Connect | Attempting the TCP handshake |
| Active | TCP failed; retrying. A confusingly named problem state |
| OpenSent | OPEN sent, awaiting the peer's OPEN |
| OpenConfirm | OPENs exchanged, awaiting KEEPALIVE |
| Established | Operational; UPDATEs may flow |
The trap worth memorising: "Active" does not mean working. It means the TCP connection is failing and BGP is retrying. A session flapping between Idle, Connect, and Active indicates the peer is unreachable, a firewall is blocking port 179, or the configured address is wrong. Only Established means the session is up.
BGP defines four message types:
OPEN — sent once, immediately after the TCP handshake. It carries the local AS number, the BGP identifier (router ID), the proposed hold time, and capabilities (32-bit ASN support, address families, route refresh, graceful restart). If the AS numbers do not match what each side expects, or the hold times are incompatible, the session fails here.
UPDATE — the workhorse. It carries withdrawn routes, path attributes, and NLRI (Network Layer Reachability Information) — the prefixes being advertised. One UPDATE can announce many prefixes sharing the same attributes, which is a significant efficiency: a provider advertising 10,000 customer prefixes with identical attributes sends far fewer messages than one per prefix.
KEEPALIVE — a 19-byte header-only message, sent every one-third of the hold time (default: 60 s keepalive, 180 s hold). If no message arrives within the hold time, the session drops and all routes learned from that peer are withdrawn — which at internet scale means a million routes disappearing and reconverging.
NOTIFICATION — sent when an error occurs, and it always closes the session. Its error code and subcode tell you exactly what went wrong, making it the single most useful diagnostic in BGP.
Because a session reset is so expensive, two mechanisms exist to avoid it. Route Refresh (RFC 2918) lets a router ask its peer to resend its routes after a policy change, instead of bouncing the session. Graceful Restart lets forwarding continue on stale routes while the BGP process restarts.
Examples
The BGP state machine — note where "Active" sits
Idle
| start
v
Connect ---TCP fails---> Active ---retry---+
| | |
| TCP established +---------------+
v ^ FLAPPING HERE = peer unreachable,
OpenSent firewall blocking 179, or wrong address.
| OPEN received "Active" is a PROBLEM state, not a good one.
v
OpenConfirm
| KEEPALIVE received
v
Established <-- the ONLY state in which routing information flows
Message flow over the life of a session
R1 R2
|------ TCP SYN (port 179) ------------------>|
|<----- TCP SYN-ACK --------------------------|
|------ TCP ACK ----------------------------->| TCP up
|
|------ OPEN (AS 64500, hold 180, caps) ----->|
|<----- OPEN (AS 64501, hold 180, caps) ------|
|------ KEEPALIVE -------------------------->|
|<----- KEEPALIVE ---------------------------| ESTABLISHED
|
|<===== UPDATE x N (the full table) ==========| initial bulk transfer
| | ~1M prefixes
|------ KEEPALIVE (every 60 s) -------------->|
|<----- UPDATE (only CHANGES from now on) ----|
|
|<----- NOTIFICATION (on any error) ----------| session CLOSES
One UPDATE can carry many prefixes — if attributes match
UPDATE
withdrawn routes: 10.9.0.0/16
path attributes: ORIGIN=IGP, AS_PATH=64501 64502, NEXT_HOP=203.0.113.1
NLRI: 10.1.0.0/16, 10.2.0.0/16, 10.3.0.0/16, ...
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ALL share the attributes above -> one message.
Different attributes would require separate UPDATEs.
BGP_STATES = [
("Idle", "refusing connections; start event needed"),
("Connect", "attempting the TCP handshake"),
("Active", "TCP FAILED, retrying — a PROBLEM state despite the name"),
("OpenSent", "OPEN sent, awaiting the peer's OPEN"),
("OpenConfirm", "OPENs exchanged, awaiting KEEPALIVE"),
("Established", "OPERATIONAL — the only state where UPDATEs flow"),
]
for state, meaning in BGP_STATES:
flag = " <<< the only healthy steady state" if state == "Established" else ""
flag = " <<< misleading name" if state == "Active" else flag
print(f"{state:<14}{meaning}{flag}")
BGP_MESSAGES = {
"OPEN": (19+10, "AS number, BGP ID, hold time, capabilities"),
"UPDATE": ("variable", "withdrawn routes + path attributes + NLRI"),
"KEEPALIVE": (19, "header only; sent at hold_time/3"),
"NOTIFICATION": (21, "error code + subcode; ALWAYS closes the session"),
}
print()
for msg, (size, purpose) in BGP_MESSAGES.items():
print(f"{msg:<14}{str(size):<10}{purpose}")
# OPEN 29 AS number, BGP ID, hold time, capabilities
# UPDATE variable withdrawn routes + path attributes + NLRI
# KEEPALIVE 19 header only; sent at hold_time/3
# NOTIFICATION 21 error code + subcode; ALWAYS closes the session
# Timer relationships, and the cost of a session drop.
def bgp_timers(hold_time=180):
return {"hold_time_s": hold_time,
"keepalive_s": hold_time // 3,
"max_silence_before_drop_s": hold_time,
"missed_keepalives_tolerated": 2}
print()
for hold in (180, 90, 9):
t = bgp_timers(hold)
print(f"hold={t['hold_time_s']:>4}s -> keepalive every {t['keepalive_s']:>3}s, "
f"tolerates {t['missed_keepalives_tolerated']} missed keepalives")
# hold= 180s -> keepalive every 60s, tolerates 2 missed keepalives
# hold= 90s -> keepalive every 30s, tolerates 2 missed keepalives
# hold= 9s -> keepalive every 3s, tolerates 2 missed keepalives
# Aggressive timers detect failure faster but risk dropping a healthy
# session during a brief CPU spike — and a drop is VERY expensive.
def session_drop_cost(prefixes, install_rate_per_s=120_000):
return {"routes_withdrawn": prefixes,
"reconvergence_s": round(prefixes / install_rate_per_s, 1),
"note": "every prefix must be withdrawn, then relearned and reinstalled"}
print()
for p in (1_000, 100_000, 1_000_000):
c = session_drop_cost(p)
print(f"{p:>9,} prefixes -> {c['reconvergence_s']:>5.1f} s to reconverge after a reset")
# 1,000 prefixes -> 0.0 s to reconverge after a reset
# 100,000 prefixes -> 0.8 s to reconverge after a reset
# 1,000,000 prefixes -> 8.3 s to reconverge after a reset
# Why one UPDATE per attribute-set matters at scale.
def updates_needed(prefixes, distinct_attribute_sets, max_nlri_per_update=400):
per_set = -(-prefixes // distinct_attribute_sets)
msgs_per_set = -(-per_set // max_nlri_per_update)
return distinct_attribute_sets * msgs_per_set
print()
for sets in (1, 10, 1000):
msgs = updates_needed(100_000, sets)
print(f"100,000 prefixes over {sets:>4} attribute set(s) -> {msgs:>6,} UPDATE messages")
# 100,000 prefixes over 1 attribute set(s) -> 250 UPDATE messages
# 100,000 prefixes over 10 attribute set(s) -> 250 UPDATE messages
# 100,000 prefixes over 1000 attribute set(s) -> 1,000 UPDATE messages
# Prefixes sharing attributes pack efficiently; highly varied policy
# (many distinct attribute sets) costs more messages.
NOTIFICATION_CODES = {
(1, 2): "OPEN: unsupported version",
(2, 2): "OPEN: bad peer AS — the configured remote-as does not match",
(2, 6): "OPEN: unacceptable hold time",
(3, 1): "UPDATE: malformed attribute list",
(4, 0): "Hold Timer Expired — no message arrived within the hold time",
(6, 2): "Cease: administrative shutdown",
(6, 1): "Cease: MAXIMUM PREFIX LIMIT EXCEEDED",
}
print()
for (code, sub), meaning in NOTIFICATION_CODES.items():
print(f"code {code} subcode {sub}: {meaning}")
# Session state — read the State column carefully.
$ vtysh -c "show ip bgp summary"
# Neighbor V AS MsgRcvd MsgSent Up/Down State/PfxRcd
# 203.0.113.1 4 1299 184211 9112 3w2d 974213
# ^^^^^^ a NUMBER here
# means Established
# 198.51.100.7 4 64510 0 92 never Active
# ^^^^^^ NOT working:
# TCP is failing
# Diagnosing a stuck "Active" session, in order:
$ ping -c1 198.51.100.7 # 1. is the peer reachable?
$ nc -zv 198.51.100.7 179 # 2. is port 179 open?
# nc: connect to 198.51.100.7 port 179 (tcp) failed: Connection refused
$ vtysh -c "show ip bgp neighbors 198.51.100.7" | grep -iE "state|last reset"
# BGP state = Active
# Last reset 00:04:12, due to BGP Notification received (Cease/Connection rejected)
# NOTIFICATION tells you exactly what broke:
$ vtysh -c "show ip bgp neighbors 203.0.113.1" | grep -A3 "Notification"
# Last reset: Notification sent (OPEN Message Error/Bad Peer AS)
# ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ code 2 sub 2:
# our 'remote-as' does not match what the peer actually announced.
# Watch a session come up:
$ sudo tcpdump -i eth0 -n tcp port 179 -v
# IP 10.0.0.1.51234 > 203.0.113.1.179: Flags [S] <- TCP handshake
# IP 10.0.0.1.51234 > 203.0.113.1.179: BGP, OPEN, length 53
# Version 4, my AS 64500, Holdtime 180s, ID 10.0.0.1
# Capability: Multiprotocol Extensions IPv4 Unicast
# Capability: 32-Bit AS Number, 64500
# IP 203.0.113.1.179 > 10.0.0.1.51234: BGP, KEEPALIVE, length 19
# IP 203.0.113.1.179 > 10.0.0.1.51234: BGP, UPDATE, length 4096 <- bulk transfer
# Protecting the session (a TCP session is attackable; an OSPF one is not):
# router bgp 64500
# neighbor 203.0.113.1 password SharedSecret <- TCP MD5
# neighbor 203.0.113.1 ttl-security hops 1 <- GTSM (Chapter 5)
# neighbor 203.0.113.1 maximum-prefix 1000000 90 <- cap; warn at 90%
# Avoid resetting the session when policy changes:
$ vtysh -c "clear ip bgp 203.0.113.1 soft in"
# Uses Route Refresh: the peer RESENDS its routes. The session stays up.
$ vtysh -c "clear ip bgp 203.0.113.1"
# HARD reset: drops the session, withdraws ~1M routes. Avoid on production.
$ vtysh -c "show ip bgp neighbors 203.0.113.1" | grep -i refresh
# Route refresh: advertised and received(old & new)
Key takeaways
- BGP runs over TCP port 179, giving reliable ordered delivery for a table of ~1M prefixes — no IGP does this.
- Peers are explicitly configured (no discovery) and need IP reachability first, usually via an IGP.
- States: Idle → Connect → Active → OpenSent → OpenConfirm → Established.
- "Active" is a problem state — TCP is failing and retrying; only Established means the session works.
- Four messages: OPEN (AS, router ID, hold time, capabilities), UPDATE (withdrawals, attributes, NLRI), KEEPALIVE (19 bytes, every hold/3), NOTIFICATION (always closes the session).
- One UPDATE can carry many prefixes sharing the same attributes, which is a major efficiency.
- Defaults: 60 s keepalive, 180 s hold; losing a session withdraws every route learned from that peer.
- Route Refresh re-requests routes after a policy change without a reset; Graceful Restart keeps forwarding while BGP restarts.
- Because the session is TCP, it is attackable — protect it with TCP MD5/AO, GTSM, and maximum-prefix limits.
🧪 Practice
- List the BGP states in order and explain why "Active" indicates a problem rather than healthy operation.
- A peer shows
Activewith no prefixes received. Give three checks in the order you would perform them.- Compute the keepalive interval for hold times of 180, 60, and 30 seconds, and state the risk of aggressive values.
- Interview-style: Why does BGP run over TCP when every IGP avoids it? (Hint: consider the volume of data in a full table and what would have to be rebuilt inside BGP if TCP were not there.)
Path Attributes: AS-PATH, NEXT-HOP, LOCAL-PREF, MED
Theory
An IGP advertises a prefix and a number. BGP advertises a prefix and a set of attributes describing the path — where it has been, how to reach it, and how much the local AS wants it. Policy is expressed by manipulating these attributes, so they are the vocabulary of all inter-domain routing.
Attributes are classified by whether an implementation must understand them and whether unknown ones survive propagation:
| Category | Must recognise? | Must propagate? |
|---|---|---|
| Well-known mandatory | Yes | Yes — must be in every UPDATE |
| Well-known discretionary | Yes | Optional |
| Optional transitive | No | Yes, marked partial |
| Optional non-transitive | No | No — dropped if unrecognised |
That last distinction is what makes BGP extensible: a new optional transitive attribute (communities, for instance) passes unharmed through routers that have never heard of it.
AS-PATH (well-known mandatory) is the list of ASes a route has traversed. It serves two purposes, and the first is the more important:
Loop prevention. An AS that sees its own number in a received AS-PATH rejects the route. This is the mechanism that replaces TTL and split horizon at the inter-domain level, and it is complete: a route cannot loop through an AS twice. It is why BGP is called a path-vector protocol — it advertises the path, not just a distance.
Path selection. A shorter AS-PATH is preferred, but only late in the decision process, well below policy.
AS-PATH is also the main lever for inbound traffic engineering via AS-PATH prepending — adding your own ASN several times to make a path look longer and therefore less attractive. It is crude (other ASes' policy may ignore length entirely) but widely used, because inbound traffic is otherwise controlled by other people's decisions.
NEXT-HOP (well-known mandatory) is the IP address to forward toward. Its handling is the source of a classic failure:
- On eBGP, the advertising router sets NEXT-HOP to its own address.
- On iBGP, the NEXT-HOP is preserved unchanged — so an internal router receives a next hop belonging to an external router, which its IGP may not know how to reach. The route is then inaccessible and never installed (Chapter 6.1's recursive resolution).
The standard fix is next-hop-self on iBGP sessions, which rewrites the next hop to the advertising router's own (IGP-reachable) address.
LOCAL_PREF (well-known discretionary) expresses how much this AS wants to use a path for outbound traffic. Higher is better, default 100. It is propagated only within the AS — never to eBGP peers — which makes it exactly the right tool for AS-wide policy: set it at the edge, and every router in the AS agrees on which exit to prefer.
Since LOCAL_PREF is evaluated very early in the decision process, it overrides AS-PATH length, which is how an operator makes a longer free path beat a shorter paid one.
MED (Multi-Exit Discriminator) (optional non-transitive) is a hint to a neighbouring AS about which of several entry points to prefer. Lower is better. It is used when two ASes share multiple links and one wants to influence where the other sends traffic.
MED is weak by design: it is a suggestion, it is compared only between routes from the same neighbouring AS by default, and many providers simply ignore it. The contrast with LOCAL_PREF is the cleanest way to remember both:
| LOCAL_PREF | MED | |
|---|---|---|
| Controls | Outbound traffic | Inbound traffic (as a hint) |
| Direction | Your decision, authoritative | Their decision, advisory |
| Better value | Higher | Lower |
| Scope | Within your AS only | To the neighbouring AS only |
| Honoured? | Always (it is yours) | Often ignored |
Two more attributes matter in practice. ORIGIN (well-known mandatory) records how the route entered BGP — IGP (i), EGP (e), or Incomplete (?, typically redistributed). Communities (optional transitive) are 32-bit tags with no inherent meaning, used to signal policy: providers publish community values that customers can set to request behaviours such as "do not export to peer X" or "prepend twice toward AS Y". Communities are how large-scale BGP policy is actually operated.
Examples
AS-PATH: loop prevention that needs no TTL
AS 64500 originates 10.1.0.0/16
-> AS 64501 receives: AS-PATH = [64500]
-> AS 64502 receives: AS-PATH = [64501, 64500]
-> AS 64500 receives: AS-PATH = [64502, 64501, 64500]
^^^^^
64500 sees ITSELF in the path -> REJECT. Loop impossible.
This is complete, not probabilistic: a route can never traverse
the same AS twice.
LOCAL_PREF vs MED — the two directions of control
YOUR AS 64500 THEIR AS 64501
+-----------------+ +-----------------+
| |--- link A -------| |
| [R1] [R2] | | |
| |--- link B -------| |
+-----------------+ +-----------------+
OUTBOUND (your traffic leaving): set LOCAL_PREF on your routers.
link A route LOCAL_PREF 200 <- your routers all prefer link A
link B route LOCAL_PREF 100 Authoritative: it is your network.
INBOUND (their traffic arriving): set MED on what you advertise.
advertise via A with MED 10
advertise via B with MED 50 <- you SUGGEST they use A
They may ignore this entirely. It is a hint, not a control.
The iBGP NEXT-HOP trap
external peer 203.0.113.1
|
[R1] eBGP: learns 10.9.0.0/16, NEXT_HOP = 203.0.113.1
| iBGP to R2: NEXT_HOP PRESERVED as 203.0.113.1
[R2]
|
R2 asks its IGP: how do I reach 203.0.113.1?
The IGP has no idea — that address is in the OTHER AS.
-> route marked (inaccessible), NOT installed, traffic black-holed.
FIX on R1: neighbor <R2> next-hop-self
-> R1 rewrites NEXT_HOP to its own loopback, which the IGP knows.
ATTRIBUTES = {
"ORIGIN": ("well-known mandatory", "how the route entered BGP: i / e / ?"),
"AS_PATH": ("well-known mandatory", "ASes traversed; LOOP PREVENTION"),
"NEXT_HOP": ("well-known mandatory", "where to forward"),
"LOCAL_PREF": ("well-known discretionary", "OUTBOUND preference; higher wins; local AS only"),
"MED": ("optional non-transitive", "INBOUND hint to a neighbour; LOWER wins"),
"COMMUNITY": ("optional transitive", "32-bit policy tags; survives unknown routers"),
}
print(f"{'attribute':<13}{'category':<28}purpose")
for a, (cat, purpose) in ATTRIBUTES.items():
print(f"{a:<13}{cat:<28}{purpose}")
# AS-PATH loop prevention — complete, not heuristic.
def accept_route(as_path, my_asn):
if my_asn in as_path:
return False, f"REJECTED: AS{my_asn} already in AS-PATH {as_path} — loop"
return True, f"accepted: AS-PATH {as_path}, length {len(as_path)}"
print()
for path in [[64501, 64500], [64502, 64501, 64500], [64502, 64499]]:
ok, why = accept_route(path, my_asn=64500)
print(f" {why}")
# accepted: AS-PATH [64501, 64500], length 2
# accepted: AS-PATH [64502, 64501, 64500], length 3
# accepted: AS-PATH [64502, 64499], length 2
# (from AS64500's own perspective the first two WOULD be rejected —
# shown here evaluated at AS64500 to make the rule concrete:)
print()
for path in [[64501, 64500], [64502, 64499]]:
ok, why = accept_route(path, my_asn=64501)
print(f" at AS64501: {why}")
# at AS64501: REJECTED: AS64501 already in AS-PATH [64501, 64500] — loop
# at AS64501: accepted: AS-PATH [64502, 64499], length 2
# LOCAL_PREF beats AS-PATH length — how a longer free path wins.
def select(routes):
# Simplified: LOCAL_PREF first (higher), then AS-PATH length (shorter).
return max(routes, key=lambda r: (r["local_pref"], -len(r["as_path"])))
routes = [
{"via": "provider (paid)", "local_pref": 100, "as_path": [1299, 64999]},
{"via": "peer (free)", "local_pref": 200, "as_path": [64510, 64511, 64512, 64999]},
]
best = select(routes)
print(f"\nselected: {best['via']} (LOCAL_PREF {best['local_pref']}, "
f"AS-PATH length {len(best['as_path'])})")
print("the LONGER path won because LOCAL_PREF is evaluated first.")
# selected: peer (free) (LOCAL_PREF 200, AS-PATH length 4)
# the LONGER path won because LOCAL_PREF is evaluated first.
# AS-PATH prepending: the blunt instrument for INBOUND traffic.
def prepend(as_path, my_asn, times):
return [my_asn] * times + as_path
base = [64500]
print()
for n in (1, 2, 4):
p = prepend(base, 64500, n)
print(f"prepend x{n}: AS-PATH {p} (length {len(p)}) "
f"-> {'primary' if n == 1 else 'less attractive'}")
# prepend x1: AS-PATH [64500] (length 1) -> primary
# prepend x2: AS-PATH [64500, 64500] (length 2) -> less attractive
# prepend x4: AS-PATH [64500, 64500, 64500, 64500] (length 4) -> less attractive
# Crude: ASes that decide on LOCAL_PREF never even look at length.
# The NEXT_HOP handling difference, and the failure it causes.
def next_hop_on_advertise(session_type, learned_next_hop, my_address, next_hop_self=False):
if session_type == "ebgp":
return my_address, "eBGP: next hop set to my own address"
if next_hop_self:
return my_address, "iBGP with next-hop-self: rewritten to my loopback"
return learned_next_hop, ("iBGP: next hop PRESERVED — the receiver's IGP "
"may not be able to reach it")
print()
for stype, nhs in [("ebgp", False), ("ibgp", False), ("ibgp", True)]:
nh, why = next_hop_on_advertise(stype, "203.0.113.1", "10.255.0.1", nhs)
print(f"{stype:<5} next-hop-self={str(nhs):<5} -> NEXT_HOP {nh:<14} ({why})")
# ebgp next-hop-self=False -> NEXT_HOP 10.255.0.1 (eBGP: next hop set to my own address)
# ibgp next-hop-self=False -> NEXT_HOP 203.0.113.1 (iBGP: next hop PRESERVED — the receiver's IGP may not be able to reach it)
# ibgp next-hop-self=True -> NEXT_HOP 10.255.0.1 (iBGP with next-hop-self: rewritten to my loopback)
def installable(next_hop, igp_known_prefixes):
import ipaddress
reachable = any(ipaddress.ip_address(next_hop) in ipaddress.ip_network(p)
for p in igp_known_prefixes)
return "INSTALLED" if reachable else "(inaccessible) — NOT installed, traffic black-holed"
igp = ["10.255.0.0/16", "10.0.0.0/8"]
print()
for nh in ("10.255.0.1", "203.0.113.1"):
print(f"NEXT_HOP {nh:<14} -> {installable(nh, igp)}")
# NEXT_HOP 10.255.0.1 -> INSTALLED
# NEXT_HOP 203.0.113.1 -> (inaccessible) — NOT installed, traffic black-holed
# Reading a route's full attribute set:
$ vtysh -c "show ip bgp 10.9.0.0/16"
# BGP routing table entry for 10.9.0.0/16
# Paths: (2 available, best #1, table default)
# 64510 64511 64999
# ^^^^^^^^^^^^^^^^^ AS_PATH — read left to right = nearest to origin
# 203.0.113.1 from 203.0.113.1 (203.0.113.1)
# ^^^^^^^^^^^ NEXT_HOP
# Origin IGP, metric 50, localpref 200, valid, external, best
# ^^^ ^^ MED ^^^ LOCAL_PREF
# Community: 64500:100 64500:666
# ^^^^^^^^^^^^^^^^^^^^ policy tags
# Setting LOCAL_PREF to control OUTBOUND traffic (your authoritative choice):
# route-map PREFER_PEER permit 10
# set local-preference 200 <- higher = preferred, AS-wide
# router bgp 64500
# neighbor 198.51.100.1 route-map PREFER_PEER in
# Setting MED to hint about INBOUND traffic (they may ignore it):
# route-map HINT_ENTRY permit 10
# set metric 50 <- MED; LOWER is preferred
# router bgp 64500
# neighbor 203.0.113.1 route-map HINT_ENTRY out
# AS-PATH prepending — the other inbound lever:
# route-map MAKE_LESS_ATTRACTIVE permit 10
# set as-path prepend 64500 64500 64500
$ vtysh -c "show ip bgp neighbors 203.0.113.1 advertised-routes" | head -3
# Network Next Hop Metric LocPrf Weight Path
# *> 10.1.0.0/16 10.0.0.1 0 0 64500 64500 64500 i
# ^^^^^^^^^^^^^^^^^^ padded
# The next-hop-self fix — essentially mandatory on iBGP sessions:
# router bgp 64500
# neighbor 10.255.0.2 next-hop-self
# Diagnosing the trap:
$ vtysh -c "show ip bgp 10.9.0.0/16" | grep -i inaccessible
# 203.0.113.1 (inaccessible) from 10.255.0.1 (10.255.0.1)
# ^^^^^^^^^^^^^^ the IGP cannot reach this external address
# Communities in action — a provider's published policy:
# 64500:100 = learned from customer
# 64500:200 = learned from peer
# 64500:666 = blackhole this prefix (DDoS mitigation)
# 64500:1299 = prepend once toward AS1299
$ vtysh -c "show ip bgp community 64500:666"
# Network Next Hop Metric LocPrf Weight Path
# *> 192.0.2.5/32 10.0.0.9 0 100 0 i
# ^ a customer requested blackholing of this /32 under attack
Key takeaways
- BGP advertises path attributes, not a metric; policy is expressed by manipulating them.
- Categories: well-known mandatory, well-known discretionary, optional transitive (survives unknown routers), optional non-transitive (dropped).
- AS_PATH provides complete loop prevention — an AS rejects any route containing its own number — making BGP a path-vector protocol.
- AS-PATH prepending is the crude but common lever for inbound traffic engineering.
- NEXT_HOP is rewritten on eBGP but preserved on iBGP, so internal routers may find it unreachable — fixed with
next-hop-self. - LOCAL_PREF controls outbound traffic, higher wins, stays inside the AS, and is evaluated early enough to override AS-PATH length.
- MED hints at preferred inbound entry points, lower wins, and is frequently ignored — it is advisory, not authoritative.
- ORIGIN records how the route entered BGP (
i,e,?); communities are 32-bit policy tags and are how large-scale BGP policy is actually operated.
🧪 Practice
- State which attribute controls outbound versus inbound traffic, and whether higher or lower is preferred for each.
- Explain why an iBGP-learned route shows
(inaccessible)and give the one-line configuration that fixes it.- A route with AS-PATH
[64501, 64500]arrives at AS64500. State what happens and which mechanism is responsible.- Interview-style: You want more inbound traffic on link A than link B from the same provider. Name two mechanisms and explain why neither is guaranteed to work. (Hint: inbound traffic is decided by the other AS — ask which of your signals they are obliged to honour.)
BGP Route Selection Process
Theory
A BGP router frequently holds many paths to the same prefix. Exactly one becomes best — installed in the FIB and advertised onward. The selection runs as an ordered sequence of tie-breakers: the first step that distinguishes the candidates decides, and later steps are never consulted.
The canonical order (Cisco's numbering; other vendors differ slightly at the margins):
| # | Criterion | Prefer | Notes |
|---|---|---|---|
| 0 | NEXT_HOP reachable? | — | Unreachable paths are invalid and never considered |
| 1 | WEIGHT | Highest | Cisco-proprietary, local to one router, never advertised |
| 2 | LOCAL_PREF | Highest | The main policy lever; AS-wide |
| 3 | Locally originated | Self | network/aggregate beats a learned route |
| 4 | AS_PATH length | Shortest | The first step resembling a distance metric |
| 5 | ORIGIN | IGP < EGP < Incomplete | i beats e beats ? |
| 6 | MED | Lowest | Only between paths from the same neighbouring AS |
| 7 | eBGP over iBGP | eBGP | External beats internal |
| 8 | IGP metric to NEXT_HOP | Lowest | "Hot potato" — nearest exit |
| 9 | Oldest path | Oldest | A stability tie-break, not a quality one |
| 10 | Lowest router ID | Lowest | Deterministic final fallback |
| 11 | Lowest neighbour address | Lowest | Absolute last resort |
Several features of this list deserve attention.
Policy comes before distance. WEIGHT and LOCAL_PREF are steps 1 and 2; AS-PATH length is step 4. An operator's preference overrides path length every time — which is exactly how a longer free path beats a shorter paid one.
Step 8 is "hot potato" routing and it has real economic meaning. Among otherwise equal paths, the router picks the one whose next hop is nearest in IGP terms — handing traffic off to the neighbouring AS as quickly as possible, so it travels on someone else's backbone rather than yours. Most networks do this by default because carrying traffic costs money. Cold potato routing (deliberately carrying traffic further inside your own network before handing it off) is used when you want to control quality, typically by content providers.
Step 9 prefers the oldest path, which optimises for stability rather than quality. A newly arrived, possibly better path loses to an established one, avoiding churn from routes that may be flapping. This surprises people expecting "best" to mean "best".
MED's comparison is restricted. By default MED is compared only between paths from the same neighbouring AS, because the values are not meaningful across different ASes — AS 64501's MED 50 and AS 64502's MED 10 measure different things. bgp always-compare-med removes the restriction and is generally discouraged.
By default BGP installs only the single best path, unlike IGPs which install all equal-cost paths automatically. maximum-paths enables BGP multipath, and it requires the candidates to be equal through step 8 as well as sharing the same AS-PATH length (bgp bestpath as-path multipath-relax loosens this for multihoming).
Examples
The decision cascade — the FIRST distinguishing step decides
4 candidate paths for 10.9.0.0/16:
path NEXT_HOP WEIGHT LOCAL_PREF AS_PATH ORIGIN MED type
---- -------- ------ ---------- ------- ------ --- ----
A reachable 0 100 [1299] i 50 eBGP
B reachable 0 200 [1,2,3] i 10 eBGP
C UNREACHABLE 0 300 [5] i 0 eBGP
D reachable 0 200 [7,8] i 20 iBGP
step 0: C's NEXT_HOP is unreachable -> C ELIMINATED (despite LP 300!)
step 1: WEIGHT all 0 -> no decision
step 2: LOCAL_PREF: A=100, B=200, D=200 -> A ELIMINATED
step 3: none locally originated -> no decision
step 4: AS_PATH length: B=3, D=2 -> B ELIMINATED
WINNER: D. Steps 5-11 never run.
Note the shortest AS_PATH did NOT win overall (C had length 1)
and the highest LOCAL_PREF did not either — reachability came first.
Hot potato vs cold potato (step 8)
Your AS, two exits to the same destination:
[ingress]---IGP cost 10---[exit West]---> destination
|
+-------IGP cost 90---[exit East]---> destination
HOT POTATO (default): choose exit West, IGP cost 10.
Hand traffic off ASAP -> it crosses THEIR backbone, not yours.
Cheaper for you.
COLD POTATO: deliberately choose East.
Carry the traffic yourself, closer to the destination.
Costs you bandwidth; you control quality for longer.
Content providers and CDNs do this deliberately.
ORIGIN_RANK = {"i": 0, "e": 1, "?": 2} # IGP < EGP < Incomplete
def bgp_best_path(paths, local_router_id="10.0.0.1"):
"""Ordered tie-breakers; the first distinguishing step decides."""
steps = []
# STEP 0: reachability is a gate, not a tie-break.
valid = [p for p in paths if p["next_hop_reachable"]]
dropped = [p["name"] for p in paths if not p["next_hop_reachable"]]
if dropped:
steps.append(f"step 0 NEXT_HOP reachable: eliminated {dropped}")
if not valid:
return None, steps
def narrow(cands, key, label, reverse=False):
best = max(c[key] for c in cands) if reverse else min(c[key] for c in cands)
kept = [c for c in cands if c[key] == best]
if len(kept) < len(cands):
steps.append(f"{label}: kept {[c['name'] for c in kept]}")
return kept
valid = narrow(valid, "weight", "step 1 WEIGHT (highest)", reverse=True)
if len(valid) == 1: return valid[0], steps
valid = narrow(valid, "local_pref", "step 2 LOCAL_PREF (highest)", reverse=True)
if len(valid) == 1: return valid[0], steps
valid = narrow(valid, "local_origin","step 3 locally originated", reverse=True)
if len(valid) == 1: return valid[0], steps
for p in valid:
p["_aslen"] = len(p["as_path"])
valid = narrow(valid, "_aslen", "step 4 AS_PATH length (shortest)")
if len(valid) == 1: return valid[0], steps
for p in valid:
p["_origin"] = ORIGIN_RANK[p["origin"]]
valid = narrow(valid, "_origin", "step 5 ORIGIN (i < e < ?)")
if len(valid) == 1: return valid[0], steps
# STEP 6: MED, compared only WITHIN the same neighbouring AS.
by_as = {}
for p in valid:
by_as.setdefault(p["as_path"][0] if p["as_path"] else None, []).append(p)
kept = []
for peer_as, group in by_as.items():
lowest = min(g["med"] for g in group)
kept += [g for g in group if g["med"] == lowest]
if len(kept) < len(valid):
steps.append(f"step 6 MED (lowest, same peer AS only): kept {[c['name'] for c in kept]}")
valid = kept
if len(valid) == 1: return valid[0], steps
for p in valid:
p["_ebgp"] = 1 if p["type"] == "ebgp" else 0
valid = narrow(valid, "_ebgp", "step 7 eBGP over iBGP", reverse=True)
if len(valid) == 1: return valid[0], steps
valid = narrow(valid, "igp_metric", "step 8 IGP metric to NEXT_HOP (HOT POTATO)")
if len(valid) == 1: return valid[0], steps
valid = narrow(valid, "age", "step 9 oldest path (STABILITY)", reverse=True)
if len(valid) == 1: return valid[0], steps
valid = narrow(valid, "router_id", "step 10 lowest router ID")
return valid[0], steps
def mk(name, **kw):
base = dict(name=name, next_hop_reachable=True, weight=0, local_pref=100,
local_origin=0, as_path=[64501], origin="i", med=0,
type="ebgp", igp_metric=10, age=100, router_id="10.0.0.9")
base.update(kw)
return base
paths = [
mk("A", local_pref=100, as_path=[1299], med=50),
mk("B", local_pref=200, as_path=[1, 2, 3], med=10),
mk("C", local_pref=300, as_path=[5], next_hop_reachable=False),
mk("D", local_pref=200, as_path=[7, 8], med=20, type="ibgp"),
]
best, steps = bgp_best_path(paths)
for s in steps:
print(s)
print(f"\nBEST PATH: {best['name']}")
# step 0 NEXT_HOP reachable: eliminated ['C']
# step 2 LOCAL_PREF (highest): kept ['B', 'D']
# step 4 AS_PATH length (shortest): kept ['D']
#
# BEST PATH: D
# C had the HIGHEST local_pref (300) and the SHORTEST as_path, and still
# lost — step 0 eliminated it before any comparison happened.
# Hot potato: step 8 decided by IGP distance to the exit.
hot = [mk("exit-West", igp_metric=10), mk("exit-East", igp_metric=90)]
best, steps = bgp_best_path(hot)
print(f"\nhot potato -> {best['name']} (IGP metric {best['igp_metric']})")
print("traffic leaves your AS as early as possible: THEIR backbone carries it.")
# hot potato -> exit-West (IGP metric 10)
# traffic leaves your AS as early as possible: THEIR backbone carries it.
# Step 9 prefers the OLDEST path — stability over optimality.
age = [mk("established", age=50000), mk("just-arrived", age=5)]
best, steps = bgp_best_path(age)
print(f"\noldest-path tie-break -> {best['name']} (age {best['age']}s)")
print("a newer, possibly better path LOSES. This prevents churn from flapping routes.")
# oldest-path tie-break -> established (age 50000s)
# a newer, possibly better path LOSES. This prevents churn from flapping routes.
# Which path won, and why — the '>' marks best.
$ vtysh -c "show ip bgp 10.9.0.0/16"
# BGP routing table entry for 10.9.0.0/16
# Paths: (3 available, best #2, table default)
# 64510 64999
# 203.0.113.1 from 203.0.113.1 (203.0.113.1)
# Origin IGP, metric 50, localpref 100, valid, external
# ^^^ lost on LOCAL_PREF
# 64520 64999
# 198.51.100.1 from 198.51.100.1 (198.51.100.1)
# Origin IGP, metric 10, localpref 200, valid, external, best
# ^^^ WON at step 2
# 64530 64540 64999
# 192.0.2.1 (inaccessible) from 10.255.0.2 (10.255.0.2)
# ^^^^^^^^^^^^^^ eliminated at step 0, whatever its attributes
$ vtysh -c "show ip bgp" | head -6
# Network Next Hop Metric LocPrf Weight Path
# *> 10.1.0.0/16 203.0.113.1 0 100 0 64510 i
# * 10.1.0.0/16 198.51.100.1 0 100 0 64520 64510 i
# ^ '*' = valid, '>' = best. Only the best is installed and re-advertised.
# Influencing selection, in decreasing order of bluntness:
# WEIGHT — one router only, never advertised (Cisco):
# neighbor 203.0.113.1 weight 200
# LOCAL_PREF — AS-wide, the standard policy lever:
# route-map SET_LP permit 10
# set local-preference 200
# AS-PATH prepend — affects OTHER ASes' step 4:
# set as-path prepend 64500 64500
# Enable multipath — BGP does NOT do this by default, unlike IGPs:
# router bgp 64500
# maximum-paths 4 <- eBGP multipath
# maximum-paths ibgp 4
# bgp bestpath as-path multipath-relax <- allow different AS-PATHs of equal
# LENGTH (needed for multihoming)
$ vtysh -c "show ip bgp 10.9.0.0/16" | grep -c multipath
# 2
# Hot vs cold potato:
$ vtysh -c "show ip bgp 10.9.0.0/16" | grep -A1 "best"
# Origin IGP, localpref 100, valid, external, best
# Last update: ...
$ vtysh -c "show ip route 203.0.113.1" | grep metric
# Known via "ospf", distance 110, metric 10 <- step 8 used THIS number
# Deterministic selection (avoid order-of-arrival dependence):
# router bgp 64500
# bgp deterministic-med
# bgp bestpath compare-routerid <- skip the "oldest path" tie-break
Key takeaways
- BGP selects one best path through an ordered sequence of tie-breakers; the first distinguishing step decides.
- Step 0 is a gate: an unreachable NEXT_HOP eliminates a path regardless of how good its attributes are.
- WEIGHT (step 1) is Cisco-proprietary and local to one router; LOCAL_PREF (step 2) is the AS-wide policy lever.
- Policy precedes distance — AS-PATH length is only step 4, so operator preference always wins.
- ORIGIN ranks
i < e < ?; MED is compared only within the same neighbouring AS by default. - Step 8 is hot-potato routing: prefer the exit nearest in IGP metric, handing traffic off early so someone else carries it. Cold potato deliberately does the opposite.
- Step 9 prefers the oldest path, optimising for stability rather than quality, which prevents churn from flapping routes.
- BGP installs only the best path by default;
maximum-paths(withas-path multipath-relaxfor multihoming) is needed for multipath.
🧪 Practice
- List the first five selection steps in order and state which of AS-PATH length and LOCAL_PREF is evaluated first.
- Given paths with LOCAL_PREF 150/AS-PATH 4 and LOCAL_PREF 100/AS-PATH 1, determine the winner and the deciding step.
- Explain hot-potato routing and give the business reason most networks use it by default.
- Interview-style: A path has the highest LOCAL_PREF and the shortest AS-PATH, yet is not selected. Give the most likely explanation. (Hint: one check happens before any attribute is compared — and it depends on a protocol other than BGP.)
eBGP vs iBGP and Route Reflectors
Theory
BGP runs in two modes, and although the messages are identical, the rules differ fundamentally.
eBGP runs between routers in different ASes. It is the policy boundary. Peers are usually directly connected, TTL is 1 by default (a security property, reinforced by GTSM), the NEXT_HOP is rewritten to the advertising router, and the local ASN is prepended to AS_PATH.
iBGP runs between routers in the same AS. Its purpose is to distribute externally learned routes internally. Peers are usually not directly connected — they peer between loopback addresses, reachable via the IGP, so the session survives any single link failure. AS_PATH is not prepended (the route has not left the AS), and NEXT_HOP is preserved (hence next-hop-self, from the previous topic).
The defining rule of iBGP:
A router will not re-advertise a route learned via iBGP to another iBGP peer.
This is iBGP's loop prevention. AS_PATH cannot help internally, because the ASN is not prepended within the AS — so there is no way to detect a loop. The rule prevents loops absolutely, and it has a severe consequence: every iBGP router must peer directly with every other, a full mesh.
A full mesh requires n(n-1)/2 sessions, which grows quadratically:
| Routers | iBGP sessions |
|---|---|
| 5 | 10 |
| 10 | 45 |
| 20 | 190 |
| 50 | 1,225 |
| 100 | 4,950 |
Each session consumes memory, CPU, and configuration, and adding one router means touching every existing router. This does not scale.
Two solutions exist.
Route reflectors (RFC 4456) relax the re-advertisement rule for designated routers. An RR may reflect routes between its clients, so clients need only peer with the RR rather than with each other. Sessions drop from n(n-1)/2 to roughly n.
New loop-prevention attributes replace the rule that was relaxed:
- ORIGINATOR_ID — the router ID of the route's originator inside the AS. A router receiving its own ID discards the route.
- CLUSTER_LIST — the list of clusters the route has passed through. An RR seeing its own cluster ID discards the route.
These are exactly AS_PATH's loop-prevention idea, applied within the AS.
The reflection rules are precise and worth memorising, because they determine whether a design works:
| Route learned from | Reflected to |
|---|---|
| Client | All clients and all non-clients |
| Non-client | Clients only |
| eBGP peer | Everyone |
Redundancy is achieved with multiple RRs, typically sharing a CLUSTER_ID. The main design hazard is that an RR reflects only its own best path, so clients may not learn paths the RR did not choose — which can cause suboptimal routing and reduce path diversity. BGP Add-Path addresses this by allowing multiple paths to be advertised for one prefix.
Confederations (RFC 5065) are the alternative: split the AS into sub-ASes that run eBGP between them while appearing as a single AS externally. Sub-AS numbers are stripped from the AS_PATH at the confederation boundary. Confederations are more flexible but more complex, and route reflectors are far more common in practice.
Examples
The iBGP split-horizon rule, and why the full mesh follows
R1 --- R2 --- R3 (all in AS 64500)
R1 learns 10.9.0.0/16 via eBGP and advertises it to R2 over iBGP.
R2 must NOT re-advertise it to R3 (iBGP-learned -> no iBGP re-advertise).
-> R3 never learns the route unless R1 peers with R3 DIRECTLY.
Hence the FULL MESH:
R1------R2
\ /
\ /
R3 3 routers -> 3 sessions
10 routers -> 45 sessions
100 routers -> 4,950 sessions
Route reflector: sessions collapse from quadratic to linear
FULL MESH (6 routers, 15 sessions): WITH AN RR (6 routers, 5 sessions):
R1----R2 [RR]
|\ /| \ / | | \
| \/ | R3 / | | \
| /\ | | R1 R2 R3 R4 R5
R4---R5-+ ^ clients peer ONLY with the RR
Reflection rules — what goes where
learned from a CLIENT:
[client1] --> [RR] --> all clients AND all non-clients (everyone)
learned from a NON-CLIENT (a normal iBGP peer):
[non-client] --> [RR] --> CLIENTS ONLY
^ non-clients are assumed to be full-meshed
with each other already
learned from eBGP:
[external] --> [RR] --> everyone
Loop prevention that replaces the relaxed rule
route passes RR1 (cluster 1.1.1.1) -> CLUSTER_LIST = [1.1.1.1]
route passes RR2 (cluster 2.2.2.2) -> CLUSTER_LIST = [2.2.2.2, 1.1.1.1]
route returns to RR1 -> RR1 sees 1.1.1.1 in CLUSTER_LIST
-> DISCARD. Same idea as AS_PATH.
def full_mesh_sessions(n):
return n * (n - 1) // 2
def rr_sessions(n, num_rrs=1):
clients = n - num_rrs
rr_to_rr = full_mesh_sessions(num_rrs)
return clients * num_rrs + rr_to_rr
print(f"{'routers':>8}{'full mesh':>12}{'1 RR':>8}{'2 RRs':>8}{'reduction (2 RRs)':>20}")
for n in (5, 10, 20, 50, 100, 500):
fm, r1, r2 = full_mesh_sessions(n), rr_sessions(n, 1), rr_sessions(n, 2)
print(f"{n:>8}{fm:>12,}{r1:>8,}{r2:>8,}{100*(1-r2/fm):>19.1f}%")
# routers full mesh 1 RR 2 RRs reduction (2 RRs)
# 5 10 4 7 30.0%
# 10 45 9 17 62.2%
# 20 190 19 37 80.5%
# 50 1,225 49 97 92.1%
# 100 4,950 99 197 96.0%
# 500 124,750 499 997 99.2%
# eBGP vs iBGP behaviour, side by side.
def process_route(session_type, route, my_asn, my_address, next_hop_self=False):
out = dict(route)
if session_type == "ebgp":
out["as_path"] = [my_asn] + route["as_path"] # prepend our ASN
out["next_hop"] = my_address # rewrite
out["readvertise_to_ibgp"] = True
else:
out["as_path"] = route["as_path"] # NOT prepended
out["next_hop"] = my_address if next_hop_self else route["next_hop"]
out["readvertise_to_ibgp"] = False # THE iBGP RULE
return out
r = {"prefix": "10.9.0.0/16", "as_path": [64999], "next_hop": "203.0.113.1"}
print()
for stype in ("ebgp", "ibgp"):
o = process_route(stype, r, my_asn=64500, my_address="10.255.0.1")
print(f"{stype:<6} AS_PATH={str(o['as_path']):<18} NEXT_HOP={o['next_hop']:<14} "
f"re-advertise to iBGP peers: {o['readvertise_to_ibgp']}")
# ebgp AS_PATH=[64500, 64999] NEXT_HOP=10.255.0.1 re-advertise to iBGP peers: True
# ibgp AS_PATH=[64999] NEXT_HOP=203.0.113.1 re-advertise to iBGP peers: False
# Route reflector logic, including the loop-prevention attributes.
class RouteReflector:
def __init__(self, router_id, cluster_id):
self.router_id, self.cluster_id = router_id, cluster_id
self.clients, self.non_clients = set(), set()
def receive(self, route, from_peer, peer_type):
# Loop prevention replacing the relaxed re-advertisement rule.
if route.get("originator_id") == self.router_id:
return [], "DISCARD: ORIGINATOR_ID is my own router ID"
if self.cluster_id in route.get("cluster_list", []):
return [], f"DISCARD: my cluster {self.cluster_id} already in CLUSTER_LIST"
out = dict(route)
out.setdefault("originator_id", from_peer)
out["cluster_list"] = [self.cluster_id] + route.get("cluster_list", [])
if peer_type == "ebgp":
targets = self.clients | self.non_clients
why = "from eBGP -> reflect to EVERYONE"
elif peer_type == "client":
targets = (self.clients | self.non_clients) - {from_peer}
why = "from a CLIENT -> reflect to clients AND non-clients"
else:
targets = self.clients
why = "from a NON-CLIENT -> reflect to CLIENTS ONLY"
return sorted(targets), why
rr = RouteReflector("10.255.0.1", cluster_id="1.1.1.1")
rr.clients = {"R2", "R3", "R4"}
rr.non_clients = {"RR2"}
route = {"prefix": "10.9.0.0/16", "as_path": [64999]}
print()
for from_peer, ptype in [("R2", "client"), ("RR2", "non-client"), ("ext", "ebgp")]:
targets, why = rr.receive(route, from_peer, ptype)
print(f"from {from_peer:<5} ({ptype:<10}) -> {str(targets):<28} {why}")
# from R2 (client ) -> ['R3', 'R4', 'RR2'] from a CLIENT -> reflect to clients AND non-clients
# from RR2 (non-client) -> ['R2', 'R3', 'R4'] from a NON-CLIENT -> reflect to CLIENTS ONLY
# from ext (ebgp ) -> ['R2', 'R3', 'R4', 'RR2'] from eBGP -> reflect to EVERYONE
# The loop-prevention attributes doing their job.
looped = {"prefix": "10.9.0.0/16", "as_path": [64999], "cluster_list": ["1.1.1.1"]}
targets, why = rr.receive(looped, "RR2", "non-client")
print(f"\nreturning route -> {why}")
# returning route -> DISCARD: my cluster 1.1.1.1 already in CLUSTER_LIST
# The RR's limitation: it reflects only ITS OWN best path.
def paths_seen_by_client(all_paths, rr_best, add_path=False):
return all_paths if add_path else [rr_best]
all_paths = ["via-exit-A", "via-exit-B", "via-exit-C"]
print()
for ap in (False, True):
seen = paths_seen_by_client(all_paths, "via-exit-A", ap)
label = "with Add-Path" if ap else "standard RR"
print(f"{label:<16} client sees {len(seen)} of {len(all_paths)} paths: {seen}")
# standard RR client sees 1 of 3 paths: ['via-exit-A']
# with Add-Path client sees 3 of 3 paths: ['via-exit-A', 'via-exit-B', 'via-exit-C']
# Reduced path diversity is the main design cost of route reflection.
# iBGP peering on LOOPBACKS, so the session survives a link failure:
# router bgp 64500
# neighbor 10.255.0.2 remote-as 64500 <- same ASN = iBGP
# neighbor 10.255.0.2 update-source Loopback0 <- ESSENTIAL for iBGP
# neighbor 10.255.0.2 next-hop-self
# neighbor 203.0.113.1 remote-as 1299 <- different ASN = eBGP
# ! eBGP is normally directly connected; no update-source needed.
# ! For multihop eBGP: neighbor 203.0.113.1 ebgp-multihop 2
# Configuring a route reflector:
# router bgp 64500
# neighbor 10.255.0.2 route-reflector-client
# neighbor 10.255.0.3 route-reflector-client
# neighbor 10.255.0.4 route-reflector-client
# bgp cluster-id 1.1.1.1 <- shared by redundant RRs
$ vtysh -c "show ip bgp neighbors 10.255.0.2" | grep -i reflector
# Route-Reflector Client
$ vtysh -c "show ip bgp 10.9.0.0/16"
# 64999
# 10.255.0.9 from 10.255.0.2 (10.255.0.9)
# Origin IGP, localpref 100, valid, internal, best
# Originator: 10.255.0.9, Cluster list: 1.1.1.1
# ^^^^^^^^^^ who first injected it into our AS
# ^^^^^^^ which RRs it passed
# Telling eBGP from iBGP at a glance:
$ vtysh -c "show ip bgp summary"
# Neighbor V AS MsgRcvd MsgSent Up/Down State/PfxRcd
# 203.0.113.1 4 1299 184211 9112 3w2d 974213 <- eBGP
# 10.255.0.2 4 64500 14021 13998 8w1d 1204 <- iBGP
# ^^^^^ same as our local AS
$ vtysh -c "show ip bgp 10.9.0.0/16" | grep -E "internal|external"
# Origin IGP, localpref 100, valid, internal, best <- learned via iBGP
# Origin IGP, localpref 100, valid, external <- learned via eBGP
# Add-Path restores the diversity a route reflector removes:
# router bgp 64500
# address-family ipv4 unicast
# neighbor 10.255.0.2 addpath-tx-all-paths
$ vtysh -c "show ip bgp 10.9.0.0/16" | grep -c "10.255"
# 3 # three paths now visible to the client, not just the RR's favourite
# Verifying the iBGP rule empirically: a route learned via iBGP is NOT
# re-advertised to another iBGP peer.
$ vtysh -c "show ip bgp neighbors 10.255.0.3 advertised-routes" | grep 10.9
# (empty on a non-RR router — the rule in action)
Key takeaways
- eBGP runs between different ASes: prepends the local ASN to AS_PATH, rewrites NEXT_HOP, TTL 1 by default.
- iBGP runs within one AS: no AS_PATH prepend, NEXT_HOP preserved, peers on loopbacks with
update-sourceso sessions survive link failures. - The core rule: a route learned via iBGP is never re-advertised to another iBGP peer — iBGP's only loop prevention, since AS_PATH cannot help internally.
- That rule forces a full mesh of
n(n-1)/2sessions: 100 routers need 4,950 sessions. - Route reflectors relax the rule for designated routers, cutting sessions to roughly
n; loops are prevented instead by ORIGINATOR_ID and CLUSTER_LIST. - Reflection rules: from a client → everyone; from a non-client → clients only; from eBGP → everyone.
- An RR reflects only its own best path, reducing path diversity — Add-Path restores it.
- Confederations are the alternative (sub-ASes running eBGP internally), more flexible but far less commonly deployed.
🧪 Practice
- Compute full-mesh iBGP sessions for 30 routers, then with two route reflectors, and give the percentage reduction.
- State the three reflection rules and explain why a non-client's route goes only to clients.
- Explain why iBGP peers use loopback addresses while eBGP peers normally use interface addresses.
- Interview-style: Why does iBGP forbid re-advertising iBGP-learned routes, when eBGP happily re-advertises everything? (Hint: ask what stops an eBGP loop, and whether that same mechanism changes as a route moves inside one AS.)
Routing Policy and Traffic Engineering
Theory
Policy is BGP's purpose. An IGP asks "what is the shortest path?"; BGP asks "which path am I willing to use, and which am I willing to tell others about?" Every BGP deployment is fundamentally a policy configuration.
Policy is applied at three points, and keeping them distinct is essential:
Inbound filtering — what to accept from a peer. This is the security boundary and must never be omitted. Best-path manipulation — which accepted path to prefer. Outbound filtering — what to advertise, to whom.
The tools are route maps (or policy statements), which match on prefix lists, AS-PATH regular expressions, or communities, and then permit, deny, or modify attributes.
Traffic engineering divides sharply by direction, and the asymmetry is the key insight:
Outbound traffic is easy because you control it completely. LOCAL_PREF is the instrument: set it at the edge based on which peer a route came from, and every router in your AS agrees on the preferred exit. This is fully authoritative.
Inbound traffic is hard because it is decided by other ASes' policies, which you neither see nor control. You can only send signals and hope:
| Technique | Mechanism | Reliability |
|---|---|---|
| AS-PATH prepending | Make a path look longer | Moderate — ignored by anyone deciding on LOCAL_PREF |
| MED | Hint between entry points | Weak — often ignored; same-AS only |
| Communities | Ask the provider to act | Good — if they publish and honour them |
| More-specific advertisement | Advertise a /24 on the preferred link | Strong — longest prefix match always wins |
| Selective advertisement | Advertise a prefix only where wanted | Strong, but removes redundancy |
The most reliable method is the more-specific advertisement, because it exploits longest prefix match (Chapter 5), which no policy can override. Advertise 10.1.0.0/16 on both links and 10.1.0.0/17 on the preferred one, and traffic for that half arrives where you want. The cost is a larger global routing table, which is why deaggregation for traffic engineering is a tragedy of the commons — locally rational, globally expensive.
Communities are how serious policy is operated at scale. A provider publishes values customers can set, such as "prepend twice toward AS1299", "do not export to peers", or "blackhole this prefix". This gives the customer remote control over the provider's policy without either party editing configuration per change. Large communities (RFC 8092) extend the original 32-bit format to accommodate 32-bit ASNs.
Three filtering practices are mandatory rather than optional, and omitting them causes real incidents:
Filter inbound. Reject RFC 1918 space, default routes you did not ask for, your own prefixes, and anything not expected from that peer. Accepting everything is how route leaks propagate.
Filter outbound. Advertise only your own prefixes and your customers'. This is the valley-free rule made concrete, and failing to do it is exactly how large route leaks happen — an AS accidentally advertises its provider's routes to another provider and attracts a continent's traffic into a network that cannot carry it.
Set maximum-prefix limits. Cap what a peer may send, so a misconfiguration on their side cannot exhaust your memory.
Examples
Outbound is yours; inbound is theirs
OUTBOUND — fully under your control:
routes from peer -> LOCAL_PREF 200
routes from provider-> LOCAL_PREF 100
Every router in your AS now prefers the peer. Authoritative. Done.
INBOUND — controlled by strangers:
You advertise. They decide. You can only influence:
prepend -> only helps if THEY compare AS-PATH length at all
MED -> only if THEY honour it (often not)
community-> only if THEY publish and act on it
/24 more-specific -> ALWAYS works: longest prefix match cannot be
overridden by policy anywhere on the internet
The more-specific trick, and its global cost
You have 10.1.0.0/16 and two links.
link A: advertise 10.1.0.0/16 AND 10.1.0.0/17
link B: advertise 10.1.0.0/16 AND 10.1.128.0/17
Traffic for 10.1.0.5 -> matches /17 via A -> arrives on A
Traffic for 10.1.200.5 -> matches /17 via B -> arrives on B
If A fails, the /16 still covers everything -> redundancy preserved.
COST: 3 global routing entries instead of 1, in EVERY DFZ router
on earth. Rational for you; expensive for everyone.
Route leak: what missing outbound filters cause
[Provider A] ---- [small AS] ---- [Provider B]
^
Correct: advertise only OWN + CUSTOMER prefixes to both.
Leak: advertise A's full table to B.
B now believes the small AS is a path to the whole internet.
Traffic floods in. The small AS's links saturate instantly.
Large parts of the internet become unreachable.
This has happened repeatedly at global scale. The fix is a single
outbound prefix filter that should always have been there.
def apply_policy(route, peer_type):
"""Outbound TE: LOCAL_PREF by relationship. Fully authoritative."""
LP = {"customer": 300, "peer": 200, "provider": 100}
out = dict(route)
out["local_pref"] = LP[peer_type]
return out
print("outbound preference by relationship:")
for pt in ("customer", "peer", "provider"):
r = apply_policy({"prefix": "10.9.0.0/16"}, pt)
print(f" from {pt:<9} -> LOCAL_PREF {r['local_pref']} "
f"({'prefer: they pay us' if pt=='customer' else 'free' if pt=='peer' else 'costs money'})")
# outbound preference by relationship:
# from customer -> LOCAL_PREF 300 (prefer: they pay us)
# from peer -> LOCAL_PREF 200 (free)
# from provider -> LOCAL_PREF 100 (costs money)
# Inbound techniques, ranked by whether they actually work.
INBOUND_TE = {
"AS-PATH prepending": (0.6, "ignored by ASes that decide on LOCAL_PREF"),
"MED": (0.3, "advisory; same-neighbour-AS only; often ignored"),
"Communities": (0.9, "works IF the provider publishes and honours them"),
"More-specific prefix": (1.0, "longest prefix match CANNOT be overridden"),
"Selective advertise": (1.0, "absolute, but sacrifices redundancy"),
}
print()
print(f"{'technique':<24}{'reliability':>12} caveat")
for name, (rel, caveat) in sorted(INBOUND_TE.items(), key=lambda kv: -kv[1][0]):
print(f"{name:<24}{rel:>11.0%} {caveat}")
# technique reliability caveat
# More-specific prefix 100% longest prefix match CANNOT be overridden
# Selective advertise 100% absolute, but sacrifices redundancy
# Communities 90% works IF the provider publishes and honours them
# AS-PATH prepending 60% ignored by ASes that decide on LOCAL_PREF
# MED 30% advisory; same-neighbour-AS only; often ignored
# The more-specific technique, and the global table cost.
import ipaddress
def more_specific_plan(block, links):
net = ipaddress.ip_network(block)
halves = list(net.subnets(prefixlen_diff=1))
plan = {}
for link, half in zip(links, halves):
plan[link] = [str(net), str(half)] # /16 everywhere + one /17
return plan, sum(len(v) for v in plan.values()) - (len(links) - 1)
plan, entries = more_specific_plan("10.1.0.0/16", ["link-A", "link-B"])
print()
for link, adverts in plan.items():
print(f"{link}: advertise {adverts}")
print(f"global routing entries created: {entries} (was 1)")
# link-A: advertise ['10.1.0.0/16', '10.1.0.0/17']
# link-B: advertise ['10.1.0.0/16', '10.1.128.0/17']
# global routing entries created: 3 (was 1)
def arrives_on(dst, plan):
d = ipaddress.ip_address(dst)
best_len, best_link = -1, None
for link, adverts in plan.items():
for a in adverts:
n = ipaddress.ip_network(a)
if d in n and n.prefixlen > best_len:
best_len, best_link = n.prefixlen, link
return best_link, best_len
print()
for dst in ("10.1.0.5", "10.1.200.5"):
link, plen = arrives_on(dst, plan)
print(f"traffic for {dst:<12} arrives on {link} (matched /{plen})")
# traffic for 10.1.0.5 arrives on link-A (matched /17)
# traffic for 10.1.200.5 arrives on link-B (matched /17)
# Mandatory inbound filtering — the security boundary.
BOGONS = ["10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16", "127.0.0.0/8",
"169.254.0.0/16", "0.0.0.0/8", "224.0.0.0/4", "240.0.0.0/4"]
def inbound_filter(prefix, my_prefixes, max_len=24):
net = ipaddress.ip_network(prefix)
for b in BOGONS:
if net.subnet_of(ipaddress.ip_network(b)):
return False, f"REJECT: bogon/private space ({b})"
for mine in my_prefixes:
if net.subnet_of(ipaddress.ip_network(mine)):
return False, f"REJECT: this is OUR prefix ({mine}) — possible hijack"
if net.prefixlen > max_len:
return False, f"REJECT: /{net.prefixlen} more specific than /{max_len}"
if net.prefixlen == 0:
return False, "REJECT: default route not requested"
return True, "accept"
print()
for p in ["203.0.113.0/24", "10.5.0.0/16", "198.51.100.0/25", "0.0.0.0/0", "192.0.2.0/24"]:
ok, why = inbound_filter(p, my_prefixes=["192.0.2.0/24"])
print(f"{p:<20}{why}")
# 203.0.113.0/24 accept
# 10.5.0.0/16 REJECT: bogon/private space (10.0.0.0/8)
# 198.51.100.0/25 REJECT: /25 more specific than /24
# 0.0.0.0/0 REJECT: default route not requested
# 192.0.2.0/24 REJECT: this is OUR prefix (192.0.2.0/24) — possible hijack
# Outbound filtering: the valley-free rule, enforced as code.
def outbound_filter(route_source, advertise_to):
if route_source in ("own", "customer"):
return True, "own/customer prefixes go to everyone"
if advertise_to == "customer":
return True, "customers get the full table"
return False, f"REJECT: {route_source} -> {advertise_to} would be a ROUTE LEAK"
print()
for src in ("own", "customer", "peer", "provider"):
for dst in ("peer", "provider"):
ok, why = outbound_filter(src, dst)
if not ok:
print(f" {src:<9} -> {dst:<9} {why}")
# peer -> peer REJECT: peer -> peer would be a ROUTE LEAK
# peer -> provider REJECT: peer -> provider would be a ROUTE LEAK
# provider -> peer REJECT: provider -> peer would be a ROUTE LEAK
# provider -> provider REJECT: provider -> provider would be a ROUTE LEAK
# Outbound TE: LOCAL_PREF by relationship — the standard pattern.
# ip community-list standard CUSTOMER permit 64500:100
# route-map FROM_CUSTOMER permit 10
# set local-preference 300
# set community 64500:100
# route-map FROM_PEER permit 10
# set local-preference 200
# set community 64500:200
# route-map FROM_PROVIDER permit 10
# set local-preference 100
# Mandatory inbound filtering — never omit this:
# ip prefix-list FROM_PEER_OK seq 5 deny 0.0.0.0/0
# ip prefix-list FROM_PEER_OK seq 10 deny 10.0.0.0/8 le 32
# ip prefix-list FROM_PEER_OK seq 15 deny 192.0.2.0/24 le 32 <- our own space
# ip prefix-list FROM_PEER_OK seq 20 permit 0.0.0.0/0 ge 8 le 24
# router bgp 64500
# neighbor 198.51.100.1 prefix-list FROM_PEER_OK in
# neighbor 198.51.100.1 maximum-prefix 500000 90 restart 15
# ^^^^^^ cap; warn at 90%
# Outbound filtering — what prevents a route leak:
# ip as-path access-list 1 permit ^$ <- our own originated routes
# ip as-path access-list 1 permit ^64510_ <- our customer AS64510
# router bgp 64500
# neighbor 203.0.113.1 filter-list 1 out
# Using a provider's published communities for inbound TE:
# route-map TO_PROVIDER permit 10
# set community 1299:70 additive <- e.g. "prepend 2x toward peers"
# ! Each provider publishes its own list; check their documentation.
$ whois -h whois.radb.net AS1299 | grep -i remarks | head -3
# remarks: 1299:XXX - communities for traffic engineering
# Verify what you are actually sending and receiving:
$ vtysh -c "show ip bgp neighbors 203.0.113.1 advertised-routes" | wc -l
# 42 # should be OUR prefixes + customers' only. If this is
# ~1,000,000 you are leaking a full table — fix it NOW.
$ vtysh -c "show ip bgp neighbors 203.0.113.1 received-routes" | head -3
# AS-PATH regex filtering:
$ vtysh -c "show ip bgp regexp ^64510_" # routes originating from AS64510
$ vtysh -c "show ip bgp regexp _64999$" # routes ORIGINATED by AS64999
Key takeaways
- Policy is applied at three points: inbound filtering, best-path manipulation, and outbound filtering.
- Outbound traffic engineering is easy and authoritative — set LOCAL_PREF by relationship and your whole AS agrees.
- Inbound traffic engineering is unreliable because other ASes decide; you can only send signals.
- Inbound tools, by reliability: more-specific advertisement (always works — longest prefix match cannot be overridden), selective advertisement, communities, AS-PATH prepending, then MED.
- More-specific advertisement adds entries to every DFZ router on earth — locally rational, globally costly.
- Communities are how large-scale policy is operated; providers publish values customers can set to request actions remotely.
- Inbound filtering is mandatory: reject bogons, private space, default routes, your own prefixes, and over-specific prefixes.
- Outbound filtering is mandatory: advertise only your own and your customers' prefixes — omitting this is exactly how route leaks occur.
- Always set maximum-prefix limits so a peer's misconfiguration cannot exhaust your memory.
🧪 Practice
- Explain why outbound traffic engineering is reliable while inbound is not, naming the attribute that makes outbound authoritative.
- Design a more-specific advertisement plan to split
172.20.0.0/16across two links, and state the global routing table cost.- List five prefix categories that must be rejected inbound from a peer, with a reason for each.
- Interview-style: An AS accidentally advertises its provider's full table to another provider and large parts of the internet become unreachable. Name the incident type and the single filter that would have prevented it. (Hint: ask which routes that AS was ever entitled to advertise to a non-customer.)
BGP Hijacking and RPKI
Theory
BGP was designed in 1989 among a small number of mutually trusting research networks. It contains no mechanism to verify that an AS is entitled to announce a prefix. Any AS can announce any prefix, and by default its neighbours will believe it. That property has never been fixed at the protocol level; it is mitigated by layers added around BGP.
BGP hijacking is the exploitation of this. Three forms:
Prefix hijacking. An AS announces a prefix it does not own. If the announcement propagates, traffic destined for the legitimate owner arrives at the hijacker instead.
More-specific hijacking. The effective version. Announce a /24 inside someone's /16, and longest prefix match (Chapter 5) makes it win globally, regardless of any policy or path length. This is why most DFZ routers refuse prefixes longer than /24 — it bounds the granularity of the attack.
AS-PATH manipulation. Forge the path to appear adjacent to the legitimate origin, making the announcement look plausible.
Motives vary: traffic interception (the hijacker forwards traffic onward after inspecting or modifying it, so the outage that would reveal the attack never occurs), denial of service, spam from hijacked address space, and cryptocurrency theft by hijacking prefixes hosting wallet services. Many incidents are also simple misconfiguration — a fat-fingered prefix or a missing filter — and are indistinguishable from an attack at the protocol level.
The defences form layers, none complete on its own.
IRR (Internet Routing Registry) databases let operators publish route objects declaring which AS may originate which prefix. Filters can be generated from them. The weakness is that IRR data is often stale and unauthenticated — anyone can register objects in some registries.
RPKI (Resource Public Key Infrastructure, RFC 6480) is the cryptographic answer. RIRs issue certificates binding address blocks to their holders. A holder creates a ROA (Route Origin Authorization) — a signed statement saying "AS X may originate prefix P, with maximum length L." Routers fetch validated ROA data from a local RPKI validator (over the RTR protocol) and classify each received route:
| State | Meaning |
|---|---|
| Valid | A ROA exists and the origin AS and prefix length match |
| Invalid | A ROA exists but the origin AS is wrong, or the prefix is longer than maxLength |
| NotFound | No ROA covers this prefix |
ROV (Route Origin Validation) is the enforcement step: drop Invalid routes. Adoption has grown substantially — most large transit providers now drop Invalids, which meaningfully reduces the reach of an origin hijack.
RPKI's crucial limitation must be stated clearly: it validates only the origin AS, not the path. An attacker who forges an AS-PATH ending in the legitimate origin produces a route that is RPKI-Valid while still being a hijack. Closing that gap requires BGPsec (RFC 8205), which cryptographically signs each AS hop — but its per-hop signature verification cost and the need for near-universal adoption have kept deployment negligible.
A common configuration error deserves its own mention: setting maxLength too loosely. A ROA for 10.0.0.0/16 with maxLength 24 authorises anyone's announcement of any /17 through /24 inside it, provided the origin AS matches — which re-opens the more-specific attack against yourself. The guidance is to set maxLength equal to the prefix length you actually announce.
The complementary practice is MANRS (Mutually Agreed Norms for Routing Security), which specifies four actions every operator should take: filtering, anti-spoofing, coordination, and global validation (publishing routing data and ROAs).
Examples
The more-specific hijack — why it works globally
Legitimate: AS64500 announces 10.1.0.0/16
Hijacker: AS64666 announces 10.1.5.0/24
Every router on the internet, for destination 10.1.5.9:
10.1.0.0/16 matches (16 bits)
10.1.5.0/24 matches (24 bits) <-- LONGEST PREFIX MATCH WINS
-> traffic goes to AS64666, everywhere, regardless of policy,
AS-PATH length, or how far away the hijacker is.
No routing policy can override this. Longest prefix match is
evaluated before every attribute.
RPKI validation states
ROA on file: prefix 10.1.0.0/16, origin AS64500, maxLength 16
received: 10.1.0.0/16 origin AS64500 -> VALID (matches exactly)
received: 10.1.0.0/16 origin AS64666 -> INVALID (wrong origin AS)
received: 10.1.5.0/24 origin AS64500 -> INVALID (longer than maxLength)
received: 10.1.5.0/24 origin AS64666 -> INVALID (both wrong) -> DROPPED
received: 203.0.113.0/24 origin AS999 -> NOTFOUND (no ROA; still accepted)
The maxLength trap
ROA: 10.0.0.0/16, AS64500, maxLength 24 <- TOO LOOSE
Now ANY announcement of a /17 .. /24 inside 10.0.0.0/16 that claims
origin AS64500 validates as VALID. An attacker forging the origin AS
gets a VALID more-specific hijack.
ROA: 10.0.0.0/16, AS64500, maxLength 16 <- CORRECT
Only the exact /16 validates. More-specifics become INVALID.
What RPKI does NOT protect against
real path: AS64500 (origin) <- AS1299 <- you
forged path: AS64666 announces 10.1.0.0/16 with AS_PATH [64666, 64500]
^^^^^ appended
RPKI checks the ORIGIN (rightmost AS) = 64500. Matches the ROA.
-> validates as VALID. Still a hijack.
Only BGPsec, which signs each hop, would catch this.
class ROA:
def __init__(self, prefix, origin_asn, max_length=None):
import ipaddress
self.net = ipaddress.ip_network(prefix)
self.origin_asn = origin_asn
self.max_length = max_length if max_length is not None else self.net.prefixlen
def validate(prefix, origin_asn, roas):
"""RFC 6811 Route Origin Validation."""
import ipaddress
net = ipaddress.ip_network(prefix)
covering = [r for r in roas if net.subnet_of(r.net)]
if not covering:
return "NotFound", "no ROA covers this prefix — accepted, unverified"
for r in covering:
if r.origin_asn == origin_asn and net.prefixlen <= r.max_length:
return "Valid", f"matches ROA (AS{r.origin_asn}, maxLength {r.max_length})"
reasons = []
for r in covering:
if r.origin_asn != origin_asn:
reasons.append(f"origin AS{origin_asn} != authorised AS{r.origin_asn}")
if net.prefixlen > r.max_length:
reasons.append(f"/{net.prefixlen} exceeds maxLength /{r.max_length}")
return "Invalid", "; ".join(dict.fromkeys(reasons))
roas = [ROA("10.1.0.0/16", 64500, max_length=16)]
print(f"{'prefix':<18}{'origin':<10}{'state':<11}reason")
for prefix, asn in [("10.1.0.0/16", 64500), ("10.1.0.0/16", 64666),
("10.1.5.0/24", 64500), ("10.1.5.0/24", 64666),
("203.0.113.0/24", 64999)]:
state, why = validate(prefix, asn, roas)
print(f"{prefix:<18}AS{asn:<8}{state:<11}{why}")
# prefix origin state reason
# 10.1.0.0/16 AS64500 Valid matches ROA (AS64500, maxLength 16)
# 10.1.0.0/16 AS64666 Invalid origin AS64666 != authorised AS64500
# 10.1.5.0/24 AS64500 Invalid /24 exceeds maxLength /16
# 10.1.5.0/24 AS64666 Invalid origin AS64666 != authorised AS64500; /24 exceeds maxLength /16
# 203.0.113.0/24 AS64999 NotFound no ROA covers this prefix — accepted, unverified
# The maxLength trap, demonstrated.
print()
for maxlen in (16, 24):
r = [ROA("10.1.0.0/16", 64500, max_length=maxlen)]
state, _ = validate("10.1.128.0/24", 64500, r)
verdict = "SAFE: more-specifics are Invalid" if state == "Invalid" else \
"DANGEROUS: any /24 claiming AS64500 validates as Valid"
print(f"maxLength {maxlen}: a /24 inside the /16 is {state:<9} -> {verdict}")
# maxLength 16: a /24 inside the /16 is Invalid -> SAFE: more-specifics are Invalid
# maxLength 24: a /24 inside the /16 is Valid -> DANGEROUS: any /24 claiming AS64500 validates as Valid
# Longest prefix match is why the more-specific hijack works everywhere.
import ipaddress
def who_gets_traffic(dst, announcements):
d = ipaddress.ip_address(dst)
best = None
for prefix, asn in announcements:
n = ipaddress.ip_network(prefix)
if d in n and (best is None or n.prefixlen > best[0].prefixlen):
best = (n, asn)
return best
announcements = [("10.1.0.0/16", "AS64500 (legitimate)"),
("10.1.5.0/24", "AS64666 (HIJACKER)")]
print()
for dst in ("10.1.5.9", "10.1.9.9"):
net, asn = who_gets_traffic(dst, announcements)
print(f"traffic for {dst:<12} -> {asn:<24} (matched /{net.prefixlen})")
# traffic for 10.1.5.9 -> AS64666 (HIJACKER) (matched /24)
# traffic for 10.1.9.9 -> AS64500 (legitimate) (matched /16)
# The hijack captures only the /24 — but does so on EVERY router worldwide.
# RPKI's blind spot: a forged AS-PATH still validates.
def path_forgery_check(prefix, as_path, roas):
origin = as_path[-1] # RPKI checks the RIGHTMOST AS only
state, why = validate(prefix, origin, roas)
real_announcer = as_path[0]
return state, real_announcer, origin
print()
state, announcer, origin = path_forgery_check("10.1.0.0/16", [64666, 64500], roas)
print(f"AS-PATH [64666, 64500]: RPKI state = {state}")
print(f" RPKI validated the ORIGIN (AS{origin}) — which is correct.")
print(f" But AS{announcer} actually announced it. RPKI cannot see this.")
print(" Only BGPsec, signing every hop, would detect the forgery.")
# AS-PATH [64666, 64500]: RPKI state = Valid
# RPKI validated the ORIGIN (AS64500) — which is correct.
# But AS64666 actually announced it. RPKI cannot see this.
# Only BGPsec, signing every hop, would detect the forgery.
# The defence stack — no single layer is sufficient.
DEFENCES = {
"Prefix filters (IRR-derived)": (0.7, "stale and weakly authenticated data"),
"maximum-prefix limits": (0.4, "limits blast radius, not correctness"),
"RPKI ROV (drop Invalid)": (0.8, "ORIGIN only — forged paths still pass"),
"BGPsec (path signing)": (1.0, "negligible deployment; high CPU cost"),
"Monitoring (BGPStream/alerts)":(0.5, "detects after the fact, does not prevent"),
}
print()
print(f"{'defence':<32}{'coverage':>10} limitation")
for name, (cov, lim) in DEFENCES.items():
print(f"{name:<32}{cov:>9.0%} {lim}")
# Check your own prefixes' RPKI status:
$ whois -h whois.bgpmon.net " --roa 15169 8.8.8.0/24"
# 0 - Valid
# ROA Details
# Origin ASN: AS15169
# Prefix: 8.8.8.0/24
# Max Length: 24
# Run a validator (Routinator) and feed routers over RTR:
$ routinator server --rtr 0.0.0.0:3323 --http 0.0.0.0:8323
$ routinator vrps | head -3
# ASN,IP Prefix,Max Length,Trust Anchor
# AS13335,1.0.0.0/24,24,apnic
# AS15169,8.8.8.0/24,24,arin
# Point the router at the validator and enforce ROV:
# router bgp 64500
# rpki server tcp 10.0.0.50 port 3323 refresh 600
# bgp rpki
# !
# route-map RPKI_CHECK deny 10
# match rpki invalid <- DROP Invalid routes: this is enforcement
# route-map RPKI_CHECK permit 20
# match rpki notfound
# set local-preference 90 <- accept, but slightly deprefer unverified
# route-map RPKI_CHECK permit 30
# match rpki valid
# set local-preference 110
# neighbor 203.0.113.1 route-map RPKI_CHECK in
$ vtysh -c "show bgp rpki prefix-table" | head -4
# RPKI/RTR prefix table
# Prefix MaxLen Origin-AS
# 1.0.0.0/24 24 13335
# 8.8.8.0/24 24 15169
$ vtysh -c "show ip bgp 10.1.5.0/24" | grep -i rpki
# rpki validation-state: invalid
# ^ this route would be dropped by the route-map above
# Publish ROAs for YOUR prefixes via your RIR's portal, then verify:
$ curl -s "https://stat.ripe.net/data/rpki-validation/data.json?resource=AS64500&prefix=192.0.2.0/24" \
| python3 -c "import json,sys; d=json.load(sys.stdin)['data']; print(d['status'])"
# valid
# Monitor for hijacks of your own space — detection still matters:
# BGPStream / BGPalerter / RIPE RIS live feeds
$ bgpalerter # alerts on unexpected origins for your prefixes
# Filter over-specific prefixes — bounds hijack granularity:
# ip prefix-list SANE permit 0.0.0.0/0 ge 8 le 24
# ^^^^ nothing longer than /24
Key takeaways
- BGP has no built-in mechanism to verify that an AS may announce a prefix; any AS can announce anything.
- More-specific hijacking is the effective form: a /24 inside a /16 wins globally under longest prefix match, which no policy can override.
- Most DFZ routers reject prefixes longer than /24, which bounds the attack's granularity.
- Many incidents are misconfiguration, which is protocol-indistinguishable from an attack.
- IRR route objects allow filter generation but are often stale and weakly authenticated.
- RPKI binds prefixes to origin ASes with signed ROAs; ROV classifies routes Valid / Invalid / NotFound, and enforcement means dropping Invalid.
- Set
maxLengthequal to the prefix you actually announce — a loose maxLength re-enables the more-specific attack against yourself. - RPKI validates only the origin AS, not the path: a forged AS-PATH ending in the real origin validates as Valid.
- BGPsec signs every hop and would close that gap, but deployment is negligible.
- MANRS defines the operator baseline: filtering, anti-spoofing, coordination, and global validation.
🧪 Practice
- Given a ROA for
192.0.2.0/24, AS64500, maxLength 24, determine the validation state of that prefix from AS64500, from AS64666, and of192.0.2.128/25from AS64500.- Explain why announcing a /24 inside someone's /16 succeeds globally, naming the forwarding rule responsible.
- Explain why a ROA with
maxLength 24on a /16 is dangerous, and state the correct value.- Interview-style: A route passes RPKI validation as Valid but is still a hijack. How? (Hint: ask exactly which AS in the path RPKI examines, and whether anything prevents another AS from appending that one to a forged path.)
<a id="65-multicast-and-advanced-forwarding"></a>
6.5 Multicast and Advanced Forwarding
This closing subchapter covers forwarding that is not ordinary destination-based unicast: delivering one stream to many receivers, and forwarding on labels rather than addresses.
Multicast Addressing and Groups
Theory
Chapter 5 introduced multicast as a delivery model. Here the concern is routing: what has to change when the destination address identifies a group rather than an interface.
The economics are the motivation. Sending a 5 Mb/s video stream to 1,000 receivers by unicast requires 5 Gb/s at the source and 1,000 copies across every shared link. With multicast the source sends one copy, and routers duplicate packets only where the paths to receivers diverge. Each link carries one copy regardless of how many receivers lie beyond it.
Three things distinguish multicast routing from unicast routing, and each forces a design change.
The destination address is not a location. 239.1.1.1 identifies a group, not a host. Nothing in it says where receivers are, so a router cannot look it up in a table built from topology.
Receivers must be discovered, not computed. Hosts join groups, and routers must learn which of their links have interested receivers. That is what IGMP and MLD do (next topic).
Forwarding is source-rooted and reverse. Multicast builds a distribution tree and forwards away from the source, checking each packet against the path back toward it — Reverse Path Forwarding (RPF). A packet arriving on an interface that is not the one the router would use to reach the source is dropped. Since a multicast packet is replicated, an ordinary forwarding loop would multiply packets without bound; RPF is what makes that impossible.
The IPv4 multicast address space is 224.0.0.0/4, subdivided by scope:
| Range | Scope | Use |
|---|---|---|
224.0.0.0/24 | Link-local, never forwarded | 224.0.0.1 all hosts, 224.0.0.2 all routers, 224.0.0.5/6 OSPF, 224.0.0.9 RIPv2, 224.0.0.18 VRRP |
224.0.1.0 – 238.255.255.255 | Globally scoped | Internet-wide applications |
232.0.0.0/8 | SSM | Source-Specific Multicast |
233.0.0.0/8 | GLOP | ASN-derived allocation |
239.0.0.0/8 | Administratively scoped | Private — the multicast analogue of RFC 1918 |
Note that TTL 1 is not what protects 224.0.0.0/24 — routers are required never to forward that range regardless of TTL.
Mapping to Ethernet introduces a subtlety worth knowing. An IPv4 multicast group maps to a MAC address of 01:00:5e plus the low 23 bits of the group address. But the group has 28 significant bits (32 minus the 1110 prefix), so 5 bits are lost and 32 groups collapse onto one MAC address. A host joined to one group receives frames for 31 others and must discard them in software. This matters on segments with many groups. IPv6 avoids it by using 33:33 plus the low 32 bits.
SSM (Source-Specific Multicast) deserves emphasis because it is the modern default. Traditional ASM (Any-Source Multicast) lets any host send to a group, which requires rendezvous infrastructure to connect unknown sources to receivers, and allows anyone to inject traffic into a group. SSM has receivers specify the (source, group) pair — written (S,G) — so the tree is built directly toward a known source. No rendezvous point is needed, and unwanted sources cannot inject. For one-to-many applications such as IPTV and market data, SSM is simpler and safer, and it is what new deployments should use.
Examples
Why multicast scales: copies are made where paths diverge
UNICAST to 4 receivers: MULTICAST to 4 receivers:
[source] [source]
|||| 4 copies on the | 1 copy
|||| first link |
[R1] [R1]
/||\ / \ duplicated HERE
/ || \ / \ (paths diverge)
[A][B][C][D] [R2] [R3]
/ \ / \
source link: 4 x 5 Mb/s = 20 Mb/s [A] [B] [C] [D]
source link: 1 x 5 Mb/s = 5 Mb/s
EVERY link: exactly one copy
Reverse Path Forwarding — the loop preventer
source S is reachable via interface eth0 (per the UNICAST table).
multicast packet from S arrives on eth0 -> RPF PASS -> replicate out
all other
outgoing interfaces
multicast packet from S arrives on eth1 -> RPF FAIL -> DROP
Without RPF, a loop would not merely circulate a packet — it would
REPLICATE it at every hop, multiplying without bound until links died.
The 32:1 MAC overlap (IPv4 only)
group 224.1.1.1 = 1110 0000 0000 0001 0000 0001 0000 0001
\__/ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| low 23 bits -> the MAC
5 bits DISCARDED
224.1.1.1 -> 01:00:5e:01:01:01
225.1.1.1 -> 01:00:5e:01:01:01 <- SAME MAC
239.129.1.1 -> 01:00:5e:01:01:01 <- SAME MAC
... 32 groups share each MAC address.
A host joined to 224.1.1.1 receives frames for all 32 and discards
the unwanted 31 in software.
import ipaddress
MULTICAST_RANGES = [
("224.0.0.0/24", "link-local control — NEVER forwarded by routers"),
("232.0.0.0/8", "SSM (Source-Specific Multicast)"),
("233.0.0.0/8", "GLOP (ASN-derived)"),
("239.0.0.0/8", "administratively scoped (private, like RFC 1918)"),
("224.0.0.0/4", "globally scoped multicast"),
]
WELL_KNOWN = {
"224.0.0.1": "all hosts on this subnet",
"224.0.0.2": "all routers on this subnet",
"224.0.0.5": "OSPF AllSPFRouters",
"224.0.0.6": "OSPF AllDRouters",
"224.0.0.9": "RIPv2",
"224.0.0.13": "PIM",
"224.0.0.18": "VRRP",
"224.0.0.251": "mDNS",
}
def classify(group):
g = ipaddress.ip_address(group)
if not g.is_multicast:
return "NOT a multicast address"
for cidr, name in MULTICAST_RANGES:
if g in ipaddress.ip_network(cidr):
extra = f" [{WELL_KNOWN[group]}]" if group in WELL_KNOWN else ""
return name + extra
return "unclassified"
for g in ["224.0.0.5", "224.0.0.251", "232.1.2.3", "239.192.0.1", "225.1.1.1"]:
print(f"{g:<16}{classify(g)}")
# 224.0.0.5 link-local control — NEVER forwarded by routers [OSPF AllSPFRouters]
# 224.0.0.251 link-local control — NEVER forwarded by routers [mDNS]
# 232.1.2.3 SSM (Source-Specific Multicast)
# 239.192.0.1 administratively scoped (private, like RFC 1918)
# 225.1.1.1 globally scoped multicast
# The 32:1 MAC collision.
def multicast_mac(group):
g = int(ipaddress.ip_address(group))
low23 = g & 0x7FFFFF # only 23 bits survive
return "01:00:5e:" + ":".join(f"{b:02x}" for b in low23.to_bytes(3, "big"))
print()
for g in ["224.1.1.1", "225.1.1.1", "239.129.1.1", "224.129.1.1"]:
print(f"{g:<16} -> {multicast_mac(g)}")
print("all four map to the SAME MAC: 5 of the 28 significant bits are discarded.")
# 224.1.1.1 -> 01:00:5e:01:01:01
# 225.1.1.1 -> 01:00:5e:01:01:01
# 239.129.1.1 -> 01:00:5e:01:01:01
# 224.129.1.1 -> 01:00:5e:01:01:01
# all four map to the SAME MAC: 5 of the 28 significant bits are discarded.
print(f"\ngroups sharing each MAC: 2^5 = {2**5}")
print(f"IPv6 uses 33:33 + the low 32 bits -> no collision at all")
# Bandwidth: unicast versus multicast, at the source and per link.
def stream_cost(receivers, mbps, receivers_per_leaf=25):
leaves = -(-receivers // receivers_per_leaf)
return {
"unicast_source": receivers * mbps,
"multicast_source": mbps,
"unicast_core_link": receivers * mbps,
"multicast_core_link": mbps,
"multicast_total_copies": leaves,
}
print()
print(f"{'receivers':>10}{'unicast src':>14}{'mcast src':>12}{'ratio':>10}")
for n in (10, 100, 1_000, 10_000):
c = stream_cost(n, 5)
print(f"{n:>10,}{c['unicast_source']:>11,} Mb/s{c['multicast_source']:>9} Mb/s"
f"{c['unicast_source']//c['multicast_source']:>9,}x")
# receivers unicast src mcast src ratio
# 10 50 Mb/s 5 Mb/s 10x
# 100 500 Mb/s 5 Mb/s 100x
# 1,000 5,000 Mb/s 5 Mb/s 1,000x
# 10,000 50,000 Mb/s 5 Mb/s 10,000x
# Reverse Path Forwarding.
def rpf_check(source, arrived_on, unicast_table):
expected = unicast_table.get(source)
if expected is None:
return False, f"DROP: no unicast route to source {source}"
if arrived_on != expected:
return False, (f"DROP: arrived on {arrived_on}, but the route to {source} "
f"is via {expected} — RPF FAIL")
return True, f"PASS: arrived on {expected}, the RPF interface — replicate onward"
unicast = {"10.1.1.5": "eth0", "10.2.2.9": "eth1"}
print()
for src, iface in [("10.1.1.5", "eth0"), ("10.1.1.5", "eth1"), ("10.9.9.9", "eth0")]:
ok, why = rpf_check(src, iface, unicast)
print(f"source {src:<12} on {iface:<6} -> {why}")
# source 10.1.1.5 on eth0 -> PASS: arrived on eth0, the RPF interface — replicate onward
# source 10.1.1.5 on eth1 -> DROP: arrived on eth1, but the route to 10.1.1.5 is via eth0 — RPF FAIL
# source 10.9.9.9 on eth0 -> DROP: no unicast route to source 10.9.9.9
# ASM vs SSM.
print()
for model, needs_rp, who_can_send, join in [
("ASM (*,G)", True, "ANY host — injection is possible", "group only"),
("SSM (S,G)", False, "only the specified source", "source AND group"),
]:
print(f"{model:<12} rendezvous point: {str(needs_rp):<6} senders: {who_can_send:<34} "
f"receiver specifies: {join}")
# ASM (*,G) rendezvous point: True senders: ANY host — injection is possible receiver specifies: group only
# SSM (S,G) rendezvous point: False senders: only the specified source receiver specifies: source AND group
# Groups this host has joined:
$ ip maddr show eth0
# 2: eth0
# inet 224.0.0.1 <- all-hosts, always joined
# inet 224.0.0.251 <- mDNS (avahi)
# inet 239.255.255.250 <- SSDP/UPnP discovery
# inet6 ff02::1
# Multicast routing state — note the RPF interface on every entry:
$ ip mroute show
# (10.1.1.5, 239.1.1.1) Iif: eth0 Oifs: eth1 eth2
# ^^^^^^^^^^^^^^^^^^^^ ^^^^ ^^^^^^^^^
# the (S,G) pair RPF iface outgoing: one copy per interface
$ vtysh -c "show ip mroute"
# Source Group Proto Input Output TTL Uptime
# 10.1.1.5 239.1.1.1 PIM eth0 eth1 1 00:14:22
# ^^^^ must match the unicast route
# to 10.1.1.5, or packets are dropped
# Diagnosing an RPF failure — the most common multicast fault:
$ vtysh -c "show ip rpf 10.1.1.5"
# RPF information for 10.1.1.5
# RPF interface: eth0
# RPF neighbor: 10.0.0.2
# RPF route/mask: 10.1.1.0/24
# RPF type: unicast (ospf)
# ^^^^^^^ multicast forwarding depends on the UNICAST table.
# Asymmetric unicast routing therefore BREAKS multicast.
$ vtysh -c "show ip mroute count" | grep -i rpf
# RPF failed: 18241 packets
# ^^^^^ traffic is arriving on the wrong interface
# Send and receive a test stream:
$ iperf -s -u -B 239.1.1.1 -i 1 # receiver joins the group
$ iperf -c 239.1.1.1 -u -T 32 -t 30 -i 1 # sender; -T sets TTL
# ^^^^^ TTL must exceed the hop count, or the
# stream dies before reaching receivers
# The MAC collision, observed:
$ python3 -c "
import ipaddress
for g in ['224.1.1.1','225.1.1.1','239.129.1.1']:
v=int(ipaddress.ip_address(g))&0x7fffff
print(g,'-> 01:00:5e:'+':'.join(f'{b:02x}' for b in v.to_bytes(3,'big')))"
# 224.1.1.1 -> 01:00:5e:01:01:01
# 225.1.1.1 -> 01:00:5e:01:01:01
# 239.129.1.1 -> 01:00:5e:01:01:01
# Enable SSM — the recommended model for new deployments:
# ip pim ssm range 232.0.0.0/8
# ! receivers join (S,G) directly; no rendezvous point required
Key takeaways
- Multicast sends one copy from the source; routers replicate only where paths diverge, so each link carries one copy regardless of receiver count.
- The destination address names a group, not a location, so receivers must be discovered (IGMP/MLD) rather than computed.
- Reverse Path Forwarding (RPF) drops any packet not arriving on the interface used to reach the source — essential because a multicast loop would replicate, not merely circulate.
- Multicast forwarding depends on the unicast routing table, so asymmetric unicast routing breaks multicast.
- Key ranges:
224.0.0.0/24link-local (never forwarded, whatever the TTL),232.0.0.0/8SSM,239.0.0.0/8administratively scoped (private). - IPv4 maps a group to
01:00:5e+ the low 23 bits, discarding 5 of the 28 significant bits, so 32 groups share one MAC; IPv6 uses33:33+ 32 bits and avoids this. - ASM (*,G) allows any sender and needs rendezvous infrastructure; SSM (S,G) specifies the source, needs no RP, and prevents unwanted injection — the right default for new deployments.
🧪 Practice
- Compute the Ethernet MAC for
239.10.20.30and name one other group that maps to the same MAC.- Explain why RPF is essential for multicast when unicast forwarding gets by with TTL alone.
- Compute source bandwidth for a 12 Mb/s stream to 2,500 receivers under unicast and multicast.
- Interview-style: Multicast works in one direction but fails after a unicast routing change, with no multicast configuration altered. Explain. (Hint: ask which table the RPF check consults, and what happens when the path to the source moves to a different interface.)
IGMP and MLD
Theory
A router replicating multicast onto a link must know whether anyone there actually wants the traffic. Without that knowledge it would either flood every group everywhere or deliver nothing. IGMP (Internet Group Management Protocol) is the signalling between hosts and their local router for IPv4; MLD (Multicast Listener Discovery) is the IPv6 equivalent, carried inside ICMPv6.
The division of labour is worth stating precisely, because it is a common confusion: IGMP/MLD operate on one link only. They tell the local router which groups have listeners on that segment. Building the tree between routers is a different protocol's job — PIM, covered next.
The mechanics are simple: hosts send Membership Reports to join, the router sends periodic Queries to check whether anyone is still interested, and hosts respond. If no report arrives for a group within the timeout, the router stops forwarding it.
The versions differ in ways that matter operationally:
| Version | Added |
|---|---|
| IGMPv1 | Join and query only — no leave message, so departure waited for a timeout (up to ~3 minutes) |
| IGMPv2 | Leave Group messages and group-specific queries — fast leave |
| IGMPv3 | Source filtering: INCLUDE/EXCLUDE source lists — required for SSM |
| MLDv1 | IPv6 equivalent of IGMPv2 |
| MLDv2 | IPv6 equivalent of IGMPv3 |
Two mechanisms in the protocol design repay attention.
Report suppression. When a router queries a group, hosts start a random timer. If a host hears another host's report for the same group first, it cancels its own — the router only needs to know that at least one listener exists, not how many. On a segment with 500 receivers this turns 500 reports into roughly one. (IGMPv3 removes suppression, because per-source state requires each host's full picture.)
The querier election. If several routers share a segment, the one with the lowest IP address becomes the querier; the rest stay silent. This prevents duplicate queries.
The complementary switch feature is IGMP snooping, and its absence is the single most common cause of multicast problems on a LAN. A switch is a layer 2 device; a multicast frame has a multicast destination MAC that is in no MAC address table, so the switch floods it to every port — turning multicast into broadcast and defeating the entire point. IGMP snooping has the switch listen to IGMP messages passing through and build a map of which ports have listeners for which groups, forwarding only there.
Snooping needs a querier to function, because it learns from the report traffic that queries provoke. If no multicast router exists on the segment, the switch must run an IGMP querier itself — otherwise reports stop, snooping state times out, and the switch reverts to flooding. A high-rate stream (IPTV, market data) flooded to every port can saturate a segment.
Examples
Who does what: IGMP is local, PIM is between routers
[host]---[host]---[host] [router]======[router]======[source]
\ | / |
\ | / |
+--[switch]--+ |
| |
+--- IGMP ------------>+
^ hosts <-> local router, ONE LINK ONLY
|
+--- PIM ------------> upstream
^ between routers, builds the tree
IGMP exchange on a segment
router --- Query (224.0.0.1, "any listeners?") -----> all hosts
hostA --- Report (239.1.1.1) -----------------------> router
hostB --- (hears A's report, CANCELS its own) ------- (silent)
hostC --- (hears A's report, CANCELS its own) ------- (silent)
^ REPORT SUPPRESSION: the router needs to know
"someone wants it", not "how many".
hostA --- Leave Group (239.1.1.1) ------------------> router [v2+]
router --- Group-Specific Query ---------------------> "anyone else?"
...silence... -> stop forwarding 239.1.1.1 to this link
Without IGMP snooping: multicast becomes broadcast
WITHOUT snooping: WITH snooping:
[switch] [switch]
/ | | \ / | | \
A B C D A B C D
(all four receive) (only A and C joined; only they receive)
A 100 Mb/s IPTV stream flooded to 48 ports = 4.8 Gb/s of wasted
switching capacity, and every host's NIC interrupting on frames
it must discard in software.
IGMP_VERSIONS = {
1: ("join, query", "NO leave message — departure waits for a ~3 min timeout"),
2: ("join, query, LEAVE", "fast leave via group-specific query"),
3: ("join, query, leave, SOURCE FILTERING", "INCLUDE/EXCLUDE lists — REQUIRED for SSM"),
}
for v, (features, note) in IGMP_VERSIONS.items():
print(f"IGMPv{v}: {features:<42}{note}")
# IGMPv1: join, query NO leave message — departure waits for a ~3 min timeout
# IGMPv2: join, query, LEAVE fast leave via group-specific query
# IGMPv3: join, query, leave, SOURCE FILTERING INCLUDE/EXCLUDE lists — REQUIRED for SSM
class Segment:
"""One link: hosts, a querier, and the router's forwarding decision."""
def __init__(self, query_interval=125, timeout=260):
self.memberships = {} # group -> {host: last_report_time}
self.query_interval = query_interval
self.timeout = timeout
def report(self, host, group, now):
self.memberships.setdefault(group, {})[host] = now
def leave(self, host, group, now):
"""IGMPv2+: explicit leave, then a group-specific query."""
if group in self.memberships:
self.memberships[group].pop(host, None)
if not self.memberships[group]:
del self.memberships[group]
return f"{group}: last listener left -> STOP forwarding to this link"
return f"{group}: {len(self.memberships[group])} listener(s) remain"
return f"{group}: no state"
def expire(self, now):
stopped = []
for group in list(self.memberships):
live = {h: t for h, t in self.memberships[group].items()
if now - t < self.timeout}
if live:
self.memberships[group] = live
else:
del self.memberships[group]
stopped.append(group)
return stopped
def forwarding(self):
return sorted(self.memberships)
seg = Segment()
seg.report("hostA", "239.1.1.1", now=0)
seg.report("hostB", "239.1.1.1", now=0)
seg.report("hostC", "239.2.2.2", now=0)
print(f"\nforwarding to this link: {seg.forwarding()}")
print(seg.leave("hostA", "239.1.1.1", now=10))
print(seg.leave("hostB", "239.1.1.1", now=20))
print(f"forwarding now: {seg.forwarding()}")
# forwarding to this link: ['239.1.1.1', '239.2.2.2']
# 239.1.1.1: 1 listener(s) remain
# 239.1.1.1: last listener left -> STOP forwarding to this link
# forwarding now: ['239.2.2.2']
# Report suppression: many listeners, one report.
def reports_sent(listeners, suppression):
return 1 if suppression else listeners
print()
for n in (5, 50, 500):
for supp in (True, False):
label = "with suppression" if supp else "without (IGMPv3)"
print(f"{n:>4} listeners, {label:<20} -> {reports_sent(n, supp):>4} report(s) per query")
# 5 listeners, with suppression -> 1 report(s) per query
# 5 listeners, without (IGMPv3) -> 5 report(s) per query
# 50 listeners, with suppression -> 1 report(s) per query
# 50 listeners, without (IGMPv3) -> 50 report(s) per query
# 500 listeners, with suppression -> 1 report(s) per query
# 500 listeners, without (IGMPv3) -> 500 report(s) per query
# Querier election: lowest IP wins.
def elect_querier(routers):
return min(routers, key=lambda ip: [int(o) for o in ip.split(".")])
print(f"\nquerier among {['10.0.0.5','10.0.0.2','10.0.0.9']}: "
f"{elect_querier(['10.0.0.5','10.0.0.2','10.0.0.9'])} (lowest IP)")
# querier among ['10.0.0.5', '10.0.0.2', '10.0.0.9']: 10.0.0.2 (lowest IP)
# IGMP snooping: the difference between multicast and broadcast on a LAN.
def switch_forwarding(ports, joined_ports, snooping):
return sorted(ports) if not snooping else sorted(joined_ports)
ports = [f"Gi1/0/{i}" for i in range(1, 49)]
joined = ["Gi1/0/3", "Gi1/0/17"]
print()
for snoop in (False, True):
out = switch_forwarding(ports, joined, snoop)
label = "snooping ON" if snoop else "snooping OFF"
wasted = (len(out) - len(joined)) * 100
print(f"{label:<14} stream sent to {len(out):>2} port(s) "
f"wasted switching capacity: {wasted:>5,} Mb/s")
# snooping OFF stream sent to 48 port(s) wasted switching capacity: 4,600 Mb/s
# snooping ON stream sent to 2 port(s) wasted switching capacity: 0 Mb/s
# IGMPv3 source filtering — what makes SSM possible.
def igmpv3_accept(group, source, filter_mode, source_list):
if filter_mode == "INCLUDE":
return source in source_list, f"INCLUDE {source_list}"
return source not in source_list, f"EXCLUDE {source_list}"
print()
for mode, lst, src in [("INCLUDE", ["10.1.1.5"], "10.1.1.5"),
("INCLUDE", ["10.1.1.5"], "10.9.9.9"),
("EXCLUDE", ["10.9.9.9"], "10.1.1.5")]:
ok, desc = igmpv3_accept("232.1.1.1", src, mode, lst)
print(f"{desc:<26} source {src:<12} -> {'ACCEPT' if ok else 'REJECT'}")
# INCLUDE ['10.1.1.5'] source 10.1.1.5 -> ACCEPT
# INCLUDE ['10.1.1.5'] source 10.9.9.9 -> REJECT
# EXCLUDE ['10.9.9.9'] source 10.1.1.5 -> ACCEPT
# Rejecting unwanted sources at the receiver is exactly what SSM needs.
# Group membership as the router sees it:
$ vtysh -c "show ip igmp groups"
# Interface Address Group Mode Timer Srcs V Uptime
# eth1 10.0.1.1 239.1.1.1 EXCL 00:03:41 0 3 00:22:14
# ^^^^ IGMPv3 filter mode
$ ip maddr show eth0 | head -4
# The IGMP exchange on the wire:
$ sudo tcpdump -i eth0 -n igmp
# IP 10.0.1.1 > 224.0.0.1: igmp query v3 <- router's query
# IP 10.0.1.50 > 224.0.0.22: igmp v3 report, 1 group record(s)
# ^^^^^^^^^^ IGMPv3 reports go to 224.0.0.22, not the group
# IP 10.0.1.51 > 224.0.0.2: igmp leave 239.1.1.1 <- IGMPv2 leave
# IGMP snooping on the switch — verify it is ON and learning:
$ show ip igmp snooping
# Global IGMP Snooping configuration:
# IGMP snooping : Enabled
# IGMPv3 snooping : Enabled
# Report suppression : Enabled
$ show ip igmp snooping groups
# Vlan Group Type Version Port List
# 10 239.1.1.1 igmp v3 Gi1/0/3, Gi1/0/17
# ^^^^^^^^^^^^^^^^^ ONLY these two ports
# receive the stream
# The classic failure: snooping on, but no querier anywhere.
$ show ip igmp snooping querier
# Vlan IP Address IGMP Version Port
# 10 -- -- --
# ^^ NO QUERIER. Snooping state will time out and the switch
# reverts to FLOODING the stream to all 48 ports.
# FIX — let the switch be the querier when no mrouter exists:
# ip igmp snooping querier
# ip igmp snooping vlan 10 querier address 10.0.10.1
# Force a version (v3 is required for SSM):
# interface GigabitEthernet0/1
# ip igmp version 3
# ipv6 equivalent:
# ipv6 mld version 2
# Join a group from a host to test:
$ python3 -c "
import socket, struct
s=socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
mreq=struct.pack('4sl', socket.inet_aton('239.1.1.1'), socket.INADDR_ANY)
s.setsockopt(socket.IPPROTO_IP, socket.IP_ADD_MEMBERSHIP, mreq)
print('joined 239.1.1.1'); input()"
Key takeaways
- IGMP (IPv4) and MLD (IPv6, inside ICMPv6) signal group membership between hosts and their local router, on one link only.
- Building the tree between routers is PIM's job, not IGMP's.
- IGMPv1 had no leave message; IGMPv2 added Leave Group and group-specific queries; IGMPv3 added source filtering and is required for SSM.
- Report suppression means one report answers a query on behalf of every listener — the router only needs to know that at least one exists.
- The querier is the router with the lowest IP address on the segment.
- IGMP snooping lets a switch read IGMP messages and forward a group only to ports with listeners; without it, multicast is flooded like broadcast.
- Snooping requires a querier to keep its state fresh — with none present, state times out and the switch reverts to flooding, which a high-rate stream can turn into a segment-wide outage.
- IGMPv3's INCLUDE/EXCLUDE source lists let a receiver accept traffic only from specified sources.
🧪 Practice
- State the feature each IGMP version added and which version SSM requires.
- Explain report suppression and compute the reports generated per query for 300 listeners with and without it.
- A 200 Mb/s stream saturates a 48-port switch even though only two hosts joined. Diagnose the cause and give two configuration fixes.
- Interview-style: IGMP snooping is enabled and multicast still floods. What would you check first? (Hint: snooping learns from the reports that something must solicit — ask what happens to that state when nothing is asking.)
PIM Sparse and Dense Mode
Theory
IGMP tells a router which groups are wanted on its own links. PIM (Protocol Independent Multicast) builds the distribution tree between routers, connecting sources to receivers across the network.
The "protocol independent" in the name is precise and important: PIM has no routing protocol of its own. It relies entirely on whatever unicast protocol is already running — OSPF, IS-IS, BGP, or static routes — to perform its RPF checks. This is why multicast breaks when unicast routing changes, and it is why PIM works identically regardless of which IGP you run.
Two fundamentally different approaches exist, and choosing correctly is the main design decision.
PIM Dense Mode (PIM-DM) uses flood and prune. It assumes receivers exist nearly everywhere: flood the traffic to the entire network, then let routers with no interested receivers send Prune messages to stop it. Pruned branches time out roughly every 3 minutes and are re-flooded, so the cycle repeats indefinitely.
This is only sensible when receivers really are dense. On any realistic network it wastes enormous bandwidth on periodic re-flooding, and PIM-DM is effectively obsolete — it appears in older documentation and small lab networks only.
PIM Sparse Mode (PIM-SM) uses explicit join, and it is what networks actually deploy. Nothing is forwarded until a router explicitly asks. The mechanism needs a meeting point, because receivers do not know where sources are and sources do not know where receivers are.
That meeting point is the RP (Rendezvous Point), a designated router known to every PIM router in the domain. The sequence:
- A receiver joins a group. Its local router sends a (*,G) Join toward the RP, building a branch of the shared tree (also called the RP tree, or RPT).
- A source starts sending. Its local router encapsulates the traffic in PIM Register messages and unicasts them to the RP.
- The RP forwards the traffic down the shared tree to receivers, and sends a (S,G) Join back toward the source so traffic can flow natively.
- Once the receiver's router sees traffic and learns the source's address, it may initiate an SPT switchover: send an (S,G) Join directly toward the source, building a shortest-path tree, then prune itself off the shared tree.
The switchover matters because the shared tree is rarely optimal — traffic detours via the RP, which may be far from both source and receiver. The SPT is the direct path. Cisco switches over after the first packet by default; the threshold is configurable, and setting it to infinity keeps traffic on the shared tree permanently (useful for very many low-rate groups, where per-(S,G) state would be expensive).
Notation to keep straight:
| Notation | Meaning |
|---|---|
| (*,G) | Shared tree — traffic from any source to group G, via the RP |
| (S,G) | Source tree — traffic from source S to group G, direct |
RP placement and redundancy is the main operational concern in PIM-SM, since the RP is a single point of failure for group setup. Options: static RP (simple, must be configured identically everywhere), Auto-RP (Cisco), BSR (Bootstrap Router) (standards-based), and Anycast RP — the best modern answer, where several routers share the same RP address announced by the IGP, with MSDP synchronising source information between them.
PIM-SSM removes the RP entirely. Because the receiver specifies the source via IGMPv3, the router sends an (S,G) Join directly toward it. No RP, no register process, no shared tree, no MSDP. For one-to-many applications, SSM is simpler, faster, and more secure — and is the recommended model for new deployments.
Examples
Dense mode: flood everywhere, then prune, then do it all again
t=0 source sends -> FLOODED to every router in the network
t=0.1 routers with no receivers send PRUNE
t=0.2 traffic now flows only where wanted
t=180 prunes EXPIRE -> FLOOD EVERYTHING AGAIN
t=180.1 prune again...
^^^^^^^^^^^^^^^ every 3 minutes, forever. Obsolete for a reason.
Sparse mode: nothing flows until someone asks
step 1: receiver joins
[receiver]--IGMP-->[R3] --(*,G) Join--> [RP]
^ shared tree branch built
step 2: source starts
[source]-->[R1] --PIM Register (unicast, encapsulated)--> [RP]
step 3: RP pulls the traffic natively and feeds the shared tree
[RP] --(S,G) Join--> [R1] traffic: source -> R1 -> RP -> R3 -> receiver
^^^^^^^^ detour via RP
step 4: SPT switchover — R3 learns S and cuts the corner
[R3] --(S,G) Join--> ... --> [R1]
[R3] --(S,G) Prune toward RP-->
traffic: source -> R1 -> R3 -> receiver DIRECT, optimal
Shared tree vs source tree
(*,G) shared tree (S,G) source tree
[RP] [source]
/ \ |
[R2] [R3] [R1]
| |
[receiver] [R3]
|
path: S->R1->RP->R3->rcvr [receiver]
(may be a long detour) path: S->R1->R3->rcvr
(shortest, but more state:
one entry PER SOURCE)
class PIMRouter:
def __init__(self, name, rp, unicast_next_hop):
self.name, self.rp = name, rp
self.unicast = unicast_next_hop # dest -> next hop (from the IGP)
self.mroute = {} # (source_or_*, group) -> state
def receiver_joins(self, group):
"""(*,G): build a branch of the SHARED tree toward the RP."""
nh = self.unicast.get(self.rp, "UNREACHABLE")
self.mroute[("*", group)] = {"tree": "shared (RPT)", "toward": self.rp,
"upstream": nh}
return f"(*,{group}) Join sent toward RP {self.rp} via {nh}"
def spt_switchover(self, source, group):
"""(S,G): cut the corner — join the source directly, prune off the RPT."""
nh = self.unicast.get(source, "UNREACHABLE")
self.mroute[(source, group)] = {"tree": "shortest path (SPT)",
"toward": source, "upstream": nh}
self.mroute.pop(("*", group), None)
return (f"({source},{group}) Join sent toward the SOURCE via {nh}; "
f"pruned off the shared tree")
def show(self):
for (s, g), st in self.mroute.items():
print(f" ({s},{g}) {st['tree']:<20} upstream {st['upstream']}")
r3 = PIMRouter("R3", rp="10.255.0.100",
unicast_next_hop={"10.255.0.100": "10.0.0.2", "10.1.1.5": "10.0.0.6"})
print(r3.receiver_joins("239.1.1.1")); r3.show()
print()
print(r3.spt_switchover("10.1.1.5", "239.1.1.1")); r3.show()
# (*,239.1.1.1) Join sent toward RP 10.255.0.100 via 10.0.0.2
# (*,239.1.1.1) shared (RPT) upstream 10.0.0.2
#
# (10.1.1.5,239.1.1.1) Join sent toward the SOURCE via 10.0.0.6; pruned off the shared tree
# (10.1.1.5,239.1.1.1) shortest path (SPT) upstream 10.0.0.6
# Dense vs sparse: the bandwidth difference.
def pim_traffic(mode, routers, receivers, mbps, reflood_interval=180, window=3600):
if mode == "dense":
refloods = window // reflood_interval
flooded_mb = refloods * routers * mbps * 3 # ~3 s of flooding each
return {"floods_per_hour": refloods,
"wasted_mb_per_hour": flooded_mb - receivers * mbps * 3 * refloods}
return {"floods_per_hour": 0, "wasted_mb_per_hour": 0}
print()
for mode in ("dense", "sparse"):
t = pim_traffic(mode, routers=200, receivers=5, mbps=10)
print(f"PIM-{mode:<7} floods/hour: {t['floods_per_hour']:>2} "
f"wasted: {t['wasted_mb_per_hour']:>8,} MB/hour")
# PIM-dense floods/hour: 20 wasted: 117,000 MB/hour
# PIM-sparse floods/hour: 0 wasted: 0 MB/hour
# 200 routers, 5 actual receivers: dense mode floods the whole network
# 20 times an hour to reach five hosts.
# Shared tree vs SPT: path cost and state cost.
def tree_comparison(src_to_rp, rp_to_rcvr, src_to_rcvr, sources):
return {
"shared_tree_hops": src_to_rp + rp_to_rcvr,
"spt_hops": src_to_rcvr,
"shared_tree_state_entries": 1, # one (*,G)
"spt_state_entries": sources, # one (S,G) per source
}
print()
c = tree_comparison(src_to_rp=6, rp_to_rcvr=5, src_to_rcvr=3, sources=50)
for k, v in c.items():
print(f" {k:<28}{v}")
print(" SPT is shorter but costs one state entry PER SOURCE — which is why")
print(" a high SPT threshold is used when there are very many low-rate groups.")
# shared_tree_hops 11
# spt_hops 3
# shared_tree_state_entries 1
# spt_state_entries 50
# RP options, by robustness.
RP_OPTIONS = {
"Static RP": (0.3, "simple; must be configured identically everywhere; SPOF"),
"Auto-RP": (0.6, "Cisco-proprietary; RP announced via 224.0.1.39/40"),
"BSR": (0.7, "standards-based RP election"),
"Anycast RP": (1.0, "several routers share one RP address; MSDP syncs sources"),
}
print()
for name, (score, note) in sorted(RP_OPTIONS.items(), key=lambda kv: kv[1][0]):
print(f"{name:<12}{score:>6.0%} {note}")
# SSM removes the RP problem entirely.
print()
for model in ("ASM (PIM-SM)", "SSM (PIM-SSM)"):
if model.startswith("ASM"):
print(f"{model:<16} needs: RP, Register process, shared tree, MSDP for redundancy")
else:
print(f"{model:<16} needs: NOTHING but IGMPv3 — (S,G) Join straight to the source")
# ASM (PIM-SM) needs: RP, Register process, shared tree, MSDP for redundancy
# SSM (PIM-SSM) needs: NOTHING but IGMPv3 — (S,G) Join straight to the source
# Enable PIM sparse mode (dense mode should not be used):
# ip multicast-routing
# interface GigabitEthernet0/1
# ip pim sparse-mode
# ip igmp version 3
# ip pim rp-address 10.255.0.100
# ^^^^^^^^^^^^^^^^^^^ must be identical on EVERY PIM router
$ vtysh -c "show ip pim rp-info"
# RP address group/prefix-list OIF I am RP
# 10.255.0.100 224.0.0.0/4 eth1 no
# Multicast routing state — watch (*,G) become (S,G):
$ vtysh -c "show ip mroute"
# (*, 239.1.1.1) <- shared tree, toward the RP
# Incoming interface: eth1
# Outgoing interface list: eth2
# (10.1.1.5, 239.1.1.1) <- source tree after SPT switchover
# Incoming interface: eth3
# ^^^^^^^^^^^^^^^^^^ note: a DIFFERENT interface — the direct path
# Outgoing interface list: eth2
# Cisco flags tell the whole story:
$ show ip mroute 239.1.1.1
# (*, 239.1.1.1), 00:14:22/stopped, RP 10.255.0.100, flags: SJC
# ^^^ S=sparse,
# J=join SPT, C=connected
# (10.1.1.5, 239.1.1.1), 00:02:11/00:02:48, flags: JT
# ^^ T = using the SPT
# Incoming interface: GigabitEthernet0/3, RPF nbr 10.0.0.6
# Control the SPT switchover threshold:
# ip pim spt-threshold 0 <- switch after the first packet (default)
# ip pim spt-threshold infinity <- stay on the shared tree forever
# (saves state with many low-rate groups)
# Anycast RP — the robust design: same address on several routers.
# interface Loopback0
# ip address 10.255.0.100 255.255.255.255 <- identical on RP1 and RP2
# interface Loopback1
# ip address 10.255.0.1 255.255.255.255 <- unique, for MSDP
# ip msdp peer 10.255.0.2 connect-source Loopback1
# ip pim rp-address 10.255.0.100
$ show ip msdp summary
# MSDP Peer Status Summary
# Peer Address AS State Uptime/Downtime SA Count
# 10.255.0.2 64500 Up 2w4d 41
# SSM — no RP at all:
# ip pim ssm range 232.0.0.0/8
$ vtysh -c "show ip mroute" | grep 232
# (10.1.1.5, 232.1.1.1) <- straight to an (S,G); no shared tree existed
Key takeaways
- PIM builds the multicast tree between routers; it is protocol independent — it has no routing protocol and uses the existing unicast table for RPF.
- PIM Dense Mode floods and prunes, re-flooding every ~3 minutes; it is effectively obsolete.
- PIM Sparse Mode forwards nothing until an explicit join, using an RP (Rendezvous Point) as the meeting place for sources and receivers.
- Sequence: receiver sends (*,G) Join toward the RP; source's router Registers with the RP; RP joins toward the source; the receiver's router performs an SPT switchover to the direct path.
- (*,G) is the shared tree via the RP; (S,G) is the source tree, shorter but requiring one state entry per source.
- A high SPT threshold keeps traffic on the shared tree to save state when there are many low-rate groups.
- RP options in increasing robustness: static, Auto-RP, BSR, and Anycast RP with MSDP — the modern choice.
- PIM-SSM eliminates the RP, the Register process, the shared tree, and MSDP entirely, and is recommended for new one-to-many deployments.
🧪 Practice
- Contrast flood-and-prune with explicit-join, and state why dense mode is no longer deployed.
- Describe the four steps from a receiver joining to traffic flowing on the shortest-path tree.
- Explain what (*,G) and (S,G) mean, and give one advantage of each tree.
- Interview-style: Why does PIM-SSM need no Rendezvous Point? (Hint: the RP exists so receivers can find sources they do not know about — ask what IGMPv3 lets the receiver specify.)
MPLS Labels and Label Switched Paths
Theory
Every forwarding mechanism so far has examined the destination IP address and performed a longest-prefix lookup. MPLS (Multiprotocol Label Switching) replaces that with an exact-match lookup on a short fixed-length label.
The original motivation was speed: in the 1990s, longest-prefix matching in software was slow, while exact-match lookup on a 20-bit label was fast. Hardware advances eliminated that advantage entirely — modern ASICs do LPM at line rate. MPLS survived for completely different reasons, and this is the key point to understand: it enables traffic engineering, VPNs, and fast reroute, none of which destination-based IP forwarding can express.
An MPLS label is a 32-bit shim inserted between the layer 2 and layer 3 headers — hence "layer 2.5":
| Field | Bits | Purpose |
|---|---|---|
| Label | 20 | The forwarding identifier (values 0–15 reserved) |
| TC / EXP | 3 | Traffic class, for QoS |
| S | 1 | Bottom-of-stack flag |
| TTL | 8 | Loop prevention, as in IP |
The S bit enables label stacking — multiple labels on one packet, which is what makes MPLS VPNs possible: an outer label carries the packet across the backbone while an inner label identifies the customer.
The roles and operations:
| Role | Position | Operation |
|---|---|---|
| Ingress LER / PE | Edge, entering | PUSH a label based on an IP lookup |
| LSR / P | Core | SWAP the label — no IP lookup at all |
| Egress LER / PE | Edge, leaving | POP the label and forward by IP |
The critical property: core routers never look at the IP header. A provider's core can forward customer traffic without holding a single customer route — or even any internet routes. This is what "BGP-free core" means, and it is a major operational simplification.
An LSP (Label Switched Path) is the resulting path, and it is unidirectional — bidirectional traffic needs two LSPs.
PHP (Penultimate Hop Popping) is a small but universal optimisation. If the last router had to pop the label and then do an IP lookup, it would perform two lookups. Instead the second-to-last router pops the label (signalled by the reserved implicit-null label 3), so the egress router receives a plain IP packet and does one lookup.
Labels are distributed by a protocol:
| Protocol | Purpose |
|---|---|
| LDP | Distributes labels following the IGP's shortest path — simple, most common |
| RSVP-TE | Signals explicit paths with bandwidth reservation — traffic engineering |
| BGP-LU | Carries labels in BGP, for inter-AS and seamless MPLS |
| Segment Routing | Distributes labels in the IGP itself — no separate protocol at all |
The applications that keep MPLS deployed:
L3VPN (RFC 4364) is the dominant use. Customer routes live in per-customer VRFs, are carried across the backbone in MP-BGP with route distinguishers (making overlapping customer address space unique) and route targets (controlling which VRFs import which routes). Customers get private routing with overlapping RFC 1918 space, and the provider core stays unaware of them.
Traffic engineering lets an operator route a specific flow along a chosen path with reserved bandwidth — impossible with destination-based forwarding, which always follows the IGP's shortest path.
FRR (Fast Reroute) pre-computes backup LSPs, giving sub-50 ms failover in hardware.
Examples
The label shim: layer 2.5
+----------+----------------+----------+---------+
| Ethernet | MPLS label(s) | IP header| payload |
+----------+----------------+----------+---------+
\______________/
32 bits each:
| label (20) | TC (3) | S (1) | TTL (8) |
^ S=1 marks the LAST label
Stacked labels (an L3VPN packet):
+----------+--------+--------+----------+---------+
| Ethernet | L=1001 | L=2002 | IP | payload |
| | S=0 | S=1 | | |
+----------+--------+--------+----------+---------+
^outer: transport ^inner: which CUSTOMER VRF
swapped hop by hop untouched until the egress PE
An LSP end to end, with PHP
[CE]--IP-->[PE1]------[P1]------[P2]------[PE2]--IP-->[CE]
PUSH 100 SWAP SWAP receives
100->200 200->pop plain IP
^^^^^^^^
PENULTIMATE HOP POPS (label 3 =
implicit-null), so PE2 does ONE
lookup instead of two.
P1 and P2 never examine the IP header. They hold NO customer routes
and need no BGP at all -> "BGP-free core".
Why MPLS survived after hardware made LPM fast
speed: no longer an advantage (ASICs do LPM at line rate)
traffic eng: route a FLOW along a CHOSEN path -> impossible with IP
L3VPN: overlapping customer address space, isolated
fast reroute: pre-computed backups, <50 ms in hardware
BGP-free core: core routers hold no internet or customer routes
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
THESE are why it is still everywhere.
def encode_label(label, tc=0, s=0, ttl=255):
"""A 32-bit MPLS shim: label(20) | TC(3) | S(1) | TTL(8)."""
return (label << 12) | (tc << 9) | (s << 8) | ttl
def decode_label(word):
return {"label": word >> 12, "tc": (word >> 9) & 0b111,
"bottom_of_stack": (word >> 8) & 1, "ttl": word & 0xFF}
shim = encode_label(1001, tc=5, s=1, ttl=64)
print(f"encoded: 0x{shim:08x}")
print(f"decoded: {decode_label(shim)}")
# encoded: 0x003e9b40
# decoded: {'label': 1001, 'tc': 5, 'bottom_of_stack': 1, 'ttl': 64}
RESERVED = {0: "IPv4 explicit null", 2: "IPv6 explicit null",
3: "implicit null (triggers PHP)", 13: "GAL", 14: "OAM alert"}
print()
for v, meaning in RESERVED.items():
print(f"label {v:>2}: {meaning}")
print(f"usable label range: 16 .. {2**20 - 1:,}")
# label 0: IPv4 explicit null
# label 2: IPv6 explicit null
# label 3: implicit null (triggers PHP)
# label 13: GAL
# label 14: OAM alert
# usable label range: 16 .. 1,048,575
class LSR:
"""A label switching router. The core NEVER looks at the IP header."""
def __init__(self, name, role):
self.name, self.role = name, role
self.lfib = {} # incoming label -> (operation, outgoing label, iface)
self.fib = {} # prefix -> (label to push, iface) — ingress only
def program_swap(self, in_label, out_label, iface):
self.lfib[in_label] = ("SWAP", out_label, iface)
def program_pop(self, in_label, iface):
self.lfib[in_label] = ("POP", None, iface)
def program_push(self, prefix, label, iface):
self.fib[prefix] = (label, iface)
def forward(self, packet):
if packet["labels"]:
top = packet["labels"][0]
if top not in self.lfib:
return packet, f"{self.name}: no LFIB entry for label {top} -> DROP"
op, out, iface = self.lfib[top]
if op == "SWAP":
packet["labels"][0] = out
return packet, f"{self.name}: SWAP {top} -> {out}, out {iface} (no IP lookup)"
packet["labels"].pop(0)
return packet, f"{self.name}: POP {top}, out {iface} (PHP)"
# No label: this is the ingress. Do ONE IP lookup, then push.
label, iface = self.fib.get(packet["dst_prefix"], (None, None))
if label is None:
return packet, f"{self.name}: no route -> DROP"
packet["labels"].insert(0, label)
return packet, f"{self.name}: IP lookup, PUSH {label}, out {iface}"
pe1 = LSR("PE1", "ingress"); p1 = LSR("P1", "core")
p2 = LSR("P2", "core"); pe2 = LSR("PE2", "egress")
pe1.program_push("10.9.0.0/16", 100, "eth1")
p1.program_swap(100, 200, "eth2")
p2.program_pop(200, "eth3") # PHP: pop one hop early
packet = {"dst_prefix": "10.9.0.0/16", "labels": []}
print()
for router in (pe1, p1, p2):
packet, msg = router.forward(packet)
print(f" {msg:<58} labels now {packet['labels']}")
print(f" PE2: receives a plain IP packet -> ONE IP lookup -> deliver")
# PE1: IP lookup, PUSH 100, out eth1 labels now [100]
# P1: SWAP 100 -> 200, out eth2 (no IP lookup) labels now [200]
# P2: POP 200, out eth3 (PHP) labels now []
# PE2: receives a plain IP packet -> ONE IP lookup -> deliver
# L3VPN: route distinguishers make overlapping customer space unique.
def vpn_prefix(rd, prefix):
return f"{rd}:{prefix}"
print()
customers = [("64500:1", "CustomerA"), ("64500:2", "CustomerB")]
for rd, name in customers:
print(f"{name}: 10.1.0.0/24 -> VPNv4 {vpn_prefix(rd, '10.1.0.0/24')}")
print("identical customer prefixes, globally unique in MP-BGP.")
# CustomerA: 10.1.0.0/24 -> VPNv4 64500:1:10.1.0.0/24
# CustomerB: 10.1.0.0/24 -> VPNv4 64500:2:10.1.0.0/24
# identical customer prefixes, globally unique in MP-BGP.
# The core's routing burden, with and without MPLS.
def core_routes(customers, routes_each, mpls):
return {"core_router_entries": 0 if mpls else customers * routes_each,
"needs_bgp_in_core": not mpls}
print()
for mpls in (False, True):
c = core_routes(500, 200, mpls)
label = "with MPLS L3VPN" if mpls else "without MPLS"
print(f"{label:<18} core holds {c['core_router_entries']:>7,} customer routes, "
f"BGP in core: {c['needs_bgp_in_core']}")
# without MPLS core holds 100,000 customer routes, BGP in core: True
# with MPLS L3VPN core holds 0 customer routes, BGP in core: False
# The core forwards on LABELS, so it needs only IGP routes to the PEs.
# Enable MPLS with LDP:
# mpls ip
# interface GigabitEthernet0/1
# mpls ip
# mpls label protocol ldp
$ vtysh -c "show mpls ldp neighbor"
# AF ID State Remote Address Uptime
# ipv4 10.255.0.2:0 OPERATIONAL 10.255.0.2 3d04h
# The LFIB — what the core actually forwards on:
$ show mpls forwarding-table
# Local Outgoing Prefix Bytes Label Outgoing Next Hop
# Label Label or Tunnel Id Switched interface
# 100 200 10.9.0.0/16 184221934 Gi0/2 10.0.0.6
# ^^^ SWAP 100 -> 200. No IP lookup happened.
# 300 Pop Label 10.255.0.9/32 9128422 Gi0/3 10.0.0.10
# ^^^^^^^^^ PHP: this is the penultimate hop
# The label stack on the wire:
$ sudo tcpdump -i eth0 mpls -v -c 1
# MPLS (label 1001, exp 0, [S], ttl 63)
# ^^^^^^^^^^ ^^^ bottom of stack
# MPLS (label 2002, exp 0, ttl 63)
# IP 10.1.1.5 > 10.9.0.9: ICMP echo request
# Two labels: outer transport, inner VPN.
# L3VPN configuration — per-customer VRFs:
# vrf definition CustomerA
# rd 64500:1
# route-target export 64500:1
# route-target import 64500:1
# interface GigabitEthernet0/5
# vrf forwarding CustomerA
# ip address 10.1.0.1 255.255.255.0
# router bgp 64500
# address-family vpnv4
# neighbor 10.255.0.2 activate
# neighbor 10.255.0.2 send-community extended
$ show ip route vrf CustomerA | head -3
# Routing Table: CustomerA
# B 10.2.0.0/24 [200/0] via 10.255.0.2, 00:14:22
# ^^^^^^^^^^^^ the remote PE's loopback
$ show bgp vpnv4 unicast all | head -4
# Network Next Hop Metric LocPrf Weight Path
# Route Distinguisher: 64500:1 (default for vrf CustomerA)
# *>i 10.2.0.0/24 10.255.0.2 0 100 0 ?
# ^^ RD makes overlapping customer prefixes unique
# Traffic engineering with RSVP-TE — a path IP routing could never choose:
# interface Tunnel1
# ip unnumbered Loopback0
# tunnel mode mpls traffic-eng
# tunnel destination 10.255.0.9
# tunnel mpls traffic-eng bandwidth 500000
# tunnel mpls traffic-eng path-option 1 explicit name AVOID_CONGESTED_LINK
$ show mpls traffic-eng tunnels brief | head -3
# TUNNEL NAME DESTINATION UP IF DOWN IF STATE/PROT
# R1_t1 10.255.0.9 - Gi0/2 up/up
Key takeaways
- MPLS forwards on an exact-match 20-bit label instead of a longest-prefix IP lookup; the label is a 32-bit shim at "layer 2.5".
- Shim fields: Label (20), TC/EXP (3) for QoS, S (1) bottom-of-stack, TTL (8); the S bit enables label stacking.
- Operations: PUSH at the ingress LER, SWAP in the core LSR, POP at the egress — core routers never inspect the IP header.
- An LSP is unidirectional; bidirectional traffic needs two.
- PHP (Penultimate Hop Popping) has the second-to-last router pop the label (signalled by implicit-null, label 3) so the egress does only one lookup.
- Label distribution: LDP (follows the IGP), RSVP-TE (explicit paths with bandwidth reservation), BGP-LU, and Segment Routing.
- MPLS's original speed advantage disappeared with modern ASICs; it survives for L3VPN, traffic engineering, fast reroute, and the BGP-free core.
- L3VPN uses VRFs, route distinguishers (making overlapping customer prefixes unique) and route targets (controlling import/export) over MP-BGP.
🧪 Practice
- Decode the MPLS shim
0x003e9b40into label, TC, S bit, and TTL.- Trace a packet through ingress, two core routers, and egress, naming the operation at each and where PHP occurs.
- Explain how two customers can both use
10.1.0.0/24on the same provider backbone without conflict.- Interview-style: Modern ASICs do longest-prefix matching at line rate, so MPLS's speed advantage is gone. Why is it still deployed everywhere? (Hint: name a forwarding decision that depends on something other than the destination address — destination-based IP routing cannot express it.)
Segment Routing Overview
Theory
MPLS delivered traffic engineering and VPNs, but at a cost in protocol machinery: LDP to distribute labels, RSVP-TE to signal engineered paths, and per-LSP state in every router along every path. A network with thousands of engineered tunnels holds enormous distributed state, and every router must maintain it consistently.
Segment Routing (SR, RFC 8402) removes that state by applying source routing, done correctly. The ingress router encodes the entire path into the packet as a list of segments. Transit routers simply execute the instruction on top of the list. No per-flow state exists anywhere in the network except at the ingress.
If that sounds like IPv4 source routing (Chapter 5), the differences are decisive. IPv4's LSRR was an option that forced packets onto the slow path, could specify arbitrary hops including ones the network never sanctioned, and was a security disaster. Segment Routing uses labels the network itself advertises, stays on the fast path, and confines paths to segments the operator has published. The idea was sound; the IPv4 encoding was not.
The segment types:
| Segment | Meaning | Advertised by |
|---|---|---|
| Prefix-SID | "Go to this prefix by the shortest path" — globally significant | The IGP |
| Node-SID | A Prefix-SID for a router's loopback — identifies a node | The IGP |
| Adjacency-SID | "Use this specific link" — locally significant | The IGP |
| Anycast-SID | Shared by several nodes; reach the nearest | The IGP |
| Binding-SID | Expands to a whole list of segments | The IGP |
The most consequential architectural change is that SIDs are distributed by the IGP itself — as new TLVs in IS-IS or new opaque LSAs in OSPF. LDP and RSVP-TE are no longer needed. One protocol does routing and label distribution together, which removes an entire class of failure (the notorious LDP/IGP synchronisation problem, where an IGP path came up before its labels existed and traffic black-holed).
Segments combine to express paths of any precision:
- A single Node-SID gives shortest-path forwarding — exactly equivalent to LDP, with no LDP.
- A list of Node-SIDs steers through waypoints while taking shortest paths between them.
- Adjacency-SIDs pin specific links, giving hop-by-hop control.
SRv6 is the IPv6 variant. Instead of MPLS labels it uses IPv6 addresses as segment identifiers, carried in a Segment Routing Header (SRH) — an IPv6 extension header (Chapter 5). This eliminates MPLS from the data plane entirely: SRv6 packets are ordinary IPv6 packets, routable across any IPv6 network. It also enables network programming, where a SID encodes not just a location but a function to execute there.
The practical advantages:
Massive state reduction. Per-flow state exists only at the ingress, so the core scales with topology size rather than with the number of tunnels.
TI-LFA (Topology Independent LFA) provides guaranteed sub-50 ms protection for every destination — unlike classic LFA, which could not always find a loop-free alternate. It works by pushing a segment list that steers around the failure.
SDN-friendliness. A controller with a full topology view can compute a path and install it by giving the ingress a segment list — no signalling to any other router.
Its costs: a deeper label stack (hardware limits how many labels can be pushed), and the need for a controller or PCE to compute non-trivial paths. Segment Routing is the clear direction of travel for new service provider and large data centre deployments.
Examples
RSVP-TE vs Segment Routing: where the state lives
RSVP-TE — every router on every path holds per-LSP state:
[PE1]---[P1]---[P2]---[P3]---[PE2]
S S S S S S = per-LSP state
1,000 tunnels x 5 routers = 5,000 state entries to maintain,
signal, refresh, and keep consistent.
SEGMENT ROUTING — state only at the ingress:
[PE1]---[P1]---[P2]---[P3]---[PE2]
S - - - -
1,000 tunnels = 1,000 entries, ALL at the ingress.
Core routers just execute the top segment. No signalling protocol.
A segment list encoding a path
topology: [PE1]--[P1]--[P2]--[PE2]
\ /
+--[P3]--+ (the shortest path)
want: force traffic via P1 and P2, avoiding P3.
ingress pushes: [ Node-SID(P1)=16001 ]
[ Node-SID(P2)=16002 ]
[ Node-SID(PE2)=16005 ]
at P1: top segment is mine -> POP -> next is 16002 -> shortest path to P2
at P2: top segment is mine -> POP -> next is 16005 -> shortest path to PE2
at PE2: POP -> deliver
No router other than PE1 holds any state about this path.
SR-MPLS vs SRv6
SR-MPLS: +--------+-------+-------+----------+
|Ethernet| 16001 | 16005 | IP packet| labels as SIDs
+--------+-------+-------+----------+
SRv6: +--------+--------------+------------------+---------+
|Ethernet| IPv6 header | SRH (segment list)| payload |
+--------+--------------+------------------+---------+
^ IPv6 ADDRESSES as SIDs.
No MPLS anywhere; routable over
any plain IPv6 network.
class SRNetwork:
"""SIDs are advertised by the IGP. No LDP, no RSVP-TE."""
def __init__(self, srgb_base=16000):
self.srgb_base = srgb_base # Segment Routing Global Block
self.node_sids = {}
self.adj_sids = {}
def assign_node_sid(self, node, index):
self.node_sids[node] = self.srgb_base + index
return self.node_sids[node]
def assign_adj_sid(self, node, neighbor, label):
self.adj_sids[(node, neighbor)] = label # LOCALLY significant
return label
net = SRNetwork()
for i, node in enumerate(["PE1", "P1", "P2", "P3", "PE2"], start=1):
net.assign_node_sid(node, i)
net.assign_adj_sid("P1", "P2", 24012)
print("Node-SIDs (globally significant, advertised by the IGP):")
for node, sid in net.node_sids.items():
print(f" {node:<5} {sid}")
print(f"Adjacency-SID P1->P2 (LOCAL to P1): {net.adj_sids[('P1','P2')]}")
# Node-SIDs (globally significant, advertised by the IGP):
# PE1 16001
# P1 16002
# P2 16003
# P3 16004
# PE2 16005
# Adjacency-SID P1->P2 (LOCAL to P1): 24012
def build_segment_list(net, waypoints, destination):
"""Encode a path as segments. Shortest path BETWEEN waypoints."""
return [net.node_sids[w] for w in waypoints] + [net.node_sids[destination]]
shortest = build_segment_list(net, [], "PE2")
engineered = build_segment_list(net, ["P1", "P2"], "PE2")
print(f"\nshortest path : {shortest} (one segment = plain shortest path, no LDP)")
print(f"engineered path: {engineered} (waypoints forced; avoids P3)")
# shortest path : [16005] (one segment = plain shortest path, no LDP)
# engineered path: [16002, 16003, 16005] (waypoints forced; avoids P3)
def forward(segment_list, at_node, net):
"""A transit router executes only the TOP segment."""
if not segment_list:
return segment_list, f"{at_node}: no segments -> deliver by IP"
top = segment_list[0]
owner = next((n for n, s in net.node_sids.items() if s == top), None)
if owner == at_node:
rest = segment_list[1:]
nxt = next((n for n, s in net.node_sids.items() if s == rest[0]), "IP") if rest else "IP"
return rest, f"{at_node}: top SID {top} is mine -> POP -> next target {nxt}"
return segment_list, f"{at_node}: SID {top} belongs to {owner} -> forward by shortest path"
print()
segs = list(engineered)
for node in ["PE1", "P1", "P2", "PE2"]:
segs, msg = forward(segs, node, net)
print(f" {msg:<62} stack now {segs}")
# PE1: SID 16002 belongs to P1 -> forward by shortest path stack now [16002, 16003, 16005]
# P1: top SID 16002 is mine -> POP -> next target P2 stack now [16003, 16005]
# P2: top SID 16003 is mine -> POP -> next target PE2 stack now [16005]
# PE2: top SID 16005 is mine -> POP -> next target IP stack now []
# State reduction: the headline benefit.
def state_entries(tunnels, path_length, technology):
if technology == "RSVP-TE":
return tunnels * path_length # every router on every path
return tunnels # ingress only
print()
print(f"{'tunnels':>9}{'RSVP-TE state':>16}{'SR state':>11}{'reduction':>12}")
for t in (100, 1_000, 10_000):
r, s = state_entries(t, 5, "RSVP-TE"), state_entries(t, 5, "SR")
print(f"{t:>9,}{r:>16,}{s:>11,}{100*(1-s/r):>11.0f}%")
# tunnels RSVP-TE state SR state reduction
# 100 500 100 80%
# 1,000 5,000 1,000 80%
# 10,000 50,000 10,000 80%
# The reduction equals (path_length - 1)/path_length, and the core
# holds NO per-tunnel state at all — which is the real win.
# Protocols eliminated.
print()
for tech, protos in [("MPLS + LDP + RSVP-TE", ["IGP", "LDP", "RSVP-TE"]),
("Segment Routing", ["IGP (carries SIDs itself)"])]:
print(f"{tech:<24} protocols required: {protos}")
# MPLS + LDP + RSVP-TE protocols required: ['IGP', 'LDP', 'RSVP-TE']
# Segment Routing protocols required: ['IGP (carries SIDs itself)']
# Removing LDP also removes the LDP/IGP synchronisation black-hole problem.
# TI-LFA: guaranteed protection, unlike classic LFA.
def protection_coverage(scheme, topology_type):
if scheme == "classic LFA":
return {"ring": 0.55, "mesh": 0.90}[topology_type]
return 1.0
print()
for scheme in ("classic LFA", "TI-LFA"):
for topo in ("ring", "mesh"):
print(f"{scheme:<14} on a {topo:<5} topology: "
f"{protection_coverage(scheme, topo):>5.0%} of destinations protected")
# classic LFA on a ring topology: 55% of destinations protected
# classic LFA on a mesh topology: 90% of destinations protected
# TI-LFA on a ring topology: 100% of destinations protected
# TI-LFA on a mesh topology: 100% of destinations protected
# TI-LFA pushes a segment list that steers AROUND the failure, so a
# loop-free backup always exists regardless of topology.
# Enable Segment Routing in the IGP — no LDP, no RSVP-TE:
# segment-routing mpls
# global-block 16000 23999 <- the SRGB, identical network-wide
# router isis CORE
# segment-routing mpls
# segment-routing prefix-sid-map advertise-local
# interface Loopback0
# isis prefix-sid index 1 <- Node-SID = SRGB base + 1 = 16001
$ show segment-routing mpls connected-prefix-sid-map
# Prefix SID Algo Source
# 10.255.0.1/32 1 0 Local
# 10.255.0.2/32 2 0 ISIS
# ^ the IGP itself carries the SID. No label protocol.
$ show mpls forwarding-table labels 16002
# Local Outgoing Prefix Outgoing Next Hop
# Label Label or Tunnel Id interface
# 16002 16002 10.255.0.2/32 Gi0/2 10.0.0.6
# ^^^^^ the SAME label network-wide: Node-SIDs are GLOBAL,
# so the label does not change hop to hop (unlike LDP)
# An engineered path — configured ONLY at the ingress:
# segment-routing traffic-eng
# policy AVOID_P3
# color 100 end-point ipv4 10.255.0.5
# candidate-paths
# preference 100
# explicit segment-list VIA_P1_P2
# segment-list VIA_P1_P2
# index 10 mpls label 16002
# index 20 mpls label 16003
# index 30 mpls label 16005
# ! No other router is configured or signalled. At all.
$ show segment-routing traffic-eng policy all
# Name: AVOID_P3 Color: 100 End-point: 10.255.0.5
# Status: Admin: up Operational: up
# Candidate-paths:
# Preference 100 (explicit):
# 16002 [Prefix-SID, 10.255.0.2]
# 16003 [Prefix-SID, 10.255.0.3]
# 16005 [Prefix-SID, 10.255.0.5]
# TI-LFA — guaranteed sub-50 ms protection for every prefix:
# router isis CORE
# fast-reroute per-prefix ti-lfa
$ show isis fast-reroute summary
# Prefixes protected: 1,247 / 1,247 (100%)
# ^^^^^ classic LFA could not promise this
# SRv6 — no MPLS in the data plane at all:
# segment-routing srv6
# locators
# name MAIN
# prefix 2001:db8:cafe::/48
$ show segment-routing srv6 sid
# SID Behavior Context
# 2001:db8:cafe:1:: End
# 2001:db8:cafe:1:1:: End.X Gi0/2, 10.0.0.6
$ sudo tcpdump -i eth0 -n ip6 -v | grep -i srh
# srh (segments-left=2, [0]2001:db8:cafe:3::, [1]2001:db8:cafe:2::)
# ^ an ordinary IPv6 extension header; any IPv6 router can carry it
Key takeaways
- Segment Routing encodes the path in the packet as a list of segments, so per-flow state exists only at the ingress.
- Unlike IPv4 source routing, SR uses labels the network advertises, stays on the fast path, and confines paths to operator-sanctioned segments.
- Segment types: Prefix-SID and Node-SID (global, shortest path to a prefix/node), Adjacency-SID (local, a specific link), Anycast-SID, Binding-SID.
- SIDs are distributed by the IGP itself, so LDP and RSVP-TE are eliminated — along with the LDP/IGP synchronisation black-hole problem.
- A single Node-SID reproduces LDP-style shortest-path forwarding; a list steers through waypoints; Adjacency-SIDs pin exact links.
- SRv6 uses IPv6 addresses as SIDs in a Segment Routing Header, removing MPLS from the data plane and enabling network programming.
- TI-LFA gives guaranteed sub-50 ms protection for every destination, which classic LFA could not.
- Costs: deeper label stacks (hardware push limits) and reliance on a controller or PCE for non-trivial path computation.
🧪 Practice
- Explain why Segment Routing eliminates per-LSP state in core routers and compute the state for 5,000 tunnels over 6-hop paths under RSVP-TE versus SR.
- Distinguish a Node-SID from an Adjacency-SID in terms of scope and what each forces.
- Construct a segment list forcing traffic through two waypoints before the destination, and describe what each router does with it.
- Interview-style: Segment Routing is source routing, which IPv4 deprecated as a security disaster. Why is it acceptable now? (Hint: compare who chooses the permitted hops in each design, and whether the packet stays on the hardware fast path.)
<a id="7-the-transport-layer"></a>