Two exits, one idle link: a hands-on look at BGP ADD-PATH

Here is a situation I have seen more than once. A data center has two border leaves, each connected to a different WAN gateway. Both links are up, both BGP sessions are established, the monitoring is green. And one of the two border links has carried almost nothing for months. Someone suggests buying bigger optics for the busy one.

Nothing is broken. BGP is doing exactly what it was designed to do. This lab rebuilds that situation on a laptop, finds the cause, and fixes it, one knob at a time. On the way, it shows when the popular fix, ADD-PATH, matters, when it does nothing, and the one thing it is often sold for but does not do.

What you will see

• Behind a route reflector, ADD-PATH alone does not move a single packet, and neither does multipath-relax alone. Together, they put the idle link to work.
• In an eBGP fabric, ADD-PATH gives you visibility and nothing else.
• ADD-PATH does not make failover faster. In one design it creates a short forwarding loop, caught here in a packet capture.

The rule behind the idle link

A BGP router advertises one path per prefix: its best. A route reflector is no exception. It can know about two exits and still tell its clients about only one.

Think of a receptionist who knows two colleagues able to help you, but only ever gives out one name. You will always walk to the same office, however busy it is.

ADD-PATH (RFC 7911) lets the receptionist hand over the whole list. Technically, it adds a 4-byte Path ID in front of each prefix, so a second announcement for the same prefix is added next to the first instead of replacing it.

Both neighbors must agree to it when the session opens. In the capture, the reflector offers to send and receive, and the leaf offers to receive:

The lab

Two designs, both on FRR 10.2.1 (the routing stack inside Cumulus Linux and SONiC) with containerlab, and real iperf3 traffic between two hosts.

The main design puts the route reflector off the forwarding path, as most virtual or dedicated reflectors are. leaf1 is wired to both border leaves. Behind them, the destination network is reachable through two gateways in different ASes, as when you are dual-homed to two WAN providers.

Everything below comes from the lab: router outputs, traffic counters, and packet captures. The whole lab, with the captures, is on GitHub.

Step 1: the idle link

The reflector knows both exits. leaf1 hears about one.

Eight iperf3 streams, about 65 Gb/s: 100% on eth1, 0% on eth2. border2 and its link are perfectly healthy and completely idle.

Step 2: try multipath-relax. Nothing changes

The usual first reflex is to enable multipath on the leaf:

leaf1(config)# router bgp 65000
leaf1(config-router)# bgp bestpath as-path multipath-relax

Still one path, still one next hop, still 100% on eth1. Multipath can only balance across paths the router knows about, and leaf1 knows one. Turn it back off for the next step.

Step 3: try ADD-PATH. Nothing changes either

Now the feature everyone recommends, on the reflector:

rr1(config)# router bgp 65000
rr1(config-router)# address-family ipv4 unicast
rr1(config-router-af)# neighbor 10.255.0.11 addpath-tx-all-paths

On the wire, the reflector now sends the same prefix twice, with Path IDs 3 and 4, and two different next hops, border1 and border2:

leaf1 receives both. And still forwards on one link:

Look closely at the two paths: neither says multipath. By default BGP only load-shares across paths whose AS paths are identical, and these are not: 65098 65100 and 65099 65100. The leaf can see the second exit; it is not allowed to use it. Traffic: 100% on eth1.

Step 4: both knobs. Both links

Turn multipath-relax back on at leaf1, with ADD-PATH still on at the reflector:

Two next hops, two interfaces, and traffic on both links: 63% and 37% with eight flows. Per-flow hashing is never perfectly even with so few flows; that is the hash collision problem, not a lab error.

Here are the four measurements side by side:

ADD-PATH gets the second path to the leaf. multipath-relax lets the leaf use it. Each one alone changes nothing. If your two exits lead to the same neighboring AS, the AS paths match and ADD-PATH alone is enough; with two providers, it is not.

The counter-example: an eBGP fabric

Same feature, different design. In a typical eBGP fabric, every switch has its own AS, and every hop rewrites the next hop to itself.

The spine learns both exits itself. Enabling multipath-relax on the spine is enough to spread traffic across both border links: 63% and 37% in this run. Then I turned on ADD-PATH from the spine to the leaf. The leaf receives two paths:

Two paths, two AS paths, and one next hop: 10.0.1.1, the spine. Whichever path the leaf picks, the packet goes to the same place.

ADD-PATH gave the leaf visibility: useful for troubleshooting and telemetry, but not a single new forwarding choice. In an eBGP fabric, the knob that creates ECMP is multipath-relax, on every tier.

The selection policy is a design choice

ADD-PATH does not have to send everything. FRR, like most implementations, can send the best path per neighboring AS instead. In this lab that still sends both paths, because the two exits lead to different neighboring ASes, 65098 and 65099. In an earlier version of the lab, with both exits behind the same gateway AS, the same setting sent only one path, and the second link went idle again.

Same command, opposite result. Decide what each receiver needs (every path, the best per AS, or the best few) before you turn it on.

The myth: “ADD-PATH gives you fast failover”

This is the claim I wanted to test. With a backup path already in the table, surely failover is faster?

I failed border1’s link to its gateway three times in each of three setups, with a ping flow at 100 packets per second hashed through border1. Each lost ping is about 10 ms of outage.

Setup	Pings lost	TTL exceeded
No ADD-PATH	14, 15, 15	0
ADD-PATH to the leaf	14, 15, 15	0
ADD-PATH to the leaf and both borders	15, 15, 15	6, 6, 6

Convergence did not change: about 150 ms in every case. That is the time it takes border1’s withdrawal to travel through the reflector to the leaf. Having more paths does not make that journey shorter.

The last row is worse: giving the borders backup paths created a forwarding loop. border1’s backup exit is border2, but the only way to reach border2 is back through leaf1, which is still sending the traffic to border1. The capture on the leaf1–border1 link shows a single ping bouncing between the two routers:

63 crossings of the same cable in 0.3 milliseconds, then the TTL runs out. The full timeline, from the same capture:

For comparison, the same failure in the eBGP fabric costs about 60 ms: the spine peers directly with both borders and already holds both paths. A reflector adds a hop of propagation.

The lesson: a backup path only speeds up failover if it is reachable without going back through the router that is still sending you the traffic. If it is not, protect the ingress instead: fast failure detection (BFD), and next-hop tracking so the leaf drops a path the moment its exit fails.

So is the second path a safety net?

A fair question, and one I was asked while preparing this lab: if the router already holds two paths, does it keep forwarding when one fails, without waiting for BGP?

Yes, when the failure is the router’s own link. I failed leaf1’s link to border1 while a ping flow used it, and counted on h2 the requests that actually arrived:

leaf1 holds	Lost toward h2	Lost on the round trip
Two paths (ADD-PATH and multipath-relax)	0, 0, 0	15, 15, 15
One path	11, 11, 12	15, 15, 15

With two paths installed, the kernel removes the dead next hop from the ECMP group the moment the interface goes down, and the flow moves to eth2. BGP is not involved. With one path, the leaf waits about 115 ms for the IGP and BGP to remove the path and for the reflector to send the other one.

No, when the failure is somewhere else. The round trip still lost about 150 ms in both cases. For the return traffic, the failure is remote: the wan router keeps hashing some replies toward border1, which can no longer reach the leaf, until the withdrawal arrives. The border1 gateway test above is the same story. A path you hold is only as fresh as the last update about it.

And watch the next-hop tracking. The first time I ran this test, the leaf lost five seconds of traffic, two paths or not. After eth1 went down, BGP still considered border1’s loopback reachable, through the lab’s management network default route: FRR had ip nht resolve-via-default turned on. The dead path stayed valid and traffic went into the management network until the iBGP session timed out. Any default or summary route that covers your BGP next hops can do the same. The lab now sets no ip nht resolve-via-default on every fabric router.

When does ADD-PATH change forwarding?

Design	Does ADD-PATH change forwarding?	What you actually need
eBGP fabric, next hop rewritten at every hop	No: paths collapse onto one next hop	multipath-relax on every tier
iBGP, reflector in the forwarding path	Little: the reflector balances anyway	iBGP multipath on the reflector
iBGP, off-path reflector, exits in the same AS	Yes	ADD-PATH and maximum-paths ibgp
iBGP, off-path reflector, exits in different ASes	Only with multipath-relax	ADD-PATH, maximum-paths ibgp and multipath-relax
Overlays (EVPN, MPLS, SRv6)	Yes: next hops are tunnel endpoints	ADD-PATH with a selection policy chosen on purpose

More paths cost memory and update processing on every receiver. Send only what the receiver can use.

Run it yourself

You need Linux with Docker and containerlab, and about 4 GB of RAM. From the lab folder:

cd phase-b-ibgp-rr && containerlab deploy -t topology.clab.yml && cd ..
python3 verify.py b --failover      # every step above, checked and measured (9 checks)
python3 capture.py b                # the packet captures (redeploy first)

The guide walks through each step by hand, for presenting it live. The .pcap files are in captures/. Open them in Wireshark: it decodes the Path IDs because the OPEN messages are in the same capture.

This lab was prompted by the ADD-PATH discussion in AI Data Center Network Design and Technologies (Subramaniam, Styszynski and Tambakuwala, Pearson, 2026). The topology, configurations, captures and results are my own. Configurations are for a lab; validate them against your release before using them in production.