At 06:30 on a Tuesday, the market data team at Halden Markets brings up the day’s feeds. By 16:30 the network team will have been paged six times, and not one of those pages will have been caused by a link going down.
Halden is fictional. Everything that happens to it below is real: each incident is a failure mode I built, ran and measured in a lab, on IOS-XR and NX-OS, with a small tool that publishes and checks sequence-numbered market data the way a feed handler does. The story is there to keep the order of things honest. In a trading firm, multicast problems don’t arrive as chapters of a textbook. They arrive as “the book is wrong” at 09:31.
New to multicast, or rusty on the details? Multicast from the ground up explains every mechanism used here (IGMP, PIM, RPs, SSM, BIDIR) with captures from the same lab, and the first mention of each term below links to it.
Follow the day from the feeds coming up to the post-mortem. Each chapter tells you what went wrong, explains the piece of multicast behind it, proves it in the lab with CLI output and packet captures, and ends with what I’d do about it.
In short
- A/B redundancy is a property of paths, not feeds. With NX-OS’s default ECMP hash, whether A and B of a channel shared a spine depended on how the venue numbered its groups: none of 8 channels with one numbering, all 8 with another, and after a restart of the same fabric the answer changed again. A PIM failure that OSPF never saw cost both copies of two channels for 74 seconds. With an A plane and a B plane, the same failure cost nothing.
- SSM is the right default for exchange feeds. Cold joins delivered the first message in 79–200 ms with no loss in 10 trials. ASM’s switch to the source tree lost one to seven messages right after the first in 3 of 10 trials.
- The extranet boundary is a filter you must write. Without one, an uncontracted group came in and an internal group went out. One IOS-XR
boundarystopped both. - No querier, no membership. In a VLAN without PIM or a snooping querier, the strategy’s membership expired between 150 and 250 seconds after it joined.
- Pre-joined trees make subscriptions instant. First message in 3–11 ms instead of 69–717 ms.
- Bursts, not averages, drop messages. The queue formula predicted 1,100 and 3,500 losses; the emulated port dropped 1,097 and 3,493.
- BIDIR keeps state flat; Anycast RP makes RP failure a non-event. 32 senders cost ASM 32 entries per router and BIDIR one; losing an anycast RP cost nothing, and a phantom-RP failover cost a third of a second.
The cast: Halden’s network, built in the lab
| Role | Node | Platform |
|---|---|---|
| The exchange | EXCH: one Linux host, two NICs; feed A out of one, feed B out of the other, same sequence numbers | Linux container |
| Extranet edges | XR-A (handoff A), XR-B (handoff B; also the RP for the venue’s ASM groups) | IOS-XR 7.11.1 (XRv 9000) |
| The fabric | SPINE1, SPINE2, LEAF1, LEAF2 | NX-OS 10.4(2) (Nexus 9000v) |
| Strategy servers | ALGO1 (A/B arbiter), ALGO2 (behind SLOW, a Linux bridge with a token-bucket shaper) | Linux containers |
| Internal publishers | TP1 (ticker plant, L2-only VLAN 200), OMS1 (order management, on LEAF2) | Linux containers |
OSPF and PIM sparse mode run everywhere; SSM uses the default 232.0.0.0/8 range. Three RP arrangements sit side by side: the venue’s ASM range 239.1.0.0/16 uses XR-B as its RP, the internal ASM range 239.100.0.0/16 uses an anycast RP on both spines, and the BIDIR range 239.50.1.0/24 uses a phantom RP. The configurations, scripts and every raw result are in the lab repository.
The tool, mdfeed, is about 200 lines of Python. The publisher stamps every message with a sequence number and its send time and sends each one once per feed. The subscriber joins the way a feed handler does, source-specific for SSM and any-source for ASM, and reports what a handler’s gap detector would see: time from join to first message, gaps, duplicates and reordering, and with --arbitrate, the merged A/B stream where the first copy of each sequence number wins.
What a virtual lab can and can’t tell you. Nexus 9000v forwards in software. Above a few hundred packets per second it queues and then drops, so every test runs at 100 messages per second per feed or less, and latencies here are milliseconds that say nothing about real hardware. What the lab does prove is protocol behaviour: which tree carries a packet, what state a router builds, what a failure does to the sequence numbers and how long recovery takes. None of that depends on the ASIC.
06:30 — The feeds come up
Halden’s feed handlers start before anyone else is in. Each one subscribes to the channels its strategies need, A and B for every channel, and the network has to build a tree for each.
What’s happening: SSM, A/B feeds and sequence numbers
Every exchange message carries a sequence number. A feed handler checks each one against the last, and a missing number means the order book is wrong until the gap is repaired by a TCP replay or a snapshot. That’s why venues send everything twice. CME’s MDP 3.0 documentation spells out the arbitration it recommends: listen to Feed A and Feed B, process packets by sequence number, discard a number you’ve already processed, and treat a gap as a loss on both feeds, which is when recovery starts. Nasdaq’s MoldUDP64 takes the same idea a step further with a re-request server you ask for anything you missed.
For exchange feeds, the tree to build is SSM, Source-Specific Multicast (RFC 4607). The venue publishes the source address of every channel, so the receiver can ask for exactly (S,G): this source, this group. Its leaf sends a PIM join hop by hop towards S, and data flows down the shortest path from the very first packet. There’s no rendezvous point, no register and no switchover. 232.0.0.0/8 is reserved for SSM, and the host asks with IGMPv3 (RFC 4604), naming the source in the report.
In the lab
On the wire, the subscription is an IGMPv3 report naming the group and the source, sent twice (IGMP repeats unsolicited reports in case one is lost). When the subscriber leaves fifteen seconds later, it reports block source, and the querier checks with group-and-source-specific queries before pruning anything:
ALGO1 eth0: IGMP
1 0.000000 10.20.100.11 → 224.0.0.22 IGMPv3 58 Membership Report / Group 232.1.1.1, new source {10.10.1.10}
2 0.100004 10.20.100.11 → 224.0.0.22 IGMPv3 58 Membership Report / Group 232.1.1.1, new source {10.10.1.10}
3 15.063918 10.20.100.11 → 224.0.0.22 IGMPv3 58 Membership Report / Group 232.1.1.1, block source {10.10.1.10}
4 15.067428 10.20.100.1 → 232.1.1.1 IGMPv3 60 Membership Query, specific for group 232.1.1.1, source {10.10.1.10}
5 15.999941 10.20.100.11 → 224.0.0.22 IGMPv3 58 Membership Report / Group 232.1.1.1, block source {10.10.1.10}
6 16.003166 10.20.100.1 → 232.1.1.1 IGMPv3 60 Membership Query, specific for group 232.1.1.1, source {10.10.1.10}
7 16.068365 10.20.100.1 → 232.1.1.1 IGMPv3 60 Membership Query, specific for group 232.1.1.1, source {10.10.1.10}
8 17.004839 10.20.100.1 → 232.1.1.1 IGMPv3 60 Membership Query, specific for group 232.1.1.1, source {10.10.1.10}
Internet Group Management Protocol
[IGMP Version: 3]
Type: Membership Report (0x22)
Reserved: 00
Checksum: 0xe4e6 [correct]
[Checksum Status: Good]
Reserved: 0000
Num Group Records: 1
Group Record : 232.1.1.1 Allow New Sources
Record Type: Allow New Sources (5)
Aux Data Len: 0
Num Src: 1
Multicast Address: 232.1.1.1
Source Address: 10.10.1.10
LEAF1 turns it into a PIM (S,G) join towards the source, sent only up the uplink its RPF check chose:
LEAF1 Ethernet1/1 (to SPINE1): PIM
1 0.000000 10.0.0.9 → 224.0.0.13 PIMv2 68 Join/Prune
3 17.061429 10.0.0.9 → 224.0.0.13 PIMv2 68 Join/Prune
Protocol Independent Multicast
0010 .... = Version: 2
.... 0011 = Type: Join/Prune (3)
Reserved byte(s): 00
Checksum: 0xd6cc [correct]
[Checksum Status: Good]
PIM Options
Upstream-neighbor: 10.0.0.8
Address Family: IPv4 (1)
Encoding Type: Native (0)
Unicast: 10.0.0.8
Reserved byte(s): 00
Num Groups: 1
Holdtime: 210
Group 0
Group 0: 232.1.1.1/32
Address Family: IPv4 (1)
Encoding Type: Native (0)
Flags: 0x00
0... .... = Bidirectional PIM: Not set
.000 000. = Reserved: 0x00
.... ...0 = Admin Scope Zone: Not set
Masklen: 32
Group: 232.1.1.1
Num Joins: 1
IP address: 10.10.1.10/32 (S)
Address Family: IPv4 (1)
Encoding Type: Native (0)
Flags: 0x04, Sparse
0000 0... = Reserved: 0x00
.... .1.. = Sparse: Set
.... ..0. = WildCard: Not set
.... ...0 = Rendezvous Point Tree: Not set
Masklen: 32
Source: 10.10.1.10
Num Prunes: 0
And what arrives is the same message twice, one copy per feed, with identical sequence numbers in the first eight bytes of the payload:
ALGO1 eth0: feed A and feed B (UDP 30001); payload = 8-byte sequence number, 8-byte send time in ns
0.000 s 10.10.1.10 -> 232.1.1.1 TTL 13 seq 93 sent at 1793205080685828081 ns (000000000000005d18e2be8523b457f1...)
0.001 s 10.10.2.10 -> 232.1.2.1 TTL 13 seq 93 sent at 1793205080685828081 ns (000000000000005d18e2be8523b457f1...)
0.199 s 10.10.1.10 -> 232.1.1.1 TTL 13 seq 94 sent at 1793205080885830543 ns (000000000000005e18e2be852fa0238f...)
0.200 s 10.10.2.10 -> 232.1.2.1 TTL 13 seq 94 sent at 1793205080885830543 ns (000000000000005e18e2be852fa0238f...)
0.399 s 10.10.1.10 -> 232.1.1.1 TTL 13 seq 95 sent at 1793205081085829878 ns (000000000000005f18e2be853b8be2f6...)
0.399 s 10.10.2.10 -> 232.1.2.1 TTL 13 seq 95 sent at 1793205081085829878 ns (000000000000005f18e2be853b8be2f6...)
0.598 s 10.10.2.10 -> 232.1.2.1 TTL 13 seq 96 sent at 1793205081285829990 ns (000000000000006018e2be854777a566...)
0.599 s 10.10.1.10 -> 232.1.1.1 TTL 13 seq 96 sent at 1793205081285829990 ns (000000000000006018e2be854777a566...)
0.802 s 10.10.1.10 -> 232.1.1.1 TTL 13 seq 97 sent at 1793205081485830177 ns (000000000000006118e2be8553636821...)
0.802 s 10.10.2.10 -> 232.1.2.1 TTL 13 seq 97 sent at 1793205081485830177 ns (000000000000006118e2be8553636821...)
1.001 s 10.10.1.10 -> 232.1.1.1 TTL 13 seq 98 sent at 1793205081685830212 ns (000000000000006218e2be855f4f2a44...)
1.002 s 10.10.2.10 -> 232.1.2.1 TTL 13 seq 98 sent at 1793205081685830212 ns (000000000000006218e2be855f4f2a44...)
Takeaway: SSM keeps the exchange feeds simple: one join, one tree, nothing in the middle to fail. The rest of the day is about everything around it.
07:45 — A new venue, an old feed handler
Halden is onboarding a second venue. It publishes ASM groups and expects its customers to use an RP. On the same morning, a vendor’s feed handler for the first venue refuses to receive anything: it can only join with IGMPv2.
What’s happening: ASM and the switch to the source tree
Any-Source Multicast (RFC 7761) is the original PIM sparse-mode model. The receiver asks for a group without naming a source, so its router joins a shared tree towards a rendezvous point every router agrees on. The source’s first-hop router registers with the RP: it wraps the data in unicast and sends it to the RP, which then joins towards the source. Data flows down the shared tree. Then, by default, the receiver’s last-hop router learns the source address from the data, joins the shortest-path tree straight to the source, and prunes that source off the shared tree.
That switchover is where trading people should look. For a moment two trees exist and two copies can be in flight. NX-OS switches on the first packet unless ip pim spt-threshold infinity holds a list of groups on the shared tree, which avoids the switchover at the price of a longer path through the RP.
In the lab: the register on the wire
When the venue starts sending an ASM group, XR-A, the first-hop router, registers it with the RP, XR-B: the data travels inside unicast PIM Register messages. The RP answers with a Register-Stop, either because it has joined the source natively or because nobody wants the group yet. After that, XR-A doesn’t stop asking: just before its register-suppression timer runs out it sends a Null-Register, a Register with only a header and no data, so the RP can say whether it wants the traffic again. This capture caught the RP’s Register-Stop and one of those Null-Registers:
XR-A Gi0/0/0/1 (to SPINE1): PIM other than hellos
5 17.400657 10.0.0.2 → 10.255.0.2 PIMv2 62 Register
XR-A Gi0/0/0/2 (to SPINE2): PIM other than hellos
1 0.000000 10.0.0.3 → 224.0.0.13 PIMv2 68 Join/Prune
3 12.188491 10.255.0.2 → 10.0.0.2 PIMv2 60 Register-stop
5 17.052952 10.0.0.3 → 224.0.0.13 PIMv2 68 Join/Prune
Protocol Independent Multicast
0010 .... = Version: 2
.... 0001 = Type: Register (1)
Reserved byte(s): 00
Checksum: 0x9eff [correct]
[Checksum Status: Good]
PIM Options
Flags: 0x40000000
0... .... .... .... .... .... .... .... = Border: No
.1.. .... .... .... .... .... .... .... = Null-Register: Yes
0000 .... = IP Version: Dummy Header (0)
Dummy HeaderIPv4
Source: 10.10.1.10
Group: 239.1.12.1
In the lab: what the switchover costs
The exchange was already publishing 50 messages per second per group when ALGO1 joined, the way a strategy subscribes mid-session. Every trial joined a group nobody had joined yet, so each join was cold.
mode: spt (LEAF1 default: switch to the SPT on the first packet)
kind group join->first msg first seq received missing duplicates reordered
SSM 232.1.10.1 94.1 ms 520 497 0 0 0
SSM 232.1.10.2 85.3 ms 1394 487 0 0 0
SSM 232.1.10.3 88.7 ms 2251 495 0 0 0
SSM 232.1.10.4 86.2 ms 3108 497 0 0 0
SSM 232.1.10.5 97.2 ms 3966 497 0 0 0
ASM 239.1.10.1 137.3 ms 4825 484 0 0 0
ASM 239.1.10.2 110.7 ms 5662 489 7 0 0 gaps: seq 5663-5669 at 268 ms after join
ASM 239.1.10.3 88.9 ms 6518 497 0 0 0
ASM 239.1.10.4 230.2 ms 7382 490 0 0 0
ASM 239.1.10.5 137.0 ms 8234 495 0 0 0
mode: infinity (LEAF1 ip pim spt-threshold infinity for 239.1.0.0/16)
kind group join->first msg first seq received missing duplicates reordered
ASM 239.1.11.1 103.4 ms 547 496 0 0 0
ASM 239.1.11.2 145.4 ms 1372 496 0 0 0
ASM 239.1.11.3 129.9 ms 2230 493 0 0 0
ASM 239.1.11.4 107.4 ms 3086 496 0 0 0
ASM 239.1.11.5 166.5 ms 3945 493 0 0 0
| Design | Join to first message | Cold joins that lost messages |
|---|---|---|
| SSM | 79–200 ms | 0 of 10 |
| ASM, switch to the source tree on the first packet (NX-OS default) | 89–230 ms | 3 of 10: one to seven messages right after the first |
ASM, spt-threshold infinity (shared tree only) |
103–167 ms | 0 of 5 |
The ASM losses sit in exactly one place: the messages straight after the first one, within about 300 ms of the join, while the leaf moves from the shared tree to the source tree. Mid-switchover, LEAF1 holds both trees at once, the shared tree towards the RP through SPINE1 and the source tree through SPINE2, while XR-B, the RP, is still receiving the data as decapsulated registers:
LEAF1# show ip mroute 239.1.10.1
IP Multicast Routing Table for VRF "default"
(*, 239.1.10.1/32), uptime: 00:00:06, igmp ip pim
Incoming interface: Ethernet1/1, RPF nbr: 10.0.0.8
Outgoing interface list: (count: 1)
Vlan100, uptime: 00:00:06, igmp
(10.10.1.10/32, 239.1.10.1/32), uptime: 00:00:06, ip mrib pim
Incoming interface: Ethernet1/2, RPF nbr: 10.0.0.10
Outgoing interface list: (count: 1)
Vlan100, uptime: 00:00:06, mrib
XR-B# show mrib route 239.1.10.1
(*,239.1.10.1) RPF nbr: 10.255.0.2 Flags: C RPF
Up: 00:00:07
Incoming Interface List
Decapstunnel0 Flags: A, Up: 00:00:07
Outgoing Interface List
GigabitEthernet0/0/0/1 Flags: F NS, Up: 00:00:07
(10.10.1.10,239.1.10.1) RPF nbr: 10.255.0.2 Flags: L RPF
Up: 00:14:37
Incoming Interface List
Decapstunnel0 Flags: A, Up: 00:00:07
In the lab: the IGMPv2 feed handler
An IGMPv2 join can’t name a source, so a join for an SSM group is useless on its own. SSM mapping on the leaf (ip igmp ssm-translate) supplies the source the host couldn’t.
ALGO1 played the old handler: a small script sent genuine IGMPv2 reports for 232.1.1.1, with no source, while tcpdump counted what reached the host:
IGMPv2 host reports 232.1.1.1 (no source), without SSM mapping: 0 packets of 232.1.1.1 reached ALGO1 in 15 s
LEAF1# show ip igmp groups vlan 100
IGMP Connected Group Membership for Interface "Vlan100" - 0 total entries
Type: S - Static, D - Dynamic, L - Local, T - SSM Translated, H - Host Proxy
* - Cache Only
Group Address Type Interface Uptime Expires Last Reporter
LEAF1# show ip mroute 232.1.1.1
IP Multicast Routing Table for VRF "default"
(*, 232.0.0.0/8), uptime: 00:48:21, pim ip
Incoming interface: Null, RPF nbr: 0.0.0.0
Outgoing interface list: (count: 0)
IGMPv2 host reports 232.1.1.1 (no source), with ip igmp ssm-translate 232.1.1.0/24 10.10.1.10: 271 packets of 232.1.1.1 reached ALGO1 in 15 s
LEAF1# show ip igmp groups vlan 100
IGMP Connected Group Membership for Interface "Vlan100" - 1 total entries
Type: S - Static, D - Dynamic, L - Local, T - SSM Translated, H - Host Proxy
* - Cache Only
Group Address Type Interface Uptime Expires Last Reporter
232.1.1.1
10.10.1.10 DT Vlan100 00:00:08 00:04:11 10.20.100.11
LEAF1# show ip mroute 232.1.1.1
IP Multicast Routing Table for VRF "default"
(10.10.1.10/32, 232.1.1.1/32), uptime: 00:00:08, igmp ip pim
Incoming interface: Ethernet1/1, RPF nbr: 10.0.0.8
Outgoing interface list: (count: 1)
Vlan100, uptime: 00:00:08, igmp
Without mapping, the leaf keeps no membership and builds no tree: an IGMPv2 join for an SSM group is ignored, and nothing arrives. With one line of SSM mapping, the same v2 join becomes an (S,G) membership, marked T for SSM-translated, and the feed flows.
Takeaway: use SSM wherever the venue publishes sources. Where you must run ASM, put the RP at your edge and decide deliberately between switching to the source tree (the default, and a gap risk at every new subscription) and holding the shared tree (no switchover, longer path). Fix IGMPv2-only handlers with SSM mapping, then fix the handlers.
08:59 — One minute to the open
A strategy that only trades the open subscribes to forty channels at 08:59:30. Every subscription builds a tree hop by hop from the leaf to the venue edge, all at once, all in the busiest minute of the day.
What’s happening: pre-joined trees
A tree that already exists when the strategy subscribes costs almost nothing: the data is at the leaf, and subscribing just adds a port to IGMP snooping. NX-OS’s ip igmp static-oif on the subscriber VLAN makes the leaf join on behalf of hosts that haven’t asked yet. The cost is bandwidth on every link the tree covers, all day long.
In the lab
Five channels were pre-joined on LEAF1 with ip igmp static-oif, five others left cold, all published at 50 messages per second. ALGO1 then subscribed to each in turn and timed the first message:
channel kind join->first msg
232.1.20.1 pre-joined 3.6 ms
232.1.21.1 cold 99.7 ms
232.1.20.2 pre-joined 4.2 ms
232.1.21.2 cold 69.3 ms
232.1.20.3 pre-joined 3.4 ms
232.1.21.3 cold 117.7 ms
232.1.20.4 pre-joined 11.3 ms
232.1.21.4 cold 717.4 ms
232.1.20.5 pre-joined 179.9 ms
232.1.21.5 cold 564.7 ms
LEAF1# show running-config interface Vlan100
interface Vlan100
no shutdown
ip address 10.20.100.1/24
ip ospf passive-interface
ip router ospf 1 area 0.0.0.0
ip pim sparse-mode
ip igmp version 3
ip igmp static-oif 232.1.20.1 source 10.10.1.10
LEAF1# show ip mroute 232.1.20.1
IP Multicast Routing Table for VRF "default"
(10.10.1.10/32, 232.1.20.1/32), uptime: 00:00:21, static ip pim
Incoming interface: Ethernet1/1, RPF nbr: 10.0.0.8
Outgoing interface list: (count: 1)
Vlan100, uptime: 00:00:21, static
Pre-joined channels delivered the first message in 3 to 11 ms, with one outlier at 180 ms; cold channels took 69 to 717 ms, because every one of them needed a PIM join to travel to the venue edge and the data to travel back. The static entry shows up in the mroute table as static: the leaf is pulling the channel whether anyone is subscribed or not.
Takeaway: pre-join the channels your strategies need at the open. Bandwidth is cheap next to a slow first minute.
09:00 — The open
Average utilisation on the 10G link to Halden’s busiest strategy server sits at 30%. At 09:00:00.2, its feed handler logs a burst of gaps on both A and B.
What’s happening: queue arithmetic
Averages hide bursts. When a burst arrives faster than an egress port can send it, the difference queues, and whatever doesn’t fit in the buffer is dropped. A burst of B bytes arriving at Rin into a port that drains at Rout takes B/Rin to arrive, and meanwhile the port sends Rout × B/Rin. The queue peaks at:
Qpeak = B × (1 − Rout / Rin)
and anything above the buffer is lost. 1 MB arriving at line rate from a 25G source into a 10G port needs 600 kB of buffer; into a 1G port, 960 kB. The same arithmetic gives serialization delay, the time one packet occupies the wire: 1,000 bytes take 8 µs at 1G, 0.8 µs at 10G and 0.32 µs at 25G.
In the lab (a Linux emulation)
A virtual switch can’t show buffer behaviour, so this one is a Linux emulation and I’ll say so plainly. SLOW is a Linux bridge between LEAF1 and ALGO2, with a token-bucket filter (tc tbf) on its ALGO2-facing port: a 100 Mb/s egress port with a fixed buffer. The publisher runs on SLOW itself, so bursts arrive at host speed and meet the slow port directly. Each burst is N back-to-back 1,000-byte messages, every 500 ms for 10 seconds; the table puts the queue formula next to what the qdisc dropped and what the subscriber’s gap detector saw:
SLOW eth1: tbf rate 100mbit, 1000-byte messages, bursts every 500 ms for 10 s (Linux emulation of a slow egress port)
buffer B burst msgs bursts R_in Mb/s peak queue kB (model) predicted drops tbf dropped ALGO2 missing gap events
64000 32 20 857 30 0 0 0 0
64000 64 20 1085 61 0 0 0 0
64000 128 20 1154 122 1100 1097 1092 91
64000 256 20 1318 247 3500 3493 3491 265
256000 32 20 851 30 0 0 0 0
256000 64 20 1053 61 0 0 0 0
256000 128 20 1105 122 0 0 0 0
256000 256 20 1131 244 0 0 0 0
SLOW# tc -s qdisc show dev eth1 (buffer 64000 B, bursts of 256)
qdisc tbf 8004: root refcnt 2 rate 100Mbit burst 3000b lat 4.88ms
Sent 1696258 bytes 1643 pkt (dropped 3493, overlimits 6590 requeues 0)
backlog 0b 0p requeues 0
The model predicted 1,100 and 3,500 lost messages for the two bursts that overflow a 64 kB buffer; the queue dropped 1,097 and 3,493, and the subscriber counted 1,092 and 3,491 missing. Quadruple the buffer and the same bursts lose nothing. The averages never changed: 20 bursts in 10 seconds is a few megabits per second on a 100 Mb/s port.
Takeaway: size buffers and port speeds for the burst, not the average. Do the arithmetic before you buy the switch.
10:15 — The spine that didn’t fail
At 10:15 the A and B feeds for two of Halden’s channels stop at the same moment. Nothing is down. Both spines answer pings, OSPF is full everywhere, and the arbitrated stream for those two channels has a 74-second hole in it. The other channels didn’t notice anything.
What’s happening: ECMP decides where A and B go
A/B redundancy is a property of the path, not of the feed. In an ECMP fabric, the path of each (S,G) is picked by a hash. NX-OS’s default, ip multicast multipath s-g-hash, picks the RPF interface from the source, the group and the next hops, and A and B of the same channel have different sources and groups, so they land wherever the hash puts them.
In the lab: the venue’s numbering decides
With the publisher idle, ALGO1 joined channels 1 to 8 of feed A (232.1.1.i) and four possible numberings of feed B, and I read which spine LEAF1 chose for each (S,G):
| Feed B numbering | Channels where A and B share a spine |
|---|---|
| 232.1.2.i | none |
| 232.1.3.i | all 8 |
| 232.2.1.i | 6 of 8 |
| 232.1.129.i | none |
Same fabric, same default, same sources. Only the venue’s numbering of the B groups differs, and it moves fate-sharing from none to all. You can’t design around a hash you don’t control.
It doesn’t even hold still. Three weeks later I restarted the lab for another project and read the same placements again, with the same configuration, sources and groups. With B on 232.1.2.i, the numbering that had shared nothing, channel 1’s A and B both crossed SPINE2. With B on 232.1.3.i, the one that had shared everything, three of the four channels I checked still shared a spine and one didn’t. I didn’t chase why the hash moved; this is NX-OSv, and I’d check on hardware before reading anything into the detail. The lesson doesn’t depend on it: a spine assignment you checked once is an observation, not a design.
In the lab: a failure routing doesn’t see
With B numbered 232.1.3.i, channels 1 and 3 hashed entirely onto SPINE1:
LEAF1# show ip mroute (RPF interface per (S,G), filtered)
(10.10.1.10, 232.1.1.1) RPF Ethernet1/1 -> SPINE1
(10.10.2.10, 232.1.3.1) RPF Ethernet1/1 -> SPINE1
(10.10.1.10, 232.1.1.2) RPF Ethernet1/2 -> SPINE2
(10.10.2.10, 232.1.3.2) RPF Ethernet1/2 -> SPINE2
(10.10.1.10, 232.1.1.3) RPF Ethernet1/1 -> SPINE1
(10.10.2.10, 232.1.3.3) RPF Ethernet1/1 -> SPINE1
(10.10.1.10, 232.1.1.4) RPF Ethernet1/2 -> SPINE2
(10.10.2.10, 232.1.3.4) RPF Ethernet1/2 -> SPINE2
Then PIM broke on SPINE1’s link to LEAF1 (no ip pim sparse-mode) while OSPF stayed up. This is the class of failure that hurts most: a PIM process problem, a policy, a configuration change, anything that breaks multicast while unicast looks healthy. LEAF1 lost its PIM neighbour on that link, but RPF follows unicast routing, and unicast still pointed at SPINE1:
LEAF1# show ip pim neighbor
PIM Neighbor Status for VRF "default"
Neighbor Interface Uptime Expires DR Bidir- BFD ECMP Redirect
Priority Capable State Capable
10.0.0.10 Ethernet1/2 00:14:34 00:01:28 1 yes n/a no
LEAF1# show ip ospf neighbors
OSPF Process ID 1 VRF default
Total number of neighbors: 2
Neighbor ID Pri State Up Time Address Interface
10.255.0.11 1 FULL/ - 00:24:56 10.0.0.8 Eth1/1
10.255.0.12 1 FULL/ - 00:14:28 10.0.0.10 Eth1/2
ALGO1$ mdfeed sub --arbitrate
A1: rx 2365 missing 1885 in 1 gaps dup 0 reord 0
gap seq 579-2463 (1885 msgs) seen at t+103.443 s
B1: rx 2392 missing 1858 in 1 gaps dup 0 reord 0
gap seq 579-2436 (1858 msgs) seen at t+102.358 s
A2: rx 4250 missing 0 in 0 gaps dup 0 reord 0
B2: rx 4250 missing 0 in 0 gaps dup 0 reord 0
A3: rx 2464 missing 1786 in 1 gaps dup 0 reord 0
gap seq 579-2364 (1786 msgs) seen at t+99.478 s
B3: rx 2492 missing 1758 in 1 gaps dup 0 reord 0
gap seq 579-2336 (1758 msgs) seen at t+98.361 s
A4: rx 4250 missing 0 in 0 gaps dup 0 reord 0
B4: rx 4250 missing 0 in 0 gaps dup 0 reord 0
A/B 1: rx 4757 missing 1858 in 1 gaps dup 2365 reord 0
gap seq 579-2436 (1858 msgs) seen at t+102.358 s
A/B 2: rx 8500 missing 0 in 0 gaps dup 4250 reord 0
A/B 3: rx 4956 missing 1758 in 1 gaps dup 2464 reord 0
gap seq 579-2336 (1758 msgs) seen at t+98.361 s
A/B 4: rx 8500 missing 0 in 0 gaps dup 4250 reord 0
Channels 1 and 3 lost both copies of the same 1,858 consecutive messages, about 74 seconds at 25 per second, and the arbiter could do nothing: the merged stream has the same gap. Channels 2 and 4 didn’t lose a message.
In the lab: two planes
Two lines on LEAF1 pin every A source to SPINE1 and every B source to SPINE2:
ip mroute 10.10.1.0/24 10.0.0.8
ip mroute 10.10.2.0/24 10.0.0.10
The same failure:
ALGO1$ mdfeed sub --arbitrate
A1: rx 1741 missing 2509 in 1 gaps dup 0 reord 0
gap seq 577-3085 (2509 msgs) seen at t+128.352 s
B1: rx 4250 missing 0 in 0 gaps dup 0 reord 0
A2: rx 1785 missing 2465 in 1 gaps dup 0 reord 0
gap seq 577-3041 (2465 msgs) seen at t+126.59 s
B2: rx 4250 missing 0 in 0 gaps dup 0 reord 0
A3: rx 1837 missing 2413 in 1 gaps dup 0 reord 0
gap seq 577-2989 (2413 msgs) seen at t+124.51 s
B3: rx 4250 missing 0 in 0 gaps dup 0 reord 0
A4: rx 1890 missing 2360 in 1 gaps dup 0 reord 0
gap seq 577-2936 (2360 msgs) seen at t+122.389 s
B4: rx 4250 missing 0 in 0 gaps dup 0 reord 0
A/B 1: rx 5991 missing 0 in 0 gaps dup 1741 reord 0
A/B 2: rx 6035 missing 0 in 0 gaps dup 1785 reord 0
A/B 3: rx 6087 missing 0 in 0 gaps dup 1837 reord 0
A/B 4: rx 6140 missing 0 in 0 gaps dup 1890 reord 0
Feed A went dark on every channel, feed B carried every message, and the merged streams lost nothing.
Three more things these runs showed:
- A dark plane is silent. With planes, losing A cost the strategies nothing, which is the point, and also the danger: Halden is now running on one feed and nobody has noticed. Alarm on per-feed gaps, not only arbitrated ones.
- Recovery isn’t instant when the fault clears. After PIM came back, the hashed channels took about 19 seconds to recover and the pinned A plane about 40. PIM rebuilds on its periodic join refresh: plan on tens of seconds.
- Clean failures are fast. When SPINE1 shut its port to LEAF1 instead (LEAF1’s side kept carrier), OSPF’s two-way check removed the link from LEAF1’s SPF within milliseconds, PIM followed, and the colliding channels lost 9 messages on both feeds. Still an arbitrated gap, just a short one. It’s the partial failures that last.
Takeaway: build an A plane and a B plane, physically if you can. Where A and B must share devices, pin RPF per plane instead of trusting the hash, and monitor each feed on its own.
11:30 — The order bus
Halden’s order-management system publishes order and position updates to every strategy, and every strategy publishes its own state back. Two hundred senders, all on the same few groups. The fabric team notices the multicast tables on the spines growing every time a strategy is added.
What’s happening: BIDIR and the phantom RP
In ASM, every sender becomes an (S,G) entry on every router along its path, with its own register exchange. Bidirectional PIM (RFC 5015) builds a single (*,G) tree rooted at the RP address instead, and data flows both up and down it: no registers, no source trees, no per-source state. On each link, routers elect a designated forwarder (DF) per RP address, and only the DF forwards traffic towards the RP.
The RP address doesn’t even have to exist. In the phantom RP design it’s an unused address on a subnet, the RP link. Two routers advertise that subnet with different mask lengths (here SPINE1 a /30, SPINE2 a /29); longest match makes SPINE1’s link primary, and if it disappears, SPINE2’s takes over. The price is the path: traffic always heads towards the RP link before it comes back down.
In the lab
OMS1 sent from 32 source addresses at once, to one ASM group (239.100.1.50, anycast RP) and to one BIDIR group (239.50.1.50, phantom RP), while ALGO1 subscribed to both. The multicast table on every router told the story, summarised from show ip mroute <group> summary:
LEAF2 ASM 239.100.1.50/32 source entries: 32
LEAF2 BIDIR 239.50.1.0/24 source entries: 0
SPINE1 ASM 239.100.1.50/32 source entries: 32
SPINE1 BIDIR 239.50.1.50/32 source entries: 0
SPINE2 ASM 239.100.1.50/32 source entries: 32
SPINE2 BIDIR 239.50.1.0/24 source entries: 0
LEAF1 ASM 239.100.1.50/32 source entries: 32
LEAF1 BIDIR 239.50.1.50/32 source entries: 0
Every ASM router on the path holds one (S,G) per sender, so the table grows with every engine you add. BIDIR holds one entry for the whole range and none per source: it doesn’t care how many senders there are. Both delivered every message.
SPINE1-LEAF1 link: PIM DF election messages (time, sender, subtype, RP, route metric to the RP)
20.452 s 10.0.0.9 Winner RP 10.254.0.1 metric 110/41
21.550 s 10.0.0.8 Offer RP 10.254.0.1 metric 0/0
21.552 s 10.0.0.9 Backoff RP 10.254.0.1 metric 110/41
21.847 s 10.0.0.9 Offer RP 10.254.0.1 metric infinite
21.909 s 10.0.0.8 Offer RP 10.254.0.1 metric 0/0
22.010 s 10.0.0.8 Offer RP 10.254.0.1 metric 0/0
22.110 s 10.0.0.8 Offer RP 10.254.0.1 metric 0/0
22.151 s 10.0.0.8 Winner RP 10.254.0.1 metric 0/0
SPINE2-LEAF1 link: PIM DF election messages (time, sender, subtype, RP, route metric to the RP)
25.849 s 10.0.0.10 Winner RP 10.254.0.1 metric 0/0
27.860 s 10.0.0.11 Offer RP 10.254.0.1 metric 110/41
27.862 s 10.0.0.10 Winner RP 10.254.0.1 metric 0/0
27.881 s 10.0.0.10 Winner RP 10.254.0.1 metric 110/42
27.983 s 10.0.0.11 Offer RP 10.254.0.1 metric 110/41
27.986 s 10.0.0.10 Backoff RP 10.254.0.1 metric 110/42
28.957 s 10.0.0.10 Pass RP 10.254.0.1 metric 110/42
28.962 s 10.0.0.11 Winner RP 10.254.0.1 metric 110/41
Protocol Independent Multicast
0010 .... = Version: 2
.... 1010 = Type: DF election (10)
0010 .... = DF Subtype: DF Winner (2)
.... 0000 = DF reserved: 0
Checksum: 0xc9e0 [correct]
[Checksum Status: Good]
PIM Options
RP: 10.254.0.1
Address Family: IPv4 (1)
Encoding Type: Native (0)
Unicast: 10.254.0.1
DF Metric Preference: 0
Metric: 0
On the wire, the DF election is a short conversation per link. The capture covers both halves of the test: SPINE1’s /30 disappearing and coming back. On the SPINE2–LEAF1 link, SPINE2 (10.0.0.10) had been the Winner while it owned the only RP link. When SPINE1’s /30 returned, LEAF1 (10.0.0.11) suddenly had the better route to the RP and sent an Offer; SPINE2 answered with a Backoff, then a Pass, handing the role to LEAF1, which announced itself Winner. A metric of infinite is a router saying it has no route to the RP at all. One DF per link, re-elected by metric whenever the route to the RP changes: that’s all BIDIR needs to stay loop-free without any per-source state.
Then the phantom RP failover: OMS1 sent 50 messages per second on the bus while SPINE1 withdrew its /30 RP link. Before, LEAF1 reaches the RP link through SPINE1 and its uplink to SPINE1 is the RPF interface:
LEAF1# show ip route 10.254.0.0/24 longer-prefixes
IP Route Table for VRF "default"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>
10.254.0.0/29, ubest/mbest: 1/0
*via 10.0.0.10, Eth1/2, [110/41], 00:19:31, ospf-1, intra
10.254.0.0/30, ubest/mbest: 1/0
*via 10.0.0.8, Eth1/1, [110/41], 00:19:32, ospf-1, intra
LEAF1# show ip pim df 10.254.0.1
Bidir-PIM Designated Forwarder Information for VRF "default"
RP Address (ordinal) RP Metric Group Range
10.254.0.1 (1) [110/41] 239.50.1.0/24
Interface DF Address DF State DF Metric DF Uptime
Vlan100 10.20.100.1 Winner [110/41] 00:19:31
Lo0 10.255.0.21 Winner [110/41] 00:19:31
Eth1/1 10.0.0.8 Loser [0/0] 00:19:43 (RPF)
Eth1/2 10.0.0.11 Winner [110/41] 00:19:31
Twenty seconds after the /30 disappeared, only SPINE2’s /29 is left, the RPF interface has moved to SPINE2, and every link has a new designated forwarder:
LEAF1# show ip route 10.254.0.0/24 longer-prefixes
IP Route Table for VRF "default"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>
10.254.0.0/29, ubest/mbest: 1/0
*via 10.0.0.10, Eth1/2, [110/41], 00:20:02, ospf-1, intra
LEAF1# show ip pim df 10.254.0.1
Bidir-PIM Designated Forwarder Information for VRF "default"
RP Address (ordinal) RP Metric Group Range
10.254.0.1 (1) [110/41] 239.50.1.0/24
Interface DF Address DF State DF Metric DF Uptime
Vlan100 10.20.100.1 Winner [110/41] 00:20:02
Lo0 10.255.0.21 Winner [110/41] 00:20:02
Eth1/1 10.0.0.9 Winner [110/41] 00:00:24
Eth1/2 10.0.0.10 Loser [0/0] 00:00:25 (RPF)
SPINE2# show ip pim df 10.254.0.1
Bidir-PIM Designated Forwarder Information for VRF "default"
RP Address (ordinal) RP Metric Group Range
10.254.0.1 (1) [0/0] 239.50.1.0/24
Interface DF Address DF State DF Metric DF Uptime
Lo0 10.255.0.12 Winner [0/0] 00:27:00
Lo1 10.254.1.1 Winner [0/0] 00:27:00
Lo2 0.0.0.0 Loser [0/0] 00:00:27 (RPF)
Eth1/1 10.0.0.3 Winner [0/0] 00:00:27
Eth1/2 10.0.0.7 Winner [0/0] 00:00:26
Eth1/3 10.0.0.10 Winner [0/0] 00:00:27
Eth1/4 10.0.0.14 Winner [0/0] 00:00:27
ALGO1$ mdfeed sub
BUS: rx 2483 missing 17 in 1 gaps dup 0 reord 0
gap seq 1127-1143 (17 msgs) seen at t+27.783 s
The switch from the /30 to the /29 cost 17 messages, about a third of a second at that rate: the time for OSPF to withdraw the route and for every link to re-elect its DF. Fast, but not free, which is why the RP link belongs on the most stable devices you have.
Takeaway: for many-to-many groups, BIDIR keeps the state flat however many senders join. Put the RP link on the spines everyone crosses anyway, so the path stretch costs nothing.
12:40 — The RP goes away
Halden’s internal ASM groups use the spines as RPs. At 12:40, SPINE1 is taken out for maintenance.
What’s happening: Anycast RP
With ASM, the RP is a dependency for every new source and receiver. Anycast RP puts the same RP address on several routers. Unicast routing takes each register and join to the nearest one, and the RPs tell each other about the sources they’ve learnt, with MSDP (RFC 3446) or, as NX-OS’s ip pim anycast-rp does, with PIM itself (RFC 4610).
In the lab
Both spines carry 10.254.1.1 and know each other as anycast-RP peers. OMS1 published two groups; ALGO1 was already subscribed to the first when SPINE1 lost its RP address (interface loopback1 shut), and ALGO2 subscribed to the second only after the failure:
LEAF1# show ip route 10.254.1.1
IP Route Table for VRF "default"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>
10.254.1.1/32, ubest/mbest: 1/0
*via 10.0.0.10, Eth1/2, [110/41], 00:16:52, ospf-1, intra
SPINE2# show ip mroute 239.100.1.2
IP Multicast Routing Table for VRF "default"
(10.20.30.11/32, 239.100.1.2/32), uptime: 00:00:44, pim ip
Incoming interface: Ethernet1/4, RPF nbr: 10.0.0.15, internal
Outgoing interface list: (count: 0)
ALGO1$ existing subscriber
EXISTING: rx 1581 missing 0 in 0 gaps dup 0 reord 0
ALGO2$ new group joined after the failure
NEW: rx 697 missing 0 in 0 gaps dup 0 reord 0
The existing stream didn’t lose a message, and the subscription made after the failure delivered its first message in 68 ms with no loss. The RP address simply moved to SPINE2, which now holds the shared tree and the source state for the new group.
Takeaway: if you run internal ASM, run Anycast RP. An RP failure should be a non-event.
14:05 — The ticker plant goes quiet
Halden’s ticker plant republishes normalised prices to the strategies in its own VLAN. No routing is needed, so nobody enabled PIM on that VLAN. In testing it worked. In production, strategies started reporting a dead feed a few minutes after each restart.
What’s happening: IGMP snooping needs a querier
A switch running IGMP snooping forwards a group only to the ports that asked for it, and it learns that from IGMP reports. Hosts send reports when they join, then only in answer to queries. Without a querier, nothing asks, and the switch eventually forgets. Cisco’s NX-OS guide is explicit on both halves: snooping expires a membership after three missed general queries, and when PIM isn’t enabled on the VLAN, you must configure an IGMP snooping querier.
What happens to the traffic once the membership is gone depends on the switch. RFC 4541, the IGMP snooping considerations, calls it an unregistered packet: a switch must forward it on ports where a multicast router is attached, and may flood it to every port. In a VLAN with no router, that’s the difference between a feed that dies and a feed that suddenly reaches every server in the VLAN.
In the lab
ALGO1 joined once. LEAF1’s snooping table over time:
LEAF1# show ip igmp snooping groups vlan 200 (t = 20 s into the feed; ALGO1 joined 3 s before it started)
200 239.200.1.1 v3 D Eth1/6
LEAF1# show ip igmp snooping groups vlan 200 (t = 150 s into the feed; ALGO1 joined 3 s before it started)
200 239.200.1.1 v3 D Eth1/6
LEAF1# show ip igmp snooping groups vlan 200 (t = 250 s into the feed; ALGO1 joined 3 s before it started)
(no entries)
LEAF1# show ip igmp snooping groups vlan 200 (t = 300 s into the feed; ALGO1 joined 3 s before it started)
(no entries)
LEAF1# show ip igmp snooping groups vlan 200 (t = 380 s into the feed; ALGO1 joined 3 s before it started)
(no entries)
The membership was there at 150 seconds and gone by 250. On the wire, the VLAN shows ALGO1’s reports when it joins and then silence: nobody ever asks again.
ALGO1 eth1 (VLAN 200), 150 s, no querier: IGMP
1 0.000000 10.20.200.11 → 224.0.0.22 IGMPv3 54 Membership Report / Join group 239.200.1.1 for any sources
2 0.823944 10.20.200.11 → 224.0.0.22 IGMPv3 54 Membership Report / Join group 239.200.1.1 for any sources
3 150.067969 10.20.200.11 → 224.0.0.22 IGMPv3 54 Membership Report / Leave group 239.200.1.1
4 150.359976 10.20.200.11 → 224.0.0.22 IGMPv3 54 Membership Report / Leave group 239.200.1.1
ALGO1 eth1 (VLAN 200), 150 s, snooping querier 10.20.200.254: IGMP
1 0.000000 10.20.200.11 → 224.0.0.22 IGMPv3 54 Membership Report / Join group 239.200.1.1 for any sources
2 0.456027 10.20.200.11 → 224.0.0.22 IGMPv3 54 Membership Report / Join group 239.200.1.1 for any sources
3 20.577354 10.20.200.254 → 224.0.0.1 IGMPv3 60 Membership Query, general
4 20.579130 10.20.200.254 → 224.0.0.1 IGMPv3 60 Membership Query, general
5 28.808028 10.20.200.11 → 224.0.0.22 IGMPv3 54 Membership Report / Join group 239.200.1.1 for any sources
6 144.176656 10.20.200.254 → 224.0.0.1 IGMPv3 60 Membership Query, general
7 144.179095 10.20.200.254 → 224.0.0.1 IGMPv3 60 Membership Query, general
8 144.616063 10.20.200.11 → 224.0.0.22 IGMPv3 54 Membership Report / Join group 239.200.1.1 for any sources
9 150.072050 10.20.200.11 → 224.0.0.22 IGMPv3 54 Membership Report / Leave group 239.200.1.1
10 150.920023 10.20.200.11 → 224.0.0.22 IGMPv3 54 Membership Report / Leave group 239.200.1.1
With a snooping querier configured on the VLAN (vlan configuration 200 / ip igmp snooping querier 10.20.200.254), the same test keeps its membership, refreshed by every query:
LEAF1# show ip igmp snooping groups vlan 200 (t = 20 s into the feed)
200 239.200.1.1 v3 D Eth1/6
LEAF1# show ip igmp snooping querier vlan 200
200 10.20.200.254 v3 00:01:57 Switch querier
LEAF1# show ip igmp snooping groups vlan 200 (t = 150 s into the feed)
200 239.200.1.1 v3 D Eth1/6
LEAF1# show ip igmp snooping querier vlan 200
200 10.20.200.254 v3 00:01:50 Switch querier
LEAF1# show ip igmp snooping groups vlan 200 (t = 250 s into the feed)
200 239.200.1.1 v3 D Eth1/6
LEAF1# show ip igmp snooping querier vlan 200
200 10.20.200.254 v3 00:00:10 Switch querier
LEAF1# show ip igmp snooping groups vlan 200 (t = 300 s into the feed)
200 239.200.1.1 v3 D Eth1/6
LEAF1# show ip igmp snooping querier vlan 200
200 10.20.200.254 v3 00:01:25 Switch querier
LEAF1# show ip igmp snooping groups vlan 200 (t = 380 s into the feed)
200 239.200.1.1 v3 D Eth1/6
LEAF1# show ip igmp snooping querier vlan 200
200 10.20.200.254 v3 00:00:04 Switch querier
A limit of the virtual switch. Nexus 9000v’s software data plane floods multicast in the VLAN regardless of the snooping table: a port that never joined received all 8,200 packets. So the lab shows the membership expiring exactly as documented, but not what a hardware switch does with the traffic afterwards. That part depends on the platform’s unregistered-multicast behaviour.
Takeaway: every VLAN with IGMP snooping needs a querier: PIM on the SVI, or ip igmp snooping querier under vlan configuration.
15:30 — Something extra from the venue
A strategy server starts receiving a group nobody at Halden has heard of. The venue had started publishing a test channel, and one of Halden’s developers had subscribed to it. Worse, the post-mortem finds that a host on the venue side of the handoff could have joined Halden’s internal order bus.
What’s happening: the extranet boundary
The handoff to a venue is a multicast border between two organisations. Accept only the channels in the contract, send nothing out, and write it down as a boundary on the edge interface, not as an assumption about what the venue will send or what your hosts will ask for.
In the lab
The exchange published the contracted channel 232.1.1.1 and an uncontracted one, 232.9.9.9; ALGO1 asked for both. Meanwhile OMS1 published an internal group, 239.100.1.1, and a host on the exchange side of XR-A asked for it. Without a boundary, everything flowed:
ALGO1$ mdfeed sub (contracted A channel and the uncontracted 232.9.9.9)
A: rx 600 missing 0 in 0 gaps dup 0 reord 0
ROGUE: rx 600 missing 0 in 0 gaps dup 0 reord 0
EXCH$ mdfeed sub (exchange side asks for internal 239.100.1.1)
INTERNAL: rx 885 missing 0 in 0 gaps dup 0 reord 0
XR-A# show mrib route 232.9.9.9
(10.10.1.10,232.9.9.9) RPF nbr: 10.10.1.10 Flags: RPF
Up: 00:00:15
Incoming Interface List
GigabitEthernet0/0/0/0 Flags: A, Up: 00:00:15
Outgoing Interface List
GigabitEthernet0/0/0/1 Flags: F NS, Up: 00:00:15
XR-A# show mrib route 232.1.1.1
(10.10.1.10,232.1.1.1) RPF nbr: 10.10.1.10 Flags: RPF
Up: 00:00:15
Incoming Interface List
GigabitEthernet0/0/0/0 Flags: A, Up: 00:00:15
Outgoing Interface List
GigabitEthernet0/0/0/1 Flags: F NS, Up: 00:00:15
XR-A# show mrib route 239.100.1.1
(*,239.100.1.1) RPF nbr: 10.0.0.3 Flags: C RPF
Up: 00:00:16
Incoming Interface List
GigabitEthernet0/0/0/2 Flags: A NS, Up: 00:00:16
Outgoing Interface List
GigabitEthernet0/0/0/0 Flags: F NS LI, Up: 00:00:16
(10.20.30.11,239.100.1.1) RPF nbr: 10.0.0.3 Flags: RPF
Up: 00:00:16
Incoming Interface List
GigabitEthernet0/0/0/2 Flags: A, Up: 00:00:16
Outgoing Interface List
GigabitEthernet0/0/0/0 Flags: F NS, Up: 00:00:16
With a boundary on XR-A’s handoff that permits only the contracted ranges:
ipv4 access-list EXCH-A-CONTRACT
10 permit ipv4 any 232.1.0.0 0.0.255.255
20 permit ipv4 any 239.1.0.0 0.0.255.255
multicast-routing
address-family ipv4
interface GigabitEthernet0/0/0/0
boundary EXCH-A-CONTRACT
ALGO1$ mdfeed sub (contracted A channel and the uncontracted 232.9.9.9)
A: rx 600 missing 0 in 0 gaps dup 0 reord 0
ROGUE: rx 0 missing 0 in 0 gaps dup 0 reord 0
EXCH$ mdfeed sub (exchange side asks for internal 239.100.1.1)
INTERNAL: rx 0 missing 0 in 0 gaps dup 0 reord 0
XR-A# show mrib route 232.9.9.9
(10.10.1.10,232.9.9.9) RPF nbr: 10.10.1.10 Flags: RPF
Up: 00:00:15
Outgoing Interface List
GigabitEthernet0/0/0/1 Flags: F NS, Up: 00:00:15
XR-A# show mrib route 232.1.1.1
(10.10.1.10,232.1.1.1) RPF nbr: 10.10.1.10 Flags: RPF
Up: 00:00:15
Incoming Interface List
GigabitEthernet0/0/0/0 Flags: A, Up: 00:00:15
Outgoing Interface List
GigabitEthernet0/0/0/1 Flags: F NS, Up: 00:00:15
XR-A# show mrib route 239.100.1.1
(*,239.100.1.1) RPF nbr: 10.0.0.3 Flags: C RPF
Up: 00:00:16
Incoming Interface List
GigabitEthernet0/0/0/2 Flags: A NS, Up: 00:00:16
Outgoing Interface List
GigabitEthernet0/0/0/0 Flags: LI, Up: 00:00:16
(10.20.30.11,239.100.1.1) RPF nbr: 10.0.0.3 Flags: RPF
Up: 00:01:30
Incoming Interface List
GigabitEthernet0/0/0/2 Flags: A, Up: 00:01:30
The contracted channel still arrives. The uncontracted (S,G) exists in XR-A’s MRIB but has no incoming interface: XR-A won’t accept it from the venue. The internal group lists the handoff interface with LI, local interest, and no F: XR-A knows a host there wants it and won’t forward it.
Takeaway: an explicit boundary on every extranet interface, permitting only contracted ranges, in both directions.
16:30 — The post-mortem
Six pages, no link failures. Here’s what I’d build so that none of them happen.
| Decision | My choice | Why |
|---|---|---|
| Exchange feeds | SSM, IGMPv3 on the hosts | No RP and no switchover; 0 of 10 cold joins lost messages |
| Venues that require ASM | RP at your edge; hold the shared tree only if the longer path is acceptable | The switchover lost messages in 3 of 10 cold joins |
| IGMPv2-only handlers | SSM mapping on the leaf, then replace the handler | An IGMPv2 join can’t name the source |
| The open | Pre-joined trees for the channels strategies need | First message in 3–11 ms pre-joined, against 69–717 ms cold |
| Subscriber ports | Size with B × (1 − Rout / Rin) | Bursts decide loss, not averages |
| A and B | Two planes, physical where possible, RPF pinned otherwise | The default hash made fate-sharing depend on the venue’s numbering |
| Internal many-to-many | BIDIR with a phantom RP | With 32 senders, ASM held 32 (S,G) entries per router; BIDIR held one |
| Internal ASM | Anycast RP | An RP failure cost 0 messages; a group joined after it worked first time |
| L2-only VLANs | A snooping querier, always | Memberships expire after three missed queries |
| Extranet edge | An explicit boundary permitting only contracted ranges |
Without it, traffic crossed in both directions |
| Monitoring | Per-feed gap counters, not only arbitrated ones | A dark plane is invisible to the strategies |
What would change my mind
- A venue that offers only ASM. Then the RP is theirs or yours, and holding the shared tree may be worth the longer path to avoid switchover gaps.
- Hardware results. The lab proves protocol behaviour. Switchover timing, DF re-election and buffer behaviour on a real ASIC are worth measuring on the platform you’ll run.
- More spines. Four spines don’t automatically separate A and B better: the hash still decides unless you pin it.
References
- CME Group, MDP 3.0 - Incremental Feed Arbitration and MDP 3.0 - Dissemination: Feed A and Feed B, arbitration by sequence number, recovery when a number is missing from both.
- Nasdaq, MoldUDP64 protocol specification: sequence numbers in every packet and re-requests for missed messages.
- RFC 7761, PIM Sparse Mode; RFC 4607, Source-Specific Multicast; RFC 4604, IGMPv3 and MLDv2 for SSM; RFC 5015, Bidirectional PIM; RFC 4610, Anycast-RP using PIM; RFC 3446, Anycast RP using PIM and MSDP.
- RFC 3376, IGMPv3; RFC 4541, IGMP and MLD snooping switches: unregistered traffic to router ports, optionally flooded; RFC 2365, administratively scoped multicast.
- Cisco, Nexus 9000 NX-OS Multicast Routing Configuration Guide 10.6(x): Configuring PIM and PIM6:
ip multicast multipathmodes and thes-g-hashdefault, Bidir platform support and limits, anycast-RP,spt-threshold infinity, SSM range. - Cisco, Nexus 9000 NX-OS Multicast Routing Configuration Guide 10.6(x): Configuring IGMP Snooping: the snooping querier requirement and membership expiry after three missed general queries.
- Linux, tc-tbf(8): the token-bucket filter used to emulate a slow egress port.
- Multicast from the ground up: the mechanisms behind every chapter, with captures from the same lab.
- Lab code, configurations and raw results: networkfactory-labs.
Halden Markets is fictional; the lab and its results are real. Configurations are illustrative: validate them on your own platforms and software versions in a lab before production use. Latency figures come from software forwarding in a virtual lab and don’t represent hardware. Cisco, IOS-XR, NX-OS and Nexus are trademarks of Cisco Systems, Inc. CME Group and MDP are trademarks of CME Group Inc. Nasdaq and MoldUDP64 are trademarks of Nasdaq, Inc.