At 06:30 on a Tuesday, the market data team at Halden Markets brings up the day’s feeds. By 16:30 the network team will have been paged six times, and not one of those pages will have been caused by a link going down.

Halden is fictional. Everything that happens to it below is real: each incident is a failure mode I built, ran and measured in a lab, on IOS-XR and NX-OS, with a small tool that publishes and checks sequence-numbered market data the way a feed handler does. The story is there to keep the order of things honest. In a trading firm, multicast problems don’t arrive as chapters of a textbook. They arrive as “the book is wrong” at 09:31.

New to multicast, or rusty on the details? Multicast from the ground up explains every mechanism used here (IGMP, PIM, RPs, SSM, BIDIR) with captures from the same lab, and the first mention of each term below links to it.

Follow the day from the feeds coming up to the post-mortem. Each chapter tells you what went wrong, explains the piece of multicast behind it, proves it in the lab with CLI output and packet captures, and ends with what I’d do about it.

In short

  • A/B redundancy is a property of paths, not feeds. With NX-OS’s default ECMP hash, whether A and B of a channel shared a spine depended on how the venue numbered its groups: none of 8 channels with one numbering, all 8 with another, and after a restart of the same fabric the answer changed again. A PIM failure that OSPF never saw cost both copies of two channels for 74 seconds. With an A plane and a B plane, the same failure cost nothing.
  • SSM is the right default for exchange feeds. Cold joins delivered the first message in 79–200 ms with no loss in 10 trials. ASM’s switch to the source tree lost one to seven messages right after the first in 3 of 10 trials.
  • The extranet boundary is a filter you must write. Without one, an uncontracted group came in and an internal group went out. One IOS-XR boundary stopped both.
  • No querier, no membership. In a VLAN without PIM or a snooping querier, the strategy’s membership expired between 150 and 250 seconds after it joined.
  • Pre-joined trees make subscriptions instant. First message in 3–11 ms instead of 69–717 ms.
  • Bursts, not averages, drop messages. The queue formula predicted 1,100 and 3,500 losses; the emulated port dropped 1,097 and 3,493.
  • BIDIR keeps state flat; Anycast RP makes RP failure a non-event. 32 senders cost ASM 32 entries per router and BIDIR one; losing an anycast RP cost nothing, and a phantom-RP failover cost a third of a second.

The cast: Halden’s network, built in the lab

The lab with its addresses. EXCH sends feed A from 10.10.1.10 to XR-A and feed B from 10.10.2.10 to XR-B, two IOS-XR edges (loopbacks 10.255.0.1 and .2; XR-A has the boundary, XR-B is the RP for 239.1.0.0/16). Both edges connect to two NX-OS spines, SPINE1 and SPINE2 (10.255.0.11 and .12), which both carry anycast RP 10.254.1.1 and the BIDIR RP link, /30 on SPINE1 and /29 on SPINE2. The spines connect to LEAF1 (10.255.0.21; VLAN 100 10.20.100.1/24, where it is PIM DR and IGMP querier, and VLAN 200 at Layer 2 only) and LEAF2 (10.255.0.22; VLAN 300). Routed links are 10.0.0.x/31 and each end shows its port and x: XR-A .0 to SPINE1 .1, XR-A .2 to SPINE2 .3, XR-B .4 to SPINE1 .5, XR-B .6 to SPINE2 .7, SPINE1 .8 to LEAF1 .9, SPINE2 .10 to LEAF1 .11, SPINE1 .12 to LEAF2 .13, SPINE2 .14 to LEAF2 .15. LEAF1 serves ALGO1 (10.20.100.11), ALGO2 behind the SLOW bridge (10.20.100.12) and the ticker plant TP1 (10.20.200.5); LEAF2 serves OMS1 (10.20.30.11)

Role Node Platform
The exchange EXCH: one Linux host, two NICs; feed A out of one, feed B out of the other, same sequence numbers Linux container
Extranet edges XR-A (handoff A), XR-B (handoff B; also the RP for the venue’s ASM groups) IOS-XR 7.11.1 (XRv 9000)
The fabric SPINE1, SPINE2, LEAF1, LEAF2 NX-OS 10.4(2) (Nexus 9000v)
Strategy servers ALGO1 (A/B arbiter), ALGO2 (behind SLOW, a Linux bridge with a token-bucket shaper) Linux containers
Internal publishers TP1 (ticker plant, L2-only VLAN 200), OMS1 (order management, on LEAF2) Linux containers

OSPF and PIM sparse mode run everywhere; SSM uses the default 232.0.0.0/8 range. Three RP arrangements sit side by side: the venue’s ASM range 239.1.0.0/16 uses XR-B as its RP, the internal ASM range 239.100.0.0/16 uses an anycast RP on both spines, and the BIDIR range 239.50.1.0/24 uses a phantom RP. The configurations, scripts and every raw result are in the lab repository.

The tool, mdfeed, is about 200 lines of Python. The publisher stamps every message with a sequence number and its send time and sends each one once per feed. The subscriber joins the way a feed handler does, source-specific for SSM and any-source for ASM, and reports what a handler’s gap detector would see: time from join to first message, gaps, duplicates and reordering, and with --arbitrate, the merged A/B stream where the first copy of each sequence number wins.

What a virtual lab can and can’t tell you. Nexus 9000v forwards in software. Above a few hundred packets per second it queues and then drops, so every test runs at 100 messages per second per feed or less, and latencies here are milliseconds that say nothing about real hardware. What the lab does prove is protocol behaviour: which tree carries a packet, what state a router builds, what a failure does to the sequence numbers and how long recovery takes. None of that depends on the ASIC.

06:30 — The feeds come up

Halden’s feed handlers start before anyone else is in. Each one subscribes to the channels its strategies need, A and B for every channel, and the network has to build a tree for each.

What’s happening: SSM, A/B feeds and sequence numbers

Every exchange message carries a sequence number. A feed handler checks each one against the last, and a missing number means the order book is wrong until the gap is repaired by a TCP replay or a snapshot. That’s why venues send everything twice. CME’s MDP 3.0 documentation spells out the arbitration it recommends: listen to Feed A and Feed B, process packets by sequence number, discard a number you’ve already processed, and treat a gap as a loss on both feeds, which is when recovery starts. Nasdaq’s MoldUDP64 takes the same idea a step further with a re-request server you ask for anything you missed.

For exchange feeds, the tree to build is SSM, Source-Specific Multicast (RFC 4607). The venue publishes the source address of every channel, so the receiver can ask for exactly (S,G): this source, this group. Its leaf sends a PIM join hop by hop towards S, and data flows down the shortest path from the very first packet. There’s no rendezvous point, no register and no switchover. 232.0.0.0/8 is reserved for SSM, and the host asks with IGMPv3 (RFC 4604), naming the source in the report.

Three multicast tree models side by side: SSM, where the receiver’s leaf joins (S,G) straight towards the source; ASM, where the first-hop router registers with the RP, receivers join a shared tree rooted at the RP, and the last-hop router then switches to the source tree; and BIDIR, where one (*,G) tree rooted at the RP link carries traffic from any source up towards the RP and down to the receivers, with one designated forwarder per link

In the lab

On the wire, the subscription is an IGMPv3 report naming the group and the source, sent twice (IGMP repeats unsolicited reports in case one is lost). When the subscriber leaves fifteen seconds later, it reports block source, and the querier checks with group-and-source-specific queries before pruning anything:

ALGO1 eth0: IGMP
    1   0.000000 10.20.100.11 → 224.0.0.22   IGMPv3 58 Membership Report / Group 232.1.1.1, new source {10.10.1.10}
    2   0.100004 10.20.100.11 → 224.0.0.22   IGMPv3 58 Membership Report / Group 232.1.1.1, new source {10.10.1.10}
    3  15.063918 10.20.100.11 → 224.0.0.22   IGMPv3 58 Membership Report / Group 232.1.1.1, block source {10.10.1.10}
    4  15.067428  10.20.100.1 → 232.1.1.1    IGMPv3 60 Membership Query, specific for group 232.1.1.1, source {10.10.1.10}
    5  15.999941 10.20.100.11 → 224.0.0.22   IGMPv3 58 Membership Report / Group 232.1.1.1, block source {10.10.1.10}
    6  16.003166  10.20.100.1 → 232.1.1.1    IGMPv3 60 Membership Query, specific for group 232.1.1.1, source {10.10.1.10}
    7  16.068365  10.20.100.1 → 232.1.1.1    IGMPv3 60 Membership Query, specific for group 232.1.1.1, source {10.10.1.10}
    8  17.004839  10.20.100.1 → 232.1.1.1    IGMPv3 60 Membership Query, specific for group 232.1.1.1, source {10.10.1.10}
Internet Group Management Protocol
    [IGMP Version: 3]
    Type: Membership Report (0x22)
    Reserved: 00
    Checksum: 0xe4e6 [correct]
    [Checksum Status: Good]
    Reserved: 0000
    Num Group Records: 1
    Group Record : 232.1.1.1  Allow New Sources
        Record Type: Allow New Sources (5)
        Aux Data Len: 0
        Num Src: 1
        Multicast Address: 232.1.1.1
        Source Address: 10.10.1.10

LEAF1 turns it into a PIM (S,G) join towards the source, sent only up the uplink its RPF check chose:

LEAF1 Ethernet1/1 (to SPINE1): PIM
    1   0.000000     10.0.0.9 → 224.0.0.13   PIMv2 68 Join/Prune
    3  17.061429     10.0.0.9 → 224.0.0.13   PIMv2 68 Join/Prune
Protocol Independent Multicast
    0010 .... = Version: 2
    .... 0011 = Type: Join/Prune (3)
    Reserved byte(s): 00
    Checksum: 0xd6cc [correct]
    [Checksum Status: Good]
    PIM Options
        Upstream-neighbor: 10.0.0.8
            Address Family: IPv4 (1)
            Encoding Type: Native (0)
            Unicast: 10.0.0.8
        Reserved byte(s): 00
        Num Groups: 1
        Holdtime: 210
        Group 0
            Group 0: 232.1.1.1/32
                Address Family: IPv4 (1)
                Encoding Type: Native (0)
                Flags: 0x00
                    0... .... = Bidirectional PIM: Not set
                    .000 000. = Reserved: 0x00
                    .... ...0 = Admin Scope Zone: Not set
                Masklen: 32
                Group: 232.1.1.1
            Num Joins: 1
                IP address: 10.10.1.10/32 (S)
                    Address Family: IPv4 (1)
                    Encoding Type: Native (0)
                    Flags: 0x04, Sparse
                        0000 0... = Reserved: 0x00
                        .... .1.. = Sparse: Set
                        .... ..0. = WildCard: Not set
                        .... ...0 = Rendezvous Point Tree: Not set
                    Masklen: 32
                    Source: 10.10.1.10
            Num Prunes: 0

And what arrives is the same message twice, one copy per feed, with identical sequence numbers in the first eight bytes of the payload:

ALGO1 eth0: feed A and feed B (UDP 30001); payload = 8-byte sequence number, 8-byte send time in ns
  0.000 s  10.10.1.10 -> 232.1.1.1  TTL 13  seq    93  sent at 1793205080685828081 ns  (000000000000005d18e2be8523b457f1...)
  0.001 s  10.10.2.10 -> 232.1.2.1  TTL 13  seq    93  sent at 1793205080685828081 ns  (000000000000005d18e2be8523b457f1...)
  0.199 s  10.10.1.10 -> 232.1.1.1  TTL 13  seq    94  sent at 1793205080885830543 ns  (000000000000005e18e2be852fa0238f...)
  0.200 s  10.10.2.10 -> 232.1.2.1  TTL 13  seq    94  sent at 1793205080885830543 ns  (000000000000005e18e2be852fa0238f...)
  0.399 s  10.10.1.10 -> 232.1.1.1  TTL 13  seq    95  sent at 1793205081085829878 ns  (000000000000005f18e2be853b8be2f6...)
  0.399 s  10.10.2.10 -> 232.1.2.1  TTL 13  seq    95  sent at 1793205081085829878 ns  (000000000000005f18e2be853b8be2f6...)
  0.598 s  10.10.2.10 -> 232.1.2.1  TTL 13  seq    96  sent at 1793205081285829990 ns  (000000000000006018e2be854777a566...)
  0.599 s  10.10.1.10 -> 232.1.1.1  TTL 13  seq    96  sent at 1793205081285829990 ns  (000000000000006018e2be854777a566...)
  0.802 s  10.10.1.10 -> 232.1.1.1  TTL 13  seq    97  sent at 1793205081485830177 ns  (000000000000006118e2be8553636821...)
  0.802 s  10.10.2.10 -> 232.1.2.1  TTL 13  seq    97  sent at 1793205081485830177 ns  (000000000000006118e2be8553636821...)
  1.001 s  10.10.1.10 -> 232.1.1.1  TTL 13  seq    98  sent at 1793205081685830212 ns  (000000000000006218e2be855f4f2a44...)
  1.002 s  10.10.2.10 -> 232.1.2.1  TTL 13  seq    98  sent at 1793205081685830212 ns  (000000000000006218e2be855f4f2a44...)

Takeaway: SSM keeps the exchange feeds simple: one join, one tree, nothing in the middle to fail. The rest of the day is about everything around it.

07:45 — A new venue, an old feed handler

Halden is onboarding a second venue. It publishes ASM groups and expects its customers to use an RP. On the same morning, a vendor’s feed handler for the first venue refuses to receive anything: it can only join with IGMPv2.

What’s happening: ASM and the switch to the source tree

Any-Source Multicast (RFC 7761) is the original PIM sparse-mode model. The receiver asks for a group without naming a source, so its router joins a shared tree towards a rendezvous point every router agrees on. The source’s first-hop router registers with the RP: it wraps the data in unicast and sends it to the RP, which then joins towards the source. Data flows down the shared tree. Then, by default, the receiver’s last-hop router learns the source address from the data, joins the shortest-path tree straight to the source, and prunes that source off the shared tree.

That switchover is where trading people should look. For a moment two trees exist and two copies can be in flight. NX-OS switches on the first packet unless ip pim spt-threshold infinity holds a list of groups on the shared tree, which avoids the switchover at the price of a longer path through the RP.

In the lab: the register on the wire

When the venue starts sending an ASM group, XR-A, the first-hop router, registers it with the RP, XR-B: the data travels inside unicast PIM Register messages. The RP answers with a Register-Stop, either because it has joined the source natively or because nobody wants the group yet. After that, XR-A doesn’t stop asking: just before its register-suppression timer runs out it sends a Null-Register, a Register with only a header and no data, so the RP can say whether it wants the traffic again. This capture caught the RP’s Register-Stop and one of those Null-Registers:

XR-A Gi0/0/0/1 (to SPINE1): PIM other than hellos
    5  17.400657     10.0.0.2 → 10.255.0.2   PIMv2 62 Register
XR-A Gi0/0/0/2 (to SPINE2): PIM other than hellos
    1   0.000000     10.0.0.3 → 224.0.0.13   PIMv2 68 Join/Prune
    3  12.188491   10.255.0.2 → 10.0.0.2     PIMv2 60 Register-stop
    5  17.052952     10.0.0.3 → 224.0.0.13   PIMv2 68 Join/Prune
Protocol Independent Multicast
    0010 .... = Version: 2
    .... 0001 = Type: Register (1)
    Reserved byte(s): 00
    Checksum: 0x9eff [correct]
    [Checksum Status: Good]
    PIM Options
        Flags: 0x40000000
            0... .... .... .... .... .... .... .... = Border: No
            .1.. .... .... .... .... .... .... .... = Null-Register: Yes
        0000 .... = IP Version: Dummy Header (0)
        Dummy HeaderIPv4
        Source: 10.10.1.10
        Group: 239.1.12.1

In the lab: what the switchover costs

The exchange was already publishing 50 messages per second per group when ALGO1 joined, the way a strategy subscribes mid-session. Every trial joined a group nobody had joined yet, so each join was cold.

mode: spt (LEAF1 default: switch to the SPT on the first packet)
kind  group         join->first msg   first seq   received   missing   duplicates   reordered
SSM   232.1.10.1          94.1 ms         520        497         0            0           0
SSM   232.1.10.2          85.3 ms        1394        487         0            0           0
SSM   232.1.10.3          88.7 ms        2251        495         0            0           0
SSM   232.1.10.4          86.2 ms        3108        497         0            0           0
SSM   232.1.10.5          97.2 ms        3966        497         0            0           0
ASM   239.1.10.1         137.3 ms        4825        484         0            0           0
ASM   239.1.10.2         110.7 ms        5662        489         7            0           0   gaps: seq 5663-5669 at 268 ms after join
ASM   239.1.10.3          88.9 ms        6518        497         0            0           0
ASM   239.1.10.4         230.2 ms        7382        490         0            0           0
ASM   239.1.10.5         137.0 ms        8234        495         0            0           0
mode: infinity (LEAF1 ip pim spt-threshold infinity for 239.1.0.0/16)
kind  group         join->first msg   first seq   received   missing   duplicates   reordered
ASM   239.1.11.1         103.4 ms         547        496         0            0           0
ASM   239.1.11.2         145.4 ms        1372        496         0            0           0
ASM   239.1.11.3         129.9 ms        2230        493         0            0           0
ASM   239.1.11.4         107.4 ms        3086        496         0            0           0
ASM   239.1.11.5         166.5 ms        3945        493         0            0           0
Design Join to first message Cold joins that lost messages
SSM 79–200 ms 0 of 10
ASM, switch to the source tree on the first packet (NX-OS default) 89–230 ms 3 of 10: one to seven messages right after the first
ASM, spt-threshold infinity (shared tree only) 103–167 ms 0 of 5

The ASM losses sit in exactly one place: the messages straight after the first one, within about 300 ms of the join, while the leaf moves from the shared tree to the source tree. Mid-switchover, LEAF1 holds both trees at once, the shared tree towards the RP through SPINE1 and the source tree through SPINE2, while XR-B, the RP, is still receiving the data as decapsulated registers:

LEAF1# show ip mroute 239.1.10.1
IP Multicast Routing Table for VRF "default"

(*, 239.1.10.1/32), uptime: 00:00:06, igmp ip pim 
  Incoming interface: Ethernet1/1, RPF nbr: 10.0.0.8
  Outgoing interface list: (count: 1)
    Vlan100, uptime: 00:00:06, igmp

(10.10.1.10/32, 239.1.10.1/32), uptime: 00:00:06, ip mrib pim 
  Incoming interface: Ethernet1/2, RPF nbr: 10.0.0.10
  Outgoing interface list: (count: 1)
    Vlan100, uptime: 00:00:06, mrib
XR-B# show mrib route 239.1.10.1

(*,239.1.10.1) RPF nbr: 10.255.0.2 Flags: C RPF
  Up: 00:00:07
  Incoming Interface List
    Decapstunnel0 Flags: A, Up: 00:00:07
  Outgoing Interface List
    GigabitEthernet0/0/0/1 Flags: F NS, Up: 00:00:07

(10.10.1.10,239.1.10.1) RPF nbr: 10.255.0.2 Flags: L RPF
  Up: 00:14:37
  Incoming Interface List
    Decapstunnel0 Flags: A, Up: 00:00:07

In the lab: the IGMPv2 feed handler

An IGMPv2 join can’t name a source, so a join for an SSM group is useless on its own. SSM mapping on the leaf (ip igmp ssm-translate) supplies the source the host couldn’t.

ALGO1 played the old handler: a small script sent genuine IGMPv2 reports for 232.1.1.1, with no source, while tcpdump counted what reached the host:

IGMPv2 host reports 232.1.1.1 (no source), without SSM mapping: 0 packets of 232.1.1.1 reached ALGO1 in 15 s
LEAF1# show ip igmp groups vlan 100
IGMP Connected Group Membership for Interface "Vlan100" - 0 total entries
Type: S - Static, D - Dynamic, L - Local, T - SSM Translated, H - Host Proxy
      * - Cache Only
Group Address      Type Interface              Uptime    Expires   Last Reporter
LEAF1# show ip mroute 232.1.1.1
IP Multicast Routing Table for VRF "default"

(*, 232.0.0.0/8), uptime: 00:48:21, pim ip 
  Incoming interface: Null, RPF nbr: 0.0.0.0
  Outgoing interface list: (count: 0)

IGMPv2 host reports 232.1.1.1 (no source), with ip igmp ssm-translate 232.1.1.0/24 10.10.1.10: 271 packets of 232.1.1.1 reached ALGO1 in 15 s
LEAF1# show ip igmp groups vlan 100
IGMP Connected Group Membership for Interface "Vlan100" - 1 total entries
Type: S - Static, D - Dynamic, L - Local, T - SSM Translated, H - Host Proxy
      * - Cache Only
Group Address      Type Interface              Uptime    Expires   Last Reporter
232.1.1.1
  10.10.1.10       DT   Vlan100                00:00:08  00:04:11  10.20.100.11
LEAF1# show ip mroute 232.1.1.1
IP Multicast Routing Table for VRF "default"

(10.10.1.10/32, 232.1.1.1/32), uptime: 00:00:08, igmp ip pim 
  Incoming interface: Ethernet1/1, RPF nbr: 10.0.0.8
  Outgoing interface list: (count: 1)
    Vlan100, uptime: 00:00:08, igmp

Without mapping, the leaf keeps no membership and builds no tree: an IGMPv2 join for an SSM group is ignored, and nothing arrives. With one line of SSM mapping, the same v2 join becomes an (S,G) membership, marked T for SSM-translated, and the feed flows.

Takeaway: use SSM wherever the venue publishes sources. Where you must run ASM, put the RP at your edge and decide deliberately between switching to the source tree (the default, and a gap risk at every new subscription) and holding the shared tree (no switchover, longer path). Fix IGMPv2-only handlers with SSM mapping, then fix the handlers.

08:59 — One minute to the open

A strategy that only trades the open subscribes to forty channels at 08:59:30. Every subscription builds a tree hop by hop from the leaf to the venue edge, all at once, all in the busiest minute of the day.

What’s happening: pre-joined trees

A tree that already exists when the strategy subscribes costs almost nothing: the data is at the leaf, and subscribing just adds a port to IGMP snooping. NX-OS’s ip igmp static-oif on the subscriber VLAN makes the leaf join on behalf of hosts that haven’t asked yet. The cost is bandwidth on every link the tree covers, all day long.

In the lab

Five channels were pre-joined on LEAF1 with ip igmp static-oif, five others left cold, all published at 50 messages per second. ALGO1 then subscribed to each in turn and timed the first message:

channel           kind        join->first msg
232.1.20.1        pre-joined       3.6 ms
232.1.21.1        cold            99.7 ms
232.1.20.2        pre-joined       4.2 ms
232.1.21.2        cold            69.3 ms
232.1.20.3        pre-joined       3.4 ms
232.1.21.3        cold           117.7 ms
232.1.20.4        pre-joined      11.3 ms
232.1.21.4        cold           717.4 ms
232.1.20.5        pre-joined     179.9 ms
232.1.21.5        cold           564.7 ms
LEAF1# show running-config interface Vlan100



interface Vlan100
  no shutdown
  ip address 10.20.100.1/24
  ip ospf passive-interface
  ip router ospf 1 area 0.0.0.0
  ip pim sparse-mode
  ip igmp version 3
  ip igmp static-oif 232.1.20.1 source 10.10.1.10
LEAF1# show ip mroute 232.1.20.1
IP Multicast Routing Table for VRF "default"

(10.10.1.10/32, 232.1.20.1/32), uptime: 00:00:21, static ip pim 
  Incoming interface: Ethernet1/1, RPF nbr: 10.0.0.8
  Outgoing interface list: (count: 1)
    Vlan100, uptime: 00:00:21, static

Pre-joined channels delivered the first message in 3 to 11 ms, with one outlier at 180 ms; cold channels took 69 to 717 ms, because every one of them needed a PIM join to travel to the venue edge and the data to travel back. The static entry shows up in the mroute table as static: the leaf is pulling the channel whether anyone is subscribed or not.

Takeaway: pre-join the channels your strategies need at the open. Bandwidth is cheap next to a slow first minute.

09:00 — The open

Average utilisation on the 10G link to Halden’s busiest strategy server sits at 30%. At 09:00:00.2, its feed handler logs a burst of gaps on both A and B.

What’s happening: queue arithmetic

Averages hide bursts. When a burst arrives faster than an egress port can send it, the difference queues, and whatever doesn’t fit in the buffer is dropped. A burst of B bytes arriving at Rin into a port that drains at Rout takes B/Rin to arrive, and meanwhile the port sends Rout × B/Rin. The queue peaks at:

Qpeak = B × (1 − Rout / Rin)

and anything above the buffer is lost. 1 MB arriving at line rate from a 25G source into a 10G port needs 600 kB of buffer; into a 1G port, 960 kB. The same arithmetic gives serialization delay, the time one packet occupies the wire: 1,000 bytes take 8 µs at 1G, 0.8 µs at 10G and 0.32 µs at 25G.

In the lab (a Linux emulation)

A virtual switch can’t show buffer behaviour, so this one is a Linux emulation and I’ll say so plainly. SLOW is a Linux bridge between LEAF1 and ALGO2, with a token-bucket filter (tc tbf) on its ALGO2-facing port: a 100 Mb/s egress port with a fixed buffer. The publisher runs on SLOW itself, so bursts arrive at host speed and meet the slow port directly. Each burst is N back-to-back 1,000-byte messages, every 500 ms for 10 seconds; the table puts the queue formula next to what the qdisc dropped and what the subscriber’s gap detector saw:

SLOW eth1: tbf rate 100mbit, 1000-byte messages, bursts every 500 ms for 10 s (Linux emulation of a slow egress port)
buffer B  burst msgs  bursts  R_in Mb/s  peak queue kB (model)  predicted drops  tbf dropped  ALGO2 missing  gap events
   64000          32      20        857                     30                0            0              0           0
   64000          64      20       1085                     61                0            0              0           0
   64000         128      20       1154                    122             1100         1097           1092          91
   64000         256      20       1318                    247             3500         3493           3491         265
  256000          32      20        851                     30                0            0              0           0
  256000          64      20       1053                     61                0            0              0           0
  256000         128      20       1105                    122                0            0              0           0
  256000         256      20       1131                    244                0            0              0           0
SLOW# tc -s qdisc show dev eth1   (buffer 64000 B, bursts of 256)
qdisc tbf 8004: root refcnt 2 rate 100Mbit burst 3000b lat 4.88ms 
 Sent 1696258 bytes 1643 pkt (dropped 3493, overlimits 6590 requeues 0) 
 backlog 0b 0p requeues 0

The model predicted 1,100 and 3,500 lost messages for the two bursts that overflow a 64 kB buffer; the queue dropped 1,097 and 3,493, and the subscriber counted 1,092 and 3,491 missing. Quadruple the buffer and the same bursts lose nothing. The averages never changed: 20 bursts in 10 seconds is a few megabits per second on a 100 Mb/s port.

Takeaway: size buffers and port speeds for the burst, not the average. Do the arithmetic before you buy the switch.

10:15 — The spine that didn’t fail

At 10:15 the A and B feeds for two of Halden’s channels stop at the same moment. Nothing is down. Both spines answer pings, OSPF is full everywhere, and the arbitrated stream for those two channels has a 74-second hole in it. The other channels didn’t notice anything.

What’s happening: ECMP decides where A and B go

A/B redundancy is a property of the path, not of the feed. In an ECMP fabric, the path of each (S,G) is picked by a hash. NX-OS’s default, ip multicast multipath s-g-hash, picks the RPF interface from the source, the group and the next hops, and A and B of the same channel have different sources and groups, so they land wherever the hash puts them.

In the lab: the venue’s numbering decides

With the publisher idle, ALGO1 joined channels 1 to 8 of feed A (232.1.1.i) and four possible numberings of feed B, and I read which spine LEAF1 chose for each (S,G):

Feed B numbering Channels where A and B share a spine
232.1.2.i none
232.1.3.i all 8
232.2.1.i 6 of 8
232.1.129.i none

Same fabric, same default, same sources. Only the venue’s numbering of the B groups differs, and it moves fate-sharing from none to all. You can’t design around a hash you don’t control.

It doesn’t even hold still. Three weeks later I restarted the lab for another project and read the same placements again, with the same configuration, sources and groups. With B on 232.1.2.i, the numbering that had shared nothing, channel 1’s A and B both crossed SPINE2. With B on 232.1.3.i, the one that had shared everything, three of the four channels I checked still shared a spine and one didn’t. I didn’t chase why the hash moved; this is NX-OSv, and I’d check on hardware before reading anything into the detail. The lesson doesn’t depend on it: a spine assignment you checked once is an observation, not a design.

In the lab: a failure routing doesn’t see

With B numbered 232.1.3.i, channels 1 and 3 hashed entirely onto SPINE1:

LEAF1# show ip mroute  (RPF interface per (S,G), filtered)
  (10.10.1.10, 232.1.1.1)  RPF Ethernet1/1 -> SPINE1
  (10.10.2.10, 232.1.3.1)  RPF Ethernet1/1 -> SPINE1
  (10.10.1.10, 232.1.1.2)  RPF Ethernet1/2 -> SPINE2
  (10.10.2.10, 232.1.3.2)  RPF Ethernet1/2 -> SPINE2
  (10.10.1.10, 232.1.1.3)  RPF Ethernet1/1 -> SPINE1
  (10.10.2.10, 232.1.3.3)  RPF Ethernet1/1 -> SPINE1
  (10.10.1.10, 232.1.1.4)  RPF Ethernet1/2 -> SPINE2
  (10.10.2.10, 232.1.3.4)  RPF Ethernet1/2 -> SPINE2

Then PIM broke on SPINE1’s link to LEAF1 (no ip pim sparse-mode) while OSPF stayed up. This is the class of failure that hurts most: a PIM process problem, a policy, a configuration change, anything that breaks multicast while unicast looks healthy. LEAF1 lost its PIM neighbour on that link, but RPF follows unicast routing, and unicast still pointed at SPINE1:

LEAF1# show ip pim neighbor
PIM Neighbor Status for VRF "default"
Neighbor        Interface            Uptime    Expires   DR       Bidir-  BFD    ECMP Redirect
                                                         Priority Capable State     Capable
10.0.0.10       Ethernet1/2          00:14:34  00:01:28  1        yes     n/a     no
LEAF1# show ip ospf neighbors
 OSPF Process ID 1 VRF default
 Total number of neighbors: 2
 Neighbor ID     Pri State            Up Time  Address         Interface
 10.255.0.11       1 FULL/ -          00:24:56 10.0.0.8        Eth1/1 
 10.255.0.12       1 FULL/ -          00:14:28 10.0.0.10       Eth1/2
ALGO1$ mdfeed sub --arbitrate
              A1: rx    2365  missing  1885 in 1 gaps  dup 0  reord 0
                 gap seq 579-2463 (1885 msgs) seen at t+103.443 s
              B1: rx    2392  missing  1858 in 1 gaps  dup 0  reord 0
                 gap seq 579-2436 (1858 msgs) seen at t+102.358 s
              A2: rx    4250  missing     0 in 0 gaps  dup 0  reord 0
              B2: rx    4250  missing     0 in 0 gaps  dup 0  reord 0
              A3: rx    2464  missing  1786 in 1 gaps  dup 0  reord 0
                 gap seq 579-2364 (1786 msgs) seen at t+99.478 s
              B3: rx    2492  missing  1758 in 1 gaps  dup 0  reord 0
                 gap seq 579-2336 (1758 msgs) seen at t+98.361 s
              A4: rx    4250  missing     0 in 0 gaps  dup 0  reord 0
              B4: rx    4250  missing     0 in 0 gaps  dup 0  reord 0
           A/B 1: rx    4757  missing  1858 in 1 gaps  dup 2365  reord 0
                 gap seq 579-2436 (1858 msgs) seen at t+102.358 s
           A/B 2: rx    8500  missing     0 in 0 gaps  dup 4250  reord 0
           A/B 3: rx    4956  missing  1758 in 1 gaps  dup 2464  reord 0
                 gap seq 579-2336 (1758 msgs) seen at t+98.361 s
           A/B 4: rx    8500  missing     0 in 0 gaps  dup 4250  reord 0

Channels 1 and 3 lost both copies of the same 1,858 consecutive messages, about 74 seconds at 25 per second, and the arbiter could do nothing: the merged stream has the same gap. Channels 2 and 4 didn’t lose a message.

In the lab: two planes

Two lines on LEAF1 pin every A source to SPINE1 and every B source to SPINE2:

ip mroute 10.10.1.0/24 10.0.0.8
ip mroute 10.10.2.0/24 10.0.0.10

The same failure:

ALGO1$ mdfeed sub --arbitrate
              A1: rx    1741  missing  2509 in 1 gaps  dup 0  reord 0
                 gap seq 577-3085 (2509 msgs) seen at t+128.352 s
              B1: rx    4250  missing     0 in 0 gaps  dup 0  reord 0
              A2: rx    1785  missing  2465 in 1 gaps  dup 0  reord 0
                 gap seq 577-3041 (2465 msgs) seen at t+126.59 s
              B2: rx    4250  missing     0 in 0 gaps  dup 0  reord 0
              A3: rx    1837  missing  2413 in 1 gaps  dup 0  reord 0
                 gap seq 577-2989 (2413 msgs) seen at t+124.51 s
              B3: rx    4250  missing     0 in 0 gaps  dup 0  reord 0
              A4: rx    1890  missing  2360 in 1 gaps  dup 0  reord 0
                 gap seq 577-2936 (2360 msgs) seen at t+122.389 s
              B4: rx    4250  missing     0 in 0 gaps  dup 0  reord 0
           A/B 1: rx    5991  missing     0 in 0 gaps  dup 1741  reord 0
           A/B 2: rx    6035  missing     0 in 0 gaps  dup 1785  reord 0
           A/B 3: rx    6087  missing     0 in 0 gaps  dup 1837  reord 0
           A/B 4: rx    6140  missing     0 in 0 gaps  dup 1890  reord 0

Feed A went dark on every channel, feed B carried every message, and the merged streams lost nothing.

Two panels: with the default ECMP hash and B numbered 232.1.3.x, A and B of channels 1 and 3 both cross SPINE1, so a PIM failure on SPINE1 loses both copies for about 74 seconds; with an A plane through SPINE1 and a B plane through SPINE2, the same failure loses feed A but not feed B, and the arbitrated stream loses nothing

Three more things these runs showed:

  • A dark plane is silent. With planes, losing A cost the strategies nothing, which is the point, and also the danger: Halden is now running on one feed and nobody has noticed. Alarm on per-feed gaps, not only arbitrated ones.
  • Recovery isn’t instant when the fault clears. After PIM came back, the hashed channels took about 19 seconds to recover and the pinned A plane about 40. PIM rebuilds on its periodic join refresh: plan on tens of seconds.
  • Clean failures are fast. When SPINE1 shut its port to LEAF1 instead (LEAF1’s side kept carrier), OSPF’s two-way check removed the link from LEAF1’s SPF within milliseconds, PIM followed, and the colliding channels lost 9 messages on both feeds. Still an arbitrated gap, just a short one. It’s the partial failures that last.

Takeaway: build an A plane and a B plane, physically if you can. Where A and B must share devices, pin RPF per plane instead of trusting the hash, and monitor each feed on its own.

11:30 — The order bus

Halden’s order-management system publishes order and position updates to every strategy, and every strategy publishes its own state back. Two hundred senders, all on the same few groups. The fabric team notices the multicast tables on the spines growing every time a strategy is added.

What’s happening: BIDIR and the phantom RP

In ASM, every sender becomes an (S,G) entry on every router along its path, with its own register exchange. Bidirectional PIM (RFC 5015) builds a single (*,G) tree rooted at the RP address instead, and data flows both up and down it: no registers, no source trees, no per-source state. On each link, routers elect a designated forwarder (DF) per RP address, and only the DF forwards traffic towards the RP.

The RP address doesn’t even have to exist. In the phantom RP design it’s an unused address on a subnet, the RP link. Two routers advertise that subnet with different mask lengths (here SPINE1 a /30, SPINE2 a /29); longest match makes SPINE1’s link primary, and if it disappears, SPINE2’s takes over. The price is the path: traffic always heads towards the RP link before it comes back down.

In the lab

OMS1 sent from 32 source addresses at once, to one ASM group (239.100.1.50, anycast RP) and to one BIDIR group (239.50.1.50, phantom RP), while ALGO1 subscribed to both. The multicast table on every router told the story, summarised from show ip mroute <group> summary:

LEAF2   ASM   239.100.1.50/32    source entries:  32
LEAF2   BIDIR 239.50.1.0/24      source entries:   0
SPINE1  ASM   239.100.1.50/32    source entries:  32
SPINE1  BIDIR 239.50.1.50/32     source entries:   0
SPINE2  ASM   239.100.1.50/32    source entries:  32
SPINE2  BIDIR 239.50.1.0/24      source entries:   0
LEAF1   ASM   239.100.1.50/32    source entries:  32
LEAF1   BIDIR 239.50.1.50/32     source entries:   0

Every ASM router on the path holds one (S,G) per sender, so the table grows with every engine you add. BIDIR holds one entry for the whole range and none per source: it doesn’t care how many senders there are. Both delivered every message.

SPINE1-LEAF1 link: PIM DF election messages (time, sender, subtype, RP, route metric to the RP)
 20.452 s  10.0.0.9    Winner   RP 10.254.0.1  metric 110/41
 21.550 s  10.0.0.8    Offer    RP 10.254.0.1  metric 0/0
 21.552 s  10.0.0.9    Backoff  RP 10.254.0.1  metric 110/41
 21.847 s  10.0.0.9    Offer    RP 10.254.0.1  metric infinite
 21.909 s  10.0.0.8    Offer    RP 10.254.0.1  metric 0/0
 22.010 s  10.0.0.8    Offer    RP 10.254.0.1  metric 0/0
 22.110 s  10.0.0.8    Offer    RP 10.254.0.1  metric 0/0
 22.151 s  10.0.0.8    Winner   RP 10.254.0.1  metric 0/0
SPINE2-LEAF1 link: PIM DF election messages (time, sender, subtype, RP, route metric to the RP)
 25.849 s  10.0.0.10   Winner   RP 10.254.0.1  metric 0/0
 27.860 s  10.0.0.11   Offer    RP 10.254.0.1  metric 110/41
 27.862 s  10.0.0.10   Winner   RP 10.254.0.1  metric 0/0
 27.881 s  10.0.0.10   Winner   RP 10.254.0.1  metric 110/42
 27.983 s  10.0.0.11   Offer    RP 10.254.0.1  metric 110/41
 27.986 s  10.0.0.10   Backoff  RP 10.254.0.1  metric 110/42
 28.957 s  10.0.0.10   Pass     RP 10.254.0.1  metric 110/42
 28.962 s  10.0.0.11   Winner   RP 10.254.0.1  metric 110/41
Protocol Independent Multicast
    0010 .... = Version: 2
    .... 1010 = Type: DF election (10)
    0010 .... = DF Subtype: DF Winner (2)
    .... 0000 = DF reserved: 0
    Checksum: 0xc9e0 [correct]
    [Checksum Status: Good]
    PIM Options
        RP: 10.254.0.1
            Address Family: IPv4 (1)
            Encoding Type: Native (0)
            Unicast: 10.254.0.1
        DF Metric Preference: 0
        Metric: 0

On the wire, the DF election is a short conversation per link. The capture covers both halves of the test: SPINE1’s /30 disappearing and coming back. On the SPINE2–LEAF1 link, SPINE2 (10.0.0.10) had been the Winner while it owned the only RP link. When SPINE1’s /30 returned, LEAF1 (10.0.0.11) suddenly had the better route to the RP and sent an Offer; SPINE2 answered with a Backoff, then a Pass, handing the role to LEAF1, which announced itself Winner. A metric of infinite is a router saying it has no route to the RP at all. One DF per link, re-elected by metric whenever the route to the RP changes: that’s all BIDIR needs to stay loop-free without any per-source state.

Then the phantom RP failover: OMS1 sent 50 messages per second on the bus while SPINE1 withdrew its /30 RP link. Before, LEAF1 reaches the RP link through SPINE1 and its uplink to SPINE1 is the RPF interface:

LEAF1# show ip route 10.254.0.0/24 longer-prefixes
IP Route Table for VRF "default"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>

10.254.0.0/29, ubest/mbest: 1/0
    *via 10.0.0.10, Eth1/2, [110/41], 00:19:31, ospf-1, intra
10.254.0.0/30, ubest/mbest: 1/0
    *via 10.0.0.8, Eth1/1, [110/41], 00:19:32, ospf-1, intra
LEAF1# show ip pim df 10.254.0.1
Bidir-PIM Designated Forwarder Information for VRF "default"

RP Address (ordinal)   RP Metric        Group Range
10.254.0.1 (1)         [110/41]         239.50.1.0/24

  Interface            DF Address       DF State   DF Metric    DF Uptime
  Vlan100              10.20.100.1      Winner     [110/41]     00:19:31
  Lo0                  10.255.0.21      Winner     [110/41]     00:19:31
  Eth1/1               10.0.0.8         Loser      [0/0]        00:19:43  (RPF)
  Eth1/2               10.0.0.11        Winner     [110/41]     00:19:31

Twenty seconds after the /30 disappeared, only SPINE2’s /29 is left, the RPF interface has moved to SPINE2, and every link has a new designated forwarder:

LEAF1# show ip route 10.254.0.0/24 longer-prefixes
IP Route Table for VRF "default"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>

10.254.0.0/29, ubest/mbest: 1/0
    *via 10.0.0.10, Eth1/2, [110/41], 00:20:02, ospf-1, intra
LEAF1# show ip pim df 10.254.0.1
Bidir-PIM Designated Forwarder Information for VRF "default"

RP Address (ordinal)   RP Metric        Group Range
10.254.0.1 (1)         [110/41]         239.50.1.0/24

  Interface            DF Address       DF State   DF Metric    DF Uptime
  Vlan100              10.20.100.1      Winner     [110/41]     00:20:02
  Lo0                  10.255.0.21      Winner     [110/41]     00:20:02
  Eth1/1               10.0.0.9         Winner     [110/41]     00:00:24
  Eth1/2               10.0.0.10        Loser      [0/0]        00:00:25  (RPF)
SPINE2# show ip pim df 10.254.0.1
Bidir-PIM Designated Forwarder Information for VRF "default"

RP Address (ordinal)   RP Metric        Group Range
10.254.0.1 (1)         [0/0]            239.50.1.0/24

  Interface            DF Address       DF State   DF Metric    DF Uptime
  Lo0                  10.255.0.12      Winner     [0/0]        00:27:00
  Lo1                  10.254.1.1       Winner     [0/0]        00:27:00
  Lo2                  0.0.0.0          Loser      [0/0]        00:00:27  (RPF)
  Eth1/1               10.0.0.3         Winner     [0/0]        00:00:27
  Eth1/2               10.0.0.7         Winner     [0/0]        00:00:26
  Eth1/3               10.0.0.10        Winner     [0/0]        00:00:27
  Eth1/4               10.0.0.14        Winner     [0/0]        00:00:27
ALGO1$ mdfeed sub
             BUS: rx    2483  missing    17 in 1 gaps  dup 0  reord 0
                 gap seq 1127-1143 (17 msgs) seen at t+27.783 s

The switch from the /30 to the /29 cost 17 messages, about a third of a second at that rate: the time for OSPF to withdraw the route and for every link to re-elect its DF. Fast, but not free, which is why the RP link belongs on the most stable devices you have.

Takeaway: for many-to-many groups, BIDIR keeps the state flat however many senders join. Put the RP link on the spines everyone crosses anyway, so the path stretch costs nothing.

12:40 — The RP goes away

Halden’s internal ASM groups use the spines as RPs. At 12:40, SPINE1 is taken out for maintenance.

What’s happening: Anycast RP

With ASM, the RP is a dependency for every new source and receiver. Anycast RP puts the same RP address on several routers. Unicast routing takes each register and join to the nearest one, and the RPs tell each other about the sources they’ve learnt, with MSDP (RFC 3446) or, as NX-OS’s ip pim anycast-rp does, with PIM itself (RFC 4610).

In the lab

Both spines carry 10.254.1.1 and know each other as anycast-RP peers. OMS1 published two groups; ALGO1 was already subscribed to the first when SPINE1 lost its RP address (interface loopback1 shut), and ALGO2 subscribed to the second only after the failure:

LEAF1# show ip route 10.254.1.1
IP Route Table for VRF "default"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>

10.254.1.1/32, ubest/mbest: 1/0
    *via 10.0.0.10, Eth1/2, [110/41], 00:16:52, ospf-1, intra
SPINE2# show ip mroute 239.100.1.2
IP Multicast Routing Table for VRF "default"

(10.20.30.11/32, 239.100.1.2/32), uptime: 00:00:44, pim ip 
  Incoming interface: Ethernet1/4, RPF nbr: 10.0.0.15, internal
  Outgoing interface list: (count: 0)
ALGO1$ existing subscriber
        EXISTING: rx    1581  missing     0 in 0 gaps  dup 0  reord 0
ALGO2$ new group joined after the failure
             NEW: rx     697  missing     0 in 0 gaps  dup 0  reord 0

The existing stream didn’t lose a message, and the subscription made after the failure delivered its first message in 68 ms with no loss. The RP address simply moved to SPINE2, which now holds the shared tree and the source state for the new group.

Takeaway: if you run internal ASM, run Anycast RP. An RP failure should be a non-event.

14:05 — The ticker plant goes quiet

Halden’s ticker plant republishes normalised prices to the strategies in its own VLAN. No routing is needed, so nobody enabled PIM on that VLAN. In testing it worked. In production, strategies started reporting a dead feed a few minutes after each restart.

What’s happening: IGMP snooping needs a querier

A switch running IGMP snooping forwards a group only to the ports that asked for it, and it learns that from IGMP reports. Hosts send reports when they join, then only in answer to queries. Without a querier, nothing asks, and the switch eventually forgets. Cisco’s NX-OS guide is explicit on both halves: snooping expires a membership after three missed general queries, and when PIM isn’t enabled on the VLAN, you must configure an IGMP snooping querier.

What happens to the traffic once the membership is gone depends on the switch. RFC 4541, the IGMP snooping considerations, calls it an unregistered packet: a switch must forward it on ports where a multicast router is attached, and may flood it to every port. In a VLAN with no router, that’s the difference between a feed that dies and a feed that suddenly reaches every server in the VLAN.

In the lab

ALGO1 joined once. LEAF1’s snooping table over time:

LEAF1# show ip igmp snooping groups vlan 200      (t = 20 s into the feed; ALGO1 joined 3 s before it started)
200   239.200.1.1        v3   D     Eth1/6
LEAF1# show ip igmp snooping groups vlan 200      (t = 150 s into the feed; ALGO1 joined 3 s before it started)
200   239.200.1.1        v3   D     Eth1/6
LEAF1# show ip igmp snooping groups vlan 200      (t = 250 s into the feed; ALGO1 joined 3 s before it started)
(no entries)
LEAF1# show ip igmp snooping groups vlan 200      (t = 300 s into the feed; ALGO1 joined 3 s before it started)
(no entries)
LEAF1# show ip igmp snooping groups vlan 200      (t = 380 s into the feed; ALGO1 joined 3 s before it started)
(no entries)

The membership was there at 150 seconds and gone by 250. On the wire, the VLAN shows ALGO1’s reports when it joins and then silence: nobody ever asks again.

ALGO1 eth1 (VLAN 200), 150 s, no querier: IGMP
    1   0.000000 10.20.200.11 → 224.0.0.22   IGMPv3 54 Membership Report / Join group 239.200.1.1 for any sources
    2   0.823944 10.20.200.11 → 224.0.0.22   IGMPv3 54 Membership Report / Join group 239.200.1.1 for any sources
    3 150.067969 10.20.200.11 → 224.0.0.22   IGMPv3 54 Membership Report / Leave group 239.200.1.1
    4 150.359976 10.20.200.11 → 224.0.0.22   IGMPv3 54 Membership Report / Leave group 239.200.1.1
ALGO1 eth1 (VLAN 200), 150 s, snooping querier 10.20.200.254: IGMP
    1   0.000000 10.20.200.11 → 224.0.0.22   IGMPv3 54 Membership Report / Join group 239.200.1.1 for any sources
    2   0.456027 10.20.200.11 → 224.0.0.22   IGMPv3 54 Membership Report / Join group 239.200.1.1 for any sources
    3  20.577354 10.20.200.254 → 224.0.0.1    IGMPv3 60 Membership Query, general
    4  20.579130 10.20.200.254 → 224.0.0.1    IGMPv3 60 Membership Query, general
    5  28.808028 10.20.200.11 → 224.0.0.22   IGMPv3 54 Membership Report / Join group 239.200.1.1 for any sources
    6 144.176656 10.20.200.254 → 224.0.0.1    IGMPv3 60 Membership Query, general
    7 144.179095 10.20.200.254 → 224.0.0.1    IGMPv3 60 Membership Query, general
    8 144.616063 10.20.200.11 → 224.0.0.22   IGMPv3 54 Membership Report / Join group 239.200.1.1 for any sources
    9 150.072050 10.20.200.11 → 224.0.0.22   IGMPv3 54 Membership Report / Leave group 239.200.1.1
   10 150.920023 10.20.200.11 → 224.0.0.22   IGMPv3 54 Membership Report / Leave group 239.200.1.1

With a snooping querier configured on the VLAN (vlan configuration 200 / ip igmp snooping querier 10.20.200.254), the same test keeps its membership, refreshed by every query:

LEAF1# show ip igmp snooping groups vlan 200      (t = 20 s into the feed)
200   239.200.1.1        v3   D     Eth1/6
LEAF1# show ip igmp snooping querier vlan 200
200   10.20.200.254    v3        00:01:57     Switch querier
LEAF1# show ip igmp snooping groups vlan 200      (t = 150 s into the feed)
200   239.200.1.1        v3   D     Eth1/6
LEAF1# show ip igmp snooping querier vlan 200
200   10.20.200.254    v3        00:01:50     Switch querier
LEAF1# show ip igmp snooping groups vlan 200      (t = 250 s into the feed)
200   239.200.1.1        v3   D     Eth1/6
LEAF1# show ip igmp snooping querier vlan 200
200   10.20.200.254    v3        00:00:10     Switch querier
LEAF1# show ip igmp snooping groups vlan 200      (t = 300 s into the feed)
200   239.200.1.1        v3   D     Eth1/6
LEAF1# show ip igmp snooping querier vlan 200
200   10.20.200.254    v3        00:01:25     Switch querier
LEAF1# show ip igmp snooping groups vlan 200      (t = 380 s into the feed)
200   239.200.1.1        v3   D     Eth1/6
LEAF1# show ip igmp snooping querier vlan 200
200   10.20.200.254    v3        00:00:04     Switch querier

A limit of the virtual switch. Nexus 9000v’s software data plane floods multicast in the VLAN regardless of the snooping table: a port that never joined received all 8,200 packets. So the lab shows the membership expiring exactly as documented, but not what a hardware switch does with the traffic afterwards. That part depends on the platform’s unregistered-multicast behaviour.

Takeaway: every VLAN with IGMP snooping needs a querier: PIM on the SVI, or ip igmp snooping querier under vlan configuration.

15:30 — Something extra from the venue

A strategy server starts receiving a group nobody at Halden has heard of. The venue had started publishing a test channel, and one of Halden’s developers had subscribed to it. Worse, the post-mortem finds that a host on the venue side of the handoff could have joined Halden’s internal order bus.

What’s happening: the extranet boundary

The handoff to a venue is a multicast border between two organisations. Accept only the channels in the contract, send nothing out, and write it down as a boundary on the edge interface, not as an assumption about what the venue will send or what your hosts will ask for.

In the lab

The exchange published the contracted channel 232.1.1.1 and an uncontracted one, 232.9.9.9; ALGO1 asked for both. Meanwhile OMS1 published an internal group, 239.100.1.1, and a host on the exchange side of XR-A asked for it. Without a boundary, everything flowed:

ALGO1$ mdfeed sub (contracted A channel and the uncontracted 232.9.9.9)
               A: rx     600  missing     0 in 0 gaps  dup 0  reord 0
           ROGUE: rx     600  missing     0 in 0 gaps  dup 0  reord 0
EXCH$ mdfeed sub (exchange side asks for internal 239.100.1.1)
        INTERNAL: rx     885  missing     0 in 0 gaps  dup 0  reord 0
XR-A# show mrib route 232.9.9.9

(10.10.1.10,232.9.9.9) RPF nbr: 10.10.1.10 Flags: RPF
  Up: 00:00:15
  Incoming Interface List
    GigabitEthernet0/0/0/0 Flags: A, Up: 00:00:15
  Outgoing Interface List
    GigabitEthernet0/0/0/1 Flags: F NS, Up: 00:00:15
XR-A# show mrib route 232.1.1.1

(10.10.1.10,232.1.1.1) RPF nbr: 10.10.1.10 Flags: RPF
  Up: 00:00:15
  Incoming Interface List
    GigabitEthernet0/0/0/0 Flags: A, Up: 00:00:15
  Outgoing Interface List
    GigabitEthernet0/0/0/1 Flags: F NS, Up: 00:00:15
XR-A# show mrib route 239.100.1.1

(*,239.100.1.1) RPF nbr: 10.0.0.3 Flags: C RPF
  Up: 00:00:16
  Incoming Interface List
    GigabitEthernet0/0/0/2 Flags: A NS, Up: 00:00:16
  Outgoing Interface List
    GigabitEthernet0/0/0/0 Flags: F NS LI, Up: 00:00:16

(10.20.30.11,239.100.1.1) RPF nbr: 10.0.0.3 Flags: RPF
  Up: 00:00:16
  Incoming Interface List
    GigabitEthernet0/0/0/2 Flags: A, Up: 00:00:16
  Outgoing Interface List
    GigabitEthernet0/0/0/0 Flags: F NS, Up: 00:00:16

With a boundary on XR-A’s handoff that permits only the contracted ranges:

ipv4 access-list EXCH-A-CONTRACT
 10 permit ipv4 any 232.1.0.0 0.0.255.255
 20 permit ipv4 any 239.1.0.0 0.0.255.255
multicast-routing
 address-family ipv4
  interface GigabitEthernet0/0/0/0
   boundary EXCH-A-CONTRACT
ALGO1$ mdfeed sub (contracted A channel and the uncontracted 232.9.9.9)
               A: rx     600  missing     0 in 0 gaps  dup 0  reord 0
           ROGUE: rx       0  missing     0 in 0 gaps  dup 0  reord 0
EXCH$ mdfeed sub (exchange side asks for internal 239.100.1.1)
        INTERNAL: rx       0  missing     0 in 0 gaps  dup 0  reord 0
XR-A# show mrib route 232.9.9.9

(10.10.1.10,232.9.9.9) RPF nbr: 10.10.1.10 Flags: RPF
  Up: 00:00:15
  Outgoing Interface List
    GigabitEthernet0/0/0/1 Flags: F NS, Up: 00:00:15
XR-A# show mrib route 232.1.1.1

(10.10.1.10,232.1.1.1) RPF nbr: 10.10.1.10 Flags: RPF
  Up: 00:00:15
  Incoming Interface List
    GigabitEthernet0/0/0/0 Flags: A, Up: 00:00:15
  Outgoing Interface List
    GigabitEthernet0/0/0/1 Flags: F NS, Up: 00:00:15
XR-A# show mrib route 239.100.1.1

(*,239.100.1.1) RPF nbr: 10.0.0.3 Flags: C RPF
  Up: 00:00:16
  Incoming Interface List
    GigabitEthernet0/0/0/2 Flags: A NS, Up: 00:00:16
  Outgoing Interface List
    GigabitEthernet0/0/0/0 Flags: LI, Up: 00:00:16

(10.20.30.11,239.100.1.1) RPF nbr: 10.0.0.3 Flags: RPF
  Up: 00:01:30
  Incoming Interface List
    GigabitEthernet0/0/0/2 Flags: A, Up: 00:01:30

The contracted channel still arrives. The uncontracted (S,G) exists in XR-A’s MRIB but has no incoming interface: XR-A won’t accept it from the venue. The internal group lists the handoff interface with LI, local interest, and no F: XR-A knows a host there wants it and won’t forward it.

Takeaway: an explicit boundary on every extranet interface, permitting only contracted ranges, in both directions.

16:30 — The post-mortem

Six pages, no link failures. Here’s what I’d build so that none of them happen.

Decision My choice Why
Exchange feeds SSM, IGMPv3 on the hosts No RP and no switchover; 0 of 10 cold joins lost messages
Venues that require ASM RP at your edge; hold the shared tree only if the longer path is acceptable The switchover lost messages in 3 of 10 cold joins
IGMPv2-only handlers SSM mapping on the leaf, then replace the handler An IGMPv2 join can’t name the source
The open Pre-joined trees for the channels strategies need First message in 3–11 ms pre-joined, against 69–717 ms cold
Subscriber ports Size with B × (1 − Rout / Rin) Bursts decide loss, not averages
A and B Two planes, physical where possible, RPF pinned otherwise The default hash made fate-sharing depend on the venue’s numbering
Internal many-to-many BIDIR with a phantom RP With 32 senders, ASM held 32 (S,G) entries per router; BIDIR held one
Internal ASM Anycast RP An RP failure cost 0 messages; a group joined after it worked first time
L2-only VLANs A snooping querier, always Memberships expire after three missed queries
Extranet edge An explicit boundary permitting only contracted ranges Without it, traffic crossed in both directions
Monitoring Per-feed gap counters, not only arbitrated ones A dark plane is invisible to the strategies

What would change my mind

  • A venue that offers only ASM. Then the RP is theirs or yours, and holding the shared tree may be worth the longer path to avoid switchover gaps.
  • Hardware results. The lab proves protocol behaviour. Switchover timing, DF re-election and buffer behaviour on a real ASIC are worth measuring on the platform you’ll run.
  • More spines. Four spines don’t automatically separate A and B better: the hash still decides unless you pin it.

References

Halden Markets is fictional; the lab and its results are real. Configurations are illustrative: validate them on your own platforms and software versions in a lab before production use. Latency figures come from software forwarding in a virtual lab and don’t represent hardware. Cisco, IOS-XR, NX-OS and Nexus are trademarks of Cisco Systems, Inc. CME Group and MDP are trademarks of CME Group Inc. Nasdaq and MoldUDP64 are trademarks of Nasdaq, Inc.