“EVPN is a standard, so the vendors interoperate.” It’s the sentence that closes many design reviews, and it’s half true. The route types are standard. The defaults around them aren’t: how route targets are derived, which VLANs are reserved, what has to be carved in hardware before a feature will even configure, how a leaf treats a host that has never spoken.

Multi-vendor EVPN isn’t an exotic case. Acquisitions merge two data centers. A second source is a procurement rule. AI clusters often run one vendor in the GPU back-end and another in the front-end and storage networks. So I did what I’d do for a customer: wrote the requirements, made the design decisions, built it from one plan, and then validated every service on the wire, one at a time, before trying to break it. The lab is Cisco NX-OS on one side and NVIDIA Cumulus Linux on the other.

In short

  • It works with standard EVPN. A stretched VLAN, routing between subnets on different vendors, one anycast gateway, a VM moving between vendors, and servers dual-homed to two leaves without MLAG.
  • The design decisions are where interop is won. Four values must be explicit and identical: route targets, the anycast gateway MAC, the L3 VNI, and an EVPN next hop that’s never rewritten. Defaults differ between vendors.
  • The protocol is fast; the edges aren’t. EVPN withdrew a failed leaf’s routes in 0.2 s. Every second of loss I measured came from LACP, BGP hold timers or spanning tree.
  • Two mistakes break it quietly. A 1500-byte underlay link and automatic route targets both leave BGP up and monitoring green while traffic fails.

The method

Six steps: requirements, design decisions, build, validate one service at a time (stretched VLAN, routing, mobility, multihoming), break it on purpose, operate

Validation goes one service at a time on purpose. When something fails, you want to know which layer failed, and a capture that shows it.

Step 1: requirements

Requirement What it means for the design
One tenant across two fabrics, NX-OS and Cumulus One VRF, one L3 VNI, route targets both vendors agree on
Some subnets on both sides, some on one side only Stretched L2 VNIs plus routed subnets (symmetric IRB)
The same gateway for every host, wherever it sits One anycast gateway MAC on every leaf of every vendor
VMs can move between the fabrics MAC mobility must work across vendors
Critical servers survive the loss of a leaf Dual-homing: EVPN multihoming, no MLAG
Failures must be measured, not assumed Captures on one clock, a baseline check that can be rerun

The toolkit, before the design

EVPN is a BGP address family that tells every switch where every host and subnet lives, so the data plane doesn’t have to learn it by flooding. VXLAN is the data plane: the original Ethernet frame inside a UDP packet, sent from one switch (a VTEP, VXLAN tunnel endpoint) to another.

Three EVPN route types: type-3 builds the flood list, type-2 carries a host’s MAC and IP, type-5 carries a subnet. The attributes that travel with them: route target, two labels, router MAC, MAC mobility. VXLAN adds 50 bytes: outer Ethernet, outer IPv4, UDP 4789 and an 8-byte header with a 24-bit VNI

  • Type-3, inclusive multicast. “I am a VTEP for this segment.” Every switch builds a flood list from these: a broadcast is sent as one copy to each VTEP on the list. That’s ingress replication.
  • Type-2, MAC/IP advertisement. “This MAC, and this IP, are behind me.” It replaces flood-and-learn, feeds ARP suppression and carries /32 host routes.
  • Type-5, IP prefix. “This subnet is behind me.” It lets a switch route towards a subnet before any host in it has been learned.
  • Two more appear with multihoming, type-1 and type-4; they come in step 4d.

Two kinds of VNI (VXLAN network identifier) matter. An L2 VNI carries one VLAN. An L3 VNI carries routed traffic for one tenant VRF. With symmetric IRB (integrated routing and bridging, RFC 9135), routing happens at both ends: the ingress leaf routes into the L3 VNI, the egress leaf routes out of it.

Step 2: design decisions

Two fabrics. NX-OS: two spines in AS 65200, leaves NX-LEAF-01, NX-LEAF-02 and LEAF-BORDER. Cumulus: two spines in AS 65100, leaves LEAF-3, LEAF-4, LEAF-5. LEAF-BORDER E1/1 connects to LEAF-3 swp4 over 10.255.0.0/31 with eBGP for IPv4 and EVPN. SERVER-7 is bonded to LEAF-4 and LEAF-5

NX-OS fabric Cumulus fabric
Software Nexus 9000v, NX-OS 10.4(2)F Cumulus VX 5.4.0
Platform EVE-NG Pro 7.2, running as a VM on Proxmox VE 8.4 on a Dell PowerEdge R640 (2 × Xeon Gold 6248, 440 GiB RAM) same EVE-NG lab
Underlay eBGP over numbered /31 links eBGP unnumbered
ASNs spines 65200, one ASN per leaf spines 65100, one ASN per leaf
VTEPs 10.0.1.10 to 10.0.1.12 10.0.0.3 to 10.0.0.5

The fabrics meet on one /31 between LEAF-BORDER and LEAF-3, with a single eBGP session carrying the underlay loopbacks and EVPN. Every leaf ends up with a VXLAN tunnel to every other leaf, whichever vendor it is.

VLAN VNI Subnet Lives on
10 10010 10.10.10.0/24 both fabrics
20 10020 10.20.20.0/24 Cumulus only
30 10030 10.30.30.0/24 NX-OS only
3001 50001 none: transit VLAN for routed traffic, no ports every leaf

Four decisions make two vendors one fabric:

1. Route targets set by hand, the same on both vendors. A route target (RT) decides which switches import a route. Left automatic, every leaf derives it from its own ASN, and here every leaf has a different one. Cumulus copes by importing automatic RTs as a wildcard; NX-OS doesn’t. Step 5 shows what happens.

2. The EVPN next hop is never rewritten. In an eBGP fabric every hop is a new AS, and by default eBGP sets itself as next hop. For EVPN, the next hop is the VTEP that VXLAN tunnels to, so it must survive end to end. The Cumulus spines kept it unchanged with no extra configuration; on NX-OS it took a route map with set ip next-hop unchanged, and retain route-target all on the spines, which have no tenant of their own.

3. One anycast gateway MAC on both vendors. Every leaf answers for the subnet’s .1 with the same MAC. Cumulus calls it VRR, NX-OS calls it the anycast gateway, and nothing makes them match unless you set the same MAC on both.

4. A transit VLAN for the L3 VNI: 3001. Every leaf needs a VLAN to attach the L3 VNI to. Nothing plugs into it: it has no ports, only an interface inside the VRF. The tag never reaches the wire, only VNI 50001 does, so the number is local to each switch. I first picked 4001; NX-OS refused it (invalid vlans (reserved values)), because it reserves 3968 to 4095 by default.

Key point. Interop is decided by the values you make explicit. Anything left to a default is a value each vendor picks differently.

Step 3: build from one plan

Every ASN, address, VNI, route target and MAC comes from one Python file that generates both vendors’ configurations. Two hand-written configurations drift; one plan doesn’t. The same intent, in each vendor’s words (illustrative, generated by the lab):

# Cumulus Linux (NVUE)
nv set vrf TENANT1 evpn vni 50001
nv set vrf TENANT1 evpn vlan 3001
nv set vrf TENANT1 router bgp route-export to-evpn route-target 65000:50001
nv set vrf TENANT1 router bgp route-import from-evpn route-target 65000:50001
nv set bridge domain br_default vlan 10 vni 10010
nv set evpn vni 10010 route-target both 65000:10010
nv set interface vlan10 ip vrr address 10.10.10.1/24
nv set interface vlan10 ip vrr mac-address 00:00:5e:00:01:01
! Cisco NX-OS
fabric forwarding anycast-gateway-mac 0000.5e00.0101
vlan 3001
  vn-segment 50001
interface Vlan3001
  vrf member TENANT1
  ip forward
vrf context TENANT1
  vni 50001
  address-family ipv4 unicast
    route-target both 65000:50001
    route-target both 65000:50001 evpn
interface nve1
  member vni 10010
    suppress-arp
    ingress-replication protocol bgp
  member vni 50001 associate-vrf
evpn
  vni 10010 l2
    route-target import 65000:10010
    route-target export 65000:10010

Step 4a: validate the stretched VLAN

The question. SERVER-6, behind a Cumulus leaf, and CLIENT-16, behind an NX-OS leaf, share VLAN 10. How much flooding crosses the interconnect, and does ARP suppression remove it?

ARP suppression lets a leaf answer an ARP request itself, from its EVPN table. I ran the same ping three times, capturing on the interconnect and on both access ports at once.

SERVER-6’s ARP is answered locally by LEAF-3 from the EVPN MAC+IP table; CLIENT-16’s by NX-LEAF-02; a table shows ARP copies crossing the link per phase: 3 and 1, then 0 and 6, then 0 and 0

Phase a: suppression is on, and the ARP is flooded anyway. I expected a formality: Cumulus turns ARP suppression on by default. It wasn’t. SERVER-6’s ARP crossed the interconnect three times, one copy per NX-OS VTEP:

Three VXLAN packets from 10.0.0.3 to 10.0.1.10, 10.0.1.12 and 10.0.1.11, each carrying the same ARP request

Neither LEAF-3 nor NX-LEAF-02 had a MAC+IP route for CLIENT-16: it had never sent a packet a leaf could learn its IP from.

LEAF-3’s ARP cache and NX-LEAF-02’s MAC-IP table list 10.10.10.106 and 10.10.10.108, but not 10.10.10.116

Key point. ARP suppression answers from the EVPN table. A silent host isn’t in it, so its first ARP is flooded whatever you configure.

Decode: outer IP 10.0.0.3 to 10.0.1.10, UDP destination port 4789, VNI 10010, inner Ethernet to broadcast, ARP request

Phase b: the host speaks once. One ping from CLIENT-16 to its gateway, and NX-LEAF-02 advertises a type-2 MAC+IP route. LEAF-3 now answers SERVER-6’s ARP itself, in 0.65 ms:

On SERVER-6’s port: ARP request, reply from LEAF-3 with CLIENT-16’s MAC 0.65 ms later, then the ping

The other direction still floods: NX-LEAF-02 has no suppression yet.

On the interconnect: SERVER-6’s ping crosses directly, CLIENT-16’s ARP crosses three times

Phase c: suppression on both vendors, and a reload first. On NX-OS, suppress-arp wouldn’t even configure:

ERROR: Please configure TCAM region for Ingress ARP-Ether ACL before configuring ARP supression.

The lab borrows space from the IPv4 router ACL region (unused here), carves an arp-ether region and reloads the three NX-OS leaves. After that, only the pings cross the interconnect:

On the interconnect: six echo requests and six replies, no ARP

Phase ARP from SERVER-6 crossing the link ARP from CLIENT-16 crossing the link
a: silent host, NX-OS suppression off 3 1 (a unicast re-check)
b: host spoke once, NX-OS suppression off 0 6 (two ARPs, three copies each)
c: suppression on both vendors 0 0

Key point. On a platform that needs the TCAM carve, enabling ARP suppression is a reload of every leaf. Do it at first deployment, not in a later change window.

Step 4b: validate routing between vendors

The question. SERVER-7’s VLAN 20 exists only on Cumulus; SERVER-11’s VLAN 30 only on NX-OS. The packet has to be routed by one vendor, carried across, and routed again by the other. What does it look like?

SERVER-7 sends to the gateway MAC with TTL 64; LEAF-4 routes into VNI 50001 from its router MAC to LEAF-BORDER’s, TTL 63; LEAF-BORDER routes to SERVER-11, TTL 62

Both hosts resolve their .1 to the same MAC, one answered by Cumulus, one by NX-OS:

SERVER-7 and SERVER-11 both see 00:00:5e:00:01:01; LEAF-4 shows VRR with that MAC; LEAF-BORDER shows fabric forwarding anycast-gateway-mac 0000.5e00.0101

On the interconnect, the routed packet travels in VNI 50001, from LEAF-4’s router MAC to LEAF-BORDER’s, with the inner TTL already at 63. SERVER-11 receives TTL 62: one routing hop per vendor. That’s symmetric IRB, visible in two numbers.

Decode: outer 10.0.0.4 to 10.0.1.10, UDP 4789, VNI 50001, inner Ethernet between the two router MACs, inner TTL 63

LEAF-4 learns LEAF-BORDER’s router MAC from BGP. A host route carries two labels (L2 VNI and L3 VNI), two route targets and a Router MAC community; a leaf without the VLAN imports only the routed half.

LEAF-BORDER’s entry for 10.20.20.107: labels 10020 and 50001, route targets 65000:10020 and 65000:50001, Router MAC, imported into TENANT1

The first packet to a silent host is lost, by design. With no ARP entry for SERVER-11, LEAF-4 has only the /24 type-5 route. The first packet reaches LEAF-BORDER, which ARPs and drops the packet that triggered it; from then on a /32 takes every packet straight through.

Request 1 crosses but gets no reply; LEAF-BORDER ARPs for 10.30.30.111 and request 2 arrives

Key point. One lost packet to a quiet host is a glean, not a fault. Health checks that send a single probe will raise false alarms.

Step 4c: validate VM mobility

The question. A VM migrates and keeps its MAC and IP. Here SERVER-8 “moves” from Cumulus LEAF-5 to NX-OS NX-LEAF-01 and back, while two hosts ping it every 100 ms.

SERVER-8 moves from LEAF-5 to NX-LEAF-01 and back; MAC Mobility sequence 0, 1, 2; 900 ms and 700 ms lost; flapping triggers duplicate detection

EVPN decides between two locations with the MAC Mobility extended community (RFC 7432): every move increments a sequence number and the highest wins. Both vendors agreed, step by step:

LEAF-3 and NX-LEAF-02 show sequence 0, then 1 via 10.0.1.11, then 2 via 10.0.0.5

BGP UPDATEs: the new location announced with sequence 1, the old one withdrawn 0.27 s later, then the reverse

Each move cost 700 to 900 ms of pings, almost all of it the move itself (port down, gratuitous ARP, the new leaf learning the MAC). Moving the host eight times in 40 seconds made both vendors raise a duplicate-host alarm; Cumulus uses the RFC 7432 default of five moves in 180 seconds.

LEAF-5: Duplicate, detected, max-moves 5, time 180; NX-LEAF-01 log L2RIB_DUPLICATE_HOST_DURING_LOCAL_UPDATE

Key point. Detection isn’t protection. By default Cumulus only flags and logs; neither switch froze the MAC here. Decide on freezing, and make every vendor behave the same way.

Step 4d: dual-home the servers (EVPN multihoming)

The question. Critical servers must survive the loss of a leaf. For years the answer was MLAG: two switches pretending to be one, a peer link, a proprietary pairing, a limit of two switches. EVPN multihoming does the same job with standard BGP routes. Does it hold up?

SERVER-7 is now bonded (LACP) to LEAF-4 and LEAF-5.

SERVER-7 bonded to LEAF-4 and LEAF-5, which present LACP system ID 44:38:39:be:ef:07 and share Ethernet segment 03:44:38:39:be:ef:07:00:00:01; LEAF-4 is the designated forwarder; remote leaves send to either leaf

Three ideas carry the feature:

  • The Ethernet segment (ES). The links that connect one device to several leaves, with a 10-byte identifier, the ESI, that every leaf on the segment shares.
  • All-active and aliasing. Both leaves forward for the server; remote leaves can send to either one.
  • One designated forwarder (DF). Flooded traffic must reach the server once, so the leaves elect one DF per segment to deliver it.

The configuration, the same on both leaves except the port and the DF preference (illustrative):

# Cumulus Linux 5.4 (NVUE), LEAF-4
nv set evpn multihoming enable on                            # on every VTEP
nv set evpn dad enable off                                   # not supported with multihoming
nv set interface swp1,swp2 evpn multihoming uplink on        # uplink tracking
nv set interface bond1 bond member swp3
nv set interface bond1 bond lacp-rate fast
nv set interface bond1 bridge domain br_default access 20
nv set interface bond1 bridge domain br_default stp admin-edge on
nv set interface bond1 bridge domain br_default stp bpdu-guard on
nv set interface bond1 evpn multihoming segment local-id 1
nv set interface bond1 evpn multihoming segment mac-address 44:38:39:be:ef:07
nv set interface bond1 evpn multihoming segment df-preference 50000   # LEAF-5 keeps 32767

Two leaves, one LACP partner. Both leaves send LACP with the segment MAC as their system ID, so the server sees one partner and the bond forms in about 2.4 seconds:

LACP on both links: both leaves answer with actor system 44:38:39:be:ef:07; both sides Collecting and Distributing at 2.2 to 2.4 s

Each leaf builds the ESI from the same inputs: type 3, the system MAC and the local ID. Same inputs, same ESI.

LEAF-4: ESI 03:44:38:39:be:ef:07:00:00:01, DF status df, DF preference 50000, peer 10.0.0.5 at 32767

What EVPN advertises for a segment. Four kinds of route, captured as the segment came up:

Type-1 EAD per EVI for aliasing; type-4 Ethernet Segment for discovery and DF election; type-1 EAD per ES for mass withdrawal; type-2 with the ESI

The type-4 Ethernet Segment route carries an ES-Import route target, so only leaves on this segment import it, and the DF Election community: algorithm 2, Highest-Preference in RFC 9785.

Type-4 decode: ESI type 3, ES-Import route target 44:38:39:be:ef:07, DF Election algorithm 2, preference 32767

The type-1 routes come in two flavours: one per segment, tag MAX-ET (4294967295) with the ESI Label marking it all-active, for mass withdrawal; one per EVI, tag 0, for aliasing.

EAD per ES with Ethernet tag 4294967295 and ESI Label all-active; EAD per EVI with tag 0

A remote leaf puts it together: the server’s MAC points at the segment, its /32 at both leaves.

LEAF-3: ESI remote via 10.0.0.4 and 10.0.0.5; MAC with Remote ES; route to 10.20.20.107 via both

The NX-OS side. Nexus 9000 can’t host a segment, but Cisco documents it as a remote VTEP for segments hosted by other leaves (10.4(1)F, Nexus 9300-EX and newer), with maximum-paths under the EVPN address family. It accepted the routes, then used one leaf: the two paths have different AS paths, and NX-OS needed bestpath as-path multipath-relax again inside the tenant VRF before it installed both.

NX-OS: EAD received all-active; one path until multipath-relax inside the VRF plus a refresh, then two

Key point. “It works” and “it uses both leaves” are different tests. Check the route table for two next hops from every remote vendor.

Aliasing on the wire. Sixteen parallel TCP flows from each side, counted by outer destination: both segment leaves carried traffic.

LEAF-3 split 6 and 11 flows between the two leaves; NX-OSv split 8 and 9 across 17 source IPs

Flooded traffic, the hard part. Both segment leaves receive a broadcast; the server must get one copy. I counted broadcasts on both of the server’s links:

Panel A, from a remote leaf: only the DF should deliver; on Cumulus VX the non-DF delivered too. Panel B, from a host on LEAF-5: delivered once by local bias

Remote broadcast: both leaves delivered, a duplicate. Broadcast from SERVER-9 on LEAF-5: delivered once

  • Split horizon by local bias works. A broadcast from a host on LEAF-5 reached SERVER-7 once: LEAF-5 delivered it, LEAF-4 dropped the copy from its segment peer. VXLAN has no room for an MPLS split-horizon label, so this source check is how EVPN does it over IP tunnels (RFC 8365, RFC 9746).
  • DF filtering didn’t, on Cumulus VX. A broadcast from LEAF-3 reached the server twice. The election ran and moved when I changed the preference; the non-DF filter was never programmed on the virtual switch. NVIDIA lists multihoming for its Spectrum ASICs, so this one I’d only judge on hardware.

Key point. Test multihoming by counting broadcasts on the server’s links. It’s the only test that shows both rules working.

Step 5: break it on purpose

Failures, timed on one clock

Every capture ran on the EVE-NG host, so all timestamps share one clock. Four hosts pinged SERVER-7 every 100 ms; each case ran twice.

Link failure: withdrawal at +0.2 s, server drops the link at +2.4 to +3.1 s, recovery free with STP edge. Isolation: protodown at once, withdrawal at +7.4 to +9.1 s on the BGP hold timer, 180 s startup delay

Event timelines from the captures: loss on failure and recovery without STP edge, none on recovery with it; isolation loss until the withdrawal at +9.06 s

  • A server link fails. LEAF-4 withdrew its segment routes 0.2 s later. The loss, 2.3 to 3.0 s, was the server hashing flows onto the dead link until fast LACP timed out. In EVE the far end of a downed link keeps carrier; a real cable signals loss of light at once.
  • The link comes back. The first runs lost another 2 to 2.8 s while spanning tree took the port through learning. With stp admin-edge on and BPDU guard: nothing lost.
  • A leaf loses its uplinks. Uplink tracking took the segment link down at once, but the fabric kept sending to LEAF-4 until the spines’ BGP hold timer expired (9 s here): 7.3 to 8.6 s of loss. BFD on fabric links catches what a link doesn’t signal.
  • The uplinks come back. LEAF-4 stayed out of the segment for the multihoming startup delay, 180 seconds, so it wouldn’t attract traffic before EVPN reconverged.

Key point. The failover inside EVPN is fast. The outage a server sees is set by the timers around it: LACP, BGP or BFD, and spanning tree.

VXLAN adds 50 bytes, so a 1500-byte host packet needs 1550 on every underlay link. I set the interconnect to 1500 on both ends.

At MTU 1500: 1428-byte pings fine, 1500-byte pings 100% loss, TCP 0 bytes, BGP up, no errors or drops

Small pings pass, BGP never flaps, the interface shows no errors. Meanwhile full-size packets and every TCP transfer die, and no “fragmentation needed” comes back. When LEAF-3 only routed another leaf’s VXLAN packet, it fragmented the outer packet and the receiving VTEP dropped the fragments, as the VXLAN specification allows (RFC 7348, section 4.3):

Outer packets split into 1514-byte and 84-byte fragments

Mistake 2: automatic route targets

I removed the explicit route target for VNI 10010 on the Cumulus leaves.

With automatic RTs, Cumulus exports 65103:10010; NX-OS has no Cumulus host in its L2RIB; Cumulus to NX-OS pings 100% loss

The failure is one-way, which makes it hard to diagnose. Inside Cumulus everything still works, because it imports automatic RTs as *:VNI. NX-OS drops every Cumulus route (65103:10010 isn’t 65000:10010). And Cumulus still imports the NX-OS routes, so half the tables look right.

MTU 1500 on the left, automatic route targets on the right, both invisible to the usual health checks

Key point. In both cases the control plane looks healthy while the data plane fails. Monitor the data plane, with full-size packets.

Step 6: operate

The interop details, side by side

Area What happened What to do
L3 VNI transit VLAN NX-OS rejected 4001 (3968-4095 reserved) Pick a VLAN free on every platform; only the VNI must match
Route targets Automatic: NX-OS imported nothing from Cumulus Explicit and identical everywhere
EVPN next hop over eBGP Must stay the original VTEP NX-OS: next-hop unchanged, retain route-target all on spines
Anycast gateway Each vendor has its own name and default One MAC, set explicitly on every leaf
ARP suppression on NX-OS TCAM carve and reload first Carve at deployment
Silent hosts First ARP flooded, first routed packet lost Expected; keep it out of alarms
Duplicate hosts Detected, not frozen Align freezing and recovery across vendors
ECMP to a multihomed server NX-OS used one leaf until VRF multipath-relax Check two next hops from every vendor
DF filtering Not enforced on Cumulus VX Validate on hardware
Recovery after link up 2-2.8 s lost to spanning tree STP edge plus BPDU guard on server ports
Leaf isolation Waited for the 9 s BGP hold timer BFD on fabric links
Underlay MTU 1500 on one link: TCP dead, no alarms Host MTU + 50 everywhere, tested

Two more, specific to the virtual platforms: NX-OSv 9000 keeps no boot variable, so a reload stops at loader > until you set boot nxos; and when NX-LEAF-02 flooded an ARP into VXLAN, the inner Ethernet source was the switch’s own MAC while the ARP payload kept the host’s. Nothing broke, but I’d check that on hardware.

Best practices

  1. Write the plan once and generate every vendor from it.
  2. Make every interop value explicit: route targets, anycast gateway MAC, L3 VNI. Keep local values like the transit VLAN the same everywhere for easier troubleshooting.
  3. Keep the EVPN next hop intact end to end, and make sure spines keep routes they don’t import.
  4. Turn on ARP suppression on every leaf of every vendor at first deployment, with any hardware preparation it needs.
  5. Test the underlay with full-size packets between hosts on different leaves: ping -M do -s 1472 on Linux.
  6. Dual-home with EVPN multihoming, the same ESI inputs on both leaves, uplink tracking on, multihoming enabled on every VTEP.
  7. Make server ports STP edge ports with BPDU guard, run fast LACP on servers, and add BFD to fabric links.
  8. Align duplicate-host behaviour across vendors, and alarm on it.
  9. Monitor the data plane, not only BGP sessions, and keep a baseline check you can rerun after every change. This lab’s verify.py runs 18.
  10. Plan maintenance around the multihoming startup delay: a leaf back from a reboot stays out of the segment for minutes.

What this lab doesn’t prove

Virtual switches forward in software. The lab shows control-plane interoperability and the behaviour of each mechanism, including one the virtual platform didn’t enforce. It doesn’t show forwarding performance, hardware timings, TCAM limits or hash quality, and the failure timings include virtual links that keep carrier when one end goes down. Treat the numbers as relative, and check your platform’s documentation.

Run it yourself

The lab is on GitHub: the EVE-NG topology, the configuration generator, one script per scenario, every capture and output shown here, and the verify.py baseline. You bring the Nexus 9000v and Cumulus VX images.

References

  • RFC 7432, BGP MPLS-Based Ethernet VPN: route types, the ESI (section 5), the Ethernet Segment route and ES-Import route target (7.4, 7.6), the ESI Label (7.5), the MAC Mobility community (7.7), mass withdrawal (8.2), aliasing (8.4), DF election (8.5), MAC mobility and duplicate detection with defaults of 5 moves in 180 seconds (15).
  • RFC 8365, A Network Virtualization Overlay Solution Using EVPN: EVPN over VXLAN, automatic route targets (5.1.2.1), split horizon by local bias (8.3.1).
  • RFC 9135, Integrated Routing and Bridging in EVPN: symmetric IRB and the Router’s MAC community.
  • RFC 9136, IP Prefix Advertisement in EVPN: the type-5 route.
  • RFC 9746, BGP EVPN Multihoming Extensions for Split-Horizon Filtering: local bias for IP-based tunnels such as VXLAN.
  • RFC 9785, Preference-Based EVPN Designated Forwarder Election: DF algorithm 2, Highest-Preference.
  • RFC 7348, VXLAN: the header, UDP port 4789, fragmentation (4.3).
  • RFC 5798, VRRP version 3: 00:00:5e:00:01:01 is the VRRP MAC for virtual router 1. The lab uses it for readability; in production, choose a locally administered MAC.
  • NVIDIA, Cumulus Linux 5.4 EVPN Enhancements: ARP suppression, duplicate address detection, automatic route targets with eBGP.
  • NVIDIA, Cumulus Linux 5.4 EVPN Multihoming: configuration, DF preference default 32767, uplink tracking, startup delay, supported platforms.
  • FRR, EVPN documentation: how automatic route targets are derived and matched.
  • Cisco, Nexus 9000 NX-OS VXLAN Configuration Guide, Release 10.4(x), including Interoperability with EVPN multi-homing using ESI.
  • The lab, with every output and capture: networkfactory-labs/evpn-multivendor.

Configurations are for a lab. Validate them against the documentation of your release before using them in production.