“EVPN is a standard, so the vendors interoperate.” It’s the sentence that closes many design reviews, and it’s half true. The route types are standard. The defaults around them aren’t: how route targets are derived, which VLANs are reserved, what has to be carved in hardware before a feature will even configure, how a leaf treats a host that has never spoken.
Multi-vendor EVPN isn’t an exotic case. Acquisitions merge two data centers. A second source is a procurement rule. AI clusters often run one vendor in the GPU back-end and another in the front-end and storage networks. So I did what I’d do for a customer: wrote the requirements, made the design decisions, built it from one plan, and then validated every service on the wire, one at a time, before trying to break it. The lab is Cisco NX-OS on one side and NVIDIA Cumulus Linux on the other.
In short
- It works with standard EVPN. A stretched VLAN, routing between subnets on different vendors, one anycast gateway, a VM moving between vendors, and servers dual-homed to two leaves without MLAG.
- The design decisions are where interop is won. Four values must be explicit and identical: route targets, the anycast gateway MAC, the L3 VNI, and an EVPN next hop that’s never rewritten. Defaults differ between vendors.
- The protocol is fast; the edges aren’t. EVPN withdrew a failed leaf’s routes in 0.2 s. Every second of loss I measured came from LACP, BGP hold timers or spanning tree.
- Two mistakes break it quietly. A 1500-byte underlay link and automatic route targets both leave BGP up and monitoring green while traffic fails.
The method
Validation goes one service at a time on purpose. When something fails, you want to know which layer failed, and a capture that shows it.
Step 1: requirements
| Requirement | What it means for the design |
|---|---|
| One tenant across two fabrics, NX-OS and Cumulus | One VRF, one L3 VNI, route targets both vendors agree on |
| Some subnets on both sides, some on one side only | Stretched L2 VNIs plus routed subnets (symmetric IRB) |
| The same gateway for every host, wherever it sits | One anycast gateway MAC on every leaf of every vendor |
| VMs can move between the fabrics | MAC mobility must work across vendors |
| Critical servers survive the loss of a leaf | Dual-homing: EVPN multihoming, no MLAG |
| Failures must be measured, not assumed | Captures on one clock, a baseline check that can be rerun |
The toolkit, before the design
EVPN is a BGP address family that tells every switch where every host and subnet lives, so the data plane doesn’t have to learn it by flooding. VXLAN is the data plane: the original Ethernet frame inside a UDP packet, sent from one switch (a VTEP, VXLAN tunnel endpoint) to another.
- Type-3, inclusive multicast. “I am a VTEP for this segment.” Every switch builds a flood list from these: a broadcast is sent as one copy to each VTEP on the list. That’s ingress replication.
- Type-2, MAC/IP advertisement. “This MAC, and this IP, are behind me.” It replaces flood-and-learn, feeds ARP suppression and carries /32 host routes.
- Type-5, IP prefix. “This subnet is behind me.” It lets a switch route towards a subnet before any host in it has been learned.
- Two more appear with multihoming, type-1 and type-4; they come in step 4d.
Two kinds of VNI (VXLAN network identifier) matter. An L2 VNI carries one VLAN. An L3 VNI carries routed traffic for one tenant VRF. With symmetric IRB (integrated routing and bridging, RFC 9135), routing happens at both ends: the ingress leaf routes into the L3 VNI, the egress leaf routes out of it.
Step 2: design decisions
| NX-OS fabric | Cumulus fabric | |
|---|---|---|
| Software | Nexus 9000v, NX-OS 10.4(2)F | Cumulus VX 5.4.0 |
| Platform | EVE-NG Pro 7.2, running as a VM on Proxmox VE 8.4 on a Dell PowerEdge R640 (2 × Xeon Gold 6248, 440 GiB RAM) | same EVE-NG lab |
| Underlay | eBGP over numbered /31 links | eBGP unnumbered |
| ASNs | spines 65200, one ASN per leaf | spines 65100, one ASN per leaf |
| VTEPs | 10.0.1.10 to 10.0.1.12 | 10.0.0.3 to 10.0.0.5 |
The fabrics meet on one /31 between LEAF-BORDER and LEAF-3, with a single eBGP session carrying the underlay loopbacks and EVPN. Every leaf ends up with a VXLAN tunnel to every other leaf, whichever vendor it is.
| VLAN | VNI | Subnet | Lives on |
|---|---|---|---|
| 10 | 10010 | 10.10.10.0/24 | both fabrics |
| 20 | 10020 | 10.20.20.0/24 | Cumulus only |
| 30 | 10030 | 10.30.30.0/24 | NX-OS only |
| 3001 | 50001 | none: transit VLAN for routed traffic, no ports | every leaf |
Four decisions make two vendors one fabric:
1. Route targets set by hand, the same on both vendors. A route target (RT) decides which switches import a route. Left automatic, every leaf derives it from its own ASN, and here every leaf has a different one. Cumulus copes by importing automatic RTs as a wildcard; NX-OS doesn’t. Step 5 shows what happens.
2. The EVPN next hop is never rewritten. In an eBGP fabric every hop is a new AS, and by default eBGP sets itself as next hop. For EVPN, the next hop is the VTEP that VXLAN tunnels to, so it must survive end to end. The Cumulus spines kept it unchanged with no extra configuration; on NX-OS it took a route map with set ip next-hop unchanged, and retain route-target all on the spines, which have no tenant of their own.
3. One anycast gateway MAC on both vendors. Every leaf answers for the subnet’s .1 with the same MAC. Cumulus calls it VRR, NX-OS calls it the anycast gateway, and nothing makes them match unless you set the same MAC on both.
4. A transit VLAN for the L3 VNI: 3001. Every leaf needs a VLAN to attach the L3 VNI to. Nothing plugs into it: it has no ports, only an interface inside the VRF. The tag never reaches the wire, only VNI 50001 does, so the number is local to each switch. I first picked 4001; NX-OS refused it (invalid vlans (reserved values)), because it reserves 3968 to 4095 by default.
Key point. Interop is decided by the values you make explicit. Anything left to a default is a value each vendor picks differently.
Step 3: build from one plan
Every ASN, address, VNI, route target and MAC comes from one Python file that generates both vendors’ configurations. Two hand-written configurations drift; one plan doesn’t. The same intent, in each vendor’s words (illustrative, generated by the lab):
# Cumulus Linux (NVUE)
nv set vrf TENANT1 evpn vni 50001
nv set vrf TENANT1 evpn vlan 3001
nv set vrf TENANT1 router bgp route-export to-evpn route-target 65000:50001
nv set vrf TENANT1 router bgp route-import from-evpn route-target 65000:50001
nv set bridge domain br_default vlan 10 vni 10010
nv set evpn vni 10010 route-target both 65000:10010
nv set interface vlan10 ip vrr address 10.10.10.1/24
nv set interface vlan10 ip vrr mac-address 00:00:5e:00:01:01
! Cisco NX-OS
fabric forwarding anycast-gateway-mac 0000.5e00.0101
vlan 3001
vn-segment 50001
interface Vlan3001
vrf member TENANT1
ip forward
vrf context TENANT1
vni 50001
address-family ipv4 unicast
route-target both 65000:50001
route-target both 65000:50001 evpn
interface nve1
member vni 10010
suppress-arp
ingress-replication protocol bgp
member vni 50001 associate-vrf
evpn
vni 10010 l2
route-target import 65000:10010
route-target export 65000:10010
Step 4a: validate the stretched VLAN
The question. SERVER-6, behind a Cumulus leaf, and CLIENT-16, behind an NX-OS leaf, share VLAN 10. How much flooding crosses the interconnect, and does ARP suppression remove it?
ARP suppression lets a leaf answer an ARP request itself, from its EVPN table. I ran the same ping three times, capturing on the interconnect and on both access ports at once.
Phase a: suppression is on, and the ARP is flooded anyway. I expected a formality: Cumulus turns ARP suppression on by default. It wasn’t. SERVER-6’s ARP crossed the interconnect three times, one copy per NX-OS VTEP:
Neither LEAF-3 nor NX-LEAF-02 had a MAC+IP route for CLIENT-16: it had never sent a packet a leaf could learn its IP from.
Key point. ARP suppression answers from the EVPN table. A silent host isn’t in it, so its first ARP is flooded whatever you configure.
Phase b: the host speaks once. One ping from CLIENT-16 to its gateway, and NX-LEAF-02 advertises a type-2 MAC+IP route. LEAF-3 now answers SERVER-6’s ARP itself, in 0.65 ms:
The other direction still floods: NX-LEAF-02 has no suppression yet.
Phase c: suppression on both vendors, and a reload first. On NX-OS, suppress-arp wouldn’t even configure:
ERROR: Please configure TCAM region for Ingress ARP-Ether ACL before configuring ARP supression.
The lab borrows space from the IPv4 router ACL region (unused here), carves an arp-ether region and reloads the three NX-OS leaves. After that, only the pings cross the interconnect:
| Phase | ARP from SERVER-6 crossing the link | ARP from CLIENT-16 crossing the link |
|---|---|---|
| a: silent host, NX-OS suppression off | 3 | 1 (a unicast re-check) |
| b: host spoke once, NX-OS suppression off | 0 | 6 (two ARPs, three copies each) |
| c: suppression on both vendors | 0 | 0 |
Key point. On a platform that needs the TCAM carve, enabling ARP suppression is a reload of every leaf. Do it at first deployment, not in a later change window.
Step 4b: validate routing between vendors
The question. SERVER-7’s VLAN 20 exists only on Cumulus; SERVER-11’s VLAN 30 only on NX-OS. The packet has to be routed by one vendor, carried across, and routed again by the other. What does it look like?
Both hosts resolve their .1 to the same MAC, one answered by Cumulus, one by NX-OS:
On the interconnect, the routed packet travels in VNI 50001, from LEAF-4’s router MAC to LEAF-BORDER’s, with the inner TTL already at 63. SERVER-11 receives TTL 62: one routing hop per vendor. That’s symmetric IRB, visible in two numbers.
LEAF-4 learns LEAF-BORDER’s router MAC from BGP. A host route carries two labels (L2 VNI and L3 VNI), two route targets and a Router MAC community; a leaf without the VLAN imports only the routed half.
The first packet to a silent host is lost, by design. With no ARP entry for SERVER-11, LEAF-4 has only the /24 type-5 route. The first packet reaches LEAF-BORDER, which ARPs and drops the packet that triggered it; from then on a /32 takes every packet straight through.
Key point. One lost packet to a quiet host is a glean, not a fault. Health checks that send a single probe will raise false alarms.
Step 4c: validate VM mobility
The question. A VM migrates and keeps its MAC and IP. Here SERVER-8 “moves” from Cumulus LEAF-5 to NX-OS NX-LEAF-01 and back, while two hosts ping it every 100 ms.
EVPN decides between two locations with the MAC Mobility extended community (RFC 7432): every move increments a sequence number and the highest wins. Both vendors agreed, step by step:
Each move cost 700 to 900 ms of pings, almost all of it the move itself (port down, gratuitous ARP, the new leaf learning the MAC). Moving the host eight times in 40 seconds made both vendors raise a duplicate-host alarm; Cumulus uses the RFC 7432 default of five moves in 180 seconds.
Key point. Detection isn’t protection. By default Cumulus only flags and logs; neither switch froze the MAC here. Decide on freezing, and make every vendor behave the same way.
Step 4d: dual-home the servers (EVPN multihoming)
The question. Critical servers must survive the loss of a leaf. For years the answer was MLAG: two switches pretending to be one, a peer link, a proprietary pairing, a limit of two switches. EVPN multihoming does the same job with standard BGP routes. Does it hold up?
SERVER-7 is now bonded (LACP) to LEAF-4 and LEAF-5.
Three ideas carry the feature:
- The Ethernet segment (ES). The links that connect one device to several leaves, with a 10-byte identifier, the ESI, that every leaf on the segment shares.
- All-active and aliasing. Both leaves forward for the server; remote leaves can send to either one.
- One designated forwarder (DF). Flooded traffic must reach the server once, so the leaves elect one DF per segment to deliver it.
The configuration, the same on both leaves except the port and the DF preference (illustrative):
# Cumulus Linux 5.4 (NVUE), LEAF-4
nv set evpn multihoming enable on # on every VTEP
nv set evpn dad enable off # not supported with multihoming
nv set interface swp1,swp2 evpn multihoming uplink on # uplink tracking
nv set interface bond1 bond member swp3
nv set interface bond1 bond lacp-rate fast
nv set interface bond1 bridge domain br_default access 20
nv set interface bond1 bridge domain br_default stp admin-edge on
nv set interface bond1 bridge domain br_default stp bpdu-guard on
nv set interface bond1 evpn multihoming segment local-id 1
nv set interface bond1 evpn multihoming segment mac-address 44:38:39:be:ef:07
nv set interface bond1 evpn multihoming segment df-preference 50000 # LEAF-5 keeps 32767
Two leaves, one LACP partner. Both leaves send LACP with the segment MAC as their system ID, so the server sees one partner and the bond forms in about 2.4 seconds:
Each leaf builds the ESI from the same inputs: type 3, the system MAC and the local ID. Same inputs, same ESI.
What EVPN advertises for a segment. Four kinds of route, captured as the segment came up:
The type-4 Ethernet Segment route carries an ES-Import route target, so only leaves on this segment import it, and the DF Election community: algorithm 2, Highest-Preference in RFC 9785.
The type-1 routes come in two flavours: one per segment, tag MAX-ET (4294967295) with the ESI Label marking it all-active, for mass withdrawal; one per EVI, tag 0, for aliasing.
A remote leaf puts it together: the server’s MAC points at the segment, its /32 at both leaves.
The NX-OS side. Nexus 9000 can’t host a segment, but Cisco documents it as a remote VTEP for segments hosted by other leaves (10.4(1)F, Nexus 9300-EX and newer), with maximum-paths under the EVPN address family. It accepted the routes, then used one leaf: the two paths have different AS paths, and NX-OS needed bestpath as-path multipath-relax again inside the tenant VRF before it installed both.
Key point. “It works” and “it uses both leaves” are different tests. Check the route table for two next hops from every remote vendor.
Aliasing on the wire. Sixteen parallel TCP flows from each side, counted by outer destination: both segment leaves carried traffic.
Flooded traffic, the hard part. Both segment leaves receive a broadcast; the server must get one copy. I counted broadcasts on both of the server’s links:
- Split horizon by local bias works. A broadcast from a host on LEAF-5 reached SERVER-7 once: LEAF-5 delivered it, LEAF-4 dropped the copy from its segment peer. VXLAN has no room for an MPLS split-horizon label, so this source check is how EVPN does it over IP tunnels (RFC 8365, RFC 9746).
- DF filtering didn’t, on Cumulus VX. A broadcast from LEAF-3 reached the server twice. The election ran and moved when I changed the preference; the non-DF filter was never programmed on the virtual switch. NVIDIA lists multihoming for its Spectrum ASICs, so this one I’d only judge on hardware.
Key point. Test multihoming by counting broadcasts on the server’s links. It’s the only test that shows both rules working.
Step 5: break it on purpose
Failures, timed on one clock
Every capture ran on the EVE-NG host, so all timestamps share one clock. Four hosts pinged SERVER-7 every 100 ms; each case ran twice.
- A server link fails. LEAF-4 withdrew its segment routes 0.2 s later. The loss, 2.3 to 3.0 s, was the server hashing flows onto the dead link until fast LACP timed out. In EVE the far end of a downed link keeps carrier; a real cable signals loss of light at once.
- The link comes back. The first runs lost another 2 to 2.8 s while spanning tree took the port through learning. With
stp admin-edge onand BPDU guard: nothing lost. - A leaf loses its uplinks. Uplink tracking took the segment link down at once, but the fabric kept sending to LEAF-4 until the spines’ BGP hold timer expired (9 s here): 7.3 to 8.6 s of loss. BFD on fabric links catches what a link doesn’t signal.
- The uplinks come back. LEAF-4 stayed out of the segment for the multihoming startup delay, 180 seconds, so it wouldn’t attract traffic before EVPN reconverged.
Key point. The failover inside EVPN is fast. The outage a server sees is set by the timers around it: LACP, BGP or BFD, and spanning tree.
Mistake 1: one link at MTU 1500
VXLAN adds 50 bytes, so a 1500-byte host packet needs 1550 on every underlay link. I set the interconnect to 1500 on both ends.
Small pings pass, BGP never flaps, the interface shows no errors. Meanwhile full-size packets and every TCP transfer die, and no “fragmentation needed” comes back. When LEAF-3 only routed another leaf’s VXLAN packet, it fragmented the outer packet and the receiving VTEP dropped the fragments, as the VXLAN specification allows (RFC 7348, section 4.3):
Mistake 2: automatic route targets
I removed the explicit route target for VNI 10010 on the Cumulus leaves.
The failure is one-way, which makes it hard to diagnose. Inside Cumulus everything still works, because it imports automatic RTs as *:VNI. NX-OS drops every Cumulus route (65103:10010 isn’t 65000:10010). And Cumulus still imports the NX-OS routes, so half the tables look right.
Key point. In both cases the control plane looks healthy while the data plane fails. Monitor the data plane, with full-size packets.
Step 6: operate
The interop details, side by side
| Area | What happened | What to do |
|---|---|---|
| L3 VNI transit VLAN | NX-OS rejected 4001 (3968-4095 reserved) | Pick a VLAN free on every platform; only the VNI must match |
| Route targets | Automatic: NX-OS imported nothing from Cumulus | Explicit and identical everywhere |
| EVPN next hop over eBGP | Must stay the original VTEP | NX-OS: next-hop unchanged, retain route-target all on spines |
| Anycast gateway | Each vendor has its own name and default | One MAC, set explicitly on every leaf |
| ARP suppression on NX-OS | TCAM carve and reload first | Carve at deployment |
| Silent hosts | First ARP flooded, first routed packet lost | Expected; keep it out of alarms |
| Duplicate hosts | Detected, not frozen | Align freezing and recovery across vendors |
| ECMP to a multihomed server | NX-OS used one leaf until VRF multipath-relax |
Check two next hops from every vendor |
| DF filtering | Not enforced on Cumulus VX | Validate on hardware |
| Recovery after link up | 2-2.8 s lost to spanning tree | STP edge plus BPDU guard on server ports |
| Leaf isolation | Waited for the 9 s BGP hold timer | BFD on fabric links |
| Underlay MTU | 1500 on one link: TCP dead, no alarms | Host MTU + 50 everywhere, tested |
Two more, specific to the virtual platforms: NX-OSv 9000 keeps no boot variable, so a reload stops at loader > until you set boot nxos; and when NX-LEAF-02 flooded an ARP into VXLAN, the inner Ethernet source was the switch’s own MAC while the ARP payload kept the host’s. Nothing broke, but I’d check that on hardware.
Best practices
- Write the plan once and generate every vendor from it.
- Make every interop value explicit: route targets, anycast gateway MAC, L3 VNI. Keep local values like the transit VLAN the same everywhere for easier troubleshooting.
- Keep the EVPN next hop intact end to end, and make sure spines keep routes they don’t import.
- Turn on ARP suppression on every leaf of every vendor at first deployment, with any hardware preparation it needs.
- Test the underlay with full-size packets between hosts on different leaves:
ping -M do -s 1472on Linux. - Dual-home with EVPN multihoming, the same ESI inputs on both leaves, uplink tracking on, multihoming enabled on every VTEP.
- Make server ports STP edge ports with BPDU guard, run fast LACP on servers, and add BFD to fabric links.
- Align duplicate-host behaviour across vendors, and alarm on it.
- Monitor the data plane, not only BGP sessions, and keep a baseline check you can rerun after every change. This lab’s
verify.pyruns 18. - Plan maintenance around the multihoming startup delay: a leaf back from a reboot stays out of the segment for minutes.
What this lab doesn’t prove
Virtual switches forward in software. The lab shows control-plane interoperability and the behaviour of each mechanism, including one the virtual platform didn’t enforce. It doesn’t show forwarding performance, hardware timings, TCAM limits or hash quality, and the failure timings include virtual links that keep carrier when one end goes down. Treat the numbers as relative, and check your platform’s documentation.
Run it yourself
The lab is on GitHub: the EVE-NG topology, the configuration generator, one script per scenario, every capture and output shown here, and the verify.py baseline. You bring the Nexus 9000v and Cumulus VX images.
References
- RFC 7432, BGP MPLS-Based Ethernet VPN: route types, the ESI (section 5), the Ethernet Segment route and ES-Import route target (7.4, 7.6), the ESI Label (7.5), the MAC Mobility community (7.7), mass withdrawal (8.2), aliasing (8.4), DF election (8.5), MAC mobility and duplicate detection with defaults of 5 moves in 180 seconds (15).
- RFC 8365, A Network Virtualization Overlay Solution Using EVPN: EVPN over VXLAN, automatic route targets (5.1.2.1), split horizon by local bias (8.3.1).
- RFC 9135, Integrated Routing and Bridging in EVPN: symmetric IRB and the Router’s MAC community.
- RFC 9136, IP Prefix Advertisement in EVPN: the type-5 route.
- RFC 9746, BGP EVPN Multihoming Extensions for Split-Horizon Filtering: local bias for IP-based tunnels such as VXLAN.
- RFC 9785, Preference-Based EVPN Designated Forwarder Election: DF algorithm 2, Highest-Preference.
- RFC 7348, VXLAN: the header, UDP port 4789, fragmentation (4.3).
- RFC 5798, VRRP version 3:
00:00:5e:00:01:01is the VRRP MAC for virtual router 1. The lab uses it for readability; in production, choose a locally administered MAC. - NVIDIA, Cumulus Linux 5.4 EVPN Enhancements: ARP suppression, duplicate address detection, automatic route targets with eBGP.
- NVIDIA, Cumulus Linux 5.4 EVPN Multihoming: configuration, DF preference default 32767, uplink tracking, startup delay, supported platforms.
- FRR, EVPN documentation: how automatic route targets are derived and matched.
- Cisco, Nexus 9000 NX-OS VXLAN Configuration Guide, Release 10.4(x), including Interoperability with EVPN multi-homing using ESI.
- The lab, with every output and capture: networkfactory-labs/evpn-multivendor.
Configurations are for a lab. Validate them against the documentation of your release before using them in production.