I was always sure that ACI and NSX-T can work together, today i tested it , i connected My NSX-T lab to my ACI lab via BGP
This is the topology:
Cisco ACI configuration:
ACI version 4.0(3d)
- At our SDDC tenant i created a new VRF named NSX-T (Because i already had a L3OUT at SDDC VRF)
- At Access-> Fabric policies I configured a new external routed domain with interfaces 1/7 at leafs 101,102 and allow VLANs 131 & 132 (a new VLAN pool)
- I created a new L3OUT named NSX-T with this configuration:
- VRF: NSX-T
- External routed domain: NSX-T-Ext-domain
- Enable BGP
- Enable Default Leak Policy (advertise default route to NSX-T)
- Node profile: Node101
- BGP peer connectivity: Address:10.101.231.101
- Enable Peer control: Bidirectional forwarding detection (BFD)
- Remote AS 65102
- Local AS 65101 (Must not match MP-BGP EVPN AS, if it’s match BGP neighbor status will be IDLE)
- BGP timers: 180 , 60 (if you do not enable BFD , configure Keepalive – 1, hold – 3)
- Create BFD Interface profile with timers: 999, 999, 999, multiplier 3 (This is the MAX for ACI, the min for NSX-T is 1000 :), but it’s working)
- Networks – subnets (also called external EPG) – 0.0.0.0/0
- Enable only ‘External Subnets for the External EPG’
- And the same for Node102
NSX-T configuration
NSX-T version 2.4
- Configure new regular vSwitch at each ESX at the cluster that have NSX-T Edge with the relevant physical port and relevant VLAN for the edge Uplink
- Configured 2 uplinks at Tier-0 via Edge-1 and Edge-2
- BGP timers: 180 , 60 (if you do not enable BFD , configure Keepalive – 1, hold – 3)
- BFD timers : Interval – 1000, multiplier – 3 (1000 is the minimum for Physical uplinks)
- At Tier-0 Configure route redistribute of Tier-1 Connected Subnets
BGP neighborship
And… It’s working
NSX-T
ACI
Troubleshooting
ACI
SSH to the relevant leaf and Check BGP neigbhors
Leaf-102# show ip bgp summary vrf SDDC:NSX-T
BGP summary information for VRF SDDC:NSX-T, address family IPv4 Unicast
BGP router identifier 10.101.255.102, local AS number 65001
BGP table version is 12, IPv4 Unicast config peers 1, capable peers 1
5 network entries and 5 paths using 800 bytes of memory
BGP attribute entries [5/720], BGP AS path entries [1/10]
BGP community entries [0/0], BGP clusterlist entries [1/4]
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.101.232.102 4 65102 7248 7193 12 0 0 00:07:33 1
SSH to the relevant leaf and check route table:
Leaf-102# show ip route vrf SDDC:NSX-T
IP Route Table for VRF “SDDC:NSX-T”
‘‘ denotes best ucast next-hop ‘*’ denotes best mcast next-hop
‘[x/y]’ denotes [preference/metric]
‘%’ in via output denotes VRF
10.101.171.0/24, ubest/mbest: 1/0
*via 10.101.232.102%SDDC:NSX-T, [20/0], 07:10:57, bgp-65001, external, tag 65101
10.101.231.0/24, ubest/mbest: 1/0
*via 10.1.112.64%overlay-1, [200/0], 00:37:56, bgp-65001, internal, tag 65001
10.101.232.0/24, ubest/mbest: 1/0, attached, direct
*via 10.101.232.254, vlan74, [1/0], 1d03h, direct
10.101.232.254/32, ubest/mbest: 1/0, attached
*via 10.101.232.254, vlan74, [1/0], 1d03h, local, local
10.101.249.0/24, ubest/mbest: 1/0, attached, direct, pervasive
*via 10.1.224.66%overlay-1, [1/0], 1d02h, static, tag 4294967294
10.101.255.101/32, ubest/mbest: 1/0
*via 10.1.112.64%overlay-1, [1/0], 00:37:56, bgp-65001, internal, tag 65001
10.101.255.102/32, ubest/mbest: 2/0, attached, direct
*via 10.101.255.102, lo3, [1/0], 1d03h, local, local
*via 10.101.255.102, lo3, [1/0], 1d03h, direct
SSH to the relevant leaf and check BFD status:
Leaf-102# show bfd neighbors vrf SDDC:NSX-T
OurAddr NeighAddr LD/RD RH/RS Holdown(mult) State Int Vrf
10.101.232.254 10.101.232.102 1090519042/385914307 Up 3000(3) Up Vlan74 SDDC:NSX-T
NSX-T
SSH to the edge and check witch VRF uses Tier-0
edge2> get logical-router
Logical Router
UUID VRF LR-ID Name Type Ports
736a80e3-23f6-5a2d-81d6-bbefb2786666 0 0 TUNNEL 3
c2a31082-fc16-4268-a510-2c35740a980c 3 3080 SR-Tier0 SERVICE_ROUTER_TIER0 6
26f6cc48-ad93-4b22-beee-7a1b8e030d8a 4 3075 SR-Tier1 SERVICE_ROUTER_TIER1 5
edge2> vrf 3
edge2(tier0_sr)>
Check BGP neighbors
edge2(tier0_sr)> get bgp neighbor summary
BFD States: NC – Not configured, AC – Activating,DC – Disconnected
AD – Admin down, DW – Down, IN – Init,UP – Up
BGP summary information for VRF default for address-family: ipv4Unicast
Router ID: 10.101.232.102 Local AS: 65102
Neighbor AS State Up/DownTime BFD InMsgs OutMsgs InPfx OutPfx
169.254.0.130 65102 Estab 04:24:00 NC 40706 40715 4 4
10.101.232.254 65101 Estab 00:16:00 UP 11551 11642 1 2
10.101.231.254 65101 Activ never NC 0 0 0 0
Check BFD sessions
edge2(tier0_sr)> get bfd-sessions
BFD Session
Dest_port : 3784
Diag : No Diagnostic
Encap : vlan
Forwarding : last true (current true)
Interface : 54e35cab-c821-4f9d-aed1-f93e042ad08c
Keep-down : false
Last_cp_diag : No Diagnostic
Last_cp_rmt_diag : No Diagnostic
Last_cp_rmt_state : up
Last_cp_state : up
Last_fwd_state : UP
Last_local_down_diag : No Diagnostic
Last_remote_down_diag : No Diagnostic
Last_up_time : 2019-04-01 18:58:57
Local_address : 10.101.232.102
Local_discr : 385914307
Min_rx_ttl : 255
Multiplier : 3
Received_remote_diag : No Diagnostic
Received_remote_state : up
Remote_address : 10.101.232.254
Remote_admin_down : false
Remote_diag : No Diagnostic
Remote_discr : 1090519041
Remote_min_rx_interval : 999
Remote_min_tx_interval : 999
Remote_multiplier : 3
Remote_state : up
Router : c2a31082-fc16-4268-a510-2c35740a980c
Router_down : false
Rx_cfg_min : 1000
Rx_interval : 1000
Service-link : false
Session_type : LR_PORT
State : up
Tx_cfg_min : 1000
Tx_interval : 1000
Check routing table
edge2(tier0_sr)> get route bgp
Flags: t0c – Tier0-Connected, t0s – Tier0-Static, B – BGP,
t0n – Tier0-NAT, t1s – Tier1-Static, t1c – Tier1-Connected,
t1n: Tier1-NAT, t1l: Tier1-LB VIP, t1ls: Tier1-LB SNAT,
t1d: Tier1-DNS FORWARDER, > – selected route, * – FIB route
Total number of routes: 3
b > * 0.0.0.0/0 [20/0] via 10.101.232.254, uplink-277, 00:00:46
b 169.254.0.128/25 [200/0] via 169.254.0.130, inactive, 00:29:31
b > * 10.101.231.0/24 [200/0] via 169.254.0.130, inter-sr-279, 00:29:31
NSX-T edge – packet capture – to use with wireshark
set capture session 1 interface fp-eth0 direction dual
set capture session 1 file capture1.pcap
the file is saved at – /var/vmware/nsx/file-store/ , in order to copy this file via WinSCP, you need first to enable service SSH at CLI and then login via console as root and enable remote root login at sshd_config file
Summery
ACI to NSX-T BGP is working 🙂
for this physical interface (1/7) i did not used VMM domain at ACI
Last thought – what is the East-West packet size ? (it’s Geneve over VXLAN)