Table of Contents:
- Introduction
- Adding 9364C-GX switch to existing ACI Fabric
- Adding N9364C-GX switch as a Spine during brand new/initial ACI Fabric deployment
- References
Introduction
Cisco Nexus 9364C-GX can be used as either a ACI Spine or ACI leaf, giving you great flexibility. However, you should be aware of how to complete Fabric Discovery / joining the Node to the ACI fabric.
The diagram below shows the front panel and ports available on the N9364C-GX
Figure 1: N9364-C Front Panel Ports
Adding N9364C-GX switch to existing ACI Fabric
- The default mode of the N9364C-GX Switch with ACI Code is to function as a leaf.
- The uplink ports (fabric connectivity) on that leaf switch are the last 2 ports (63 and 64).
- Ports 1 through 62 are downlink ports (host connectivity, APIC also could connect there) on that leaf switch
📙 The N9364C-GX has 2 built in SFP ports where you can connect your APICs to. Each APIC has 2 connections (active/standby). In case you wanted to connect your cluster of 3 APICs to 2 Leaf Role N9364C-GX, you would need an extra SFP Port in each of the leaves. For this you can use QSA-QSFP to SFP converter. - The N9364C-GX role can be changed from the APIC to be a spine (default mode is Leaf). Once done, it needs to be rebooted. Once it comes back up, All 64 ports become Fabric ports (where you can connect leaves to). It is best not to connect leaves to ports 61 and 62. These can be converted to downlinks for IPN/ISN connectivity when doing Multipod/Multisite.
Joining N9364C-GX as a ACI Leaf
Given that port 63 and port 64 are the Fabric Ports for that leaf, for that node to join the fabric as a leaf, connect those 2 ports to existing spines and the node will get discovered as a leaf. You can then go to the APIC UI and register that node to the fabric (give it name and nodeID). The Leaf will show up as active and healthy.
Joining N9364C-GX as a ACI Spine
If you wanted the N9364C-GX to function as a Spine you would connect port 63/64 to existing leaf ports. The Node will get discovered as a Leaf. Remember in the ACI Fabric (Clos topology), Leaves connect to Spines. Leaves don’t connect to Leaves (generally), so this is an invalid topology.
- You would then have to go to APIC UI, Right Click on that Node and choose Spine as the role.
- Wait for a few minutes to ensure that the APIC has pushed the config to the node. Then reboot that Node. Once the Node comes up it will come up as a spine and be active.
- Go ahead and register the node to the fabric.
- This node will now function as a spine and you can connect any port on this spine to other leaves
📙 Remember that the last 2 ports are for Fabric Discovery purposes.
Adding N9364C-GX switch as a Spine during brand new/initial ACI Fabric deployment
This section is explicity the reason why I am writing this article. I got pulled in by a customer because they were having issues with this.
To start off their brand new ACI Fabric Deployment, their initial/starting Fabric topology is as shown below.
Figure 2: Initial/Starting point for building brand new ACI Fabric (for customer)
Problems Encountered:
- The first problem was that they had connected their N9364C-GX Spines to port E1/60 from the leaf. Since the N9364C-GX is by default a leaf switch with port 63 and 64 being the fabric discovery ports, the N9364C-GX was not getting discovered. After reading the Cisco documenation we connected the ports on the N9364C-GX ports 64 to the leaf as shown in the diagram above.
- Meanwile we also noticed that the customer had configured the infra VLAN to be vlan 1001 which is normally not a good idea (generally 3967 is a common good choice). For that reason we decided to clean up the configuration on the APIC and the registered leaf and start from scratch.
- Once the APIC rebooted, they did the intial config on it again (this time with Vlan 3967 as the Infra Vlan)
- After a while the Leaf showed up as discovered but it never went to active state. We waited for a long time and realzied that this was not working. On further analysis we realized that the connected N9364C-GX nodes were at their default leaf mode configuration. This was an invalid topology since Real Leaf was connected to 3 N9364C-GX which were supposed to be spines but were in default Leaf mode.
Analysis and Solution:
- For this brand new Fabric bringup, we disconnected the N9364C-GX Nodes (default Leaves).
- We then rebooted the real Leaf and the Leaf came up and became active in a few minutes.
- We then registered the Leaf to the ACI Fabric (from APIC UI)
- Once the first Leaf was registered and happy, we connected back the N9364C-GX (port 64) back to the leaf.
- From here the procedure was the same as adding N9364C-GX as spines to an exiting ACI Fabric. The N9364C-GX, got discovered as leaves. We changed to role of them to Spines from the APIC UI and rebooted them. They came back up again as spines and went active. We registered the Spines to the Fabric and all was done.
📙 all was done for bringing up the ACI Fabric. Customer now needs to do their initial best practice configurations, for fabric etc, etc, before onboarding tenants
What is the reason on VLAN 1001 is not a good idea to be used as Infra VLAN?
Infra Vlan should be a vlan that is not already used by customer or reserved by common devices. Vlan 3967 has been commonly used with customers since NXOS has 3968 – 4094 reserved. Vlan 3967 may not be appropriate if there are UCS-B FI-6454 in the network. Vlan 3914 may be a more appropriate choice.
Technically Vlan 1001 could be used, but now you are using up a Vlan from the middle of the vlan number. The infra Vlan is present in every leaf. This would mean that Vlan 1001 would now be present on every leaf and you cannot use Vlan 1001 for user Vlan Pools. When you define Vlan Pools for user vlans, you will need to remember to omit Vlan 1001 from any Vlan pool. This would be hard to remember and would cause you to create disontigious Vlan Pools. Much better to be organized and have the Infra Vlan at the end.
Thanks! It certainly makes sense regarding platform reserved VLAN considerations. However my understanding of the infr VLAN is it could be any VLAN as far as it is not extended outside of the fabric, such as using virtual switch outside as TEP. If Vlan 1001 as the infr vlan is isolated to the fabric only, It wouldn’t restrict user VLAN pool to include 1001.
Generally speaking a Vlan encapsulation should map to only 1 EPG in a given leaf unless using Local Scope which has other implications. Remember that the leaves that have APIC connectivity already have the Infra Vlan traversing the front panel ports which you can observe ” vsh_lc -c ‘show system internal eltmc info vlan brief’ “. There are very strong reasons why best practices should be followed from day 0.