Nutanix Spec Sheet Errata

The official Nutanix G6 & G7 generation NX appliance spec sheets are frequently updated and are getting better with every iteration. Unfortunately there are still issues with them. Some information in the NX appliance spec sheets is confusing and showing conflicting or wrong information and should be corrected. Some of the aspects of NX appliances are documented in fairly shallow manner and could benefit from more comprehensive documentation.

While building the DPTPB Nutanix Dynamic Visio shapes, the issues found in the spec sheets caused some headaches and extra work. Sometimes these issues went unnoticed and resulted in bugs with the dynamic shapes, which in turn required additional development cycles to address the issues. To avoid making the same mistakes again, I decided to write them down.

For average Nutanix end-user most of these issues are fairly minor, but finding errors in the spec sheets makes you wonder what other issues are there, still not found and undocumented.

I’ve notified Nutanix about few of these issues, but for some reason or other they haven’t (yet) updated the spec sheets. Hopefully this post will help Nutanix to correct some of the issues…

This blog post is based on the information found in G6 and G7 generation single-node and multi-node spec sheets:

System-Specs-G6-Multinode, Feb 11, 2020
System-Specs-G6-Single-Node, Jan 25, 2020
System-Specs-G7-Multinode, Feb 26, 2020
System-Specs-G7-Single-Node, Feb 25, 2020

These spec sheets are available on the Nutanix support portal (login required).

Generic Issues

Wrong form factor NICs and PCIe Expansions Slots

Usually NICs and PCIe expansion slots come in two form factors, low-profile (LP) or full-height (FH). As NICs are installed into the PCIe expansion slots, naturally the form factor of the PCIe slot have to match with the form factor of the NIC or otherwise the NIC will not fit into the PCIe slot or will not fully cover the opening in the PCIe slot.

PCIe slot form factors

Unfortunately it is seems that Nutanix has overlooked this piece of information and states (wrongly) in the G6 & G7 spec sheets that ALL PCIe expansion slots are low-profile form factor.

Example 1: Expansion Slot for NX-1175S-G7, G7 spec sheet, page 9

Example 2: Expansion Slot for NX-3155G-G7, G7 spec sheet, page 18

Example 3: Expansion Slot for NX-8150-G7, G7 spec sheet, page 30

Back panel examples & PCIe expansion slots for NX appliances

By looking at “Back panel” examples one can clearly see that the PCIe expansions slots are not equally wide, meaning that the narrower ones are low-profile and the wider ones are full-height.

Example 4: Back panel NX-8150-G7, G7 spec sheet, page 26

Example 4: Notes

You can clearly see that
- NIC1 is narrower than the rest
  - i.e it is low-profile form factor
- NIC2 & NIC3 are wider than NIC1
  - i.e they are full-height form factor

The wrong usage of form factors in the spec sheets can be confirmed also by looking at Supermicro documentation for similar hardware as NX-8150-G7 (NX appliance hardware is supplied by Supermicro)

Example 5: Expansion slot form factors in Supermicro documentation

Example 5: Notes

Supermicro documentation confirms
NIC1/Slot4 is LP (low-profile) form factor
NIC2/Slot5 and NIC3/Slot7 are FH (full-height) form factor

NIC form factors

No specification of NIC form factor in written format in the spec sheets, however in supporting images NICs are shown in both FH and LP form factors.

Example 6: NICs in both LP and FH form factor, G6 / G7 spec sheets

Example 6: Notes

First three NICs are Low-profile (LP) and have low-profile metal brackets
The last NIC (25GbE) is Full-Height (FH) and has full-height metal bracket
Typically you can acquire different form factor variants for NICs
- The actual NIC hardware doesn’t usually change
- Only different metal bracket is used

The actual NIC and PCIe slot form factors in use with NX appliances

Depending on NX model, it may be supported to use only LP form factor PCIe slots, only FH form factor PCIe slots or both LP and FH PCIe slots.

To make matters more confusing, some of the NX models (NX-3155/5155/8150/8155) share the same back panel and look exactly the same when viewed from behind without any NICs, but some of those models only support using FH PCIe slots, while others support using both LP and FH PCIe slots. Again not stated in the spec sheets in written form, you have to figure this out by looking at some of the images in the spec sheets.

NX Models with only LP PCIe slots in use
- NX-1065-G6/G7
- NX-3060-G6/G7
NX Models with only FH PCIe slots in use
- NX-8035-G6/G7
- NX-5155-G6
- NX-8155-G6/G7
NX Models with both LP and FH PCIe slots in use
- NX-1175S-G6/G7
- NX-3155G-G6/G7
- NX-3170-G6
- NX-8150-G7
- NX-8170-G7

NIC options

Unfortunately “NIC options” images used in the spec sheets show most of the time confusing information. Showing also FH NICs when the NX platform only supports LP NICs and the other way around showing also LP NICs when the NX platform only supports FH NICs. It seems that they are basically using a variation of the same image to document the NIC options, as seen in examples below.

Example 1: NX-3060-G7 NIC options, G7 spec sheet, page 14

NX-3060-G7 only supports NICs in Low-Profile (LP) form factor
Yet:
- Quad-Port 10GbE spf+ is shown with Full-Height (FH) form factor
- Dual-Port 25GbE spf28 is shown with Full-Height (FH) form factor

Example 2: NX-8035-G7 NIC options, G7 spec sheet, page 25

NX-8035-G7 only supports NICs in Full-Height (FH) form factor
Yet:
- Dual-Port 10G BaseT is shown in LP form factor
- Dual-Port 10GbE spf+ is shown in LP form factor
- Dual-Port 40Gbe Qsfp28 is shown in LP form factor
Basically the same image as used with NX-3060-G7

Conclusion of wrong form factor NICs:

Wrong form factor NICs or NIC brackets are not usually problem with factory-installed units. Naturally when NX appliances are assembled at the factory, a correct form factor for NIC is selected or otherwise it wouldn’t fit in the designated PCIe slot…

But assuming wrong form factor might be a problem when NICs are ordered separately, either for expanding NX appliances with additional NICs or upgrading the existing NICs. With some NX platforms there are both LP and FH form factor PCIe slots in use and you might end up ordering wrong form-factor NICs.

Most of the time the NIC itself does not change when used with different form factors, but the metal bracket is different. I am not sure if separately ordered factory supplied NICs come with user changeable (LP and FH) form factor brackets or if they are sold with only a specific form factor bracket (LP or FH). If the latter, buyer be aware…

As I don’t have access to Nutanix Salesforce configurator any more, I can’t check available NIC options and their form factors, comments regarding this are welcome…

Nearly no NIC orientation information available

Once again one of those things that is not documented in written format, you can figure this out by looking at some of the supporting images found in the spec sheets.

When using multiple NICs with most NX platforms NICs run in two orientations.

0° orientation
- The PCIe connector is on the left side of the NIC to match the motherboard PCIe connector (when looking at the back panel )
180° orientation
- The PCIe connector is on the right side of the NIC to match the motherboard PCIe connector (when looking at the back panel )
Since there are no 0° and 180° variants of NICs available in real life, you make a 180° orientation NIC from 0° orientation NIC by flipping the NIC by 180° when installing the NIC.
The orientation names “0°” and “180°” are just names, not something that are widely used or somehow “official” terms, I just picked them as they make sense and makes it easier to understand the rest of this blog post.

Example: NX-8155-G7 blow up image, G7 spec sheet, page 39

Since we are now looking the NX appliance from the front
- left and right are swapped
- 0° orientation
  - The PCIe connector is on the right side of the NIC to match the motherboard PCIe connector
- 180° orientation
  - The PCIe connector is on the left side of the NIC to match the motherboard PCIe connector (when looking at the back panel )

NX Models with just one NIC orientation

NX-8035-Gx
- All NICs are installed in 0° orientation

NX models with both 0° and 180° orientation NICs

All other NX models when using more than one NIC
there are no hard rules that NIC1 is always in 0°orientation or 180°
the orientation of specific NICn varies based on NX platform used

Example 1: NX-8035-G7 NIC orientation, G7 spec sheet, page 24

NICs are in the same orientation
All NICs are in FH form factor

Example 2: NX-8155-G7 NIC orientation, G7 spec sheet, page 37

NIC1 (on the left) is in different orientation
than
NIC2 (on the right) and NIC3 (in the middle)
All NICs are in FH form factor

Example 3: NX-8150-G7 NIC orientation, G7 spec sheet, page 26

NIC1 (on the left) is in different orientation
- Little bit hard to figure this out, see the next example 3.1
than
NIC2 & NIC 3 (on the right)
While the back plane view for NX-8150-G7 and NX-8155-G7 is exactly the same when NOT populated with NICs, they have different PCIe slot usage order and assigned PCIe slots and they look different when populated with NICs
NX-8155-G7 uses only FH form factor NICs, while NX-8150-G7 uses both LP and FH form factor NICs

Example 3.1

Here is the same example as above using my DPTPB Dynamic shapes
Oops, it seems that Nutanix has used “Quad -port 10Gbe spf+” adapters in their example, which are not supported with NX-8150-G7 platform according to the spec sheet, maybe the image in the spec sheet should be updated with supported NICs
The automatic placement of NICs with DPTPB Dynamic shapes does not allow using unsupported NICs, but you can override the automation by manually placing also unsupported NICs in the PCI slots
The image used in the previous example is fit fuzzy it is fairly hard to see the orientation of the NICs
- You can clearly see that NIC2 & NIC3 are in the same orientation
- But what about NIC1?
  - If you look close enough, you can see that the status lights are on top of the ports with NIC1 and below the ports with NIC2 & NIC3
  - Meaning that NIC1 has different orientation than NIC2 & NIC3

So what is the actual problem with different NIC orientations and lack of documentation regarding this?

With 0° orientation NIC, port 1 is typically the LEFTMOST port
Once you flip the NIC by 180° when installing the NIC into 180° orientation PCIe slot
- the port numbers flip as well
And with 180° orientation NIC, port 1 is typically the RIGHTMOST port
Or it might be the other way around
- With 0° orientation port 1 is the RIGHTMOST port
- but the important part is that the port numbers flip when the NIC is flipped by 180° when installing it to 180° orientation PCIe slot
- And port 1 becomes the LEFTMOST port in 180° orientation
Using port numbering in the spec sheets also for NIC ports would solve this issue
For the rest of the documentation I am assuming that
- With 0° orientation NIC, port 1 is the LEFTMOST port
- With 0° orientation NIC, port 2 is the RIGHTMOST port
- With 180° orientation NIC, port 1 is the RIGHTMOST port
- With 180° orientation NIC, port 2 is the LEFTMOST port

Example 4: NX-8150-G7 with 2x10GbE sfp+ NIC / node, ports numbered

With NIC1
- LP form factor NIC on the left
- port 1 is the rightmost port
- port 2 is the leftmost port
- NIC1 is in 180° orientation
With NIC2
- FH form factor NIC on the right
- port 1 is the LEFTmost port
- port 2 is the RIGHTmost port
- NIC2 is in 0° orientation
(Image made with DPTPB Dynamic Shapes, not available in the spec sheets)

Example 5: NX-8150-G7 with 2x10GbE sfp+ NIC / node

If this was factory installed unit, how would you know
- which NIC port is which port number?
  - With some of the actual physical NICs the ports are numbered
  - But sometimes are not numbered
  - And sometimes it is hard to see port numbers in dimly lit data centers
- what is the orientation of the NIC (0° or 180°)?
  - You would have to open the covers and find out where the motherboard PCI slot connectors are
  - With FH NICs ports are usually closer to PCI connector
    - i.e if the ports are on the left, it is a 0° NIC
    - i.e if the ports are on the right, it is a 180° NIC
  - With LP NICs is not so easy to figure out
(Image made with DPTPB Dynamic Shapes, not available in the spec sheets)

Example 6: Messy cabling

What if you
- designed your system prior to having the system at your disposal?
- and there was no documentation of NIC port numbers?
- and you had no idea that the NIC orientation and port numbering might change when the NIC is installed?
- and documented ports used for cabling based on those assumptions?
Well, you might mess up your cabling, while assuming that the left most NIC port is always port 1
And run cable from NIC1 port 2 to switch 1
when you were supposed to run NIC1 port 1 to switch 1
(Image made with DPTPB Dynamic Shapes, not available in the spec sheets)

Conclusion of Nearly no NIC orientation information available

With two possible NIC orientations (and resulting two NIC port number orientations), it is possible that you might run cables from nodes to switches using different NIC ports than you thought you were using.

Will it be a real problem in real life? Maybe, maybe not.

Most likely when everything is working and you have multiple links with load balancing, you would probably not notice that the port numbers are reverted with some of the NICs.

But consider a situation where:

you have only two links
one of your links goes down
to troubleshoot you are asked to replace a cable

Yes, there are status lights with NICs, but the meaning of NIC status lights vary from NIC model to NIC model (well documented in the spec sheets) and you might misinterpret the status light or might just rely on the fact that port 1 is usually the leftmost port. Since your data center is dimly lit, you also missed the stamped port number on the NIC.

As a result you might mistakenly:

Pull the cable from port 1
When you were supposed to pull the cable from port 2.
Pulling the cable from the only working link
Isolating your node totally from the network and the cluster….
Causing HA to kick in and move VMs to nodes that are still able to communicate
Your cluster might not be able to sustain other node loss…

It wouldn’t hurt to document the NIC orientations or at least NIC port numbers in the spec sheets. Some of the motherboard ports are already numbered, why not NIC ports as well?

Wrong NIC transceiver types used, sfp+ vs sfp28, qsfp+ vs qsfp28

While not a major issue, once again makes the documentation look sloppy…

Let’s recap correct usage of terms: sfp+/sfp28/qsfp+/qsfp28

sfp+ = 10GbE
sfp28 = 25GbE
qsfp+ = 40GbE (or 50/56GbE, 4x10GbE)
qsfp28 = 100GbE (or 2x50GbE, 4x25GbE)
Source: https://community.fs.com/blog/sfp-vs-sfp-vs-sf-p28-vs-qsfp-vs-qsf-p28-what-are-the-differences.html

Unfortunately the Nutanix spec sheets use these terms in various and very confusing combinations.

Example1: NX-8155-G6 ports, G6 spec sheet text, page 52

Example 1: Notes

25 GbE should be SFP28yes, you can connect a sfp+ transceiver to sfp28 port, but it only operates at spf+ speed, 10GbE
40 GbE should be QSFP+
- The text used in the example above just plainly wrong

Example2: NX-8155-G7 ports, G7 spec sheet text, pages 37-38

Maybe G7 documentation is more accurate?

Example 2: Notes

Nope, the same errors with 25 Gbe and 40Gbe

Example 3: NX-8155-G7 ports, G7 spec sheet image, page 38

Maybe the image just BELOW the text is better

Example 3: Notes

Yes, they got it right in the image
But, somehow failed to fix the text which sits JUST ON TOP of this image…
As a side note, all NX-8155-G7 NICs are FH form factor, yet 3/4 NICs shown are LP form factor…

Example 4: NX-8035-G7 ports, G7 spec sheet text, page 25