NVIDIA® Network Interface Cards Using the mlx5 Driver

NVIDIA hardware, such as ConnectX® devices like the ConnectX-5, may require special configuration for use with TNSR. These devices use the mlx5 driver in DPDK, which supports numerous devices in the ConnectX and BlueField product lines. The limitations noted on this page affect all devices supported by this driver.

Note

This product line and related devices were formerly sold under the brand “Mellanox®” which was acquired by NVIDIA in 2020. Current hardware uses only the NVIDIA name. Hardware, drivers, and documentation may still reference the Mellanox branding.

See also

For a list of devices the mlx5 driver supports, see the DPDK documentation <https://doc.dpdk.org/guides-24.11/nics/mlx5.html>.

mlx5 Driver Jumbo Frame Support

Network interface cards which use the mlx5 driver will not pass jumbo frames unless the default MTU is set to 9000 and dataplane dpdk no-multi-seg is unset.

To utilize jumbo frames on any network interface card using the mlx5 driver, the default MTU must be set to 9000, then other interfaces can have a different MTU set individually.

For example, to utilize jumbo frames on internal interfaces, first change the default to 9000:

tnsr(config)# dataplane ethernet default-mtu 9000

And then configure any interface which should not use jumbo frames with its appropriate lower MTU:

tnsr(config)# interface WAN
tnsr(config-interface)# mtu 1500
tnsr(config-interface)# exit

Any interface without a manually configured MTU will use the default MTU, which in this example is 9000, and this allows the mlx5 driver to use jumbo frames.

Tip

The default MTU cannot be set above 9000. However, once the default is set to 9000, individual interfaces can be set with a higher MTU.

After setting the default MTU, ensure multi-segment buffers are not disabled:

tnsr(config)# no dataplane dpdk no-multi-seg
tnsr(config)# service dataplane restart

Note

This is the default setting in TNSR. However, this setting was formerly a requirement for mlx5 driver devices on previous versions of TNSR, so it may be set in existing configurations.

mlx5 Driver Firmware Requirements

Network interface cards using the mlx5 driver in the MT27800 family are currently shipping with firmware revision 16.26.1040 which is not compatible with TNSR.

The incompatible firmware can be identified by errors in the log (e.g. sudo vppctl show errors), such as:

net_mlx5: probe of PCI device [PCI ID] aborted after encountering an error: Operation not supported

Or:

Interface WAN error -12: Unknown error -12
net_mlx5: Failed to query QP using DevX
net_mlx5: Fail to query port 0 Tx queue 0 QP TIS transport domain
net_mlx5: port 0 Tx queue allocation failed: Cannot allocate memory
Device with port_id=0 already stopped

Firmware version 16.24.1000 is compatible with TNSR, and can be manually downgraded using the following procedure.

First, identify the first PCI ID of the card in question. This can be found by looking in the boot logs, the output of a utility such as lspci, or similar methods. The ID will take the form of xx:yy.z, for example 65:00.0. The ID will be used in the following set of commands.

Next, download and decompress the appropriate firmware:

$ curl -LO http://www.mellanox.com/downloads/firmware/fw-ConnectX5-rel-16_24_1000-MCX516A-CCA_Ax-UEFI-14.17.11-FlexBoot-3.5.603.bin.zip
$ unzip fw-ConnectX5-rel-16_24_1000-MCX516A-CCA_Ax-UEFI-14.17.11-FlexBoot-3.5.603.bin.zip

Now install the firmware tool and perform the firmware downgrade:

$ sudo apt install mstflint
$ sudo mstconfig q > mst.log
$ sudo mstfwmanager -d <PCI ID> -u -f -i fw-ConnectX5-rel-16_24_1000-MCX516A-CCA_Ax-UEFI-14.17.11-FlexBoot-3.5.603.bin

Replace <PCI ID> in the last command with the first PCI ID of the card.

With the appropriate firmware loaded on the card, it will no longer produce errors, and it will be usable by TNSR.

Warning

Reboot after performing the firmware downgrade to ensure the card is fully reinitialized with the appropriate firmware.