Mellanox ConnectX-5 Firmware Requirements

Mellanox ConnectX-5 network interface cards (mlx5) in the MT27800 family are currently shipping with firmware revision 16.26.1040 which is not compatible with TNSR.

The incompatible firmware can be identified by errors in the log (e.g. sudo vppctl show errors), such as:

net_mlx5: probe of PCI device [PCI ID] aborted after encountering an error: Operation not supported

Or:

Interface WAN error -12: Unknown error -12
net_mlx5: Failed to query QP using DevX
net_mlx5: Fail to query port 0 Tx queue 0 QP TIS transport domain
net_mlx5: port 0 Tx queue allocation failed: Cannot allocate memory
Device with port_id=0 already stopped

Firmware version 16.24.1000 is compatible with TNSR, and can be manually downgraded using the following procedure.

First, identify the first PCI ID of the card in question. This can be found by looking in the boot logs, the output of a utility such as lspci, or similar methods. The ID will take the form of xx:yy.z, for example 65:00.0. The ID will be used in the following set of commands.

Next, download and decompress the appropriate firmware:

$ curl -LO http://www.mellanox.com/downloads/firmware/fw-ConnectX5-rel-16_24_1000-MCX516A-CCA_Ax-UEFI-14.17.11-FlexBoot-3.5.603.bin.zip
$ unzip fw-ConnectX5-rel-16_24_1000-MCX516A-CCA_Ax-UEFI-14.17.11-FlexBoot-3.5.603.bin.zip

Now install the Mellanox firmware tool and perform the firmware downgrade:

$ sudo apt install mstflint
$ sudo mstconfig q > mst.log
$ sudo mstfwmanager -d <PCI ID> -u -f -i fw-ConnectX5-rel-16_24_1000-MCX516A-CCA_Ax-UEFI-14.17.11-FlexBoot-3.5.603.bin

Replace <PCI ID> in the last command with the first PCI ID of the card.

With the appropriate firmware loaded on the card, it will no longer produce errors and it will be usable by TNSR.

Warning

Reboot after performing the firmware downgrade to ensure the card is fully reinitialized with the appropriate firmware.