Setup QAT Compatible Hardware

TNSR Supports hardware compatible with Intel® QuickAssist Technology, also known as QAT, for accelerating cryptographic and compression operations.

This hardware can be found in CPIC cards as well as many C3000 and Skylake Xeon systems. Netgate 1541 and Netgate 1537 hardware has an add-on option for a CPIC card.

QAT accelerates several different types of encryption and hashing, including AES and SHA operations. It does not currently accelerate ChaCha20-Poly1305 on most hardware.

Setup Process

Enable SR-IOV in the BIOS

SR-IOV is required for QAT to function in TNSR. SR-IOV enables Virtual Functions which are required for binding by crypto devices.

The procedure to enable SR-IOV varies by platform. Generally this involves rebooting the hardware and entering the BIOS setup, making the change, and then saving and rebooting. The exact location of the SR-IOV option also varies in different BIOS implementations.

Note

Netgate devices which ship with a CPIC card preinstalled will have this step completed at the factory, but double check the BIOS to ensure it is set as expected.

VT-d/IOMMU

From here there are two compatible combinations of settings in the BIOS for QAT. The user can choose one or the other of:

  • VT-d and IOMMU both disabled

  • VT-d and IOMMU both enabled

Either of these choices is sufficient to allow the dataplane to utilize QAT.

Disable VT-d in the BIOS

Certain combinations of hardware may experience problems with QAT when VT-d is enabled in the BIOS. As such, the best practice is to disable VT-t in the BIOS for the best possible experience with QAT. Netgate 1537 and Netgate 1541 devices with a DH895xcc QAT CPIC card installed are known to have this limitation.

Some hardware also requires disabling IOMMU and may function better with VT-d disabled. For example, hardware requiring the vfio-pci PMD which runs with noiommu mode internally.

The procedure to disable VT-d varies by platform. The setting is typically located under Advanced > Chipset Configuration > North Bridge > IIO > VT-d or along a similar path.

If VT-d and QAT are incompatible, the problem can manifest in different ways, including:

  • IPsec tunnels may come up but drop packets or otherwise fail to pass traffic.

  • Errors may appear on the console when the dataplane tries to send buffers to the QAT device:

    [110772.063766] DMAR: [DMA Read] Request device [04:01.0] fault addr 406482000 [fault
   reason 06] PTE Read access is not set
[110773.059440] DMAR: DRHD: handling fault status reg 102

Enable IOMMU in the Kernel Command Line

If VT-d is enabled in the BIOS, IOMMU (Input–Output Memory Management Unit) must also be enabled in the kernel command line at boot time for QAT to function. It functions similar to PCI passthrough, allowing the dataplane to access the QAT device.

TNSR CLI Method

The simplest method to enable IOMMU in the kernel is via the TNSR CLI.

The following procedure adds the IOMMU command line parameters, saves the configuration, and reboots the device.

tnsr(config)# system kernel arguments manual intel_iommu=on iommu=pt
Changes to kernel command line arguments will take effect after system is rebooted.
tnsr(config)# configuration copy running startup
tnsr(config)# reboot now
Reboot initiated. Are you sure? [yes/no]
yes
Manual Method

The same procedue can also be performed manually in the shell.

To enable IOMMU in grub:

  • Open /etc/default/grub in a text editor (as root or with sudo)

  • Locate the line starting with GRUB_CMDLINE_LINUX

  • Check if that line includes intel_iommu=on iommu=pt

  • If those parameters are not included on the line, append them to the end, before the end quote.

  • Save and exit the text editor

  • Run the following command:

    sudo grub-mkconfig -o /boot/grub/grub.cfg

  • Reboot the device

Change the interface driver

Next, change the TNSR dataplane interface driver to one which supports QAT. Currently this is primarily the igb_uio driver. The vfio-pci driver (Interface Drivers) will not operate with certain QAT devices by default:

tnsr# configure
tnsr(config)# dataplane dpdk uio-driver igb_uio

Note

As of TNSR 23.06, vfio-pci is the default driver.

Bypass vfio-pci Compatibility Checks

The vfio-pci driver has compatibility issues with certain QAT devices, including DH895x, C3xxx, and C62x devices. As such, by default it has a deny list which prevents these devices from being activated when using the vfio-pci driver.

There is a method to forcefully bypass the deny list for environments where the vfio-pci driver is a hard requirement.

Warning

The impact of trying to use vfio-pci with one of these devices is unknown and the current best practice is to use igb_uio with these QAT devices.

There is a kernel command line argument which disables the VFIO driver deny list behavior, vfio_pci.disable_denylist=1. This can be added to existing kernel arguments managed in the TNSR CLI:

tnsr(config)# system kernel arguments manual intel_iommu=on iommu=pt vfio_pci.disable_denylist=1
Changes to kernel command line arguments will take effect after system is rebooted.
tnsr(config)# configuration copy running startup
tnsr(config)# reboot now
Reboot initiated. Are you sure? [yes/no]
yes

Warning

The system kernel arguments manual command replaces any existing manual arguments. Make sure to check for and include any other options set using that command before replacing the content.

Activating the setting requires a full reboot to boot the kernel with the new command line argument.

Configure the QAT PCI device in TNSR

Next, configure the QAT device in TNSR.

To configure this device, first locate its PCI ID. TNSR will print the PCI ID when viewing possible parameters for dataplane devices:

tnsr(config)# dataplane dpdk dev ?
0000:03:00.0           Ethernet controller: Intel Corporation Ethernet Connection X552 10 GbE SFP+
0000:03:00.1           Ethernet controller: Intel Corporation Ethernet Connection X552 10 GbE SFP+
0000:04:00.0           Co-processor: Intel Corporation DH895XCC Series QAT
0000:05:00.0           Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01) ( Active Interface eno1 )
0000:05:00.1           Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)

In this instance, the following line from the output is for the QAT device:

0000:04:00.0 Co-processor: Intel Corporation DH895XCC Series QAT

The first value printed on the line is the PCI ID, 0000:04:00.0.

This varies by hardware, and may appear with a slightly different string, such as:

0000:01:00.0          Co-processor: Intel Corporation Atom Processor C3000 Series QuickAssist Technology (rev 11)

Now, tell TNSR the device at that address is a crypto device:

tnsr(config)# dataplane dpdk dev 0000:04:00.0 crypto

If TNSR is running in a virtual machine and the QAT device is passed through from the hypervisor host system using SR-IOV, use crypto-vf at the end of the command instead. When the device is defined with crypto-vf, the dataplane uses the Virtual Function (VF) instead of the Physical Function (PF), since the PF is not directly available in a virtual machine.

Note

Typically a VF can be identified by the string Virtual Function printed in the device description listed by dataplane dpdk dev ?. Some platforms may not make this distinction visible to TNSR, so the general guideline is to use crypto-vf when running in a virtual machine and crypto otherwise.

Note

TNSR will only display device types which are usable by the dataplane. This means:

  • If a PF is available, it is usable by the dataplane and will appear in the device list.

  • If a VF is available without a corresponding PF, the VF is usable by the dataplane and will appear in the device list.

  • If both a VF and corresponding PF are available, only the PF is usable by the dataplane and thus only the PF will appear in the device list.

Activate and check the settings

When viewing the XML configuration with show configuration running, it will contain settings similar to the following example. Note that if other dataplane options are present in the configuration, those will also be visible. Here is how it looks once configured:

<dataplane-config>
   <dpdk>
      <dev>
         <id>0000:04:00.0</id>
         <device-type>crypto</device-type>
      </dev>
      <uio-driver>igb_uio</uio-driver>
   </dpdk>
</dataplane-config>

After configuring the crypto device and uio driver, TNSR will commit the settings to the dataplane configuration.

To activate the new settings, restart the dataplane.

tnsr(config)# service dataplane restart
tnsr(config)# exit
tnsr#

Lastly, using the dataplane shell command, verify that VPP can see the crypto device, and that it is being used to handle cryptographic operations:

tnsr# dataplane shell sudo vppctl show crypto engines
[...]
dpdk_cryptodev      100     DPDK Cryptodev Engine
tnsr# dataplane shell sudo vppctl show crypto async handlers
Algo                        Type            Handler
aes-128-gcm-aad8            async-encrypt   sw_scheduler dpdk_cryptodev*
                            async-decrypt   sw_scheduler dpdk_cryptodev*
[...]

The output of those commands may vary slightly depending on hardware and TNSR version. In both commands, look for the presence of dpdk_cryptodev.

Troubleshooting

If the QAT device does not appear in the show crypto async handlers output, then VPP can not see the crypto device. To correct this, first verify the QAT drivers are loaded, VFs exist for the QAT device, and grub BOOT_IMAGE is passing the necessary iommu parameters.

Verify IOMMU parameters:

$ dmesg | grep iommu

The following parameters should appear somewhere on the BOOT_IMAGE line in the dmesg output:

intel_iommu=on iommu=pt

Verify that the QAT drivers are loaded in the operating system:

$ lsmod | grep qat
qat_dh895xccvf         13281  0
qat_dh895xcc           13510  0
intel_qat             141755  2 qat_dh895xccvf,qat_dh895xcc
dh_generic             13286  1 intel_qat
rsa_generic            18819  1 intel_qat
authenc                17776  1 intel_qat

Verify Virtual Functions (VFs) exist for the QAT device:

$ sudo lspci | grep -i 'co-processor'

The number of listings are dependent on how many threads VPP uses to process packets. At minimum there will be at least three entries, but there may be many more. The lines will look similar to this example:

04:00.0 Co-processor: Intel Corporation DH895XCC Series QAT
04:01.0 Co-processor: Intel Corporation DH895XCC Series QAT Virtual Function
04:01.1 Co-processor: Intel Corporation DH895XCC Series QAT Virtual Function

Note

Some platforms expand the QAT acronym to QuickAssist Technology. If lspci does not recognize the specific chipset, the list may include a device ID such as 19e3, 18ef, or 37c9 instead of the string QAT Virtual Function. Refer to the list of QAT device IDs from DPDK to see if one matches.

TNSR stores the device Physical Function (PF), 04:00.0 for example, in its configuration because the VFs do not yet exist at boot time. They are created by clixon-backend when it processes the crypto device. Then, the allocated VFs on the PF have their addresses written to startup.conf.

The VFs are bound to igb_uio because igb_uio is a driver which allows a userspace process to do RDMA on buffers that are used by a PCI device.

If the drivers are loaded and the VFs show under lspci, then verify /etc/vpp/startup.conf has the appropriate dpdk settings. The igb_uio driver must be present and the PCI IDs of TNSR interfaces along with one of the VFs for the QAT device:

dpdk {
    uio-driver igb_uio
    dev 0000:04:01.0
    dev 0000:05:00.1
    dev 0000:03:00.0
    dev 0000:03:00.1
}

If that looks correct, verify igb_uio is being used by the QAT VF and interfaces:

$ sudo vppctl show pci all | grep igb_uio
0000:03:00.0   0  8086:15ac   2.5 GT/s x1  igb_uio
0000:03:00.1   0  8086:15ac   2.5 GT/s x1  igb_uio
0000:04:01.0   0  8086:0443   unknown      igb_uio
0000:05:00.1   0  8086:1521   5.0 GT/s x4  igb_uio

Physical TNSR interfaces may also be present in that output in addition to the QAT VF ID, which matches the QAT VF ID configured for dpdk in /etc/vpp/startup.conf.

Note

As with lspci, not every QAT VF device is recognized by name, so match up the devices by PCI ID. Additionally, some PF devices will not show igb_uio but the device-appropriate QAT driver instead.

For example, the following QAT PF and VF devices are present on a properly working C3XXX system with QAT:

0000:01:00.0   0  8086:19e2   5.0 GT/s x16  c3xxx
0000:01:01.0   0  8086:19e3   unknown       igb_uio
0000:01:01.1   0  8086:19e3   unknown       c3xxxvf

If any of those tests do not provide the expected output, then reboot the system and check again. Ensure the TNSR services and VPP are running, and then check the VPP QAT status again.

$ sudo vppctl show crypto engines
$ sudo vppctl show crypto async handlers

If there is still no dpdk_cryptodev shown in the output of either command, verify the PCI ID for the crypto device specified in TNSR is accurate. It must be the first PCI ID displayed by sudo lspci | grep -i 'co-processor'. Then verify the PCI ID of the next listing in that output (first VF device) is specified in /etc/vpp/startup.conf properly and also the same PCI ID seen by VPP when running:

$ sudo vppctl show pci all