Tip
This is the documentation for the 20.02 version. Looking for the documentation of the latest version? Have a look here.
Setup QAT Compatible Hardware¶
TNSR Supports hardware compatible with Intel® QuickAssist Technology, also known as QAT, for accelerating cryptographic and compression operations.
This hardware can be found in CPIC cards as well as many C3000 and Skylake Xeon systems. Netgate XG-1541 and XG-1537 hardware has an add-on option for a CPIC card.
Setup Process¶
Enable SR-IOV in the BIOS¶
SR-IOV is required for QAT to function in TNSR. SR-IOV enables Virtual Functions which are required for binding by crypto devices.
The procedure to enable SR-IOV varies by platform. Generally this involves rebooting the hardware and entering the BIOS setup, making the change, and then saving and rebooting. The exact location of the SR-IOV option also varies in different BIOS implementations.
Note
Netgate devices which ship with a CPIC card preinstalled will have this step completed at the factory, but double check the BIOS to ensure it is set as expected.
Disable VT-d in the BIOS¶
Certain combinations of hardware may experience problems with QAT when VT-d is enabled in the BIOS. As such, we recommend disabling VT-t in the BIOS for the best possible experience with QAT. Netgate XG-1537 and XG-1541 devices with a DH895xcc QAT CPIC card installed are known to have this limitation.
Note
One exception to this is the SG-5100 device, which is capable of using QAT while VT-d is active in its BIOS.
The procedure to disable VT-d varies by platform. The setting is typically located under Advanced > Chipset Configuration > North Bridge > IIO > VT-d or along a similar path.
If VT-d and QAT are incompatible, the problem can manifest in a few different ways, including:
IPsec tunnels may come up but drop packets or otherwise fail to pass traffic.
Errors may appear on the console when the dataplane tries to send buffers to the QAT device:
[110772.063766] DMAR: [DMA Read] Request device [04:01.0] fault addr 406482000 [fault reason 06] PTE Read access is not set [110773.059440] DMAR: DRHD: handling fault status reg 102
The number of used “in flight” buffer resources will continually increase as traffic attempts to pass through IPsec, as observed under
used_resources
in the output of theshell sudo vppctl show dpdk crypto devices
CLI command.
Enable IOMMU in grub¶
IOMMU (Input–Output Memory Management Unit), which in this context is also known
as Intel VT-d, must be enabled in grub
for QAT to function. It functions
similar to PCI passthrough, allowing the dataplane to access the QAT device.
To enable IOMMU in grub
:
Open
/etc/default/grub
in a text editor (as root or withsudo
)Locate the line starting with
GRUB_CMDLINE_LINUX
Check if that line includes
intel_iommu=on iommu=pt
If those parameters are not included on the line, append them to the end, before the end quote.
Save and exit the text editor
Run one following commands (depending on how the device boots):
Legacy:
sudo grub2-mkconfig -o /boot/grub2/grub.cfg
UEFI:
sudo grub2-mkconfig -o /boot/efi/EFI/centos/grub.cfg
Reboot the device
Change the uio driver to igb_uio¶
Next, change the TNSR dataplane uio
driver to igb_uio
:
tnsr# configure
tnsr(config)# dataplane dpdk uio-driver igb_uio
Note
As of TNSR 20.02, igb_uio
is the default driver.
Configure the QAT PCI device in TNSR¶
Next, configure the QAT device in TNSR.
To configure this device, first locate its PCI ID. TNSR will print the PCI ID when viewing possible parameters for dataplane devices
tnsr(config)# dataplane dpdk dev ?
0000:03:00.0 Ethernet controller: Intel Corporation Ethernet Connection X552 10 GbE SFP+
0000:03:00.1 Ethernet controller: Intel Corporation Ethernet Connection X552 10 GbE SFP+
0000:04:00.0 Co-processor: Intel Corporation DH895XCC Series QAT
0000:05:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01) ( Active Interface eno1 )
0000:05:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
In this instance, the following line from the output is for the QAT device:
0000:04:00.0 Co-processor: Intel Corporation DH895XCC Series QAT
The first value printed on the line is the PCI ID, 0000:04:00.0
.
Now, tell TNSR the device at that address is a crypto
device:
tnsr(config)# dataplane dpdk dev 0000:04:00.0 crypto
If TNSR is running in a virtual machine and the QAT device is passed through
from the hypervisor host system using SR-IOV, use crypto-vf
at the end of
the command instead. When the device is defined with crypto-vf
, the
dataplane uses the Virtual Function (VF) instead of the Physical Function (PF),
since the PF is not directly available in a virtual machine.
Note
Typically a VF can be identified by the string Virtual Function
printed in the device description listed by dataplane dpdk dev ?
. Some
platforms may not make this distinction visible to TNSR, so the general
guideline is to use crypto-vf
when running in a virtual machine and
crypto
otherwise.
Note
TNSR will only display device types which are usable by the dataplane. This means:
If a PF is available, it is usable by the dataplane and will appear in the device list.
If a VF is available without a corresponding PF, the VF is usable by the dataplane and will appear in the device list.
If both a VF and corresponding PF are available, only the PF is usable by the dataplane and thus only the PF will appear in the device list.
Activate and check the settings¶
When viewing the XML configuration with show configuration running
, it will
contain settings similar to the following example. Note that if other dataplane
options are present in the configuration, those will also be visible. Here is
how it looks once configured:
<dataplane-config>
<dpdk>
<dev>
<id>0000:04:00.0</id>
<device-type>crypto</device-type>
</dev>
<uio-driver>igb_uio</uio-driver>
</dpdk>
</dataplane-config>
After configuring the crypto
device and uio
driver, TNSR will commit the
settings to the dataplane configuration.
To activate the new settings, restart the dataplane.
tnsr(config)# service dataplane restart
tnsr(config)# exit
tnsr#
Lastly, using the shell
command, verify that VPP can see the crypto device:
tnsr# shell sudo vppctl show dpdk crypto devices
0000:04:00.0_qat_sym crypto_qat up
numa_node 0, max_queues 2
free_resources 0, used_resources 1
SYMMETRIC_CRYPTO, SYM_OPERATION_CHAINING, HW_ACCELERATED, IN_PLACE_SGL, OOP_SGL_IN_SGL_OUT, OOP_SGL_IN_LB_OUT, OOP_LB_IN_SGL_OUT, OOP_LB_IN_LB_OUT
Cipher: none, aes-cbc-128, aes-cbc-192, aes-cbc-256, aes-ctr-128, aes-ctr-192, aes-ctr-256, aes-gcm-128, aes-gcm-192, aes-gcm-256
Auth: none, md5-96, sha1-96, sha-256-96, sha-256-128, sha-384-192, sha-512-256
Troubleshooting¶
If the QAT device does not appear in the show dpdk crypto devices
output, or
it only shows an AES-NI device, then VPP can not see the crypto device. To
correct this, first verify the QAT drivers are loaded, VFs exist for the QAT
device, and grub BOOT_IMAGE is passing the necessary iommu parameters.
Verify IOMMU parameters:
$ dmesg | grep iommu
The following parameters should appear somewhere on the BOOT_IMAGE line in the
dmesg
output:
intel_iommu=on iommu=pt
Verify that the QAT drivers are loaded in the operating system:
$ lsmod | grep qat
qat_dh895xccvf 13281 0
qat_dh895xcc 13510 0
intel_qat 141755 2 qat_dh895xccvf,qat_dh895xcc
dh_generic 13286 1 intel_qat
rsa_generic 18819 1 intel_qat
authenc 17776 1 intel_qat
Verify Virtual Functions (VFs) exist for the QAT device:
$ lspci | grep QAT | wc -l
The number of listings are dependent on how many threads VPP uses to process packets. At minimum there will be at least three entries, but there may be many more. The lines will look similar to this example:
04:00.0 Co-processor: Intel Corporation DH895XCC Series QAT
04:01.0 Co-processor: Intel Corporation DH895XCC Series QAT Virtual Function
04:01.1 Co-processor: Intel Corporation DH895XCC Series QAT Virtual Function
TNSR stores the device Physical Function (PF), 04:00.0
for example, in its
configuration because the VFs do not yet exist at boot time. They are created
by clixon-backend
when it processes the crypto
device. Then, the
allocated VFs on the PF have their addresses written to startup.conf
.
The VFs are bound to igb_uio
because igb_uio
is a driver which allows a
userspace process to do RDMA on buffers that are used by a PCI device.
If the drivers are loaded and the VFs show under lspci
, then verify
/etc/vpp/startup.conf
has the appropriate dpdk
settings. The igb_uio
driver must be present and the PCI IDs of TNSR interfaces along with one of the
VFs for the QAT device:
dpdk {
uio-driver igb_uio
dev 0000:04:01.0
dev 0000:05:00.1
dev 0000:03:00.0
dev 0000:03:00.1
}
If that looks correct, verify igb_uio
is being used by the QAT VF and
interfaces:
$ sudo vppctl show pci all | grep igb_uio
0000:03:00.0 0 8086:15ac 2.5 GT/s x1 igb_uio
0000:03:00.1 0 8086:15ac 2.5 GT/s x1 igb_uio
0000:04:01.0 0 8086:0443 unknown igb_uio
0000:05:00.1 0 8086:1521 5.0 GT/s x4 igb_uio
Physical TNSR interfaces will display there in addition to the QAT VF ID, which
matches the QAT VF ID configured for dpdk
in /etc/vpp/startup.conf
.
If any of those tests do not provide the expected output, then reboot the system and check again. Ensure the TNSR services and VPP are running, and then check the VPP QAT status again.
$ sudo vppctl show dpdk crypto devices
If there is still no output, verify the PCI ID for the crypto device specified
in TNSR is accurate. It must be the first PCI ID displayed by
lspci | grep qat
. Then verify the PCI ID of the next listing in that output
(first VF device) is specified in /etc/vpp/startup.conf
properly and also
the same PCI ID seen by VPP when running:
$ sudo vppctl show pci all | grep igb_uio