Tip

This is the documentation for the 22.06 version. Looking for the documentation of the latest version? Have a look here.

CPU Workers and Affinity¶

The dataplane has a variety of commands to fine-tune how it uses available CPU resources on the host. These commands control CPU cores TNSR will use, both the number of cores and specific cores.

Default Worker Behavior¶

The host operating system and daemons reserve core 0 by default, unless another core is reserved elsewhere and core 0 is reassigned. For example, if skip-cores is set to a value other than 0, and core 0 is assigned to another task, then the host OS will use one of the other available skipped cores.

Note

Core 0 could be used as a last resort by automatic assignment (e.g. workers with skip-cores or default main-core) if no other cores are available.

The dataplane reserves core 1 by default. Workers may not be assigned to core 1 unless the dataplane is moved to a different core using main-core as detailed in the next section. An exception to this is a single core system in which case, the main-core would be 0 since it is the only available core.

The dataplane does not use any additional cores by default until configured using one of the available methods (workers, corelist-workers, coremask-workers).

Determining the Number of Workers¶

When workers are configured, the workers handle traffic while the main thread handles coordination and other tasks. As such, workers can improve the total potential throughput significantly at a cost of increased memory requirements.

Given that the host OS and dataplane should each run on their own core, ideally the amount of cores used by workers should be 2 less than the total number of cores. However, it is possible for the host OS and main thread to share a core, leaving all but one core available for workers.

On systems with smaller numbers of cores (e.g. 4), having the host OS and main thread on the same core allows the remaining cores to all be used by workers to process traffic, which can potentially offer a significant speed increase on those platforms.

If the host OS core includes CPU-intensive daemons such as strongSwan (IPsec) or FRR (BGP, OSPF), the main core could be bogged down as a result. The load on the daemons can vary significantly depending on the configuration and usage. The only way to know for certain if a configuration would be better with the extra worker in this case or not is to test each scenario in the same environment.

Worker Configuration¶

dataplane cpu corelist-workers <first> [- <last>]:

Defines a specific list of CPU cores to be used by the dataplane. The command supports adding single cores to the list at a time, or ranges of cores. Run the command multiple times with different core numbers or ranges to define the full list of cores to utilize. When removing items with no, the command accepts a specific core to remove from the list.

dataplane cpu coremask-workers <mask>:

Similar to corelist-workers, but the cores are defined as a hexadecimal mask instead of a list. For example, 0x0000000000C0000C

dataplane cpu main-core <n>:

Assigns the main dataplane process to a specific CPU core. The default core is 1.

Core 0 is used by the operating system. Ideally the main dataplane process should use a separate core, but core 0 can be shared with the main dataplane process if there is enough CPU power to spare on that core.

dataplane cpu skip-cores <n>:

Defines the number of cores to skip when creating additional worker threads, in the range of 0 to the highest available core number. The first <n> cores will not be used by worker threads.

Note

This does not affect the core used by the main thread, which is set by dataplane cpu main-core <n>. A skipped core can, however, be used by the host OS instead of core 0.

Warning

This option is incompatible with interface RX queue core pinning. To utilize interface RX queue core pinning, define a list of cores using either corelist-workers or coremask-workers instead.

dataplane cpu workers <n>:

Defines the number of worker threads to create for the dataplane. The placement of workers on CPU cores is handled automatically but can be influenced by skip-cores.

Note

The number of worker threads is in addition to the main process. For example, with a worker count of 4, the dataplane will use one main process with four worker threads, for a total of five threads.

The maximum value for workers is 1 less than the total number of available cores.

Warning

This option is incompatible with interface RX queue core pinning. To utilize interface RX queue core pinning, define a list of cores using either corelist-workers or coremask-workers instead.

Note

The workers, corelist-workers and coremask-workers methods are all mutually exclusive. Only one of these methods of defining workers is allowed in the configuration at any time.

Worker Example¶

This example sets four additional worker threads, and instructs the dataplane to skip one core when assigning worker threads to cores:

tnsr(config)# dataplane cpu workers 4
tnsr(config)# dataplane cpu skip-cores 1
tnsr(config)# service dataplane restart

Worker Status¶

The show dataplane cpu threads command displays the current dataplane process list, including the core usage and process IDs. This output corresponds to the example above:

tnsr(config)# show dataplane cpu threads
ID Name     Type    PID  LCore Core Socket
-- -------- ------- ---- ----- ---- ------
 0 vpp_main         2330     1    0      0
 1 vpp_wk_0 workers 2346     2    2      0
 2 vpp_wk_1 workers 2347     3    3      0
 3 vpp_wk_2 workers 2348     4    4      0
 4 vpp_wk_3 workers 2349     5    5      0

The output includes the following columns:

id:: Dataplane thread ID.
name:: Name of the dataplane process.
type:: The type of thread, which will be blank for the main process.
pid:: The host OS process ID for each thread.
LCore:: The logical core used by the process.
Core:: The CPU core used by the process.
Socket:: The CPU socket associated with the core used by the process.