CPU Workers and Affinity

The dataplane has a variety of commands to fine-tune how it uses available CPU resources on the host. These commands control CPU cores TNSR will use, both the number of cores and specific cores.

See also

Cores defined here may also be pinned to interface receive (RX) queues, provided that cores are defined using corelist-workers. See Interface Configuration Options for details.

Warning

When adding workers, tuning memory values may be required, especially for the statistics segment. See Memory Usage and Tuning for details.

Default Worker Behavior

The host operating system and daemons reserve core 0 by default, unless another core is reserved elsewhere and core 0 is reassigned. For example, if skip-cores is set to a value other than 0, and core 0 is assigned to another task, then the host OS will use one of the other available skipped cores.

Note

Core 0 could be used as a last resort by automatic assignment (e.g. workers with skip-cores or default main-core) if no other cores are available.

Core affinity is disabled by default in the dataplane, thus the dataplane does not reserve any cores until configured to do so with dataplane cpu main-core. Additionally, the dataplane does not use any additional cores by default until configured using one of the available methods (workers, corelist-workers).

Determining the Number of Workers

When workers are configured, the workers handle traffic while the main thread handles coordination and other tasks. As such, workers can improve the total potential throughput significantly at a cost of increased memory requirements.

Given that the host OS and dataplane should each run on their own core, ideally the amount of cores used by workers should be 2 less than the total number of cores. However, it is possible for the host OS and main thread to share a core, leaving all but one core available for workers.

On systems with smaller numbers of cores (e.g. 4), having the host OS and main thread on the same core allows the remaining cores to all be used by workers to process traffic, which can potentially offer a significant speed increase on those platforms.

If the host OS core includes CPU-intensive daemons such as strongSwan (IPsec) or FRR (BGP, OSPF), the main core could be bogged down as a result. The load on the daemons can vary significantly depending on the configuration and usage. The only way to know for certain if a configuration would be better with the extra worker in this case or not is to test each scenario in the same environment.

Worker Configuration

dataplane cpu corelist-workers <first> [- <last>]:

Defines a specific list of CPU cores to be used by the dataplane. The command supports adding single cores to the list at a time, or ranges of cores. Run the command multiple times with different core numbers or ranges to define the full list of cores to utilize. When removing items with no, the command accepts a specific core to remove from the list.

See also

To ensure the OS does not schedule processes on these cores, set them as isolated CPUs as described in Automatic CPU Isolation.

dataplane cpu main-core <n>:

Enables core affinity and assigns the main dataplane process to a specific CPU core.

Warning

New configurations of TNSR 22.10 have core affinity disabled by default. Core affinity must be enabled before any other dataplane cpu properties can be used.

Core 0 is used by the operating system. Ideally the main dataplane process should use a separate core, but core 0 can be shared with the main dataplane process if there is enough CPU power to spare on that core.

Note

TNSR version 22.06 and earlier enabled core affinity by default and had a default main core of 1, or 0 on systems with a single core. When upgrading to TNSR version 22.10 or later TNSR will add a main-core value to the configuration equivalent to the previous default for the platform.

dataplane cpu skip-cores <n>:

Defines the number of cores to skip when creating additional worker threads, in the range of 0 to the highest available core number. The first <n> cores will not be used by worker threads.

Note

This does not affect the core used by the main thread, which is set by dataplane cpu main-core <n>. A skipped core can, however, be used by the host OS instead of core 0.

Warning

This option is incompatible with interface RX queue core pinning. To utilize interface RX queue core pinning, define a list of cores using corelist-workers instead.

dataplane cpu workers <n>:

Defines the number of worker threads to create for the dataplane. The placement of workers on CPU cores is handled automatically but can be influenced by skip-cores.

Note

The number of worker threads is in addition to the main process. For example, with a worker count of 4, the dataplane will use one main process with four worker threads, for a total of five threads.

The maximum value for workers is 1 less than the total number of available cores.

Warning

This option is incompatible with interface RX queue core pinning. To utilize interface RX queue core pinning, define a list of cores using corelist-workers instead.

Note

Core affinity must be enabled with dataplane cpu main-core <n> before any other dataplane cpu properties can be used.

The workers and corelist-workers methods are mutually exclusive. Only one of these methods of defining workers is allowed in the configuration at any time.

Worker Example

This example enables core affinity by assigning the main core to core 1, sets four additional worker threads, and instructs the dataplane to skip one core when assigning worker threads to cores:

tnsr(config)# dataplane cpu main-core 1
tnsr(config)# dataplane cpu workers 4
tnsr(config)# dataplane cpu skip-cores 1
tnsr(config)# service dataplane restart

Worker Status

The show dataplane cpu threads command displays the current dataplane process list, including the core usage and process IDs.

The output includes the following columns:

id:

Dataplane thread ID.

name:

Name of the dataplane process.

type:

The type of thread, which will be blank for the main process.

pid:

The host OS process ID for each thread.

LCore:

The logical core used by the process.

Core:

The CPU core used by the process.

Socket:

The CPU socket associated with the core used by the process.

If core affinity is disabled, some of the fields will show n/a as they are not assigned to specific cores:

# show dataplane cpu threads
ID Name     Type PID      LCore       Core     Socket
-- -------- ---- --- ---------- ---------- ----------
 0 vpp_main      819        n/a        n/a        n/a

This output corresponds to the previous example with four workers:

tnsr(config)# show dataplane cpu threads
ID Name     Type    PID  LCore Core Socket
-- -------- ------- ---- ----- ---- ------
 0 vpp_main         2330     1    0      0
 1 vpp_wk_0 workers 2346     2    2      0
 2 vpp_wk_1 workers 2347     3    3      0
 3 vpp_wk_2 workers 2348     4    4      0
 4 vpp_wk_3 workers 2349     5    5      0