CPU Workers and Affinity

The dataplane has a variety of commands to fine-tune how it uses available CPU resources on the host. These commands control CPU cores TNSR will use, both the number of cores and specific cores.

See also

Cores defined here may also be pinned to interface receive (RX) queues, provided that cores are defined using either the corelist-workers or coremask-workers methods. See Interface Configuration Options for details.

Default Worker Behavior

The host operating system and daemons reserve core 0 by default, unless another core is reserved elsewhere and core 0 is reassigned. For example, if skip-cores is set to a value other than 0, and core 0 is assigned to another task, then the host OS will use one of the other available skipped cores.

Note

Core 0 could be used as a last resort by automatic assignment (e.g. workers with skip-cores or default main-core) if no other cores are available.

The dataplane reserves core 1 by default. Workers may not be assigned to core 1 unless the dataplane is moved to a different core using main-core as detailed in the next section. An exception to this is a single core system in which case, the main-core would be 0 since it is the only available core.

The dataplane does not use any additional cores by default until configured using one of the available methods (workers, corelist-workers, coremask-workers). Given that two cores are reserved for the host OS and dataplane, the amount of cores available for use by workers is always 2 less than the total number of cores.

Worker Configuration

dataplane cpu corelist-workers <first> [- <last>]

Defines a specific list of CPU cores to be used by the dataplane. The command supports adding single cores to the list at a time, or ranges of cores. Run the command multiple times with different core numbers or ranges to define the full list of cores to utilize. When removing items with no, the command accepts a specific core to remove from the list.

dataplane cpu coremask-workers <mask>

Similar to corelist-workers, but the cores are defined as a hexadecimal mask instead of a list. For example, 0x0000000000C0000C

dataplane cpu main-core <n>

Assigns the main dataplane process to a specific CPU core. The default core is 1, and core 0 should not be used as it is reserved for the operating system.

dataplane cpu scheduler-policy (batch|fifo|idle|other|rr)

Defines a specific scheduler policy for worker thread processor usage allocation

batch

Scheduling batch processes. Uses dynamic priorities based on nice values in the host OS, but always gives the thread a small scheduling penalty so that other processes take precedence.

fifo

First in-first out scheduling. Will preempt other types of threads and threads with a lower priority.

idle

Scheduling very low priority jobs.

other

Default Linux time-sharing scheduling. Uses dynamic priorities based on nice values in the host OS, similar to batch but without the built-in penalty.

rr

Round-robin scheduling. Similar to fifo but each thread is time-limited

dataplane cpu scheduler-priority <n>

For the fifo and rr scheduler policies, this number sets the priority of processes for the dataplane. It can be any number between 1 (low) and 99 (high).

dataplane cpu skip-cores <n>

Defines the number of cores to skip when creating additional worker threads, in the range of 0 to the highest available core number. The first <n> cores will not be used by worker threads.

Note

This does not affect the core used by the main thread, which is set by dataplane cpu main-core <n>. A skipped core can, however, be used by the host OS instead of core 0.

Warning

This option is incompatible with interface RX queue core pinning. To utilize interface RX queue core pinning, define a list of cores using either corelist-workers or coremask-workers instead.

dataplane cpu workers <n>

Defines the number of worker threads to create for the dataplane. The placement of workers on CPU cores is handled automatically but can be influenced by skip-cores.

Note

The number of worker threads is in addition to the main process. For example, with a worker count of 4, the dataplane will use one main process with four worker threads, for a total of five threads.

The maximum value for workers is 2 less than the total number of available cores. This is because one core is reserved for the host, and one for the dataplane itself.

Warning

This option is incompatible with interface RX queue core pinning. To utilize interface RX queue core pinning, define a list of cores using either corelist-workers or coremask-workers instead.

Note

The workers, corelist-workers and coremask-workers methods are all mutually exclusive. Only one of these methods of defining workers is allowed in the configuration at any time.

Worker Example

This example sets four additional worker threads, and instructs the dataplane to skip one core when assigning worker threads to cores:

tnsr(config)# dataplane cpu workers 4
tnsr(config)# dataplane cpu skip-cores 1
tnsr(config)# service dataplane restart

Worker Status

The show dataplane cpu threads command displays the current dataplane process list, including the core usage and process IDs. This output corresponds to the example above:

tnsr(config)# show dataplane cpu threads
ID Name     Type    PID  LCore Core Socket
-- -------- ------- ---- ----- ---- ------
 0 vpp_main         2330     1    0      0
 1 vpp_wk_0 workers 2346     2    2      0
 2 vpp_wk_1 workers 2347     3    3      0
 3 vpp_wk_2 workers 2348     4    4      0
 4 vpp_wk_3 workers 2349     5    5      0

The output includes the following columns:

id

Dataplane thread ID.

name

Name of the dataplane process.

type

The type of thread, which will be blank for the main process.

pid

The host OS process ID for each thread.

LCore

The logical core used by the process.

Core

The CPU core used by the process.

Socket

The CPU socket associated with the core used by the process.