Hardware Tuning and Troubleshooting¶

The underlying operating system beneath pfSense® software can be fine-tuned in several ways. A few of these tunables are available under Advanced Options (See System Tunables). Others are outlined in the FreeBSD main page tuning(7).

The default installation includes a well-rounded set of values tuned for good performance without being overly aggressive. There are cases where hardware or drivers necessitate changing values or a specific network workload requires changes to perform optimally.

The hardware sold in the Netgate Store is tuned further since Netgate has detailed knowledge of the hardware, removing the need to rely on more general assumptions.

Note

Refer to Managing Loader Tunables for making edits to define loader tunables persistently.

Changes in loader tunables require a firewall reboot to take effect.

General Issues ¶

Filesystem Tuning ¶

For information on tuning ZFS memory usage, see ZFS Tuning.

Mbuf Exhaustion ¶

A common problem encountered by users of commodity hardware is mbuf exhaustion. To oversimplify, “mbufs” are network memory buffers; portions of RAM set aside for use by networking for moving data around.

The count of active mbufs is shown on the dashboard and is tracked by a graph under Status > Monitoring.

Disable MSIX ¶

Message Signaled Interrupts are an alternative to classic style Interrupts for retrieving data from hardware. Some cards behave better with MSI, MSIX, or classic style Interrupts, but the card will try the best available choice (MSIX, then MSI, then Interrupts).

MSIX and MSI can be disabled via loader tunables. Add the following as Loader Tunables:

hw.pci.enable_msix="0"
hw.pci.enable_msi="0"

To nudge the card to use MSI, disable only MSIX. To nudge the card to use regular Interrupts, disable both MSI and MSIX.

PPPoE with Multi-Queue NICs ¶

Network cards which support multiple queues rely on hashing to assign traffic to a particular queue. This works well with IPv4/IPv6 TCP and UDP traffic, for example, but fails with other protocols such as those used for PPPoE.

This can lead to a network card under performing with the default network settings, as noted on #4821 and FreeBSD PR 203856. This problem primarily affects systems with multiple CPUs and/or CPU cores, as those are the systems which benefit most from multiple NIC queues.

Adding a System Tunable or Loader Tunable entry for net.isr.dispatch=deferred can lead to performance gains on affected hardware.

Tuning the values of net.isr.maxthreads and net.isr.numthreads may yield additional performance gains. Generally these are best left at default values matching the number of CPU cores, but depending on the workload may work better at lower values.

Warning

In the past, deferred mode has led to issues on 32-bit platforms, such as crashes/panics, especially with ALTQ. There have been no recent reports, however, so it should be safe on current releases.

TSO/LRO ¶

The settings for Hardware TCP Segmentation Offload (TSO) and Hardware Large Receive Offload (LRO) under System > Advanced on the Networking tab default to checked (disabled) for good reason. Nearly all hardware/drivers have issues with these settings, and they can lead to throughput issues. Ensure the options are checked. Sometimes disabling via sysctl is also necessary.

Add the following as a Loader Tunable:

net.inet.tcp.tso="0"

IP Input Queue (intr_queue)¶

This will show the current setting:

sysctl net.inet.ip.intr_queue_maxlen

However, in largely loaded installations this may not be enough. Here is how to check:

sysctl net.inet.ip.intr_queue_drops

If the above shows values above 0, try doubling the current value of net.inet.ip.intr_queue_maxlen.

For example:

sysctl net.inet.ip.intr_queue_maxlen="3000"

Keep performing the above until the point is found where drops are eliminated without any adverse effects.

Afterwards, add an entry under System > Advanced, System Tunables tab to set net.inet.ip.intr_queue_maxlen to 3000

Card-Specific Issues ¶

Broadcom bce(4) Cards ¶

Several users have noted issues with certain Broadcom network cards, especially those built into Dell hardware. If bce interfaces are behaving erratically, dropping packets, or causing crashes, then the following tweaks may help.

Add the following as Loader Tunables:

kern.ipc.nmbclusters="1000000"
hw.bce.tso_enable="0"
hw.pci.enable_msix="0"

That will increase the amount of network memory buffers, disable TSO directly, and disable msix.

Packet loss with many (small) UDP packets¶

If a lot of packet loss is observed with UDP on bce cards, try changing the netisr settings. These can be set as system tunables under System > Advanced, on the System Tunables tab. On that page, add two new tunables:

net.isr.direct_force="1"
net.isr.direct="1"

Broadcom bge(4) Cards ¶

See above, but change “bce” to “bge” in the setting names.

Chelsio cxgbe(4) Cards ¶

Rate Limiting¶

By default the Chelsio driver implements rate limiting capabilies which can be undesirable for routing performance. To disable this rate limiting capability, add the following as a Loader Tunable:

hw.cxgbe.niccaps_allowed="1"

Resource Allocation¶

It is possible to disable the allocation of resources that are not related to the router so that the network adapter can use its entire set of resources for the corresponding functions:

Add the following as Loader Tunables:

hw.cxgbe.toecaps_allowed="0"
hw.cxgbe.rdmacaps_allowed="0"
hw.cxgbe.iscsicaps_allowed="0"
hw.cxgbe.fcoecaps_allowed="0"

Chelsio TCP Offload Engine (TOE)¶

There is experimental support for the Chelsio TCP Offload Engine (TOE) via the t4_tom kernel module. TOE offloads the entire TCP connection to hardware, but this also can cause problems with connection handling.

Warning

According to analysis of Chelsio TCP Offload Engine behavior by Calomel, under certain circumstances traffic handled by TOE bypasses the FreeBSD TCP stack and will not be filtered nor logged by pf.

Users have observed this behavior particularly when using reroot instead of a full reboot, which left services exposed.

While this feature could, in theory, reduce CPU usage for handling TCP connections, it should not be used in any role where security is a primary concern. For internal routing-only or private endpoint roles it may be acceptable.

Intel igb(4) and em(4) Cards ¶

Certain intel igb cards, especially multi-port cards, can easily exhaust mbufs and cause kernel panics. The following tweak will prevent this from being an issue. Add the following as a Loader Tunable:

kern.ipc.nmbclusters="1000000"

That will increase the amount of network memory buffers, allowing the driver enough headroom for its optimal operation.

Intel ix(4) Cards ¶

Autonegotiate Non-default Speeds¶

The ix(4) hardware may be capable of linking at non-default speeds such as 2.5G and 5G, depending on the chipset, installed SFP module, and so on. However, the driver will not autonegotiate at these speeds by default.

To negotiate at these speeds the driver must be configured to advertise them as supported rates.

This is managed using the sysctl Runtime Tunable OID dev.ix.<X>.advertise_speed where <X> is the interface number. For example, to adjust the advertised speeds of ix3, use dev.ix.3.advertise_speed. Use sysctl -x <oid> at the CLI to view the values in hexadecimal instead of decimal.

`ix(4)` Driver Speed Values¶
Speed	Hex	Decimal
100M	0x1	1
1G	0x2	2
10G	0x4	4
10M	0x8	8
2.5G	0x10	16
5G	0x20	32

The default value of the sysctl OID varies by hardware and SFP modules. Common defaults include 0x6 (hex) or 6 (decimal) to advertise 1G and 10G, and 0xb (hex) or 11 (decimal) to advertise 10M, 100M, and 1G.

To autonegotiate at the non-default speeds, add the value of the desired speeds to the existing value.

For example, to advertise speeds of 1G, 2.5G, 5G, and 10G, add their corresponding values: 0x2+0x10+0x20+0x4 (hex) or 2+16+32+4 (decimal) for a total of 0x36 (hex) or 54 (decimal). Values in hexadecimal must start with 0x.

When setting the value at the CLI, the command will print an error message if the value is not valid for the interface. The tunables GUI does not currently validate the content.

General Tuning¶

Add the following as Loader Tunables if the values are not already this large, or larger:

kern.ipc.nmbclusters="1000000"
kern.ipc.nmbjumbop="524288"

As a sysctl (Runtime Tunable) if the value are not already this large, or larger, or set to 0 (automatic):

hw.intr_storm_threshold="10000"

VMware vmx(4) Interfaces ¶

VMware VMXNET interfaces support multiple queues when using MSI-X. Multiple queues enable network performance to scale with the number of vCPUs and allows for parallel packet processing. Transmit and Receive descriptors may also be increased to help with throughput.

Add the following Loader Tunables:

Note

Some options have a separate set of tunables for each individual network interface. In these cases, replace <id> replace with the device ID such as 0, 1, etc. where the ID number matches the interface number. For example, tunables for vmx3 are under dev.vmx.3.

hw.pci.honor_msi_blacklist="0"
dev.vmx.<id>.iflib.override_ntxds="0,4096"
dev.vmx.<id>.iflib.override_nrxds="0,2048,0"

Save the file, then reboot and check the change with dmesg | grep -Eiw 'descriptors|queues' at a command prompt.

Flow Control ¶

In some circumstances, flow control may need to be disabled. The exact method depends on the hardware involved, as in the following examples:

These example entries are Loader Tunables:

cxgbe(4):

hw.cxgbe.pause_settings="0"

ixgbe(4) (aka ix):

hw.ix.flow_control="0"

These example entries go in System > Advanced, on the System Tunables tab (System Tunables):

Note

Some options have a separate set of tunables for each individual network interface. In these cases, replace <id> replace with the device ID such as 0, 1, etc. where the ID number matches the interface number. For example, tunables for igc3 are under dev.igc.3.

igc(4):

dev.igc.<id>.fc="0"

igb(4):

dev.igb.<id>.fc="0"

em(4):

dev.em.<id>.fc="0"

For ix and others, the flow control value can be further tuned:

0:

No Flow Control

1:

Receive Pause

2:

Transmit Pause

3:

Full Flow Control, Default