ZFS Tuning

ZFS on pfSense® software is much more robust than UFS and enables a variety of handy ZFS features, such as ZFS Boot Environments. Some ZFS features are not relevant on systems acting as a firewall, however. For example, ZFS can be tuned to use much less memory and sacrifice performance because filesystem performance is not as critical in a firewall role as compared to others.

Note

This is not a general purpose ZFS tuning guide. The suggestions in this document primarily apply to ZFS on a device acting in a firewall role. The tuning requirements for other use cases, such as for file or database servers, can be vastly different.

ZFS Memory Tuning

Installations utilizing ZFS may encounter higher than expected memory usage. This memory may appear as “Wired”, “Cache”, or “ARC” depending on how the memory usage is being checked (Memory Management). In each case the usage is the same, but may be reported differently. This memory is considered “Wired” because it cannot be paged out (e.g. swap) so it has the appearance of consuming large portions of memory that other things may need..

ZFS attempts to optimize performance with its Adaptive Replacement Cache (ARC) which can be aggressive in its consumption of RAM. The classic phrase “Free RAM is wasted RAM” also applies here, and ZFS will make good use of the memory when allowed to do so. ZFS will yield this RAM if other processes require more memory, but it may not give up memory fast enough for every use case. For example, if a process starts and attempts to allocate a large block of memory rapidly, and there is not sufficient free RAM or swap, then it may fail.

Tip

It’s normal and OK to see a little swap usage when ARC usage is high, as items swapped are generally idle and not actively used, whereas the RAM can be put to better use caching until such time as those pages are needed.

If the swap usage is high or nearly full, however, that is likely a sign that there is too much contention for memory and if the ARC usage is also high, it likely needs tuned to a much lower maximum.

ARC Maximum/Minimum

The ARC size has default limits which allow it to consume large portions of the system RAM:

Default Maximum:

The default maximum ARC size (vfs.zfs.arc.max) is automatic (0) and uses 1/2 RAM or the total RAM minus 1GB, whichever is greater.

As the default processes on pfSense software can easily consume 1GB of RAM given the opportunity, this situation can become problematic in some cases!

Default Minimum:

The default minimum ARC size (vfs.zfs.arc.min) is automatic (0) and is 1/32 of all RAM or 64M, whichever is greater.

To tune the maximum, configure a system tunable (System Tunables) for vfs.zfs.arc.max and set it to a size expressed in bytes.

Tip

For X MB RAM, use X * 1024 * 1024. For example, 1GB is 1073741824 bytes, 512M is 536870912, 256M is 268435456, 128M is 134217728, and 64M is 67108864.

The exact amount depends on the available resources, but to be conservative on systems with low RAM, set it to 25% of the total RAM or less.

The minimum rarely would need tuning, but can be lowered if necessary.

Note

The ZFS ARC limit will take effect immediately, but the OS may not free the wired memory previously used by the ARC right away. Rebooting after configuring the limits will ensure they are fully respected.

Prefetch

ZFS prefetch attempts to read more blocks than initially requested into the ARC in case they are needed in the near future. This can also cause increased ARC usage.

To disable prefetch, configure a system tunable (System Tunables) and set vfs.zfs.prefetch.disable to 1

Free Target

The ARC Free Target is the amount of free RAM under which the ARC will start to give up memory. This can be awkward to calculate since it is dealing with a remaining amount, not a total, and the fact that this must be given in pages (chunks of 4096 bytes).

For example, to trigger ARC cleanup when the system hits 60% RAM usage, this number should be 40% of total RAM bytes divided by 4096.

On a system with 4GB RAM, this would be when the system has 1.6GB free. To find the equivalent pages value, calculate it with (<percent> * <total bytes>)/<page size>.

Tip

The page size is almost always 4096, but check the value of sysctl hw.pagesize to be certain.

In this case that would be (0.40 * (1024 * 1024 * 1024 * 4))/4096 which results in 419430 pages.

Configure a system tunable (System Tunables) to set vfs.zfs.arc.free_target to the desired number of pages (e.g. 419430).