ZFS on pfSense® software is much more robust than UFS and enables a variety of handy ZFS features, such as ZFS Boot Environments. Some ZFS features are not relevant on systems acting as a firewall, however. For example, ZFS can be tuned to use much less memory and sacrifice performance because filesystem performance is not as critical in a firewall role as compared to others.
This is not a general purpose ZFS tuning guide. The suggestions in this document primarily apply to ZFS on a device acting in a firewall role. The tuning requirements for other use cases, such as for file or database servers, can be vastly different.
ZFS Memory Tuning¶
Installations utilizing ZFS may encounter higher than expected memory usage. This memory may appear as “Wired”, “Cache”, or “ARC” depending on how the memory usage is being checked. In each case the usage is the same, but may be reported differently. This memory is considered “Wired” because it cannot be paged out (e.g. swap) so it has the appearance of consuming large portions of memory that other things may need..
ZFS attempts to optimize performance with its Adaptive Replacement Cache (ARC) which can be aggressive in its consumption of RAM. The classic phrase “Free RAM is wasted RAM” also applies here, and ZFS will make good use of the memory when allowed to do so. ZFS will yield this RAM if other processes require more memory, but it may not give up memory fast enough for every use case. For example, if a process starts and attempts to allocate a large block of memory rapidly, and there is not sufficient free RAM or swap, then it may fail.
It’s normal and OK to see a little swap usage when ARC usage is high, as items swapped are generally idle and not actively used, whereas the RAM can be put to better use caching until such time as those pages are needed.
If the swap usage is high or nearly full, however, that is likely a sign that there is too much contention for memory and if the ARC usage is also high, it likely needs tuned to a much lower maximum.
The ARC size has default limits which allow it to consume large portions of the system RAM:
- Default Maximum
The default maximum ARC size (
vfs.zfs.arc.max) is automatic (
0) and uses 1/2 RAM or the total RAM minus 1GB, whichever is greater.
As the default processes on pfSense software can easily consume 1GB of RAM given the opportunity, this situation can become problematic in some cases!
- Default Minimum
The default minimum ARC size (
vfs.zfs.arc.min) is automatic (
0) and is 1/32 of all RAM or 64M, whichever is greater.
To tune the maximum, configure a system tunable
(System Tunables Tab) for
vfs.zfs.arc.max and set it to a size
expressed in bytes.
X MB RAM, use
X * 1024 * 1024. For example, 1GB is
bytes, 512M is
536870912, 256M is
268435456, 128M is
and 64M is
The exact amount depends on the available resources, but to be conservative on systems with low RAM, set it to 25% of the total RAM or less.
The minimum rarely would need tuning, but can be lowered if necessary.
The ZFS ARC limit will take effect immediately, but the OS may not free the wired memory previously used by the ARC right away. Rebooting after configuring the limits will ensure they are fully respected.
ZFS prefetch attempts to read more blocks than initially requested into the ARC in case they are needed in the near future. This can also cause increased ARC usage.
To disable prefetch, configure a system tunable
(System Tunables Tab) and set
The ARC Free Target is the amount of free RAM under which the ARC will start to give up memory. This can be awkward to calculate since it is dealing with a remaining amount, not a total, and the fact that this must be given in pages (chunks of 4096 bytes).
For example, to trigger ARC cleanup when the system hits 60% RAM usage, this number should be 40% of total RAM bytes divided by 4096.
On a system with 4GB RAM, this would be when the system has 1.6GB free. To find
the equivalent pages value, calculate it with
(<percent> * <total bytes>)/<page size>.
The page size is almost always
4096, but check the value of
hw.pagesize to be certain.
In this case that would be
(0.40 * (1024 * 1024 * 1024 * 4))/4096 which
Configure a system tunable (System Tunables Tab) to set
vfs.zfs.arc.free_target to the desired number of pages (e.g.