Memory Management¶

FreeBSD manages memory by dividing it into multiple areas or “pools” and moving memory pages between these areas as needed when allocating memory and taking other actions. These areas are visible in places such as the memory graphs (Memory Graph), system utilities such as top (System Activity (Top)), in sysctl OIDs, and elsewhere.

Memory Pools¶

The different memory pools in FreeBSD currently are:

Active: Active (in use) memory pages referenced by userland (non-kernel). If this memory is unreferenced, after a while it will eventually be marked inactive.
Inactive: Memory pages which were in use but have not been referenced recently are considered Inactive. Inactive pages can return to active status if they are referenced again. If the pages are “clean” (not in need of swapping or writing to disk), they may be freed directly. Otherwise they are considered “dirty” and may be moved to the Laundry queue.
Free: Memory available for immediate use.
Cache: Memory used by the operating system for caching. On systems using ZFS, this is the ZFS ARC cache (23.05+). On UFS systems, it is the UFS directory hash.

Note

On current versions this is only present in the memory graphs, and not reported in other places such as top.

The OS will attempt to use RAM when possible for caching rather than allowing it to sit idle and free, so the percentage of free RAM will often appear lower than expected.

Typically cache memory, such as the ZFS ARC, will be released by the OS when there is a shortage of free memory for other processes. However this may not happen fast enough if a process attempts to allocate a large chunk of memory all at once. In environments where that is a frequent concern, a good practice is to limit the ZFS ARC size as described in ZFS Tuning.
Wire: Memory allocated by the kernel, including the kernel itself, which cannot be paged/swapped and cannot be freed until explicitly released.

Note

In the OS and system utilities such as top, the ZFS ARC cache and UFS buffers sizes are included in wired memory. The top utility also prints a separate line breaking down ZFS ARC usage.

In the graphs on pfSense Plus software version 23.05 and later, however, these values are removed from the Wired total and graphed separately. ZFS ARC usage is graphed under Cache and UFS buffers are graphed under Buffers.
UserWire: Similar to Wired, but memory wired by user processes, not the kernel.
Laundry: Memory pages which are considered “dirty” and are due to be “cleaned”. This means they may be eligible for swap (still in use but not recently referenced) or written to disk (e.g. file cache). After cleaning, the pages may be returned to active memory if referenced recently, or to an inactive state otherwise.
Buffers: Memory used for UFS buffers.

Memory Usage Notes¶

Free RAM is Wasted RAM¶

A common mantra among FreeBSD developers is “Free RAM is Wasted RAM”. While free RAM is available for use by new/large processes, letting that RAM sit idle and unused is wasting the potential of the system. Allowing the OS to use that free RAM for caching will result in overall higher performance with minimal, if any, consequences.

If a system appears to have low amounts of free memory but is not experiencing any memory allocation failures, the best practice is usually to leave it alone and let the OS use what it can.

On the other hand, if there are problems running scheduled tasks or certain daemons because of low memory, then it may be best to reduce the amount of RAM available for caching. On ZFS systems, see ZFS Tuning for advice.

ZFS Disk Activity Increases Memory Usage¶

On systems using ZFS, large volumes of filesystem activity can lead to an increase in ZFS ARC usage which consumes memory temporarily to boost filesystem performance.

The most common scenario where this becomes a factor is during the first boot after a upgrading to a newer version of pfSense software. During an upgrade, large portions of the operating system are rewritten on disk, which can lead to a large chunk of wired memory being allocated to the ZFS ARC to speed up the process.

A similar effect can be observed from other tasks which scan or check large portions of the disk, such as from packages or certain periodic jobs.

In many cases this is harmless because the operating system will yield memory as needed if other processes require more memory. However, that does not always happen quickly. See ZFS Tuning for advice on limiting the memory available for the ZFS ARC and tuning the thresholds at which it will yield memory.

Rebooting the firewall after one of these events will return the reported memory usage to a normal level.

Not All Swap Usage is Bad¶

Installations containing disk partitions dedicated to swap can expand the amount of memory usable by the operating system. Using swap allows idle or unused areas of allocated memory to be relocated to disk instead of RAM. If those areas are requested again, they can be read back into RAM on demand. This is useful for ensuring processes do not run out of memory, but since it involves using disks, it is significantly slower than RAM.

Given that knowledge, one might assume that any usage of swap is bad and should be avoided at all costs, however, that is not universally true.

For example, there are daemons which run at all times but only periodically require the resources they have allocated. While the process is idle, there is little harm in allowing it to be swapped in favor of something that needs RAM at that moment, or even for caching.

Swap consumption generally only becomes a concern when there is significant usage, such as near 50% or higher. At that point it becomes necessary to inspect everything consuming memory to see what can be reduced or tuned.

Additionally, allowing the operating system to have swap space enables the ability to save kernel panic text dumps which facilitate debugging kernel problems. Without swap space to use as a dump device, the operating system will print the crash dump to the console and cannot store it anywhere.