Troubleshooting OS Issues with a Debug Kernel

Starting with pfSense® Plus software version 23.01 there is a new package containing an operating system kernel and module set built with additional debugging enabled. This is known as a “Debug Kernel”.

Note

In the past, there was a similar package which installed debug symbols but it did not include a full debug kernel or modules.

Installing the Debug Kernel

The debug kernel can be installed from the command line:

# pkg update
# pkg install -y pfSense-kernel-debug-pfSense

Warning

The debug kernel consumes a significant amount of disk space, ensure there is at least 1GB free.

Once installed, the debug kernel and associated module files are located in /boot/kernel.debug and debug symbols are under /usr/lib/debug/boot/kernel/.

Booting the Debug Kernel

After installing the debug kernel it can be activated in one of several ways:

Loader Menu

From the loader menu:

  • Select menu option 6 until it says kernel.debug

  • Select menu option 1 or press the Enter key to boot

This will boot the debug kernel one time and the next boot will revert back to the default kernel.

Loader Prompt

To reach a loader prompt, select menu option 3 from the loader menu or press the space bar during the kernel spinner on systems without a loader menu.

At the loader prompt, enter the following at the OK prompt:

boot kernel.debug

This will boot the debug kernel one time and the next boot will revert back to the default kernel.

Loader Configuration

To persistently boot from the debug kernel, create or edit /boot/loader.conf.local and add the following line:

kernel="kernel.debug"

Save and exit, then reboot the firewall as usual from the GUI or console menu.

The next boot and all subsequent boots will use the debug kernel.

Warning

Do not leave the debug kernel persistently active when making a firmware upgrade. The upgrade process relies on booting from the default non-debug kernel. The package may remain installed, but it should not be active in loader.conf.local during the upgrade process.

To temporarily switch back to the default non-debug kernel, use the loader menu or loader prompt as described in the previous sections, but boot kernel instead of kernel.debug.

To permanently switch back to the non-debug kernel, remove the kernel line from /boot/loader.conf.local.

Removing the Debug Kernel

To remove the debug kernel, first boot from the non-debug kernel. Then, check for and remove any reference to the debug kernel in /boot/loader.conf.local.

Finally, remove the debug kernel package from the command line:

pkg delete pfSense-kernel-debug-pfSense