Troubleshooting Upgrades

This document describes methods of troubleshooting problems firewalls may encounter when attempting to run a pfSense® software upgrade.

Check Disk Space and ZFS Boot Environments

Running out of space during an upgrade can lead to a variety of failure states that are difficult to fully document. The upgrade process attempts to estimate the amount of disk space it will need but the calculations are not always accurate enough to prevent problems during the upgrade process itself.

If there are any problems with an upgrade one of the best places to start is by cleaning up disk usage where possible. On systems running pfSense Plus software on ZFS, ZFS Boot Environments can consume significant disk space over time if they are not cleaned up. See Check and Clean Up ZFS Boot Environments for details. Additionally, if there are packages installed, package logs and data in particular may be consuming large amounts of disk space.

Low Memory Hardware and AWS/Azure Instances

Hardware with 1 GiB or less available memory may have issues upgrading depending on which features, services, or packages are running. This includes some Netgate hardware such as the Netgate 1100 when running with ZFS and/or certain services/packages. For the best chance of success in these cases, temporarily disable any non-critical services before starting the upgrade.

Tip

A Pre-Upgrade Reboot can also temporarily reduce memory used for ZFS caching, which can help in this situation as well.

pfSense Plus software can no longer run on AWS “.nano” size instances as they lack sufficient RAM to upgrade properly. Attempting to upgrade a “.nano” instance to pfSense Plus software version 24.03 will fail before the upgrade is performed. Migrate the instance to a “.micro” or larger size before attempting to upgrade, or redeploy instead.

Similar to the above, pfSense Plus software can no longer run on Azure A0 instances. Migrate to instances with more memory.

Remove / Dismount any Installation Media

Some users leave an installation disk plugged in or mounted for various reasons, but this can interfere with the upgrade process. Be sure to dismount and remove any installation media such as a USB thumb drive, optical disk, or ISO in a virtual drive.

A particularly problematic case is when a non-UEFI system has UEFI boot media inserted, the upgrade process may attempt to upgrade the boot loader on the installer and fail:

Number of packages to be reinstalled: 1
[1/1] Reinstalling pfSense-boot-23.09...
[1/1] Extracting pfSense-boot-23.09: .......... done
mount_msdosfs: /dev/msdosfs/EFISYS: Read-only file system
pkg-static: POST-INSTALL script failed
 failed.
__RC=1 __REBOOT_AFTER=10

Cosmetic Problems Post-Upgrade

If cosmetic problems occur after performing an upgrade, this is nearly always due to stale browser cache entries for CSS, JavaScript, or other files where the browser does not pull the updated version. Force a refresh of the page in the browser (e.g. Shift+Reload or Ctrl+F5) or clear the browser cache to resolve the issues.

Upgrade Log

pfSense-upgrade keeps a log of the last update attempt, which may contain additional useful information. This log is located at /conf/upgrade_log.latest.txt. Please include the contents of this log with any post or support request when requesting assistance with upgrade problems.

Upgrade not Offered / Library Errors

If the update system does not offer an upgrade to the most recent version, the upgrade will not proceed, or the upgrade process encounters errors with shared libraries, take the following steps:

  • Navigate to System > Updates

  • Set Branch to Previous stable version

  • Wait a few moments for the upgrade check to complete

  • Optional: Confirm that the latest version of pfSense-upgrade is present using pkg-static info -x pfSense-upgrade.

    If the correct version is not present, wait a bit longer and check again as that package may be updating in the background.

  • Set Branch to Latest stable version

  • Wait a few moments for the upgrade check to complete

If the upgrade is still not offered, refresh the repository configuration and upgrade script by running the following commands from the console or shell:

# pkg-static clean -ay; pkg-static install -fy pkg pfSense-repo pfSense-upgrade

If that procedure results in an error, or the upgrade is still not offered, then attempt to update to an intermediate version. For example, to get from Plus 2.4.5-p1 to Plus 21.05 or later, upgrade to Plus 21.02.x before proceeding to later versions if a direct upgrade does not succeed. In these cases, the appropriate version will be visible as a branch to select.

Repository Metadata Version Errors

If pkg is unable to update and complains about the repository metadata version, the pkg utility may need to be updated manually to version 1.13.x or later.

Example metadata version error:

>>> Updating repositories metadata...
Updating pfSense-core repository catalogue...
pkg-static: repository meta /var/db/pkg/pfSense-core.meta has wrong version 2
pkg-static: Repository pfSense-core load error: meta cannot be loaded No error: 0

To correct the problem, manually bootstrap pkg from an ssh or console shell:

# pkg-static bootstrap -f

Warning

Do not run the above command from Diagnostics > Command as it requires interactive input. If ssh and console shells are unavailable, use this variation instead:

env ASSUME_ALWAYS_YES=yes pkg-static bootstrap -f

Rewrite Repository Information

In some cases the repository information may need to be rewritten:

  • Navigate to System > Updates

  • Set the Branch to Latest Development Snapshots

  • Wait for the page to refresh

  • Set the Branch to Latest stable version

If the update still does not appear, run the commands above from the console or shell.

CLI Troubleshooting

If the GUI update is not functioning as expected, there are a handful of shell commands that can help gather information or resolve problems.

Force pkg Metadata Update

Often times DNS or connectivity problems will prevent the firewall from finding updates. A quick way to verify this is to force a pkg metadata update:

# pkg-static update -f

This command forces an update and will print errors if problems are found, a few potential errors include:

No address record:

The firewall cannot resolve the update server hostname. This could be a problem with DNS from the firewall itself, or connectivity from the firewall to the Internet in general, such as a missing or incorrect default route.

No route to host:

The firewall cannot reach the update server because it cannot find a route there. Most likely, the firewall is missing its default route or the WAN with the default route is down.

Operation timed out:

The firewall was unable to download a file in a timely manner. This is most likely due to degraded connectivity between the firewall and the update servers. It could also be a routing issue, or a problem with IPv6 on the firewall (See IPv6 Connectivity Problems).

An error occured while fetching package:

A general error that could have a few different causes. It may indicate that pkg does not trust the package servers. Try running certctl rehash from the console, a root shell prompt, or via Diagnostics > Command Prompt. This will allow pkg to utilize the system certificates until the next reboot.

Authentication error:

There is a proxy between the firewall and the update servers and it requires authentication. Move the firewall so it is not behind a proxy, or configure the proxy under System > Advanced, Miscellaneous tab.

No trusted public keys found:

The firewall is attempting to update from the wrong repository. Ensure the correct branch is selected as mentioned in Rewrite Repository Information. May require a reinstall to resolve. For CE installations, try the following command:

# fetch -qo /usr/local/share/pfSense/keys/pkg/trusted/ \
 https://raw.githubusercontent.com/pfsense/pfsense/RELENG_2_4_5/src/usr/local/share/pfSense/keys/pkg/trusted/pkg.pfsense.org.20160406

Debug pkg Metadata Update

If the previous command does not yield any meaningful information, try running the command in debug mode:

: pkg-static -d update

The output may yield additional useful messages, as in the following example:

DBG(1)[69233]> pkg initialized
Updating pfSense-core repository catalogue...
DBG(1)[69233]> PkgRepo: verifying update for pfSense-core
DBG(1)[69233]> PkgRepo: need forced update of pfSense-core
DBG(1)[69233]> Pkgrepo, begin update of '/var/db/pkg/repo-pfSense-core.sqlite'
DBG(1)[69233]> Request to fetch pkg+https://pkg.pfsense.org/pfSense_v2_7_1_amd64-core/meta.conf
DBG(1)[69233]> curl_open
DBG(1)[69233]> Fetch: fetcher used: pkg+https
DBG(1)[69233]> curl> fetching https://pkg.pfsense.org/pfSense_v2_7_1_amd64-core/meta.conf
DBG(1)[69233]> CURL> attempting to fetch from , left retry 3

* Couldn't find host pkg01-atx.netgate.com in the .netrc file; using defaults
*   Trying 208.123.73.209:443...
* Connected to pkg01-atx.netgate.com (208.123.73.209) port 443
* ALPN: curl offers http/1.1
*  CAfile: none
*  CApath: /etc/ssl/certs/
* SSL certificate problem: self-signed certificate in certificate chain
* Closing connection
DBG(1)[69233]> CURL> attempting to fetch from , left retry 2

That error suggests a problem with pkg-static trusting the package server, so in this case, try running certctl rehash from the console, a root shell prompt, or via Diagnostics > Command Prompt to allow pkg to utilize the system certificates until the next reboot.

Manual Update Check

To run a manual update check from the CLI:

# pfSense-upgrade -d -c

When run successfully, this command will print a line stating that a new version is available, and the version number of the available update. Errors displayed during that process are likely to be the same as those covered in Force pkg Metadata Update.

packages.netgate.com Has no A/AAAA Record

pkg does not use A/AAAA records. It uses service (SRV) records. The update server meta names such as packages.netgate.com are not meant to be accessed directly using a browser.

To find the actual update servers, lookup the SRV record for the host:

$ host -t srv _https._tcp.packages.netgate.com
_https._tcp.packages.netgate.com has SRV record 10 10 443 pkg01-atx.netgate.com.
_https._tcp.packages.netgate.com has SRV record 10 10 443 pkg00-atx.netgate.com.

$ host pkg01-atx.netgate.com
pkg01-atx.netgate.com has address 208.123.73.209
pkg01-atx.netgate.com has IPv6 address 2610:160:11:18::209

$ host pkg00-atx.netgate.com
pkg00-atx.netgate.com has address 208.123.73.207
pkg00-atx.netgate.com has IPv6 address 2610:160:11:18::207

Accessing the hosts using their direct hostnames will work with a browser:

$ curl --output /dev/null --silent --head --fail \
  "https://pkg00-atx.netgate.com/pfSense_v2_6_0_amd64-core/meta.txz"
$ echo $?
0

IPv6 Connectivity Problems

If IPv6 is configured on the firewall, the pfSense software will prefer to use it when performing an update. There are cases when a firewall may have broken IPv6 connectivity, however, that contribute to problems updating. This could manifest as a timeout or routing error when upgrading.

Typically the operating system will attempt to fall back to IPv4, but the extra time this takes could also lead to a timeout.

The firewall can be configured to prefer IPv4 to eliminate this as a potential cause. See Controlling IPv6 Preference for traffic from the firewall itself for details.

Alternately, from ssh or a console shell, force the upgrade to use IPv4 manually:

# pfSense-upgrade -4

Segmentation Fault in pkg

Certain cryptographic hardware can have a software-induced race condition which leads to a problematic state. In this state, pkg will crash with a segmentation fault:

1085486128:error:14099044:SSL routines:ssl3_send_client_verify:internal error:
Child process pid=30149 terminated abnormally: Segmentation fault

In this case, the device must be powered off and back on to recover. A warm reboot is not sufficient to reset the hardware.

  • Navigate to Diagnostics > Halt System

  • Click fa-stop-circle Halt

  • Wait for the device to shut down. Monitor the console to ensure that the shutdown completes.

  • Unplug the power adapter

  • Plug the power adapter back in

Forced pkg Reinstall

Forcing a reinstallation of all packages may resolve problems that otherwise may require a full reinstall. This is not ideal, as a clean install is more likely to have a positive result, but that is not always an option in every situation (e.g. remote install with no console access).

To forcefully reinstall all packages, take the following steps:

  • Make a backup

  • Clean the repository and forcefully reinstall pkg, repo data, and the upgrade script:

    # pkg-static clean -ay; pkg-static install -fy pkg pfSense-repo pfSense-upgrade

  • Force a reinstall of everything:

    # pkg-static upgrade -f

  • Review the list of changes and enter y to proceed

  • Manually reboot the firewall

Last Resort

If nothing else works then a reinstall will eliminate any possibility of problems related to the upgrade itself.

pfSense software supports multiple options to easily restore the configuration. The fastest method is Recover config.xml as discussed in Automatically Restore Configuration During Installation. Using that method, the pfSense software installation can pick up the existing configuration from the existing install and use it, eliminating the need for any manual restore process. The firewall will boot up after installation with the old settings and reinstall packages as needed.