Troubleshooting Clock Issues

Time and clock issues are relatively common on hardware, but on firewalls they are critical, especially if the firewall is performing tasks involving validating certificates as part of a PKI infrastructure.

Proper time synchronization is an absolute necessity on systems which do not have a battery onboard to preserve their date and time settings when power is removed.

Not only will getting this all in line help with critical system tasks, but it also ensures that the log files on the firewall are properly timestamped, which aids with troubleshooting, record keeping, and general system management.

Time Keeping Problems

Hardware can have significant problems keeping time, though such problems are generally isolated to older, low-quality hardware. All PC clocks will drift to some extent, but some hardware can drift as much as one minute for every couple minutes that pass and rapidly get out of sync. NTP is designed to periodically update the system time to account for normal drift. It cannot reasonably correct clocks that drift significantly. This is very uncommon, but there are a few methods that can potentially work around these problems.

The best way to avoid time keeping problems is to use quality hardware that has been tested and does not experience these issues, such as official appliances found in the Netgate Store.

There are several items to check if hardware has significant time keeping problems.

Network Time Protocol

By default, pfSense software attempts to synchronize its time using the ntp.org Network Time Protocol (NTP) server pool. This ensures an accurate date and time on the firewall, and will accommodate normal clock drift. If the firewall date and time are incorrect, ensure NTP synchronization is functioning. The most common problem preventing synchronization is the lack of proper DNS configuration on the firewall. If the firewall cannot resolve hostnames, NTP synchronization will fail. The results of synchronization are shown at boot time in the system log, and the status of the NTP clock synchronization can be viewed at Status > NTP. The NTP Status widget for the Dashboard also offers basic information about the NTP server selected for use by the firewall.

BIOS Updates

Netgate has observed older hardware that ran fine for years on Windows encounter major timekeeping problems once redeployed on FreeBSD (and by consequence, pfSense software). The systems were running a BIOS version several revisions out of date. One of the revisions addressed a timekeeping issue that apparently never affected Windows. Applying the BIOS update fixed the problem. The first thing to check is to make sure the hardware in question has the latest available BIOS.

PNP OS settings in BIOS

Other hardware might have time keeping difficulties in FreeBSD and pfSense software unless PNP OS in the BIOS was set to No. If the BIOS does not have a PNP OS configuration option, look for an OS setting and set it to Other.

Disable ACPI

A few BIOS vendors have produced ACPI (Advanced Configuration and Power Interface) implementations which are buggy at best and dangerous at worst. On more than one occasion Netgate has encountered hardware that would not boot or run properly while ACPI support is active.

While ACPI support can be disabled in the BIOS, and in the OS, this is not a best practice and as such hardware that requires such changes should be replaced.

Adjust Timecounter Hardware Setting

On rare systems, the kern.timecounter.hardware sysctl value may need to be changed to correct an inaccurate clock. This is known to be an issue with older versions of VMware such as ESX 5.0 in combination with an amd64-based pfSense software or FreeBSD image. That special case was a bug in the hypervisor that is fixed in ESX 5.1 and later.

On these systems the default timecounter will eventually stop the clock from ticking, causing problems with encryption, VPNs, and services in general. On other systems, the clock may skew by large amounts with the wrong timecounter.

To change the timecounter, browse to System > Advanced, on the System Tunables tab and add an entry to set kern.timecounter.hardware to i8254

This will make the system use the i8254 timecounter chip, which typically keeps good time but may not be as fast as other methods. The other timecounter choices will be explained later in this section.

If the system keeps time properly after making this change, leave the tunable entry in place to make this change permanent. If the change did not help, remove the tunable or try another value.

Depending on the platform and hardware, there may also be other timecounters to try. For a list of available timecounters found on a firewall, execute the following command:

# sysctl kern.timecounter.choice

The firewall will print a list of available timecounters and their “quality” as reported by FreeBSD:

kern.timecounter.choice: TSC-low(1000) ACPI-safe(850) i8254(0) dummy(-1000000)

Try any of those four values for the sysctl kern.timecounter.hardware setting. In terms of “quality” in this listing, the larger the number the better, but the actual usability varies from system to system.

TSC:

A counter on the CPU, but is tied to the clock rate and is not readable by other CPUs. It can be used in bare metal SMP systems but it requires that TSCs on all CPUs be synchronized. It cannot be used reliably on systems with variable-speed CPUs or virtualized system with multiple CPUs.

i8254:

A clock chip found in most hardware, which tends to be safe but can have performance drawbacks.

ACPI-safe:

If it is properly supported by the hardware, this is a good choice because it does not suffer from the performance limitations of i8254, but in practice its accuracy and speed vary widely depending on the implementation.

ACPI-fast:

A faster implementation of the ACPI timecounter available on hardware that does not suffer from known ACPI issues.

HPET:

High Precision Event Timer available in some hardware. When available, it is generally considered a good source of accurate time counting.

This and more information on FreeBSD Timecounters can be found in the paper Timecounters: Efficient and precise timekeeping in SMP kernels by Poul-Henning Kamp of the FreeBSD Project, and in the FreeBSD source code.

Adjust the Kernel Timer Frequency

In rare cases adjusting the kernel timer frequency, or kern.hz kernel tunable, can help performance or stability. This is especially true on virtualized environments. The default is 1000, but in some cases 100, 50, or even 10 will be a better value depending on the system. When pfSense software is installed in VMware, it detects it and automatically sets this tunable to 100, which should work fine in nearly all cases with VMware products.

To adjust this setting, add the following line, with the new value, as a Loader Tunable:

kern.hz=100

GPS Time Synchronization

To aid in maintaining an accurate clock, GPS time synchronization is also provided by pfSense software on supported hardware. Certain serial or USB GPS devices are supported, and in combination with external time servers, they can help keep the clock accurate. For more detail, see NTPD.