Testing High Availability

With all of the configuration complete, the time has come for testing. Tests for each aspect of the system are listed below. If any of the tests fails, review the configuration and consult Troubleshooting High Availability for assistance.

Verify General Functionality

Setup a client on the LAN and ensure that it receives a DHCP IP address and that it shows the LAN CARP VIP as its gateway and DNS server. Verify that the client can reach the Internet and otherwise function as expected.

Verify XMLRPC Sync is working

XMLRPC Configuration Synchronization can be tested several ways. The easiest method is to make a change to any supported area on the primary, such as a firewall rule, and then see if the change is reflected on the secondary after a few moments.

The manual method for forcing a synchronization task to test XMLRPC is to visit Status > Filter Reload on the primary node and click Force Config Sync. The status will change briefly and then if everything is working properly, a message will be displayed indicating the sync completed successfully.

Verify CARP is working

Visit Status > CARP on both nodes to check if CARP is functional. The primary node will display “MASTER” for all CARP VIPs and the secondary will display “BACKUP” for all CARP VIPs. If the status screen indicates that CARP is disabled, press the Enable CARP button.

Verify State Synchronization is working

The Status > CARP page lists pfsync nodes which give an indication of the state synchronization status. The values may not always match identically on both nodes, but they will be close. If the two are very different, it can indicate a problem with state synchronization. If they are identical or nearly identical, then state synchronization is working.

Testing Failover

A manual failover test may be initiated in one of four ways:

  1. Click Temporarily Disable CARP on Status > CARP on the primary node. This will disable CARP temporarily, and if the primary node is rebooted it will turn back on. Click Enable CARP to turn it back on.

  2. Click Enter Persistent CARP Maintenance Mode on Status > CARP on the primary node. This will disable CARP persistently, even if the primary node is rebooted. To exit maintenance mode, click Leave Persistent CARP Maintenance Mode to enable CARP once again.

  3. Unplug a network cable from an interface with a CARP VIP present, such as WAN or LAN. This will trigger a failover event. Plug the cable back in to recover.

  4. Shut down or reboot the primary node.

During any of the above tests, visit Status > CARP on the secondary to confirm that the CARP VIPs have taken over and show a “MASTER” status.

Before, during, and after triggering a failover, test connections from a client on the LAN through to the Internet to ensure connectivity works at each step. Downloading a file, streaming audio, or streaming video will most likely continue uninterrupted. VoIP-based phone calls may have a slight disruption as they are not buffered like the others.

Also have a client attempt to obtain an IP address by DHCP while running from the secondary.

If VPNs or other services have been configured, check those during the test as well to ensure the VPN established on the secondary node and continues to pass traffic.

Once the primary node has returned to “MASTER” status, ensure everything continues to work.