Testing High Availability

Since the goal of HA is high availability, thorough testing before placing a cluster into production is a must. The most important part of that testing is making sure that the HA peers will failover gracefully during outages.

If any actions in this section do not work as expected, review the configuration and consult Troubleshooting High Availability for assistance.

Verify General Functionality

Setup a client on the LAN and ensure that it receives a DHCP IP address and that it shows the LAN CARP VIP as its gateway and DNS server. Verify that the client can reach the Internet and otherwise function as expected.

Verify XMLRPC Sync is working

XMLRPC Configuration Synchronization can be tested several ways. The easiest method is to make a change to any supported area on the primary, such as a firewall rule, and then see if the change is reflected on the secondary after a few moments.

The manual method for forcing a synchronization task to test XMLRPC is to visit Status > Filter Reload on the primary node and click Force Config Sync. The status will change briefly and then if everything is working properly, a message will be displayed indicating the sync completed successfully.

Verify CARP is working

Visit Status > CARP on both nodes to check if CARP is functional. The primary node will display “MASTER” for all CARP VIPs and the secondary will display “BACKUP” for all CARP VIPs. If the status screen indicates that CARP is disabled, press the Enable CARP button.

Verify State Synchronization is working

The Status > CARP page includes State Synchronization Status which lists Filter Host ID values for entries in the state table. If the Filter Host ID in the High Availability settings has been changed recently, it may show both old and new values from the primary and secondary nodes. Over time the list should only reflect the current values of the Filter Host ID of each node in the cluster.

If the lists are identical or nearly identical, then state synchronization is working. If the list does not contain an entry for the Filter Host ID of the other node, then states are not being synchronized.

Testing Failover

A manual failover test may be initiated in one of four ways:

  1. Click Temporarily Disable CARP on Status > CARP on the primary node. This will disable CARP temporarily, and if the primary node is rebooted it will turn back on. Click Enable CARP to turn it back on.

  2. Click Enter Persistent CARP Maintenance Mode on Status > CARP on the primary node. This will disable CARP persistently, even if the primary node is rebooted. To exit maintenance mode, click Leave Persistent CARP Maintenance Mode to enable CARP once again.

    Note

    Wait a few moments and refresh the page after entering or leaving maintenance mode. The page often reloads faster than CARP election can take place, so the immediate view of the status is likely incorrect.

  3. Unplug a network cable from an interface with a CARP VIP present, such as WAN or LAN. This will trigger a failover event. Plug the cable back in to recover.

  4. Shut down or reboot the primary node.

During any of the above tests, visit Status > CARP on the secondary to confirm that the CARP VIPs have taken over and show a “MASTER” status.

Before, during, and after triggering a failover, test connections from a client on the LAN through to the Internet to ensure connectivity works at each step. Downloading a file, streaming audio, or streaming video will most likely continue uninterrupted. VoIP-based phone calls may have a slight disruption as they are not buffered like the others.

Also have a client attempt to obtain an IP address by DHCP while running from the secondary.

If VPNs or other services have been configured, check those during the test as well to ensure the VPN established on the secondary node and continues to pass traffic.

Once the primary node has returned to “MASTER” status, ensure everything continues to work.