Configuring a HA Cluster

Note

The WAN and LAN should be configured to static addresses prior to configuring a HA Cluster. Please see High Availability Prerequisites for IP address details.

This is the heart of the process, making the changes that will link the systems and allow them to function together.

Setup Sync Interface

Before proceeding, the Sync interfaces on the cluster nodes must be configured. Sync IP Address Assignments lists the addresses to use for the Sync interfaces on each node.

  1. Navigate to Interfaces and choose the interface to use on the SYNC port

  2. Check Enable Interface

  3. Enter SYNC for the Description

  4. Set IPv4 Configuration Type to Static IPv4

  5. Set IPv4 address to 172.16.1.2 when configuring the primary node, or 172.16.1.3 when configuring the secondary node

  6. Select 24 for the subnet mask in the CIDR drop-down next to IPv4 address

  7. Do not check Block private networks or Block bogon networks

  8. Click Save

  9. Click Apply Changes

Once that procedure has been completed on the primary node, perform it again on the secondary node with the appropriate IPv4 address value. Remember they must be the same on both nodes.

After configuring the sync interface, the interface assignments should have one labeled SYNC.

Add Firewall Rules for Synchronization

To complete the Sync interface configuration, firewall rules must be added to both nodes to allow synchronization.

At a minimum, the firewall rules must pass the configuration synchronization traffic (by default, HTTPS on port 443) and pfsync traffic. In most cases, a simple “allow all” style rule is used. For this guide, both will be shown and it will also serve as an indicator that synchronization is working.

On the primary node:

Set up a rule to allow configuration synchronization:

  1. Navigate to Firewall > Rules on the SYNC tab

  2. Click button_add_top at the top of the list to create a new rule

  3. Set Action to Pass

  4. Set Source to SYNC Net

  5. Set Destination to SYNC Address

  6. Set Destination port range to 443 or choose HTTPS (443) from the drop-down selector

  7. Set Description to Allow configuration synchronization

  8. Click Save

Set up a rule to allow state synchronization:

  1. Click button_add_end at the top of the list to create another new rule

  2. Set Action to Pass

  3. Set Protocol to pfsync

  4. Set Source to SYNC Net

  5. Set Destination to any

  6. Set Description to Allow state synchronization

  7. Click Save

Set up a rule to allow ICMP echo (ping) for Diagnostics:

  1. Click button_add_end at the top of the list to create another new rule

  2. Set Action to Pass

  3. Set Protocol to ICMP

  4. Set Source to SYNC Net

  5. Set Destination to SYNC Net

  6. Set Description to Allow ICMP echo (ping) for Diagnostics

  7. Click Save

  8. Click Apply Changes

When complete, the rules will look like the following, which also includes a rule to allow ICMP echo (ping) for diagnostic purposes.

../../_images/ha-sync-rules.png

Example Sync Interface Firewall Rules

On the secondary node:

  1. Navigate to Firewall > Rules on the SYNC tab

  2. Click button_add_top at the top of the list to create a new rule

  3. Set Action to Pass

  4. Set Protocol to any

  5. Set Source to SYNC Net

  6. Set Destination to any

  7. Set Description to Temp rule for sync

  8. Click Save

  9. Click Apply Changes

Note

The rule on the secondary is different, but that is intended at this point. Once the first configuration synchronization has taken place, the temporary rule on the secondary will be replaced by the rules from the primary. Seeing that the rules on the Sync interface changed is a good indicator that it worked!

Configure pfsync

State synchronization using pfsync must be configured on both the primary and secondary nodes to function.

First on the primary node and then on the secondary, perform the following:

  1. Navigate to System > High Avail. Sync

  2. Check Synchronize States

  3. Set Synchronize Interface to SYNC

  4. Set pfsync Synchronize Peer IP to the other node. Set this to 172.16.1.3 when configuring the primary node, or 172.16.1.2 when configuring the secondary node

  5. Click Save

Configure XMLRPC

Warning

Configuration synchronization must only be configured on the primary node. Never activate options in this section on the secondary node of a two-member cluster.

On the primary node only, perform the following:

  1. Navigate to System > High Avail. Sync

  2. Set Synchronize Config to IP to the secondary node’s Sync interface IP address, 172.16.1.3

  3. Set Remote System Username to admin.

    Note

    This can work with accounts other than admin, but doing so requires special account privileges and configuration steps.

  4. Set Remote System Password to the admin user account password and be sure to confirm the password.

  5. Check the boxes for each area to synchronize to the secondary node. For this guide, as with most configurations, all boxes are checked.

  6. Click Save

As a quick confirmation that the synchronization worked, on the secondary node navigate to Firewall > Rules on the SYNC tab. The rules entered on the primary are now there, and the temporary rule is gone.

The two nodes are now linked for configuration synchronization! Changes made to the primary node in supported areas will be synchronized to the secondary whenever a change is made.

Warning

Do not make changes to the secondary in areas set to be synchronized! These changes will be overwritten the next time the primary node performs a synchronization.

Add CARP VIPs

Now that the configuration synchronization is complete, the CARP Virtual IP addresses need only be added to the primary node and they will be automatically copied to the secondary. For this demonstration, two CARP VIPs will be added: One for WAN, and one for LAN.

  1. Navigate to Firewall > Virtual IPs on the primary node.

  2. Click button_add_top at the top of the list to create a new VIP

  3. Set Type to CARP

  4. Set Interface to WAN

  5. Enter the WAN CARP VIP into the IP Address(es) section Address box and pick the appropriate subnet mask. For this example, enter 198.51.100.200 and 24 (See WAN IP Address Assignments).

  6. Enter a random password in Virtual IP Password. This need only match between the two nodes, which will be handled by synchronization.

  7. Select an unused VHID Group as determined in Determine CARP VHID Availability.

    Note

    A common tactic is to make the VHID match the last octet of the IP address, so in this case 200 would be chosen.

  8. Set the Advertising Frequency to a Base of 1 and a Skew of 0. This value will be automatically adjusted when it is copied to the secondary.

  9. Enter a Description such as WAN CARP VIP.

  10. Click Save

  11. Click Apply Changes

The Base and Skew together determine how often a CARP heartbeat is sent. The value of Base adds whole seconds and should match between the two nodes. The Skew value adds 1/256th of a second increments. The primary node should always have a Skew of 0 or 1. The secondary node must be higher, typically 100+. That adjustment is handled automatically by the configuration synchronization process.

Note

If CARP appears to be too sensitive to latency on a given network, adjusting the Base by adding one second at a time is recommended until stability is achieved.

Repeat the above process for the LAN CARP VIP:

  1. Navigate to Firewall > Virtual IPs

  2. Click button_add_top at the top of the list to create a new VIP

  3. Set Type to CARP

  4. Set Interface to LAN

  5. Enter the LAN CARP VIP into the IP Address(es) section Address box and pick the appropriate subnet mask. For this example, enter 192.168.1.1 and 24 (See LAN IP Address Assignments).

  6. Enter a random password in Virtual IP Password.

  7. Select an unused VHID Group as determined in Determine CARP VHID Availability.

  8. Set the Advertising Frequency to a Base of 1 and a Skew of 0.

  9. Enter a Description such as LAN CARP VIP.

  10. Click Save

  11. Click Apply Changes

If there are any additional IP addresses in the WAN subnet that will be used for purposes such as 1:1 NAT, port forwards, VPNs, etc, they may be added now as well.

Check Firewall > Virtual IPs on the secondary node to ensure that the VIPs synchronized as expected.

The Virtual IP addresses on both nodes will look like the following if the process was successful.

../../_images/ha-carp-vips.png

CARP Virtual IP Address List

Check CARP Status

Now visit Status > CARP on both nodes to confirm the proper status. The primary node should indicate MASTER status for all VIPs, and the secondary node should indicate BACKUP status for all VIPs. If the status is incorrect, see Troubleshooting High Availability.

../../_images/ha-status-pri.png

CARP VIP Status on Primary

../../_images/ha-status-sec.png

CARP VIP Status on Secondary

Setup Manual Outbound NAT

Now it is time to put the new CARP VIPs to use. The NAT settings will synchronize so these changes need only be made to the primary node.

  1. Navigate to Firewall > NAT, Outbound tab on the primary node

  2. Change the Mode to Manual Outbound NAT rule generation

  3. Click Save, the rule list will be populated with rules equivalent to what was in use for the default, Automatic Outbound NAT.

    Note

    If no rules appear in the list, ensure the WAN has a gateway selected under Interfaces > WAN

  4. Click fa-pencil to edit the rule for the LAN subnet

  5. Set Translation to the WAN CARP VIP, 198.51.100.200 in this example.

  6. Click Save

  7. Repeat that edit for each rule in the list except the rules with a source of 127.0.0.0/8.

  8. Click Apply Changes

  9. Visit Firewall > NAT, Outbound tab on the secondary node to ensure the rule changes are reflected there.

../../_images/ha-out-nat.png

Outbound NAT Rules for LAN with CARP VIP

Warning

If additional local interfaces are added later, such as a second LAN, DMZ, etc, and that interface uses private IP addresses, then additional manual outbound NAT rules must be added at that time.

Other NAT Concerns

If there are any port forwards to be added using the WAN CARP VIP, they may be added now using Firewall > NAT, Port Forward tab. Port forwards will work the same as usual, but the Destination will be the WAN CARP VIP.

Setup DHCP

The DHCP server daemons on the cluster nodes need adjustments so that they can work together. The changes will synchronize from the primary to the secondary, so as with the VIPs and Outbound NAT, these changes need only be made on the primary node.

  1. Navigate to Services > DHCP Server, LAN* tab.

  2. Set the DNS Server to the LAN CARP VIP, here 192.168.1.1

  3. Set the Gateway to the LAN CARP VIP, here 192.168.1.1

  4. Set the Failover Peer IP to the actual LAN IP address of the secondary node, here 192.168.1.3

  5. Click Save

Setting the DNS Server and Gateway to a CARP VIP ensures that the local clients are talking to the failover address and not directly to either node. This way if the primary fails, the local clients will continue talking to the secondary node.

The Failover Peer IP allows the daemon to communicate with the peer directly in this subnet to exchange data such as lease information. When the settings synchronize to the secondary, this value is adjusted automatically so the secondary points back to the primary.

On both nodes, visit Status > DHCP Leases to see the status. A section will be displayed at the top containing the failover pool status, one line will be shown for each local interface pool. When the two nodes are working properly, both will indicate a “normal” status.

../../_images/ha-dhcp-status.png

DHCP Failover Status

VPNs and Other Services

When configuring a VPN, such as OpenVPN or IPsec, pick a WAN CARP VIP as the Interface for the VPN and ensure the remote peer also builds the other side of the tunnel using the CARP VIP as the peer address.

For other local services, packages, etc. likewise a CARP VIP is recommended for binding if the service will work on both nodes.

High Availability support in packages varies. Check the package documentation for information on if, or how, various aspects of High Availability work with a specific package.

Additional Interfaces

Additional local interfaces may also be configured, repeating some of the previous steps as needed:

  1. Assign the interface on both nodes identically

  2. Enable the interface on both nodes, using different IP addresses within the same subnet

  3. Add a CARP VIP inside the new subnet (Primary node only)

  4. Add firewall rules (Primary node only)

  5. Add Manual Outbound NAT for a source of the new subnet, utilizing the CARP VIP for translation (Primary node only)

  6. Configure the DHCP server for the new subnet, utilizing the CARP VIP for DNS and Gateway roles (Optional, Primary node only)