Configuring a High Availability Cluster

Note

The WAN and LAN should be configured to static addresses prior to configuring a HA Cluster. Please see High Availability Prerequisites for IP address details.

This is the heart of the process, making the changes that will link the systems and allow them to function together.

Setup Sync Interface

Before proceeding, the Sync interfaces on the cluster nodes must be configured. Sync IP Address Assignments lists the addresses to use for the Sync interfaces on each node.

1. Navigate to Interfaces and choose the interface to use on the SYNC port

2. Check Enable Interface

3. Enter SYNC for the Description

4. Set IPv4 Configuration Type to Static IPv4

5. Set IPv4 address to 172.16.1.2 when configuring the primary node, or 172.16.1.3 when configuring the secondary node

6. Select 24 for the subnet mask in the CIDR drop-down next to IPv4 address

7. Do not check Block private networks or Block bogon networks

8. Click Save

9. Click Apply Changes

Once that procedure has been completed on the primary node, perform it again on the secondary node with the appropriate IPv4 address value. Remember they must be the same on both nodes.

After configuring the sync interface, the interface assignments should have one labeled SYNC.

Add Firewall Rules for Synchronization

To complete the Sync interface configuration, firewall rules must be added to both nodes to allow synchronization.

At a minimum, the firewall rules must pass the configuration synchronization traffic (by default, HTTPS on port 443) and pfsync traffic. In most cases, a simple “allow all” style rule is used. For this guide, both will be shown and it will also serve as an indicator that synchronization is working.

On the primary node:

Set up a rule to allow configuration synchronization:

1. Navigate to Firewall > Rules on the SYNC tab

2. Click at the top of the list to create a new rule

3. Set Action to Pass

4. Set Source to SYNC Net

5. Set Destination to SYNC Address

6. Set Destination port range to 443 or choose HTTPS (443) from the drop-down selector

7. Set Description to Allow configuration synchronization

8. Click Save

Set up a rule to allow state synchronization:

1. Click at the top of the list to create another new rule

2. Set Action to Pass

3. Set Protocol to pfsync

4. Set Source to SYNC Net

5. Set Destination to any

6. Set Description to Allow state synchronization

7. Click Save

Set up a rule to allow ICMP for diagnostics, e.g. echo (ping) requests:

1. Click at the top of the list to create another new rule

2. Set Action to Pass

3. Set Protocol to ICMP

4. Set Source to SYNC Net

5. Set Destination to SYNC Net

6. Set Description to Allow ICMP for diagnostics

7. Click Save

8. Click Apply Changes

When complete, the rules will look like the following, which also includes a rule to allow ICMP for diagnostic purposes.

Example Sync Interface Firewall Rules

On the secondary node:

1. Navigate to Firewall > Rules on the SYNC tab

2. Click at the top of the list to create a new rule

3. Set Action to Pass

4. Set Protocol to any

5. Set Source to SYNC Net

6. Set Destination to any

7. Set Description to Temp rule for sync

8. Click Save

9. Click Apply Changes

Note

The rule on the secondary is different, but that is intended at this point. Once the first configuration synchronization has taken place, the temporary rule on the secondary will be replaced by the rules from the primary. Seeing that the rules changed on the Sync interface is a good indicator that it worked!

Configure pfsync

State synchronization using pfsync must be configured on both the primary and secondary nodes to function.

First on the primary node and then on the secondary, perform the following:

1. Navigate to System > High Avail. Sync

2. Check Synchronize States

3. Set Synchronize Interface to SYNC

4. Set Filter Host ID to a unique ID per node, e.g. 1 on the primary node and 2 on the secondary.

5. Set pfsync Synchronize Peer IP to the other node. Set this to 172.16.1.3 when configuring the primary node, or 172.16.1.2 when configuring the secondary node

6. Click Save

Configure XMLRPC

Warning

Configuration synchronization must only be configured on the primary node. Never activate options in this section on the secondary node of a two-member cluster.

On the primary node only, perform the following:

1. Navigate to System > High Avail. Sync

2. Set Synchronize Config to IP to the Sync interface IP address of the secondary node: 172.16.1.3

3. Set Remote System Username to admin.

   Note

   This can work with accounts other than admin, but doing so requires special account privileges and configuration steps.

4. Set Remote System Password to the admin user account password and be sure to confirm the password.

5. Check the boxes for each area to synchronize to the secondary node. For this guide, as with most configurations, all boxes are checked.

6. Click Save

As a quick confirmation that the synchronization worked, on the secondary node navigate to Firewall > Rules on the SYNC tab. The rules entered on the primary are now there, and the temporary rule is gone.

The two nodes are now linked for configuration synchronization! Changes made to the primary node in supported areas will be synchronized to the secondary whenever a change is made.

Warning

Do not make changes to the secondary in areas set to be synchronized! These changes will be overwritten the next time the primary node performs a synchronization.

Note

If cerficiates are included in XMLRPC synchronization, the primary node will synchronize its GUI certificate to the secondary. After that has synchronized, it must be manually selected on the secondary node under System > Advanced on the Admin Access tab. Otherwise the secondary may regenerate a new certificate any time the GUI starts as its expected entry is missing. Alternately, import the GUI certificate for the secondary on the primary so they are both present on both nodes, then select it for use on the secondary again after it synchronizes.

Add CARP VIPs

Now that the configuration synchronization is complete, the CARP Virtual IP addresses need only be added to the primary node and they will be automatically copied to the secondary. For this demonstration, two CARP VIPs will be added: One for WAN, and one for LAN.

1. Navigate to Firewall > Virtual IPs on the primary node.

2. Click Add at the bottom of the list to create a new VIP

3. Set Type to CARP

4. Set Interface to WAN

5. Enter the WAN CARP VIP into the IP Address(es) section Address box and pick the appropriate subnet mask. For this example, enter 198.51.100.200 and 24 (See WAN IP Address Assignments).

6. Enter a random password in Virtual IP Password. This need only match between the two nodes, which will be handled by synchronization.

7. Select an unused VHID Group as determined in Determine CARP VHID Availability.

   Note

   A common tactic is to make the VHID match the last octet of the IP address, so in this case 200 would be chosen.

8. Set the Advertising Frequency to a Base of 1 and a Skew of 1. This value will be automatically adjusted when it is copied to the secondary.

9. Enter a Description such as WAN CARP VIP.

10. Click Save

11. Click Apply Changes

The Base and Skew together determine how often a CARP heartbeat is sent. The value of Base adds whole seconds and should match between the two nodes. The Skew value adds 1/256th of a second increments. The primary node should always have a Skew of 1 or 0. The secondary node must be higher, typically 100+. That adjustment is handled automatically by the configuration synchronization process.

Note

If CARP appears to be too sensitive to latency on a given network, adjusting the Base by adding one second at a time is recommended until stability is achieved.

Repeat the above process for the LAN CARP VIP:

1. Navigate to Firewall > Virtual IPs

2. Click Add at the bottom of the list to create a new VIP

3. Set Type to CARP

4. Set Interface to LAN

5. Enter the LAN CARP VIP into the IP Address(es) section Address box and pick the appropriate subnet mask. For this example, enter 192.168.1.1 and 24 (See LAN IP Address Assignments).

6. Enter a random password in Virtual IP Password.

7. Select an unused VHID Group as determined in Determine CARP VHID Availability.

8. Set the Advertising Frequency to a Base of 1 and a Skew of 1.

9. Enter a Description such as LAN CARP VIP.

10. Click Save

11. Click Apply Changes

If there are any additional IP addresses in the WAN subnet that will be used for purposes such as 1:1 NAT, port forwards, VPNs, etc, they may be added now as well.

Tip

Instead of adding multiple CARP VIPs in a single segment, add multiple IP addresses on a single interface as IP Alias type VIPs using the CARP VIP for that interface as the parent. This reduces heartbeat traffic and also ensures they all fail over and back as a group.

Check Firewall > Virtual IPs on the secondary node to ensure that the VIPs synchronized as expected.

The Virtual IP addresses on both nodes will look like the following if the process was successful.

CARP Virtual IP Address List

Check CARP and State Synchronization Status

Now visit Status > CARP on both nodes to confirm the proper status.

The primary node should indicate MASTER status for all VIPs, and the secondary node should indicate BACKUP status for all VIPs.

The State Synchronization Status area should show the chosen Filter Host ID for each node if the states have synchronized sucessfully.

CARP VIP Status and State Synchronization Status on Primary

CARP VIP Status and State Synchronization Status on Secondary

If the status is incorrect, see Troubleshooting High Availability.

Setup Outbound NAT

Now it is time to put the new CARP VIPs to use. The Outbound NAT settings will synchronize so these changes need only be made to the primary node.

1. Navigate to Firewall > NAT, Outbound tab on the primary node

2. Change the Mode to Hybrid Outbound NAT

3. Click Save

This mode allows the default NAT rules to stay in place, but be overridden by manually created rules for CARP. This way it becomes unnecessary to manage entries for the firewall itself (which should not use the CARP VIP for NAT) and also can make the process less disruptive when adding new interfaces later.

• Click below the Mappings section to add a new rule

• Configure the rule as follows:

  Interface

  WAN

  Address Family

  IPv4

  Source

  Network, then enter the LAN subnet, e.g. 192.168.1.0/24

  Translation, Address

  Choose the WAN CARP VIP from the Address drop-down, 198.51.100.200

  Description

  LAN to WAN

  Leave all other options at the default values.

• Click Save

• Click below the Mappings section to add another new rule to the top of the list for IPsec pass-through.

  Note

  This is optional and may be omitted if there will not be any LAN clients which connect to external IPsec servers which lack NAT-T support.

• Configure the rule as follows:

  Interface

  WAN

  Address Family

  IPv4

  Source

  Network, then enter the LAN subnet, e.g. 192.168.1.0/24

  Destination

  Type

  Any

  Port

  500

  Translation

  Address

  Choose the WAN CARP VIP from the Address drop-down, 198.51.100.200

  Port or Range

  Check Static Port

  Description

  ISAKMP - LAN to WAN

  Leave all other options at the default values.

• Click Save

• Click Apply Changes

Warning

If an additional local interface is added later, such as a second LAN, DMZ, etc, and that interface uses private IP addresses, then additional outbound NAT rules must be added at that time. Similarly, if additional WANs are added later, they will also need a set of outbound NAT rules.

Danger

Do not use an overly permissive source on outbound NAT rules! If an outbound NAT rule matches traffic from the firewall using a VIP, it can break several protocols and cause instability. Use only local/internal networks as the source on NAT rules, or an alias containing them.

Using Localhost as a source is also appropriate but the translation address for traffic from localhost should be an interface address and not a CARP VIP.

Outbound NAT Rules for LAN with CARP VIP

Other NAT Concerns

If there are any port forwards to be added using the WAN CARP VIP, they may be added now using Firewall > NAT, Port Forward tab. Port forwards will work the same as usual, but the Destination will be the WAN CARP VIP.

Setup DHCP

The DHCP server daemons on the cluster nodes need adjustments so that they can work together. The changes will synchronize from the primary to the secondary, so as with the VIPs and Outbound NAT, these changes need only be made on the primary node.

1. Navigate to Services > DHCP Server, LAN* tab.

2. Set the DNS Server to the LAN CARP VIP, here 192.168.1.1

3. Set the Gateway to the LAN CARP VIP, here 192.168.1.1

4. Set the Failover Peer IP to the actual LAN IP address of the secondary node, here 192.168.1.3

5. Click Save

Setting the DNS Server and Gateway to a CARP VIP ensures that the local clients are talking to the failover address and not directly to either node. This way if the primary fails, the local clients will continue talking to the secondary node.

The Failover Peer IP allows the daemon to communicate with the peer directly in this subnet to exchange data such as lease information. When the settings synchronize to the secondary, this value is adjusted automatically so the secondary points back to the primary.

On both nodes, visit Status > DHCP Leases to see the status. A section will be displayed at the top containing the failover pool status, one line will be shown for each local interface pool. When the two nodes are working properly, both will indicate a “normal” status.

DHCP Failover Status

VPNs and Other Services

When configuring a VPN, such as OpenVPN or IPsec, pick a WAN CARP VIP as the Interface for the VPN and ensure the remote peer also builds the other side of the tunnel using the CARP VIP as the peer address.

For other local services, packages, etc. likewise a CARP VIP is recommended for binding if the service will work on both nodes.

High Availability support in packages varies. Check the package documentation for information on if, or how, various aspects of High Availability work with a specific package.

Additional Interfaces

Additional local interfaces may also be configured, repeating some of the previous steps as needed:

1. Assign the interface on both nodes identically

2. Enable the interface on both nodes, using different IP addresses within the same subnet

3. Add a CARP VIP inside the new subnet (Primary node only)

4. Add firewall rules (Primary node only)

5. Add Outbound NAT rules for a source of the new subnet, utilizing the CARP VIP for translation (Primary node only)

6. Configure the DHCP server for the new subnet, utilizing the CARP VIP for DNS and Gateway roles (Optional, Primary node only)