High Availability Configuration Example

This recipe describes a typical pfSense® software high availability (HA) cluster configuration with two nodes (primary and secondary) containing three interfaces: WAN, LAN, and Sync. This is logically equivalent to a deployment with two interfaces (WAN and LAN), with the Sync interface carrying synchronization data between the primary and secondary nodes.

Note

This example covers both IPv4 and IPv6 configuration.

If static IPv6 assignments are not available, set IPv6 to None on all interfaces and skip any configuration tasks specific to IPv6.

See also

Review High Availability before following this recipe.

For help diagnosing issues with HA, see:

• Troubleshooting High Availability

• Troubleshooting High Availability Clusters in Virtual Environments

• Troubleshooting VPN Connectivity to a High Availability Secondary Node

Determine IP Address Assignments

The first task is to plan IP address assignments. A good strategy is to number the addresses as follows:

• Lowest usable IP address in the subnet as the CARP VIP address.

• The next IP address as the primary node interface IP address.

• The next IP address as the secondary node interface IP address.

This specific suggestion is optional. The best practice is to use a consistent and logical scheme to make design and administration simpler, but it is ultimately a matter of preference. For example, some network designers prefer to place routers at the end of a subnet instead of the beginning.

WAN IP Addresses

Select the WAN IP addresses from those assigned by the ISP. For this example, the assignments are listed in in Table WAN Interface IP Address Assignments.

For IPv4, the WAN subnet for the HA pair is 198.51.100.0/24. The WAN IPv4 addresses for the cluster are 198.51.100.200 through 198.51.100.202.

For IPv6, the WAN prefix is 2001:db8::/64 and the ISP is routing a prefix of 2001:db8:1:df30::/60 for local networks. The WAN IPv6 addresses for the cluster are 2001:db8::200/64 through 2001:db8::202/64.

Note

Though there is a whole /64 to utilize, this example uses the same ending values in both IPv4 and IPv6 for consistency and to make similarities more apparent.

WAN Interface IP Address Assignments

IP Address	Usage
198.51.100.0/24	WAN IPv4 subnet
198.51.100.1	WAN ISP IPv4 Gateway
198.51.100.200/24	CARP shared IPv4 address
198.51.100.201/24	Primary node WAN IPv4 address
198.51.100.202/24	Secondary node WAN IPv4 address
2001:db8::/64	WAN IPv6 Prefix
2001:db8::1	WAN ISP IPv6 Gateway
2001:db8:1:df30::/60	Routed IPv6 Prefix
2001:db8::200/64	CARP shared IPv6 address
2001:db8::201/64	Primary node WAN IPv4 address
2001:db8::202/64	Secondary node WAN IPv6 address

Note

In this case the ISP would route the IPv6 prefix (2001:db8:1:df30::/60) to the IPv4 WAN CARP VIP, 2001:db8::200. Requirements vary by ISP. If the ISP prefers to route to a link-local address, add a CARP VIP on the WAN interface using a link-local address for this purpose.

LAN IP Addresses

The LAN IPv4 subnet is 192.168.1.0/24 and the LAN IPv6 prefix is 2001:db8:1:df30::/64. For this example, the LAN IP addresses will be assigned as shown in Table LAN Interface IP Address Assignments.

LAN Interface IP Address Assignments

IP Address	Usage
192.168.1.1/24	CARP shared IPv4 address
192.168.1.2/24	Primary node LAN IPv4 address
192.168.1.3/24	Secondary node LAN IPv4 address
2001:db8:1:df30::1/64	CARP shared IPv6 address
2001:db8:1:df30::2/64	Primary node LAN IPv6 address
2001:db8:1:df30::3/64	Secondary node LAN IPv6 address
fe80::1:1/64	CARP shared IPv6 Link-Local Address

Note

The CARP IPv6 link-local address in this example uses fe80::1:1/64 as the fe80::1/64 address is reserved for use by pfSense software in certain scenarios and can conflict. Using a different address avoids any potential problems.

Sync Interface IP Addresses

The IP addresses on the Sync interface are used only for communication between the firewalls. As there are no other devices on the Sync network, there is no need for a shared CARP VIP on the Sync interface, so it is not necessary to add a VIP on that interface.

This example uses the IPv4 subnet 172.16.1.0/24 and IPv6 prefix 2001:db8:1:df31::/64 for the Sync interface. The example only uses two IP addresses, but the IPv4 subnet uses a /24 to be consistent with the other internal interface (LAN). For the last octet of the IP addresses, use the same last octet as the LAN IP addresses for consistency.

Sync Interface IP Address Assignments

IP Address	Usage
172.16.1.2/24	Primary node Sync IPv4 address
172.16.1.3/24	Secondary node Sync IPv4 address
2001:db8:1:df31::2/64	Primary node Sync IPv6 address
2001:db8:1:df31::3/64	Secondary node Sync IPv6 address

Example Cluster Diagram

Figure Example High Availability Cluster Network Diagram shows the layout of this example HA cluster. The primary and secondary nodes each have identical connections to the WAN and LAN, and a crossover cable between them to connect the Sync interfaces.

See also

In this basic example, the WAN switch and LAN switch are still potential single points of failure. Switching redundancy is covered in Layer 2 Redundancy.

Example High Availability Cluster Network Diagram

Cluster Configuration Basics

Both nodes requires a few basic configuration steps before they are ready to be configured as a high availability cluster.

Danger

Do not connect both nodes to the same LAN (switch/layer 2) before both nodes have a non-conflicting LAN configuration.

Installation, interface assignment, and basic configuration

Install pfSense software on both nodes and assign the interfaces identically on both nodes.

Warning

Interfaces must be assigned in the same order on all nodes exactly. If the interface order is not identical, configuration synchronization and other tasks will not behave correctly. If any adjustments have been made to the interface assignments in the future, they must be replicated identically on both nodes.

After installation, connect to the GUI and use the Setup Wizard to configure each node with a unique hostname and non-conflicting static IP addresses.

See also

Setup Wizard

For example, the primary node could be firewall-a.example.com and the secondary could be firewall-b.example.com, or a more personalized pair of names.

Note

Avoid naming the nodes master or backup since those are states, not roles. Consider naming them primary and secondary instead.

The default configuration on WAN is for DHCP, this must be changed to a static IP address configuration, such as the example WAN addresses in WAN Interface IP Address Assignments. Be sure to configure appropriate upstream gateways for IPv4 and IPv6.

The default LAN IPv4 address is 192.168.1.1, but each node must be moved to its own unique and non-conflicting address. The default LAN IPv6 configuration is set to track WAN, which is not valid for HA, so it must be changed to a static configuration The example addresses for IPv4 and IPv6 are shown in LAN Interface IP Address Assignments.

Once each node has a unique LAN IPv4 and IPv6 address, then both nodes may be plugged into the same LAN switch.

Setup Sync Interface

Before proceeding, the Sync interfaces on the cluster nodes must be configured. Sync Interface IP Address Assignments lists the addresses to use for the Sync interfaces on each node. Once that has been completed on the primary node, perform it again on the secondary node with the appropriate address values.

To complete the Sync interface configuration, add firewall rules on both nodes to allow synchronization.

At a minimum, the firewall rules must pass configuration synchronization traffic (by default, HTTPS on port TCP 443), pfsync traffic, and Kea DHCP HA traffic on TCP ports 8765 and 8766. In most cases, a simple “allow all” style rule is sufficient, but using specific rules is a better practice.

When complete, the rules will look like the example in figure Example Sync Interface Firewall Rules, which also includes a rule to allow ICMP echo (ping) for diagnostic purposes.

Example Sync Interface Firewall Rules

The secondary does not need all of these rules initially, only a rule to allow traffic to the GUI for XMLRPC to function. The full set of rules will synchronize via XMLRPC configuration synchronization later.

Configure State Synchronization (pfsync)

State synchronization using pfsync must be configured on both the primary and secondary nodes to function.

See also

State Synchronization Settings (pfsync)

First on the primary node and then on the secondary, perform the following:

• Navigate to System > High Availability

• Check Synchronize States

• Set Synchronize Interface to SYNC

• Set a custom Filter Host ID

  For example, 1 on the primary node, 2 on the secondary node.

• Set pfsync Synchronize Peer IP to the Sync interface IPv4 address of the other node

  Set this to 172.16.1.3 on the primary node, or 172.16.1.2 on the secondary node

• Click Save

Configure Configuration Synchronization (XMLRPC)

Warning

Configuration synchronization must only be configured on the primary node. Never activate options in this section on the secondary node of a two-member cluster.

See also

Configuration Synchronization Settings (XMLPRC Sync)

On the primary node only, perform the following:

• Navigate to System > High Availability

• Set Synchronize Config to IP to the Sync interface IPv4 address on the secondary node, 172.16.1.3

• Set Remote System Username to admin

  Note

  This must either be the admin account or a user on both nodes with the “System - HA node sync” privilege.

• Set Remote System Password to the admin user account password, and repeat the value in the confirmation box.

• Check the boxes for each area to synchronize to the secondary node.

  For this guide, as with most configurations, all boxes are checked.

  Tip

  Use the Toggle All button to select all of the options at once, rather than selecting them individually.

• Click Save

As a quick confirmation that the synchronization worked, on the secondary node navigate to Firewall > Rules, SYNC tab. The rules entered on the primary should now be present on the tab, and the temporary rule has been removed.

The two nodes are now linked for configuration synchronization! Supported areas will synchronize to the secondary node whenever a change is made on the primary node.

Warning

Do not make changes to the secondary in areas set to be synchronized! These changes will be overwritten the next time the primary node performs a synchronization.

Configuring the CARP Virtual IPs

With configuration synchronization in place, the CARP Virtual IP addresses need only be added to the primary node and they will automatically copy to the secondary node.

Each VIP needs a unique VHID on a given broadcast domain/layer 2. See CARP VIP VHID Assignments for the VHIDs to use in this example.

CARP VIP VHID Assignments

CARP VIP Address	Interface	VHID
198.51.100.200/24	WAN	200
2001:db8::200/64	WAN	201
192.168.1.1/24	LAN	1
2001:db8:1:df30::1/64	LAN	2
fe80::1:1/64	LAN	3

• Navigate to Firewall > Virtual IPs on the primary node to manage CARP VIPs

• Click Add at the top of the list to create a new VIP.

  Note

  Each interface handling user traffic must have a CARP VIP, in this case WAN and LAN.

• Fill in the VIP options as described below. See CARP VIP VHID Assignments for specific settings unique to each VIP in this example. Refer to VIP Configuration Options for more information on the options in general.

Type:

CARP

Interface:

The interface upon which the VIP resides, e.g. WAN

Address(es):

The IP address and subnet mask/prefix value for the VIP. Consult CARP VIP VHID Assignments for the address values in this example.

Note

The subnet mask/prefix must match the subnet mask on the interface IP addresses. For example, the first VIP 198.51.100.200 and 24 on WAN (See WAN Interface IP Address Assignments).

Virtual IP Password:

Password for the CARP VIP. Use a random value.

This need only match between the two nodes, which will be handled by synchronization. The password and confirm password box must both be filled in and they must match.

VHID Group:

Virtual ID for the CARP VIP. Consult CARP VIP VHID Assignments for the VHID to use for each address in this example.

Tip

A common tactic is to make the VHID match the last octet of the IP address, e.g. 200 for an address ending in .200.

Advertising Frequency:

How often the node sends CARP heartbeats on its interface.

Base:

Whole seconds between Heartbeats, typically 1.

Skew:

Fractions of a second (1/256th increments).

Note

A primary node is typically set to 0 or 1, secondary nodes will be 100 or higher. This adjustment is handled automatically by XMLRPC synchronization.

Description:

Text to identify the VIP, such as WAN CARP VIP.

The above description used the WAN VIP as an example. Repeat the procedure for every required CARP VIP (See CARP VIP VHID Assignments).

If there are any additional IP addresses in the WAN subnet that will be used for purposes such as 1:1 NAT, port forwards, VPNs, etc, add them now.

Tip

For HA setups designed with multiple CARP VIPs of the same address family on the same interface, instead consider adding the additional CARP VIPs as IP alias VIPs which use a single CARP VIP as their parent interface. This reduces the number of CARP VIP heartbeats on a network segment and also allows the VIPs to fail as a group. For details, see Using IP Aliases to Reduce Heartbeat Traffic.

Click Apply Changes after completing the VIP configuration.

After adding VIPs, check Firewall > Virtual IPs on the secondary node to ensure that the VIPs synchronized as expected.

The Virtual IP addresses on both nodes will look like CARP Virtual IP Address List if the process was successful.

CARP Virtual IP Address List

Configure Outbound NAT for CARP

The next step is to configure outbound NAT so that the firewall translates IPv4 traffic from clients on the LAN to the shared IPv4 CARP VIP address on WAN as the address as it exits.

• Navigate to Firewall > NAT, Outbound tab

• Click to select Hybrid Outbound NAT rule generation

• Click Save

Hybrid mode allows the default NAT rules to stay in place, but also allows the default rules to be overridden by manually created rules. This way it becomes unnecessary to manage entries for the firewall itself (which should not use the CARP VIP for NAT) and it can also make the process less disruptive when adding new interfaces later.

Create outbound NAT rules for internal subnet sources to work with the CARP IP address.

• Click below the Mappings section to add a new rule

• Configure the rule as follows:

  Interface:

  WAN

  Address Family:

  IPv4

  Source:

  LAN Subnets

  Translation, Address:

  Choose the WAN CARP IPv4 VIP from the Address drop-down, 198.51.100.200

  Description:

  LAN to WAN

  Leave all other options at the default values.

• Click Save

• Click below the Mappings section to add another new rule to the top of the list for IPsec pass-through.

  Note

  This is optional and may be omitted if there will not be any LAN clients connecting to external IPsec servers which lack NAT-T support.

• Configure the rule as follows:

  Interface:

  WAN

  Address Family:

  IPv4

  Source:

  LAN Subnets

  Destination:

  Type:

  Any

  Port:

  500

  Translation:

  Address:

  Choose the WAN CARP IPv4 VIP from the Address drop-down, 198.51.100.200

  Port or Range:

  Check Static Port

  Description:

  ISAKMP - LAN to WAN

  Leave all other options at the default values.

• Click Save

• Click Apply Changes

Warning

If an additional local interface is added later, such as a second LAN, DMZ, etc, and that interface uses private IP addresses, then additional outbound NAT rules must be added to translate those sources. Similarly, if additional WANs are added later, they will also need a set of outbound NAT rules.

Danger

Do not use an overly permissive source on outbound NAT rules! If an outbound NAT rule matches traffic from the firewall using a VIP, it can break several protocols and cause instability. Use only local/internal networks as the source on NAT rules, or an alias containing those local private networks.

Any rules using Localhost as a source must use an interface address for translation and not a CARP VIP.

When complete, the rule changes will look like those found in Outbound NAT Rules for LAN with CARP VIP

Outbound NAT Rules for LAN with CARP VIP

Modifying the DHCP Server

The DHCP server daemons on the cluster nodes need adjustments so that they can work together. The changes will synchronize from the primary to the secondary, so as with the VIPs and Outbound NAT, these changes need only be made on the primary node.

Note

This guide assumes the Kea DHCP Backend is active as it is the only backend which supports HA for both IPv4 and IPv6 (Server Backend).

Set the Backend

On both nodes, ensure the Kea DHCP backend is active:

• Navigate to System > Advanced, Networking tab

• Set the Server Backend to Kea DHCP in the DHCP Options section if it is not already selected

• Click Save if the value was changed

Configure DHCP for IPv4

On the primary node only, enable HA for IPv4 DHCP in the Kea Settings:

• Navigate to Services > DHCP Server, Settings tab

• Check Enable high availability

• Set Node Role to Primary

• Set Local Name to a custom value, e.g. ha-primary

• Set Local Address to the IP address of the sync interface on the primary node, e.g. 172.16.1.2

• Set Remote Name to the name of the secondary node, e.g. ha-secondary.

• Set Remote Address to the IP address of the Sync interface on the secondary node, e.g. 172.16.1.3

• Configure TLS support to encrypt the DHCP HA traffic (optional)

  The Sync interface is directly connected in this example so there is no risk of the traffic being intercepted, but using encryption is still a best practice. Consult TLS Transport for more information on configuring TLS for DHCP HA. Note that TLS settings do not synchronize and must be configured separately on each node.

• Click Save

Note

XMLRPC configuration synchronization will adjust these options appropriately when copying the settings to the secondary node.

Still on the primary node only, configure the LAN interface DHCP settings to use the LAN CARP IPv4 VIP:

• Navigate to Services > DHCP Server, LAN tab

• Set the DNS Server to the LAN CARP VIP, here 192.168.1.1

• Set the Gateway to the LAN CARP VIP, here 192.168.1.1

• Click Save

Setting the DNS Server and Gateway to a CARP VIP ensures that the local clients are talking to the failover address and not directly to either node. This way if the primary fails, the local clients will continue talking to the secondary node.

Configure DHCP for IPv6

Configure DHCPv6 HA

On the primary node only, enable HA for IPv6 DHCP in the Kea Settings:

• Navigate to Services > DHCPv6 Server, Settings tab

• Check Enable high availability

• Set Node Role to Primary

• Set Local Name to a custom value, e.g. ha-primary

• Set Local Address to the IP address of the sync interface on the primary node, e.g. 2001:db8:1:df31::2

• Set Remote Name to the name of the secondary node, e.g. ha-secondary.

• Set Remote Address to the IP address of the Sync interface on the secondary node, e.g. 2001:db8:1:df31::3

• Configure TLS support to encrypt the DHCP HA traffic (optional)

  The Sync interface is directly connected in this example so there is no risk of the traffic being intercepted, but using encryption is still a best practice. Consult TLS Transport for more information on configuring TLS for DHCP HA. Note that TLS settings do not synchronize and must be configured separately on each node.

• Click Save

Note

XMLRPC configuration synchronization will adjust these options appropriately when copying the settings to the secondary node.

Configure DHCPv6 Interface Settings

Still on the primary node only, configure the LAN interface DHCPv6 settings to use the LAN IPv6 CARP VIP for DNS.

• Navigate to Services > DHCPv6 Server, LAN tab

• Set the DNS Server to the LAN CARP VIP, here 2001:db8:1:df30::1

• Click Save

Configure IPv6 Router Advertisements

Finally, configure IPv6 Router Advertisements (RA) to use the LAN CARP IPv6 Link-Local VIP for the IPv6 gateway:

• Navigate to Services > Router Advertisement, LAN tab

• Set Router Mode to Managed so that clients will use DHCPv6 to obtain addresses and other settings.

• Set RA Interface to the LAN CARP IPv6 Link-Local VIP, fe80::1:1

• Check Enable DNS

• Check Mirror DHCPv6 to have the RA daemon advertise the same DNS settings as the DHCPv6 server

• Click Save

Finish Up & Test

At this point the cluster is configured and ready to use. Review the testing procedure in Verifying Failover Functionality to confirm it is operating properly.

Once everything tests OK, make configuration backups of both nodes.