High Availability Prerequisites

Achieving a redundant configuration requires meeting a few prerequisites, which this document describes.

Assumptions

This guide assumes that:

  • The cluster contains only two nodes (primary and secondary).

  • Both cluster nodes are using identical hardware.

  • Both nodes have a factory default configuration and there are no existing settings.

Warning

Do not connect both nodes to the same LAN (switch/layer 2) before both nodes have a non-conflicting LAN configuration at the end of this section. Otherwise there will be an IP address conflict and communication with each node individually will not be possible until the conflict is resolved.

Updates and Patches

Before starting, ensure both nodes are running the latest available version of pfSense® Plus software. Additionally, the best practice is to install the System Patches package and apply any recommended patches, especially those relevant to HA.

Determine the Synchronization Interface

One interface on each cluster node is dedicated for synchronization tasks. This is typically referred to as the “Sync” interface, and it is used for configuration synchronization, pfsync state synchronization, and some other function-specific synchronization tasks. Any available interface may be used for this role, but the best practice is to use a dedicated interface directly connected between the two nodes.

Warning

This should not be called a “CARP” interface as it is not involved with CARP. CARP heartbeats happen on each interface with a CARP VIP; CARP traffic and failover actions do not utilize the Sync interface.

Interface Assignments

Warning

Interfaces must be assigned in the same order on all nodes exactly. If the interface order is not identical, configuration synchronization and other tasks will not behave correctly. If any adjustments have been made to the interface assignments in the future, they must be replicated identically on both nodes.

IP Address Requirements

A High Availability cluster using CARP needs three IP addresses in each subnet along with a separate unused subnet for the Sync interface. For WANs, this means that each WAN requires a /29 subnet or larger for an optimal configuration. Each node uses One IP address, plus a shared CARP VIP address for failover. The synchronization interface only requires one IP address per node.

The IP addresses used in this guide are shown in the following tables, substitute the real IP addresses as needed.

WAN Interface IP Address Assignments

IP Address

Usage

198.51.100.0/24

WAN IPv4 subnet

198.51.100.1

WAN ISP IPv4 Gateway

198.51.100.200/24

CARP shared IPv4 address

198.51.100.201/24

Primary node WAN IPv4 address

198.51.100.202/24

Secondary node WAN IPv4 address

2001:db8::/64

WAN IPv6 Prefix

2001:db8::1

WAN ISP IPv6 Gateway

2001:db8:1:df30::/60

Routed IPv6 Prefix

2001:db8::200/64

CARP shared IPv6 address

2001:db8::201/64

Primary node WAN IPv4 address

2001:db8::202/64

Secondary node WAN IPv6 address

Note

In this case the ISP would route the IPv6 prefix (2001:db8:1:df30::/60) to the IPv4 WAN CARP VIP, 2001:db8::200. Requirements vary by ISP. If the ISP prefers to route to a link-local address, add a CARP VIP on the WAN interface using a link-local address for this purpose.

LAN Interface IP Address Assignments

IP Address

Usage

192.168.1.1/24

CARP shared IPv4 address

192.168.1.2/24

Primary node LAN IPv4 address

192.168.1.3/24

Secondary node LAN IPv4 address

2001:db8:1:df30::1/64

CARP shared IPv6 address

2001:db8:1:df30::2/64

Primary node LAN IPv6 address

2001:db8:1:df30::3/64

Secondary node LAN IPv6 address

fe80::1:1/64

CARP shared IPv6 Link-Local Address

Note

The CARP IPv6 link-local address in this example uses fe80::1:1/64 as the fe80::1/64 address is reserved for use by pfSense software in certain scenarios and can conflict. Using a different address avoids any potential problems.

Sync Interface IP Address Assignments

IP Address

Usage

172.16.1.2/24

Primary node Sync IPv4 address

172.16.1.3/24

Secondary node Sync IPv4 address

2001:db8:1:df31::2/64

Primary node Sync IPv6 address

2001:db8:1:df31::3/64

Secondary node Sync IPv6 address

See also

This example only uses a single WAN. For multiple WANs, see the High Availability Configuration Example with Multi-WAN in the documentation for pfSense software.

Single address CARP

It is technically possible to configure an interface with a CARP VIP as the only IP address in a given subnet, but it is not generally recommended. When used on a WAN, this type of configuration will only allow communication from the primary node to the WAN, which greatly complicates tasks such as updates, package installations, gateway monitoring, or anything that requires external connectivity from the secondary node. It can be a better fit for an internal interface, however internal interfaces do not typically suffer from the same IP address limitations as a WAN, so it is still preferable to configure IP addresses on all nodes. Such a configuration is not covered in this guide.

Determine CARP VHID Availability

CARP can interfere with VRRP, HSRP, or other systems using CARP if conflicting identifiers are used. In order to ensure that a segment is clear of conflicting traffic, perform a packet capture on each interface looking for CARP/VRRP traffic. A given VHID must be unique on each layer 2, so each interface must be checked separately. The same VHID may be used on different segments so long as they are separate broadcast domains.

If any CARP or VRRP traffic is present, note the VHID/VRID and avoid using that identifier when configuring the CARP VIP VHIDs later.

This guide assumes there is no other potentially conflicting traffic present.

CARP VIP VHID Assignments

CARP VIP Address

Interface

VHID

198.51.100.200/24

WAN

200

2001:db8::200/64

WAN

201

192.168.1.1/24

LAN

1

2001:db8:1:df30::1/64

LAN

2

fe80::1:1/64

LAN

3

Setup Requirements

Using the Setup Wizard, or manually afterward, configure each firewall with a unique hostname and non-conflicting static IP addresses.

For example, the primary node could be firewall-a.example.com and the secondary could be firewall-b.example.com, or a more personalized pair of names.

Note

Avoid naming the nodes master or backup since those are states, not roles. Consider naming them based on primary and secondary instead.

The default configuration on WAN is for DHCP, this must be changed to a static IP address configuration, such as the example WAN addresses in WAN Interface IP Address Assignments. Be sure to configure appropriate upstream gateways for IPv4 and IPv6.

The default LAN IPv4 address is 192.168.1.1, but each node must be moved to its own unique and non-conflicting address. The default LAN IPv6 configuration is set to track WAN, which is not valid for HA, so it must be changed to a static configuration The example addresses for IPv4 and IPv6 are shown in LAN Interface IP Address Assignments.

Once each node has a unique LAN IPv4 and IPv6 address, then both nodes may be plugged into the same LAN switch.

Both nodes must have the GUI running on the same port and protocol. This guide assumes both use HTTPS on port 443.

The both nodes must have the same admin account password, or setup an alternate dedicated synchronization user with the System - HA node sync privilege and the same password on both nodes.

Switch / Layer 2 Configuration

CARP Concerns

In the default mode, CARP heartbeats utilize multicast and may require special handling on the switches involved with the cluster. Some switches filter, rate limit, or otherwise interfere with multicast in ways that can cause CARP to fail. Also, some switches employ port security methods which may not work properly with CARP.

At a minimum, to use multicast mode CARP VIPs the switch must:

  • Allow Multicast traffic to be sent and received without interference on ports using CARP VIPs.

  • Allow traffic to be sent and received using multiple MAC addresses.

  • Allow the CARP VIP MAC address to move between ports.

Nearly all problems with CARP failing to properly reflect the expected status are failures of the switch or other layer 2 issues, so be sure the switches are properly configured before continuing.

If multicast mode is not viable, CARP VIPs may be configured in Unicast mode on pfSense Plus software. Unicast mode sends heartbeats to a single defined peer IP address (and vice versa). This mode works across any L3 connectivity through which the heartbeats can be transmitted between the peers without relying on multicast. However, use of unicast mode on traditional infrastructure where multicast is more suitable should be avoided. In unicast mode switches may flood packets for unicast CARP VIPs to all ports, leading to significant security and performance concerns.

Port Configuration

Each node must be connected to a common, but separate, layer 2 on each interface. This means that WAN, LAN, and other interfaces must be connected to separate switches or VLANS with each node being connected to the same segments on each.

For example, the WAN ports of each node must connect to the same WAN switch, which then connects to the WAN ISP Customer Premise Equipment (CPE)/Router/Upstream link. The LAN ports would all connect to the same LAN switch, and so on. The Sync interface may be connected directly between the two nodes without a switch. See Example High Availability Cluster for an example connection layout.