AWS High Availability

The AWS High Availability package enables the use of active/standby pairs of pfSense® Plus instances which modify AWS resources in response to failover events.

The AWS High Availability package builds upon the CARP Virtual IP Address (VIP) functionality in pfSense software to provide an analogous mechanism for High Availability (HA) in the AWS Virtual Private Cloud.

The need for the AWS High Availability package stems from the unique environment of a VPC. These environments do not have a traditional layer 2 network, and thus do not support broadcast and multicast. This limitation necessitates using the AWS API for configuration of VPC resources to define routes, bind addresses, and make external IP address mappings – actions that CARP cannot do natively due to the nature of the AWS environment.

See also

How it Works

When the AWS High Availability package is installed, it extends CARP VIP MASTER events on configured CARP VIPs such that user configurable modifications to VPC resources can be made without manual intervention.

An instance be configured to modify Elastic Interface IP address assignments, AWS VPC Route Tables, and Elastic IP Allocations in response to a CARP VIP event. This allows a failover event to modify routes and map subnet IPs and Elastic IPs to the Elastic Network Adapter on the node where a CARP VIP has MASTER status.

The package performs these actions by making contact with the AWS API to make changes dynamically as needed.

Prerequisites

AWS Account Privileges

The package requires access to the AWS API, relying on a correctly configured EC2 Instance Profile to assign a Role to the instance for authorization. The Role must have the following privileges for the associated resources:

DescribeRegions

  • All resources (“*”)

DescribeRoutetables

  • All resources (“*”)

AssignPrivateIpAddresses

  • All Elastic Network Interfaces which will be configured with IPv4 IPs

ReplaceRoute

  • All Route Tables which will have routes modified in response to CARP events

AssociateAddress

  • All Elastic IP Allocations which will be modified in response to CARP events

  • All VPC subnets with which the Elastic IP will associated

  • All Elastic Network Interfaces with which the Elastic IP will be associated

  • All EC2 instances to which the Elastic Network Interfaces are assigned

CreateTags

  • All Elastic Network Interfaces which will be configured with IPv4 IPs

  • All Elastic IP Allocations which will be modified in response to CARP events

  • All Route Tables which will have routes modified in response to CARP events

DescribeTags

  • All resources (“*”)

pfSense Plus Software Configuration

Before configuring the AWS High Availability package, the following areas must be configured on pfSense Plus Software.

High Availability Synchronization

State Synchronization may be configured on both primary and secondary instances with pfsync Synchronize Peer IP values defined pointing to the other node. When configured in this way, pfsync utilizes unicast instead of the default directed multicast, which does not work on AWS VPC.

CARP Virtual IP Addresses

All AWS High Availability configuration requires at least one available unicast CARP VIP configured on an interface. Any number of IP, Route, and Elastic IP actions may be configured for a given CARP VHID.

Note

Reminder: Unicast CARP is required as the default multicast communication does not work in AWS VPC.

The configuration for a unicast CARP VIP is nearly identical to a traditional multicast CARP VIP. The difference is:

  • The CARP Mode must be set to Unicast

  • The Peer Address field to the right of the Unicast option must be set to the actual interface IP address on the HA peer node.

For example, with a CARP VIP for the WAN interface, the peer address on the VIP on the primary node is set to the WAN interface IP address of the secondary node. If configuration synchronization (XMLRPC Sync) is enabled, the primary node will automatically adjust this value when copying the VIP to the secondary node so that the VIP on the secondary uses the WAN interface IP address of the primary for its peer address.

../_images/aws-ha-unicast-carp-wan-pri-example.png

WAN Interface Unicast CARP Virtual IP Address - Primary

../_images/aws-ha-unicast-carp-wan-sec-example.png

WAN Interface Unicast CARP Virtual IP Address - Secondary

In configurations where a cluster is synchronizing across availability zones, there is no common address to all subnets, so a dummy address must be chosen for the CARP VIP. In these instances, choose an addres from the RFC 5737 documentation networks (192.0.2.0/24, 192.51.100.0/24 or 203.0.113.0/24). The purpose of the CARP VIP is entirely for failover negotiation, and should not be used as a source or destination address.

../_images/aws-ha-unicast-carp-az-example.png

WAN Interface Unicast CARP Virtual IP Address - Peer in Other AZ

Note

If peers are in different subnets, unicast CARP traffic will egress through the default route unless another route to the peer has been configured. The route may seem counterintuitive if the selected CARP interface is not the default WAN interface - in which case, absent a route to direct the traffic through a specific gateway, the unicast CARP traffic will route via the WAN interface to the default route.

Package Configuration

To configure the AWS High Availability package, navigate to System > AWS High Availability. This displays the IPs tab for the package.

../_images/aws-ha-system-menu.png

AWS High Availability System Menu Entry Location

IPs

The IPs tab controls how the package updates IP address assignments to Elastic Network Interfaces.

When configuring an IP action in the IPs tab, the user is presented with the following options:

CARP VHID

A drop-down selection to choose the CARP VIP which will trigger this route to be updated when it encounters a failover event and assumes MASTER status. This list is presented with the interface name and VHID (e.g. wan@1).

Interface

A drop-down selection to choose the interface destination of the route. Only Elastic Network Interfaces will be presented.

Private IP Address

An input field for the IP Address to associate with the Elastic Network Interface. This address must be a valid address in the VPC Subnet associated with the Elastic Network Interface.

XMLRPC Sync

A checkbox to enable synchronizing this IP action via XMLRPC. The Private IP Address will be mapped to the VPC Subnet of the Elastic Network Interface of the peer automatically, if it differs and the subnet masks are the same.

../_images/aws-ha-ip.png

AWS High Availability IP Address Entry

Routes

The Routes tab controls how the package updates route entries in AWS during failover events.

When configuring a route action in the Routes tab, the user is presented with the following options:

CARP VHID

A drop-down selection to choose the CARP VIP which will trigger this route to be updated when it encounters a failover event and assumes MASTER status. This list is presented with the VHID and interface name (e.g. wan@1).

Route Table ID

Input field to specify the AWS resource identifier for the route table to modify (e.g. rtb-05ed6e7c46531c6c3).

Route CIDR

An input field for the CIDR of the route destination address or network to be replaced in the routing table (e.g. 0.0.0.0/0 for the default route).

Interface

A drop-down selection to choose the interface destination of the route. Only Elastic Network Interfaces will be presented.

XMLRPC Sync

A checkbox to enable synchronizing this Route action via XMLRPC.

All fields are required.

../_images/aws-ha-route.png

AWS High Availability Route Entry

Elastic IP Addresses

The Elastic IPs tab associates a CARP VIP with an Elastic IP address and interface.

When configuring an Elastic IP action, the user is presented with the following options:

CARP VHID

A drop-down selection to choose the CARP VIP which will trigger this Elastic IP address to be updated when it encounters a failover event and assumes MASTER status. This list is presented with the VHID and interface name (e.g. wan@1).

Elastic IP Allocation ID

An input field for the Elastic IP Allocation ID that identifies the Elastic IP Allocation to modify. This value must be filled in by the user.

Interface

A drop-down selection to choose the interface destination of the route. Only Elastic Network Interfaces will be presented.

Private IP Address

A drop-down selection to choose the Private IP Address of the interface to which the Elastic IP address will point. Private IP Addresses presented will include addresses that are known to be associated with the Elastic Network Interface and addresses configured on the IP tab for the interface.

This value is populated automatically with the IP address of the selected CARP VIP.

XMLRPC Sync

A checkbox to enable synchronizing this Elastic IP action via XMLRPC. The Private IP Address will be mapped to the VPC Subnet of the Elastic Network Interface of the peer automatically, if it differs and the subnet masks are the same.

../_images/aws-ha-eip.png

AWS High Availability Elastic IP Address Options Tab

Behavior

When an instance assumes the MASTER status for a CARP VHID, the package takes the following actions in order:

  • Assigns all VIPs for that VHID to the Elastic Network Interface associated with the VHID.

  • Performs all configured route replacements configured for the VHID sequentially.

  • Performs all configured Elastic IP re-associations configured for the VHID sequentially.

When an action is completed on a resource, it is also updated with tags to record the time of modification and the CARP advertising frequency base and skew of the member that acted upon it.

If the package fails to complete any of these tasks via AWS API calls, it generates a system notice.

Background Service

AWS High Availability installs a small background service in the form of a script scheduled to run by minicron every 60 seconds. This script cycles through the configured actions on the system and checks that a lower priority CARP member has not modified the resources, indicating that the CARP link is not configured correctly. If this is found to be the case, a notification is generated and the resource actions are executed again to correct it.

EC2 Service Endpoints

AWS High Availability uses the AWS PHP SDK to access AWS API regional endpoints. Because these endpoints resolve to addresses outside of a VPC subnet, without additional action each pfSense® Plus instance will require a valid route through an AWS Internet Gateway or NAT Gateway that remains intact across resource actions.

AWS PrivateLink allows you to map AWS API endpoints to addresses in your VPC subnets, which avoids this scenario and keeps API endpoint traffic local to your VPC. Follow the guides at https://docs.aws.amazon.com/vpc/latest/privatelink/create-interface-endpoint.html and https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/interface-vpc-endpoints.html to create VPC Service Endpoints for EC2 in a VPC Subnet.

Note

When configuring AWS High Availability across availability zones, a subnet in each availability zone will require its own EC2 VPC Service Endpoint.

Limitations

Due to limitations in AWS, established TCP streams that are traversing through pfSense® Plus instances must be re-established after failover. AWS is not currently capable of redirecting traffic for established streams in response to changes to Route Tables, Elastic Network Interfaces, and Elastic IP Addresses.