Multi-Path Routing

Current versions of pfSense® software include kernels built with the option ROUTE_MPATH which enables multi-path routing.

This allows the routing table to contain multiple entries to the same destination, which allows for weight-based balancing of traffic including Equal-cost multi-path routing (ECMP) if all gateways for a destination are weighted the same.

Warning

Currently multi-path routing can only be utilized by the dynamic routing daemon package FRR. The base system GUI does not currently support managing multiple routes to the same destination, but support is planned for future releases.

Multi-Path Routing Behavior

Packets may only take alternate paths when they are different in some way. When there are multiple paths to a destination the operating system computes outbound flow hashing for connections to decide which path to use for a packet. This concept is similar to the LAGG Hash Algorithm. The hashing takes the 5-tuple connection property set into account: “(protocol, source address, destination address, source port, destination port)”.

For example, testing with ICMP only from one host to another with no variation may never see flows take a second path. In contrast, testing repeated TCP connections could take multiple paths if the source port is randomized. The best way to test is by using multiple sources and destinations passing through the firewall and not from the firewall itself.

View Nexthop Data

The first thing to check is the routing table to ensure that there are multiple routes to the same destination:

: netstat -rWn4
Routing tables

Internet:
Destination        Gateway            Flags   Nhop#    Mtu      Netif Expire
[...]
10.5.0.0/16        198.51.100.5       UGS         0   1500        ix3
10.5.0.0/16        203.0.113.5        UGS         0   1500        ix2

Note

The route table in the status output of the FRR package will also show multiple entries for the same destination with different via addresses.

Next, there are two items to check to verify that multi-path routing is taking effect: Nexthop data and Nexthop group data.

Check the nexthop data with netstat -4onW and/or netstat -6onW:

: netstat -4onW
Nexthop data

Internet:
Idx   Type         IFA                Gateway             Flags      Use Mtu         Netif     Addrif Refcnt Prepend
[...]
32           v4/gw 198.51.100.17      198.51.100.5       GS            0   1500        ix3               1
33           v4/gw 203.0.113.17       203.0.113.5        GS            0   1500        ix2               1

Check nexthop group data with netstat -4OnW and/or netstat -6OnW:

: netstat -4OnW
Nexthop groups data

Internet:
GrpIdx  NhIdx     Weight   Slots           Gateway     Netif  Refcnt
34        ------- ------- ------- ----------------- ---------       2
              32       1       1      198.51.100.5       ix3
              33       1       1       203.0.113.5       ix2

The outputs of those commands should show both gateways and indicate that they belong to the same “group”.