Troubleshooting Asymmetric Routing

Asymmetric routing happens when traffic between two nodes takes a different path in each direction (e.g. A->B->C, C->D->A). This can pose a problem for TCP which has strict state tracking but often does not affect “stateless” protocols such as ICMP or UDP.

Common Scenario

What happens in most cases is this:

  • Client sends a TCP SYN packet which arrives at the firewall

  • The firewall creates a state table entry

  • The firewall sends an ICMP redirect to the client letting it know to reach the target server through an alternate gateway on the same interface

  • The target server sends a TCP SYN+ACK packet to the client by a different path that does not pass through the firewall

  • The client sends its ACK and later responses to the server using the other gateway and they are not visible to the firewall

  • After 30 seconds the firewall removes its state table entry as the firewall did not see the client and server fully establish a connection

  • Some time later the route the client learned via ICMP redirect expires and the client sends another packet back to the firewall

  • Since this packet is not starting a new connection the firewall drops the packet

  • The client gets disconnected as it no longer knows how to reach the server

Automatic Fix

The Bypass firewall rules for traffic on the same interface option located under System > Advanced on the Firewall & NAT tab activates rules for traffic to/from the static route networks which are much more permissive when it comes to creating states for TCP traffic and allowing it to pass. The rules allow any TCP packets regardless of their flags to create a state, and also utilize “Sloppy” state tracking which performs a less strict state match.

Manual Fix

The same rules may be created manually by adding one on the affected interface tab (e.g. LAN), and a second rule on the Floating tab using the same interface (LAN again) to match the traffic in the out direction. The rule must be set for a protocol of TCP, under TCP flags check Any Flags, and use a State Type of Sloppy.

The options for TCP flags and State Type can be found in the Advanced Options when editing a firewall rule.

../_images/asym_rule_advanced.png

Alternate Causes

On occasion these issues can be caused by other factors that lead to asymmetric routing, such as issues with route-to or reply-to, both having to do with gateways on interface settings.

Defining gateways under System > Routing does not cause this, but rather these problems can come up when the gateway is improperly configured on the interface pages, Interfaces > WAN, Interfaces > LAN, and so on.

See also

WAN vs LAN Interfaces

Gateway set when it should not be set

If a gateway is set on an internal interface, such as LAN, it can cause problematic behavior. Setting a gateway on an internal interface will tag outbound rule on that interfacewith route-to, and inbound rules with reply-to which will cause the firewall to forward packets to the defined gateway rather than following their natural path. For WANs this is typically a good thing! For LANs it is not. Among other ill effects, it can lead to a loop of sorts where packets bounce between the firewall and the defined gateway, eventually being blocked or dropped when their TTL expires.

Gateway not set when it should be set

A gateway should usually be set on a WAN or other external-type interface settings (MPLS, IP VPN, etc.) In these cases the reply-to and route-to behavior is desired and likely required. If it is missing the packets may be blocked or dropped as they attempt to leave the wrong interface. A packet could enter via the alternate WAN, but the reply would leave by the default gateway. Similar to the effect seen when improperly using an Interface Group for WAN interfaces.