Troubleshooting Routes¶
When diagnosing traffic flow issues, one of the first things to check is the routes known to pfSense®.
Viewing Routes¶
There are two ways to view the routes: Via the WebGUI, and via the command line.
To view the routes in the WebGUI, navigate to Diagnostics > Routes and output is shown similar to Figure Route Display.
Route Display¶
The output from the command line is similar to that seen in the WebGUI:
# netstat -rWn
Routing tables
Internet:
Destination Gateway Flags Use Mtu Netif Expire
default 198.51.100.1 UGS 1822 1500 igb1
10.2.0.0/24 link#2 U 0 1500 igb0
10.2.0.1 link#2 UHS 0 16384 lo0
127.0.0.1 link#11 UH 204 16384 lo0
198.51.100.0/24 link#3 U 1181 1500 igb1
198.51.100.1 00:08:a2:09:95:b6 UHS 2789 1500 igb1
198.51.100.2 link#3 UHS 0 16384 lo0
The columns shown on these screens indicate various properties of the routes, and are explained later in this section.
Destination¶
This column contains the destination host or network. The default route for the
system is simply listed as default. Otherwise, hosts are listed as by IP
address, and networks are listed with an IP address and CIDR subnet mask.
Gateway¶
A gateway is the router through which packets going to a specific destination
are sent. If this column shows a link, such as link#1, then that network is
directly reachable by that interface and no special routing is necessary. If a
host is visible with a MAC address, then it is a locally reachable host with an
entry in the ARP table, and packets are sent there directly.
Flags¶
There are quite a few flags, all of which are covered in the FreeBSD man page for netstat(1), reproduced in Table Route Table Flags and Meanings with some modifications.
Letter |
Flag |
Meaning |
|---|---|---|
1 |
RTF_PROTO1 |
Protocol specific routing flag #1 |
2 |
RTF_PROTO2 |
Protocol specific routing flag #2 |
3 |
RTF_PROTO3 |
Protocol specific routing flag #3 |
B |
RTF_BLACKHOLE |
Discard packets during updates |
b |
RTF_BROADCAST |
Represents a broadcast address |
D |
RTF_DYNAMIC |
Created dynamically by redirect |
G |
RTF_GATEWAY |
Destination requires forwarding by intermediary |
H |
RTF_HOST |
Host entry (net otherwise) |
L |
RTF_LLINFO |
Valid protocol to link address translation |
M |
RTF_MODIFIED |
Modified dynamically (by redirect) |
R |
RTF_REJECT |
Host or net unreachable |
S |
RTF_STATIC |
Manually added |
U |
RTF_UP |
Route usable |
X |
RTF_XRESOLVE |
External daemon translates proto to link address |
For example, a route flagged as UGS is a usable route, packets are sent via
the gateway listed, and it is a static route.
Refs¶
This column counts the current number of active uses of a given route.
Use¶
This counter is the total number of packets sent via this route. This is helpful for determining if a route is actually being used, as it will continually increment as packets utilize the route.
Netif¶
The network interface used for this route.
Expire¶
For dynamic entries, this field shows how long until this route expires if it is not used again.
Using traceroute¶
Traceroute is a useful tool for testing and verifying routes and multi-WAN
functionality, among other uses. It shows each “hop” along the path a packet
travels from one end to the other, along with the latency encountered in
reaching that intermediate point. On pfSense software, a traceroute can be
performed by navigating to Diagnostics > Traceroute, or by using
traceroute at the command line. From clients running Windows, the program is
available under the name tracert.
Every IP packet contains a time-to-live (TTL) value. When a router passes a packet, it decrements the TTL by one. When a router receives a packet with a TTL of 1 and the destination is not a locally attached network, the router returns an ICMP error message “Time-to-live exceeded” and drops the packet. This is to limit the impact of routing loops, which otherwise would cause each packet to loop indefinitely.
Traceroute uses this TTL to its advantage to map the path to a specific network destination. It starts by sending the first packet with a TTL of 1. The first router (usually the default gateway) will send back an ICMP time-to-live exceeded error. The time between sending the packet and receiving the ICMP error is the time displayed, listed along with the IP address that sent the error and its reverse DNS, if any. After sending three packets with a TTL of 1 and displaying their response times, it will increment the TTL to 2 and send three more packets, noting the same information for the second hop. Traceroute increments the TTL and repeats the process until it reaches the specified destination, or exceeds the maximum number of hops.
Traceroute functions slightly differently on Windows and Unix-like operating systems (BSD, Linux, Mac OS X, Unix, etc.). Windows uses ICMP echo request packets (pings) while Unix-like systems use UDP packets by default. ICMP and UDP are layer 4 protocols, and traceroute is done at layer 3, so the protocol used is largely irrelevant except when considering a policy routing configuration. Traceroute from Windows clients will be policy routed based on which rule permits ICMP echo requests, while Unix-like clients will be routed by the rule matching the UDP ports in use.
In this example, traceroute is used to view the route to www.google.com:
# traceroute www.google.com
traceroute: Warning: www.google.com has multiple addresses; using 74.125.95.99
traceroute to www.l.google.com (74.125.95.99), 64 hops max, 40 byte packets
1 core (172.17.23.1) 1.450 ms 1.901 ms 2.213 ms
2 172.17.25.21 (172.17.25.21) 4.852 ms 3.698 ms 3.120 ms
3 bb1-g4-0-2.ipltin.ameritech.net (151.164.42.156) 3.275 ms 3.210 ms 3.215 ms
4 151.164.93.49 (151.164.93.49) 8.791 ms 8.593 ms 8.891 ms
5 74.125.48.117 (74.125.48.117) 8.460 ms 39.941 ms 8.551 ms
6 209.85.254.120 (209.85.254.120) 10.376 ms 8.904 ms 8.765 ms
7 209.85.241.22 (209.85.241.22) 19.479 ms 20.058 ms 19.550 ms
8 209.85.241.29 (209.85.241.29) 20.547 ms 19.761 ms
209.85.241.27 (209.85.241.27) 20.131 ms
9 209.85.240.49 (209.85.240.49) 30.184 ms
72.14.239.189 (72.14.239.189) 21.337 ms 21.756 ms
10 iw-in-f99.google.com (74.125.95.99) 19.793 ms 19.665 ms 20.603 ms
The output shows that it took 10 hops to get there, and the latency generally increased with each hop, which is expected.
Note
When utilizing policy routing, such as with Multi-WAN, the firewall itself may not appear as a hop in traceroute. When policy routing is employed, pf does not decrement the TTL when forwarding packets, so traceroute cannot detect it as an intermediate router.
Routes and VPNs¶
Depending on the VPN being used, a route may not display in the table for the far side. IPsec does not use the routing table, it is instead handled internally in the kernel using IPsec security policy database (SPD) entries. Static routes will never cause traffic to be directed across an IPsec connection. OpenVPN uses the system routing table and as such entries are present for networks reachable via an OpenVPN tunnel, as in the following example:
#netstat -rWn
Routing tables
Internet:
Destination Gateway Flags Use Mtu Netif Expire
default 198.51.100.1 UGS 92421 1500 em0
10.6.0.0/16 10.6.203.1 UGS 0 1500 ovpnc2
10.6.203.0/24 10.6.203.2 UGS 0 1500 ovpnc2
10.6.203.1 link#9 UH 0 1500 ovpnc2
10.6.203.2 link#9 UHS 0 16384 lo0
10.7.0.0/24 link#2 U 1260771 1500 em1
10.7.0.1 link#2 UHS 0 16384 lo0
127.0.0.1 link#7 UH 866 16384 lo0
198.51.100.0/24 link#1 U 1251477 1500 em0
198.51.100.7 link#1 UHS 0 16384 lo0
The OpenVPN interface is 10.6.203.2, with a gateway of 10.6.203.1 and
the interface is ovpnc2. The network reachable using OpenVPN in this example
is 10.6.0.0/16.
With IPsec, traceroute is not as useful as with routed setups like OpenVPN, because the IPsec tunnel itself does not have IP addresses. When running traceroute to a destination across IPsec, a timeout will be shown for the hop that is the IPsec tunnel.