Using tcpdump on the command line¶
The tcpdump program is a command line packet capture utility provided with
most UNIX and UNIX-like operating system distributions, including FreeBSD. It is
included in pfSense® software and is usable from a shell on the console or over
SSH.
The tcpdump program is an exceptionally powerful tool, but that also makes
it daunting to the uninitiated user. The tcpdump binary in FreeBSD supports
over 50 different command line flags, limitless possibilities with filter
expressions, and its man page, providing only a brief overview of all its
options, is nearly 1200 lines long and 67k.
After learning to use tcpdump, knowledge of how to interpret the data
it provides is also necessary, which can require an in-depth understanding of
networking protocols.
This section is intended to provide an introduction to this topic and leave the reader with enough knowledge for basic troubleshooting. A comprehensive review of packet capturing and interpretation of the results is outside the scope of this documentation.
See also
For those with a thirst for more than basic knowledge in this area, see Additional References for more resources.
Common tcpdump flags¶
The following table shows the most common command line flags for tcpdump.
This section contains information on each of these flags.
Flag |
Description |
|---|---|
|
Listen on |
|
Do not perform reverse DNS resolution on IP addresses |
|
Save capture in pcap format to |
|
Snap length: Amount of data to be captured from each frame |
|
Exit after receiving a specific number of packets |
|
Do not put the interface in promiscuous mode |
|
Verbose output |
|
Print link-layer header on each line |
-i flag¶
The -i flag specifies the interface on which tcpdump will listen. Use
FreeBSD interface names here, such as igb0, em0, vmx0, etc.
-n flag¶
Do not resolve IP addresses using reverse DNS. When this option is not
specified, tcpdump will perform a reverse DNS (PTR) lookup for each IP
address. This generates a significant amount of DNS traffic in captures
displaying large volumes of traffic. Disable this to avoid adding load to DNS
servers.
The best practice is to always use -n because it eliminates the delay caused
by performing the reverse lookup between when tcpdump captures a packet and
when it can display the content. Also, IP addresses are typically easier to read
and understand than their PTR records. That is a matter of personal preference,
though, and in familiar environments where the PTR records are known to provide
the actual host names of the devices, captures may be run without -n to show
the hostnames.
Another reason to use -n, is to be “sneaky.” One means of detecting packet
capturing is looking for spikes and patterns in DNS PTR lookups. Skipping the
DNS lookup will not cause any extra traffic to be generated in the process.
-w flag¶
tcpdump can save capture files in pcap format for later analysis or analysis
on another system. This is commonly done from command line only devices like
those running pfSense software so the file can be copied to a host running
Wireshark or another graphical network protocol analyzer and reviewed there.
When saving to a file using -w, the frames will not be displayed in the
terminal as they otherwise are.
See also
See Using Wireshark for more information about using Wireshark with pfSense software.
-s flag¶
By default tcpdump only saves the first 64 bytes of each frame when
capturing to a file. This is enough to contain the IP and protocol header for
most protocols, but limits the usability of capture files. By using the -s
flag, tcpdump can be told how much of the frame to capture, in bytes. This
is called the snap length.
Flag |
Description |
|---|---|
|
Capture the first 500 bytes of each frame |
|
Capture each frame in its entirety |
In most cases, using -s 0 is the best practice when capturing to a file for
analysis on another system. The only exception to this is scenarios where a
significant amount of traffic must be captured over a longer period of time. If
the information being sought is known to be in the header, the default 64
bytes of each frame may be used to get the required information while
significantly reducing the size of the resulting capture file.
-c flag¶
To capture a certain number of frames and then exit, use the -c flag.
Example usage: tcpdump will exit after capturing 100 frames by specifying
-c 100.
-p flag¶
Normally when capturing traffic with tcpdump, it puts the network interface
into promiscuous mode. When not running in promiscuous mode, the interface only
receives frames destined for its own MAC address as well as broadcast and
multicast addresses. When switched into promiscuous mode, the interface shows
every frame on the wire that arrives at the network interface. In a switched
network, this generally has little impact on the capture. In networks where the
device is connected to a vswitch also in promiscuous mode, or a hub, using
-p can significantly limit noise in the capture when the only traffic of
interest is to and from the system performing the capture.
-v flag¶
The -v flag controls the detail, or verbosity, of the output. Using more
v options yields more detail, so use -v, -vv, or -vvv to view
even more detail in the output printed to the console. This option does not
affect the detail stored in a capture file when using the -w switch, but
will instead cause the process to report the number of packets captured every
10 seconds.
-e flag¶
Normally tcpdump does not show any link layer information. Specify -e to
display the source and destination MAC addresses, and VLAN tag information for
any traffic tagged with 802.1q VLANs.
Example capture without -e¶
This capture shows the default output, containing no link layer information:
# tcpdump -ni igb1 -c 5
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on igb1, link-type EN10MB (Ethernet), capture size 96 bytes
23:18:15.830706 IP 10.0.64.210.22 > 10.0.64.15.1395: P 2023587125:2023587241(116)
ack 2091089207 win 65535
23:18:15.830851 IP 10.0.64.210.22 > 10.0.64.15.1395: P 116:232(116) ack 1 win 65535
23:18:15.831256 IP 10.0.64.15.1395 > 10.0.64.210.22: . ack 116 win 65299
23:18:15.839834 IP 10.0.64.3 > 224.0.0.18: VRRPv2, Advertisement, vrid 4, prio 0,
authtype none, intvl 1s, length 36
23:18:16.006407 IP 10.0.64.15.1395 > 10.0.64.210.22: . ack 232 win 65183
5 packets captured
Example capture using -e¶
Here the link layer information is included by using -e. Note the source and
destination MAC addresses in addition to the source and destination IP
addresses:
# tcpdump -ni igb1 -e -c 5
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on igb1, link-type EN10MB (Ethernet), capture size 96 bytes
23:30:05.914958 00:0c:29:0b:c3:ed > 00:13:d4:f7:73:d2, ethertype IPv4 (0x0800), length 170:
10.0.64.210.22 > 10.0.64.15.1395: P 2023592509:2023592625(116) ack 2091091355 win 65535
23:30:05.915110 00:0c:29:0b:c3:ed > 00:13:d4:f7:73:d2, ethertype IPv4 (0x0800), length 170:
10.0.64.210.22 > 10.0.64.15.1395: P 116:232(116) ack 1 win 65535
23:30:05.915396 00:13:d4:f7:73:d2 > 00:0c:29:0b:c3:ed, ethertype IPv4 (0x0800), length 60:
10.0.64.15.1395 > 10.0.64.210.22: . ack 116 win 65299
23:30:05.973359 00:00:5e:00:01:04 > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 70:
10.0.64.3 > 224.0.0.18: VRRPv2, Advertisement, vrid 4, prio 0, authtype none, intvl 1s,
length 36
23:30:06.065200 00:13:d4:f7:73:d2 > 00:0c:29:0b:c3:ed, ethertype IPv4 (0x0800), length 60:
10.0.64.15.1395 > 10.0.64.210.22: . ack 232 win 65183
5 packets captured
tcpdump Filters¶
Running tcpdump without filters on most firewalls will produce so much
output that it is extremely difficult to find traffic of interest. There are
numerous filtering expressions available that limit traffic display and capture.
Host filters¶
To filter for a specific host, append host and the IP address to the
tcpdump command.
To filter for host 192.168.1.100 use the following command:
# tcpdump -ni igb1 host 192.168.1.100
The previous command captures all traffic to and from the given host. To only
capture traffic being initiated by that host, use the src directive:
# tcpdump -ni igb1 src host 192.168.1.100
Similarly, filtering for traffic destined to that IP address is possible by
specifying dst:
# tcpdump -ni igb1 dst host 192.168.1.100
Network filters¶
Network filters narrow the capture to a specific subnet using the net
expression. Following net, specify a CIDR-masked network
(192.168.1.0/24), dotted quad ( 192.168.1.1), dotted triple
(192.168.1), dotted pair ( 192.168) or simply a number ( 192). A
dotted quad is equivalent to specifying host, a dotted triple uses a subnet
mask of 255.255.255.0, a dotted pair uses 255.255.0.0, and a number
alone uses 255.0.0.0.
The best practice for filtering by network is to use a CIDR masked subnet prefix
specification as an argument to net:
# tcpdump -ni igb1 src net 172.16.0.0/12
Alternately, omit parts of an address to use the assumed masks mentioned previously.
The following command displays traffic to or from any host with a
192.168.1.x IP address:
# tcpdump -ni igb1 net 192.168.1
The next command will capture traffic to or from any host with a 10.x.x.x IP
address:
# tcpdump -ni igb1 net 10
Those examples will capture all traffic to or from the specified network. The
src or dst keywords may be used the same as with host filters to
capture only traffic initiated by or destined to the specified network:
# tcpdump -ni igb1 src net 10
Protocol and port filters¶
Narrowing down by host or network can be inadequate to eliminate unnecessary traffic from a capture. Or the source or destination of traffic may not be significant, and all traffic of a certain type should be captured. In other cases, filtering out all traffic of a specific type can reduce noise.
TCP and UDP port filters¶
To filter on TCP and UDP ports, use the port directive. This captures both
TCP and UDP traffic using the specified port either as a source or destination
port. It can be combined with tcp or udp to specify the protocol, and
src or dst to specify a source or destination port.
Capture all HTTP traffic¶
# tcpdump -ni igb1 tcp port 80
Capture all DNS traffic¶
Capture all DNS traffic (Queries can use both UDP and TCP):
# tcpdump -ni igb1 port 53
Protocol filters¶
Specific protocols can be filtered using the proto directive or by using the
protocol name directly. Parameters passed to the proto directive can be
specified using the IP protocol number or one of the names icmp, igmp,
igrp, pim, ah, esp, carp, vrrp, udp, or tcp.
Because the normal protocol names are reserved words, they must be escaped with
one or two backslashes when used with the proto directive, depending on the
shell. The default shell available in pfSense software requires two backslashes
to escape these protocol names. If the command returns a syntax error, check
that the protocol name is properly escaped.
The following capture will show all ICMP traffic on the igb1 interface:
# tcpdump -ni igb1 proto \\icmp
Specifying carp for the protocol will capture CARP traffic but it also needs
-T carp in order to interpret the CARP packets correctly when viewing the
output using tcpdump. The GUI makes this adjustment automatically when
capturing CARP.
The following capture will show all CARP traffic on the igb1 interface,
which can be useful to ensure CARP traffic is being sent and received on the
specified interface. It also omits the proto keyword, showing that it works
on its own:
# tcpdump -i igb1 -T carp carp
Negating a filter match¶
In addition to matching specific parameters, a filter match can be negated by
specifying not in front of the filter expression. When troubleshooting
something other than CARP, and its multicast heartbeats are cluttering the
capture output, exclude it as follows:
# tcpdump -ni igb1 not proto \\carp
Combining filters¶
Any of the aforementioned filters can be combined using and or or. The
following sections provide some examples.
Display all HTTP traffic to and from a host¶
Display all HTTP traffic to or from 192.168.1.11:
# tcpdump -ni igb1 host 192.168.1.11 and tcp port 80
Display all HTTP traffic to and from multiple hosts¶
Display all HTTP traffic from either 192.168.1.11 or 192.168.1.15:
# tcpdump -ni igb1 host 192.168.1.11 or host 192.168.1.15 and tcp port 80
Filter expression usage¶
Filter expressions must come after every command line flag used. Adding any flags after a filter expression will result in a syntax error.
Incorrect ordering¶
# tcpdump -ni igb1 -T carp carp -c 2
tcpdump: syntax error
Correct ordering¶
# tcpdump -ni igb1 -T carp -c 2 carp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on igb1, link-type EN10MB (Ethernet), capture size 65535 bytes
14:50:07.426993 IP 198.51.100.12 > 224.0.0.18: CARPv2-advertise 36: vhid=11 advbase=1
advskew=0 authlen=7 counter=5449924379588860810
14:50:08.436849 IP 198.51.100.12 > 224.0.0.18: CARPv2-advertise 36: vhid=11 advbase=1
advskew=0 authlen=7 counter=5449924379588860810
2 packets captured
78 packets received by filter
0 packets dropped by kernel
More on Filters¶
This section covered the most commonly used tcpdump filter expressions, and
probably covers all the syntax most users will need. However this barely
scratches the surface of the possibilities. There are many documents on the web
that cover tcpdump in general and filtering specifically. See
Additional References at the end of this chapter for
links to more resources.
Practical Troubleshooting Examples¶
This section details best practice approaches for troubleshooting a few specific problems. There are multiple ways to approach any problem, but packet capturing can rarely be beat for its effectiveness. Examining the traffic on the wire provides a level of visibility into what is actually happening on the network
Port forward not working¶
In this example, a new port forward is failing to respond to a request from a host on the Internet. The troubleshooting steps outlined in Troubleshooting NAT Port Forwards offers one way to approach this, but sometimes packet capturing is the only or easiest way to find the source of the problem.
Start from WAN¶
First, make sure the traffic is getting to the WAN interface. Start a
tcpdump session on the WAN interface, and watch for the traffic:
# tcpdump -ni igb1 tcp port 5900
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on igb1, link-type EN10MB (Ethernet), capture size 96 bytes
11:14:02.444006 IP 172.17.11.9.37219 > 10.0.73.5.5900: S 3863112259:3863112259(0)
win 65535 <mss 1260,nop,nop,sackOK>
In this case, a packet comes in from the WAN, so it is making it that far. Note
that the first part of the TCP handshake, a packet with only SYN set (the S
shown), is reaching the firewall. If the port forward was working, a SYN ACK
(S.) packet would be shown in reply to the SYN. With no return traffic
visible, it could be a firewall rule or the target system may be unreachable –
turned off, not listening on the specified port, host firewall blocking the
traffic, etc.
Check Internal Interface¶
The next step would be to run a tcpdump session on the internal interface
associated with the port forward:
# tcpdump -ni igb0 tcp port 5900
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on igb0, link-type EN10MB (Ethernet), capture size 96 bytes
11:14:38.339926 IP 172.17.11.9.2302 > 192.168.30.5.5900: S 1481321921:1481321921(0)
win 65535 <mss 1260,nop,nop,sackOK>
Looking at the internal traffic, the connection left the inside interface and the local IP address was translated correctly. If this local address matches what was expected, then both the port forward and the firewall rule are working properly, and connectivity to the local PC must be confirmed by other means. If no output was displayed, then there is a problem with the firewall rule or the port forward may have been incorrectly defined. For this example, the target system was unplugged.
IPsec tunnel will not connect¶
tcpdump has some awareness of the protocols being used, which can be very
helpful in figuring out problems with IPsec tunnels. The next few examples will
show how certain error conditions may present themselves when monitoring with
tcpdump. The IPsec logs are usually more helpful, but this can confirm what
is actually being seen by the firewall. For encrypted traffic such as IPsec,
packet capturing of the traffic is of less value as the payload of the captured
packets cannot be examined without additional parameters, but it is helpful to
determine if traffic from the remote end is reaching the firewall and which
phases complete.
This first tunnel has an unreachable peer:
# tcpdump -ni igb1 host 192.168.10.6
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on igb1, link-type EN10MB (Ethernet), capture size 96 bytes
19:11:11.542976 IP 192.168.10.5.500 > 192.168.10.6.500: isakmp: phase 1 I agg
19:11:21.544644 IP 192.168.10.5.500 > 192.168.10.6.500: isakmp: phase 1 I agg
This tunnel attempt has a mismatched PSK, notice how it attempts to move to phase 2, but then stops:
# tcpdump -ni igb1 host 192.168.10.6
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on igb1, link-type EN10MB (Ethernet), capture size 96 bytes
19:15:05.566352 IP 192.168.10.5.500 > 192.168.10.6.500: isakmp: phase 1 I agg
19:15:05.623288 IP 192.168.10.6.500 > 192.168.10.5.500: isakmp: phase 1 R agg
19:15:05.653504 IP 192.168.10.5.500 > 192.168.10.6.500: isakmp: phase 2/others I inf[E]
Now Phase 1 is OK but there is a mismatch in the Phase 2 information. It will repeatedly attempt phase 2 traffic but there will not be any traffic in the tunnel:
# tcpdump -ni igb1 host 192.168.10.6
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on igb1, link-type EN10MB (Ethernet), capture size 96 bytes
19:17:18.447952 IP 192.168.10.5.500 > 192.168.10.6.500: isakmp: phase 1 I agg
19:17:18.490278 IP 192.168.10.6.500 > 192.168.10.5.500: isakmp: phase 1 R agg
19:17:18.520149 IP 192.168.10.5.500 > 192.168.10.6.500: isakmp: phase 1 I agg
19:17:18.520761 IP 192.168.10.6.500 > 192.168.10.5.500: isakmp: phase 2/others R inf[E]
19:17:18.525474 IP 192.168.10.5.500 > 192.168.10.6.500: isakmp: phase 2/others I inf[E]
19:17:19.527962 IP 192.168.10.5.500 > 192.168.10.6.500: isakmp: phase 2/others I oakley-quick[E]
Finally, a fully working tunnel with two-way traffic after Phase 1 and Phase 2 have completed!:
# tcpdump -ni igb1 host 192.168.10.6
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on igb1, link-type EN10MB (Ethernet), capture size 96 bytes
21:50:11.238263 IP 192.168.10.5.500 > 192.168.10.6.500: isakmp: phase 1 I agg
21:50:11.713364 IP 192.168.10.6.500 > 192.168.10.5.500: isakmp: phase 1 R agg
21:50:11.799162 IP 192.168.10.5.500 > 192.168.10.6.500: isakmp: phase 1 I agg
21:50:11.801706 IP 192.168.10.5.500 > 192.168.10.6.500: isakmp: phase 2/others I inf[E]
21:50:11.812809 IP 192.168.10.6.500 > 192.168.10.5.500: isakmp: phase 2/others R inf[E]
21:50:12.820191 IP 192.168.10.5.500 > 192.168.10.6.500: isakmp: phase 2/others I oakley-quick[E]
21:50:12.836478 IP 192.168.10.6.500 > 192.168.10.5.500: isakmp: phase 2/others R oakley-quick[E]
21:50:12.838499 IP 192.168.10.5.500 > 192.168.10.6.500: isakmp: phase 2/others I oakley-quick[E]
21:50:13.168425 IP 192.168.10.5 > 192.168.10.6: ESP(spi=0x09bf945f,seq=0x1), length 132
21:50:13.171227 IP 192.168.10.6 > 192.168.10.5: ESP(spi=0x0a6f9257,seq=0x1), length 132
21:50:14.178820 IP 192.168.10.5 > 192.168.10.6: ESP(spi=0x09bf945f,seq=0x2), length 132
21:50:14.181210 IP 192.168.10.6 > 192.168.10.5: ESP(spi=0x0a6f9257,seq=0x2), length 132
21:50:15.189349 IP 192.168.10.5 > 192.168.10.6: ESP(spi=0x09bf945f,seq=0x3), length 132
21:50:15.191756 IP 192.168.10.6 > 192.168.10.5: ESP(spi=0x0a6f9257,seq=0x3), length 132
Traffic traversing an IPsec tunnel¶
Traffic can also be observed traversing IPsec tunnels by capturing on the
enc0 interface. This can help determine if traffic is attempting to reach
the far end by using the tunnel. All traffic for all IPsec tunnels appears on
the enc0 interface.
In the following example, a host on one side of the tunnel is successfully sending an ICMP echo request (ping) to the far side, and receiving replies:
# tcpdump -ni enc0
tcpdump: WARNING: enc0: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enc0, link-type ENC (OpenBSD encapsulated IP), capture size 65535 bytes
15:52:46.151098 (authentic,confidential): SPI 0xcd77e085: IP 10.3.0.1 > 10.7.0.1:
ICMP echo request, id 44640, seq 0, length 64
15:52:46.151814 (authentic,confidential): SPI 0xc0afb14d: IP 10.7.0.1 > 10.3.0.1:
ICMP echo reply, id 44640, seq 0, length 64
15:52:47.154243 (authentic,confidential): SPI 0xcd77e085: IP 10.3.0.1 > 10.7.0.1:
ICMP echo request, id 44640, seq 1, length 64
15:52:47.154843 (authentic,confidential): SPI 0xc0afb14d: IP 10.7.0.1 > 10.3.0.1:
ICMP echo reply, id 44640, seq 1, length 64
If traffic was not properly entering the tunnel, no output would be shown. If there is a firewall or internal routing issue on the far side, traffic will appear leaving but nothing will show returning.
Troubleshooting Outbound NAT¶
For complex environments where Manual Outbound NAT is needed, tcpdump can be
of great assistance in troubleshooting the Outbound NAT configuration. One good
capture to use is to look for traffic with private IP addresses on the WAN
interface, as everything on WAN should be have NAT applied and appear to be a
public IP address. The following capture will display any traffic with RFC 1918
IP addresses as the source or destination. This will show any traffic that is
not matching one of the outbound NAT rules, providing information to help review
the Outbound NAT configuration to find the problem:
# tcpdump -ni igb1 net 10 or net 192.168 or net 172.16.0.0/12