Tip

This is the documentation for the 20.02 version. Looking for the documentation of the latest version? Have a look here.

Troubleshooting¶

This section contains commonly encountered issues with TNSR and methods to resolve them.

Ping and traceroute do not function without host OS default route¶

Utilities such as ping and traceroute will send traffic using the host OS routing table by default unless a specific source address is passed to the command. See Diagnostic Routing Behavior for details.

Unrecognized routes in a routing table¶

TNSR automatically populates routing tables with necessary entries that may not appear to directly correspond with manually configured addresses. See Common Routes for details.

Services do not receive traffic on an interface with NAT enabled¶

When NAT is enabled, by default TNSR will drop traffic that doesn’t match an existing NAT session or static NAT rule. This includes traffic for services on TNSR such as IPsec and BGP. To allow this traffic, see NAT Forwarding.

NAT session limits / “Create NAT session failed” error¶

The default limit for NAT sessions per IP address in the dataplane is 10240. If the number of sessions from a client IP address, including TNSR itself, exceeds that value, then new connections will fail. This value can be changed by using the dataplane nat max-translations-per-user command as described in Advanced Dataplane Configuration: NAT.

ACL rules do not match NAT traffic as expected¶

When NAT is active, ACL rules are always processed before NAT on interfaces where NAT is applied, in any direction. This behavior is different from some other products, such as pfSense. See ACL and NAT Interaction for details.

Some Traffic to the host OS management interface is dropped¶

TNSR includes a default set of Netfilter rules which secure the management interface. Only certain ports are allowed by default. See Default Allowed Traffic for details. To allow more traffic, create host ACLs as described in Host ACLs.

Locked out by NACM Rules¶

If TNSR access is lost due to the NACM configuration, access can be regained by following the directions in Regaining Access if Locked Out by NACM.

How to gain access to the root account¶

By default, the root account has interactive login disabled, which is the best practice. This can be changed by resetting the root password using sudo from another administrator account, or in the ISO installer. See Default Accounts and Passwords for details.

Console Messages Obscure Prompts¶

When connected to the console of a TNSR device, such as the serial console, the kernel may output messages to the terminal which obscure prompts or other areas of the screen. This is normal and an expected effect when using the console directly.

To work around this intended behavior, use one of the following methods:

Press Ctrl-L to clear or redraw the screen without the messages.
Press Enter to receive a new prompt.
Run sudo dmesg -D from a shell prompt or with the TNSR shell command, which will disable kernel output to all consoles.
Connect to the TNSR device using SSH instead of the console.

OSPF Neighbors Stuck in ExStart State¶

When attempting to form an adjacency between two OSPF (Open Shortest Path First v2 (OSPF)) neighbors, if the neighbor status appears to be stuck in the ExStart state, the most likely cause is an MTU mismatch between the routers.

To solve this problem, adjust the MTU values of the interfaces actively participating in OSPF on all routers to match. If this is not possible, try using the mtu-ignore option on active OSPF interfaces.

Large packets fail to pass over IPsec¶

Encapsulated packets which are larger than the default-data-size buffer will be dropped by the dataplane. On older versions of TNSR this may even result in a dataplane crash. The size of this buffer is 2048 by default, which will pass packets up to approximately 2000 bytes. Since the default MTU is 1500, this issue is not apparent in many cases. However, when attempting to pass jumbo frames over IPsec, this becomes a problem. To pass 9000 byte frames over IPsec, increase the buffer size to 16384 and restart the dataplane.

tnsr(config)# dataplane buffers default-data-size 16384
tnsr(config)# service dataplane restart

IPsec packets are dropped or fail to pass with QAT enabled¶

There is a known incompatibility between QAT and VT-d on some platforms which can prevent IPsec traffic from passing when QAT acceleration is enabled. See Disable VT-d in the BIOS for details.

Console DMA / PTE Read access Error Messages¶

Errors similar to the following may appear on the console:

[110772.063766] DMAR: [DMA Read] Request device [04:01.0] fault addr 406482000
   [fault reason 06] PTE Read access is not set
[110773.059440] DMAR: DRHD: handling fault status reg 102

The cause is likely an incompatibility between an enabled QAT device and VT-d in the BIOS. See Disable VT-d in the BIOS for details.

Diagnosing Service Issues¶

If a service will not stay running and the logs indicate that it is crashing, additional debugging information can be obtained from core dumps.

By default, core dumps are disabled for services. These can be individually enabled as needed by the following command:

tnsr(config)# service (backend|bgp|dataplane|dhcp|http|ike|ntp|restconf|unbound)
                 coredump (enable|disable)

The resulting core files will be written under /var/lib/systemd/coredump/.

Debugging TNSR¶

The following commands enable debugging information in various aspects of TNSR. These should only be used under direction of Netgate.

debug cli [level <n>]:

Enable debugging in clixon and cligen at the given level.

debug tnsr (clear|set|value) <flags>:

Enable debugging in TNSR. The set or clear command may be repeated multiple times to add or remove individual flag values. The value command may be used to directly set the value. The <flags> value is the logical or of all desired debugging flags.

The following flag values are available:

Flag	Value
TDBG_NONE	0x00000000
TDBG_FRR	0x00000001
TDBG_HOST	0x00000002
TDBG_KEA	0x00000004
TDBG_VPP	0x00000008
TDBG_NTP	0x00000010
TDBG_STRONGSWAN	0x00000020
TDBG_UNBOUND	0x00000040
TDBG_HTTP	0x00000080
TDBG_DELAYED_NODE	0x00001000
TDBG_DEP_GRAPH	0x00002000
TDBG_TRANSACTION	0x00004000
TDBG_ACL	0x00010000
TDBG_BGP	0x00020000
TDBG_BRIDGE	0x00040000
TDBG_INTF	0x00080000
TDBG_NEIGHBOR	0x00100000
TDBG_SUBIF	0x00200000
TDBG_SYSCTL	0x00400000
TDBG_GRE	0x00800000
TDBG_LOOPBACK	0x01000000
TDBG_ROUTE	0x02000000
TDBG_SPAN	0x04000000
TDBG_MAP	0x08000000

debug vmgmt (clear|set|value) <flags>:

Enable VPP Mgmt library debug. The set or clear command may be repeated multiple times to add or remove individual flag values. The value command may be used to directly set the value. The <flags> value is the logical or of all desired debugging flags.

The following flag values are available:

Flag	Value
VDBG_NONE	0x0000
VDBG_API_SETUP	0x0001
VDBG_API_MSG	0x0002
VDBG_ACL	0x0004
VDBG_BRIDGE	0x0008
VDBG_INTF	0x0010
VDBG_NAT	0x0020
VDBG_TAP	0x0040
VDBG_MEMIF	0x0080
VDBG_LLDP	0x0100
VDBG_GRE	0x0200
VDBG_MAP	0x0400
VDBG_ROUTE	0x0800

no debug (cli|tnsr|vmgmt):

Removes all debugging.

Diagnostic Information for Support¶

When contacting Netgate for support at https://go.netgate.com, support representatives may require information about the configuration and state of a TNSR installation. TNSR includes a utilty, tnsr-diag, which gathers diagnostic information automatically. The archive it produces can then be included in communications with Netgate support.

The command to collect this data may be run from a shell using sudo tnsr-diag, or from within the TNSR CLI using shell sudo tnsr-diag.

For example:

tnsr# shell sudo tnsr-diag
Collecting...
Diagnostic data stored at /tmp/tnsr-diag-2020-02-13-161340.zip

As shown in the example output above, the data is stored in a compressed archive in /tmp named tnsr-diag-<timestamp>.zip. This file may be copied from the TNSR device using scp.

Warning

The archive generated by this utility may contain sensitive information about the router configuration and environment. Review its contents before transmitting the information, and always use a secure communications method.