Tip

This is the documentation for the 20.02 version. Looking for the documentation of the latest version? Have a look here.

Troubleshooting

This section contains commonly encountered issues with TNSR and methods to resolve them.

Ping and traceroute do not function without host OS default route

Utilities such as ping and traceroute will send traffic using the host OS routing table by default unless a specific source address is passed to the command. See Diagnostic Routing Behavior for details.

Unrecognized routes in a routing table

TNSR automatically populates routing tables with necessary entries that may not appear to directly correspond with manually configured addresses. See Common Routes for details.

Services do not receive traffic on an interface with NAT enabled

When NAT is enabled, by default TNSR will drop traffic that doesn’t match an existing NAT session or static NAT rule. This includes traffic for services on TNSR such as IPsec and BGP. To allow this traffic, see NAT Forwarding.

NAT session limits / “Create NAT session failed” error

The default limit for NAT sessions per IP address in the dataplane is 10240. If the number of sessions from a client IP address, including TNSR itself, exceeds that value, then new connections will fail. This value can be changed by using the dataplane nat max-translations-per-user command as described in Advanced Dataplane Configuration: NAT.

ACL rules do not match NAT traffic as expected

When NAT is active, ACL rules are always processed before NAT on interfaces where NAT is applied, in any direction. This behavior is different from some other products, such as pfSense. See ACL and NAT Interaction for details.

Some Traffic to the host OS management interface is dropped

TNSR includes a default set of Netfilter rules which secure the management interface. Only certain ports are allowed by default. See Default Allowed Traffic for details. To allow more traffic, create host ACLs as described in Host ACLs.

Locked out by NACM Rules

If TNSR access is lost due to the NACM configuration, access can be regained by following the directions in Regaining Access if Locked Out by NACM.

How to gain access to the root account

By default, the root account has interactive login disabled, which is the best practice. This can be changed by resetting the root password using sudo from another administrator account, or in the ISO installer. See Default Accounts and Passwords for details.

Console Messages Obscure Prompts

When connected to the console of a TNSR device, such as the serial console, the kernel may output messages to the terminal which obscure prompts or other areas of the screen. This is normal and an expected effect when using the console directly.

To work around this intended behavior, use one of the following methods:

  • Press Ctrl-L to clear or redraw the screen without the messages.

  • Press Enter to receive a new prompt.

  • Run sudo dmesg -D from a shell prompt or with the TNSR shell command, which will disable kernel output to all consoles.

  • Connect to the TNSR device using SSH instead of the console.

OSPF Neighbors Stuck in ExStart State

When attempting to form an adjacency between two OSPF (Open Shortest Path First v2 (OSPF)) neighbors, if the neighbor status appears to be stuck in the ExStart state, the most likely cause is an MTU mismatch between the routers.

To solve this problem, adjust the MTU values of the interfaces actively participating in OSPF on all routers to match. If this is not possible, try using the mtu-ignore option on active OSPF interfaces.

Large packets fail to pass over IPsec

Encapsulated packets which are larger than the default-data-size buffer will be dropped by the dataplane. On older versions of TNSR this may even result in a dataplane crash. The size of this buffer is 2048 by default, which will pass packets up to approximately 2000 bytes. Since the default MTU is 1500, this issue is not apparent in many cases. However, when attempting to pass jumbo frames over IPsec, this becomes a problem. To pass 9000 byte frames over IPsec, increase the buffer size to 16384 and restart the dataplane.

tnsr(config)# dataplane buffers default-data-size 16384
tnsr(config)# service dataplane restart

IPsec packets are dropped or fail to pass with QAT enabled

There is a known incompatibility between QAT and VT-d on some platforms which can prevent IPsec traffic from passing when QAT acceleration is enabled. See Disable VT-d in the BIOS for details.

Console DMA / PTE Read access Error Messages

Errors similar to the following may appear on the console:

[110772.063766] DMAR: [DMA Read] Request device [04:01.0] fault addr 406482000
   [fault reason 06] PTE Read access is not set
[110773.059440] DMAR: DRHD: handling fault status reg 102

The cause is likely an incompatibility between an enabled QAT device and VT-d in the BIOS. See Disable VT-d in the BIOS for details.

Diagnosing Service Issues

If a service will not stay running and the logs indicate that it is crashing, additional debugging information can be obtained from core dumps.

By default, core dumps are disabled for services. These can be individually enabled as needed by the following command:

tnsr(config)# service (backend|bgp|dataplane|dhcp|http|ike|ntp|restconf|unbound)
                 coredump (enable|disable)

The resulting core files will be written under /var/lib/systemd/coredump/.

Debugging TNSR

The following commands enable debugging information in various aspects of TNSR. These should only be used under direction of Netgate.

debug cli [level <n>]

Enable debugging in clixon and cligen at the given level.

debug tnsr (clear|set|value) <flags>

Enable debugging in TNSR. The set or clear command may be repeated multiple times to add or remove individual flag values. The value command may be used to directly set the value. The <flags> value is the logical or of all desired debugging flags.

The following flag values are available:

Flag

Value

TDBG_NONE

0x00000000

TDBG_FRR

0x00000001

TDBG_HOST

0x00000002

TDBG_KEA

0x00000004

TDBG_VPP

0x00000008

TDBG_NTP

0x00000010

TDBG_STRONGSWAN

0x00000020

TDBG_UNBOUND

0x00000040

TDBG_HTTP

0x00000080

TDBG_DELAYED_NODE

0x00001000

TDBG_DEP_GRAPH

0x00002000

TDBG_TRANSACTION

0x00004000

TDBG_ACL

0x00010000

TDBG_BGP

0x00020000

TDBG_BRIDGE

0x00040000

TDBG_INTF

0x00080000

TDBG_NEIGHBOR

0x00100000

TDBG_SUBIF

0x00200000

TDBG_SYSCTL

0x00400000

TDBG_GRE

0x00800000

TDBG_LOOPBACK

0x01000000

TDBG_ROUTE

0x02000000

TDBG_SPAN

0x04000000

TDBG_MAP

0x08000000

debug vmgmt (clear|set|value) <flags>

Enable VPP Mgmt library debug. The set or clear command may be repeated multiple times to add or remove individual flag values. The value command may be used to directly set the value. The <flags> value is the logical or of all desired debugging flags.

The following flag values are available:

Flag

Value

VDBG_NONE

0x0000

VDBG_API_SETUP

0x0001

VDBG_API_MSG

0x0002

VDBG_ACL

0x0004

VDBG_BRIDGE

0x0008

VDBG_INTF

0x0010

VDBG_NAT

0x0020

VDBG_TAP

0x0040

VDBG_MEMIF

0x0080

VDBG_LLDP

0x0100

VDBG_GRE

0x0200

VDBG_MAP

0x0400

VDBG_ROUTE

0x0800

no debug (cli|tnsr|vmgmt)

Removes all debugging.

Diagnostic Information for Support

When contacting Netgate for support at https://go.netgate.com, support representatives may require information about the configuration and state of a TNSR installation. TNSR includes a utilty, tnsr-diag, which gathers diagnostic information automatically. The archive it produces can then be included in communications with Netgate support.

The command to collect this data may be run from a shell using sudo tnsr-diag, or from within the TNSR CLI using shell sudo tnsr-diag.

For example:

tnsr# shell sudo tnsr-diag
Collecting...
Diagnostic data stored at /tmp/tnsr-diag-2020-02-13-161340.zip

As shown in the example output above, the data is stored in a compressed archive in /tmp named tnsr-diag-<timestamp>.zip. This file may be copied from the TNSR device using scp.

Warning

The archive generated by this utility may contain sensitive information about the router configuration and environment. Review its contents before transmitting the information, and always use a secure communications method.