Tip

This is the documentation for the 21.07 version. Looking for the documentation of the latest version? Have a look here.

Updating TNSR

Warning

TNSR Home+Lab installations cannot be updated. Reinstall with TNSR Business or install a new version of TNSR Home+Lab. The operating system may be updated, but not TNSR.

With a signed client certificate from Netgate in place, TNSR has access to the Netgate software repositories which contain important updates to TNSR. These updates can be retrieved using the package command in the TNSR CLI, or dnf in the host OS shell.

Note

Updating TNSR will also update the operating system. Even when there are no TNSR updates available, it is a good practice to periodically perform an update to obtain important operating system updates such as security vulnerability mitigations.

See also

Most of this document covers in-place updates. For information on updating by redeploying/reinstalling, see Upgrading by Redeploying TNSR.

Pre-Upgrade Tasks

Before updating TNSR, perform the following tasks:

  • Read through the Netgate TNSR Releases release notes for the new version to identify relevant changes in behavior which may require special actions before or after the upgrade

  • Make sure the signed certificate is in place (Install the certificate)

  • Make sure the TNSR instance has working Internet connectivity

  • Have installation media ready for the new version of TNSR software. Problems during an in-place update may require reinstallation of TNSR software.

  • Take a backup of the running and startup configurations, plus other important files such as the signed certificate and keys (Configuration Backups)

  • If TNSR is running as a virtual machine, take a snapshot

Tip

Though it is optional, best practices for updating include a pre-upgrade reboot. This reboot ensures that the hardware and installation are functional before attempting the upgrade. This can help identify potential hardware and other issues, such as storage failures, so they do not present themselves unexpectedly during the upgrade.

Updating to TNSR 20.08 from Previous Versions

TNSR 20.08 is based on CentOS 8, while previous versions of TNSR software were based on CentOS 7.x. Due to this major operating system change, updating to TNSR 20.08 requires special handling.

Warning

While Netgate has tested common update scenarios, updating in-place from a CentOS 7 base to CentOS 8 base may not work in all installations.

Installing TNSR 20.08 directly and then restoring the TNSR configuration data is a safer approach. However, that method requires physical access or equivalent out-of-band access and is potentially more time consuming. See Upgrading by Redeploying TNSR for details.

Before starting, update the current installation of TNSR software to version 20.02 as described in Updates via the TNSR CLI. Upgrading from previous versions of TNSR software has not been tested.

The following shell script will upgrade TNSR software installations in place from CentOS 7 to CentOS 8 and TNSR software version 20.08. Download the script to the router, inspect and verify its contents, then execute the script from the console (serial, video, IPMI/OOB access).

Warning

The script must not be run using an SSH session connected through TNSR, as TNSR services will be stopped and started during the upgrade, breaking connectivity between the client and TNSR.

Download: upgrade_centos_7_to_8.sh
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
#!/bin/sh

_fail() {
	echo "Upgrade process has failed!  Please contact support"
}

YUM="yum -y"
DNF="dnf -y"
CENTOS_RPM_SERVER="http://mirror.centos.org/centos/8/BaseOS/x86_64/os/Packages"
CENTOS_VERSION="8.2-2.2004.0.1"

if [ "$(rpm -q kernel | wc -l)" = "1" ]; then
	echo "Only one kernel found, upgrading current system"
	${YUM} upgrade
	echo "Reboot your system and run this script again"
	exit 0
fi

echo "This script will upgrade your TNSR instance to a new major"
echo "version based on CentOS 8.  This operation is irreversible."
echo "Make sure you backup your configuration files before start."
echo ""
read -p "Are you ready to proceed? (y/N) " -e -n 1 _proceed

if [ "${_proceed}" != "y" -a "${_proceed}" != "Y" ]; then
	echo "Aborting..."
	exit 0
fi

# Install items that are part of CentOS 8 Core group so they will be upgraded
# at distro-sync step
${YUM} install epel-release dnf tnsr-release NetworkManager \
	NetworkManager-team NetworkManager-tui dnf-plugins-core \
	rng-tools sssd-common sssd-kcm libibverbs

systemctl disable NetworkManager >/dev/null 2>&1 || true

# Adjust dnf.conf
if ! grep -q 'clean_requirements_on_remove=False' /etc/dnf/dnf.conf; then
	sed -i -e '/clean_requirements_on_remove/d' /etc/dnf/dnf.conf
	echo 'clean_requirements_on_remove=False' >> /etc/dnf/dnf.conf
fi

# At this point tnsr/tnsr-release packages will be removed due to yum-utils
# dependency
rpm -e --nodeps yum-utils
${DNF} remove yum yum-metadata-parser '*rpmconf*' redhat-rpm-config

mv -f /etc/yum /etc/yum.backup_tnsr

# Remove unnecessary kernels
TO_BE_REMOVED=$(dnf repoquery --installonly --latest-limit=-2 -q)
if [ -n "${TO_BE_REMOVED}" ]; then
	${DNF} remove ${TO_BE_REMOVED}
fi

trap '' HUP INT TERM
trap _fail EXIT

set -e

${DNF} --disablerepo=* install \
	${CENTOS_RPM_SERVER}/centos-repos-${CENTOS_VERSION}.el8.x86_64.rpm \
	${CENTOS_RPM_SERVER}/centos-release-${CENTOS_VERSION}.el8.x86_64.rpm \
	${CENTOS_RPM_SERVER}/centos-gpg-keys-${CENTOS_VERSION}.el8.noarch.rpm

${DNF} --disablerepo=* upgrade \
	https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm

# Cleanup all local files
dnf -y clean all

# Remove CentOS 7 kernel used to boot
rpm -e --nodeps $(rpm -q kernel-$(uname -r))

# Remove conflicting packages
rpm -e --nodeps sysvinit-tools

${DNF} --releasever=8 makecache

# Enable nginx 1.16 module
${DNF} --releasever=8 module enable nginx:1.16

# Cache all packages that will be used to upgrade
${DNF} --releasever=8 --allowerasing --setopt=deltarpm=false --best \
	--downloadonly distro-sync
${DNF} --releasever=8 --allowerasing --setopt=deltarpm=false --best \
	--downloadonly groupinstall Core
${DNF} --releasever=8 --allowerasing --setopt=deltarpm=false --best \
	--downloadonly install tnsr-release libibverbs

# Cleanup old dpdk stuff
rpm -e --nodeps dpdk
rm -rf /var/lib/dkms/dpdk

# Upgrade CentOS
${DNF} --releasever=8 --allowerasing --setopt=deltarpm=false --best \
	-C distro-sync

# Enable legacy network service
systemctl disable NetworkManager >/dev/null 2>&1 || true
systemctl enable network

trap "-" HUP INT TERM EXIT

echo "Upgrade is complete!  Reboot your system!"

When the script completes, reboot the router and monitor the console for errors.

Warning

In certain cases, the first reboot after upgrading to CentOS 8 may also require an extra dataplane restart to ensure interfaces have full connectivity. This can be performed from the shell with sudo tnsrctl restart or from the TNSR CLI with config then service dataplane restart. Alternately, performing another reboot will restart into the expected state.

If the upgrade failed, or it is otherwise not running and passing traffic properly after the upgrade, access the console and check the state of the system from there. Before contacting support, review TNSR 20.08 Release Notes for relevant changes in behavior which may require manual adjustments. For assistance, please contact Netgate TAC.

Updating after TNSR 20.08

The following methods are not a viable upgrade procedures to reach TNSR 20.08 from earlier versions. They may be used to upgrade from TNSR 20.08 to later versions as well as to obtain regular updates from CentOS.

AppStream Repository Priority

Before proceeding with the upgrade, check the CentOS AppStream repository priority. It should be set to a priority value of 100 to ensure that appropriate TNSR packages are used instead of CentOS versions with the same name.

Tip

The commands below are performed from a host shell, but may also be run from the TNSR cli using the host shell command.

First, look to see if the priority is already set:

$ grep priority /etc/yum.repos.d/CentOS-Linux-AppStream.repo
priority=100

In the above example, the priority is already set.

Note

If the grep command reports a No such file or directory error, try again with /etc/yum.repos.d/CentOS-AppStream.repo. If that succeeds, use that path in the following command as well.

If the grep command returns no output, add the priority as follows:

$ sudo bash -c 'echo "priority=100" >> /etc/yum.repos.d/CentOS-Linux-AppStream.repo'

After running that command, check again with grep or inspect the contents with cat or similar commands to ensure the priority line is present with the correct value.

Updates via the TNSR CLI

The easiest way to update TNSR is from within the TNSR CLI itself.

tnsr# package upgrade

That command will download and apply all available updates. Afterward, exit the CLI and start it again.

Note

There will be no output from this command until the process completely finishes, which may take a few minutes for larger updates.

Updating via the shell

TNSR can also be updated from the command line using the host OS package management commands, in this case, dnf:

$ sudo dnf -y module disable BaseOS
$ sudo dnf clean expire-cache
$ sudo dnf -y upgrade

Update Script

The following shell script may be used to keep TNSR and CentOS updated. In addition to the updates it also makes a local backup before performing the update.

Download: updatetnsr.sh
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
#!/bin/sh

# Time to make the backups
mkdir -p ~/tnsr-config-backup
sudo cp -p /var/tnsr/running_db ~/tnsr-config-backup/running_db-`date +%Y%m%d%H%M%S`.xml
sudo cp -p /var/tnsr/startup_db ~/tnsr-config-backup/startup_db-`date +%Y%m%d%H%M%S`.xml

# Update all RPMs
sudo dnf -y module disable BaseOS
sudo dnf clean expire-cache
sudo dnf -y upgrade

# Restart TNSR services to ensure a clean and consistent run
sudo tnsrctl restart

Post-Upgrade Reboot

TNSR upgrades include kernel updates, driver updates, and other operating system component updates. As such, the best practice is to reboot after upgrading these to ensure the device is running the proper kernel and is using a consistent set of updated system components.

The reboot procedure is covered in Rebooting the Router.

Updating the Configuration Database

Automatic Configuration Update

When upgrading from TNSR 20.02 to a later version, TNSR has its own automatic configuration upgrade procedures which accommodate changes made to the configuration database structure between versions.

Warning

The TNSR configuration upgrade only alters the running configuration database and not the startup database. After starting TNSR the first time post-upgrade, validate the running configuration. If the running configuration is OK, copy it to the startup configuration:

tnsr# config
tnsr(config)# configuration copy running startup

Configuration Update for Previous Versions of TNSR

When upgrading from a version older than 20.02, or restoring a configuration backup from before 20.02, additional steps may be necessary to update the configuration database.

If TNSR fails to start due to a change in the configuration, use the included configuration database migration utility, /usr/bin/tnsr-db-update, to check for and correct potential problems with older configuration files.

Warning

TNSR must be stopped before attempting to run this script, which modifies the running database at /var/tnsr/running_db.

The /usr/bin/tnsr-db-update script must be run as root either directly or via sudo:

$ sudo /usr/bin/tnsr-db-update

The utility will make a backup of the configuration before making alterations. This backup is placed in /var/lib/tnsr/db-backups/tnsr-<version>/running_db-<timestamp>.

If TNSR still will not start after running this utility, then the configuration database may contain errors which were not able to be corrected by the script.

Once all errors are corrected and TNSR starts successfully, copy the running configuration to the startup configuration.

Manual Configuration Update

Any errors which could not be corrected by the automatic script must be corrected by hand. Alternately, the configuration databases may be erased and recreated from scratch.

To attempt manual corrections, check the system logs after attempting to start TNSR for information about which configuration entries are causing the failure. View the logs with sudo systemctl status clixon-backend.service, sudo journalctl -xelu clixon-backend.service and sudo journalctl -xe.

A log entry for a configuration problem could look like the following example:

clixon_backend: startup_failsafe: 297: Database error: Startup failed and no
  Failsafe database found, exiting
clixon_backend: <rpc-reply><rpc-error>
  <error-type>application</error-type>
  <error-tag>unknown-element</error-tag><error-info>
  <bad-element>someinvalidtag</bad-element></error-info>
  <error-severity>error</error-severity>
  <error-message>namespace is: urn:ietf:params:xml:ns:netconf:base:1.0</error-message>
  </rpc-error></rpc-reply>

To correct such problems, edit the configuration in /var/tnsr/running_db (e.g. sudo vi /var/tnsr/running_db), erase or adjust the offending tag or configuration section, copy the repaired configuration to /var/tnsr/startup_db, and attempt to start TNSR again. Repeat until no errors are reported and TNSR starts normally.

To erase the configuration database, remove its files from /var/tnsr:

$ sudo rm /var/tnsr/*_db

After removing the configuration and starting TNSR, the TNSR configuration will need to be created again manually from scratch using the CLI or RESTCONF. Open the contents of a configuration backup in a text editor to use as a guide.

Additional Reboot / Update Verification

After performing a TNSR update and updating the configuration, administrators may wish to perform a reboot of the router to ensure it starts up correctly with the expected configuration.

This practice ensurs that the router performs as expected at startup during an upgrade maintenance window.

Update Troubleshooting

If the TNSR CLI method does not work, use the shell method instead.

If either method prints an error referring to a broken package database, recover it as follows:

$ mkdir -p ~/tmp/
$ sudo mv /var/lib/rpm/__db* ~/tmp/
$ sudo rpm --rebuilddb
$ sudo dnf clean all

Upgrading by Redeploying TNSR

Rather than performing an in-place update of a TNSR installation, administrators may instead choose to deploy a fresh instance of TNSR using the new version. This practice is typical of environments such as cloud providers or virtual machines, but may be performed for ISO installations and others as well.

In those cases, follow this general procedure:

  • Take a backup of the configuration

  • Deploy a new instance of TNSR using the installation instructions for the chosen platform

  • Restore the configuration

  • Update the configuration (Updating the Configuration Database)

See Configuration Backups for details on saving and restoring configuration backups, and review Updating the Configuration Database for important information about updating the configuration for a new version of TNSR.