Tip
This is the documentation for the 22.06 version. Looking for the documentation of the latest version? Have a look here.
Prometheus Exporter¶
TNSR includes a Prometheus exporter which supports statistical data from the
dataplane (VPP) only. This data is typically fed into Grafana. When active,
the service listens for connections on TCP port 9482
.
See also
Configuring Grafana and its supporting systems is outside the scope of this documentation. Consult its documentation for details.
Warning
The Prometheus service on TNSR does not perform authentication or encryption. Only transmit data across trusted network paths, and do not expose the service to untrusted networks.
Data exported by Prometheus may be sensitive in nature, so protect access to the service with appropriate standard ACLs if the service is running in the dataplane namespace. Do not open access to the Prometheus port unilaterally. The default host ACL ruleset allows access to the Prometheus port.
Configuring Prometheus¶
- prometheus <namespace> enable:
Enables the Prometheus Exporter service in either the
host
ordataplane
namespace.Warning
Using the
host
namespace is more secure as limits the exposure of the service to host OS management networks.After enabling the service, start it as described in Service Control.
- prometheus <namespace> disable:
Disables the Prometheus Exporter service in the given namespace.
- prometheus <namespace> filter <regex> [<regex> […]]:
Adds one or more regular expression filters which limit the data exposed by the service. For details about how these filters work, see Prometheus Filtering.
- no prometheus <namespace> filter <regex>:
Removes a filter.
Prometheus Filtering¶
The data exposed by Prometheus to clients can be limited by filtering. Limiting in this manner can increase performance by reducing the amount of data sent to clients and the amount that clients must process as a consequence. By default, all data in the dataplane statistics segment is exported which can be sizable on large deployments.
Prometheus filters consist of one or more POSIX style regular expression (“regex”) patterns which will result in data matching any of the given patterns as they are considered to have an implicit boolean “OR” between them.
To see the full list of statistical items which the dataplane can export run the following command in a shell:
$ sudo vpp_get_stats ls [<regex pattern>]
Note
Though the statistical items in the dataplane appear to be structured similar to a filesystem, matching does not work like shell filesystem globs, but as regular expression patterns only.
Filter patterns do not include automatic anchoring. Patterns can match any
part of a given string. For example, the pattern /a
will match /a
,
b/a
, b/a/c
, and so on. Regular expression anchoring can limit this as
needed: To only match a string starting with /a
use ^/a
, to match as
string ending in /a
, use /a$
, and to match a string exactly anchor it
both ways, such as ^/a$
.
Using wildcard style matching in filters must also follow regular expression
syntax. The filter ab*
matches the a
followed by b
repeated zero or
more times. Thus it will match a
, ab
, abb
, and abbb
, but not
ac
. To match ab
followed by anything else, use ab.*
which will match
ab
plus any single character (.
) repeated zero or more times.
See also
Comprehensive coverage of regular expression pattern usage is beyond the scope of this documentation. There are a variety of resources on the Internet which cover regular expression usage and syntax. See regular expression page on Wikipedia for a good starting point.
Warning
When querying TNSR for Prometheus data, the items returned do not use /
as a separator, but _
. For example, the dataplane item
/sys/vector_rate_per_worker
will be in the Prometheus data as
_sys_vector_rate_per_worker
. Always craft filter expressions to match the
dataplane version of the desired items, not the format in Prometheus
output.
Querying Prometheus Data¶
The URL for metrics is: http://<IP address>:9482/metrics
In the URL, <IP address>
is an IP address on an interface in the appropriate
namespace. For example, if Prometheus is running in the host
namespace, the
IP address would be from a host OS management interface on TNSR.
Note
This service is not meant to be queried by a web browser. Use a client which
understands Prometheus data, or a client such as curl
which will print
the data returned by the service when testing.
Note
The Prometheus statistics daemon uses an AF_INET6 socket which can accept
connections from both IPv4 and IPv6 clients. When viewing the daemon process,
it will show tcp6
, IPv6
, or similar strings in the output. This is
normal and does not indicate a problem.
$ sudo netstat -lntp | grep prometheus
tcp6 0 0 :::9482 :::* LISTEN 2985/vpp_prometheus
$ sudo lsof +c15 -iTCP -sTCP:LISTEN | grep prometheus
vpp_prometheus_ 2985 root 3u IPv6 39319 0t0 TCP *:9482 (LISTEN)
The dataplane tracks some metrics served by Prometheus by interface name and others by index. The Prometheus output includes entries which map the index numbers to their corresponding names. This can be used to correlate data in the monitoring system consuming the data:
# TYPE _if_names_info gauge
_if_names_info{index="0",name="local0"} 1
_if_names_info{index="1",name="WAN"} 1
_if_names_info{index="2",name="LAN"} 1
_if_names_info{index="3",name="CorpNet"} 1
_if_names_info{index="4",name="DMZ"} 1
_if_names_info{index="5",name="Guest"} 1
Protecting Prometheus¶
As mentioned in the warning at the start of this section, the Prometheus service does not have its own encryption or authentication. As such, the primary ways to protect the service are:
- Isolate the service to the appropriate namespace
This is typically the
host
namespace, but some TNSR configurations do not have host interfaces.- Protect the network paths carrying Prometheus data
Use a directly connected secure path, such as a local management network. If the data must be transmitted remotely, encrypt the path between Prometheus and the host collecting its data, for example, with an IPsec tunnel.
- Restrict access with ACLs
Limit access to the prometheus server on TCP port
9482
using access lists. When running in thehost
namespace, use host ACLs and when running in thedataplane
namespace use standard ACLs.Note
The default set of host ACLs denies access to the service, but take care when crafting rules to only permit access from authorized hosts or networks.
This example host ACL permits access to the Prometheus service from one IPv4 host,
198.51.100.244
:tnsr(config)# host acl prometheus tnsr(config-host-acl)# sequence 10 tnsr(config-host-acl)# rule 10 tnsr(config-host-acl-rule)# action permit tnsr(config-host-acl-rule)# description Allow Prometheus tnsr(config-host-acl-rule)# match ip protocol tcp tnsr(config-host-acl-rule)# match ip port destination 9482 tnsr(config-host-acl-rule)# match ip version 4 tnsr(config-host-acl-rule)# match ip address source 198.51.100.244/32 tnsr(config-host-acl-rule)# exit tnsr(config-host-acl)# exit tnsr(config)#