Configuring Telemetry for timing

 

Introduction

This page provides information on configuring telemetry on the router and the client for streaming and storing raw logs on an external linux machine.

 

Pre-requisites

• Telemetry client is installed on Ubuntu linux VM. You can use the following Ubuntu OVA image, which already has the telemetry client applications installed.
  Note: The Cisco CLVS service by default provides Cisco Enterprise Linux. The telemetry client applications are tested only on Ubuntu. While requesting custom (Ubuntu) OS installation from the CLVS team, they require a linux machine image (OVA) file built from hypervisors like VMware Fusion. At the time of writing, the CLVS hypervisors could only load OVA files up to version vmx-11. Therefore, if you are building an OVA image you may want to set the supported hypervisor version vmx-11 or lower in the compatibiity-settings of your hypervisor.
• The router Mgmt interface is configured with an external IP through which it can reach the Ubuntu linux VM. 

 

XR configuration

 

telemetry model-driven
 destination-group telem_client

 ! Provide IP address and port on the machine where the collector is listening.
  address-family ipv4 10.65.44.195 port 57000
  encoding self-describing-gpb
  protocol tcp
 !
!
sensor-group my-servo
 sensor-path Cisco-IOS-XR-gnss-oper:gnss-receiver/nodes
 sensor-path Cisco-IOS-XR-ptp-oper:ptp/ptp-pd-oper:platform/servo
 sensor-path Cisco-IOS-XR-ptp-oper:ptp/ptp-pd-oper:platform/apr-log
 sensor-path Cisco-IOS-XR-ptp-oper:ptp/ptp-pd-oper:platform/servo-statistics
!

subscription servo-sub
 sensor-group-id my-servo sample-interval 5000
 destination-id telem_client
 source-interface MgmtEth0/RP0/CPU0/0
!
!
! Enable streaming of APR logs
ptp
 log
  servo driver
!
!

Pipeline collector

pipeline-gnmi as a collector for the telemetry client.

 

Installation

git clone https://github.com/cisco-ie/pipeline-gnmi.git

cd pipeline-gnmi

make build

 

Configuration

Pipeline uses two configuration files:

- pipeline_cisco.conf

- metrics_cisco.json

 

pipeline_cisco.conf

[cisco-input]
stage = xport_input
type = tcp
encap = st
listen = : 57000      # Should be same as the port number configured in XR configuration

[mymetrics]
stage = xport_output
type = metrics
file = metrics_cisco.json    # Attached to this page
datachanneldepth = 1000
output = influx
influx = http://localhost:8086     
database = ptp_db     # Influx database name used by pipeline for storing the received data
workers = 10

Running pipeline

<pipeline-gnmi directory>/bin/pipeline_linux_amd64 --config pipeline_cisco.conf

 

Database - Influx

 

Installation

wget https://dl.influxdata.com/influxdb/releases/influxdb_1.8.6_amd64.deb
sudo dpkg -i influxdb_1.8.6_amd64.deb
sudo service influxdb start

 

Configuration

influx       # Enter influx database shell

# Create database which wraps around data after 8 hours
create database ptp_db with duration 8h       # Use same database name as specified in pipeline_cisco.conf file

Other useful commands

influx     # Enter influx database shell
show databases      # List databases

use ptp_db             # Enter into a database

# List tables in the database
show measurements      
Cisco-IOS-XR-gnss-oper:gnss-receiver/nodes/node/receivers/receiver
Cisco-IOS-XR-ptp-oper:ptp/Cisco-IOS-XR-ptp-pd-oper:platform/apr-log
Cisco-IOS-XR-ptp-oper:ptp/Cisco-IOS-XR-ptp-pd-oper:platform/servo
Cisco-IOS-XR-ptp-oper:ptp/Cisco-IOS-XR-ptp-pd-oper:platform/servo-statistics

# Show columns in a table
show field keys from "Cisco-IOS-XR-ptp-oper:ptp/Cisco-IOS-XR-ptp-pd-oper:platform/servo"   
fieldKey fieldType
-------- ---------
lock-status integer
mean-path-delay integer
offset-from-master integer
phase-accuracy-last integer
servo-mode integer

#Retrieve data from the table
select "lock-status","offset-from-master" from "Cisco-IOS-XR-ptp-oper:ptp/Cisco-IOS-XR-ptp-pd-oper:platform/servo" order by desc limit 5
name: Cisco-IOS-XR-ptp-oper:ptp/Cisco-IOS-XR-ptp-pd-oper:platform/servo
time                lock-status offset-from-master
----                ----------- ------------------
1626669329152000000 2           2
1626669324152000000 2           2
1626669319153000000 2           2
1626669314152000000 2           2
1626669309152000000 2           2

Grafana

Installation

sudo apt-get install -y adduser libfontconfig1
wget https://dl.grafana.com/oss/release/grafana_8.0.5_amd64.deb

sudo dpkg -i grafana_8.0.5_amd64.deb
sudo /bin/systemctl start grafana-server

 

Launching grafana

Point the web-browser to http://<Ubuntu-VM-IP>:3000

 

References

Influx query language documentation: https://docs.influxdata.com/influxdb/v1.8/query_language/

 

ASR9000/XR: How to use Port Spanning or Port Mirroring

 

Introduction

This document provides some extra documentation and use cases on the use of port spanning or port mirroring.

You can monitor traffic passing in & out of a set of L2 or L3 Ethernet interfaces (including bundle-Ether).

 

Core Issue

ASR 9000 is the only platform implementing SPAN on XR (Only support on ethernet linecards, not on SIP-700.)

 

You can use SPAN/Mirror in the follow scenarios

- L2 & L3 interfaces.
- Local,  R-SPAN, and PW-SPAN only (no ER SPAN.)
- Scale limits:
    8 monitor sessions
    800 total source ports
    1.5 Gig bidirectional replication limit toward fabric for bundle interfaces and 10 Gig ports.
    Guideline:  ~ 10% - 15% total bandwidth can be mirrored system-wide
- Source ports:  Physical, EFPs, and bundles interfaces (L2 & L3)
- Destination ports:  Ethernet interfaces, EFPs, and PW-SPAN. (No bundle) [ only L2 transport interfaces are supported as destination ports]

- Ability to use ACL's to define which traffic is to be captured

- Capture multicast traffic is possible

 

Note: some of the functionality mentioned are enhancements to the XR 4.0.1 release, this document assumes you are using this release or later.

 

A good reference on the terminology of SPAN/Mirror can be found here:

http://www.cisco.com/en/US/docs/switches/lan/catalyst6500/ios/12.2SX/configuration/guide/span.pdf

 

 

SPAN order of operation

SPAN mirrors what is on the wire
For ingress, this means packets are mirrored before QOS, ACL, and encapsulation rewrite operations.
For egress, this means packets are mirrored after QOS, ACL, and encapsulation rewrite operations.

 

Partial Packet Mirroring

User can configure to mirror first 64 upto 256 bytes of the packet.
Note: The actual mirrored packet will be the configured size plus 4-byte trailling CRC.

 

Sample config:

 

interface GigabitEthernet0/6/0/20 l2transport
  monitor-session PW
  mirror first 100  <==  valid range: [64, 256], inclusively
  !
!

 

Note:  The mirrored packet received at sniffer will have the size of 104
               (4-byte of trailing CRC added by transmit MAC layer.)

 

 

ACL based Mirroring

 

“permit/deny” determines the behavior of the regular traffic (forwarded or dropped)
“capture” determines whether the packet is mirrored to the SPAN destination.

 

On SPAN: mirror traffic on the wire (regardless with or without ACL.)

      ACL on ingress direction:
           SPAN will mirror traffic even regular traffic dropped by ACL:  Always mirror!
     ACL on egress direction
          Will mirror if regular traffic is forwarded (Permit)
          Will not mirror if regular traffic is dropped (Deny.)

 

Inconsistent configurations:
“acl” is configured on SPAN source port but
   ACL has no “capture” keyword:
    No traffic gets mirrored. 
“acl” is NOT configured on SPAN source port but
   ACL has “capture” keyword:
    Mirroring traffic as normal, no ACL performed.

 

The ACL can also be an L2 ACL :

 

ethernet-services access-list esacl_t2
10 deny 1234.5678.90ab 0000.0000.0000 any capture

 

 

L3 Spanning Example

monitor-session TEST
destination interface GigabitEthernet0/1/0/2 (<<<< this is NP3)
!
interface GigabitEthernet0/1/0/14  (<<<< this is NP2)
ipv4 address 5.5.1.1 255.255.255.0
monitor-session TEST
  acl
!
load-interval 30
ipv4 access-group span ingress
!
ipv4 access-list span
10 permit ipv4 any host 1.1.1.10 capture
15 permit ipv4 any host 239.1.1.1 capture
20 permit ipv4 any host 2.2.2.100
30 permit ipv4 any any

 

Sample TRAFFIC GEN: (sending multicast in this example)
tgn rate 1000
L2-dest-addr 0100.5E01.0101
L2-src-addr 0003.A0FD.28A8
L3-src-addr 5.5.1.2
L3-dest-addr 239.1.1.1

 

Checking NP2: (the port that we are spanning)
Show global stats counters for NP2, revision v3

 

Read 12 non-zero NP counters:
Offset  Counter                                         FrameValue   Rate (pps)
-------------------------------------------------------------------------------
  22  PARSE_ENET_RECEIVE_CNT                                  5478        1001
  31  PARSE_INGRESS_DROP_CNT                                     3           1
  33  RESOLVE_INGRESS_DROP_CNT                                5474        1000
(there is no mcast recipient for this mcast addr, but we’re still replicating, see red line)
  40  PARSE_INGRESS_PUNT_CNT                                     1           0
  50  MODIFY_RX_SPAN_CNT                                      5475        1000
  54  MODIFY_FRAMES_PADDED_CNT                                5475        1000
  68  RESOLVE_INGRESS_L3_PUNT_CNT                                1           0
104  LOOP                                                       1           0
224  PUNT_STATISTICS                                            9           2
480  RESOLVE_IPM4_ING_RTE_DROP_CNT                           5475        1000
565  UIDB_TCAM_MISS_AGG_DROP                                    3           1
570  UIDB_TCAM_MISS_PORT4_DROP_FOR_HOST                         3           0

 

NP3 is the span monitor interface:
Show global stats counters for NP3, revision v3

 

Read 16 non-zero NP counters:
Offset  Counter                                         FrameValue   Rate (pps)
-------------------------------------------------------------------------------
  22  PARSE_ENET_RECEIVE_CNT                                    36           0
  23  PARSE_FABRIC_RECEIVE_CNT                               79656        1000
  30  MODIFY_ENET_TRANSMIT_CNT                               79655        1000

 

Packets received from fabric and sent off to the Ethernet on the span port!

 

 

PW SPAN example

For PW span to work, you need to define a local monitor session with a destination pseudo wire. You apply that span session to the interface of interest and define an xconnect group that also leverages that span session as one of the pw ends.

 

On the remote side where the PW terminates, you just configure regular VPWS.

Here an example:

 

 

On the Local Side, besides my Span configuration, there is also a local cross connect between the interested session we want to span over the PW

 

l2vpn

xconnect group TEST
  p2p TEST
   interface GigabitEthernet0/1/0/39

   ! port 39 is the port where we apply the span on.
   interface GigabitEthernet0/1/0/20.100
  ! this is just a random AC to have traffic flowing between the spanned port.
!

 

AC configuration:

interface GigabitEthernet0/1/0/20.100 l2transport
encapsulation dot1q 100
rewrite ingress tag pop 1 symmetric
! the tag is popped because the other XCON end is a plain ethernet without vlan. The explanation and use cases of tag popping can be found a related

! Tech note article.

 

 

Configuration on the remote side:

 

Regular VPWS configuration:

 

RP/0/RSP0/CPU0:A9K-TOP#sh run l2vpn
l2vpn
xconnect group PW-SPAN
  p2p PW-SPAN_1
   interface GigabitEthernet0/0/0/39
   neighbor 2.2.2.2 pw-id 1
   !
  !
!
interface GigabitEthernet0/0/0/39
load-interval 30
transceiver permit pid all
l2transport
!
!

 

the neighbor in the l2vpn configuration is the LDP neighbor ID
between which the PW is built.

 

Show on remote side:
RP/0/RSP0/CPU0:A9K-TOP#show l2vpn xcon group PW-SPAN det

 

Group PW-SPAN, XC PW-SPAN_1, state is up; Interworking none
  AC: GigabitEthernet0/0/0/39, state is up
    Type Ethernet
    MTU 1500; XC ID 0x4000a; interworking none
    Statistics:
      packets: received 0, sent 16570475
      bytes: received 0, sent 994228500

! packets received from the PW are sent out hte Attachment circuit's interface. The analyzer is connected to G0/0/0/39
  PW: neighbor 2.2.2.2, PW ID 1000, state is up ( established )
    PW class not set, XC ID 0x4000a
    Encapsulation MPLS, protocol LDP
    PW type Ethernet, control word disabled, interworking none
    PW backup disable delay 0 sec
    Sequencing not set

 

      MPLS         Local                          Remote
      ------------ ------------------------------ -----------------------------
      Label        16002                          16027
      Group ID     0xa40                          0x2
      Interface    GigabitEthernet0/0/0/39        PW/TM/MS
      MTU          1500                           1500
      Control word disabled                       disabled
      PW type      Ethernet                       Ethernet
      VCCV CV type 0x2                            0x2
                   (LSP ping verification)        (LSP ping verification)
      VCCV CC type 0x6                            0x6
                   (router alert label)           (router alert label)
                   (TTL expiry)                   (TTL expiry)
      ------------ ------------------------------ -----------------------------
    MIB cpwVcIndex: 4294705162
    Create time: 04/04/2011 14:36:42 (00:20:07 ago)
    Last time status changed: 04/04/2011 14:36:42 (00:20:07 ago)
    Statistics:
      packets: received 16570475, sent 0
      bytes: received 994228500, sent 0

! Packets received on the Pseudo Wire from the SPAN port

 

 

NOTE: Pseudo Wire counters on the span side are not incrementing.That is the XCON group "cisco" in this picture config example.

This is intentional. You can review the SPANNING also with this command:

 

RP/0/RSP1/CPU0:A9K-BOTTOM#sh monitor-session counters

Monitor-session PW_TM_MS
  GigabitEthernet0/1/0/39
    Rx replicated: 58488205 packets, 3743245120 octets
    Tx replicated: 58488206 packets, 3743245184 octets
    Non-replicated: 0 packets, 0 octets

 

R-SPAN configuration:

R-SPAN is natively support with the capability of ASR9000 to do vlan imposition:

 

monitor-session MS2

destination interface gig0/2/0/19.10

!

interface gig0/2/0/12.10 l2transport

encapsulation dot1q 10 <<< Monitoring vlan 10 traffic

monitor-session MS2

!

interface gig0/2/0/19.10 l2transport (*)

encapsulation dot1q 100 <<< VLAN 100 will get imposed.

!

 

 

(*) Monitor destination could be any supported destination interface regardless of monitor source

 

 

 

 

Related Information

n/a

Xander Thuijs, CCIE #6775

Sr. Tech Lead ASR9000

https://community.cisco.com/t5/service-providers-documents/asr9000-xr-how-to-use-port-spanning-or-port-mirroring/ta-p/3108031

Support of: unhide viptela_internal

 

Introduction

 

Starting 20.4 release, we have removed the support for unhide viptela_internal command which let (TAC) Engineers troubleshoot the customer issues.    unhide viptela_internal is no longer a valid command and any of the previously hidden commands that remain for field, use are either “support” commands, are have been made fully supported commands.   

 

 

Background

 

There are MANY hidden commands.  If you go to a Viptela device CLI you will not see “show internal” or “request internal” or “tools internal”.  But if you type “unhide viptela_internal” and then provide the password ”  ", you will then be able to see those.  And underneath them are many more commands, all usually hidden and none of them are documented.  This is considered a security violation under Cisco rules.  Because this is not documented, it is considered a back door.  Because there is a password, it appears to be a more serious back door.  And this password has been posted (by others) online. 

 

Also note, in 19.2.3, 20.1.2. 20.3.1 and 20.3.2, we no longer user "unhite viptela_internal" to access.   Instead, use "unhide full".  The password is the same as used with viptela_internal.  See CSCvt00497  for more information.

 

With CSCwa45995: We are removing all traces of "unhide viptela internal" from the cEdge platform.   As part of removing hidden config (which could be exposed via unhide command), some commands were missed on polaris.  With this CDETS, we will be removing all instances of "viptela_internal" hidegroup from the code.  

 

External Notification

 

The following CCO link is posted externally.  

For the reasons mentioned above, the password and the list of hidden commands are published in below link.

 

https://www.cisco.com/c/dam/en/us/td/docs/routers/sdwan/Internal-Commands/Troubleshooting-Commands-f...

 

 

 

20.4 and after

 

vEdge# unhide viptela_internal
Error: unknown hide group

 

Any of the previously hidden commands that remain for field use, are moved under the support option 

There may be some commands that could be missing.     See below for more information.

 

vEdge# tools support ?
Possible completions:
  fp-dump   Perform fp-dump on a network interface
vEdge# 

 

vEdge# show support ?     
Possible completions:
  cellular       cellular support commands
  cloudexpress   cloudexpress support commands
  control        DTLS support shell commands
  dhcp           DHCP support commands
  dnsd           dnsd support commands
  dpi            dpi support commands
  filter         filter support commands
  fp             Fast-path support commands
  ftm            ftm support commands
  nat            nat support commands
  omp            OMP support commands
  pim            pim support commands
  resolv         resolvd support commands
  tracker        tracker support commands
  ttm            TTM support commands
  vrrp           VRRP support commands
vEdge# 

 

vEdge# request support ?
Possible completions:
  cellular               
  debug-malloc           Malloc-trim in a daemon
  fp                     
  router-advertisement   Enable/Disable Ipv6 Router Advertisements tx/rx interface
  software               
  tcpopt                 
  vdebug                 Control vdebug RAM disk logging
vEdge# 

 

For UnPinning of flows on vE2K

vEdge# request support fp unpin-flows

 

Moving the deivce to vManaged mode or not

Currently there is no option to move the device in or out of vManage mode.  This option is not directly available to the customer.  It requires the use of 'unhide viptela_internal', and then from config mode running 'no system is-vmanaged'.

In 20.4, this is missing.     CSCvx23574  is opened to track this.   This will address for both cEdge and vEdge platforms.

 

 

Capturing (existing) Internal commands

 

Below are the tools, show and request internal commands as taken from 20.3.1 node.

 

show internal

 

vEdge# show  internal ?
Possible completions:
  admin-tech     Admin-tech commands
  app-route      
  cellular       
  cfgmgr         Configuration Manager shell commands
  cflowd         
  cloudexpress   cloudexpress commands
  control        DTLS shell commands
  cxp-app        
  dbgd           
  dhcp           DHCP shell commands
  dnsd           dnsd commands
  dot1x          
  dpi            dpi commands
  filter         
  flow-db        Flow Database
  flow-summary   Flow Database Summary
  fp             Fast-path shell commands
  fpm            
  ftm            
  gps            
  igmp           
  nat            
  omp            OMP shell commands
  pim            
  policy         Policy shell commands
  resolv         
  rtm            RTM shell commands
  server-app     
  snmp           SNMP shell commands
  sysmgr         
  system         
  tcpopt-db      
  tcpopt-tcpd    
  tracker        Tracker shell commands
  ttm            TTM shell commands
  tunnel         
  vrrp           VRRP shell commands
  wlan           
  zbf            
vEdge# 

 

request internal

 

vEdge# request internal ?
Possible completions:
  cloudexpress      Cloudexpress related tools command
  embargo           vEdge embargo check
  fec               
  fp-dump           Perform fp-dump on a network interface
  ftm               
  interface-reset   
  live-core         Generate non-disruptive coredump of a running process
  malloc-trim       Malloc-trim in a daemon
  reset             Reset system or logs
  software          
  tcpopt            
  vdebug            Control vdebug RAM disk logging
  vedge-cloud       vEdge cloud internal commands
vEdge#

 

tools internal

 

vEdge# tools internal ?
Possible completions:
  clean_db            Remove vManage data
  csr_read            Reading cavium registers.
  csr_write           Writing into cavium registers.
  ethtool             ethtool
  firmware-printenv   Display environment variables.
  fp-dump             Perform fp-dump on a network interface
  hostapd_cli         hostapd_cli
  i2cdetect           i2cdetect tool.(Only for Mips)
  i2cdump             i2cdump tool.
  i2cget              i2cget tool. (only for Mips)
  i2cset              i2cset tool.
  mdio-read           mdio-read
  mdio-write          mdio-write
  mii-tool            mii-tool
  oui-lookup          Perform OUI lookup for show arp.
  poe-tool            poe-tool
  process_id          Find process ID.
  remove_tenancy      Remove Tenancy file on vManage
  tlv_tool            TLV tool.(Only for Mips)
  touch_test_root     Create or remove /usr/share/viptela/test_root for allowing any root cert for sw vedges.
  tracker             Add Latency on the interface for tracker packets
  valgrind_tool       Enable valgrind on a process.
vEdge#