Wednesday, March 31, 2021

Lightspeed family get ready doc

Lightspeed

The Lightspeed program is the next-generation of route processors and line cards for the ASR9K family. These new cards will be based on Lightspeed NPU and Skybolt fabric. New set of commons, RP/RSPs and Skybolt based fabric cards, shall also be included as part of this project enabling these newer cards to operate at their full capacity.

Lightspeed - the fourth generation ASR9k Hardware and has the following set of new linecards

32 port 100G linecard: A99-32X100GE-SE& A99-32X100GE-TR
16 port 100G linecard: A9K-16X100GE-SE& A9K-16X100GE-TR
8 port 100G linecard: A9K-8X100GE-X-SE & A9K-8X100GE-X-TR

Following are the release timelines for the various PID on Lightspeed family

7-fabric linecards:
- A99-32X100GE–TR (supported in 6.5.15)
- A99-32X100GE–SE (supported with TR scale in 6.5.15. SE scale support in 6.5.2)
5-fabric linecards:
- A9K-16X100GE–TR (supported in 6.5.15)
- A9K-16X100GE–SE (supported with TR scale in 6.5.15. SE scale support in 6.5.2)
- A9K-8X100GE-X-TR (supported in 6.5.15)

A9K-8X100GE-X-SE (supported with TR scale in 6.5.15. SE scale support in 6.5.2)

* Note these versions are eXR only (64bit version of XR)

The key hardware capabilities for the Lightpseed hardware are listed below

New in-house developed NPU
420G bandwidth (bi-directional), 300Mpps+ full duplex forwarding per LightSpeed NPU
- (Tomahawk: 240G / 150Mpps+)
8Tb to 4.2Tb per slot fabric capacity
- depending on chassis type
22 billion transistors
2M MAC addresses
6M v4 or 6M v6 routes
48K queues per NPU

Difference between Tomahawk and Lightspeed

TBD (Internal consumption )

7- Fabric vs 5-Fabric

This denotes the requirement of how many FC3’s (fabric card) are required in the system.

For a 7-Fabric line card you need to have 7 FC3’s installed in the system to take the effectively utilize the Lightspeed bandwidth capacity.

For ASR9922 & ASR 9912

32x100GE 7-fabric linecard: You need to install 7 SFC3s
16x100GE 5-fabric linecard: You need to install min 4 SFC3s
8x100GE 5-fabric linecard: You need to install min 4 SFC3s

For ASR9910 & ASR9906

32x100GE 7-fabric linecard: You need to install 2 RSP5s and 5 SFC3s
16x100GE 5-fabric linecard: You need to install 2 RSP5s and min 2 SFC3s
8x100GE 5-fabric linecard: You need to install 2 RSP5s and min 2 SFC3s

The 7 fabric cards also need the A99-HighBandwidth fabric mode enabled to avoid policing the FIA to ~93gig per interface

Information on ASR 9000 Fabric Modes

If you use a 7 fabric card without that fabric mode you will see the following syslog when the LC comes online.

%FABRIC-FIA-1-RATE_LIMITER_ON : Set|fialc[4320]|0x108a000|Insufficient fabric capacity for card types in 
use -FIA egress rate limiter applied

The following table illustrates the same :

Linecard Type	Fabrics
32 Port	7	7xSFC3 - or - 2xRSP5 & 5xSFC3
16 Port	4	4xSFC3 - or - 2xRSP5 & 2xSFC3
8 Port	4	4xSFC2/SFC3 - or - 2xRSP5 & 2xSFC3

Line card Architecture - 32X100 GE

Key highlights on the LC architecture for 32X100 GE

3.2T linerate QSFP28 LAN linecard: SE & TR variant (SE has TR scale with 6.5.15. Full SE scale with 6.5.2)
No hardware support for OTN, MACsec and FlexE
7-Fabric card

Works in 9922, 9912, 9910, 9906 & 9904 chassis

Requires LightSpeed commons

RP3
SFC3
RSP5

Linecard Performance

ASR 9922, 9912, 9910 & 9906 chassis

3.2T linerate with fabric redundancy

ASR 9904 chassis

3.2T linerate with dual RSP5
1.8T throughput with single RSP5

Line card Architecture - 16X100 GE

1.6T linerate QSFP28 LAN linecard: SE & TR variant (SE has TR scale with 6.5.15. Full SE scale with 6.5.2)
No hardware support for OTN, MACsec and FlexE
5-Fabric card

Works in all ASR9k modular chassis

Can work with Tomahawk commons (RP2)

Rules are explained later in this deck

Linecard Performance with LightSpeed commons

ASR 9922, 9912, 9910 & 9906 chassis

1.6T linerate with fabric redundancy

ASR 9904 chassis

1.6T linerate with dual RSP5
1.4T throughput with single RSP5

ASR 9010 & 9006 chassis

1.6T linerate with dual RSP5
900G throughput with single RSP5

Lightspeed Commons

Following are list of the Common PID;s that will support Lightspeed family of line cards:

RP/RSP

• A99-RP3-SE - ASR 9900 Route Processor 3 for Service Edge

• A99-RP3-TR - ASR 9900 Route Processor 3 for Packet Transport

• A9K-RSP5-SE - ASR 9000 Route Switch Processor 5 for Service Edge

• A9K-RSP5-TR -ASR 9000 Route Switch Processor 5 for Packet Transport

Fabric Cards

• A99-SFC3 ASR 9900 Series Switch Fabric Card 3

• A99-SFC3-S ASR 9910 Series Switch Fabric Card 3 (ASR 9910 Shockwave)

• A99-SFC3-T ASR 9906 Series Switch Fabric Card 3 (ASR 9906 Torchwood)

Fan

• ASR-9922-FAN-V3 ASR 9922 Fan Tray v3

•

Route Switch Processors and Route Processors

RSP used in ASR9910/9906/9904/9006/9010, RP in ASR9922/9912

	RSP880 A99-RSP	RP2	RSP5	RP3
Description	3rd Gen RP and Fabric ASIC		4th Gen RP and Fabric ASIC
Switch Fabric Bandwidth	400G + 400G (9006/9010) 700G + 700G (9904) 200G + 200G + 1.0T (9906/9910)	1.2Tb + 200G (separated fabric card)	900G + 900G (9006/9010) 1.8T + 1.8T (9904) 600G + 600G + 3.0T (9906/9910)	3.6Tb + 600G (separated fabric card)
Processor	Intel x86 (Ivy Bridge EP) 8 Core 2GHz		Intel x86 (Skylake EP) 8 Core 2GHz
RAM	-TR: 16GB -SE: 32GB		-TR: 16GB -SE: 40GB
SSD	2 x 32GB Slim SATA		2 x 128GB Slim SATA
Punt BW	40GE		40GE

ASR 9000 Switch Fabric Overview

ASR9922/12: 7-fabric System w/ LightSpeed and Tomahawk

The 7-fabric LCs use all 7 fabrics if all other LCs in the system are 7 fabricand all LightSpeed and Tomahawk LCs in this mode interoperate at full BW.

ASR99xx: 5-fabric System w/ LightSpeed and Tomahawk

A 7 Fabric LC ( i.e 32 X 100 G ) uses only 5 fabric when there is a 5-Fab lightspeed or tomahawk LC in the system.

Lightspeed NPU Internals

LightSpeed supports up to 500Gbps of line side interfaces comprised of a configurable combination of 10G/25G/40G/100G/400G Ethernet ports and/or 100G/400G/500G Interlaken interfaces. In addition to this there are two dedicated 10GE ports for punt/inject as well as a PDMA port (via a PCI gen3 interface) from the local CPU. Note that packets sourced locally (either via punt/inject 10GE ports or the PCI interface) can only come in to the ingress side of the ASIC, and packets destined to the local CPU (punt/inject or PCI) must go through the egress side.

There are several local datapaths.

A 20Gbps/8Mpps local loopback path which loops back from the ingress datapath to the egress datapath bypassing the fabric interface
A 60Gbps/25Mpps recycle path that loops back from the egress datapath back into the egress datapath (note this feature is not available in the ingress direction)

A 250Gbps service loopback path which loops back from the egress datapath to the ingress datapath bypassing the L2 line interface

Brief description of each functional block is listed below:

L2 Super Block–Line side interfaces plus preliminary L2 processing
- Flexible Header Parsing
- Protocol Decode and Validation
- Bundle Mapping
- Satellite Encap Termination
- Priority Classification
i/eGPB–
- ingress/egress Global Packet Buffer, split as ingress and egress GPB – meaning ingress and egress traffic can’t be switched directly within same PPE. It has to go through loopback path to reach counterpart.
- 9MB to store packet under processing. V
- Uidb, Racetrack and My MAC lookup moved in from L2 block.
DST–Manages the assignment of packets to 4608 PPE threads based on affinity

PA–Processor Array, has up to 3584 threads to process received packet and runs u-code, uses DMA for h/w assists and TLU for table lookups

FLB–Flow Lock Block, enforce packet order in the same flow. Support up to 1024 flows.
i/eGTR–ingress/egress Gather Block, gather up the packet from iGPB memory for sends the packet to Fabric or Line
EPE–Embedded Processing Element, 96MB SRAM. Store
- Used for Stats, Policer, C code stack & thread local memory
DPE (DRAM): 2x4GB
Internal TCAM:5 MB
New hash-based PLUalgorithm with 9MB SRAM ( Refer Processor Prefix Lookup)
Local loopback used for ingress punt & inject to wire
Service loopback used for egress to fabric
Recycle from egress TM to eGPB
PCIe Gen3 Host Processor Interface

Life of packet inside the LS NPU

1) L2 - Block :

An untagged frame is received on an Ethernet port of the L2 Block
Decodes the L2 encapsulation, determines that the L3 Type is IPv4, determines the L3 Offset, decodes the IPv4 header
Gets the frame type i.e encapsulation type
The Packet Classification Module (PCM) of the L2 Block detects a My-MAC match on the destination MAC address and prioritizes the frame based on the TOS value in the IPv4 header

2) iGPB :

Prior to sending the packet to GPM, the BAF performs the Ingress UIDB Classification TCAM Lookup and a Protocol Decode TCAM Lookup

The Ingress UIDB Classification uses the embedded TCAM
The Protocol Decode TCAM Lookup produces a vector that is used to accelerate PPE execution, For e.g an “IPv4 Unicast Normal, No Exceptions”, for an IPV4 packet.
Some of the fields in the Protocol Decode TCAM Lookup Key are Mark Bits, Port Bits, Frame Type, L3 Type, DA Bits, and Decode Bits

3) iGPB :

The packet is forwarded to a PPE for processing, with BAF Header is delivered to the PPE along with the packet.
BAF Headers are accessible to the PPE software via DMEM. some of the more interesting info provided in the BAF Header includes Traffic Type, Frame Type, L3 Type, L3 Offset, Mark Bits, Port Bits, DA Bits, Decode Bits, Protocol Decode Vector, BAF Port, Logical Port, and Timestamp

4) PPE: Data plane software execution begins in the main packet processing loop .

5) PPE: Depending on the packet was received from a line side interface by checking a field in the Distribution Header, and then switches on the Protocol Decode Vector, which results in a call to the ingress IPv4 processing function.

6) PPE: Depending on the packet type the appropriate packet processing will take place

Fabric header is built in this stage and packets are flushed to fabric.

7) Packet is transmitted across fabric to egress LC

LightSpeed Migration

In service RSP5 or RP3 Migration is documented here:

https://www.cisco.com/c/en/us/td/docs/iosxr/asr9000/hardware-install/hig/b-asr9k-hardware-installati...

QOS

QOS Deployment Guide

The ASR9k QOS Deployment Guide has been updated. There are still more updates coming so keep checking the officiall doc EDCS–1226762

Attaching version 45.1 here but check on docs.cisco.com for the latest version.

QOS Dynamic Packet Buffering

http://xrdocs.io/asr9k/blogs/2018-09-06-dynamic-packet-buffering/

Optics

See the datasheet for the supported optics.

a99-32x100ge Data Sheet

Also note that usage of 100GER4-L-S QSFP in the 32x100GE card.

It is a known limitation on the 32x100G LC only as these optics dissipate a lot more heat than the others.

These are the ports supported - the rest are disabled in software:

Ports : 12,13,14,15 and 28,29,30,31

Feature Parity

Keep track with the PM team: https://salesconnect.cisco.com/#/program/PAGE-10319

SW and HW Release Dashboard - https://apps.na.collabserv.com/communities/service/html/communitystart?communityUuid=a89c8115-466a-4...

Screen Shot 2018-10-24 at 3.17.03 PM.png

Wednesday, February 3, 2021

Password Recovery - NCS5K

Introduction

These instructions will erase the config and software restoring the router to factory defaults.

At present, there is no way to keep the config and reset the password.

Requirements

A USB Drive, big enough to fit the boot files ... 6.0.0 is about 810MB.
.tar file with the USB boot files ... I used NCS5000-iosxr-k9-6.0.0.tar
Physical access.
Console access.
30 minutes and two reboots. This doesn't add the packages necessary for operation.

Procedure

1. Understand this will erase the config on the router.

2. Download a software image via cisco.com, it doesn't need to be the image currently on the router.

3. Extract the .tar. Inside you should see a folder similar to ncs5k-usb-boot-6.0.0

4. Copy the contents of the folder, boot and EFI to a Fat32 formatted USB drive. Make sure the boot and EFI folders are in the root directory of the USB drive.

5. Power down the NCS5K, and insert the USB drive.

6. While connected to the CONSOLE, hold down the ESC key.

7. This leads to the BIOS boot prompt, boot the USB device.

8. Wait.

9. Create a default username and password.

Verification

I've included timestamps so you can get a feel for how long this takes.

CISCO NCS5K XR ROUTER Ver 1.8
Booting from Primary Bios 
Code Signing Results: 0x0 
Using Upgrade FPGA
FPGA Revision        : 0x10
FPGA ID             : 0x1
FPGA Date           : 0x20141016
Reset Cause Register: 0x80000000
SPI Flash Boot Timer Ctrl Status Register  : 0xe0ff
Persistent Power Debug Register1 : 0x0
Persistent Power Debug Register2 : 0x0
FPGA Remote System Update Status Register: 0x20
Detected CISCO MIFPGA
FPGA Update Status  : 0x20
                                                                             A9
       Aptio Setup Utility - Copyright (C) 2015 American Megatrends, Inc.       
    Cisco Boot Options                                                          
ÚÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÂÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄ¿
³  Boot Override                                     ³                         ³
³  UEFI: SMART eUSB HS-SD/MMC                        ³                         ³
³  UEFI: Built-in EFI IPXE                           ³                         ³
³  UEFI: Built-in EFI Shell                          ³                         ³
³  UEFI: SanDisk Extreme 0001                        ³                         ³
³                                                    ³                         ³
³                                                    ³                         ³
³                                                    ³                         ³
³                                                    ³                         ³
³                                                    ³ÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄ³
³                                                    ³><: Select Screen        ³
³                                                    ³: Select Item          ³
³                                                    ³Enter: Select            ³
³                                                    ³+/-: Change Opt.         ³
³                                                    ³F1: General Help         ³
³                                                    ³ESC: Exit                ³
³                                                    ³                         ³
³                                                    ³                         ³
³                                                    ³                         ³
ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÁÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ
        Version 2.17.1246. Copyright (C) 2015 American Megatrends, Inc.        
                                                                             AB
PASSED SecureBoot verification

GNU GRUB version 2.00

13:18:22.867 § 
13:18:22.867 § GNU GRUB version 2.00
13:18:22.867 § Press F2 to goto grub Menu..
13:18:24.899 § Booting from USB..
13:18:24.899 § Loading Kernel..
13:18:25.642 § Loading initrd..
13:20:17.307 § Initrd, addr=0xcdbd4000, size=0x3241b768
13:20:23.183 § [    5.762902] i8042: No controller found
13:20:23.197 § mkdir: cannot create directory '/run': File exists
13:20:23.220 § Starting udev
13:20:25.695 § Populating dev cache
13:20:25.968 § Running postinst /etc/rpm-postinsts/100-dnsmasq...
13:20:26.099 § update-rc.d: /etc/init.d/run-postinsts exists during rc.d purge (continuing)
13:20:26.195 §  Removing any system startup links for run-postinsts ...
13:20:26.273 §   /etc/rcS.d/S99run-postinsts
13:20:26.386 § Configuring network interfaces... done.
13:20:26.497 § Starting system message bus: dbus.
13:20:26.612 § Starting OpenBSD Secure Shell server: sshd
13:20:26.627 §   generating ssh RSA key...
13:20:26.787 §   generating ssh ECDSA key...
13:20:26.794 §   generating ssh DSA key...
13:20:26.842 §   generating ssh ED25519 key...
13:20:27.286 § done.
13:20:27.621 § Starting rpcbind daemon...done.
13:20:28.507 § Starting libvirtd daemon: [  OK  ]
13:20:28.711 § Starting system log daemon...0
13:20:29.089 § Starting kernel log daemon...0
13:20:29.131 § Starting HPA's tftpd: in.tftpd-hpa
13:20:29.141 § .
13:20:29.141 § Starting internet superserver: xinetd.
13:20:29.141 § Starting S.M.A.R.T. daemon: smartd (failed)
13:20:29.198 § Wed Oct 19 21:40:49 UTC 2016: Detected /iso/host.iso
13:20:29.251 § mount: /dev/loop0 is write-protected, mounting read-only
13:20:29.311 § Wed Oct 19 21:40:49 UTC 2016: Mounted /iso/host.iso to /tmp/isomnt.wamZqW
13:20:29.458 § Wed Oct 19 21:40:49 UTC 2016: Found /tmp/isomnt.wamZqW/rpm/ncs5k-sysadmin-hostos-6.0.0-r600.host.x86_64.rpm in host.iso
13:20:29.506 § Wed Oct 19 21:40:49 UTC 2016: Installing /tmp/isomnt.wamZqW/rpm/ncs5k-sysadmin-hostos-6.0.0-r600.host.x86_64.rpm
13:20:29.546 § Preparing packages for installation...
13:20:29.695 § ncs5k-sysadmin-hostos-6.0.0-r600.host.x86_64
13:20:29.740 § Wed Oct 19 21:40:49 UTC 2016: Passing control to /etc/rc.d/init.d/pxe_install.sh start
13:20:29.741 § [    9.250969] sd 5:0:0:0: [sdb] No Caching mode page found
13:20:29.755 § [    9.273498] sd 5:0:0:0: [sdb] No Caching mode page found
13:20:29.756 § [    9.304257] sd 5:0:0:0: [sdb] No Caching mode page found
13:20:29.768 § Wed Oct 19 21:40:49 UTC 2016: Preparing disk for PLATFORM=ncs5k:
13:20:29.799 § Number of disks detected: 2: Disk Check for: /dev/sdb 
13:20:29.800 § Sourcing /etc/sysconfig/udev
13:20:29.854 § Starting udev: [  OK  ]
13:20:29.864 § Wed Oct 19 21:40:49 UTC 2016: 
13:20:29.865 § Wed Oct 19 21:40:49 UTC 2016: Secondary disk is not present
13:20:29.875 § Wed Oct 19 21:40:49 UTC 2016: Installer will install image on ieusb
13:20:29.876 § Wed Oct 19 21:40:49 UTC 2016: Removing old volumes
13:20:30.598 § Wed Oct 19 21:40:49 UTC 2016: Inside Volume-Cleaning Function
13:20:30.604 § Wed Oct 19 21:40:49 UTC 2016: Removed LVM for Panini
13:20:30.612 § Wed Oct 19 21:40:49 UTC 2016: Removed LVM for App-Host
13:20:30.620 § Wed Oct 19 21:40:49 UTC 2016: Removed App-Vol Grp
13:20:30.668 § Wed Oct 19 21:40:50 UTC 2016: Removed Panini Vol-Grp
13:20:30.672 § Wed Oct 19 21:40:50 UTC 2016: Formatted App-Volume
13:20:30.696 § Wed Oct 19 21:40:50 UTC 2016: Removed PV
13:20:30.697 § Wed Oct 19 21:40:50 UTC 2016: Exiting from the Volume Cleaning Section
13:20:30.716 § Wed Oct 19 21:40:50 UTC 2016: 
13:20:30.976 § Wed Oct 19 21:40:50 UTC 2016: ---Starting to prepare ieusb---
13:20:30.985 § Wed Oct 19 21:40:52 UTC 2016: Creating partitions, BOOT=550MB, LVM=22500MB, EFI=30MB
13:20:31.008 § Wed Oct 19 21:40:53 UTC 2016: Partition creation on /dev/ieusb took 3 seconds
13:20:31.642 § Wed Oct 19 21:40:54 UTC 2016: File system creation on /dev/ieusb1 took 1 seconds
13:20:31.649 § Wed Oct 19 21:40:54 UTC 2016: Install boot image on /dev/ieusb1
13:21:06.870 § Wed Oct 19 21:41:29 UTC 2016: Starting Calvados patch for LXC for hostos
13:21:07.013 § Wed Oct 19 21:41:29 UTC 2016: Uninstalling rpm gdb
13:21:07.117 § error: failed to open /etc/mtab: No such file or directory
13:21:11.177 § Wed Oct 19 21:41:30 UTC 2016 (/etc/init.d/calvados_patch_lxc_iso.sh): Disable DHCP on host eth, eth1
13:21:11.186 § NCS5K: Complete Patch Host
13:21:11.219 § Wed Oct 19 21:41:30 UTC 2016: Finished Calvados patch for LXC
13:21:12.395 § Wed Oct 19 21:41:35 UTC 2016: Installing host image size of 459M took 41 seconds
13:21:12.547 § Wed Oct 19 21:41:35 UTC 2016: 
13:21:12.555 § Wed Oct 19 21:41:35 UTC 2016: ---Starting to prepare host logical volume---
13:21:59.756 § Wed Oct 19 21:42:21 UTC 2016: Starting Calvados patch for LXC for hostos
13:21:59.775 § Wed Oct 19 21:42:21 UTC 2016: Uninstalling rpm gdb
13:21:59.985 § error: failed to open /etc/mtab: No such file or directory
13:22:00.127 § Wed Oct 19 21:42:22 UTC 2016 (/etc/init.d/calvados_patch_lxc_iso.sh): Disable DHCP on host eth, eth1
13:22:00.242 § NCS5K: Complete Patch Host
13:22:01.583 § Wed Oct 19 21:42:22 UTC 2016: Finished Calvados patch for LXC
13:22:21.073 § Wed Oct 19 21:42:43 UTC 2016: 
13:22:21.074 § Wed Oct 19 21:42:43 UTC 2016: ---Starting to prepare calvados logical volume---
13:22:23.153 § Wed Oct 19 21:42:44 UTC 2016: Create sub partition on /dev/panini_vol_grp/calvados_lv0
13:22:30.035 § Wed Oct 19 21:42:46 UTC 2016: Create data sub partition on /dev/panini_vol_grp/calvados_data_lv0
13:22:44.332 § Wed Oct 19 21:43:07 UTC 2016: File system creation on /dev/panini_vol_grp/calvados_lv0 took 19 seconds
13:22:44.480 § Wed Oct 19 21:43:07 UTC 2016: Install sysadmin-vm image on /dev/panini_vol_grp/calvados_lv0
13:23:00.051 § Wed Oct 19 21:43:14 UTC 2016: sysadmin-vm: RP based installation
13:23:14.419 § Wed Oct 19 21:43:37 UTC 2016: Install RPM with --excludepath=/opt/cisco/calvados/packages/ncs5k-sysadmin-hostos-6.0.0-r600/lc --excludepath=/opt/cisco/calvados/packages/ncs5k-sysadmin-hostos-6.0.0-r600/sc --excludepath=/opt/cisco/calvados/packages/ncs5k-sysadmin-hostos-6.0.0-r600/xc --excludepath=/opt/cisco/calvados/packages/ncs5k-sysadmin-hostos-6.0.0-r600/fc
13:23:22.105 § Wed Oct 19 21:43:45 UTC 2016: Install RPM with --excludepath=/opt/cisco/calvados/packages/ncs5k-sysadmin-mgbl-6.0.0-r600/lc --excludepath=/opt/cisco/calvados/packages/ncs5k-sysadmin-mgbl-6.0.0-r600/sc --excludepath=/opt/cisco/calvados/packages/ncs5k-sysadmin-mgbl-6.0.0-r600/xc --excludepath=/opt/cisco/calvados/packages/ncs5k-sysadmin-mgbl-6.0.0-r600/fc
13:23:29.563 § !Wed Oct 19 21:43:52 UTC 2016: Install RPM with --excludepath=/opt/cisco/calvados/packages/ncs5k-sysadmin-ncs5k-6.0.0-r600/lc --excludepath=/opt/cisco/calvados/packages/ncs5k-sysadmin-ncs5k-6.0.0-r600/sc --excludepath=/opt/cisco/calvados/packages/ncs5k-sysadmin-ncs5k-6.0.0-r600/xc --excludepath=/opt/cisco/calvados/packages/ncs5k-sysadmin-ncs5k-6.0.0-r600/fc
13:23:37.112 § Wed Oct 19 21:43:59 UTC 2016: Install RPM with --excludepath=/opt/cisco/calvados/packages/ncs5k-sysadmin-shared-6.0.0-r600/lc --excludepath=/opt/cisco/calvados/packages/ncs5k-sysadmin-shared-6.0.0-r600/sc --excludepath=/opt/cisco/calvados/packages/ncs5k-sysadmin-shared-6.0.0-r600/xc --excludepath=/opt/cisco/calvados/packages/ncs5k-sysadmin-shared-6.0.0-r600/fc
13:23:45.881 § Wed Oct 19 21:44:08 UTC 2016: Install RPM with --excludepath=/opt/cisco/calvados/packages/ncs5k-sysadmin-system-6.0.0-r600/lc --excludepath=/opt/cisco/calvados/packages/ncs5k-sysadmin-system-6.0.0-r600/sc --excludepath=/opt/cisco/calvados/packages/ncs5k-sysadmin-system-6.0.0-r600/xc --excludepath=/opt/cisco/calvados/packages/ncs5k-sysadmin-system-6.0.0-r600/fc
13:23:58.434 § Wed Oct 19 21:44:17 UTC 2016: Install RPM with --excludepath=/opt/cisco/calvados/packages/ncs5k-sysadmin-topo-6.0.0-r600/lc --excludepath=/opt/cisco/calvados/packages/ncs5k-sysadmin-topo-6.0.0-r600/sc --excludepath=/opt/cisco/calvados/packages/ncs5k-sysadmin-topo-6.0.0-r600/xc --excludepath=/opt/cisco/calvados/packages/ncs5k-sysadmin-topo-6.0.0-r600/fc
13:24:08.143 § sed: can't read /tmp/partmnt.8awMzI/etc/sysconfig/network: No such file or directory
13:24:08.151 § Wed Oct 19 21:44:31 UTC 2016: Starting Calvados patch for LXC for sysadmin-vm
13:24:08.236 § Wed Oct 19 21:44:31 UTC 2016: Uninstalling rpm gdb
13:24:08.307 § error: failed to open /etc/mtab: No such file or directory
13:24:09.038 § Wed Oct 19 21:44:31 UTC 2016: Uninstalling rpm smartmontools
13:24:09.081 § error: failed to open /etc/mtab: No such file or directory
13:24:09.817 § Wed Oct 19 21:44:32 UTC 2016: Finished Calvados patch for LXC
13:24:17.785 § Wed Oct 19 21:44:40 UTC 2016: Installing sysadmin-vm image size of 566M took 93 seconds
13:24:19.747 § Wed Oct 19 21:44:42 UTC 2016: 
13:24:19.825 § Wed Oct 19 21:44:42 UTC 2016: ---Starting to prepare repository---
13:24:24.870 § Wed Oct 19 21:44:47 UTC 2016: File system creation on /dev/ieusb2 took 4 seconds
13:24:24.937 § Wed Oct 19 21:44:47 UTC 2016: Copying /iso/host.iso to repository /iso directory
13:24:25.117 § Wed Oct 19 21:44:48 UTC 2016: Copying /iso/ncs5k-sysadmin.iso to repository /iso directory
13:24:32.617 § Wed Oct 19 21:44:55 UTC 2016: Copying /iso/ncs5k-xr.iso to repository /iso directory
13:24:57.168 § Wed Oct 19 21:45:20 UTC 2016: Copying all ISOs to repository took 33 seconds
13:25:08.340 § Wed Oct 19 21:45:31 UTC 2016: Install EFI on /dev/ieusb4
13:25:09.909 § 44107 blocks
13:25:09.958 § 44109 blocks
13:25:12.989 § Wed Oct 19 21:45:35 UTC 2016: Install finished on ieusb
13:25:13.222 § Automatic rebooting system after installation ...
13:25:13.223 § Syncing file system
13:25:15.703 § [  298.589761] reboot: Restarting system
13:25:27.263 § 
13:25:27.263 § CISCO NCS5K XR ROUTER Ver 1.8
13:25:27.264 § Booting from Primary Bios 
13:25:27.264 § Code Signing Results: 0x0 
13:25:27.264 § Using Upgrade FPGA
13:25:27.264 § FPGA Revision        : 0x10
13:25:27.265 § FPGA ID             : 0x1
13:25:27.265 § FPGA Date           : 0x20141016
13:25:27.265 § Reset Cause Register: 0x80000000
13:25:27.265 § SPI Flash Boot Timer Ctrl Status Register  : 0xe0ff
13:25:27.265 § Persistent Power Debug Register1 : 0x0
13:25:27.266 § Persistent Power Debug Register2 : 0x0
13:25:27.266 § FPGA Remote System Update Status Register: 0x20
13:25:27.290 § Detected CISCO MIFPGA
13:25:27.291 § FPGA Update Status  : 0x20
13:25:32.169 § PASSED SecureBoot verification
13:25:32.180 § 
13:25:32.180 § 
13:25:32.180 § GNU GRUB version 2.00
13:25:32.180 § Press F2 to goto grub Menu..
13:25:34.258 § Booting from Disk..
13:25:34.677 § Loading Kernel..
13:25:35.513 § Loading initrd..
13:25:36.977 § Initrd, addr=0xff69a000, size=0x955730
13:25:38.857 § [    1.466598] i8042: No controller found
13:25:39.902 § NCS5K init
13:25:42.303 § [    4.936658] sd 5:0:0:0: [sdb] No Caching mode page found
13:25:42.306 § [    4.960572] sd 5:0:0:0: [sdb] No Caching mode page found
13:25:42.306 § [    4.991701] sd 5:0:0:0: [sdb] No Caching mode page found
13:25:44.841 § Attached devices:
13:25:44.841 § Host: scsi4 Channel: 00 Id: 00 Lun: 00
13:25:44.850 §   Vendor: SanDisk  Model: Extreme          Rev: 0001
13:25:44.850 §   Type:   Direct-Access                    ANSI  SCSI revision: 06
13:25:44.858 § Host: scsi5 Channel: 00 Id: 00 Lun: 00
13:25:44.859 §   Vendor: SMART    Model: eUSB  HS-SD/MMC  Rev: 2.11
13:25:44.870 §   Type:   Direct-Access                    ANSI  SCSI revision: 00
13:25:44.871 § Number of disks detected: 2: Disk Check for: /dev/sdb 
13:25:44.871 § Detected /dev/sdb
13:25:44.871 § NCS5K init: End
13:25:45.753 § mount: mount point /run does not exist
13:25:45.753 § Switching to new root and running init.
13:25:45.753 § switch_root: failed to mount moving /run to /sysroot/run: No such file or directory
13:25:45.754 § switch_root: forcing unmount of /run
13:25:46.172 § mkdir: cannot create directory '/run': File exists
13:25:47.633 § Sourcing /etc/sysconfig/udev
13:25:51.292 § Starting udev: [  OK  ]
13:25:51.293 § mount: none is already mounted or /dev/pts busy
13:25:51.293 §        none is already mounted on /sys
13:25:51.293 §        none is already mounted on /dev
13:25:51.294 §        none is already mounted on /dev/pts
13:25:51.294 §        none is already mounted on /dev/shm
13:25:51.294 § Setting hostname localhost:  [  OK  ]
13:25:51.302 § Checking filesystems:[  OK  ]
13:25:51.463 §  Successfully executed rc sysinit 
13:25:53.687 § Starting udev
13:25:56.718 § Populating dev cache
13:25:57.228 § Running postinst /etc/rpm-postinsts/100-dnsmasq...
13:25:57.237 § update-rc.d: /etc/init.d/run-postinsts exists during rc.d purge (continuing)
13:25:57.237 §  Removing any system startup links for run-postinsts ...
13:25:57.238 §   /etc/rcS.d/S99run-postinsts
13:25:57.273 § Configuring network interfaces... done.
13:25:57.646 § Starting system message bus: dbus.
13:25:57.722 § Starting OpenBSD Secure Shell server: sshd
13:25:57.723 §   generating ssh RSA key...
13:25:57.977 §   generating ssh ECDSA key...
13:25:57.977 §   generating ssh DSA key...
13:25:57.978 §   generating ssh ED25519 key...
13:25:57.988 § done.
13:25:58.022 § Starting rpcbind daemon...done.
13:26:02.060 § Starting kdump:[  OK  ]
13:26:02.359 § Starting libvirtd daemon: [  OK  ]
13:26:02.560 § Starting system log daemon...0
13:26:02.655 § Starting kernel log daemon...0
13:26:02.677 § Starting HPA's tftpd: in.tftpd-hpa
13:26:02.687 § .
13:26:02.703 § Starting internet superserver: xinetd.
13:26:02.723 § net.ipv4.ip_forward = 1
13:26:02.843 § Starting S.M.A.R.T. daemon: smartd (failed)
13:26:03.401 § Starting Lighttpd Web Server: lighttpd.
13:26:03.467 § Starting crond: OK
13:26:03.494 § Starting cgroup-init
13:26:03.612 § Starting programs for RP on hostos: [  OK  ]
13:26:03.612 § kernel.sched_latency_ns = 20000000
13:26:03.616 § kernel.sched_min_granularity_ns = 4000000
13:26:03.640 § Load the igb module
13:26:04.370 § Bringing up eth0 ... 
13:26:08.522 § Network ieobc_br defined from /etc/init/ieobc_br_network.xml
13:26:08.522 § 
13:26:09.255 § Network local_br defined from /etc/init/local_br_network.xml
13:26:09.256 § 
13:26:09.344 § Network xr_local_br defined from /etc/init/xr_local_br_network.xml
13:26:09.344 § 
13:26:09.782 § Network ieobc_br started
13:26:09.787 § 
13:26:09.871 § Network local_br started
13:26:09.871 § 
13:26:10.200 § Network xr_local_br started
13:26:10.200 § 
13:26:11.334 § Creating default host password file
13:26:11.353 § 
13:27:39.881 § 
13:27:39.881 § #### Waiting for 'default-sdr--1' lxc to be ready. ####
13:27:39.882 § 
13:27:39.891 § If you want to connect to available console, press 'Ctrl-o' <'o' as in orange>

[output omitted] 

13:34:25.196 § 
13:34:25.197 § Connecting to 'default-sdr--1' console
13:34:31.771 § ÿûÿûÿûÿýmount: can't find /tmp in /etc/fstab
13:34:32.927 § Setting hostname host:  [  OK  ]
13:34:32.928 § Checking filesystems:[  OK  ]
13:34:33.018 §  Successfully executed rc sysinit 
13:34:38.456 § Mount /dev/mapper/panini_vol_grp-ssd_disk1_xr_1 at /misc/disk1
13:34:46.706 § Running postinst /etc/rpm-postinsts/100-dnsmasq...
13:34:46.711 § update-rc.d: /etc/init.d/run-postinsts exists during rc.d purge (continuing)
13:34:46.711 §  Removing any system startup links for run-postinsts ...
13:34:46.711 §   /etc/rcS.d/S99run-postinsts
13:34:46.731 § Configuring network interfaces... done.
13:34:46.955 § Starting system message bus: dbus.
13:34:47.018 § Starting OpenBSD Secure Shell server: sshd
13:34:47.019 §   generating ssh RSA key...
13:34:47.221 §   generating ssh ECDSA key...
13:34:47.222 §   generating ssh DSA key...
13:34:47.238 §   generating ssh ED25519 key...
13:34:47.254 § done.
13:34:47.287 § Starting rpcbind daemon...done.
13:34:47.354 § Libvirt not initialized for container instance
13:34:47.366 § Starting system log daemon...0
13:34:47.381 § Starting HPA's tftpd: in.tftpd-hpa
13:34:47.382 § .
13:34:47.397 § Starting internet superserver: xinetd.
13:34:47.436 § Starting crond: OK
13:34:47.511 § Starting programs for RP on xr-vm: [  OK  ]
13:34:51.513 § [warn] Epoll ADD(8) on fd 50 failed.  Old events were 0; read change was 0 (none); write change was 0 (none); error change was 1 (add): Operation not permitted
13:35:07.401 § 
13:35:07.401 § 
13:35:07.401 § ios con0/RP0/CPU0 is now available
13:35:07.401 § 
13:35:07.402 § 
13:35:07.402 § 
13:35:07.402 § 
13:35:07.402 § 
13:35:07.402 § Press RETURN to get started.
13:35:07.403 § 
13:35:21.016 § 

[output omitted] 

13:37:15.130 § 
13:37:16.307 §   Enter root-system username: root
13:37:17.238 §   Enter secret: 
13:37:20.367 § Use the 'configure' command to modify this configuration.
13:37:20.368 § User Access Verification
13:37:20.368 § 
13:37:22.525 § Username: root
13:37:23.340 § Password: 
13:37:23.366 § 
13:37:23.367 § 
13:37:25.059 § RP/0/RP0/CPU0:ios#show run
13:37:25.104 § Wed Oct 19 21:57:44.794 UTC
13:37:25.374 § Building configuration...
13:37:27.264 § !! IOS XR Configuration version = 6.0.0
13:37:27.264 § !! Last configuration change at Wed Oct 19 21:57:38 2016 by SYSTEM
13:37:27.264 § !
13:37:27.264 § username root
13:37:27.265 §  group root-lr
13:37:27.266 §  group cisco-support
13:37:27.279 §  secret 5 $1$.krS$mkholzmdUdW6gHjOgDyPp/
13:37:27.279 § !
13:37:27.279 § interface MgmtEth0/RP0/CPU0/0
13:37:27.279 §  shutdown
13:37:27.280 § !
13:37:27.280 § interface MgmtEth0/RP0/CPU0/1
13:37:27.280 §  shutdown

[output omitted]

References

NCS5K Documentation - Disaster Recovery

NCS5K Documentation - Create a Bootable USB Drive

NCS5K Documentation - Boot the Router using USB

Above process did only work partially for me:

- in BIOS prompt menu USB stick was not 'visible': in BIOS 'Boot from File' menu there should be a 'usb' option to select and to start the usb-boot-image from (eg "Built-in GrubEFI USB Device (USB DISK 2.0)" or similar).

- turned out that the FPD/bootloader version was not sufficient (show hw-module fpd

0/RP0 NC55-RP-E 1.1 Bootloader CURRENT 1.18 1.18

0/RP0 NC55-RP-E 1.1 IOFPGA CURRENT 0.21 0.21

0/RP0 NC55-RP-E 1.1 OMGFPGA CURRENT 0.48 0.48

0/RP1 NC55-RP-E 1.1 Bootloader NEED UPGD 1.14 0.00

0/RP1 NC55-RP-E 1.1 IOFPGA CURRENT 0.21 0.21

0/RP1 NC55-RP-E 1.1 OMGFPGA CURRENT 0.48 0.48

So suposedly, if that version would have been higher, USB should have been visible in 'boot from file' menu.

Chicken-and-egg problem: without upgrade no USB boot possible, but if RP is not up it cannot be upgraded.

In the end we solved this by inserting the 'faulty' RP in a 'working' chassis (with a running/'active' RP) and flashed/sync'd the 'faulty' RP to the working one using this (admin!) command:

sysadmin-vm:0_RP0# hw-module location 0/RP1 bootmedia network reload

NB: in order to execute that command in admin mode the RP has to be visible on the active RP (show platform)!

NB: On a NCS5500 RP there are two USB slots (just under the console port). Apparently the TOP slot has to be used for this USB recovery procedure. the BOTTOM one is only for console access (TBC!).

Thursday, December 31, 2020

Linux online Expansion Harddisk

step 1

Frist edit harddisk on ESXi or VCSA

setp 2

then login the Linux server， check “fdisk -l” command.

root@eve-ng:~# fdisk -l

Disk /dev/sdb: 800 GiB, 858993459200 bytes, 1677721600 sectors

Units: sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

if Disk /dev/sdx eq your set harddisk, please go to setp 3.

if disk not eq set harddisk, please use "partprobe /dev/sdx" command tell kernel re-read the harddisk.

root@eve-ng:~# partprobe /dev/sdb

root@eve-ng:~# fdisk -l

Disk /dev/sdb: 900 GiB, 966367641600 bytes, 1887436800 sectors

Units: sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

setp 3

Expansion PV

root@eve-ng:~# pvdisplay /dev/sdb

--- Physical volume ---

PV Name /dev/sdb

VG Name eve-ng-vg

PV Size <800.00 GiB / not usable 3.00 MiB

Allocatable yes (but full)

PE Size 4.00 MiB

Total PE 204799

Free PE 0

Allocated PE 204799

PV UUID KqNZmM-AvG0-68yt-WBXa-DMvO-jorh-nCUzyH

root@eve-ng:~# pvresize /dev/sdb

Physical volume "/dev/sdb" changed

1 physical volume(s) resized / 0 physical volume(s) not resized

root@eve-ng:~# pvdisplay /dev/sdb

--- Physical volume ---

PV Name /dev/sdb

VG Name eve-ng-vg

PV Size <900.00 GiB / not usable 3.00 MiB

Allocatable yes

PE Size 4.00 MiB

Total PE 230399

Free PE 25600

Allocated PE 204799

PV UUID KqNZmM-AvG0-68yt-WBXa-DMvO-jorh-nCUzyH

root@eve-ng:~#

step 4

Check VG

root@eve-ng:~# vgdisplay eve-ng-vg

--- Volume group ---

VG Name eve-ng-vg

System ID

Format lvm2

Metadata Areas 2

Metadata Sequence No 11

VG Access read/write

VG Status resizable

MAX LV 0

Cur LV 2

Open LV 2

Max PV 0

Cur PV 2

Act PV 2

VG Size 949.99 GiB

PE Size 4.00 MiB

Total PE 243198

Alloc PE / Size 217598 / 849.99 GiB

Free PE / Size 25600 / 100.00 GiB

VG UUID M1Jx3e-XvsF-HMDR-HOi5-9vur-FakF-GWfMMl

Setp 5

extend LV

root@eve-ng:~# lvextend -l +100%FREE /dev/eve-ng-vg/root

Size of logical volume eve-ng-vg/root changed from <849.04 GiB (217353 extents) to <949.04 GiB (242953 extents).

Logical volume eve-ng-vg/root successfully resized.

root@eve-ng:~# lvdisplay /dev/eve-ng-vg/root

--- Logical volume ---

LV Path /dev/eve-ng-vg/root

LV Name root

VG Name eve-ng-vg

LV UUID X7yq2F-TVJR-58Tw-mOt2-3RVi-CegE-4hZjZn

LV Write Access read/write

LV Creation host, time eve-ng, 2020-06-26 04:21:37 +0800

LV Status available

# open 1

LV Size <949.04 GiB

Current LE 242953

Segments 2

Allocation inherit

Read ahead sectors auto

- currently set to 256

Block device 253:0

root@eve-ng:~#

Step 6

resize LV

root@eve-ng:~# df -h

Filesystem Size Used Avail Use% Mounted on

udev 63G 0 63G 0% /dev

tmpfs 13G 20M 13G 1% /run

/dev/mapper/eve--ng--vg-root 836G 801G 470M 100% /

tmpfs 63G 0 63G 0% /dev/shm

tmpfs 5.0M 0 5.0M 0% /run/lock

tmpfs 63G 0 63G 0% /sys/fs/cgroup

overlay 836G 801G 470M 100% /var/lib/docker/overlay2/17794a2f84f5ddf30ed690cc8b0e50753b385c0d845fd60bf9fc5e3a3f037aeb/merged

tmpfs 13G 0 13G 0% /run/user/0

root@eve-ng:~# resize2fs /dev/eve-ng-vg/root

resize2fs 1.44.1 (24-Mar-2018)

Filesystem at /dev/eve-ng-vg/root is mounted on /; on-line resizing required

old_desc_blocks = 107, new_desc_blocks = 119

The filesystem on /dev/eve-ng-vg/root is now 248783872 (4k) blocks long.

root@eve-ng:~# df -h

Filesystem Size Used Avail Use% Mounted on

udev 63G 0 63G 0% /dev

tmpfs 13G 20M 13G 1% /run

/dev/mapper/eve--ng--vg-root 934G 801G 95G 90% /

tmpfs 63G 0 63G 0% /dev/shm

tmpfs 5.0M 0 5.0M 0% /run/lock

tmpfs 63G 0 63G 0% /sys/fs/cgroup

overlay 934G 801G 95G 90% /var/lib/docker/overlay2/17794a2f84f5ddf30ed690cc8b0e50753b385c0d845fd60bf9fc5e3a3f037aeb/merged

tmpfs 13G 0 13G 0% /run/user/0

root@eve-ng:~#

Done

Sunday, December 6, 2020

Cisco IOS/IOS-XE show clock与show logging记录的时间不一致。

请在设备上配置如下命令后，保存即可：

service timestamps log datetime localtime