vManage Configuration DB slowness

 In NMS (vManage) releases 20.3.1, 20.3.2, 20.3.2.1, 20.4.1, some customers started experiencing slowness. The slowness was in the form of general user experience: Navigating through vManage pages was slow, GUI pages sometimes did not come up and timed out, and tasks submitted such as template push took significantly longer.

 

After investigation of why some customers were experiencing that, a resolution was identified where a vManage configuration-db setting change to be made. This setting change will become permanent in NMS (vManage) software 20.3.3, 20.4.1 IR, 20.5.1 and subsequent releases.

 

The setting change was made under CSCvw76649.

 

The change causes the query planner to use the ”slotted” runtime, which is different from the default runtime setting. It results in additional optimizations causing improvements to both the performance and memory usage of the query.

 

Given that slowness is a generic symptom, there is a need to identify whether this issue in the bug above is the needed remedy. To check, view the file /var/log/nms/query.log and check the planning phase duration. Example is "planning: 5000" which means that the planning took 5000 msec. Any query with planning phase taking around 5000 msec is considered very high. And the setting introduced in this change is appropriate.

 

The change is done by stopping app servers on all cluster instances, stopping config-dbs on all cluster instances, viewing the file /opt/web-app/etc/config_templates/neo4j.conf of all config-db instances, then adding the following at the end of the file: "unsupported.cypher.runtime=slotted". Then starting config-dbs. When checking the file /var/log/nms/debug.log on config-db instances, you should see the following string being printed out: “unsupported.cypher.runtime=slotted”. Finally, start app-servers.

Policy scale estimates on vEdge devices

 VEDGE policy scale depends on the amount of system memory in the fast path carved out for filters/policy.

 

Filter means memory is used for policy filters.

 

Number of policies and system memory scale depends on the platform:

 

vEdge-5000, vEdge-2000, vEdge-1000: Max number of sequences in the policies is 512 and Max filter blocks - 1024*1024 (Size 128MB, each block 128 Bytes)

vEdge100B-2GB:Max number of sequences in the policies is 512, filter blocks -1024*1024

vEdge-100-1GB:Max number of sequences in the policies is 256 and Max filter blocks: 16*1024

vEdge Cloud Medium:Max number of sequences in the policies is 256 and Max filter blocks: 64*1024

vEdge Cloud Small:Max number of sequences in the policies is 256 and Max filter blocks: 6*1024

 

 

Device type	Max sequences	Max filter 128B-blocks
V100B-2GB	512	1024*1024
V1000	512	1024*1024
V2000	512	1024*1024
V5000	512	1024*1024
vEdge-Cloud 2 vCPU/2G RAM	256	64*1024
vEdge-Cloud 4 vCPU/4G RAM	512	1024*1024
vEdge-Cloud 8 vCPU/8G RAM	512	1024*1024

 

128MB will be able to support large number of sequences. Example 100 policies, each policy with 1K sequences.

 

• Limits
• V100B-2GB/V1k/2k/5k: 512 sequences OR 1024*1024 = 1,048,576 filter blocks of 128 bytes (dedicated memory of 134,217,728 bytes ~ 128MB or 1,073,741,824 bits > ~1024 Mbits)
• Vedge cloud: 256 sequences max OR 64*1024 = 65536 filter blocks of 128 bytes (dedicated memory of 8388608 bytes or 671088646 bits > ~64 Mbits).
• Usage
• 1 match statement takes 1 or more filter blocks depending on its complexity
• 1 action statement takes 1 or more filter blocks depending on its complexity

All the policies in a system shares available filter blocks.

Number of filter blocks required for policy depends on the combination of # of sequences / match / actions.

 

Example: For a platform with 128MB filter memory can support 1 policy with 200 sequences, or 5 policies with 40 sequences each etc.

 

Memory required per policy = Number of sequences * number of match field types (example src-ip, dest-ip, src-port, dest-port etc.) * number of match fields (example: Number of IP prefixes, num ports etc.)* number of action fileds .

 

Number of policies and sequences per policy depends on the customer policy configuration.

 

Part of 19.3 release, you can get the usage with below operational command. 

vm1# show policy filter-memory-usage

 

FILTER  FILTER  BLOCKS  BLOCKS 

USED    MAX     USED    MAX    

--------------------------------

0       256     0       6144

Verify TLS/DTLS version supported by SDWAN Controllers

 

Introduction

 

This document provides details on how to find/verify TLS/DTLS versions supported by any Cisco SDWAN Controllers

 

Major TLS and DTLS Version

 

Major TLS versions are

• TLSv1.0
• TLSv1.1
• TLSv1.2
• TLSv1.3 (not supported by Cisco SDWAN Controllers yet)

DTLS is nothing but Datagram TLS which is a UDP Implementation of TLS.  

Major DTLS versions are

• DTLSv1.0 (UDP Implementation of TLSv1.1)
• DTLVv1.2 (UDP Implementation of TLSv1.2)

Note: There is no DTLSv1.1

 

Solution/Verification

 

To verify the version of TLS/DTLS version supported by the SDWAN contollers, we need to have a PC with OpenSSL 1.0.2 or above.

Alternatively, we can also try this from the vManage vShell where required OpenSSL library is used

 

To find if a particular TLS/DTLS version is enabled on the server, we can send the TLS/DTLS Client Hello with a particular version as per below format

 

 

For TLSv1
   openssl s_client -connect <server-name>:<server-port> -tls1

For TLSv1.1
   openssl s_client -connect <server-name>:<server-port> -tls1_1

For TLSv1.2
   openssl s_client -connect <server-name>:<server-port> -tls1_2

For DTLSv1
   openssl s_client -connect <server-name>:<server-port> -dtls1

For DTLSv1.2
   openssl s_client -connect <server-name>:<server-port> -dtls1_2

 

 

ServerName is the SDWAN Controller's VPN 0 IP address or FQDN (incase of vBond)

Server Port is the SDWAN Controller's port number to which the TLS connection is established.

 

From the outputs for above, if the server does not support the desired TLS/DTLS version on which the Client Hello was sent, the server will not send the server ceritifcate and we will see message - "no peer certificate available"

 

If the server (SDWAN Controller) supports the version of TLS/DTLS version on which the Client Hello was sent, the server will send the server certificate in its response.

 

Example:

 

In the below example, we find that the vSmart did not send server certificates for a TLS Client Hello sent on TLSv1.0

Hence, TLSv1.0 is not supported by the vSmart

 

 

vManage:~$ openssl s_client -connect 10.197.250.75:23456 -tls1
CONNECTED(00000004)
140372110214848:error:1409442E:SSL routines:ssl3_read_bytes:tlsv1 alert protocol version:s3_pkt.c:1498:SSL alert number 70
140372110214848:error:1409E0E5:SSL routines:ssl3_write_bytes:ssl handshake failure:s3_pkt.c:659:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 0 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1
    Cipher    : 0000
    Session-ID:
    Session-ID-ctx:
    Master-Key:
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1589733451
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
---
vManage:~$

 

 

In the below example, we find that the vSmart has sent server certificates for a TLS Client Hello sent on TLSv1.2

Hence, TLSv1.2 is supported by the vSmart

 

 

vManage:~$ openssl s_client -connect 10.197.250.75:23456 -tls1_2
CONNECTED(00000004)
depth=0 C = US, ST = California, L = San Jose, O = "Cisco Systems, Inc.", OU = titan-viptela, CN = vsmart-c7eaf390-98d5-4aa2-a991-cc451f86f8cd-1.viptela.com
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 C = US, ST = California, L = San Jose, O = "Cisco Systems, Inc.", OU = titan-viptela, CN = vsmart-c7eaf390-98d5-4aa2-a991-cc451f86f8cd-1.viptela.com
verify error:num=21:unable to verify the first certificate
verify return:1
---
Certificate chain
 0 s:/C=US/ST=California/L=San Jose/O=Cisco Systems, Inc./OU=titan-viptela/CN=vsmart-c7eaf390-98d5-4aa2-a991-cc451f86f8cd-1.viptela.com
   i:/O=Cisco/OU=Albireo/CN=Viptela SubCA
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIFqTCCA5GgAwIBAgIUUXveczeJhorYJWQow6GLfg8YGFIwDQYJKoZIhvcNAQEL
BQAwOjEOMAwGA1UECgwFQ2lzY28xEDAOBgNVBAsMB0FsYmlyZW8xFjAUBgNVBAMM
DVZpcHRlbGEgU3ViQ0EwHhcNMjAwNTE3MTYxOTM2WhcNMjAwODE3MTYyOTAwWjCB
rzELMAkGA1UEBhMCVVMxEzARBgNVBAgMCkNhbGlmb3JuaWExETAPBgNVBAcMCFNh
biBKb3NlMRwwGgYDVQQKDBNDaXNjbyBTeXN0ZW1zLCBJbmMuMRYwFAYDVQQLDA10
aXRhbi12aXB0ZWxhMUIwQAYDVQQDDDl2c21hcnQtYzdlYWYzOTAtOThkNS00YWEy
LWE5OTEtY2M0NTFmODZmOGNkLTEudmlwdGVsYS5jb20wggEiMA0GCSqGSIb3DQEB
AQUAA4IBDwAwggEKAoIBAQDjc5SJFAEBmWdcnQdFpl+mYdKoSvtja3euo6G0ulK4
LSzYDL1huXDzjw+MbEpgXS3fwp6gGu46C+HoAiBzkVRyDCz/kbii95mL5w63lWYf
kRPkvPF/nRCPDjy3ITBfuDqJI+UFsueO1b2mGl/Jre6kv91LNYBlFlBTKqzl5NpL
yVHdDH+t2zRlFsiWS37/VnOfj1rQ9Y2C99+RVoYagTipo3wfTghSb4PKi/ZnLTJA
kOUUJXCPhm5lxZjqc3uaBcSjS31F2B5JZzMXSz7dn4d1ndeH+oHMhd3lM2vkXkr+
E6NOrstMgnHWXkxv5wwq57M7v6wl8zjxr98iCc3BCyGnAgMBAAGjggEvMIIBKzAM
BgNVHRMBAf8EAjAAMB8GA1UdIwQYMBaAFOzZ2kaSWq3vZfoQTgkziLp3X9fvMIGB
BggrBgEFBQcBAQR1MHMwQgYIKwYBBQUHMAKGNmh0dHA6Ly9jcmwuaW50ZXJuYWxo
b3N0bmFtZXMuY29tL0NJU0NPVklQVEVMQVNVQkNBLmNydDAtBggrBgEFBQcwAYYh
aHR0cDovL29jc3AuaW50ZXJuYWxob3N0bmFtZXMuY29tMEcGA1UdHwRAMD4wPKA6
oDiGNmh0dHA6Ly9jcmwuaW50ZXJuYWxob3N0bmFtZXMuY29tL0NJU0NPVklQVEVM
QVNVQkNBLmNybDAdBgNVHQ4EFgQU3HBlp59mPTOG8ktHPIwgVYmQ7SAwDgYDVR0P
AQH/BAQDAgWgMA0GCSqGSIb3DQEBCwUAA4ICAQB/Sb6FkiAYfd8B7D4UzvoALkSk
/0mqUksOWMKwR+ZIK1X8hKIFqPLGEsvPseuTMnoXBN5znzN3jXrgAytMII/r/Hsy
C2T7KlRD7nkQqZTWe8jsYvNFu21XGh/x4Pfaoi93FxFU9bHlRBlTLUB/DhVq+dbf
ygrq1X/SseYuarpL0xSFbmVwGExq3MRx35nBOt9z22a9tNdG4XsE9XEGqlOjJJ9O
JfnGUlJ5MOuk0axKnqmITsW3dxS5b0j3EItsxqAc0Lw1co2oSDaPi0h3K4MlUVmB
T1Q9mtgIgdvyvUkxnlCuw7avNoz52eMtNjv5W2WMarfcKQlbqN+M473tniS6PBYb
rfURXQ8EO8DVNrTDKY+1+nKmuY4wEuaILDWjpHDpPiPLaz6PRs7UU6GzNTfuHOX6
CcKEeB3LTTzrp66UlkyEpck5R4X9G+D72nww/+pC7CxCDxdnhQsCKp4fK8pCpDIc
b8D3UY62e2WpaLg8MAZEkkN9vzLWW7Xih4XwhP9Mls4vTDaZ8ZHj8flB2ZKl06fU
Aj7/7foDw4VMuBm/9F64YE8ZNnr7Xtqv3g+aq9USETeUdFQ7MIRbM9XnzjyEhrvO
9ae4Sp2ZAtsdvhjcu4Qv/5sqJr5X8b6AeywB7WSBvBu9KWzyw+/s1rnvgc1iB1q8
22cSUbQ+rpJ9btzLJA==
-----END CERTIFICATE-----
subject=/C=US/ST=California/L=San Jose/O=Cisco Systems, Inc./OU=titan-viptela/CN=vsmart-c7eaf390-98d5-4aa2-a991-cc451f86f8cd-1.viptela.com
issuer=/O=Cisco/OU=Albireo/CN=Viptela SubCA
---
No client certificate CA names sent
Client Certificate Types: RSA sign, DSA sign, ECDSA sign
Requested Signature Algorithms: RSA+SHA512:DSA+SHA512:ECDSA+SHA512:RSA+SHA384:DSA+SHA384:ECDSA+SHA384:RSA+SHA256:DSA+SHA256:ECDSA+SHA256:RSA+SHA224:DSA+SHA224:ECDSA+SHA224:RSA+SHA1:DSA+SHA1:ECDSA+SHA1
Shared Requested Signature Algorithms: RSA+SHA512:DSA+SHA512:ECDSA+SHA512:RSA+SHA384:DSA+SHA384:ECDSA+SHA384:RSA+SHA256:DSA+SHA256:ECDSA+SHA256:RSA+SHA224:DSA+SHA224:ECDSA+SHA224:RSA+SHA1:DSA+SHA1:ECDSA+SHA1
Peer signing digest: SHA512
Server Temp Key: ECDH, P-521, 521 bits
---
SSL handshake has read 2043 bytes and written 453 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID:
    Session-ID-ctx:
    Master-Key: 80AAA41228ADD93446784BB2D1103DE2ABD3C9076669074D3A4D2E2EDDA713F01CCE848A145342AF40DE7011353BBD53
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1589733724
    Timeout   : 7200 (sec)
    Verify return code: 21 (unable to verify the first certificate)
---                                                   

 

 

In the below example, we find that the vManage has sent server certificates for a DTLS Client Hello sent on DTLSv1.2

Hence, DTLSv1.2 is supported by the vManage

 

vManage:~$ openssl s_client -connect 10.197.250.74:12346 -dtls1_2
CONNECTED(00000004)
depth=0 C = US, ST = California, L = San Jose, O = "Cisco Systems, Inc.", OU = titan-viptela, CN = vmanage-f32e471a-10e3-4b29-bbda-b1b20cf03297-0.viptela.com
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 C = US, ST = California, L = San Jose, O = "Cisco Systems, Inc.", OU = titan-viptela, CN = vmanage-f32e471a-10e3-4b29-bbda-b1b20cf03297-0.viptela.com
verify error:num=21:unable to verify the first certificate
verify return:1
---
Certificate chain
 0 s:/C=US/ST=California/L=San Jose/O=Cisco Systems, Inc./OU=titan-viptela/CN=vmanage-f32e471a-10e3-4b29-bbda-b1b20cf03297-0.viptela.com
   i:/O=Cisco/OU=Albireo/CN=Viptela SubCA
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIFqjCCA5KgAwIBAgIUIPaJ6YXYPgba5h+SzFNRhJPFnuMwDQYJKoZIhvcNAQEL
BQAwOjEOMAwGA1UECgwFQ2lzY28xEDAOBgNVBAsMB0FsYmlyZW8xFjAUBgNVBAMM
DVZpcHRlbGEgU3ViQ0EwHhcNMjAwNTA0MjAyMDU1WhcNMjEwNTA0MjAzMDAwWjCB
sDELMAkGA1UEBhMCVVMxEzARBgNVBAgMCkNhbGlmb3JuaWExETAPBgNVBAcMCFNh
biBKb3NlMRwwGgYDVQQKDBNDaXNjbyBTeXN0ZW1zLCBJbmMuMRYwFAYDVQQLDA10
aXRhbi12aXB0ZWxhMUMwQQYDVQQDDDp2bWFuYWdlLWYzMmU0NzFhLTEwZTMtNGIy
OS1iYmRhLWIxYjIwY2YwMzI5Ny0wLnZpcHRlbGEuY29tMIIBIjANBgkqhkiG9w0B
AQEFAAOCAQ8AMIIBCgKCAQEAv2AYQQv1r39HMpc+WT0Apggh057AmAeo2u7i8y+7
yJCH5PCM1dN3cJL5kMZ2RJ9ujImtPxwc7x63/hq6ycDC2jJ7iU6S0EzqUhW4tzdg
JP+mQoE/5/osAQd2v2xadGfLWArCihqV1uA0tzrHeSTHUUA+sRkPWP/hBA78hC2b
QoQDyhf3INSOLjO50a8jtCMj+Uo6GMVKc8N5c2clCd7WQat2IvkZfFpBvS9xR3kF
WRNhPhVkS7nwGCxe2FkAOyjY/MPWZPnGzXpd0+pF/7ANNkDvbH4v2HeIpjnXFkfL
HCFLnZLHn1S6PLdh40sjX0yZ5a2EqXTmV/rdtQt3AjHo0wIDAQABo4IBLzCCASsw
DAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBTs2dpGklqt72X6EE4JM4i6d1/X7zCB
gQYIKwYBBQUHAQEEdTBzMEIGCCsGAQUFBzAChjZodHRwOi8vY3JsLmludGVybmFs
aG9zdG5hbWVzLmNvbS9DSVNDT1ZJUFRFTEFTVUJDQS5jcnQwLQYIKwYBBQUHMAGG
IWh0dHA6Ly9vY3NwLmludGVybmFsaG9zdG5hbWVzLmNvbTBHBgNVHR8EQDA+MDyg
OqA4hjZodHRwOi8vY3JsLmludGVybmFsaG9zdG5hbWVzLmNvbS9DSVNDT1ZJUFRF
TEFTVUJDQS5jcmwwHQYDVR0OBBYEFA7dS4GwWzarEgFu2IaEFq6vbxzMMA4GA1Ud
DwEB/wQEAwIFoDANBgkqhkiG9w0BAQsFAAOCAgEAOnyyNvVrubKjOV+67/+RId8r
OOnPb0dusbHv5mirgB2qzzpddC9ZP3de+ItxG5kEb8pdt5FaUkXzh8U6Ag/7O/sv
9ygucd+KzbCeCk2HCbnQQscKq0Hd/m29bt8IpaDlqkCLy4kCriJWmRc9T4tUFW0Z
KA5HV5lKuGOXwkrPjZH2W6qOeHdH1joXJg9MSqtsq0M6F82objk1ZfAx5CzJqNAb
FeCQYcc/HIV3n3QXXLj20mmj1Zzvk+o5b7vsPlmsvJl2JmAEGxQffPX10Ih7qkxc
Iby64gQkrGCPx6W65jE5osoAuHcDHh9nit6DqcBrw3wWsI+xrxl7hDONq4MP6dbg
u/SqvTd9f5eSQE/GXAh3OYjc8OWLEfikRsOSCAP7t3qIDvoBrui/2vJ2rFGw74CG
8HfXkCnIldaKF/oJNjApByXw+Q85XUoQOJKuoEyAl+2Tn6gjYg6JamGQZZrRvNEr
1X8kM+2k9LzoNJ/50WlBYY9uNewnaUJ9MMcxt3M1Q1X53koEFTV/UZdtEUcmdslY
6z8skUY+LB/r/D8iGhc3DnuV3w9ltsq3MNvLw6iXV/OHPUIhJhvTlt4RiVxL15jJ
KHZ1FJuBO0xDr6ikp0spEMFLQvZFVuGARzksEO/bR5OQmxg1imb7PA5usXntpmJ3
75js3Lqt5dTZanjQwXE=
-----END CERTIFICATE-----
subject=/C=US/ST=California/L=San Jose/O=Cisco Systems, Inc./OU=titan-viptela/CN=vmanage-f32e471a-10e3-4b29-bbda-b1b20cf03297-0.viptela.com
issuer=/O=Cisco/OU=Albireo/CN=Viptela SubCA
---
No client certificate CA names sent
Client Certificate Types: RSA sign, DSA sign, ECDSA sign
Requested Signature Algorithms: RSA+SHA512:DSA+SHA512:ECDSA+SHA512:RSA+SHA384:DSA+SHA384:ECDSA+SHA384:RSA+SHA256:DSA+SHA256:ECDSA+SHA256:RSA+SHA224:DSA+SHA224:ECDSA+SHA224:RSA+SHA1:DSA+SHA1:ECDSA+SHA1
Shared Requested Signature Algorithms: RSA+SHA512:DSA+SHA512:ECDSA+SHA512:RSA+SHA384:DSA+SHA384:ECDSA+SHA384:RSA+SHA256:DSA+SHA256:ECDSA+SHA256:RSA+SHA224:DSA+SHA224:ECDSA+SHA224:RSA+SHA1:DSA+SHA1:ECDSA+SHA1
Peer signing digest: SHA512
Server Temp Key: ECDH, P-521, 521 bits
---
SSL handshake has read 2251 bytes and written 814 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : DTLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID:
    Session-ID-ctx:
    Master-Key: B125B1839D05E2D8C09F1922CC38629D6AC2CC623AC27AB510CC38B28C590560DE4315815A08CD35288B6AB6CB47503F
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1589734217
    Timeout   : 7200 (sec)
    Verify return code: 21 (unable to verify the first certificate)
---