Release Announcement: NSO 6.6 is here!
Check out the release highlights to see what's new
NSO DevCenterTry NSO

README v1.5.5 2024-08-29

Table of contents

1. General
   1.1 Extract the NED package
   1.2 Install the NED package
       1.2.1 Local install
       1.2.2 System install
   1.3 Configure the NED in NSO
2. Optional debug and trace setup
3. Dependencies
4. Sample device configuration
5. Built in live-status actions
6. Built in live-status show
7. Limitations
8. How to report NED issues and feature requests
9. How to rebuild a NED
10. Configure the NED to use ssh multi factor authentication
11. How to avoid out-of-sync

1. General

This document describes the extreme-xos NED.

Additional README files bundled with this NED package

Common NED Features

Custom NED Features

Verified target systems

1.1 Extract the NED package

It is assumed the NED package ncs-<NSO version>-extreme-xos-<NED version>.signed.bin has already been downloaded from software.cisco.com.

In this instruction the following example settings will be used:

  • NSO version: 6.0

  • NED version: 1.0.1

  • NED package downloaded to: /tmp/ned-package-store

  1. Extract the NED package and verify its signature:

  2. In case the signature can not be verified (for instance if no internet connection), do as below instead:

  3. The result of the extraction shall be a tar.gz file with the same name as the .bin file:

1.2 Install the NED package

There are two alternative ways to install this NED package. Which one to use depends on how NSO itself is setup.

In the instructions below the following example settings will be used:

  • NSO version: 6.0

  • NED version: 1.0.1

  • NED download directory: /tmp/ned-package-store

  • NSO run time directory: ~/nso-lab-rundir

A prerequisite is to set the environment variable NSO_RUNDIR to point at the NSO run time directory:

1.2.1 Local install

This section describes how to install a NED package on a locally installed NSO (see "NSO Local Install" in the NSO Installation guide).

It is assumed the NED package has been been unpacked to a tar.gz file as described in 1.1.

  1. Untar the tar.gz file. This creates a new sub-directory named:extreme-xos-<NED major digit>.<NED minor digit>:

  2. Install the NED into NSO, using the ncs-setup tool:

  3. Open a NSO CLI session and load the new NED package like below:

Alternatively the tar.gz file can be installed directly into NSO. Then skip steps 1 and 2 and do like below instead:

Set the environment variable NED_ROOT_DIR to point at the NSO NED package:

1.2.2 System install

This section describes how to install a NED package on a system installed NSO (see "NSO System Install" in the NSO Installation Guide).

It is assumed the NED package has been been unpacked to a tar.gz file as described in 1.1.

  1. Do a NSO backup before installing the new NED package:

  2. Start a NSO CLI session and fetch the NED package:

  3. Install the NED package (add the argument replace-existing if a previous version has been loaded):

  4. Load the NED package

1.3 Configure the NED in NSO

This section describes the steps for configuring a device instance using the newly installed NED package.

  • Start a NSO CLI session:

  • Enter configuration mode:

  • Configure a new authentication group (my-group) to be used for this device:

  • Configure a new device instance (example: dev-1):

  • If configured protocol is ssh, do fetch the host keys now:

When connecting through a proxy using SSH or TELNET

  • Finally commit the configuration

  • Verify configuration, using a sync-from.

If the sync-from was not successful, check the NED configuration again.

2. Optional debug and trace setup

It is often desirable to see details from when and how the NED interacts with the device(Example: troubleshooting)

This can be achieved by configuring NSO to generate a trace file for the NED. A trace file contains information about all interactions with the device. Messages sent and received as well as debug printouts, depending on the log level configured.

NSO creates one separate trace file for each device instance with tracing enabled. Stored in the following location:

$NSO_RUNDIR/logs/ned-extreme-xos-cli-1.0-<device name>.trace

Do as follows to enable tracing in one specific device instance in NSO:

  1. Start a NSO CLI session:

  2. Enter configuration mode:

  3. Enable trace raw:

    Alternatively, tracing can be enabled globally affecting all configured device instances:

  4. Configure the log level for printouts to the trace file:

    Alternatively the log level can be set globally affecting all configured device instances using this NED package.

The log level 'info' is used by default and the 'debug' level is the most verbose.

IMPORTANT: Tracing shall be used with caution. This feature does increase the number of IPC messages sent between the NED and NSO. In some cases this can affect the performance in NSO. Hence, tracing should normally be disabled in production systems.

An alternative method for generating printouts from the NED is to enable the Java logging mechanism. This makes the NED print log messages to common NSO Java log file.

$NSO_RUNDIR/logs/ncs-java-vm.log

Do as follows to enable Java logging in the NED

  1. Start a NSO CLI session:

  2. Enter configuration mode:

  3. Enable Java logging with level all from the NED package:

  4. Configure the NED to log to the Java logger

    Alternatively Java logging can be enabled globally affecting all configured device instances using this NED package.

IMPORTANT: Java logging does not use any IPC messages sent to NSO. Consequently, NSO performance is not affected. However, all log printouts from all log enabled devices are saved in one single file. This means that the usability is limited. Typically single device use cases etc.

3. Dependencies

This NED has the following host environment dependencies:

  • Java 1.8 (NSO version < 6.2)

  • Java 17 (NSO version >= 6.2)

  • Gnu Sed

Dependencies for NED recompile:

  • Apache Ant

  • Bash

  • Gnu Sort

  • Gnu awk

  • Grep

  • Python3 (with packages: re, sys, getopt, subprocess, argparse, os, glob)

4. Sample device configuration

NONE

5. Built in live-status actions

Any device command can be run on the device with:

The output will be printed out as it it returned by the device.

The NED has support for all operational commands by use of the 'devices device live-status exec any' action. For example:

To execute multiple commands, separate them with " ; "NOTE: Must be a white space on either side of the comma. For example:

Generally the command output parsing halts when the NED detects an operational prompt, however sometimes the command requests additional input, 'answer(s)' to questions.

Using these settings it is possible to define a separate way of interacting with the device, ignoring the default behaviour of the ned.

The state machine involves configuring the pattern to be expected, the command to execute at match and the next state:

E.g:

Example of execution:

6. Built in live-status show

NONE

7. Limitations

  • 'configure access-list filename any *'

The filename must be introduced without the .pol extension:

  • 'configure stpd ports mode dot1d ' This command seems to be auto-created and auto-deleted by the device, thus, the user must mimic the device behavior. For create, the ned will send the configure command, but for delete, the unconfigure command it's not available, and the ned will skip sending it to the device.

8. How to report NED issues and feature requests

Issues like bugs and errors shall always be reported to the Cisco NSO NED team through the Cisco Support channel:

The following information is required for the Cisco NSO NED team to be able to investigate an issue:

Do as follows to gather the necessary information needed for your device, here named 'dev-1':

  1. Enable full debug logging in the NED

  2. Configure the NSO to generate a raw trace file from the NED

  3. If the NED already had trace enabled, clear it in order to submit only relevant information

    Do as follows for NSO 6.4 or newer:

    Do as follows for older NSO versions:

  4. Run a compare-config to populate the trace with initial device config

  5. Reproduce the found issue using ncs_cli or your NSO service. Write down each necessary step in a reproduction report.

    In addition to this, it helps if you can show how it should work by manually logging into the device using SSH/TELNET and type the relevant commands showing a successful operation.

  6. Gather the reproduction report and a copy of the raw trace file containing data recorded when the issue happened.

  7. Contact the Cisco support and request to open a case. Provide the gathered files together with access details for a device that can be used by the Cisco NSO NED when investigating the issue.

Requests for new features and extensions of the NED are handled by the Cisco NSO NED team when applicable. Such requests shall also go through the Cisco support channel.

The following information is required for feature requests and extensions:

  1. Set the config on the real device including all existing dependent config and run sync-from to show it in the trace.

  2. Run sync-from # devices device dev-1 sync-from

  3. Attach the raw trace to the ticket

  4. List the config you want implemented in the same syntax as shown on the device

  5. SSH/TELNET access to a device that can be used by the Cisco NSO NED team for testing and verification of the new feature. This usually means that both read and write permissions are required. Pseudo access via tools like Webex, Zoom etc is not acceptable. However, it is ok with access through VPNs, jump servers etc as long as we can connect to the NED via SSH/TELNET.

9. How to rebuild a NED

To rebuild the NED do as follows:

When the NED has been successfully rebuilt, it is necessary to reload the package into NSO.

10. Configure the NED to use ssh multi factor authentication

This NED supports multi factor authentication (MFA) using the ssh authentication method 'keyboard-interactive'.

Some additional steps are required to enable the MFA support:

  1. Verify that your NSO version supports MFA. This is configurable as additional settings in the authentication group used by the device instance.

    Enter a NSO CLI and enter the following and do tab completion:

    If 'mfa' is displayed in the output like above, NSO has MFA support enabled. In case MFA is not supported it is necessary to upgrade NSO before proceeding.

  2. Implement the authenticator executable. The MFA feature relies on an external executable to take care of the client part of the multi factor authentication. The NED will automatically call this executable for each challenge presented by the ssh server and expects to get a proper response in return.

    The executable can be a simple shell script or a program implemented in any programming language.

    The required behaviour is like this:

    • read one line from stdin The line passed from the NED will be a semi colon separated string containing the following info:

      The elements for device name, user, password and opaque corresponds to what has been configured in NSO. The ssh server name, instruction and prompt are given by the ssh server during the authentication step.

      Each individual element in the semi colon separated list is Base64 encoded.

    • Extract the challenge based on the contents above.

    • Print a response matching the challenge to stdout and exit with code 0

    • In case a matching response can not be given do exit with code 2

    Below is a simple example of an MFA authenticator implemented in Python3:

  3. Configure the authentication group used by the device instance to enable MFA. There are two configurables available:

    • executable The path to the external multi factor authentication executable (mandatory).

    • opaque Opaque data that will passed as a cookie element to the executable (optional).

  4. Try connecting to the device.

10.1 Trouble shooting

In case of connection problems the following steps can help for debugging:

Enable the NED trace in debug level:

Try connect again

Inspect the generated trace file.

Verify that the ssh client is using the external authenticator executable:

Verify that the executable is called with the challenges presented by the ssh server:

Check for any errors reported by the NED when calling the executable

11. How to avoid out-of-sync

All delete operations are performed using standard "no" command in front of the configuration line. The ned will transform those command into expected device commands:

Also, the automatic deletion of some fields must be mirrored by the user: E.g

To keep in sync, the 'configure ***' deletion must be performed by the user:

The ned will send only what is expected, as there is no command for tag deletion.

11.1 Create and Configure

The Extreme device does various automatic changes to the running configuration. To keep in sync, the user must mimic device behavior, as the ned will not autoconfigure NSO cdb (using hooks is prohibited). The Ned will only do various operations over the device.

E.g: Deleting VR and vlan configurations, both "no create" and "no configure" commands must be sent:

11.2 Banner

A special case is the banner configuration. As this device expects a particular set of operations to set and reset the banner, in NSO, the banner string must be quoted, and no '\r' must be added, as the device adds it automatically. For eol use only '\n'. The delimiters are added/detected automatically by the ned. Configuration example:

Deleting the banner:

For multiline banner, in the commit dry-run outformat native output, the ned will add "! meta-data :: turbo :: no-prompt-after-send" tag before each line that is expected to be part of the banner text.

11.3 UPM profile

Similar to banner configuration, 'upm profile' creation expects a multiline input with the desired configuration. The configuration is ended with "."

The ned expects the configuration to be quoted in a single line and the all the command lines must be ended concatenated using a \n. The "." terminator is added automatically by the ned:

Note that quotes inside the config lines must be escaped, otherwise the input will not be correctly digested by NSO. The ned adds the '!meta-data ***' comments to signal the internal device interactor that this is a multi-line input without expecting the prompt. These comment lines are present only in prepare phase (dry-run) and are skipped when sent to the device.

11.4 Shared-secrets

The device accepts unencrypted secrets that after commit will be shown as encrypted strings. To keep in sync, the ned expects the secret to be configured using the "encrypted" prefix for both encrypted and clear text secrets. The secrets handler will keep track of it and will not generate an out-of-sync.

The ned also supports cleartext provisioning by default. This means that if the service layer has the secret encrypted with tailf:aes-cfb-128-encrypted-string type, it will pass it encrypted to the ned. Then, the ned will automatically decrypt it, and pass it as clear text password to the device (auto removing the "encrypted" label). By default, the ned will not show the decrypted secret in the dry-run native format.

Also, by default the ned-settings/extreme-xos/console/obfuscate-secret is enabled and all secrets are obfuscated from the trace files.

E.g: Service sends:

NSO will send to the Ned this command:

Then the NED will transform it to this:

The secrets handler will keep track of it, and no compare diff will appear, keeping it in sync, and being able to correctly use the device (e.g: login with test/test1234 credentials).

Example of supported strings:

11.5 Dynamic configurations

In many places, at configuration, the device allows skipping some elements and values, using a shorted command or even a macro. In these cases the configured commands become different from what is seen at 'show' leading to out-of-sync. To avoid this, the user/service must mimic the device behavior, as the ned can't and will not autofill elements in its cdb (hooks usage is prohibited).

Adding snmpv3 users with clear text passwords

This is a case where the device encrypts the password as hex-strings and also allows a short from of the configuration command. While this kind of command is accepted by the device :

when looking at show configuration output, the result is quite different:

Both passwords are encrypted as hashed hex strings and added under "auth-encrypted localized-key" and "privacy-encrypted localized-key". Also, the engine-id is added and the aes value: 128.

The ned will format the output for clear-text sending:

For NSO encrypted values, the ned will auto-decrypt the passwords and will send it to the device as clear-text. Node that in dry-run, the decrypt will not occur due to security reasons.

NSO user input:

NSO dry-run output:

NSO output (at commit):

Device final result (show config):

Note: To also support get output reinsert scenarios (e.g: retreived output is saved, device is cleared, and then device saved output is reinserted), the ned enforces at configuration an identic yang structure for passwords, i.e the user must introduce "auth/privacy-encrypted localized-key" in front of the clear-text/NSO encyrpted passwords", even if these tags re not sent to the device. Only if the passwords are of hex format (how the device exposes them in 'show configuration'), the ned will detect it and will send the full command format.

Note: as engine-id it's operational data, it's not included in the configuration model. This means that only local engine-id configurations are supported by the ned.

11.6 Default user accounts

As there are "hidden" default accounts, that are not visible in "show config" output, but there the user can modify their password, the NED adds "show accounts" output into the cdb as "create accounts admin(Access is R/W)| user (Access is RO) .

Eg: Output accounts:

This way, the default accounts will appear in NSO cdb, as a list entry in the /create/accounts list, but with empty 'encrypted' value, as password

11.7 Changing account passwords

To change an existing account password, the device expects 'configure account [encrypted] password' command.

In "show configuration" output the device does not expose "configure account" lines, like in other cases, where, the lists entries that can be modified after they are created (e.g vlan), are present both in /create and in /configure modes.

For the "account" list, only "create account" command is available in the show output and in configuration mode. To keep the NED in sync with the device, the NED will use the "create account" command both for creation and editing of already existing parameters (account passwords).

To change an account password, the user must send the same command as when creating a new account, always using the "encrypted" parameter, even when the password is in clear-text:

If there is a password change, the ned will detect it, and act as follows:

11.7.1 Encrypted string

If the 'value' is a device encrypted string (post-encrypted hash code), then the command that is sent is:

In this case, the device doesn't require the current password to be introduced. E.g:

11.7.2 Clear-text or NSO encrypted string

If the 'value' is clear-text or NSO encrypted (tailf:aes-cfb-128-encrypted-string), then the device expects the following procedure for password change:

In this case the current user's password is required to make a password change. The ned act as follows:

  • If the 'OLD password' value was previously set as clear-text/aes-cfb-128-encrypted-string by NSO (e.g at account creation), then the ned accepts the following format:

The OLD_password value is NOT needed at user input as it is already known by NSO (Modify operation: the value it's already present in cdb). The NED retrieves it from NSO cdb and uses it as input at "Current user's password" prompt. E.g:

  • If the 'OLD password' is unknown to NSO (either retreived at sync-from, or was previously set as device encrypted value, or it's hidden - the case for default accounts), then the user must provide the clear text current password as follows:

The password values must be clear-text or aes-cfb-128-encrypted-string format or any combo of these two.

E.g: User input:

decrypts to:

Device input:

Note: If NSO is aware of the oldpassword value (set in a prevous transaction), even if the old-password is introduced at user input, the NED will use the old cdb value instead. This was done to avoid cases where (at rollback), the old password switches places with the new one and the user input will be incorrect:

E.g:

Here the current password is set to "oldpassword" as in previous transaction, but this is incorrect, as the current password is now "newpassword1". This is detected by the NED and will use the current password value from its cdb ("newpassword1"):

Device input:

Warning: Note that if there are scenarios where the user changes an account password from clear-text to a post-encrypted hash value, the rollback operation will fail, as the device encrypted password is unknown to NSO, can't be decrypted, and it is used as "Current user's password" in the password change sequence: E.g:

The device expects a clear-text password. As NSO can't decrypt it, the device will generate the following error: Error: Incorrect password for user "test" ,retry or logout and then login and try again '

  • Special handling for empty password for string for "old_password": In case the old password is an empty string, to avoid NSO confusion, "\n" must be introduced instead of an empty string: e.g:

PreviousREADME-ned-settingsNextf5-bigip

Last updated

Was this helpful?