Ways to Deploy NSO

• By installation

• By using Cisco-provided container images

By Installation

Choose this way if you want to install NSO on a system. Before proceeding with the installation, decide on the install type.

Install Types

The installation of NSO comes in two variants:

By Using Cisco-Provided Container Images

Choose this way if you want to run NSO in a container, such as Docker. Visit the link below for more information.

Supporting Information

If you are evaluating NSO, you should have a designated support contact. If you have an NSO support agreement, please use the support channels specified in the agreement. In either case, do not hesitate to reach out to us if you have questions or feedback.

Installation and Deployment

Management

Release Highlights

This release includes major enhancements in the following areas:

The following actions are possible after installing NSO.

After Local Install

After System Install

If you intend to use your NSO instance for development purposes, enable the development mode using the command license smart development enable.

To uninstall Local Install, simply delete the Install Directory.

Use this page to navigate your way through the NSO documentation and access the resources most relevant to your role.

NSO Roles

An NSO deployment typically consists of the following roles:

Learn NSO

For users new to NSO or wanting to explore it further.

Work with NSO

For users working in a production-wide NSO deployment.

Administration

Operation and Usage

Development

NSO can be uninstalled using the option only if NSO is installed with --system-install option. Either part of the static files or the full installation can be removed using ncs-uninstall option. Ensure to stop NSO before uninstalling.

Executing the above command removes the Installation Directory /opt/ncs including symbolic links, Configuration Directory /etc/ncs, Run Directory /var/opt/ncs

NSO Packages contain data models and code for a specific function. It might be a NED for a specific device, a service application like MPLS VPN, a WebUI customization package, etc. Packages can be added, removed, and upgraded in run-time.

The currently loaded packages can be viewed with the following command:

Thus the above command shows that NSO currently has only one package loaded, the NED package for Cisco IOS. The output includes the name and version of the package, the minimum required NSO version, the Java components included, package build details, and finally the operational status of the package. The operational status is of particular importance - if it is anything other than up, it indicates that there was a problem with the loading or the initialization of the package. In this case, an item error-info may also be present, giving additional information about the problem. To show only the operational status for all loaded packages, this command can be used:

This section describes the various northbound programmatic APIs in NSO NETCONF, REST, and SNMP. These APIs are used by external systems that need to communicate with NSO, such as portals, OSS, or BSS systems.

NSO has two northbound interfaces intended for human usage, the CLI and the WebUI. These interfaces are described in and respectively.

There are also programmatic Java, Python, and Erlang APIs intended to be used by applications integrated with NSO itself. See for more information about these APIs.

Integrating an External System with NSO

There are two APIs to choose from when an external system should communicate with NSO:

Since all the NSO examples and README steps that come with the installer are primarily aimed at Local Install, you need to modify them to run them on a System Install.

To work with the System Install structure, this may require a little or bigger modification depending on the example.

For example, to port the example.ncs/development-guide/nano-services/basic-vrouter example to the System Install structure:

1. Make the following changes to the basic-vrouter/ncs.conf file:

2. Copy the Local Install $NCS_DIR/var/ncs/cdb/aaa_init.xml file to the basic-vrouter/ folder.

Other, more complex examples may require more ncs.conf file changes or require a copy of the Local Install default $NCS_DIR/etc/ncs/ncs.conf file together with the modification described above, or require the Local Install tool $NCS_DIR/bin/ncs-setup to be installed, as the ncs-setup command is usually not useful with a System Install. See for more information.

Client libraries connect to NSO using TCP. We tell NSO which address to use for these connections through the /ncs-config/ncs-ipc-address/ip (default value 127.0.0.1) and /ncs-config/ncs-ipc-address/port (default value 4569) elements in ncs.conf. It is possible to change these values, but it requires a number of steps to also configure the clients. Also, there are security implications, see Security Issues.

Some clients read the environment variables NCS_IPC_ADDR and NCS_IPC_PORT to determine if something other than the default is to be used, others might need to be recompiled. This is a list of clients that communicate with NSO, and what needs to be done when ncs-ipc-address is changed.

To run more than one instance of NSO on the same host (which can be useful in development scenarios), each instance needs its own IPC port. For each instance, set /ncs-config/ncs-ipc-address/port in ncs.conf to something different.

There are two more instances of ports that will have to be modified, NETCONF and CLI over SSH. The netconf (SSH and TCP) ports that NSO listens to by default are 2022 and 2023 respectively. Modify /ncs-config/netconf/transport/ssh and /ncs-config/netconf/transport/tcp, either by disabling them or changing the ports they listen to. The CLI over SSH by default listens to 2024; modify /ncs-config/cli/ssh either by disabling or changing the default port.

Restricting Access to IPC port

By default, the clients connecting to the IPC port are considered trusted, i.e. there is no authentication required, and we rely on the use of 127.0.0.1 for /ncs-config/ncs-ipc-address/ip to prevent remote access. In case this is not sufficient, it is possible to restrict access to the IPC port by configuring an access check.

The access check is enabled by setting the ncs.conf element /ncs-config/ncs-ipc-access-check/enabled to true, and specifying a filename for /ncs-config/ncs-ipc-access-check/filename. The file should contain a shared secret, i.e., a random character string. Clients connecting to the IPC port will then be required to prove that they have knowledge of the secret through a challenge handshake before they are allowed access to the NSO functions provided via the IPC port.

To provide the secret to the client libraries and inform them that they need to use the access check handshake, we have to set the environment variable NCS_IPC_ACCESS_FILE to the full pathname of the file containing the secret. This is sufficient for all the clients mentioned above, i.e., there is no need to change the application code to support or enable this check.

A common misfeature found on UNIX operating systems is the restriction that only root can bind to ports below 1024. Many a dollar has been wasted on workarounds and often the results are security holes.

Both FreeBSD and Solaris have elegant configuration options to turn this feature off. On FreeBSD:

# sysctl net.inet.ip.portrange.reservedhigh=0

The above is best added to your /etc/sysctl.conf.

Similarly, on Solaris, we can just configure this. Assuming we want to run NSO under a non-root user ncs. On Solaris, we can do that easily by granting the specific right to bind privileged ports below 1024 (and only that) to the ncs user using:

# /usr/sbin/usermod -K defaultpriv=basic,net_privaddr ncs

And check that we get what we want through:

# grep ncs /etc/user_attr
ncs::::type=normal;defaultpriv=basic,net_privaddr

Linux doesn't have anything like the above. There are a couple of options on Linux. The best is to use an auxiliary program like authbind (http://packages.debian.org/stable/authbind) or privbind (http://sourceforge.net/projects/privbind/).

These programs are run by root. To start NCS under e.g., privbind, we can do:

The above command starts NSO as the user ncs and binds to ports below 1024.

The NSO Web UI provides an intuitive northbound interface to your NSO deployment. The UI consists of individual views, each with a different purpose, such as device management, service management, commit handling, etc.

The main components of the Web UI are shown in the figure below.

The UI works by auto-rendering the underlying device and service models. This gives the benefit that the Web UI is immediately updated when new devices or services are added to the system. For example, say you have added support for a new device vendor. Then, without any programming requirements, the NSO Web UI provides the capability to configure those devices.

Browser Requirements

All modern web browsers are supported and no plug-ins are needed. The interface itself is a JavaScript Client.

Accessing the Web UI

By default, the Web UI is accessible on port 8080 of the NSO server for an NSO Local Install and port 8888 for a System Install. The port can be changed in the ncs.conf file. A user must authenticate before accessing the (web) UI.

Basic Operations

Log In

Log in to the NSO Web UI by using the username and password provided by your administrator. SSO SAML login is available if set up by your administrator. If applicable, use the SSO option to log in.

Log Out

Log out by clicking your username on the top-right corner and choosing Logout.

Help Options

Access the help options by clicking the help options icon in the UI banner. The following options are available:

• Online documentation: Access the Web UI's online help.

• Config editor help: Access help on using the configuration editor.

• Manage hidden groups: Administer hidden groups, e.g. for debugging. Read more about hide groups in .

• NSO version: Information about the version of NSO you are running.

In the Web UI, supplementary help text, whenever applicable, is available on the configuration fields and can be accessed by clicking the info icons.

Commit Manager

The Commit Manager is accessible at all times from the UI banner. Working with the Commit Manager is further described in .

One of the included scripts with an NSO installation is the ncs-setup, which makes it very easy to create instances of NSO from a Local Install. You can look at the --help or in Manual Pages for more details, but the two options we need to know are:

• --dest

CLI

Web UI

The Configuration editor view is where you view and manage aspects of your NSO deployment using the underlying YANG model, for example, to configure devices, services, packages, etc.

The Configuration Editor's home page shows all the currently loaded YANG modules in NSO, i.e., the database schema. In this view, you can also browse and manage the configuration defined by the YANG modules.

Editing Configuration Data

All NSO configuration is performed in this view. You can edit the configuration data defined YANG model directly in this view or get directed by the Web UI to this view.

Configuration Navigator

An important component of Configuration Editor is the Configuration Navigator which you can use to traverse and edit the configuration defined by the YANG model in a hierarchical tree-like fashion. This provides an efficient way to browse and configure aspects of NSO. Let's say, for example, you want to access all the devices in your deployment and choose a specific one to view and configure. In the Configuration Editor, you can do this by typing in ncs:devices in the navigator, and then choosing further guided options (automatically suggested by the Web UI), e.g., ncs:devices/device/ce0/config/....

Using the Configuration Navigator

As you navigate through the Web UI, the Configuration Navigator automatically displays and updates the path you are located at.

• To exit back to the home page from another path, click the home button.

• Click the up arrow to go back one step to the parent node.

• To fetch information about a property/component, click the info button.

• Use the TAB key to complete the config path.

Configuration Editor Tabs

When accessing an item (e.g., a device, service, etc.) using the Configuration Editor, the following tabs are visible:

• Edit Config tab, to configure the item's configuration.

• Config tab, to view configured items.

• Operdata tab, to view the operational data relevant to the item (e.g., last sync time, last modified time, etc).

• Actions tab, to apply an action to the item with specified options/parameters.

Depending on the selection of the tabs mentioned above, you may see four additional tabs in the Configuration editor view:

• Widgets tab, to view the data defined by YANG modules in different formats.

• None tab.

• Containers tab, to view container-specific information from the YANG model.

• List tab, to view list-specific information from the YANG model.

This section provides an overview of how to run the examples provided with the NSO installer. By working through the examples, the reader should get a good overview of the various aspects of NSO and hands-on experience from interacting with it.

If you already have a Local Install with existing data that you would like to convert into a System Install, the following procedure allows you to do so. However, a reverse migration from System to Local Install is not supported.

is a cloud-based approach to licensing and it simplifies the purchase, deployment, and management of Cisco software assets. Entitlements are purchased through a Cisco account via Cisco Commerce Workspace (CCW) and are immediately deposited into a Smart Account for usage. This eliminates the need to install license files on every device. Products that are smart-enabled communicate directly to Cisco to report consumption.

Cisco Smart Software Manager (CSSM) enables the management of software licenses and Smart Account from a single portal. The interface allows you to activate your product, manage entitlements, and renew and upgrade software.

A functioning Smart Account is required to complete the registration process. For detailed information about CSSM, see .

Smart Accounts and Virtual Accounts

The Service manager view is where you create, deploy, and manage services in your NSO deployment. Available services are displayed in this view by default.

Introduction to Automation

Core Concepts

When new NED versions with diverging XML namespaces are introduced, adaptations might be needed in the services for these new NEDs. But not necessarily; it depends on where in the specific NED models the ambiguities reside. Existing services might not refer to these parts of the model and in that case, they do not need any adaptations.

Finding out if and where services need adaptations can be non-trivial. An important exception is template services which check and point out ambiguities at load time (NSO startup). In Java or Python code this is harder and essentially falls back to code reviews and testing.

The changes in service code to handle ambiguities are straightforward but different for templates and code.

Template Services

In templates, there are new processing instructions if-ned-id

NSO is capable of starting user-provided Erlang applications embedded in the same Erlang VM as NSO.

The Erlang code is packaged into applications which are automatically started and stopped by NSO if they are located at the proper place. NSO will search all packages for top-level directories called erlang-lib. The structure of such a directory is the same as a standard lib directory in Erlang. The directory may contain multiple Erlang applications. Each one must have a valid .app file. See the Erlang documentation of application and app for more info.

An Erlang package skeleton can be created by making use of the ncs-make-package command:

Multiple applications can be generated by using the option --erlang-application-name NAME

The command ncs -h shows various options when starting NSO. By default, NSO starts in the background without an associated terminal. It is recommended to add NSO to the /etc/init scripts of the deployment hosts. For more information, see the in Manual Pages.

Whenever you start (or reload) the NSO daemon, it reads its configuration from ./ncs.conf or ${NCS_DIR}/etc/ncs/ncs.conf or from the file specified with the

The service manager executes in a Java VM outside of NSO. The NcsMux initializes a number of sockets to NSO at startup. These are Maapi sockets and data provider sockets. NSO can choose to close any of these sockets whenever NSO requests the service manager to perform a task, and that task is not finished within the stipulated timeout. If that happens, the service manager must be restarted. The timeout(s) are controlled by several ncs.conf parameters found under /ncs-config/japi.

The ncs-netsim program is a useful tool to simulate a network of devices to be managed by NSO. It makes it easy to test NSO packages towards simulated devices. All you need is the NSO NED packages for the devices that you need to simulate. The devices are simulated with the Tail-f ConfD product.

All the NSO examples use ncs-netsim to simulate the devices. A good way to learn how to use ncs-netsim is to study them.

Using Netsim

The ncs-netsim tool takes any number of NED packages as input. The user can specify the number of device instances per package (device type) and a string that is used as a prefix for the name of the devices. The command takes the following parameters:

Assume that you have prepared an NSO package for a device called router. (See the examples.ncs/getting-started/developing-with-ncs/0-router-network example). Also, assume the package is in ./packages/router. At this point, you can create the simulated network by:

This creates three devices; device0, device1, and device2. The simulated network is stored in the ./netsim directory. The output structure is:

There is one separate directory for every ConfD simulating the devices.

The network can be started with:

You can add more devices to the network in a similar way as it was created. E.g. if you created a network with some Juniper devices and want to add some Cisco IOS devices. Point to the NED you want to use (See {NCS_DIR}/packages/neds/) and run the command. Remember to start the new devices after they have been added to the network.

To extract the device data from the simulated network to a file in XML format:

This data is usually used to load the simulated network into NSO. Putting the XML file in the ./ncs-cdb folder will load it when NSO starts. If NSO is already started it can be reloaded while running.

The generated device data creates devices of the same type as the device being simulated. This is true for NETCONF, CLI, and SNMP devices. When simulating generic devices, the simulated device will run as a netconf device.

Under very special circumstances, one can choose to force running the simulation as a generic device with the option --force-generic.

The simulated network device info can be shown with:

Here you can see the device name, the working directory, and the port number for different services to be accessed on the simulated device (NETCONF SSH, SNMP, IPC, and direct access to the CLI).

You can reach the CLI of individual devices with:

The simulated devices actually provide three different styles of CLI:

• cli: J-Style

• cli-c: Cisco XR Style

• cli-i: Cisco IOS Style

Individual devices can be started and stopped with:

You can check the status of the simulated network. Either a short version just to see if the device is running or a more verbose with all the information.

View which packages are used in the simulated network:

It is also possible to reset the network back to the state of initialization:

When you are done, remove the network:

Using ConfD Tools with Netsim

The netsim tool includes a standard ConfD distribution and the ConfD C API library (libconfd) that the ConfD tools use. The library is built with default settings where the values for MAXDEPTH and MAXKEYLEN are 20 and 9, respectively. These values define the size of confd_hkeypath_t struct and this size is related to the size of data models in terms of depth and key lengths. Default values should be big enough even for very large and complex data models. But in some rare cases, one or both of these values might not be large enough for a given data model.

One might observe a limitation when the data models that are used by simulated devices exceed these limits. Then it would not be possible to use the ConfD tools that are provided with the netsim. To overcome this limitation, it is advised to use the corresponding NSO tools to perform desired tasks on devices.

NSO and ConfD tools and Python APIs are basically the same except for naming, the default IPC port and the MAXDEPTH and MAXKEYLEN values, where for NSO tools, the values are set to 60 and 18, respectively. Thus, the advised solution is to use the NSO tools and NSO Python API with netsim.

E.g. Instead of using the below command:

One may use:

Learn More

The README file in examples.ncs/getting-started/developing-with-ncs/0-router-network gives a good introduction on how to use ncs-netsim.

CDB implements write-ahead logging to provide durability in the datastores, appending a new log for each CDB transaction to the target datastore (A.cdb for configuration, O.cdb for operational, and S.cdb for snapshot datastore). Depending on the size and number of transactions towards the system, these files will grow in size leading to increased disk utilization, longer boot times, and longer initial data synchronization time when setting up a high-availability cluster.

Compaction is a mechanism used to reduce the size of the write-ahead logs to a minimum. It works by replacing an existing write-ahead log, which is composed by a number of consecutive transaction logs created in run-time, with a single transaction log representing the full current state of the datastore. From this perspective, it can be seen that a compaction acts similar to a write transaction towards a datastore. To ensure data integrity, 'write' transactions towards the datastore are not permitted during the time compaction takes place.

Automatic Compaction

By default, compaction is handled automatically by the CDB. After each transaction, CDB evaluates whether compaction is required for the affected data store.

This is done by examining the number of added nodes as well as the file size changes since the last performed compaction. The thresholds used can be modified in the ncs.conf file by configuring the /ncs-config/compaction/file-size-relative, /ncs-config/compaction/file-size-absolute, and /ncs-config/compaction/num-node-relative settings.

It is also possible to automatically trigger compaction after a set number of transactions by setting the /ncs-config/compaction/num-transaction property.

Manual Compaction

Compaction may require a significant amount of time, during which write transactions cannot be performed. In certain use cases, it may be preferable to disable automatic compaction by CDB and instead trigger compaction manually according to the specific needs. If doing so, it is highly recommended to have another automated system in place.

CDB CAPI provides a set of functions that may be used to create an external mechanism for compaction. See cdb_initiate_journal_compaction(), cdb_initiate_journal_dbfile_compaction(), and cdb_get_compaction_info() in in Manual Pages.

Automation of compaction can be done by using a scheduling mechanism such as CRON, or by using the NCS scheduler. See for more information.

By default, CDB may perform compaction during its boot process. This may be disabled if required, by starting NSO with the flag --disable-compaction-on-start.

Delayed Compaction

In the configuration datastore, compaction is by default delayed by 5 seconds when the threshold is reached to prevent any upcoming write transaction from being blocked. If the system is idle during these 5 seconds, meaning that there is no new transaction, the compaction will initiate. Otherwise, compaction is delayed by another 5 seconds. The delay time can be configured in ncs.conf by setting the /ncs-config/compaction/delayed-compaction-timeout property.

The Devices view provides options to manage devices and device groups in the NSO network.

Device Management

The Device management view lists the devices in the network and provides options to manage them.

This section describes some recipes, tools, and other resources that you may find useful throughout development. The topics are tailored to novice users and focus on making development with NSO a more enjoyable experience.

Development NSO Instance

Many developers prefer their own, dedicated NSO instance to avoid their work clashing with other team members. You can use either a local or remote Linux machine (such as a VM), or a macOS computer for this purpose.

The advantage of running local Linux with a GUI or macOS is that it is easier to set up the Integrated Development Environment (IDE) and other tools when they run on the same system as NSO. However, many IDEs today also allow working remotely, such as through the SSH protocol, making the choice of local versus remote less of a concern.

For development, using the so-called Local Install of NSO has some distinct advantages:

The Home view is the default view after logging in. It provides shortcuts to Devices, Services, Config editor, and Tools.

Web UI Extension Packages

Currently loaded Web UI extension packages are shown in this view under Packages. Web UI packages are used to extend the functionality of your Web UI, for example, to create additional views and functionalities. Examples are creating a view to visualize your MPLS network, etc.

NSO supports access to all northbound interfaces via IPv6, and in the most simple case, i.e. IPv6-only access, this is just a matter of configuring an IPv6 address (typically the wildcard address ::) instead of IPv4 for the respective agents and transports in ncs.conf, e.g. /ncs-config/cli/ssh/ip for SSH connections to the CLI, or /ncs-config/netconf-north-bound/transport/ssh/ip for SSH to the NETCONF agent. The SNMP agent configuration is configured via one of the other northbound interfaces rather than via ncs.conf, see in Northbound APIs. For example, via the CLI, we would set snmp agent ip to the desired address. All these addresses default to the IPv4 wildcard address 0.0.0.0.

In most IPv6 deployments, it will however be necessary to support IPv6 and IPv4 access simultaneously. This requires that both IPv4 and IPv6 addresses are configured, typically

Services are the cornerstone of network automation with NSO. A service is not just a reusable recipe for provisioning network configurations; it allows you to manage the full configuration life-cycle with minimal effort.

This section examines in greater detail how services work, how to design them, and the different ways to implement them.

In NSO, the term service has a special meaning and represents an automation construct that orchestrates create, modify, and delete of a service instance into the resulting native commands to devices in the network. In its simplest form, a service takes some input parameters and maps them to device-specific configurations. It is a recipe or a set of instructions.

Much like you can bake many cakes using a single cake recipe, you can create many service instances using the same service. But unlike cakes, having the recipe produce exactly the same output, is not very useful. That is why service instances define a set of input parameters, which the service uses to customize the produced configuration.

A network engineer on the CLI, or an API call from a northbound system, provides the values for input parameters when requesting a new service instance, and NSO uses the service recipe, called a 'service mapping', to configure the network.

A similar process takes place when deleting the service instance or modifying the input parameters. The main task of a service is therefore: from a given set of input parameters, calculate the minimal set of device operations to achieve the desired service change. Here, it is very important that the service supports any change; create, delete, and update of any service parameter.

Device configuration is usually the primary goal of a service. However, there may be other supporting functions that are expected from the service, such as service-specific actions. The complete service application, implementing all the service functionality, is packaged in an NSO service package.

The following definitions are used throughout this section:

• Service type: Often referred to simply as a service, denotes a specific type of service, such as "L2 VPN", "L3 VPN", "Firewall", or "DNS".

• Service instance: A specific instance of a service type, such as "L3 VPN for ACME" or "Firewall for user X".

• Service model: The schema definition for a service type, defined in YANG. It specifies the names and format of input parameters for the service.

• Service mapping

Service Mapping

Developing a service that transforms a service instance request to the relevant device configurations is done differently in NSO than in most other tools on the market. As a service developer, you create a mapping from a YANG service model to the corresponding device YANG model.

This is a declarative, model-to-model mapping. Irrespective of the underlying device type and its native device interface, the mapping is towards a YANG device model and not the native CLI (or any other protocol/API). As you write the service mapping, you do not have to worry about the syntax of different CLI commands or in which order these commands are sent to the device. It is all taken care of by the NSO device manager and device NEDs. Implementing a service in NSO is reduced to transforming the input data structure, described in YANG, to device data structures, also described in YANG.

Who writes the models?

• Developing the service model is part of developing the service application and is covered later in this section.

• Every device NED comes with a corresponding device YANG model. This model has been designed by the NED developer to capture the configuration data that is supported by the device.

A service application then has two primary artifacts: a YANG service model and a mapping definition to the device YANG, as illustrated in the following figure.

To reiterate:

• The mapping is not defined using workflows, or sequences of device commands.

• The mapping is not defined in the native device interface language.

This approach may seem somewhat unorthodox at first, but allows NSO to streamline and greatly simplify how you implement services.

A common problem for traditional automation systems is that a set of instructions needs to be defined for every possible service instance change. Take for example a VPN service. During a service life cycle, you want to:

1. Create the initial VPN.

2. Add a new site or leg to the VPN.

3. Remove a site or leg from the VPN.

4. Modify the parameters of a VPN leg, such as the IP addresses used.

The possible run-time changes for an existing service instance are numerous. If a developer must define instructions for every possible change, such as a script or a workflow, the task is daunting, error-prone, and never-ending.

NSO reduces this problem to a single data-mapping definition for the "create" scenario. At run-time, NSO renders the minimum resulting change for any possible change in the service instance. It achieves this with the FASTMAP algorithm.

Another challenge in traditional systems is that a lot of code goes into managing error scenarios. The NSO built-in transaction manager takes that burden away from the developer of the service application by providing automatic rollback of incomplete changes.

Another benefit of this approach is that NSO can automatically generate the northbound APIs and database schema from the YANG models, enabling a true DevOps way of working with service models. A new service model can be defined as part of a package and loaded into NSO. An existing service model can be modified and the package upgraded, and all northbound APIs and User Interfaces are automatically regenerated to reflect the new or updated models.

As described in previous sections, the CLI NEDs are almost programming-free. The NSO CLI engine takes care of parsing the stream of characters that come from "show running-config [toptag]" and also automatically produces the sequence of CLI commands required to take the system from one state to another.

A generic NED is required when we want to manage a device that neither speaks NETCONF or SNMP nor can be modeled so that ConfD - loaded with those models - gets a CLI that looks almost/exactly like the CLI of the managed device. For example, devices that have other proprietary CLIs, devices that can only be configured over other protocols such as REST, Corba, XML-RPC, SOAP, other proprietary XML solutions, etc.

In a manner similar to the CLI NED, the Generic NED needs to be able to connect to the device, return the capabilities, perform changes to the device, and finally, grab the entire configuration of the device.

The interface that a Generic NED has to implement is very similar to the interface of a CLI NED. The main differences are:

• When NSO has calculated a diff for a specific managed device, it will for CLI NEDS also calculate the exact set of CLI commands to send to the device, according to the YANG models loaded for the device. In the case of a generic NED, NSO will instead send an array of operations to perform towards the device in the form of DOM manipulations. The generic NED class will receive an array of NedEditOp objects. Each NedEditOp object contains:

  • The operation to perform, i.e. CREATED, DELETED, VALUE_SET, etc.

  • The keypath to the object in case.

  • An optional value

• When NSO wants to sync the configuration from the device to NSO, the CLI NED only has to issue a series of show running-config [toptag] commands and reply with the output received from the device. A generic NED has to do more work. It is given a transaction handler, which it must attach to over the Maapi interface. Then the NED code must - by some means - retrieve the entire configuration and write into the supplied transaction, again using the Maapi interface.

Once the generic NED is implemented, all other functions in NSO work precisely in the same manner as with NETCONF and CLI NED devices. NSO still has the capability to run network-wide transactions. The caveat is that to abort a transaction towards a device that doesn't support transactions, we calculate the reverse diff and send it to the device, i.e. we automatically calculate the undo operations.

Another complication with generic NEDs is how the NED class shall authenticate towards the managed device. This depends entirely on the protocol between the NED class and the managed device. If SSH is used to a proprietary CLI, the existing authgroup structure in NSO can be used as is. However, if some other authentication data is needed, it is up to the generic NED implementer to augment the authgroups in tailf-ncs.yang accordingly.

We must also configure a managed device, indicating that its configuration is handled by a specific generic NED. Below we see that the NED with identity xmlrpc is handling this device.

The example examples.ncs/generic-ned/xmlrpc-device in the NSO examples collection implements a generic NED that speaks XML-RPC to 3 HTTP servers. The HTTP servers run the Apache XML-RPC server code and the NED code manipulates the 3 HTTP servers using a number of predefined XML RPC calls.

A good starting point when we wish to implement a new generic NED is the ncs-make-package --generic-ned-skeleton ... command, which is used to generate a skeleton package for a generic NED.

Getting Started with a Generic NED

A generic NED always requires more work than a CLI NED. The generic NED needs to know how to map arrays of NedEditOp objects into the equivalent reconfiguration operations on the device. Depending on the protocol and configuration capabilities of the device, this may be arbitrarily difficult.

Regardless of the device, we must always write a YANG model that describes the device. The array of NedEditOp objects that the generic NED code gets exposed to is relative the YANG model that we have written for the device. Again, this model doesn't necessarily have to cover all aspects of the device.

Often a useful technique with generic NEDs can be to write a pyang plugin to generate code for the generic NED. Again, depending on the device it may be possible to generate Java code from a pyang plugin that covers most or all aspects of mapping an array of NedEditOp objects into the equivalent reconfiguration commands for the device.

Pyang is an extensible and open-source YANG parser (written by Tail-f) available at http://www.yang-central.org. pyang is also part of the NSO release. A number of plugins are shipped in the NSO release, for example $NCS_DIR/lib/pyang/pyang/plugins/tree.py is a good plugin to start with if we wish to write our own plugin.

$NCS_DIR/examples.ncs/generic-ned/xmlrpc-device is a good example to start with if we wish to write a generic NED. It manages a set of devices over the XML-RPC protocol. In this example, we have:

• Defined a fictitious YANG model for the device.

• Implemented an XML-RPC server exporting a set of RPCs to manipulate that fictitious data model. The XML-RPC server runs the Apache org.apache.xmlrpc.server.XmlRpcServer Java package.

• Implemented a Generic NED which acts as an XML-RPC client speaking HTTP to the XML-RPC servers.

The example is self-contained, and we can, using the NED code, manipulate these XML-RPC servers in a manner similar to all other managed devices.

Tweaking the Order of NedEditOp Objects

As it was mentioned earlier the NedEditOp objects are relative to the YANG model of the device, and they are to be translated into the equivalent reconfiguration operations on the device. Applying reconfiguration operations may only be valid in a certain order.

For Generic NEDs, NSO provides a feature to ensure dependency rules are being obeyed when generating a diff to commit. It controls the order of operations delivered in the NedEditOp array. The feature is activated by adding the following option to package-meta-data.xml:

When the ordered-diff flag is set, the NedEditOp objects follow YANG schema order and consider dependencies between leaf nodes. Dependencies can be defined using leafrefs and the tailf:cli-diff-after, tailf:cli-diff-create-after, tailf:cli-diff-modify-after, tailf:cli-diff-set-after, tailf:cli-diff-delete-after YANG extensions. Read more about the above YANG extensions in the Tail-f CLI YANG extensions man page.

NED Commands

A device we wish to manage using a NED usually has not just configuration data that we wish to manipulate from NSO, but the device usually has a set of commands that do not relate to configuration.

The commands on the device we wish to be able to invoke from NSO must be modeled as actions. We model this as actions and compile it using a special ncsc command to compile NED data models that do not directly relate to configuration data on the device.

The NSO example $NCS_DIR/examples.ncs/generic-ned/xmlrpc-device contains an example where the managed device, a fictitious XML-RPC device contains a YANG snippet :

When that action YANG is imported into NSO it ends up under the managed device. We can invoke the action on the device as :

The NED code is obviously involved here. All NEDs must always implement:

The command() method gets invoked in the NED, the code must then execute the command. The input parameters in the params parameter correspond to the data provided in the action. The command() method must reply with another array of ConfXMLParam objects.

The above code is fake, on a real device, the job of the command() method is to establish a connection to the device, invoke the command, parse the output, and finally reply with an ConfXMLParam array.

The purpose of implementing NED commands is usually that we want to expose device commands to the programmatic APIs in the NSO DOM tree.

The Tools view offers individual utilities that you can use to run specific tasks on your deployment, such as running compliance reports, etc.

The following tools are available:

• Insights: Gathers and displays useful statistics of your deployment.

• Package upgrade: Used to perform upgrades to the packages running in NSO.

• : Used to manage a High Availability (HA) setup in your deployment.

• : Shows current alarms/events in your deployment and provides options to manage them.

• : Shortcut to the Commit Manager.

• : Used to run compliance checks on your NSO network.

Insights

The Insights view collects and displays the following types of operational information using the /ncs:metrics data model to present useful statistics:

• Real-time data about transactions, commit queues, and northbound sessions.

• Sessions created and closed towards northbound interfaces since the last restart (CLI, JSON-RPC, NETCONF, RESTCONF, SNMP).

• Transactions since last restart (committed, aborted, and conflicting). You can select between the running and operational data stores.

• Devices and their sync statuses.

Package Upgrade

In the Package upgrade view, you can load custom packages in NSO.

The Reload button on the Packages pane is the equivalent of the packages reload command in CLI. Read more about the reload action in .

Install a Package

1. In the Package upgrade view, click Browse files to select a new package (.tar or .tar.gz) from your local disk.

2. Click Upload. The package becomes visible under the Available pane. (The Progress Trace shows the real-time progress of the upload).

3. Click Install.

Uninstall a Package

• To uninstall a package, simply click Deinstall next to the package in the Loaded packages list.

High Availability

The High Availability view is used to visualize your HA setup (rule-based or Raft).

Actions can be performed on the cluster using the Configuration editor -> Actions tab. Possible actions are further described in the High Availability documentation under .

Alarms

The Alarm manager view displays current alarms in the system. The alarms are categorized as criticals, majors, and minors and can be filtered by device.

You can run actions on an alarm by selecting it and using the run action button.

Commit Manager

The Commit manager displays notifications about commits pending to be approved. Any time a change (a transaction) is made in NSO, the Commit Manager displays a notification to review the change. You can then choose to confirm or revert the commit.

Review a Configuration Change

1. Access the Commit Manager by clicking its icon in the banner.

2. Review the available changes appearing as Current transaction. If there are errors in the change, the Commit Manager alerts you and suggests possible corrections. You can then fix them and press Re-validate to clear the errors.

3. Click Revert to undo or Commit to confirm the changes in the transaction.

Load/Save Configuration Data

Start a transaction to load or save configuration data using the Load/Save option which you can then review for commit. The following tabs are available:

• Rollback, to load data that reverts an earlier change.

• Files, to load data from a local file on your disk.

• Paste, to load data by pasting it in.

• Save, to save loaded data to a file on your local disk.

Commit Manager Tabs

In the Commit manager view, the following tabs are shown.

• changes tab, to list the changes and actions done in the system, e.g., deleting a device or changing its properties.

• errors tab, to list the errors encountered while doing changes. You can review the errors, make changes, and revalidate the error using the Re-validate option.

• warnings tab, to list the warnings encountered while doing changes.

Compliance Reporting

The Compliance reporting view is used to create and run compliance reports to check the current situation, check historical events, or both.

The following main tabs are available in this view:

• Compliance reports, to create, run, manage, and view existing compliance reports.

• Report results, to view report results and compliance report status.

Create a Compliance Report

1. In the Compliance reporting view, click Add list item .

2. In the New report pop-up, enter the report name and confirm.

3. Next, set up the compliance report using the following tabs. For a more detailed description of Compliance Reporting concepts and related configuration options, see documentation.

Run a Compliance Report

1. In the Compliance reports tab, click the Run button on the desired report.

2. Specify the following:

   • Report title

   • Historical time interval. The report runs with the maximum possible interval if you do not specify an interval.

The report's results, available from the Report results tab, show if the report was compliant or has violations. Click Show details to fetch additional details.

Before starting NSO, it is recommended to explore the installation contents.

Navigate to the newly created Installation Directory, for example:

Contents of the Installation Directory

The installation directory includes the following contents:

Documentation

Along with the binaries, NSO installs a full set of documentation available in the doc/ folder in the Installation Directory. There is also an online version of the documentation available on .

Run index.html in your browser to explore further.

Examples

Local Install comes with a rich set of examples to start using NSO.

Network Element Drivers (NEDs)

In order to communicate with the network, NSO uses NEDs as device drivers for different device types. Cisco has NEDs for hundreds of different devices available for customers, and several are included in the installer in the /packages/neds directory.

In the example below, NEDs for Cisco ASA, IOS, IOS XR, and NX-OS are shown. Also included are NEDs for other vendors including Juniper JunOS, A10, ALU, and Dell.

Install New NEDs

A large number of pre-built supported NEDs are available which can be acquired and downloaded by the customers from . Note that the specific file names and versions that you download may be different from the ones in this guide. Therefore, remember to update the paths accordingly.

Like the NSO installer, the NEDs are signed.bin files that need to be run to validate the download and extract the new code.

To install new NEDs:

1. Change to the working directory where your downloads are. The filenames indicate which version of NSO the NEDs are pre-compiled for (in this case NSO 6.0), and the version of the NED. An example output is shown below.

2. Use the sh command to run signed.bin to verify the certificate and extract the NED tar.gz and other files. Repeat for all files. An example output is shown below.

3. You now have three tar (.tar.gz) files. These are compressed versions of the NEDs. List the files to verify as shown in the example below.

Shell Scripts

The last thing to note is the files ncsrc and ncsrc.tsch. These are shell scripts for bash and tsch that set up your PATH and other environment variables for NSO. Depending on your shell, you need to source this file before starting NSO.

For more information on sourcing shell script, see the .

The SNMP agent in NSO is used mainly for monitoring and notifications. It supports SNMPv1, SNMPv2c, and SNMPv3.

The following standard MIBs are supported by the SNMP agent:

• SNMPv2-MIB

• SNMP-FRAMEWORK-MIB

• SNMP-USER-BASED-SM-MIB

This section explains the different locks that exist in NSO and how they interact. It is important to understand the architecture of NSO with its management backplane, and the transaction state machine as described in to be able to understand how the different locks fit into the picture.

Global Locks

The NSO management backplane keeps a lock on the datastore running. This lock is usually referred to as the global lock and it provides a mechanism to grant exclusive access to the datastore.

The global is the only lock that can explicitly be taken through a northbound agent, for example by the NETCONF <lock> operation, or by calling Maapi.lock()

All user code that needs to run in NSO must be part of a package. A package is basically a directory of files with a fixed file structure or a tar archive with the same directory layout. A package consists of code, YANG modules, etc., that are needed to add an application or function to NSO. Packages are a controlled way to manage loading and versions of custom applications.

Network Element Drivers (NEDs) are also packages. Each NED allows NSO to manage a network device of a specific type. Except for third-party YANG NED packages which do not contain a YANG device model by default (and must be downloaded and fixed before adding to the package), a NED typically contains a device YANG model and the code, specifying how NSO should connect to the device. For NETCONF devices, NSO includes built-in tools to help you build a NED, as described in NED Administration, that you can use if needed. Otherwise, a third-party YANG NED, if available, should be used instead. Vendors, in some cases, provide the required YANG device models but not the entire NED. In practice, all NSO instances use at least one NED. The set of used NED packages depends on the number of different device types the NSO manages.

When NSO starts, it searches for packages to load. The ncs.conf parameter /ncs-config/load-path defines a list of directories. At initial startup, NSO searches these directories for packages and copies the packages to a private directory tree in the directory defined by the /ncs-config/state-dir parameter in ncs.conf, and loads and starts all the packages found. On subsequent startups, NSO will by default only load and start the copied packages. The purpose of this procedure is to make it possible to reliably load new or updated packages while NSO is running, with a fallback to the previously existing version of the packages if the reload should fail.

In a System Install of NSO, packages are always installed (normally through symbolic links) in the packages subdirectory of the run directory, i.e. by default /var/opt/ncs/packages, and the private directory tree is created in the state subdirectory, i.e. by default /var/opt/ncs/state.

Loading Packages

Loading of new or updated packages (as well as removal of packages that should no longer be used) can be requested via the reload action - from the NSO CLI:

This request makes NSO copy all packages found in the load path to a temporary version of its private directory, and load the packages from this directory. If the loading is successful, this temporary directory will be made permanent, otherwise, the temporary directory is removed and NSO continues to use the previous version of the packages. Thus when updating packages, always update the version in the load path, and request that NSO does the reload via this action.

If the package changes include modified, added, or deleted .fxs files or .ccl files, NSO needs to run a data model upgrade procedure, also called a CDB upgrade. NSO provides a dry-run option to packages reload action to test the upgrade without committing the changes. Using a reload dry-run, you can tell if a CDB upgrade is needed or not.

The report all-schema-changes option of the reload action instructs NSO to produce a report of how the current data model schema is being changed. Combined with a dry run, the report allows you to verify the modifications introduced with the new versions of the packages before actually performing the upgrade.

For a data model upgrade, including a dry run, all transactions must be closed. In particular, users having CLI sessions in configure mode must exit to operational mode. If there are ongoing commit queue items, and the wait-commit-queue-empty parameter is supplied, it will wait for the items to finish before proceeding with the reload. During this time, it will not allow the creation of any new transactions. Hence, if one of the queue items fails with rollback-on-error option set, the commit queue's rollback will also fail, and the queue item will be locked. In this case, the reload will be canceled. A manual investigation of the failure is needed in order to proceed with the reload.

While the data model upgrade is in progress, all transactions are closed and new transactions are not allowed. This means that starting a new management session, such as a CLI or SSH connection to the NSO, will also fail, producing an error that the node is in upgrade mode.

By default, the reload action will (when needed) wait up to 10 seconds for the commit queue to empty (if the wait-commit-queue-empty parameter is entered) and reload to start.

If there are still open transactions at the end of this period, the upgrade will be canceled and the reload operation will fail. The max-wait-time and timeout-action parameters to the action can modify this behavior. For example, to wait for up to 30 seconds, and forcibly terminate any transactions that still remain open after this period, we can invoke the action as:

Thus the default values for these parameters are 10 and fail, respectively. In case there are no changes to .fxs or .ccl files, the reload can be carried out without the data model upgrade procedure, and these parameters are ignored since there is no need to close open transactions.

When reloading packages, NSO will give a warning when the upgrade looks suspicious, i.e., may break some functionality. Note that this is not a strict upgrade validation, but only intended as a hint to the NSO administrator early in the upgrade process that something might be wrong. Currently, the following scenarios will trigger the warnings:

• One or more namespaces are removed by the upgrade. The consequence of this is all data belonging to this namespace is permanently deleted from CDB upon upgrade. This may be intended in some scenarios, in which case it is advised to proceed with overriding warnings as described below.

• There are source .java files found in the package, but no matching .class files in the jars loaded by NSO. This likely means that the package has not been compiled.

• There are matching .class files with modification time older than the source files, which hints that the source has been modified since the last time the package was compiled. This likely means that the package was not re-compiled the last time the source code was changed.

If a warning has been triggered it is a strong recommendation to fix the root cause. If all of the warnings are intended, it is possible to proceed with packages reload force command.

In some specific situations, upgrading a package with newly added custom validation points in the data model may produce an error similar to no registration found for callpoint NEW-VALIDATION/validate or simply application communication failure, resulting in an aborted upgrade. See on how to proceed.

In some cases, we may want NSO to do the same operation as the reload action at NSO startup, i.e. copy all packages from the load path before loading, even though the private directory copy already exists. This can be achieved in the following ways:

• Setting the shell environment variable $NCS_RELOAD_PACKAGES to true. This will make NSO do the copy from the load path on every startup, as long as the environment variable is set. In a System Install, NSO is typically started as a systemd system service, and NCS_RELOAD_PACKAGES=true can be set in /etc/ncs/ncs.systemd.conf temporarily to reload the packages.

• Giving the option --with-package-reload to the ncs

Always use one of these methods when upgrading to a new version of NSO in an existing directory structure, to make sure that new packages are loaded together with the other parts of the new system.

Redeploying Packages

If it is known in advance that there were no data model changes, i.e. none of the .fxs or .ccl files changed, and none of the shared JARs changed in a Java package, and the declaration of the components in the package-meta-data.xml is unchanged, then it is possible to do a lightweight package upgrade, called package redeploy. Package redeploy only loads the specified package, unlike packages reload which loads all of the packages found in the load-path.

Redeploying a package allows you to reload updated or load new templates, reload private JARs for a Java package, or reload the Python code which is a part of this package. Only the changed part of the package will be reloaded, e.g. if there were no changes to Python code, but only templates, then the Python VM will not be restarted, but only templates reloaded. The upgrade is not seamless however as the old templates will be unloaded for a short while before the new ones are loaded, so any user of the template during this period of time will fail; the same applies to changed Java or Python code. It is hence the responsibility of the user to make sure that the services or other code provided by the package is unused while it is being redeployed.

The package redeploy will return true if the package's resulting status after the redeploy is up. Consequently, if the result of the action is false, then it is advised to check the operational status of the package in the package list.

Adding NED Packages

Unlike a full packages reload operation, new NED packages can be loaded into the system without disrupting existing transactions. This is only possible for new packages, since these packages don't yet have any instance data.

The operation is performed through the /packages/add action. No additional input is necessary. The operation scans all the load-paths for any new NED packages and also verifies the existing packages are still present. If packages are modified or deleted, the operation will fail.

Each NED package defines ned-id, an identifier that is used in selecting the NED for each managed device. A new NED package is therefore a package with a ned-id value that is not already in use.

In addition, the system imposes some additional constraints, so it is not always possible to add just any arbitrary NED. In particular, NED packages can also contain one or more shared data models, such as NED settings or operational data for private use by the NED, that are not specific to each version of NED package but rather shared between all versions. These are typically placed outside any mount point (device-specific data model), extending the NSO schema directly. So, if a NED defines schema nodes outside any mount point, there must be no changes to these nodes if they already exist.

Adding a NED package with a modified shared data model is therefore not allowed and all shared data models are verified to be identical before a NED package can be added. If they are not, the /packages/add action will fail and you will have to use the /packages/reload command.

The command returns true if the package's resulting status after deployment is up. Likewise, if the result for a package is false, then the package was added but its code has not started successfully and you should check the operational status of the package with the show packages package <PKG> oper-status command for additional information. You may then use the /packages/package/redeploy action to retry deploying the package's code, once you have corrected the error.

Managing Packages

In a System Install of NSO, management of pre-built packages is supported through a number of actions. This support is not available in a Local Install, since it is dependent on the directory structure created by the System Install. Please refer to the YANG submodule $NCS_DIR/src/ncs/yang/tailf-ncs-software.yang for the full details of the functionality described in this section.

Actions

Actions are provided to list local packages, to fetch packages from the file system, and to install or deinstall packages:

• software packages list [...]: List local packages, categorized into loaded, installed, and installable. The listing can be restricted to only one of the categories - otherwise, each package listed will include the category for the package.

• software packages fetch package-from-file <file>: Fetch a package by copying it from the file system, making it installable.

• software packages install package <package-name> [...]: Install a package, making it available for loading via the packages reload

There is also an upload action that can be used via NETCONF or REST to upload a package from the local host to the NSO host, making it installable there. It is not feasible to use in the CLI or Web UI, since the actual package file contents is a parameter for the action. It is also not suitable for very large (more than a few megabytes) packages, since the processing of action parameters is not designed to deal with very large values, and there is a significant memory overhead in the processing of such values.

More on Package Management

NSO Packages contain data models and code for a specific function. It might be NED for a specific device, a service application like MPLS VPN, a WebUI customization package, etc. Packages can be added, removed, and upgraded in run-time. A common task is to add a package to NSO to support a new device type or upgrade an existing package when the device is upgraded.

(We assume you have the example up and running from the previous section). Currently installed packages can be viewed with the following command:

So the above command shows that NSO currently has one package, the NED for Cisco IOS.

NSO reads global configuration parameters from ncs.conf. More on NSO configuration later in this guide. By default, it tells NSO to look for packages in a packages directory where NSO was started. So in this specific example:

As seen above a package is a defined file structure with data models, code, and documentation. NSO comes with a couple of ready-made packages: $NCS_DIR/packages/. Also, there is a library of packages available from Tail-f, especially for supporting specific devices.

Adding and Upgrading a Package

Assume you would like to add support for Nexus devices to the example. Nexus devices have different data models and another CLI flavor. There is a package for that in $NCS_DIR/packages/neds/nexus.

We can keep NSO running all the time, but we will stop the network simulator to add the Nexus devices to the simulator.

Add the nexus package to the NSO runtime directory by creating a symbolic link:

The package is now in place, but until we tell NSO to look for package changes nothing happens:

So after the packages reload operation NSO also knows about Nexus devices. The reload operation also takes any changes to existing packages into account. The data store is automatically upgraded to cater to any changes like added attributes to existing configuration data.

Simulating the New Device

Adding the New Devices to NSO

We can now add these Nexus devices to NSO according to the below sequence: