netreplica / nrx

Netreplica Exporter

---

nrx - netreplica exporter

nrx reads a network topology graph from NetBox DCIM system and exports as one of the following:

• Containerlab topology for container-based networking labs
• Cisco Modeling Labs topology for VM-based labs
• Network visualization format for Graphite or D2
• Graph data as a JSON file in Cytoscape format CYJS
• Any other user-defined format using Jinja2 templates

It can also read the topology graph previously saved as a CYJS file to convert it into other formats.

This project is in early phase. We're experimenting with the best ways to automate software network lab orchestration. If you have any feedback, questions or suggestions, please reach out to us via the Netreplica Discord server linked above, #netreplica channel in NetDev Community on Slack, or open a github issue in this repository.

Latest capabilities added

The latest releases have a significant set of the new capabilities:

• 0.5.0 PyPA packaging and distribution: pip install nrx
• 0.4.0 Ability to create new output formats without a need for nrx code changes
• 0.4.0 Mapping between NetBox platform values and node parameters via platform_map.yaml file
• 0.4.0 $HOME/.nr configuration directory with automatic initialization using --init argument

Find detailed release notes on the Releases page.

Table of contents

• Capabilities
• Compatibility
• Prerequisites
• How to install
• How to configure
  • Command-line arguments
  • Environmental variables
  • Configuration file
  • Configuration directory
• Templates
• How to use
  • Containerlab example
  • Cisco Modeling Labs example
  • Topology Visualization with Graphite
• Credits
  • Original idea and implementation
  • Copyright notice

Capabilities

Data sourcing capabilities:

• Connects to a NetBox instance over an API using a user-provided authentication token
• Exports a network topology graph with Devices that
  • belong to a Site specified with --site parameter
  • have a list of Tags specified with --tags paramater
• A combination of the two methods above is possible
• Only Devices with Roles from a customizable list will be exported
• Direct connections between Devices via Cables will be exported as topology edges
• Connections via Patch Panels and Circuits will be exported as well with help of NetBox Cable Tracing API
• Only Ethernet connections will be exported
• Device configurations will be rendered and exported if not empty
• As an alternative to sourcing live data from NetBox, imports a graph from a previously exported file in CYJS format

Export capabilities:

• Exports the graph as a Containerlab (Clab) topology definition file in YAML format
• Exports the graph as a Cisco Modeling Labs (CML) topology definition file in YAML format
• Exported device configurations can be used as startup-config for Containerlab and CML
• Exports the graph in formats for visualization with Graphite or D2
• User-defined output formats using Jinja2 templates
• Uses NetBox Device Platform slug field to identify node templates when rendering the export file
• Customizable mapping between NetBox Platform values and node parameters via platform_map.yaml file
• Creates mapping between real interface names and interface names used by the supported lab tools
• Calculates level and rank values for each node based on Device Role to help visualize the topology
• Exports the graph into CYJS format that can be later converted into a topology definition file, or used by 3rd party software

Compatibility

The following software versions were tested for compatibility with nrx:

• NetBox v3.4-v3.6. For device configuration export, v3.5 is the minimum version.
• Containerlab v0.39, but earlier and later versions should work fine
• Cisco Modeling Labs v2.5
• Netreplica Graphite v0.4.0

Prerequisites

• Python 3.9+. In the commands below we assume you have python3.9 executable. If it is under a different name, change accordingly.

• PIP

  curl -sL https://bootstrap.pypa.io/get-pip.py | python3.9 -

• Virtualenv (recommended)

  pip install virtualenv

How to install

PyPI package (recommended)

mkdir -p ~/.venv
python3.9 -m venv ~/.venv/nrx
source ~/.venv/nrx/bin/activate
pip install nrx

From source code (development)

After running the following commands, you will have a working nrx command in the current directory.

git clone https://github.com/netreplica/nrx.git --recursive
cd nrx
python3.9 -m venv venv
source venv/bin/activate
pip install -r requirements.txt

How to configure

nrx accepts the following configuration options, in the order of precedence:

1. Command-line arguments
2. Environmental variables
3. Configuration file

Command-line arguments

Command-line arguments take the highest priority.

nrx --help
usage: nrx [-h] [-v] [-d] [-I [VERSION]] [-c CONFIG] [-i INPUT] [-o OUTPUT] [-a API] [-s SITE] [-t TAGS] [-n NAME]
           [--noconfigs] [-k | --insecure] [-f FILE] [-M MAP] [-T TEMPLATES] [-D DIR]

nrx - network topology exporter by netreplica

online documentation: https://github.com/netreplica/nrx/blob/main/README.md

optional arguments:
  -h, --help                show this help message and exit
  -v, --version             show version number and exit
  -d, --debug               enable debug output
  -I, --init [VERSION]      initialize configuration directory in $HOME/.nr and exit. optionally, specify a VERSION to initialize with: -I 0.1.0
  -c, --config CONFIG       configuration file, default: $HOME/.nr/nrx.conf
  -i, --input INPUT         input source: netbox (default) | cyjs
  -o, --output OUTPUT       output format: cyjs | clab | cml | graphite | d2 or any other format supported by provided templates
  -a, --api API             netbox API URL
  -s, --site SITE           netbox site to export, cannot be combined with --sites
      --sites SITES         netbox sites to export, for multiple tags use a comma-separated list: site1,site2,site3 (uses OR logic)
  -t, --tags TAGS           netbox tags to export, for multiple tags use a comma-separated list: tag1,tag2,tag3 (uses AND logic)
  -n, --name NAME           name of the exported topology (site name or tags by default)
      --noconfigs           disable device configuration export (enabled by default)
  -k, --insecure            allow insecure server connections when using TLS
  -f, --file FILE           file with the network graph to import
  -T, --templates TEMPLATES directory with template files, will be prepended to TEMPLATES_PATH list in the configuration file
  -M, --map MAP             file with platform mappings to node parameters (default: platform_map.yaml in templates folder)
  -D, --dir DIR             save files into directory DIR (topology name is used by default). nested relative and absolute paths are OK

To pass authentication token, use configuration file or environment variable:
export NB_API_TOKEN='replace_with_valid_API_token'

Note: for security reasons, there is no argument to pass an API token. Use either an environmental variable or a configuration file.

Environmental variables

As an alternative to a configuration file, use environmental variables to provide NetBox API connection parameters.

# NetBox API URL
export NB_API_URL='https://demo.netbox.dev'
# NetBox API Token
export NB_API_TOKEN='replace_with_valid_API_token'

Configuration file

Use --config <filename> argument to specify a configuration file to use. By default, nrx uses $HOME/.nr/nrx.conf if such file exists. The sample configuration file is provided as nrx.conf. Detailed information on the configuration options can be found in CONFIGURATION.md.

Configuration directory

By default, nrx looks up for the following assets in the $HOME/.nr directory:

• Configuration file: nrx.conf, unless overridden by --config argument
• Templates: templates, which can be supplemented by additional paths with --templates argument

To initialize the configuration directory, run nrx --init. This will create the $HOME/.nr folder and populate it with a configuration file example and a compatible version of the templates.

Templates

nrx renders all topology artifacts using Jinja2 templates. The user points nrx to the set of templates to use with --templates parameter.

If --templates parameter is not provided, nrx will search for Jinja2 files in the templates folder in the current directory, as well as in $HOME/.nr/templates. You can also provide an alternative list of folders to search via TEMPLATES_PATH parameter in the configuration file.

Inside the template folders, the required Jinja2 files are taken from a subfolder matching the desired output format. For example, if the output format is clab for Containerlab, then templates are taken from clab subfolder. For Cisco Modelling Labs cml format the subfolder would be cml.

A user can create their own templates for any output format and store them in a subfolder with a format name they would use for --output argument. To make the new output format available to nrx, an entry describing basic properties of the format must be added to formats.yaml file in the templates folder.

To identify which template to use for each device in the topology, nrx uses the slug field of the device's platform field in NetBox. If a template with a name matching the platform slug exists, it would be used by default. Since naming of the platforms is unique for every NetBox deployment, it is not possible to create a generic library of templates that could work out-of-the box for all users. Instead, nrx uses a mapping file platform_map.yaml to identify which template to use for each platform, with possible additional parameters like value of the image tag for Containerlab nodes.

The full list of template search rules:

• <format>/topology.j2: template for the final topology file. Mandatory.
• <format>/nodes/<kind>.j2: templates for individual node entries in the topology file, with default.j2 being mandatory as a fallback template.
• <format>/interface_names/<kind>.j2: templates for generating emulated interface names used by each kind with default.j2 being a fallback template. Optional, as not all output formats need emulated interface names. For example, not needed for visualization output formats.
• <format>/interface_maps/<kind>.j2: templates for mappings between real interface names and emulated interface names used by this NOS kind. Optional, as not all kinds support such mappings.

The nrx repository includes a set of netreplica/templates as a submodule. See more details about available templates in the templates/README.md.

Although you can always directly customize the templates according to your needs, the platform map file often provides less intrusive way. It should be used if you need to tell nrx which templates to use for the Device Platform values in your NetBox system. Also, you can override node images to be used instead of the names specified in the templates, as well as many other node parameters. See Platform Map for details.

How to use

Start with activating venv environment. See How to install if you didn't install nrx yet.

source ~/.venv/nrx/bin/activate

If this is first time you're running nrx, you need to initialize its configuration directory. This will create the $HOME/.nr folder and populate it with a configuration file example and a compatible version of the templates. The examples below don't require a configuration file, but do require the templates to be present in the configuration directory.

nrx --init

Containerlab example

1. Run nrx --output clab to export a topology graph from NetBox in Containerlab format. See How to configure for details. Here is an example of running nrx to export a graph for NetBox Site "DM-Albany" from NetBox Demo instance:

   export NB_API_TOKEN='replace_with_valid_API_token'
   nrx --api https://demo.netbox.dev --templates templates --output clab --dir demo --site DM-Albany

2. Now you're ready to start the Containerlab topology. Here is the example for "DM-Albany" site

   sudo -E containerlab deploy -t demo/DM-Albany.clab.yaml --reconfigure

3. Without --output clab argument, nrx will save data from NetBox as a CYJS file <site_name>.cyjs

   export NB_API_TOKEN='replace_with_valid_API_token'
   nrx --api https://demo.netbox.dev --site DM-Albany --dir demo

4. If you have a CYJS file, run nrx --input cyjs --file <site>.cyjs --output clab to create a Containerlab topology file from the CYJS graph you exported in the previous step. For example, run:

   nrx --input cyjs --file demo/DM-Albany.cyjs --templates templates --output clab --dir demo

Cisco Modeling Labs example

1. Run nrx --output cml to export a topology graph from NetBox in CML format. See How to configure for details. Here is an example of running nrx to export a graph for NetBox Site "DM-Akron" from NetBox Demo instance:

   export NB_API_TOKEN='replace_with_valid_API_token'
   nrx --api https://demo.netbox.dev --templates templates --output cml --dir demo --site DM-Akron

2. Now you're ready to start the "DM-Akron" topology in CML.

   • Open your CML Dashboard in a browser
   • Choose "IMPORT"
   • Use DM-Akron.cml.yaml as a file to import. The import status should be Imported.
   • Choose "GO TO LAB". In SIMULATE menu, choose START LAB
   • Use NODES menu to monitor the status of each node

3. Without --output cml argument, nrx will save data from NetBox as a CYJS file <site_name>.cyjs

   export NB_API_TOKEN='replace_with_valid_API_token'
   nrx --api https://demo.netbox.dev --dir demo --site DM-Akron

4. If you have a CYJS file, run nrx --input cyjs --file <site>.cyjs --output cml to create a topology file from the CYJS graph you exported in the previous step. For example, run:

   nrx --input cyjs --file demo/DM-Akron.cyjs --templates templates --output cml --dir demo

Topology Visualization with Graphite

A combination of netreplica nrx and graphite tools can be used to visualize NetBox topology data. Unlike typical plugin-based visualizers, this method can work with a standard NetBox instance without any plugins installed. You also don't need an administrative access to the NetBox host in order to use this type of visualization.

Follow a two-step process:

1. Export topology data from NetBox in the Graphite format: nrx -o graphite. For example, let's export "DM-Akron" site from the NetBox Demo instance:

   export NB_API_TOKEN='replace_with_valid_API_token'
   nrx --api https://demo.netbox.dev --site DM-Akron --templates templates --output graphite

2. Start Graphite to visualize "DM-Akron" site:

   TOPOLOGY="$(pwd)/DM-Akron/DM-Akron.graphite.json"
   docker run -d -t --rm \
       --mount type=bind,source="${TOPOLOGY}",target=/htdocs/default/default.json,readonly \
       -p 8080:80 \
       --name graphite \
       netreplica/graphite:latest

   Open http://localhost:8080/graphite to see the topology. If you're running Graphite on a remote host, or inside a VM, use this helper to show a working URL:

   docker exec -t -e HOST_CONNECTION="${SSH_CONNECTION}" graphite graphite_motd.sh 8080

   The visualization should be similar to

   

   To stop Graphite, run

   docker stop graphite

If you'd like to be able to switch between multiple exported topologies without restarting Graphite, use one of the methods described in Graphite documentation.

Credits

Original idea and implementation

This is a NANOG-87 Hackathon project. The original project slides. The project team:

• Alex Bortok
• Chip Gwyn
• Toni Yannick Kalombo

The implementation is inspired by ContainerLab random labs by Renato Almeida de Oliveira.

Device configuration export

We added capabilities to export device configurations at NANOG-88 Hackathon. The project team:

• Alex Bortok
• Mau Rojas
• Ahmed Elmokashfi

Watch the demo of the project on YouTube:

Copyright notice

Copyright 2023,2024 Netreplica Team

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

nrx's Issues

Filter devices for export via tag

Currently, when two devices are on different sites and are connected via a circuit, they can't be both exported:

./nrx.py -c main.conf -s colo -o clab -d
...
Debug: Traced sw1 Ethernet21 <-> dc1-spine-1 Ethernet21/1: [[Ethernet21], colo-dc1 <> Ethernet21, [colo-dc1: Termination A], [colo-dc1: Termination Z], colo-dc1 <> Ethernet21/1, [Ethernet21/1]]
Debug: One or both devices for this connection are not in the export graph
...

We need a different way to filter devices using tags instead of site name. For example, if we have a tag wan assigned to sw1 and dc1-spine-1 devices, they should both be exported with a connection between them:

./nrx.py -c main.conf -t wan -o clab -d

Note,-t flag is currently used for --templates and needs to be reassigned for --tags as such usage would be more common, while templates path would more typically be configured in the configuration file. As a flag, -T should be used as a short for --templates

create pip package

Use tracing to build netbox topology

Currently, only Cables directly connected to Device Interfaces are exported into a topology. To cover connections via patch panels and circuits, nrx needs to use netbox tracing API.

Error: Exporting from NetBox: object has no attribute "a_terminations"

Hi !
I'm testing nrx against netbox 3.1.11 and i git the output below.

./nrx.py --output clab --config nrx.conf -d
Debug: arguments Namespace(config='nrx.conf', input='netbox', output='clab', api=None, site=None, insecure=None, debug=True, file=None, templates=None)
Debug: returned site data GENEVA
Exporting NetBox 'GENEVA' site from:            https://nbx.rare.nmaas.eu
Debug: Added device: {'id': 12, 'type': 'device', 'name': 'E513-E-YECWH-1', 'node_id': 0, 'platform': 'unknown', 'platform_name': 'unknown', 'vendor': 'unknown', 'vendor_name': 'unknown', 'model': 'wedge-100bf-32qs', 'model_name': 'Wedge 100BF-32QS', 'role': 'gp4l-node', 'device_index': 0}
Debug: E513-E-YECWH-1 Ethernet interfaces:
Debug: E513-E-YECWH-1 : sdn3 : 100gbase-x-qsfp28
Debug: Added device: {'id': 6, 'type': 'device', 'name': 'GVA0001', 'node_id': 2, 'platform': 'unknown', 'platform_name': 'unknown', 'vendor': 'unknown', 'vendor_name': 'unknown', 'model': 'wedge-100bf-32x', 'model_name': 'Wedge 100BF-32X', 'role': 'gp4l-node', 'device_index': 1}
Debug: GVA0001 Ethernet interfaces:
Debug: GVA0001 : sdn2 : 100gbase-x-qsfp28
Error: Exporting from NetBox: object has no attribute "a_terminations"

I'm using rare clab kind so I added rare.j2 in template folder:

        {{ name }}:
            kind: rare
            image: ghcr.io/rare-freertr/freertr-containerlab:latest
            labels:
                graph-icon: switch
                {% include 'clab/labels.j2' %}

Do i miss something ?

As it is a cloud service, should I ask my admin to update to netbox 3.5.0 ?

With introduction of formats.yaml cyjs as output option doesn't work

./nrx.py --site 'DM-Akron' -D /tmp 
Error: [FORMAT] Output format 'cyjs' is not found in formats.yaml under ['templates', '/Users/user/.nr/templates']

open-ended output format

ignore netbox TLS certificate verification

introduce versions

Handle large number of links

When running against a large topology with 10+ devices and many cables the export fails with the error

Error: NetBox API failure at get cables: The request failed with code 414 Request-URI Too Large but more specific details were not returned in json. Check the NetBox Logs or investigate this exception's error attribute.

Allow multi-site export

Having a single site is nice, but for backbones or other large interconnected networks, having a single topology showing all sites, or multiple sites would be helpful.

export device configurations

• get device config from netbox
• save non-empty device config as a file
• render clab topo with startup-configs
• render cml topo with embedded configs
• parameter to control config rendering
• save configs and interface maps into a subfolder

add subcommand to download templates

nrx --init to download templates and example configuration file into $HOME/.nr

export in graphite format

Direct visualization of netbox data using netreplica/graphite

nrx.py --input netbox --output graphite --site dc1

this should create a file dc1.graphite.json:

rm topology.graphite.json
cp dc1.graphite.json topology.graphite.json
TOPOLOGY="$(pwd)/topology.graphite.json"

To visualize the file

docker run -d -t --rm \
  --mount type=bind,source="${TOPOLOGY}",target=/htdocs/default/default.json,readonly \
  -p 8080:80 \
  --name graphite \
  netreplica/graphite

Open http://localhost:8080/graphite/index.html to see the visualization

To map a folder with clab-{topology.name} subdirectories, use:

CLABDIR=$(pwd)
docker run -d -t --rm \
  -v "${CLABDIR}":/htdocs/lab:ro \
  -p 8080:80 \
  --name graphite \
  netreplica/graphite:nanog86

map platforms to kinds

Create a mechanism to define which kind template should be used for platform values exported from netbox, without using symlinks in the templates. Use of symlinks forces users to maintain custom template repository copies, and this complicates the ability to pull the latest addition to the templates.

use device platform as clab kind

Platform for SW => Clab kind

• slug as platform
• name as plaform_name

Also export the following Device data for future use, for example, as Clab type when supported

• Manufacturer slug as vendor
• Manufacturer name as vendor_name
• Model slug as model
• Model name as model_name

generate interface maps based on kind not platform

avoid excessive logging like this:

Debug: [TEMPLATE] Unable to open interface map template 'clab/interface_maps/linux.j2' for platform 'linux' with path ['../../../templates']. Reason: clab/interface_maps/linux.j2

WAN network topology

Hi, Thanks for nrx development. We are managing a WAN network with mostly one hardware in each site.
What would be your recommandation in order to use nrx in that context ? Would there be in the long run:
1- a mode considering all nodes from a list of sites (e.g. sites=all)
2- consider circuit/uplink connectivity betwwen these sites ?
Thanks/Frederic

Document how to override `image` parameters

Use `platforms_map.yaml` from `.nr` folder

If a user has a customized platforms_map.yaml it would be better to store it outside of templates folder, so it won't be overwritten or conflict with git.

Since the file is searched through the same methods as the templates, would it have more sense to store the edited file under a custom folder where the custom templates could be stored as well? The custom folder could be placed under .nr by the user, or any other place, and path to it added to nrx.conf.

Alternatively, PLATFORM_MAP key in the nrx.conf could be used as well, as a replacement for -M parameter.

It might be useful to use $HOME/.nr/platform_map.yaml as a default value for PLATFORM_MAP, instead of current platform_map.yaml. [Implementation comment] didn't change the default, since it can be easily accomplished now via the combination if PLATFORM_MAP key in the nrx.conf and $HOME/.nr/platform_map.yaml value.

Is it possible to improve usability of template paths in nrx.conf by leveraging $HOME env var, instead using an absolute path? [Implementation comment] yes, implemented for several path-related parameters.

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• Update dependency Jinja2 to v3.1.4
• Update dependency urllib3 to v2.2.2
• Update dependency certifi to v2024.6.2
• Update dependency idna to v3.7
• Update dependency networkx to v3.3
• Update dependency requests to v2.32.3
• Update GitHub Artifact Actions to v4 (major) (actions/download-artifact, actions/upload-artifact)
• Update actions/setup-python action to v5
• Click on this checkbox to rebase all open PRs at once

Detected dependencies

git-submodules

.gitmodules

• templates main@e6551a5125cd511c56f0a4cf345ad5042a60e5fb

github-actions

.github/workflows/pylint.yml

• actions/checkout v4
• actions/setup-python v5

.github/workflows/pypi.yml

• actions/checkout v4
• actions/setup-python v4
• actions/upload-artifact v3
• actions/download-artifact v3

.github/workflows/systest.yml

• actions/checkout v4
• actions/setup-python v5

pep621

pyproject.toml

• charset-normalizer >=3.3.2
• idna >=3.6
• Jinja2 >=3.1.3
• MarkupSafe >=2.1.5
• networkx >=3.2.1
• pynetbox >=7.3.3
• requests >=2.31.0
• toml >=0.10.2
• urllib3 >=2.2.1
• pyyaml >=6.0.1

pip_requirements

requirements-dev.txt

requirements.txt

• certifi ==2024.2.2
• charset-normalizer ==3.3.2
• idna ==3.6
• Jinja2 ==3.1.3
• MarkupSafe ==2.1.5
• networkx ==3.2.1
• pynetbox ==7.3.3
• requests ==2.31.0
• toml ==0.10.2
• urllib3 ==2.2.1
• pyyaml ==6.0.1

---

• Check this box to trigger a request for Renovate to run again on this repository

Export Device Types from NetBox

Export Device Types from NetBox and map them to Containerlab node types. This would allow to emulate different device models of the same platform. On Containerlab side this is supported by SR Linux

do not pull configs from NB if output doesn't use them

[FR] Visualization with d2lang

Requested with a proposed implementation by @frederic-loui
netreplica/templates#12

hrx help improvements

• URL for documentation
• ENV vars for API token especially

Option to filter links between devices via tags

Even if two interconnected devices should be present in the exported topology, it is sometimes necessary to limit the list of connections between them to a certain subset. For example, in a multi-tenant setup, some links maybe dedicated for one customer, and other links to another. To filter a list of connections tags could be used.

Requestor: @frederic-loui

use per-kind templates for emulated interface names

use `templates` path instead of `.` by default