The goal of this lab is to demonstrate how to use Ansible AWX (Tower) to configure Cumulus Linux devices based on information extracted from Netbox. This demo will walk you through the following steps:
- Initial Netbox configuration — populating the base Netbox data model with device information and IP address details.
- Configuring AWX — using Netbox as an inventory source for AWX and pulling device and IPAM details from Netbox.
- Using Netbox as a configuration source of truth — populating Netbox with configuration context that will be used by Ansible playbooks to generate final device configs.
Both AWX and Netbox are deployed in the Kubernetes cluster running inside the netq-ts
server. Netbox is deployed using the bootc/netbox-chart
helm chart and AWX is deployed using the AWX operator. They both share the same Postgres database that is deployed as a part of Netbox helm chart.
NOTE: For instructions on how to build the demo, install and configure both Netbox and AWX see the see the
./air
directory.
Device/Application | sw version | username | password |
---|---|---|---|
oob-mgmt-server | Ubuntu 18.04 | ubuntu | nvidia |
netq-ts | NetQ 4.0.0 | cumulus | cumulus |
leaf01, leaf02 | CL 5.0 | cumulus | CumulusLinux! |
netbox | v3.0.11 | admin | admin |
AWX | 19.5.0 | admin | rncnVRf949WvvrZGxQKxSOE0g5bl9mFJ |
In order to connect to the web UI of Netbox and AWX, we'll use SSH port forwarding through oob-mgmt-server
. Change to the "Advanced" lab view in Air and click "Enable SSH Service". Use the following command when connecting to the oob-mgmt-server
(adjust SSH URL based on the generated host and port numbers):
ssh -L 8080:192.168.200.250:30845 -L 8081:192.168.200.250:31768 ssh://[email protected]:22708
This should make Netbox available on localhost:8080 and AWX available on localhost:8081.
The default way of interacting with Netbox is via its web UI, however, in this demo instead of including dozens of screenshots, we'll be using the netbox shell in order to configure Netbox programmatically.
Connect to the netq-ts
node and start a shell inside the citc-netbox
Pod:
ubuntu@oob-mgmt-server:~$ ssh cumulus@netq-ts
cumulus@netq-ts:~$ sudo -i
root@netq-ts:~# kubectl exec -it deploy/citc-netbox bash
source /opt/netbox/venv/bin/activate
/opt/netbox/netbox/manage.py nbshell
Start by creating the basic netbox object model layout with site, device role and manufacturer details.
Site(name="CITC", status="active").save()
site = Site.objects.get(name="CITC")
DeviceRole(name="leaf").save()
role = DeviceRole.objects.get(name="leaf")
Manufacturer(name="nvidia").save()
m = Manufacturer.objects.get(name="nvidia")
DeviceType(model="vx", manufacturer=m).save()
t = DeviceType.objects.get(model="vx")
InterfaceTemplate(name="eth0", device_type=t, mgmt_only=True).save()
InterfaceTemplate(name="lo", device_type=t).save()
for i in range(1,3):
InterfaceTemplate(name=f"swp{i}", device_type=t, type="virtual").save()
Now we can put all these details together to add the two lab devices:
Device(name="leaf01", device_role=role, site=site, device_type=t).save()
leaf01 = Device.objects.get(name="leaf01")
Device(name="leaf02", device_role=role, site=site, device_type=t).save()
leaf02 = Device.objects.get(name="leaf02")
Finally, populate Netbox IPAM with details required to configure the two lab devices:
# create mgmt vrf and prefix
VRF(name="mgmt").save()
vrf_mgmt = VRF.objects.get(name="mgmt")
Prefix(prefix="192.168.200.0/24", vrf=vrf_mgmt).save()
prefix = Prefix.objects.get(prefix="192.168.200.0/24")
# get a pointer to `eth0` interface
leaf01_eth0 = Interface.objects.get(name="eth0", device=leaf01)
leaf02_eth0 = Interface.objects.get(name="eth0", device=leaf02)
# create OOB IP and assign it to `eth0` interface
IPAddress(address="192.168.200.2", vrf=vrf_mgmt, assigned_object=leaf01_eth0 ).save()
leaf01_ip = IPAddress.objects.get(address="192.168.200.2")
leaf01.primary_ip4=leaf01_ip
leaf01.save()
IPAddress(address="192.168.200.3", vrf=vrf_mgmt, assigned_object=leaf02_eth0 ).save()
leaf02_ip = IPAddress.objects.get(address="192.168.200.3")
leaf02.primary_ip4=leaf02_ip
leaf02.save()
# create `default` vrf and assign loopback ips
VRF(name="default").save()
vrf_default = VRF.objects.get(name="default")
Prefix(prefix="10.0.1.0/24", vrf=vrf_default ).save()
leaf01_lo = Interface.objects.get(name="lo", device=leaf01)
leaf02_lo = Interface.objects.get(name="lo", device=leaf02)
IPAddress(address="10.0.1.11", vrf=vrf_default, assigned_object=leaf01_lo ).save()
IPAddress(address="10.0.1.12", vrf=vrf_default, assigned_object=leaf02_lo ).save()
This is all what we need to populate basic Netbox data. This can be verified using Netbox UI at localhost:8080.
NOTE: We're only configuring a minimal set of details about our network and not including things like interface connections or rack layouts. Although these details are helpful, they are not relevant to this demo and can be safely skipped.
In order to use Netbox as an inventory source, we need to provide a way to pass authentication details to the nb_inventory plugin. To do that, add a new credential type for netbox. From AWX dashboard navigate to Administration -> Credential Types and add a new "netbox" credential type.
---
fields:
- type: string
id: netbox_api
label: Netbox API URL
- type: string
id: netbox_token
label: Netbox Token
required:
- netbox_token
- netbox_api
---
env:
NETBOX_API: "{{ netbox_api }}
NETBOX_TOKEN: "{{ netbox_token }}
Now we can create a new credential object with the details of the local Netbox instance, i.e. URL http://citc-netbox
and token 0123456789abcdef0123456789abcdef01234567
:
We also need to create a new credential to access the Cumulus Linux devices:
The default AWX EE execution environment does not include some of the python libraries required to interact with Netbox, so we'd need to create a new one. The container image has already been pre-built, however should you decide to create a custom image, you can see how it can be done by looking at the make ee
command. For now, you can create a new execution environment with the provided pre-built image:
Now we need to tell AWX where to find our playbooks and the inventory by creating a new Project and pointing at the current git repository:
Once saved, AWX will try to fetch the latest commit and should report the job status as "Success".
Create a new Inventory called "netbox" and Navigate to the "Sources" tab to add this git repo as a source and tie it together with the previously created credentails:
Once the source is created and synced, the two lab devices should appear under the "Hosts" tab of the inventory:
Now we can run our first end-to-end test by combining all of the previously configured elements in a single job template.
This job template will execute the "debug" playbook that will do the following:
- Pull all information about Netbox devices (model, type, IP)
- Pull information about all interfaces known to Netbox
- Pull all IPAM information from Netbox
- For each device, print all known information to
stdout
This is how you can verify the details that have been collected by this playbook.
It's quite common to refer to Netbox as the "networking source of truth", however in reality its scope is limited to inventory and IP address management. In order to model the entire network device configuration state, we'll use a feature called configuration context that was designed to store JSON data associated with various Netbox objects. In our case, we'll use this to store a simple BGP configuration for both of our lab devices. We'll use the Netbox's hierarchical rendering to define common configuration for groups of network devices and use local context to override any device-specific settings.
Let's start by defining the common data model that will be shared amongst all devices with "leaf" role. Connect back to the netbox shell as it was described in step #1.
role_context = {'bgp': {'address_family': [{'name': 'ipv4_unicast',
'redistribute': [{'type': 'connected'}]}],
'asn': 65000,
'neighbors': [{'interface': 'swp1',
'peergroup': 'underlay',
'unnumbered': True},
{'interface': 'swp2',
'peergroup': 'underlay',
'unnumbered': True}],
'peergroups': [{'name': 'underlay', 'remote_as': 'external'}]},
'interfaces': [{'name': 'swp1'}, {'name': 'swp2'}]}
role = DeviceRole.objects.get(name="leaf")
ConfigContext(name="leaf", data=role_context).save()
ctx = ConfigContext.objects.get(name="leaf")
ctx.roles.add(role)
For each individual device, override the default BGP AS number:
leaf01 = Device.objects.get(name="leaf01")
leaf02 = Device.objects.get(name="leaf02")
leaf01.local_context_data = { "bgp": { "asn": 65001 }}
leaf02.local_context_data = { "bgp": { "asn": 65002 }}
leaf01.save()
leaf02.save()
To check the final state that will be rendered for "leaf01":
import json
print(json.dumps(leaf01.get_config_context(), indent=2))
Which should display:
{
"bgp": {
"asn": 65001,
"neighbors": [
{
"interface": "swp1",
"peergroup": "underlay",
"unnumbered": true
},
{
"interface": "swp2",
"peergroup": "underlay",
"unnumbered": true
}
],
"peergroups": [
{
"name": "underlay",
"remote_as": "external"
}
],
"address_family": [
{
"name": "ipv4_unicast",
"redistribute": [
{
"type": "connected"
}
]
}
]
},
"interfaces": [
{
"name": "swp1"
},
{
"name": "swp2"
}
]
}
Now that we have the data models defined in AWX, we can use it to provision our lab devices. To do that, we'll create another job template in AWX and point it at the generate.yml
playbook.
The new playbook will go through the following sequence of actions:
- Fetch configuration data from Netbox using the
nb_lookup
plugin. - Using the collected data, generate NVUE configuration file for every device.
- Apply the generated configuration file.
Here's an example of a jinja template that that pulls information from multiple sources and generates the final NVUE JSON configuration file.
{% set config = dict({"set": dict()}) %}
{% set loopback_ip = hostvars[inventory_hostname].netbox_ips | community.general.json_query('[?assigned_object.name==`lo`].address') | first %}
{% include './features/hostname.j2' %}
{# interface config #}
{% set _ = config['set'].update(dict({"interface": dict()})) %}
{% include './features/eth0.j2' %}
{% include './features/swp.j2' %}
{% include './features/loopback.j2' %}
{# bgp config #}
{% set _ = config['set'].update(dict({"router": dict()})) %}
{% include './features/bgp.j2' %}
[
{{ config | to_nice_json }}
]
Once the "provision" job has run succesfully, we should be able to ping between loopback interfaces of leaf01
and leaf02
cumulus@leaf01:mgmt:~$ ping 10.0.1.12 -I 10.0.1.11 -c 2
vrf-wrapper.sh: switching to vrf "default"; use '--no-vrf-switch' to disable
PING 10.0.1.12 (10.0.1.12) from 10.0.1.11 : 56(84) bytes of data.
64 bytes from 10.0.1.12: icmp_seq=1 ttl=64 time=0.368 ms
64 bytes from 10.0.1.12: icmp_seq=2 ttl=64 time=0.395 ms
--- 10.0.1.12 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 39ms
rtt min/avg/max/mdev = 0.368/0.381/0.395/0.023 ms