nautobot / nornir-nautobot Goto Github PK
View Code? Open in Web Editor NEWNornir inventory and plugins for Nautobot.
Home Page: https://docs.nautobot.com/projects/nornir-nautobot/en/latest/
Nornir inventory and plugins for Nautobot.
Home Page: https://docs.nautobot.com/projects/nornir-nautobot/en/latest/
Add ability to add options for groups, in a similar fashion to what the Ansible inventory module will support.
options: {
"group_by": ["tags", "sites"]
}
When working with the generate_config function the current behavior has any Exceptions from Jinja2 being masked by raising a NornirNautobotException with log message. This means that any logging that is done in a Job is unable to see the full traceback. If we instead re-raised the same Exception we'd have access to the traceback and could log it. The traceback is essential with config generation as it allows quick identification of where the error lies. For example:
The full traceback is this:
[ERROR/ForkPoolWorker-6] Host 'jcy-spine-01.infra.ntc.com': task 'template_file' failed with traceback:
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/nornir/core/task.py", line 99, in start
r = self.task(self, **self.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/nornir_jinja2/plugins/tasks/template_file.py", line 43, in template_file
text = t.render(host=task.host, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/jinja2/environment.py", line 1301, in render
self.environment.handle_exception()
File "/usr/local/lib/python3.11/site-packages/jinja2/environment.py", line 936, in handle_exception
raise rewrite_traceback_stack(source=source)
File "/opt/nautobot/git/templates/cisco_nxos.j2", line 103, in top-level template code
{% include './nxos/vlans.j2' %}
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/nautobot/git/templates/nxos/vlans.j2", line 1, in top-level template code
{% for vlan in site["vlans"] %}
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/jinja2/sandbox.py", line 303, in getitem
return obj[argument]
~~~^^^^^^^^^^
jinja2.exceptions.UndefinedError: 'site' is undefined
but the Job logs only show E1010: There was a jinja2.exceptions.UndefinedError error: 'site' is undefined
. You have to look at the STDERR logs to find the traceback. It'd be much more useful if we could log the traceback as that's where the important information is.
Right now if you go into >
mode on cisco_ios and you don't have permissions to run show run
it isn't going to try the DEVICE_SECRET
and go into enable mode to get the config.
The job itself shows successful but the backup itself in Git is saved as:
^
% Invalid input detected at '^' marker.
Seems like some interop with using platform_slug_map from golden config and overloading a platform from that map in the dispatcher.
GC user has platform map of:
"platform_slug_map": {"junos": "juniper_junos"},
And dispatcher point to netmiko juniper_junos
"dispatcher_mapping": {
"juniper_junos": "nornir_nautobot.plugins.tasks.dispatcher.juniper_junos.NautobotNornirDriver"
}
with dispatcher_mapping set to juniper_junos
getting:
Unable to find the driver for check_connectivity for platform: junos
if setting the dispatcher_mapping to junos
getting:
^
syntax error, expecting <command>.
While debugging this is because its pulling back command to run based on default key which is show run
.
and in this case add a junos
key.
Hi all.
In the Build Inventory method of nornir-nautobot we can see:
host_platform=device.platform.slug
But looking at Nautobot 1.0, it is something more like:
host_platform=device.device_type.manufacturer.slug
That is working fine...
Thanks
Regards
The get_config
methods found in the dispatcher does not close the connection per host. In golden config connections for this task are not closed until after the task has run for every host.
This can cause issues such as:
Currently when running the default get_config
we need to send in a file path to backup_file
so it can save the config to a file:
nornir-nautobot/nornir_nautobot/plugins/tasks/dispatcher/default.py
Lines 79 to 82 in 5f1236e
If we want to get the configuration but not save it we should be able to send in an empty string to disable this functionality.
Currently pyproject.toml
only supports netutils
versions <1
. Update pyproject.toml
and poetry.lock
to support the latest netutils
There should be able to build in tests for functions pretty easily for REST API driven systems.
Multiple users have had issues where the import statements need to be updated in dispatcher/default.py.
From:
from netmiko.ssh_exception import NetmikoAuthenticationException, NetmikoTimeoutException
To:
from netmiko import NetmikoAuthenticationException, NetmikoTimeoutException
Looking to bump netmiko to 4.x support
Hello,
I wasn't sure under which Github repository to leave this issue so if I should open it under the nornir_netmiko library instead let me know.
The nornir_nautobot library allows us to map non-standard platform slugs to the correct [Nornir Dispatcher class].(https://docs.nautobot.com/projects/plugin-nornir/en/latest/user/app_feature_dispatcher/#configuring-the-dispatcher)
This is very convenient, and I was attempting to take advantage of this but noticed my Nautobot Golden Config backup jobs were still failing. Here is the code trace I've done and what I believe is the root cause of the failure:
netmiko_send_command
task.napalm_to_netmiko_map = {
"ios": "cisco_ios",
"nxos": "cisco_nxos",
"nxos_ssh": "cisco_nxos",
"eos": "arista_eos",
"junos": "juniper_junos",
"iosxr": "cisco_xr",
}
However, because we are using a non-standard platform string this mapping fails and throws an exception.
Would it be possible to allow us to either also configure the napalm_to_netmiko_map to include our non-standard platform strings, or have some way of passing an overriding platform value when calling the netmiko_send_command
task and establishing the Netmiko connection.
Another possibility may be to use the Napalm driver instead, but I also believe there modifications would be necessary to pass a configured platform slug.
I am of course open to any other suggestions or ideas you may have for working around this issue and please let me know if you'd like more information.
Add flag to unpack the pynautobot object into a dictionary. Should default to True to unpack.
It's impossible for now to connect to a device through a proxy SOCKS or any bastion server, would be possible to add this feature ?
It currently would appear that devices do not allow for the secret password to actually do anything.
Nornir allows you to set defaults for defaults for Host attributes, like username and password.
This currently doesn't seem to work when using nornir_nautobot.
Consider the following code snippet
from nornir import InitNornir
nr = InitNornir(
inventory={
"plugin": "NautobotInventory",
"options": {
"nautobot_url": "https://myinstance.of.nautobot.tld",
"nautobot_token": "api_token",
},
},
)
nr.inventory.defaults.username = "username"
The expectation would be that every host in the inventory has it's username attribute set with the value "username", if no explicit value was set for that particular host.
>>> nr.inventory.hosts["a_host"].username
"username"
However the actual result is
>>> nr.inventory.hosts["a_host"].username
>>> nr.inventory.hosts["a_host"].username is None
True
For this to work properly, the inventory plugin would have to instantiate a Hosts object with a defaults attribute, which doesn't seem to happen right now.
Hallo!
if revert_in
is used in next method :
napalm_confirm_commit
needs to be used to actually confirm the commit:
from nornir_napalm.plugins.tasks import napalm_confirm_commit
if revert_in is not None:
confirm_result = task.run(task= napalm_confirm_commit)
otherwise, the system will rollback to the backup config every time.
As an owner of Check Point firewalls I want to be able to take backups with Golden Config.
This is achieved by setting the netmiko platform to linux
and send the command clish -c "show configuration"
Currently when the dispatched fails, the operator is provided with an error based on f"Unable to find the driver for {method} for platform: {task.host.platform}")
. It would be good to add a link into that error message to point to a website of what was supported from a platform and method perspective, similar to the napalm getters.
Problem occure when we have more then one line to substitute from the running config.
what happends is that for every extra line we have defined in substitute-line we also get an extra copy of the config-line.
nornir-nautobot/nornir_nautobot/plugins/tasks/dispatcher/default.py
Lines 291 to 302 in 61172c6
example issue when having 3 lines defined to match against:
Building configuration...
Building configuration...
Building configuration...
Current configuration : 76809 bytes
Current configuration : 76809 bytes
Current configuration : 76809 bytes
suggestion:
# Loop through the config lines
for line in config.split("\n"):
# Loop through the replacement list on each line in the config
match = False
for item in substitute_lines:
if re.match(pattern=fr"{item['regex_search']}", string=line):
line = line.replace(
re.match(pattern=fr"{item['regex_search']}", string=line).groups()[0], item["regex_replacement"]
)
match = True
if match is True:
new_config += line.rstrip() + "\n"
break
if match is False:
new_config += line.rstrip() + "\n"
config = new_config
Nautobot Version - 2.2.6
Using the nautobot inventory plugin for nornir, when attempting to access config context data for a device, only local config context is available. Rendered config context data is not available for each host. requirements.txt below:
anyio==4.3.0
appnope==0.1.3
asttokens==2.4.1
backcall==0.2.0
bcrypt==4.0.1
certifi==2023.7.22
cffi==1.16.0
charset-normalizer==3.3.0
colorama==0.4.6
cryptography==41.0.4
decorator==5.1.1
executing==2.0.0
future==0.18.3
h11==0.14.0
httpcore==0.17.3
httplib2==0.22.0
httpx==0.24.1
idna==3.4
ipdb==0.13.13
ipython==8.17.1
jedi==0.19.1
Jinja2==3.1.2
junos-eznc==2.6.8
lxml==4.9.3
markdown-it-py==3.0.0
MarkupSafe==2.1.3
matplotlib-inline==0.1.6
mdurl==0.1.2
mypy-extensions==1.0.0
napalm==4.1.0
ncclient==0.6.13
netaddr==0.9.0
netmiko==4.1.2
netutils==1.6.0
nornir==3.4.1
nornir-jinja2==0.2.0
nornir-napalm==0.4.0
nornir-nautobot==3.2.0
nornir-netmiko==1.0.0
nornir-utils==0.2.0
ntc_templates==4.0.0
packaging==23.2
paramiko==3.3.1
parso==0.8.3
pexpect==4.8.0
pickleshare==0.7.5
prompt-toolkit==3.0.39
ptyprocess==0.7.0
pure-eval==0.2.2
pycparser==2.21
pyeapi==1.0.2
Pygments==2.16.1
PyNaCl==1.5.0
pynautobot==2.2.0
pyparsing==3.1.1
pyserial==3.5
python-dotenv==1.0.1
PyYAML==6.0.1
requests==2.31.0
rich==13.6.0
ruamel.yaml==0.17.35
ruamel.yaml.clib==0.2.8
scp==0.14.5
six==1.16.0
sniffio==1.3.1
stack-data==0.6.3
tenacity==8.2.3
textfsm==1.1.2
traitlets==5.12.0
transitions==0.9.0
ttp==0.9.5
ttp-templates==0.3.5
typing_extensions==4.8.0
urllib3==1.26.18
wcwidth==0.2.8
yamlordereddictloader==0.4.2
Some ios device have slightly diffrent outputs and cause this to not get caught and handled.
Example
^
% Invalid input detected at '^' marker.
although the API response includes the "platform" the variable task.host.platform is not set:
{ "count": 7, "next": null, "previous": null, "results": [ { ... "platform": { ... "name": "cisco_ios", "slug": "cisco_ios" }, ...
Add in auto publishing of new releases to Pypi
Currently the nornir-netmiko version is restricted to >=0.0.0 <0.1.0
while the latest version available is 1.0.0
. I can raise a PR to update the pyproject.toml and the lock file if there is an agreement.
Check compatibility with Nautobot v2
I am getting SSL: CERTIFICATE_VERIFY_FAILED, even though I am passing parameter
"ssl_verify": False
Code
`if name=="main":
nr = InitNornir(
inventory={
"plugin": "NautobotInventory",
"options": {
"nautobot_url": "https://10.81.161.100/",
"nautobot_token": "c6798e131be53ae38fd892fb6689144ca6d89c67",
"ssl_verify": False,
},
},
runner={
"plugin": "threaded",
"options": {'num_workers': 5}
},
logging={
"enabled": False
},
)
results=nr.run(task=helper_update)`
Output Error
`(.venv) tkdebnath@ubuntu:~/EIP$ python ip_helper.py
Traceback (most recent call last):
File "/home/tkdebnath/EIP/.venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 715, in urlopen
httplib_response = self._make_request(
File "/home/tkdebnath/EIP/.venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 404, in _make_request
self._validate_conn(conn)
File "/home/tkdebnath/EIP/.venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 1058, in validate_conn
conn.connect()
File "/home/tkdebnath/EIP/.venv/lib/python3.10/site-packages/urllib3/connection.py", line 419, in connect
self.sock = ssl_wrap_socket(
File "/home/tkdebnath/EIP/.venv/lib/python3.10/site-packages/urllib3/util/ssl.py", line 453, in ssl_wrap_socket
ssl_sock = ssl_wrap_socket_impl(sock, context, tls_in_tls)
File "/home/tkdebnath/EIP/.venv/lib/python3.10/site-packages/urllib3/util/ssl.py", line 495, in _ssl_wrap_socket_impl
return ssl_context.wrap_socket(sock)
File "/usr/lib/python3.10/ssl.py", line 513, in wrap_socket
return self.sslsocket_class._create(
File "/usr/lib/python3.10/ssl.py", line 1071, in _create
self.do_handshake()
File "/usr/lib/python3.10/ssl.py", line 1342, in do_handshake
self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:1007)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/tkdebnath/EIP/.venv/lib/python3.10/site-packages/requests/adapters.py", line 486, in send
resp = conn.urlopen(
File "/home/tkdebnath/EIP/.venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 799, in urlopen
retries = retries.increment(
File "/home/tkdebnath/EIP/.venv/lib/python3.10/site-packages/urllib3/util/retry.py", line 592, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='10.81.161.100', port=443): Max retries exceeded with url: /api/ (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:1007)')))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/tkdebnath/EIP/ip_helper.py", line 51, in
nr = InitNornir(
File "/home/tkdebnath/EIP/.venv/lib/python3.10/site-packages/nornir/init_nornir.py", line 72, in InitNornir
inventory=load_inventory(config),
File "/home/tkdebnath/EIP/.venv/lib/python3.10/site-packages/nornir/init_nornir.py", line 20, in load_inventory
inv = inventory_plugin(**config.inventory.options).load()
File "/home/tkdebnath/EIP/.venv/lib/python3.10/site-packages/nornir_nautobot/plugins/inventory/nautobot.py", line 148, in load
for device in self.devices:
File "/home/tkdebnath/EIP/.venv/lib/python3.10/site-packages/nornir_nautobot/plugins/inventory/nautobot.py", line 127, in devices
self._devices = self.pynautobot_obj.dcim.devices.all()
File "/home/tkdebnath/EIP/.venv/lib/python3.10/site-packages/nornir_nautobot/plugins/inventory/nautobot.py", line 110, in pynautobot_obj
self._pynautobot_obj = pynautobot.api(
File "/home/tkdebnath/EIP/.venv/lib/python3.10/site-packages/pynautobot/core/api.py", line 116, in init
self._validate_version()
File "/home/tkdebnath/EIP/.venv/lib/python3.10/site-packages/pynautobot/core/api.py", line 120, in _validate_version
api_version = self.version
File "/home/tkdebnath/EIP/.venv/lib/python3.10/site-packages/pynautobot/core/api.py", line 145, in version
).get_version()
File "/home/tkdebnath/EIP/.venv/lib/python3.10/site-packages/pynautobot/core/query.py", line 198, in get_version
req = self.http_session.get(
File "/home/tkdebnath/EIP/.venv/lib/python3.10/site-packages/requests/sessions.py", line 602, in get
return self.request("GET", url, **kwargs)
File "/home/tkdebnath/EIP/.venv/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
File "/home/tkdebnath/EIP/.venv/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
File "/home/tkdebnath/EIP/.venv/lib/python3.10/site-packages/requests/adapters.py", line 517, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='10.81.161.100', port=443): Max retries exceeded with url: /api/ (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:1007)')))
(.venv) tkdebnath@ubuntu:~/EIP$`
Modules installed
(.venv) tkdebnath@ubuntu:~/EIP$ pip freeze anyio==4.1.0 bcrypt==4.1.1 certifi==2023.11.17 cffi==1.16.0 charset-normalizer==3.3.2 colorama==0.4.6 cryptography==41.0.7 exceptiongroup==1.2.0 future==0.18.3 h11==0.14.0 httpcore==0.17.3 httpx==0.24.1 idna==3.6 Jinja2==3.1.2 junos-eznc==2.6.8 lxml==4.9.3 MarkupSafe==2.1.3 mypy-extensions==1.0.0 napalm==4.1.0 ncclient==0.6.13 netaddr==0.9.0 netmiko==4.3.0 netutils==1.6.0 nornir==3.4.1 nornir-jinja2==0.2.0 nornir-napalm==0.4.0 nornir-nautobot==3.0.0 nornir-netmiko==1.0.1 nornir-utils==0.2.0 ntc_templates==4.0.1 packaging==23.2 paramiko==3.3.1 pycparser==2.21 pyeapi==1.0.2 PyNaCl==1.5.0 pynautobot==2.0.2 pyparsing==3.1.1 pyserial==3.5 PyYAML==6.0.1 requests==2.31.0 ruamel.yaml==0.18.5 ruamel.yaml.clib==0.2.8 scp==0.14.5 six==1.16.0 sniffio==1.3.0 textfsm==1.1.3 transitions==0.9.0 ttp==0.9.5 ttp-templates==0.3.5 typing_extensions==4.9.0 urllib3==1.26.18 yamlordereddictloader==0.4.2
Not able to install package in edit mode due to non PEP-517 compliant build-system
config in pyproject.toml. Traceback when attempting this:
pip install -e plugins/ah-nornir-nautobot
Obtaining file:///Users/anthonyhouse/git_repos/nautobot-local-env/plugins/ah-nornir-nautobot
Installing build dependencies ... done
Checking if build backend supports build_editable ... done
ERROR: Project file:///Users/anthonyhouse/git_repos/nautobot-local-env/plugins/ah-nornir-nautobot has a 'pyproject.toml' and its build backend is missing the 'build_editable' hook. Since it does not have a 'setup.py' nor a 'setup.cfg', it cannot be installed in editable mode. Consider using a build backend that supports PEP 660.
Should be:
nornir_nautobot.plugins.tasks.dispatcher.cisco_ios_xr.NautobotNornirDriver
Please update dependencies for all packages but specifically nornir-jinja2
Please update httpx
to latest version.
pynautobot has the capability to disable threading. It is disabled by default. This should be exposed as an option and accounted for.
Support a new option passed in:
enable_threading
Seeking for a solution to:
a) store platform-dependent API endpoints (equivalent or extends RUN_COMMAND_MAPPING constant in default dispatcher).
b) provides user the ability to edit on a per platform basis (add, remove, override) default platform API endpoints.
this is linked to PR:: #79
PR solution (mikrotik routeros only):
ROUTEROS_API_ENDPOINTS = [
"/system/identity",
"/interface",
"/ip/address",
"/system/ntp/client",
"/ip/dns",
"/snmp/community",
"/system/logging/action",
]
RUN_COMMAND_MAPPING = {
"default": "show run",
"cisco_nxos": "show run",
"cisco_ios": "show run",
"cisco_xr": "show run",
"juniper_junos": "show configuration | display set",
"arista_eos": "show run",
"ruckus_fastiron": "show running-config",
"mikrotik_routeros_api": ROUTEROS_API_ENDPOINTS,
}
Potential solution might be to move this to a class attribute in the dispatcher itself (proposed by @itdependsnetworks )
Some versions of cisco_ios(xe) don't have ERROR:
in the output. This can cause the backup configuration to actually be saved as
^
% Invalid input detected at '^' marker.
The nautobot-ansible plugin allows the option of overwriting the inventory host name with a virtual chassis name:
https://nautobot-ansible.readthedocs.io/en/latest/plugins/inventory_inventory.html#parameter-virtual_chassis_name, all of the host vars are from the master member.
When it comes to limiting within ansible you're then able to limit of the virtual chassis name.
I'm looking to do the same thing but with nornir-nautobot.
Currently nornir-nautobot would return only devices and not virtual-chassis but I'm looking for the ability to pass via cli something like --hostname <virtualchassis>
then be able to use an F filter to filter on the 'name' attribute to filter the inventory to just those devices matching the hostname(s) passed in via cli.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.