flxpeters / netbox-prometheus-sd

Service discovery for VMs, IPs and devices in Netbox

Netbox Prometheus SD

File based service discovery script for Prometheus. Make virtual machines, IPs and devices managed from Netbox available as Prometheus targets with labels.

Deprecation This project got obsolet since Prometheus supports HTTP based service discovery. Use https://github.com/FlxPeters/netbox-plugin-prometheus-sd as Netbox plugin to directly connect Prometheus with Netbox. I will still maintain this project based on the dependabot updates, but there will be no further feature development.

Requirement

• Python >= 3.7
• Pynetbox.

Config

The app is configured with env variables.

    NETBOX_SD_URL: https://netboxdemo.com
    NETBOX_SD_TOKEN:  72830d67beff4ae178b94d8f781842408df8069d
    NETBOX_FILTER: "{\"status\":\"active\",\"site\":\"ds9\"}"
    NETBOX_SD_FILE_PATH: /data/netbox/netbox.json
    NETBOX_SD_LOG_LEVEL: "DEBUG"
    NETBOX_SD_VERIFY_SSL: "FALSE"
    NETBOX_THREADING: "True"
    NETBOX_SD_LOOP_DELAY: "60"
    NETBOX_SD_METRICS_PORT: "8000"
    NETBOX_OBJECTS: "vm device" # space separated list of netbox objects to discover. Currently supported: vm, device and ip_address

Filters are applied as JSON which is mapped to Netbox filter criterias.
See the Netbox for more Details: https://netbox.readthedocs.io/en/stable/rest-api/filtering/

Usage

python3 netbox-prometheus-sd.py

The service discovery script requires the URL to the Netbox instance, an API token that can be generated into the user profile page of Netbox and a path to an output file.

In the Prometheus configuration, declare a new scrape job using the file_sd_configs service discovery:

- job_name: 'netbox'
  file_sd_configs:
  - files:
    - '/path/to/my/output.json'

Example

See example directory for an example on how this service discovery. Currently we use netboxdemo.com for this. This should be changed to a local Docker based Netbox stack.

Thanks & Credits

This project is based on a Script from ENIX SAS:

• https://github.com/enix/netbox-prometheus-sd
• https://enix.io/fr/blog/service-discovery-avec-netbox-et-prometheus/

netbox-prometheus-sd's Issues

Fix broken test coverage in Github actions

When running tests in Github action all coverage reports are generated.
But Sonarqube cannot pick up the results with: "Cannot resolve the file path"

Add a Docker build and publish image

In order to make this project more accessible we should build a Dockerimage and publish it to Dockerhub

Problems getting plugin to work

Hey,

I've just started looking into a problem running the plugin against netbox 3.0.10 (i think) and on making a query to the plugin we get

application.log:Fri Feb 18 01:09:26 2022 - SIGPIPE: writing to a closed pipe/socket/fd (probably the client disconnected) on request /api/plugins/prometheus-sd/devices/ (ip 127.0.0.1) !!!
application.log:Fri Feb 18 01:09:26 2022 - uwsgi_response_write_body_do(): Broken pipe [core/writer.c line 341] during GET /api/plugins/prometheus-sd/devices/ (127.0.0.1)

requests.log:[pid: 1748977|app: 0|req: 20/32] 127.0.0.1 () {54 vars in 1210 bytes} [Fri Feb 18 01:07:57 2022] GET /api/plugins/prometheus-sd/devices/ => generated 523507 bytes in 89009 msecs (HTTP/1.0 200) 9 headers in 781 bytes (0 switches on core 0)

This is happening for all api queries.

I'm a total newcomer to netbox and have just started looking at this but wondered if this is a common issue.

Unable to use IPv6-only hosts

When a device only has IPv6 configured (as primary address) it gets discarded with the following debug-log:

DEBUG:Drop vm 'vela.ortg.de' due to missing primary IPv4

Is there any reason why you are using primary_ip4 instead of primary_ip in netbox_sd/inventory.py?
I propose a fix in PR #32. This should also work for any Netbox device/vm which only has an IPv4 address configured.

Add meta labels with custom fields

A great improvement could be to add all the custom fields as meta labels.

Exemple :

__meta_netbox_cf_myfield1 = "xxxx"
__meta_netbox_cf_myfield2 = "xxxx"

If the underlying field is an object, exposing the name and slug is needed
Exemple with a custom field of type "Location" :

__meta_netbox_cf_myfieldlocation_name = "xxxx"
__meta_netbox_cf_myfieldlocation_slug = "xxxx"

Question: Multiple IPs on one device?

I have some servers with 2 IP addresses, one primary and one for out-of-band management. I'd like to scrape both IPs with Prometheus using 2 different jobs. For example, blackbox exporter on the primary, and SNMP on the out-of-band IPMI. Do you have any thoughts on how to accomplish this?

So far I've been scraping an endpoint like this:

http://netbox/api/plugins/prometheus-sd/devices?status=active&cf_prometheus_job=netbox_blackbox_ping

I could use the ip-addresses endpoint instead for the second one, but then I would lose important metadata. I'm most interested in site and location labels.

Not compatible with Netbox v4.0.2

Hey :)

I noticed that the Plugin is not compatible yet with the new Netbox versions (probably > 4.0.0)

Log files from startup of Netbox:

Traceback (most recent call last):
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/core/management/__init__.py", line 255, in fetch_command
    app_name = commands[subcommand]
               ~~~~~~~~^^^^^^^^^^^^
KeyError: 'rqworker'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/opt/netbox/netbox/manage.py", line 10, in <module>
🧬 loaded config '/etc/netbox/config/configuration.py'
🧬 loaded config '/etc/netbox/config/extra.py'
🧬 loaded config '/etc/netbox/config/logging.py'
🧬 loaded config '/etc/netbox/config/plugins.py'
    execute_from_command_line(sys.argv)
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/core/management/__init__.py", line 262, in fetch_command
    settings.INSTALLED_APPS
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/conf/__init__.py", line 89, in __getattr__
    self._setup(name)
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/conf/__init__.py", line 76, in _setup
    self._wrapped = Settings(settings_module)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/conf/__init__.py", line 190, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/opt/netbox/netbox/netbox/settings.py", line 750, in <module>
    raise e
  File "/opt/netbox/netbox/netbox/settings.py", line 743, in <module>
    plugin = importlib.import_module(plugin_name)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/opt/netbox/venv/lib/python3.11/site-packages/netbox_prometheus_sd/__init__.py", line 1, in <module>
    from extras.plugins import PluginConfig
ModuleNotFoundError: No module named 'extras.plugins'

Sadly I don't have time to investigate further / fix this myself and make a PR at the current time, but wanted to create an Issue so it's on the radar. I assume that it's a relatively easy fix, probably just a few API endpoints changed. Thank you for this project and for your hard work :)