Monitoring - UNMS - Prometheus about toronto-community-network

[makew0rld]

• Modify the script to accept a name target, and always use unms.tomesh.net

[makew0rld]

The script now lives at network/scripts/unms_exporter/.

[makew0rld]

Here's a preliminary implementation. It handles HTTP requests on /metrics, with a target. By default it runs on port 8000. I have not tested it with the actual UNMS system yet. I will also be adding more keys once I'm aware of them.

Note: I will be updating the gist above, so that link will always point to the most recent public version. It's now here.

Edit: It works, with target=unms.tomesh.net.

Server command:

UNMS_KEY=foobar python3 unms_exporter.py

or

export UNMS_KEY=foobar
python3 unms_exporter.py

Client test with curl:

curl 'localhost:8000/metrics?target=unms.tomesh.net'

Edit: targets should now be names: ?target=sn1r1

[darkdrgn2k]

Script deployed and working

https://prometheus.tomesh.net/graph?g0.range_input=1h&g0.expr=node_network_receive_rate&g0.tab=0

Prometheus config

  - job_name: 'unms'
    scrape_timeout: 30s
    metrics_path: /metrics
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 127.0.0.1:8000  # The SNMP exporter's real hostname:port.
    static_configs:
      - targets:
        - sn1a1
        - sn1a2
        - sn1r1

[darkdrgn2k]

We can add some wireless entries
/api /devices/{deviceId}/interfaces

Some examples

wireless{name="sn1a1",if="wlan",mac="00:00:00:00:00", ssid="tomesh-airmaxac-sn1a2"} 1
wireless_frequency{name="sn1a1",mac="00:0:00:00:00"} 5490
wireless_frequency_width{name="sn1a1",mac="00:0:00:00:00"} 40
wireless_transmit_power{name="sn1a1",mac="00:0:00:00:00"} 0
wireless_stations{name="sn1a1",mac="00:0:00:00:00"} 0 (count of array)
wireless_gain{name="sn1a1",mac="00:0:00:00:00"} 16

should probably sync with openwrt node exporter script (might need to modify linked script too)

http://10.100.1.1:9100/metrics
http://100.64.20.2:9100/metrics

[darkdrgn2k]

created

for now
node_cpu_usage for cpu (seems close enough)
node_ram_usage for ram

I'll check the windows version of node exporter, it doesnt have load so it may be a clue there

frequency and signal added wireless_
node_boot_time_seconds is the new name for node_boot_time according to node exporter

added ubnt_ to ubiquti specific items
added more wireless info and STATION (connected device) information

#70

[makew0rld]

Fixed in #81, now a 502 BAD GATEWAY error is returned if the cpu or ram is None.

[darkdrgn2k]

UNMS installed at https://unms.tomesh.net
Prometheus installed at https://prometheus.tomesh.net
Grafana installed at https://monitor.tomesh.net

[darkdrgn2k]

Some preliminary work.

API source for UNMS https://unms.tomesh.net/nms/api-docs/#/ (login required)

Sample json data result of api call
https://cryptpad.fr/code/#/2/code/view/wCFKR8s5GusDmWzX5PnY4Ry+sOsi655vcB5QTh+JGCo/

Python3 script to generate node exporter like results for Prometheus
https://cryptpad.fr/code/#/2/code/edit/bqvpEutOdd9bog34vEbbCrDK/

Current script returns

ram{} 32
cpu{} 1
uptime{} 498250
node_network_receive_bytes_total{device"eth4"} 0
node_network_transmit_bytes_total{device"eth4"} 0
node_network_up{device"eth4"} 0

TODO:

• Make it respond to http on /metrics the same way node exporter does
• ability to pass an NAME and show that antenna's details
• expose more info, try to keep it aligned up with node-exporter format so we can re-use dashboards maybe?

[makew0rld]

Just some clarification:

• Where can I get a reference for the node-exporter format?

• ability to pass an NAME

  • As a URL query param maybe, like were you thinking /metrics?name=foo?

[darkdrgn2k]

ability to pass a name

snmp-walker configuration uses this query string

http://192.168.251.50:9116/snmp?target=1.2.3.4&module=if_mib

and the prometheus config as

 - job_name: 'snmp'
    scrape_timeout: 30s
    metrics_path: /snmp
    params:
      module: [ubiquiti_airmax]
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 127.0.0.1:9116  # The SNMP exporter's real hostname:port.
    static_configs:
      - targets:
        - 100.64.10.2  # SNMP device.
        - 100.64.11.2  # SNMP device.

So i guess to parallel this setup i would use target

/metrics?target=foo?

[makew0rld]

Working on this now, in a private file.

[darkdrgn2k]

nodeexporter dump

[makew0rld]

I've added the mtu, signal, and dropped keys.

[benhylau]

How can I access these, the links don't work:

Prometheus installed at https://prometheus.tomesh.net
Grafana installed at https://monitor.tomesh.net

[darkdrgn2k]

@benhylau Had a miss configuration after migrating the services to the mesh vlan on the servers

Should be working now.

[makew0rld]

Now that the script is deployed, all future updates by me and others should change the VERSION variable. The initial deployment used 0.1.0.

[darkdrgn2k]

Update names: (needs research)

uptime should be node_boot_time or node_boot_time_seconds - (ref m-lab/prometheus-support#91)
ram node_memory_?????
cpu
frequency
signal

[makew0rld]

I'm not sure if node_boot_time_seconds is equivalent, as I believe it indicates the Unix time of system boot, while the current uptime key is how long the system has been up in seconds.

• For ram, maybe it can become node_memory_Active_bytes?
• cpu does not seem to have any equivalent in the dump you provided
• frequency can be removed in favour of wireless_frequency
• I'm not quite sure what signal represents, so it's hard to say. Maybe it's one of the wireless_* ones?

I will add the wireless_* keys and make the above changes if you agree, let me know.

[makew0rld]

Can this be closed? I'd say it's original goals have been completed.

[darkdrgn2k]

We need to deal with "DOWN NODES"

Incorrect values - None

node_cpu_ram None
node_cpu_usage None
node_boot_time_seconds None

but also there should maybe be an error thrown as the node is down and the data is now stale

Sample: http://100.64.21.50:8000/metrics?target=100.64.20.3

https://prometheus.tomesh.net/targets