voipgrid / opensips_exporter Goto Github PK

Version

opensips_exporter 1.1.0
opensips 2.4.5

File / Feature

Parsing any of the metrics

Expected behaviour

The metrics get parsed and output correctly.

Actual behaviour

The metrics aren't parsed and the exporter even thinks OpenSIPS is dead. (opensips_up 0)

Stacktrace / Error message

Error: unknown metric format encountered for:

Other info

n.a.

Aug 6 15:57:42 b2c opensips_exporter[23582]: 2019/08/06 15:57:42 Error encountered while reading statistics from opensips socket: write unixgram /var/run/ser-fg/opensips_exporter559675193/198.sock->/var/run/ser-fg/ser.sock: sendto: no such file or directory
Aug 6 15:57:42 b2c opensips_exporter[23582]: 2019/08/06 15:57:42 Problems registering the *processors.pkmemProcessor processor (could be due to no metrics found for this processor). Error: collector has no descriptors
Aug 6 15:57:42 b2c opensips_exporter[23582]: 2019/08/06 15:57:42 Problems registering the *processors.loadProcessor processor (could be due to no metrics found for this processor). Error: collector has no descriptors

Path:

[root@Devang opensips_exporter649748520]# cd ..
[root@Devang ser-fg]# ll
total 0
drwx------ 2 root root 40 Aug 6 16:01 opensips_exporter559675193
drwx------ 2 root root 40 Aug 6 15:06 opensips_exporter649748520
[root@Devang ser-fg]#

Version
exporter version: v1.1.1
OpenSIPS version: 1.10.5-tls

Issue:

I attempted to bump up the number of workers for opensips up to 128. This worked properly for OpenSIPS proper, but the OpenSIPS exporter stopped reporting new data.

Stacktrace / Error message
After making the change, the OpenSIPS Exporter started logging the following error in journalctl:

Oct 28 07:19:46 [hostname/redacted] opensips_exporter[36788]: 2020/10/28 07:19:46 Error encountered while reading statistics from opensips socket: read unixgram /var/run/opensips/opensips_exporter800628857
Oct 28 07:19:46 [hostname/redacted] opensips_exporter[36788]: 2020/10/28 07:19:46 Problems registering the *processors.loadProcessor processor (could be due to no metrics found for this processor). Error:
Oct 28 07:19:46 [hostname/redacted] opensips_exporter[36788]: 2020/10/28 07:19:46 Problems registering the *processors.pkmemProcessor processor (could be due to no metrics found for this processor). Error:

The following error also began appearing in /var/log/opensips.log in 30-second intervals:

2020-10-28T07:12:45.517638+00:00 [hostname/redacted] /usr/sbin/opensips[38056]: ERROR:mi_datagram:datagram_recur_write_tree: failed to write -line too long!!!
2020-10-28T07:12:45.517647+00:00 [hostname/redacted] /usr/sbin/opensips[38056]: ERROR:mi_datagram:mi_datagram_server: failed to build the response

Other info

As far as I can see I bumped into mi_datagram limit (The transported data is limited to the size of a Datagram (65K))
OpenSIPS/opensips#893

Can exporter work with mi_xmlrpc or mi_fifo instead of mi_datagram ?

https://opensips.org/Documentation/Interface-MI-2-4

Right now we only support the mi_datagram (Unix socket) protocol for retrieving metrics from openSIPS. There are other ways to get the metrics out of openSIPS, since 3.0 the official methods are:

• mi_fifo - FIFO file
• mi_http - JSON-RPC over HTTP
• mi_xmlrpc_ng - XML-RPC over HTTP

It would be great to support these protocols for people using them. Any help is welcome!

Version

exporter version: v1.0.1
OpenSIPS version: 2.4.2 LTS

File / Feature

Scraping any metric.

Expected behaviour

It returns metrics.

Actual behaviour

It does not return any OpenSIPS metric other than opensips_up & the standard Go/processor metrics.

Stacktrace / Error message

Nothing is logged

Other info

The opensips exporter is written against OpenSIPS 1.11, and with this new version it could be possible that the format for metric has changed. This has already caused issues in #2 and seems to do the same here.

Good afternoon, Ruben - apologies, I'm not entirely certain if this is a fault with my set up or in opensips_exporter, but I could sure use some help.

I'm running the latest opensips_exporter (as of a pull on 2019-12-03), with OpenSIPS v2.4.6.

I have the mi_datagram socket set up with:
modparam("mi_datagram", "socket_name", "/tmp/opensips.sock")

Which garners me a socket in /tmp/, ala:
srw-rw---- 1 opensips opensips 0 Dec 3 13:22 opensips.sock

When running the exporter I'm using the following command:
sudo ./opensips_exporter -socket /tmp/opensips.sock

Which I then attempt to pull using a curl:
curl localhost:9434/metrics

The output that opensips_exporter shows is:

Started OpenSIPS exporter, listening on :9434
Error encountered while reading statistics from opensips socket: read unixgram /tmp/opensips_exporter158593694/1.sock: i/o timeout
Problems registering the *processors.loadProcessor processor (could be due to no metrics found for this processor). Error: collector has no descriptors
Problems registering the *processors.pkmemProcessor processor (could be due to no metrics found for this processor). Error: collector has no descriptors

OpenSIPS is logging receipt of the request from opensips_exporter:
:get_statistics:#012core:#012shmem:#012net:#012uri:#012tm:#012sl:#012usrloc:#012dialog:#012registrar:#012pkmem:#012load:#012tmx:#012

Builds the tree and mi_datagram_write_node reports that it's outputting the statistics to the socket (which I can provide, but is mostly several hundred lines of it doing exactly what you'd expect it to), but then logs:
ERROR:mi_datagram:mi_datagram_server: failed to send the response: Permission denied (13)

The metrics themselves include a pile of go metrics, with the following pertinent to OpenSIPS:

# HELP opensips_up Whether the opensips exporter could read metrics from the Management Interface socket. (i.e. is OpenSIPS up)
# TYPE opensips_up gauge
opensips_up 0
# HELP opensips_usrloc_registered_users_total  Total number of AOR existing in the USRLOC memory cache for all domains.
# TYPE opensips_usrloc_registered_users_total counter
opensips_usrloc_registered_users_total 0
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 0

All of this says to me that opensips_exporter is not having problems talking to the UNIX domain socket, although I can't reconcile the "timeout" and "Permission denied" errors between opensips_exporter and OpenSIPS.

This all being said, using OpenSIPS' 'opensipsunix' tool to talk to the UNIX domain socket seems to work just fine:

printf ":get_statistics:\ndialog:" | sudo opensipsunix /tmp/opensips.sock
200 OK
dialog:active_dialogs:: 0
dialog:early_dialogs:: 0
(etc)

I'm thoroughly stumped. Please let me know what else I can provide, if anything, or just point out what kind of a cretin I've been and what I've missed.

Thank you for your time, and for the project in the first place!
(Edited for formatting to make it considerably easier to read)

Version

v1.1.1

File / Feature

Reading metrics from the OpenSIPS socket file.

Expected behaviour

It should tell me that I don't have the right permissions to read from the socket file and exit with a useful info message.

Actual behaviour

The exporter runs and spits our permission denied errors.

Stacktrace / Error message

Started OpenSIPS exporter, listening on :9434
Error encountered while reading statistics from opensips socket: read unixgram /tmp/opensips_exporter158593694/1.sock: i/o timeout
ERROR:mi_datagram:mi_datagram_server: failed to send the response: Permission denied (13)

Other info

Same error came up in #8, with the simple fix being "run it as the same user your OpenSIPS socket file is created with". It would make sense to have the exporter check the socket file permissions against the user it is running as. If they're not the same user, or don't have the proper permissions to read from the file, the exporter should exit and print a message telling them to run the exporter as the correct user.

Version

2.0.4_amd64.deb

File / Feature

opensips_exporter

Expected behaviour

opensips_exporter should connect to 43000 port where management interface is being served and should start exporting the metrics.

Actual behaviour

opensips_exporter crashes

Stacktrace / Error message

Executed the following command

opensips_exporter -protocol mi_http -http_address http://127.0.0.1:43000/opensips_mi/

2021/08/17 13:59:14 Started OpenSIPS exporter, listening on :9434
panic: inconsistent label cardinality

goroutine 71 [running]:
github.com/prometheus/client_golang/prometheus.MustNewConstMetric(...)
/home/runner/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/value.go:172
github.com/VoIPGRID/opensips_exporter/processors.usrlocProcessor.Collect(0xc000196330, 0xc0000b0ba0)
/home/runner/work/opensips_exporter/opensips_exporter/processors/usrloc_processor.go:47 +0x2e5
github.com/prometheus/client_golang/prometheus.(*Registry).Gather.func2(0xc0000ab0b4, 0xc0000b0ba0, 0x9b7680, 0xc0001aa080)
/home/runner/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/registry.go:433 +0x67
created by github.com/prometheus/client_golang/prometheus.(*Registry).Gather
/home/runner/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/registry.go:431 +0x325

Other info

The protocol for MI interaction has been shifted to JSON-RPC. Please add support to this exporter.

Version

2.0.4

File / Feature

opensips_exporter

Expected behaviour

{what should happen}

Actual behaviour

To many empty directories were created by exporter. Looks like one directory gets created every time metric url is called.

Stacktrace / Error message

drwx------. 2 opensips opensips 6 Jun 3 13:36 opensips_exporter325726256
drwx------. 2 opensips opensips 6 Jun 3 13:41 opensips_exporter350908291
drwx------. 2 opensips opensips 6 Jun 3 13:42 opensips_exporter808551783
drwx------. 2 opensips opensips 6 Jun 3 13:43 opensips_exporter898496922
drwx------. 2 opensips opensips 6 Jun 3 13:43 opensips_exporter393873803
drwx------. 2 opensips opensips 6 Jun 3 13:45 opensips_exporter877593218
drwx------. 2 opensips opensips 6 Jun 3 13:46 opensips_exporter326901880
drwx------. 2 opensips opensips 6 Jun 3 13:47 opensips_exporter367032513
drwx------. 2 opensips opensips 6 Jun 3 13:47 opensips_exporter323603611
drwx------. 2 opensips opensips 6 Jun 3 13:49 opensips_exporter329160018
drwx------. 2 opensips opensips 6 Jun 3 13:49 opensips_exporter373541283
drwx------. 2 opensips opensips 6 Jun 3 13:50 opensips_exporter388946381
drwx------. 2 opensips opensips 6 Jun 3 13:53 opensips_exporter384335268
drwx------. 2 opensips opensips 6 Jun 3 13:54 opensips_exporter837977206
drwx------. 2 opensips opensips 6 Jun 3 13:54 opensips_exporter817706333
drwx------. 2 opensips opensips 6 Jun 3 13:54 opensips_exporter331869464
drwx------. 2 opensips opensips 6 Jun 3 13:55 opensips_exporter393326540
drwx------. 2 opensips opensips 6 Jun 3 13:55 opensips_exporter828099259
drwx------. 2 opensips opensips 6 Jun 3 13:57 opensips_exporter836741874
drwx------. 2 opensips opensips 6 Jun 3 13:57 opensips_exporter348996035
drwx------. 2 opensips opensips 6 Jun 3 13:58 opensips_exporter340943942

Other info

running using cmd :
/opt/ipc/opensips/opensips_exporter -protocol mi_datagram -socket /opt/ipc/opensips/opensips.sock

Version

OpenSIPS Exporter Version: 2.0.4
OpenSIPS 2.4.8

File / Feature

This is the switch statement which looks at the type of connection and either connects via the mi_http interface, or the unix domain socket (a file on the filesystem), which would normally match the filename of the socket used in the socket_name setting of mi_datagram. However, the opensips configuration allows the socket to be defined as a network socket instead of a filename.

loadmodule "mi_datagram.so"
modparam("mi_datagram", "socket_name", "udp:127.0.0.1:8090")

Expected behaviour

The socket connection would be accepted and relevant metrics get provided when queried for.
This means that the client interprets the socket path and checks whether it is a UDP socket instead of a filename and connects and queries accordingly.

Actual behaviour

$ opensips_exporter -protocol mi_datagram -addr 127.0.0.1:9434 -socket udp:127.0.0.1:8090
2020/12/06 22:52:29 Started OpenSIPS exporter, listening on 127.0.0.1:9434
2020/12/06 22:52:31 Error encountered while reading statistics from opensips socket: write unixgram opensips_exporter422669750/1.sock->udp:127.0.0.1:8090: sendto: no such file or directory
2020/12/06 22:52:31 Problems registering the *processors.pkmemProcessor processor (could be due to no metrics found for this processor). Error: collector has no descriptors

Stacktrace / Error message

nil

Other info

nil

Version

exporter version: v1.0.1
OpenSIPS version: 2.4.2 LTS

File / Feature

Scraping without excluding processors (e.g. curl localhost:9434/metrics

Expected behaviour

Metrics should be returned

Actual behaviour

The opensips exporter crashes.

Stacktrace / Error message

No info in logs.

Other info

I am scraping a presence process. If I test each processor by including a ?collect[]=<module here>: at the end of the URL most processors work, except the load: processor. Upon hitting the load processor, it crashes the process.

Looking at the log files of OpenSIPS I can see that it returns the metrics. However the metric format has changed significantly between 1.11 (for which the exporter was written against) and 2.4.2. In all likeliness the exporter panics because it can not process the metrics.

Can you add information about profiles?

getting names and types of profile:

(opensips-cli): mi list_all_profiles 
{
    "Profiles": [
        {
            "name": "call",
            "has value": false
        },
        {
            "name": "incoming",
            "has value": true
        }
    ]
}

if profiles without values - publish only count:

(opensips-cli): mi profile_get_size call
{
    "Profile": {
        "name": "call",
        "value": null,
        "count": 100,
        "shared": "no",
        "replicated": "no"
    }
}

if profile with values - publush values with count:

(opensips-cli): mi profile_get_values incoming
[
    {
        "value": "5550001",
        "count": 1
    },
    {
        "value": "5550002",
        "count": 2
    }
]