Butler SenseOps Stats ("Butler-SOS") is a microservice publishing operational Qlik Sense metrics to InfluxDB, Prometheus and New Relic. Add Grafana for great looking dashboards and you get real-time monitoring of what happens inside a Qlik Sense environment.
Home Page: https://butler-sos.ptarmiganlabs.com
License: MIT License
Free to use, permissive open source license.
Adapt to your needs - please also consider sharing enhancements with the wider community.
Documentation is available at https://butler-sos.ptarmiganlabs.com.
Commercial getting-started services for Butler SOS are available, more info here.
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot]")
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "github-actions[bot]")
![renovate[bot] avatar](https://avatars.githubusercontent.com/in/2740?v=4 "renovate[bot]")
I get the below when trying to run. Have I just done something stupid or missed a step somewhere?
node --version is v0.10.30 running on CentOS.
/root/butler/butler/butler-sos.js:164
.then(err => {
^
SyntaxError: Unexpected token >
at Module._compile (module.js:439:25)
at Object.Module._extensions..js (module.js:474:10)
at Module.load (module.js:356:32)
at Function.Module._load (module.js:312:12)
at Function.Module.runMain (module.js:497:10)
at startup (node.js:119:16)
at node.js:906:3
Add config option for deciding how far back (seconds) log entries should be queried.
E.g. "5 minutes" etc. Use Postgres INTERVAL syntax.
Currently no proper handling of the case where Butler SOS cannot reach the user session API, resulting in errors like:
2019-09-10T19:21:55.582Z error: LOGDB: ERROR: Could not connect to Postgres log db: Error: connect ECONNREFUSED x.y.z.w:4432
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1056:14)
/Users/goran/code/butler-sos/src/lib/sessionmetrics.js:80
globals.logger.debug(USER SESSIONS: User session response from: ${response.request.href});
^
MIght be hard to fully verify the YAML config file, but at least put in place a structure where major startup-tests can be done, and add a few of the more important and/or easy to implement ones.
There can be cases where an existing (or new) InfluxDb cluster requires authentication before data will be accepted.
To handle this, a couple of config settings could be added to the YAML file, with corresponding logic in globals.js to correctly authenticate with Influxdb if credentials have been specified.
At least in verbose/debug logging it would be useful to log how long Butler SOS has been running.
Butler SOS can easily generate very large amounts of data, especially if data extract is done frequently (e.g. every 15 seconds) for large Sense environments.
This will result in a very large InfluxDB database.
Considering that Butler SOS is about operational metrics, it's usually not relevant to store more than a few weeks or months of data.
InfluxDB has a feature called "retention policies" that deal with exactly this scenario: they purge all data older than some threshold (e.g. older than 14 days or 6 weeks).
Retention policies can be manually applied to Butler SOS databases in Influxdb, that however requires using the command line influx client.
A more user friendly approach is to define one or more retention policies in the Butler SOS config file, and then automatically create those policies when re-creating the InfluxDB database.
The logic would look something like this:
Before deploying apps to production, it is important to test them. One often neglected aspect is to look in detail at how apps behave when loaded from disk into engine/RAM. How much memory does the app use when loaded when sheets are accessed in a typical end-user scenario etc?
This can be tested quite efficiently using Butler SOS.
When investigating misbehaving apps (e.g. apps consuming a lot of RAM) it would be helpful to have info-level log messages stored in Influxdb alongside the warnings and errors.
There can be a lot of info-level messages though, so this should ideally be enabled/disabled on a per-server basis.
For each server to be monitored, make it possible to specify zero or more tags that are written to Influxdb.
One usecase would be to tag certain servers as being development, some as QA and some as production servers.
Having servers tagged this way would make it easier to create Grafana variables that only include for example production servers.
Not sure if/how this can be achieved, but it would be very useful.
Group by apps loading in the web client vs apps reloading from the QMC.
Include who is reloading the app (for apps in web client) and who owns the app reloading in the QMC. Also include other relevant/interesting metadata about the apps (size etc).
FROM ….
LABEL maintainer="Göran Sander [email protected]"
LABEL description="lorem ipsum"
Let user control what log entries from log db that should be extracted and stored in Influxdb.
Use boolean properties in config file, e.g.
extractErrors: true
extractWarnings: true
extractInfo: false
When Butler-SOS.influxdbConfig.enableInfluxdb is set to false, no connection to an Influxdb database should be attempted.
Right now the code will not write any data to Influx when the config setting is set to false, but an initial conneciton attempt is still made.
Describe the bug
Not as much a bug as planning for the future.
Request is officially deprecated and should not be used any more.
Axios works great as a replacement - change Butler SOS to use it.
Log files should rotate daily.
Customisable log directory.
The docker-compose.yaml file shipped with this repo references mountaindude/butler-sos:5.0.0, but the correct image is ptarmiganlabs/butler-sos:5.0.0
Thx @TehesFR for reporting the issue!
Output heap usage to log file/console if debug mode logging is turned on.
The same InfluxDB database could be used, with tags used to separate data coming from each cluster. Grafana would then use data from InfluxDB just as for a single cluster setup.
variable e and err mismatch:
https://github.com/ptarmiganlabs/butler-sos/blob/master/butler-sos.js#L385-L388
Neither the docs nor the template production.yaml file states the fact that you need to specify port 4747 in the host entry in the config file.
Update docs and production_template.yaml.
Thx @TehesFR for reporting the issue!
Hi there,
First of all, this is a great tool, thank you for sharing !
Just for you to know, currently there is no 5.0.0 tag available on your docker image as stated in the doc, only tags 5.0.0RC1 and 5.0.0RC2.
Thanks
Best regards,
Is your feature request related to a problem? Please describe.
It is good practise to protected certificates exported from Qlik Sense with a password. Butler SOS does not support this today.
Describe the solution you'd like
Butler SOS should support - if so desired by the person setting it up - password protected certs. Password should be specified in config file.
Today there is a "host" field (for FQDN and port to connect to) and a "serverName" field (for a description of the server).
It would be useful to have a field containing just the host name, without the FQDN etc.
This would make it easier to use Grafana templates, for example.
Expose health check endpoints from container, to allow container supervisors to better monitor the state of the container.
Right now the term "host" is used in YAML config file for the FQDN of Sense server that is to be monitored. There is also a property "serverName" in the YAML file.
When data is written to InfluxDB the value in "serverName" is written to a Influx tag called "host".
Confusing? A bit.
Suggested change (this would be a breaking change, btw):
Three fields defined in the config file:
host: the FQDN or IP of the Sense server. Needs to be the same as the FQDN embedded in the cert exported from the Sense QMC. E.q. myserver.company.com
serverName: A free text name for the monitored server. Would usually be the first part of the FQDN. E.g. myserver
serverDescription: A description of the the server.
This would make it possible to create Grafana dashboards that use Grafana variables to automatically pick up any new/deleted servers, and adjust the dashboards accordingly. No more manual editing of charts when servers are added/removed.
For example:
Health metrics cannot be retrieved. Instead of being authenticated using certificates, a login page (that obviously don't contain any metrics!) is returned from Sense.
Solution: Make sure the servers.host property in config file is hostname:4747
It would be easier to read the logs if each message was prefixed with info about what part of the system wrote that particular log entry.
A simple prefix will suffice and provide an easy pointer into the source code, when further investigations are needed.
Include summary of relevant commits since last release.
Has anyone tried install the Mosquitto? I couldn't manage to get it install keeps saying cannot complete please reinstall.. Does butler have to run with a MQTT application? Any other choice?
When running node butler.node.js I receive this error:
TypeError: Cannot read property 'committed' of undefined (full output below)
I am running as root,
node.js 8.11.0, 8.11.0 and 8.11.2 tested, with the same error.
[root@qlikserver1 butler]# node butler-sos.js
2018-06-26T03:26:05.783Z - info: Starting Butler SOS
2018-06-26T03:26:05.785Z - info: Log level is: verbose
2018-06-26T03:26:05.828Z - info: Connected to Influx database.
2018-06-26T03:26:10.786Z - verbose: Event started: Statistics collection
2018-06-26T03:26:10.787Z - verbose: Getting stats for server: QlikServer2
2018-06-26T03:26:10.831Z - verbose: Received ok response from QlikServer2
/root/butler/butler/butler-sos.js:199
body.mem.committed.toString()
^
TypeError: Cannot read property 'committed' of undefined
at postHealthToMQTT (/root/butler/butler/butler-sos.js:199:14)
at Request._callback (/root/butler/butler/butler-sos.js:308:11)
at Request.self.callback (/root/butler/butler/node_modules/request/request.js:186:22)
at Request.emit (events.js:180:13)
at Request. (/root/butler/butler/node_modules/request/request.js:1163:10)
at Request.emit (events.js:180:13)
at IncomingMessage. (/root/butler/butler/node_modules/request/request.js:1085:12)
at Object.onceWrapper (events.js:272:13)
at IncomingMessage.emit (events.js:185:15)
at endReadableNT (_stream_readable.js:1106:12)
Describe the solution you'd like
It would be nice from a devops/operations perspective to be able to set config settings using environment variables. This would be particularly useful when running Butler SOS in a Docker container.
Right now the IDs of all loaded/active apps are extracted from Sense and stored in InfluxDB. That's fine, but it would also be useful to have the actual app names in InfluxDB (and Grafana of course).
How to monitor the monitor?
If Butler SOS sends a heartbeat signal (with configurable interval) to some other tool, that could be a way to ensure that Butler SOS (and other tools using the same mechanism) are alive and well.
Butler SOS stores data in InfluxDB, which supports per-database retention policies.
Document how these can be changed so that Butler SOS data is only kept for desired period of time.
Describe the bug
Extraction of session data is broken in 5.2.0.
Things such as what users have sessions open to which virtual proxy used to work fine, but are now broken.
To Reproduce
Same thing seen when running native Node.js and when running in Docker.
Expected behavior
Same behaviour as in previous version.
When the query to logdb returns data about a server which is (for some reason) not defined in the config YAML file, an error will occur:
butler-sos | 2019-04-04T15:44:35.700Z error: Log db query error: TypeError: Cannot read property 'hasOwnProperty' of undefined
butler-sos | at /nodeapp/butler-sos.js:450:32
butler-sos | at Array.forEach ()
butler-sos | at pgClient.query.then.res (/nodeapp/butler-sos.js:410:18)
butler-sos | at
butler-sos | at process._tickDomainCallback (internal/process/next_tick.js:229:7)
Solution: No data should be written to Influxdb for servers that are not defined in YAML config file.
Looks like the sample Grafana dashboard might not have any charts showing detailed user session data... Ooops if so. Add such charts to sample dashboards, or investigate why they don't show up (if they do exist in the sample Grafana dashboard file).
Thx @TehesFR for reporting the issue!
The Sense Proxy API provides information about all active user sessions, broken down per virtual proxy. Extract this info and 1) store it in InfluxDB, and 2) publish it as MQTT messages.
There are two erroneous lines in the mqttConfig part of config/default_template.yaml
is:
brokerIP:
brokerHost: 1883
should be:
brokerHost:
brokerPort: 1883
Drone is pretty slick... And more in line with the workflow used for this project.
Several inconsistencies. These have not triggered any errors during runtime, but are none the less a bit confusing.
In globals.js an Influx schema is defined and an Influx measurement called "log_event" is defined.
butler-sos.js on the other hand post data retrieved from Qlik Sense QLogs database to an Influx measurement called "log_entry".
Review InfluxDB schema and make sure it's consistently used across all code.
It would be nice to get some basic info about the underlying Qlik Sense environment written to the Butler logs, whenever Butler is started.
Things such as overall Sense version, QIX engine version, info about the hardware Sense is running on etc.
The various system accounts are likely to be of less interest when tracking user sessions.
Create a blacklist for user IDs that should be ignored.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.
