ptarmiganlabs / butler Goto Github PK

Is your feature request related to a problem? Please describe.
It's currently possible to start reload tasks via Butler's REST API
It would be useful to have a flag in the API through which partial/full reloads could be controlled.

Describe the solution you'd like
Add an optional flag in the /v4/reloadtask/{taskId}/start endpoint

If not included in the call the old behavior (full reload) should be started..

Heap and total memory used is already monitored as part of the existing uptime feature, but non-heap memory is not monitored.

Solution:

• Add it to the log output
• Store to InfluxDB

Describe the bug
There is a slight memory leak that occurred when Sense tasks were started using Butler's API endpoint for this purpose.
Nothing major but still something that should be addressed.
https://github.com/ptarmiganlabs/butler/blob/master/src/qrsUtil/senseStartTask.js

The issue occurs when the options object is mutated and then passed to the Sense APIs over https.
Also, this code does not use the config data already loaded into the config object, but rather loads the data again. Inefficient.

Suggested solution:
Refactor the code, possibly using QRS Interact module instead of https calls. Would give better CORS handling among other things.

:) sense scripts included point to http:// but should rather use lib paths or variables to be defined in Sense script

Describe the bug
Not as much a bug as a poor initial design design.

When Butler was first created, it was done to solve some immediate challenges, and there wasn't a lot of consideration put towards following established standards... "Solve the problem" was the motto.
As a result its API is as of v2 all GET based, even where PUT or POST should be used.

An example: the /v4/mqttpublishmessage can be used to send MQTT messages. It uses a GET method and all parameters as URL query parameters ( /v4/mqttpublishmessage?topic=...&message=...)
A more correct way is to place the payload in the body of the http request and use a PUT method.

Suggested change
Revise all endpoints and make them follow http standards to the degree possible.

Breaking change
This will for sure be a breaking change for many endpoints.
For that reason the API version number should also be raised to come into parity with Butler's own version number.
While Butler could be made to handle several API versions in parallel, that would significantly increase code/app size. Questionable if that's meaningful at this point.

Use same concept as in failed reload notifications, but for aborted reloads.

Notification emails should be customisable using template engine, with Slack/Teams/MQTT message can be simpler for now.

Assuming Butler can send notification emails (with script log extracts in them) when scheduled reloads fail in QSEoW, it would also be possible to save the full script logs to disk.

A new directory could be created per day, in this directory all reload scripts for failed reloads will be stored.
This would make post-mortem debugging easier, as it's otherwise quite time consuming to find a particular script log in among the many, many archived script logs in a large Sense cluster.

The sibling project Butler SOS has a good uptime monitor which tells how long the service has been running, how much memory it uses etc. It also stores this data to InfluxDB for later charting/analysis.

Add similar feature to Butler.

Update doc site wrt latest Butler version.

Hi,

We have used butler for a bit more then a year to sent slack messages from our qlik sense server. We installed it on our local qlik sense server.

We installed a new server and upgraded our qlik environment to april 2020 and I thought I will upgrade butler as well to the latest version. I followed the configuration steps and it is running when I start node butler.js from an elevated prompt.

But now I have the following issue.
I created a rest api connector in Qlik to use this to send messages to slack. But when I create the rest api connector with the following settings, copied from our previous server, it gives this error:
failed to connect to server. Please check connection parameters. see screenshots for the settings I have used

I don't have a lot of experience with this but the previous version worked great so hopefully somebody can help to give some guidance to fix this.

Kind Regards,

Hvdbunte

Describe environment:

• Windows Server 2019
• Containerisation no
• Version of Butler used: v 3.1.0
• Command used to start Butler: Node butler.js

Config file(s)
What's the content of the config file(s) you use?

Butler:
Logging configuration
logLevel: info # Log level. Possible log levels are silly, debug, verbose, info, warn, error
fileLogging: false # true/false to enable/disable logging to disk file
logDirectory: log # Subdirectory where log files are stored (no trailing / )

Heartbeats can be used to send "I'm alive" messages to any other tool, e.g. a infrastructure monitoring tool
heartbeat:
enabled: false
remoteURL: http://my.monitoring.server/some/path/
frequency: every 1 hour # https://bunkat.github.io/later/parsers.html

slackConfig:
enable: true
webhookURL: https://hooks.slack.com/services/my_webhook
loginNotificationChannel: team-bi-monitoring
taskFailureChannel: team-bi-monitoring

mqttConfig:
enable: false
brokerHost:
brokerPort: 1883
taskFailureTopic: qliksense/task_failure
taskFailureServerStatusTopic: qliksense/butler/task_failure_server
sessionStartTopic: qliksense/session/start
sessionStopTopic: qliksense/session/stop
connectionOpenTopic: qliksense/connection/open
connectionCloseTopic: qliksense/connection/close
sessionServerStatusTopic: qliksense/butler/session_server
activeUserCountTopic: qliksense/users/active/count
activeUsersTopic: qliksense/users/active/usernames

udpServerConfig:
enable: true
serverHost: qliksense-p
portSessionConnectionEvents: 9997
portTaskFailure: 9998

restServerConfig:
enable: true
serverHost: qliksense-p
serverPort: 8080

Enable/disable individual REST API endpoints. Set config item below to true to enable that endpoint.
restServerEndpointsEnable:
activeUserCount: true
activeUsers: true
slackPostMessage: true
createDir: true
createDirQVD: true
mqttPublishMessage: true
senseStartTask: true
senseAppDump: true
senseListApps: true
butlerping: true
base62ToBase16: true
base16ToBase62: true

Certificates to use when connecting to Sense. Get these from the Certificate Export in QMC.
cert:
clientCert: C:\ProgramData\Qlik\Sense\Repository\Exported Certificates.Local Certificates\client.pem
clientCertKey: C:\ProgramData\Qlik\Sense\Repository\Exported Certificates.Local Certificates\client_key.pem
clientCertCA: C:\ProgramData\Qlik\Sense\Repository\Exported Certificates.Local Certificates\root.pem
# If running Butler in a Docker container, the cert paths MUST be the following
# clientCert: /nodeapp/config/certificate/client.pem
# clientCertKey: /nodeapp/config/certificate/client_key.pem
# clientCertCA: /nodeapp/config/certificate/root.pem

configEngine:
engineVersion: 12.170.2 # Qlik Associative Engine version to use with Enigma.js. Ver 12.170.2 works with Feb 2019
host: qliksense-p
port: 4747
useSSL: true
headers:
X-Qlik-User: UserDirectory=Internal;UserId=sa_repository
rejectUnauthorized: false

configQRS:
authentication: certificates
host: qliksense-p
useSSL: true
port: 4242
headerKey: X-Qlik-User
headerValue: UserDirectory=Internal; UserId=sa_repository

configDirectories:
# enableDirectoryCreation: false
qvdPath:

gitHub:
host: api.github.com
pathPrefix: ''

Describe the bug
If heartbeats are enabled and sent to a https endpoint, the heartbeat call will fail with a "...unable to verify the first certificate" error.

Expected behavior
It should be possible to send heartbeats to either http or https endpoints.

Describe environment:
Bug verified in Node.js 14.5 on mac OS Big Sur, Docker and in Kubernetes.

Add second http server with REST endpoint for Docker healthcheck.

Should be possible to individually enable/disable features such as mqtt, Slack, udp server etc.

Is your feature request related to a problem? Please describe.
As of 4.2.1 there is decent error logging coverage, but it lacks somewhat in what details are reported about errors that occur.
A bit more context when errors occur => easier debugging and better bug reports.

Some entries are not very consistent, for example

Butler.udpServerConfig.serverIP
Butler.configEngine.server
Butler.configQRS.host
Butler.gitHub.host

Use consistent naming policy, e.g.

Butler.udpServerConfig.host
Butler.configEngine.host
Butler.configQRS.host
Butler.gitHub.host

Describe the bug
The API docs state that the endpoint is found at /v4/sensestarttask.

That used to be the case in Butler versions < 4.0, but as of v4.0 this endpoint moved to /v4/reloadtask/:taskId/start

It would be nice to get some basic info about the underlying Qlik Sense environment written to the Butler logs, whenever Butler is started.
Things such as overall Sense version, QIX engine version, info about the hardware Sense is running on etc.

File move and delete already exists, file copying should be there too...

Add support for turning Docker healthcheck on/off, as well as changing the port used.

Butler allows you to enable/disable individual API endpoints.
It would be useful if there was an API endpoint that would list which endpoints are enabled/disabled.

Given the limitations of QSEoW's built-in scheduler (ex not possible to limit jobs to certain times of the day), it would be useful to have a reliable external scheduler that knows how to talk to QSEoW.

The Butler already knows how to start Sense tasks, so if a scheduler is added it would be a nice complement to Sense's standard scheduler.

Is your feature request related to a problem? Please describe.
Featurerequest. Bigger Qlik Sense environments have separated admins and Developer, therefore most of the task failed notification mails can only be forwarded to the Developer.

Describe the solution you'd like
Butler 4.2 has an amazing flexibility for failed Task notification.What we miss: the right Person (App Owner=App Developer) needs this information, not only the Qlik Sense admin.

Describe alternatives you've considered
Hope for new Qlik Sense Upgrade with this feature...

Additional context
Qlik Sense with AD Connected have this information synced.
https://help.qlik.com/en-US/sense-developer/September2020/Subsystems/ProxyServiceAPI/Content/Sense_ProxyServiceAPI/ProxyServiceAPI-Session-Module-API-User-Get.htm

Following the instructions at https://mountaindude.github.io/butler/install-config/#installation, I installed the latest version of node.js, downloaded and extracted to c:\node\butler and ran the npm install which worked for a bit and then threw errors. My buffer was too small to capture the entire output, but I've incldued what I was able to grab below. Any suggestions?
...
| | | -- [email protected] | | -- [email protected]
| +-- [email protected]
| | +-- [email protected]
| | -- [email protected] | -- [email protected]
| +-- [email protected]
| -- [email protected] +-- [email protected] +-- [email protected] | -- [email protected]
+-- [email protected]
+-- [email protected]
| +-- [email protected]
| +-- [email protected]
| -- [email protected] | +-- [email protected] | | -- [email protected]
| +-- [email protected]
| +-- [email protected]
| +-- [email protected]
| -- [email protected] +-- [email protected] | -- [email protected]
+-- [email protected]
+-- [email protected]
+-- [email protected]
| +-- [email protected]
| +-- [email protected]
| +-- [email protected]
| +-- [email protected]
| +-- [email protected]
| -- [email protected] +-- [email protected] | +-- [email protected] | +-- [email protected] | +-- [email protected] | | -- [email protected]
| | -- [email protected] | | -- [email protected]
| -- [email protected] +-- [email protected] | +-- [email protected] | +-- [email protected] | +-- [email protected] | | +-- [email protected] | | -- [email protected]
| +-- [email protected]
| | -- [email protected] | +-- [email protected] | | -- [email protected]
| +-- [email protected]
| +-- [email protected]
| +-- [email protected]
| +-- [email protected]
| | +-- [email protected]
| | | +-- [email protected]
| | | -- [email protected] | | | -- [email protected]
| | +-- [email protected]
| | +-- [email protected]
| | -- [email protected] | -- [email protected]
| +-- [email protected]
| -- [email protected] +-- [email protected] | -- [email protected]
+-- [email protected]
| -- [email protected] +-- [email protected] | -- [email protected]
+-- [email protected]
| -- [email protected] +-- [email protected] +-- [email protected] | +-- [email protected] | +-- [email protected] | | -- [email protected]
| -- [email protected] +-- [email protected] | +-- [email protected] | | +-- [email protected] | | | -- [email protected]
| | -- [email protected] | +-- [email protected] | | -- [email protected]
| | -- [email protected] | -- [email protected]
| -- [email protected] +-- [email protected] | +-- [email protected] | +-- [email protected] | -- [email protected]
| -- [email protected] +-- [email protected] | +-- [email protected] | -- [email protected]
| -- [email protected] +-- [email protected] +-- [email protected] | -- [email protected]
| -- [email protected] +-- [email protected] +-- [email protected] | -- [email protected]
+-- [email protected]
| +-- [email protected]
| | +-- [email protected]
| | -- [email protected] | -- [email protected]
+-- [email protected]
+-- [email protected]
| +-- [email protected]
| +-- [email protected]
| | +-- [email protected]
| | +-- [email protected]
| | | +-- [email protected]
| | | -- [email protected] | | | +-- [email protected] | | | +-- [email protected] | | | -- [email protected]
| | +-- [email protected]
| | | -- [email protected] | | +-- [email protected] | | | +-- [email protected] | | | -- [email protected]
| | +-- [email protected]
| | +-- [email protected]
| | +-- [email protected]
| | | -- [email protected] | | +-- [email protected] | | | +-- [email protected] | | | | -- [email protected]
| | | +-- [email protected]
| | | | +-- [email protected]
| | | | -- [email protected] | | | +-- [email protected] | | | | -- [email protected]
| | | -- [email protected] | | -- [email protected]
| +-- [email protected]
| | +-- [email protected]
| | +-- [email protected]
| | | -- [email protected] | | -- [email protected]
| +-- [email protected]
| | +-- [email protected]
| | | -- [email protected] | | -- [email protected]
| | -- [email protected] | +-- [email protected] | +-- [email protected] | +-- [email protected] | | +-- [email protected] | | +-- [email protected] | | -- [email protected]
| +-- [email protected]
| | -- [email protected] | | +-- [email protected] | | -- [email protected]
| +-- [email protected]
| +-- [email protected]
| | -- [email protected] | | -- [email protected]
| +-- [email protected]
| | +-- [email protected]
| | +-- [email protected]
| | | -- [email protected] | | | -- [email protected]
| | +-- [email protected]
| | -- [email protected] | +-- [email protected] | | +-- [email protected] | | -- [email protected]
| +-- [email protected]
| +-- [email protected]
| | +-- [email protected]
| | +-- [email protected]
| | -- [email protected] | | -- [email protected]
| | -- [email protected] | +-- [email protected] | -- [email protected]
| +-- [email protected]
| +-- [email protected]
| | -- [email protected] | +-- [email protected] | +-- [email protected] | +-- [email protected] | | -- [email protected]
| +-- [email protected]
| | -- [email protected] | | -- [email protected]
| +-- [email protected]
| | +-- [email protected]
| | -- [email protected] | | -- [email protected]
| -- [email protected] | -- [email protected]
+-- [email protected]
| +-- [email protected]
| | +-- [email protected]
| | -- [email protected] | +-- [email protected] | -- [email protected]
+-- [email protected]
| +-- [email protected]
| | -- [email protected] | +-- [email protected] | | +-- [email protected] | | | -- [email protected]
| | -- [email protected] | +-- [email protected] | +-- [email protected] | +-- [email protected] | | -- [email protected]
| -- [email protected] +-- [email protected] | -- [email protected]
| -- [email protected] +-- [email protected] | -- [email protected]
+-- [email protected]
| +-- [email protected]
| | +-- [email protected]
| | | -- [email protected] | | +-- [email protected] | | | +-- [email protected] | | | -- [email protected]
| | -- [email protected] | +-- [email protected] | +-- [email protected] | +-- [email protected] | | -- [email protected]
| | -- [email protected] | +-- [email protected] | | +-- [email protected] | | -- [email protected]
| -- [email protected] +-- [email protected] | +-- [email protected] | | +-- [email protected] | | +-- [email protected] | | | -- [email protected]
| | +-- [email protected]
| | -- [email protected] | +-- [email protected] | | -- [email protected]
| | -- [email protected] | | +-- [email protected] | | | +-- [email protected] | | | -- [email protected]
| | +-- [email protected]
| | -- [email protected] | | +-- [email protected] | | -- [email protected]
| +-- [email protected]
| | -- [email protected] | +-- [email protected] | | -- [email protected]
| +-- [email protected]
| +-- [email protected]
| | +-- [email protected]
| | -- [email protected] | +-- [email protected] | -- [email protected]
| -- [email protected] -- [email protected]

**npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@^1.0.0 (node_modules\ch
okidar\node_modules\fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@
1.0.17: wanted {"os":"darwin","arch":"any"} (current: {"os":"win32","arch":"x64"
})
npm ERR! Windows_NT 6.2.9200
npm ERR! argv "C:\Program Files\nodejs\node.exe" "C:\Program Files\nodejs\
node_modules\npm\bin\npm-cli.js" "install"
npm ERR! node v6.9.4
npm ERR! npm v3.10.10
npm ERR! code ELIFECYCLE

npm ERR! [email protected] install: node-gyp rebuild
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] install script 'node-gyp rebuild'.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the ref package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR! node-gyp rebuild
npm ERR! You can get information on how to open an issue for this project with:
npm ERR! npm bugs ref
npm ERR! Or if that isn't available, you can get their info via:
npm ERR! npm owner ls ref
npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request:
npm ERR! C:\Node\butler\src\npm-debug.log**

C:\Node\butler\src>npm install

When running Qlik Sense Enterprise on Windows in "standard mode", you have no direct access to the file system from the app scripts. But sometimes you have a real, legitimate need to delete or move for example temporary QVD files that are no longer needed.

Suggestion:
Specify in Butler's main config file in which directories it should be possible to delete files, and between what directories file moves should be allowed. Then check that the file delete/move request coming in via the API really refers to a file in one of those directories.
That way it should be possible to avoid nasty things such as using relative paths, and that way deleting important system files etc.

Email notifications (coming in v4.2) for failed scheduled reloads use a sophisticated templating engine that allows a high level of customisation.

Use the same concept also for Slack notifications.

Hi,
I have a couple of installations made recently that have the same error:

C:\Program Files\nodejs\butler-master\src>node butler.js
fs.js:675
return binding.read(fd, buffer, offset, length, position);
^

Error: EISDIR: illegal operation on a directory, read
at Object.fs.readSync (fs.js:675:18)
at tryReadSync (fs.js:540:20)
at Object.fs.readFileSync (fs.js:583:19)
at Object. (C:\Program Files\nodejs\butler-master\src\qrsUtil\sen
seStartTask.js:14:13)
at Module._compile (module.js:653:30)
at Object.Module._extensions..js (module.js:664:10)
at Module.load (module.js:566:32)
at tryModuleLoad (module.js:506:12)
at Function.Module._load (module.js:498:3)
at Module.require (module.js:597:17)

Can't really understand what's happening.
Thanks

https://www.repostatus.org/

By hooking into the engine logs it's possible to detect any failed reload in the Sense engine.

While it's desirable to detect app reloads also outside of the scheduler context, some form of white listing is probably needed.
Otherwise every failed reload during development will trigger an alert.
Maybe it's possible to while list which user IDs alerts should be sent for... Internal system accounts that are used to run reloads would be included, leaving end user triggered reloads out.

Should be possible to keep real-time info on what virtual proxy users are connected via.

This will be useful from an operations perspective, for example when a server restart is needed.
This feature would answer the question how many users that are connected through this particular server.

Docker containers are nice, but Docker containers with built-in health checks are even better..
Use the standard health check feature of the Dockerfile.

Seems the healthcheck in v3.0.0 is not working as it should.
Unclear why at this point.

Describe the bug
In current version nothing is logged by Butler when Sens returns a non-204 result code.
Only way to see that an error (e.g. 404 returned from Sense) occurred is to turn on verbose logging. While this works, this kind of scenario should result in a clear feedback to the user calling the Butler.

Expected behavior
Butler should log an error to its own logging system, as well as return an error via the REST API call response

The instructions at https://ptarmiganlabs.github.io/butler/install-config/#install_on_docker refer to verisure_mqtt as the name of the Docker container, that is a copy-paste error..
Should be "butler".

Improve CI/CD process to be more automated, including multi architecture support.

Email notifications (coming in v4.2) for failed scheduled reloads use a sophisticated templating engine that allows a high level of customisation.

Use the same concept also for Teams notifications.

A heart beat every 10 sec would enable active monitoring of whether Butler is up or down.

Is your feature request related to a problem? Please describe.
Due to incremental development during past years, the naming of API endpoints is not very consistent.
The APIs still work, but it would of course be better if related API endpoints had similar names...

Describe the solution you'd like
The new app reload endpoint is found at /v4/app/{appId}/reload.

The older list-all-apps and app-dump endpoints are found at /v4/senselistapps and /v4/senseappdump/{appId}.
Keep those two endpoints for backward compatibility, but also add:

/v4/apps/list
/v4/app/{appId}/dump

FROM ….
LABEL maintainer="Göran Sander [email protected]"
LABEL description="lorem ipsum"

By having a key-value store in Butler and exposing set/get methods via a REST API, other systems could set values, which could then be accessed by Qlik apps' load scripts.

The same thing can be achieved using external key-value stores (Redis etc), having it in Butler would merely be a convenience.

A related enhancement would be to add some parameters to the start task endpoint, where key-value pairs could be passed as parameters to that endpoint.
The reloading app could then retrieve the key-value pairs during app reload.
A kind of parameter passing to app reloads, thus.

Log for example headers, requesting IP address etc when calls are made to the REST API endpoints.

Sending such notifications using Slack or MS Teams is great, but for companies with email driven workflows the IM channels are a complement, not the main notification channel.

Enhance the current, basic setup for email notifications (which use log4net log appenders) with features such as

• Templating for email subject and body. This makes it possible to dynamically inject info from the failed reload into the email message
• Rate limiting. Don't spam users with email notifications.
• Include script logs in notification emails.
• Flexibility wrt configuration of smtp/email servers.
• Set email priority flag
• Multiple recipients
• Links to QMC/Hub from within the notification email

Upgrading Sense Enterprise to Feb 2020 or beyond means that the Sense certificates will be recreated. The new certs must be exported and made available to Butler.
Failing to do so will cause Butler to stop working.

Update docs to highlight the above.

Would be neat if there was a real-time charting tool that would take MQTT messages as input, and plot (nice looking) charts..

Node-RED does this, but the line charts do not currently handle charts covering more than a few thousand data points (at least not in a good way).
Mozaik is looking promising... there might be others too. #

Would offer user a more structured way of examining the REST API of Butler.

Slack is nice, but lots of companies use Teams. Butler should support both.

While log appenders will forward reload failure events to Butler just fine, they are somewhat tricky to set up.
It would be a smoother user experience to have Butler use the Sense notification API and that way simply subscribe to the relevant events.

This is what Butler Notifier does, so the scaffolding is already there.

Hi,

Thanks for this add-on for the Slack integration.

I am new to this, so maybe it is an issue but maybe my code is not correct. But when I try to put in a new line in the message by using \n. The \n is just shown as text.

I tried the send.slack command from the node.slack on it's own and then it works.

Can you let me know if it is possible to add a new line in the Slack message or what I can do to fix this?

Thanks

Hey this is really impressive, thanks for sharing this. I have some functionality that is used for deploying Qlik Sense extensions through the API to Qlik Sense Enterprise via push to GIT repository (where multiple branches of the same visualisation will then exist in Qlik Sense for development and testing. Would you mind if I try adding this into butler also?

Describe the bug
When running in a multi-arch (amd64 and arm64) k8s cluster, containers fail to start on arm64 nodes.

Expected behavior
Containers should automatically start on amd64, arm64 or arm nodes.

Solution idea
The issue might be caused by the variant field in the manifest. It seems that the manifests of other (working) multi-arch images Docker images don't have that variant field. Might be worth removing it as a first attempt at resolving the issue.

Describe the bug
Not possible to start Sense tasks using the REST API in v4.2.1

Screenshots
Error message is

error: STARTTASK: Failed starting task: fbf645f0-0c92-40a4-af9a-6e3eb1d3c35c,

This is a confirmed bug that will be fixed in next release.