informatiqal / automatiqal Goto Github PK
View Code? Open in Web Editor NEWQlik Sense automation deployment framework
Home Page: https://informatiqal.com/solutions/
License: MIT License
Qlik Sense automation deployment framework
Home Page: https://informatiqal.com/solutions/
License: MIT License
Allow defining an extra workflow if the task encounter an error. Couple of usecases:
In some cases we might want to ignore the task error and continue with the execution. An example will be if we have a runbook that publishes an app to a stream. The app location can be controlled via the variables file but the stream name is static in the runbook. In this case the rubook will have a two tasks - create the stream and publish the app. But if the stream exists the create stream task will fail and the app will not be imported. In this scenario onError
will be useful. And we should be able to specify in this section to ignore any errors during the create stream task. Pseudo code:
- name: Create stream
operation: stream.create
details:
name: New stream
onError:
continue: true
If exit
is set to true
then the whole process is stopped
Automatiqal
dont know how to rollback the applied changes (too many cases to cover and one solution cant fit all cases). For this reason there should be a way to define set of user defined tasks if error is encountered. onError
block can be defined for each task. This provide enough granularity and control over the process.
Once all tasks inside the onError
block are complete the runbook will automatically (pun here :) ) exit.
Pseudo code:
- name: Import app
operation: app.import
details:
file: some/file.qvf
- name: Create stream
operation: stream.create
details:
name: New stream
onError:
comment: Stream probably exists and imported app should be deleted
tasks:
- name: Remove imported app
operation: app.remove
source: Import app
- name: Publish imported app
operation: app.publish
source: Import app
details:
stream: New stream
name: Published app
environment
parameter to be optional and if not provided windows
will be set as default
If exit
parameter is set (inside onError block) then the whole process is terminated
With the latest version of qlik-repo-api
reload task is changed and all trigger related methods are in sub property. For example: reloadTask.triggers.addSchema
, reloadTask.triggers.addComposite
, reloadTask.triggers.removeAll
.
atm Automatiqal cant handle these operations
By default Automatiqal will perform all initial checks. Might be nice to allow specifying which check the developer wants the perform.
This option will be not because of speed (the initial checks are quite quick) but to allow "workaround" if the initial checks are too strict
When there is a task with binary data and special variables (for example app.upload
) the function that replaces the special variables is changing the format of the binary data object.
Because of this all these tasks will fail. Usually with 500 error from Qlik
// w/o special variables
{
file: Buffer(...),
name: "..."
}
// with special variables
{
file: { type: "Buffer", data: [...]},
name: "..."
}
For the initial release, the following QSEoW operations should be enabled and tested:
Make sure that "allowMultiple" option is respected for every operation
Explore the possibility to have "dry run" option.
This workflow will be ran right after the initial checks and it will not make any changes to Qlik!
The idea is to "examine" the runbook and run the possible tasks. "possible" tasks are:
allowZero
and/or multiple
are defined)xxx.create
tasks will be ignoredWhen task is complete and event is send sensitive data should be masked. Talking about passwords (data connections for example).
But first some analysis should be made to identify the possible operations that are affected + how to solve this - is it by masking the data or just "ignoring" it and not sowing these properties?
It seems that vitest is easier to use with TS
Explore the idea of having basic conditions for running the task.
Something like: for Task 2 if Task1 returns more than 0 entities only then run Task 2
This issue will be used to track the tests that have to be written.
TBA
For performance/optimization purposes might be better to read all files (all tasks with files
property in details section) to be read right before the task is executed.
At the moment all these sections are replaced with the binary representation of the file.
Reading the files in advance do helps to identify any file related issues (file do not exists, file locked etc). This functionality should be kept IF its possible to read the files before the task execution.
Validate the runbook against the schema before executing anything
Fix and finish emitting debug info
When using inline variables and provide property name to extract it will not work when the value is nested in the object eg. $${Some task name#someProperty.anotherLevel.myValue}
.
The function that is parsing the inline variables should respect that and should work with nested values
automatiqal/src/RunBook/RunBook.interfaces.ts
Lines 193 to 208 in ac8f1e4
The special about these variables is that their values will not be defined bu the user and their values will be generated by Automatiqal
itself. Such variables are:
${GUID}
- generate random uuid. The uuid will exclude the -
symbol. 89621de8f6944ce6befadf2862f220fd
${TODAY}
- generate date for today. No ability to specify date format ... yet. The default will be YYYYMMDD
${NOW}
- generate timestamp. No ability to specify format. Default format will be YYYYMMDDHHmmss
${INCREMENT}
- starting from 1 each occurrence of the variable will be incremented with 1${DECREMENT}
- decrement from the current INCREMENT
value${RANDOM}
- generate random alpha-numeric string. 20 characters of length. All letters are capitalThink that Qlik SaaS API package reached a point that is "compatible" with the Automatiqal and can be integrated.
This issue for the initial (analysis) phase and to outline what needs to be done (in general)
This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.
These updates are currently rate-limited. Click on a checkbox below to force their creation now.
These updates have all been created already. Click a checkbox below to force a retry/rebase of any.
package.json
@informatiqal/automatiqal-schema ^0.9.5
ajv ^8.12.0
ajv-errors ^3.0.0
events ^3.3.0
qlik-repo-api ^0.12.0
qlik-saas-api ^0.14.4
@rollup/plugin-commonjs ^25.0.4
@rollup/plugin-json ^6.0.0
@rollup/plugin-node-resolve ^15.2.1
@rollup/plugin-typescript ^11.1.3
@types/node 20.5.9
@types/uuid ^9.0.3
dotenv 16.3.1
esm ^3.2.25
nyc 15.1.0
rollup ^3.28.1
rollup-plugin-delete 2.0.0
ts-node 10.9.1
tslib 2.6.2
typedoc 0.25.1
typescript ^5.0.4
vitest ^0.34.3
node >=14.19.1
Seems that Qlik will return sensitive data (secrets, passwords etc) if the user requesting the data have access over it ofc.
Shall we introduce an option that will hide such properties from the entity?
If the task is set with skip: true
do not perform the initial checks
Add extra property to the returned task object that will indicate if the task as part of onError workflow. Its not much of use here but it can be shown in Automatiqal CLI output
Add aditional inital check that compares the return type of the task that is used as a source for another task. (intial checks are performed before the runbook is executed). For example:
app.getAll
source
but its operation is set to operate on stream
object (for example stream.update
In the case above Automatiqal
should throw an error before start the runbook. No point to execute the runbook if it is know from the beginning that it will faile when reach the stream.upadte
task
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.