informatiqal / automatiqal Goto Github PK

Allow defining an extra workflow if the task encounter an error. Couple of usecases:

• continue and exit
• clean-up/rollback

Ignore and continue

In some cases we might want to ignore the task error and continue with the execution. An example will be if we have a runbook that publishes an app to a stream. The app location can be controlled via the variables file but the stream name is static in the runbook. In this case the rubook will have a two tasks - create the stream and publish the app. But if the stream exists the create stream task will fail and the app will not be imported. In this scenario onError will be useful. And we should be able to specify in this section to ignore any errors during the create stream task. Pseudo code:

- name: Create stream
  operation: stream.create
  details:
    name: New stream
  onError:
    continue: true

If exit is set to true then the whole process is stopped

Clean-up/rollback

Automatiqal dont know how to rollback the applied changes (too many cases to cover and one solution cant fit all cases). For this reason there should be a way to define set of user defined tasks if error is encountered. onError block can be defined for each task. This provide enough granularity and control over the process.

Once all tasks inside the onError block are complete the runbook will automatically (pun here :) ) exit.

Pseudo code:

- name: Import app
  operation: app.import
  details:
    file: some/file.qvf
- name: Create stream
  operation: stream.create
  details:
    name: New stream
  onError:
    comment: Stream probably exists and imported app should be deleted
    tasks:
      - name: Remove imported app
        operation: app.remove
        source: Import app
- name: Publish imported app
   operation: app.publish
   source: Import app
   details:
     stream: New stream
     name: Published app

environment parameter to be optional and if not provided windows will be set as default

If exit parameter is set (inside onError block) then the whole process is terminated

With the latest version of qlik-repo-api reload task is changed and all trigger related methods are in sub property. For example: reloadTask.triggers.addSchema, reloadTask.triggers.addComposite, reloadTask.triggers.removeAll.

atm Automatiqal cant handle these operations

By default Automatiqal will perform all initial checks. Might be nice to allow specifying which check the developer wants the perform.

This option will be not because of speed (the initial checks are quite quick) but to allow "workaround" if the initial checks are too strict

When there is a task with binary data and special variables (for example app.upload) the function that replaces the special variables is changing the format of the binary data object.

Because of this all these tasks will fail. Usually with 500 error from Qlik

// w/o special variables
{
  file: Buffer(...),
  name: "..."
}

// with special variables
{
  file: { type: "Buffer", data: [...]},
  name: "..."
}

For the initial release, the following QSEoW operations should be enabled and tested:

About

• about.apiDefaults
• about.apiDescription
• about.apiRelations
• about.enums
• about.openApi
• about.get

App

• app.copy
• app.get cli exmple
• app.getAll
• app.export cli exmple
• app.exportMany cli exmple
• app.publish cli exmple
• app.remove cli exmple
• app.switch
• app.update
• app.upload cli exmple

Certificate

• certificate.generate cli exmple
• certificate.distributionPathGet cli exmple

Composite Trigger

• compositeTrigger.remove cli exmple

Content Library

• contentLibrary.get cli exmple
• contentLibrary.getAll cli exmple
• contentLibrary.importFile cli exmple
• contentLibrary.importFileMany cli exmple
• contentLibrary.removeFile cli exmple
• contentLibrary.removeFileMany cli exmple
• contentLibrary.export cli exmple
• contentLibrary.exportMany cli exmple
• contentLibrary.create cli exmple
• contentLibrary.remove cli exmple
• contentLibrary.update cli exmple

Custom Property

• customProperty.create
• customProperty.update
• customProperty.remove

Data Connection

• dataConnection.get cli exmple
• dataConnection.getAll cli exmple
• dataConnection.create cli exmple
• dataConnection.remove cli exmple
• dataConnection.update cli exmple

Extension

• extension.get cli exmple
• extension.getAll cli exmple
• extension.import cli exmple
• extension.remove cli exmple
• extension.update cli exmple

External Task

• extenrnalTask.get cli exmple
• extenrnalTask.getAll cli exmple
• extenrnalTask.create cli exmple
• extenrnalTask.addTriggerComposite
• extenrnalTask.addTriggerSchema cli exmple
• extenrnalTask.addTriggerMany cli exmple
• extenrnalTask.remove cli exmple
• extenrnalTask.update cli exmple
• extenrnalTask.remove cli exmple
• extenrnalTask.start
• extenrnalTask.startSynchronous

Node

• node.get cli exmple
• node.getAll cli exmple

Reload Task

• reloadTask.get cli exmple
• reloadTask.getAll cli exmple
• reloadTask.create cli exmple
• reloadTask.addTriggerComposite
• reloadTask.addTriggerSchema cli exmple
• reloadTask.addTriggerMany cli exmple
• reloadTask.start cli exmple
• reloadTask.startSynchronous cli exmple
• reloadTask.update cli exmple
• reloadTask.remove cli exmple

Stream

• stream.create
• stream.get
• stream.getAll
• stream.remove
• stream.update

System Rule

• systemRule.get cli exmple
• systemRule.getAll cli exmple
• systemRule.getAudit cli exmple
• systemRule.create cli exmple
• systemRule.licenseCreate cli exmple
• systemRule.remove cli exmple
• systemRule.update cli exmple

Tag

• tag.create
• tag.createMany
• tag.remove
• tag.update

Task

• task.get cli exmple
• task.getAll cli exmple
• task.start cli exmple
• task.startSynchronous cli exmple
• task.stop cli exmple

User

• user.get cli exmple
• user.getAll cli exmple
• user.create cli exmple
• user.remove cli exmple
• user.update cli exmple

Virtual Proxy

• virtualProxy.create cli exmple
• virtualProxy.get cli exmple
• virtualProxy.getAll cli exmple
• virtualProxy.remove cli exmple
• virtualProxy.update cli exmple

Make sure that "allowMultiple" option is respected for every operation

Explore the possibility to have "dry run" option.
This workflow will be ran right after the initial checks and it will not make any changes to Qlik!

The idea is to "examine" the runbook and run the possible tasks. "possible" tasks are:

• xxx.get
• xxx.getAll
• for all tasks, that will change Qlik, with known objects (update/remove tasks with "filter" parameter) execute the filter command and check if the update/remove tasks will actually be performed on any object or on multiple objects (ignore if allowZero and/or multiple are defined)
• all xxx.create tasks will be ignored

When task is complete and event is send sensitive data should be masked. Talking about passwords (data connections for example).

But first some analysis should be made to identify the possible operations that are affected + how to solve this - is it by masking the data or just "ignoring" it and not sowing these properties?

It seems that vitest is easier to use with TS

Explore the idea of having basic conditions for running the task.

Something like: for Task 2 if Task1 returns more than 0 entities only then run Task 2

This issue will be used to track the tests that have to be written.

TBA

For performance/optimization purposes might be better to read all files (all tasks with files property in details section) to be read right before the task is executed.

At the moment all these sections are replaced with the binary representation of the file.

Reading the files in advance do helps to identify any file related issues (file do not exists, file locked etc). This functionality should be kept IF its possible to read the files before the task execution.

Validate the runbook against the schema before executing anything

Fix and finish emitting debug info

When using inline variables and provide property name to extract it will not work when the value is nested in the object eg. $${Some task name#someProperty.anotherLevel.myValue}.

The function that is parsing the inline variables should respect that and should work with nested values

The special about these variables is that their values will not be defined bu the user and their values will be generated by Automatiqal itself. Such variables are:

• ${GUID} - generate random uuid. The uuid will exclude the - symbol. 89621de8f6944ce6befadf2862f220fd
• ${TODAY} - generate date for today. No ability to specify date format ... yet. The default will be YYYYMMDD
• ${NOW} - generate timestamp. No ability to specify format. Default format will be YYYYMMDDHHmmss
• ${INCREMENT} - starting from 1 each occurrence of the variable will be incremented with 1
• ${DECREMENT} - decrement from the current INCREMENT value
• ${RANDOM} - generate random alpha-numeric string. 20 characters of length. All letters are capital

Think that Qlik SaaS API package reached a point that is "compatible" with the Automatiqal and can be integrated.

This issue for the initial (analysis) phase and to outline what needs to be done (in general)

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Rate-Limited

These updates are currently rate-limited. Click on a checkbox below to force their creation now.

• Update dependency typedoc to v0.25.7
• Update dependency qlik-saas-api to ^0.18.0
• Update dependency rollup to v4
• Update dependency vitest to v1
• 🔐 Create all rate-limited PRs at once 🔐

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• Update dependency @rollup/plugin-commonjs to v25.0.7
• Update dependency @rollup/plugin-node-resolve to v15.2.3
• Update dependency @rollup/plugin-typescript to v11.1.6
• Update dependency @types/uuid to v9.0.7
• Update dependency ts-node to v10.9.2
• Update dependency vitest to v0.34.6
• Update dependency @rollup/plugin-json to v6.1.0
• Update dependency @types/node to v20.11.5
• Update dependency rollup to v3.29.4
• Update dependency typescript to v5.3.3
• Click on this checkbox to rebase all open PRs at once

Detected dependencies

• Check this box to trigger a request for Renovate to run again on this repository

Seems that Qlik will return sensitive data (secrets, passwords etc) if the user requesting the data have access over it ofc.

Shall we introduce an option that will hide such properties from the entity?

If the task is set with skip: true do not perform the initial checks

Add extra property to the returned task object that will indicate if the task as part of onError workflow. Its not much of use here but it can be shown in Automatiqal CLI output

Add aditional inital check that compares the return type of the task that is used as a source for another task. (intial checks are performed before the runbook is executed). For example:

• task is using operation app.getAll
• another task is using the result of previous task as a source but its operation is set to operate on stream object (for example stream.update

In the case above Automatiqal should throw an error before start the runbook. No point to execute the runbook if it is know from the beginning that it will faile when reach the stream.upadte task