ovh / utask Goto Github PK

Finding one particular task on a long list is difficult. Some kind of search/filter primitive at the API level would be useful not to burden the graphical client.

As execution payloads are encrypted, some tag/label system could be devised to add custom metadata on top of current generated metadata (creation date, requester, state etc...)

Enrich the http action plugin with the following options:

• timeout of the request
• auth enable basic auth
• deny redirects
• http parameters

Implement a plugin "script" that forks and executes a script within the utask container.

Suggestion:

    type: script
    configuration:
        file: check-invoice.py
        argv:
            - {{.step.getInvoice.output.invoiceID}}
            - ...
        stdin: foobar
        timeout: 5s

This should execute for the step the script check-invoice.py, which needs to be present in a configured script folder within the utask container.
The script should be executable (+x), and use any interpreter as long as it is present in the utask container. The base utask image should maybe provide the most common interpreters?

There should be a way to pass data to the following steps in a structured way, it may adopt a similar strategy as the ssh plugin: the last line of the script's stdout will be json parsed, and if it succeeds will be used as the result payload for the step.

Is your feature request related to a problem? Please describe.
For example:

It's common that some remote services are designed in an async communication pattern:

• Webhook workflow:
  • Send HTTP
  • Wait for HTTP callback (usually listen on POST request from remote servers)
  • continue

Describe the solution you'd like
For example, define a generic callback endpoint that's capable of finding the task by deriving public id from the request. Upon getting the webhook, look for the task/resolution, pause it and modify the state or the step or the input of the task and reevaluate the task.

Describe alternatives you've considered
Maybe some workaround is to build something outside utask, e.g. a simple webserver that talks to utask APIs.

Additional context

Is your feature request related to a problem? Please describe.
We use uTask in our deployment software. We store some technical informations in step results but watchers are not allowed to see data.

Describe the solution you'd like
For a specific template (or maybe specific step), I would like a flag which allows watcher to see output.

Describe alternatives you've considered
The only alternative is to set all watchers as resolver...

Describe the bug
Failed to assert type of plugin 'myinit.so': expected InitializerPlugin got *plugins.InitializerPlugin

To Reproduce
Steps to reproduce the behavior:

wget https://github.com/ovh/utask/releases/latest/download/install-utask.sh
sh install-utask.sh
mkdir init/myplugin

// init/myplugin/main.go

package main                                                                  
                                                                              
import "github.com/ovh/utask/pkg/plugins"                                     
                                                                              
var Plugin = NewMyPlugin()                                                    
                                                                              
type MyPlugin struct{}                                                        
                                                                              
func NewMyPlugin() plugins.InitializerPlugin            { return &MyPlugin{} }
func (p *MyPlugin) Init(service *plugins.Service) error { return nil }        
func (p *MyPlugin) Description() string                 { return "myplugin" } 

docker-compose build
docker-compose up

Starting utask-test_db_1 ... done
Recreating utask-test_utask_1 ... done
Attaching to utask-test_db_1, utask-test_utask_1
db_1     | LOG:  database system was shut down at 2020-05-23 15:57:11 UTC
db_1     | LOG:  MultiXact member wraparound protections are now enabled
utask_1  | wait-for-it.sh: waiting 15 seconds for db:5432
utask_1  | wait-for-it.sh: db:5432 is available after 0 seconds
utask_1  | time="2020-05-23T15:59:18Z" level=fatal msg="failed to assert type of plugin 'myinit.so': expected InitializerPlugin got *plugins.InitializerPlugin"
db_1     | LOG:  database system is ready to accept connections
db_1     | LOG:  autovacuum launcher started
db_1     | LOG:  incomplete startup packet
utask-test_utask_1 exited with code 1

Expected behavior
It should pass the interface type assertion.

utask version impacted by the bug
v1.6.0

Context which you are currently running utask
Fresh installation from install-utask.sh

Additional context
Got the same error on Mac and Ubuntu.

Is your feature request related to a problem? Please describe.
When uTask invokes an other template as a subtask, the step is not very relevant. Its state is SERVER_ERROR while the template is running and if watchers list is not defined, it is a sort of black-box for them.

Describe the solution you'd like
We could be execute a subtask with two modes:

1. Mode 1: With the current behavior, but with a better status (#181) and a easy way to view subtask details (a link or something like that)

2. Mode 2: Subtask steps could be included in the main template (recursive behavior) and with this mode, subtask mecanism is hidden for the users.

Describe alternatives you've considered
In all cases, an option which define watchers list with parent watchers list could be fine, particullary when we have subtasks of subtasks.

Hi,

Copuld you please provide an example for the following flow:

1. Execute request that starts some procees --> this one i know how to do
2. Loop and check above process until it is done.

Pseudo code:
execute_process
while [ true ]
do
check_if_process_is_done
if [ done ] {
return success
} else {
sleep 20 seconds
check_if_process_is_done
}

https://github.com/ovh/utask/blame/master/.travis.yml#L35

The way we get goreleaser seems very flaky, this should be fixed

As a µTask user debugging is hard on two counts:

• validity check on templates doesn't give enough context (could be fixed by #5 run within docker build)
• error messages on blocked tasks lack aren't clear enough

Describe the bug
When a subtask is spawned, a _utask_parent_task_id tag is added on the subtask.
This tag is read-only, because we don't want users to change it, as it could also to re-run arbitrary task.
This security prevents any modification of the task, meaning we can't change the inputs or everything else.

Expected behavior
Allow edition of the task, but make sure we can't change _utask_parent_task_id.

utask version impacted by the bug
v1.8.2+
Introduced by 70cbec9

Describe the solution you'd like
Allow timeout configuration on ssh builtin plugin to control the maximum time that can be spent waiting for the end of the execution of the ssh command.

Currently we need to use some commands to release µTask.
We want to automate the release process directly through the CI.

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

I'd like to be able to configure periodic execs for a given template + set of input.

Describe the solution you'd like
A clear and concise description of what you want to happen.

Periodic execs definition:

• every X minute/hour/day
• or everyday at 3pm
• or every 3rd of each month at 2pm
• or ...
  with template: create-new-thing-XXX
  with inputs:
  foo: yyy
  baz: xxxx
  bar: zzzz

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Implement a new type of action for producing messages over a kafka topic, from within a task's step

Is your feature request related to a problem? Please describe.
When using an API that do cursor pagination, we can't use the http action to range over multiple cursors and save all datas

Describe the solution you'd like
We should have a special package to range over all pages, or some settings in the current http package to configure the pagination behaviour

Describe the solution you'd like
field templating function only looks inside the templating global context when called.
In case we want to use it in a pipe context, we should take as the first parameter the context on which to apply.

Example: (context)

"step":
  "first":
    "output": 
       "result": 
           "my-payload": "{\"common-name\":\"utask.example.org\",\"id\":32}"

To retrieve the common_name key, we could use templating like this:

{{ field `step` `first` `output` `result` `my-payload` | fromJson | fieldFrom `common-name` }}

The fieldFrom operator would take the context from the first argument, instead of the global task/resolution context; hence enabling the pipe operator to work as expected.

Describe the solution you'd like
To integrate easily different notification system, we could add a "webhook" notify package, where utask admins could declare the URL, method and input body, using templating to replace value and format the message

Variables are linked to a tasktemplate and not to a resolution: when there is some modification needed in the variable, it requires either to re-prod a new µTask version, or to search/replace all occurrences of the variable by the correct value, which is not possible everywhere.

When releasing a new version of utask, goreleaser is not pushing install-utask.sh, which breaks our installation procedure which is:

mkdir utask && cd utask
wget https://github.com/ovh/utask/releases/latest/download/install-utask.sh
sh install-utask.sh
docker-compose up

We should have a way to push this file automatically to the release, in order to avoid breaking while releasing.

Describe the bug

In a template step, I want to use the condition operator IN / NOTIN. But it doesn't work: the assertion to check if a given value is IN / NOTIN a list of items is never true so I never match this condition.

Acccording to the doc:

Note that the operators IN and NOTIN expect a list of acceptable values in the field value, instead of a single one. You can specify the separator character to use to split the values of the list using the field list_separator (default: ,). Each value of the list will be trimmed of its leading and trailing white spaces before comparison.

Here is my step:

checkPlaybookStatus:
  description: Check the previously launched Ansible playbook's status is "successful"
  dependencies: [launchPlaybook]
  action:
    type: http
    configuration:
      url: http://some-url/api/v2/jobs/{{.step.launchPlaybook.output.id}}
      method: GET
  conditions:
    - type: check
      if:
        - value: '{{.step.checkPlaybookStatus.output.status}}'
          operator: IN
          expected: pending,waiting,running
      then:
        this: TO_RETRY
      message: Ansile playbook is still running

Here the value of {{.step.checkPlaybookStatus.output.status}} is "pending", I can see it in µTask GUI.

In µTask logs I can see that my condition is not matched:

AfterRun: Step [checkPlaybookStatus] condition eval: Condition not met: expected pending to be found in list of acceptable values"

I tried to play with the format of the expected: entry in my template (using a yaml list; using another delimiter than , and specify it in list_separator; etc.) without success. The only solution I found is to divide this condition in 3 different if conditions trying to match the value with the EQ operator. It works, but using the IN operator would be much better.

Am I doing it wrong, or am I missing something in my template config ? Or is it a bug with the IN / NOTIN condition operators ?

To Reproduce
Steps to reproduce the behavior:

1. Have a template step that generate an output
2. Add a conditions block of type check with an IN operator; having at least one of the items in the expected: list that match your value
3. See the log Condition not met

Expected behavior
The condition should be matched.

utask version impacted by the bug
v1.5.0

Context which you are currently running utask
Running inside a Docker container, built with this repo Quick Start. Not using any specific plugin.

As a µtask operator, I would like for task templates to listen to a kafka topic and derive inputs from the contents of messages on that topic, in order to trigger a new process execution.

As a µTask operator, I'd like to have a quick overview of the data returned on "GET /unsecured/stats" integrated on the graphical dashboard

• I'm submitting a ...

  • bug report
  • feature request

• What is the current behavior?
  When importing templates at startup, if templates contains some subtasks, the subtask template name needs to already exists in database to be added.
  In case of a first deployment, if we have two templates like this:

name: hello-world
steps:
    say-hello-world:
        action:
            type: subtask
            configuration:
                template: say-something
                input:
                    who: 'rbeuque'
                    what: 'Hello, world!'

name: say-something
inputs:
  - name: who
    description: Who
  - name: what
    description: What
steps:
    say-it:
        action:
            type: echo
            configuration:
              output:
                who: '{{.input.who}}'
                what: '{{.input.what}}'

hello-world.yaml will be imported first, but will fail with errors, as template contain a subtask, calling template say-something, which haven't been imported yet.

• What is the expected behavior?
  Check all template name that will be imported during the batch before performing validation.

• Which version of uTask are you using?
  v1.2.2

Is your feature request related to a problem? Please describe.
I would like to provides to my scripts utask task id and resolution id. Currently, i've to give through argv but it could be fine to provide them through environment variables.

Describe the solution you'd like
Currently:

action:
  type: script
  configuration:
    file_path: my_wrapper.sh
    argv:
    - '{{ .task.task_id }}'
    - '{{ .task.resolution_id }}'

Expected:

action:
  type: script
  configuration:
    file_path: my_wrapper.sh
    environment:
       utask_task_id: '{{ .task.task_id }}'
       utask_resolution_id:  '{{ .task.resolution_id }}'

travis-ci.org is closing, we need to remove build from travis and migrate all the build/test pipeline on our CDS instance.

Describe the bug
When an output strategy is declared inside an action, when the templating fails, utask fails silently instead of returning an error.

Return of generateOutput https://github.com/ovh/utask/blob/master/engine/step/step.go#L426 is not checked.

Expected behavior
FATAL_ERROR I guess ?

utask version impacted by the bug
v1.12.0

As a µTask operator, I'd like to limit the amount of finished tasks that are left in my "live" table: this could hit the performance of the engine when collecting unfinished tasks.

The task expiration parameter in µTask's configuration allows me to set a period after which they get deleted

However, I'd still like to keep a trace of those tasks, for auditing purposes, and it is practical for me that they remain within µTask, to leverage its security guarantees

Hence, an option to "archive" finished tasks instead of outright deleting them would be valuable

Design TBD (a copy of the task and resolution tables? a new set of API endpoints to retrieve this data?)

• I'm submitting a ...

  • bug report
  • feature request

• What is the current behavior?
  When using a template with some uppercase inside the name, template is being normalized before inserted inside database.
  When the template already exists in database and needs to be updated from the template directory, template is not normalized and leads to this error:
  FATA[2020-01-15T11:29:24+01:00] pq: duplicate key value violates unique constraint "task_template_name_key"

• What is the expected behavior?
  No errors.

• Which version of uTask are you using?
  v1.2.1

• I'm submitting a ...

  • [ x] bug report

• Do you want to request a feature or report a bug?
  Bug report
  install-utask.sh provided is not correctly updated when we download it :
  Steps to reproduce :

wget https://github.com/ovh/utask/releases/latest/download/install-utask.sh
docker-compose up

• What is the current behavior?

docker-compose up
Starting utask_db_1    ... done
Starting utask_utask_1 ... done
Attaching to utask_utask_1, utask_db_1
utask_1  | time="2019-11-21T08:24:07Z" level=info msg="[DatabaseConfig] Using 50 max open connections, 30 max idle connections, 60 seconds timeout"
utask_1  | time="2019-11-21T08:24:08Z" level=fatal msg="dial tcp 172.18.0.3:5432: connect: connection refused"
db_1     | LOG:  database system was shut down at 2019-11-21 08:24:00 UTC

• What is the expected behavior?
  Application is starting

• Which version of uTask are you using?
  Latest version - 1.0.1 (see wget command)

• Please tell us about your environment

lsb_release -d
Description:	Ubuntu 18.04.2 LTS

• Other information
  This commit is not correctly applied to install-task.sh provided
  8ecc5dd

In docker-compose.yaml :

    depends_on:
      - db

is missing in utask section

After adding this part, application is starting correctly

--
I am not familiar on how install-script.sh is generated for the release, but if you have few guidelines, I will be glad to open a PR

Describe the bug
When a subtask step is set to TO_RETRY by a child step, the subtask is not restarted and step pass directly to DONE.
It should spawn a new subtask, to retry the subtask execution, and not re-use the old spawned task.

Expected behavior
New subtask is started.

utask version impacted by the bug
All

Is your feature request related to a problem? Please describe.
In a template, i'm creating a subtask. I would like to define watcher_usernames with the main template value.

Describe the solution you'd like

...
inputs:
...
steps:
...
  install-servers:
    foreach: '{{ field `step` `wait-scale` `output` `result` `result` | toJson }}'
    description: Follow server installation and do post-installation steps
    action:
      type: subtask
      configuration:
        template: install-server
        input:
          ...
        watcher_usernames: '{{ .watcher_usernames | toJson }}'
...

Describe alternatives you've considered
Currently, when i create a new task, i've an input with a list of usernames.

• I'm submitting a ...

  • bug report
  • feature request

• What is the current behavior?
  When utask is being deployed behind a ProxyPass with a path prefix, UI doesn't behave correctly because it doesn't have the correct base URL for it's own files, and to call API.

• What is the expected behavior?
  To be able to configure a path prefix while working behind a proxypass

• Which version of uTask are you using?
  v1.2.0

Helm chart with an optional development postgres DB included.

Implement slack integration: let µTask send notifications over slack chans.
Current available integrations: TaT (github.com/ovh/tat)

Note: for support questions, erase this form. Otherwise use approriate areas..

• I'm submitting a ...

  • bug report
  • feature request

• Do you want to request a feature or report a bug?

Feature request/enhancement.

• What is the current behavior?

The error value of a step action is not accessible, and cannot be used in a check.

• What is the expected behavior?

Given the following step with conditions, I would like to be able to access to the error value so that it can be used with the REGEXP operator in a check.

    myStep:
        dependencies: [myDep]
        description : Loreum Ipsum
        action:
            type: toto
        conditions:
            - type: check
              if:
              - value: '{{.step.anotherStep.error}}'
                operator: regexp
                expected: '.*error message.*'
              then:
                this: DONE
              message: foobar

• Which version of uTask are you using?

Latest.

Note: for support questions, erase this form. Otherwise use approriate areas..

• I'm submitting a ...

  • bug report
  • feature request

• What is the current behavior?
  https://github.com/ovh/utask/blob/master/pkg/plugins/builtin/subtask/subtask.go#L43
  The subtask plugin uses an internal context object to exchange data between the main controller goroutine and itself.
  In #54, we fixed a problem regarding the usage of hiphens inside the template (for the stepName). Now that we fixed it, a new bug has been introduced as the context is being json.Marshal, then, the double-quotes get escaped, which breaks the templating with error:
  failed to template context: Templating error: template: tmpl:1: unexpected "\\" in operand
  Faulty context example:
  {"task_id":"{{ if (index .step \"subTask-1\" \"output\") }}{{ index .step \"subTask-1\" \"output\" \"id\" }}{{ end }}","requester_username":"{{.task.requester_username}}"}

• Which version of uTask are you using?
  v1.2.0

• Internal reference
  CORDEV-138

Implement a new type of action for sending email messages from within a task's step

Provide a CLI utility for linting / testing templates before loading them in µtask

Is your feature request related to a problem? Please describe.
In a template, i want to exexcute a subtask with a name provided as input of the main template.

Describe the solution you'd like

...
inputs:
...
- name: installation_template
  description: Installation template
- name: installation_watchers
  description: Installation task watchers
  collection: true
...
steps:
...
  install-servers:
    foreach: '{{ field `step` `wait-scale` `output` `result` `result` | toJson }}'
    description: Follow server installation and do post-installation steps
    action:
      type: subtask
      configuration:
        template: '{{ .input.installation_template }}'
        input:
          ...
        watcher_usernames: '{{ .input.installation_watchers | toJson }}'
...

• I'm submitting a ...

  • bug report
  • feature request

• What is the current behavior?
  https://github.com/ovh/utask/blob/master/pkg/plugins/builtin/subtask/subtask.go#L43
  The subtask plugin uses an internal context object to exchange data between the main controller goroutine and itself. The templating instructions are generated in a Sprintf here, and results in e.g. {{.step.foobar.output}}.
  In the case of a foreach child step, the generated template instruction becomes {{.step.foobar-0.output}}, which makes the templating engine very unhappy because of the presence of a hyphen: failed to template context: Templating error: template: tmpl:1: bad character U+002D '-'.
  This is a common issue with texttemplate, see moby/moby#35886 helm/helm#2192 gohugoio/hugo#1474 etc.

• What is the expected behavior?
  The templating generation should be more cautious to avoid generating invalid templating instructions in that case.

• Which version of uTask are you using?
  1.1

• Please tell us about your environment

• Other information (e.g. detailed explanation, stacktraces, related issues, suggestions how to fix, links for us to have context, eg. stackoverflow, gitter, etc)

When a task requires resolver input to be resolved, resolver inputs should be fetched from the template.
We should provide a quick way to see what resolver inputs are needed to simplify creating the resolution.

• I'm submitting a ...

  • bug report
  • feature request

• What is the current behavior?
  if I configure

base_output:
    foo: bar

for a step that returns an array, my base output is ignored.

• What is the expected behavior?
  I'd like to be able to access my base output (important to keep local context in foreach step chains).
  This could mean that the base_output should be injected in each item of the result array? Then how does this work if the result array is a list of scalars (ie, list of IDs) ?

• Which version of uTask are you using?
  1.1.0

• Please tell us about your environment

• Other information (e.g. detailed explanation, stacktraces, related issues, suggestions how to fix, links for us to have context, eg. stackoverflow, gitter, etc)

• Transfer examples in a specific folder
• Init Travis CI
• Move.shfiles to hack folder, Kubernetes style
• Init code coverage badge
• Complete README draft
• Add SC

We should have a group support to handle permissions access from a LDAP, SSO, ...

allowed_resolvers_groups on templates for example.
admin_groups on configuration for example.

Provide an action to extract data from an html page

As a task template author, expressing a process in a single yaml file limits its overall complexity
Having the possibility of spreading and organizing many actions across several files would allow for more structured code
Design TBD (import modules from a "root" template? unmarshal several yaml files onto the same object? stitch their text content naively? etc...)

In task templates, some sequences of steps tend to form natural clusters, ie units of work that have a meaning as a group
The ability to "collapse" those steps into a single block would allow for better rendering of an entire task, a clearer visualization of the whole

Hi,

I am trying to implement an InitializerPlugin for http header token authentication.
I find documentation not clear enough and really struggling with this one.
The documentation of the plugins us much more clearer and lots of examples exist.
Would really appreciate simple example or additional information for the documentation.

Is your feature request related to a problem? Please describe.
When I'm using foreach, all steps are created without dependency and not sequentially.

Describe the solution you'd like
I would like to be able to create my steps sequentially, with a dependency created on the previous step.

...
variables:
  - name: my-groups
    value: '["group1", "group2"]'

steps:
  deploy-groups:
    (new)foreach: '{{ eval `my-groups` }}'
    description: Dummy
    action:
      type: echo
      configuration:
        output: '{{ .iterator }}'

This will create deploy-groups-0 and deploy-groups-1. I want that deploy-groups-0 is executed before deploy-groups-1 and that deploy-groups-1 is dependent of deploy-groups-0 (if deploy-groups-0 fails, deploy-groups-1 is not executed).

Describe alternatives you've considered
No alternative found.

Additional context
N/A