Workflow versioning about lyra

[markfuller]

A scenario to consider. A plugin upversions (say from 5.4.2 to 6.0.0) and adds one mandatory field, as well as removing another).

(The plugin exposes a typeset, which can declare a version so this is probably the plugin version.)

(1) The plugin must be able to handle all prior versions, OR specify which versions it can handle (e.g. semver range). For the latter, the identityDB would therefore probably need to record the version of the plugin creating/updating the resource alongside the identity mapping and pass it to the plugin for future (CRUD) operations (2).

The workflow itself should be able to specify the version of the typeset it pertains to (3), as well as the version (or range of versions) of the workflow engine it can run on (4). The workflow engine is not yet versioned (5)

We should try to support git SHA here too (6) - we can be inspired by go modules or similar systems here.

Numbered items are potential future tickets.

[scottyw]

@markfuller and I discussed this today a bit more.

There are a bunch of orthogonal versioning issues:

• (a) Plugins can change over time meaning that a workflow that works with one version of a plugin might not work after an upgrade/downgrade. Mark covers this with his 1, 2 & 3 I think.
• (b) Assuming plugins don't change and workflows don't change, the workflow engine might change in a way that causes older plugins or workflows to fail. That's 4 and 5 I think.
• (c) Assuming plugins don't change, a user might wish to try out iteration 4 of a workflow while continuing to use iteration 3 in other contexts. Think staging vs production.
• (d) Two users might want to collaborate on workflow A, where each individual can apply the workflow and affect the same real world infrastructure. The same two users might also wish to run workflow B in isolation from each other, where one's changes do not affect the other's infrastructure. We talked about salting identities to handle this or we could instead support many identity services and have users indicate which context is to be used for each run. The latter is a bit like Puppet environments.

Looking at (c) in more detail, we can address this in a clumsy fashion today via workflow naming e.g. "myworkflow_staging", "myworkflow_production". That's a little like the concept of environment in Puppet but managed in an adhoc fashion. Potentially we could support "namespaces" or similar in the workflow? It makes me think of r10k which relates back to #86.

We could choose to put an actual version number into workflows and have the user specify the version number at workflow application time but I think that leads to awkward semantics. Presumably if I had applied "foo v.1.2.0" and was now working on "foo v2.0.0" I would want to test v2 without affecting resources created by v1.2. Once I was satisfied with v2 I would want the exact opposite - I would want to modify the resources created by v1.2 to be changed in line with v2. Further, what does it mean for me to modify v1.2.0 of the workflow without changing the version number by hand? Or if we automatically upversion, why other with versioning at all just use a hash/git branch instead?

Looking at (d) in more detail, there's some cross over with #57 and again a slight parallel with r10k if we think of separate identity DBs like contexts/environments.

Ultimately (a), (b), (c) and (d) are distinct problems that we can (and probably should to some degree) address separately. I think there's a bit of a relationship between (c) and (d). I could imagine solving a lot of problems with commands like this:

Run a local copy of a workflow using a local identity DB
lyra apply myworkflow

Run a remote copy of a workflow using a local identity DB
lyra apply myworkflow --include https://github.com/lyraproj/handy-workflows/plugins

Run a local copy of a workflow using a shared, remote identity DB
lyra apply myworkflow --identity http://lyra-production.s3.amazonaws.com

Run a remote copy of a workflow using a shared, remote identity DB
lyra apply myworkflow --include https://github.com/lyraproj/handy-workflows/plugins --identity http://lyra-production.s3.amazonaws.com