walkerh / pipe-o-matic Goto Github PK

See https://github.com/walkerh/pipe-o-matic/blob/master/docs/pipe-o-matic-functional-specs.creole#define-a-sequential-pipeline

Need to develop the first version of a "getting started" guide. Probably the best way to do this is for a pair of developers to work through a deployment. One will do the work while the other captures a tutorial.

https://github.com/paddymul/extproc

extproc might work well as a substrate for what you are doing here with pipes.

I continue to think about how to make setup of POM easier, especially deployments.yaml. Here is a more concrete description of how I see things working- please let me know if this matches up with what you think should happen:

A command ("pmaticsetup"?) should be available to generate a new deployments.yaml. This command would take a few parameters:

• An optional location (the base directory) to write the deployments.yaml to, defaulting to somewhere like ~/.pmatic/ if not specified
• If the directory doesn't exist, it will be created.
• If it doesn't exist, a pipelines directory will be made inside the directory
• If the deployments.yaml already exists, the program will abort with an appropriate error message unless a force flag is passed (maybe '-f').

When run, pmaticsetup will parse a YAML database of application names and parameters (interrogators? as we discussed in a previous email), possibly called interrogators.yaml. These executables will be detected using 'which', and if located, added to the deployments.yaml. Deployments.yaml will list (for each program) the command to invoke it, the name of the program, the flags necessary to print out a version number, and the regex necessary to extract the version number from the command's output.

When pmaticrun is invoked without finding a deployments.yaml, it should suggest how to configure a proper directory or invoke pmaticsetup.

Possible enhancements in the future:

• When run, any applications that are found with this discovery process could be added to an existing deployments.yaml if they are not already in the file. This would allow you to update your existing deployments.yaml after installing a new application without overwriting any manual entries.
• Add a parameter to manually specify the location of the interrogators.yaml.
• Add a parameter to add search directories other than $PATH.

Use the location of the bin directory containing to top-level scripts. Assume the following deployed layout by default:

• bin/
  • pmaticrevert
  • pmaticrun
  • pmaticstatus
  • etc
• lib/
  • pmatic.py
  • yaml & other dependencies
• pmatic_base/
  • deployments.yaml
  • pipelines/

Implementation

PMATIC_BASE = absolute path of "pmatic_base"

If PYTHONPATH is not set, we actually update sys.path as if...

PYTHONPATH = absolute path of "lib"