pyinvoke / invoke Goto Github PK

Kinda forgot that part of #23. CONFIDENCE INSPIRING.

When printing text inside the Context test suite and running the tests with invoke itself, I get what looks like classic thread print problems. Running tests via spec works fine. Figure out wtf. Could swear similar test debug prints in eg Parser worked fine yesterday.

Argument-level API allows for an arg to have one or more names, but right now @task only does auto-shortflags or no-auto-shortflags without exposing the aliasing behavior to users.

Obvious choice is a map, @task(argnames={'realkwarg': ['r', 'rk']}) (turning into the parser-level flags for realkwarg being --realkwarg, --rk and -r). Not sure what this option's name should be though, argnames is kinda fugly.

One pattern that is very useful but never appeared in Fabric 1.x is solid environment variable support for configuring the tool, e.g. setting config options via the invoking environment.

This would be smart to do in Invoke, especially if done in an extensible way (e.g. client libs can say "this option can be set by env var XXX")

See subject.

% virtualenv venv
% source venv/bin/activate
% git clone git://github.com/pyinvoke/invoke.git
% cd invoke
% python setup.py install
% inv
Traceback (most recent call last):
...
  File "build/bdist.macosx-10.8-intel/egg/invoke/vendor/lexicon/alias_dict.py", line 1, in <module>
ImportError: No module named six

I can see six.py in vendor/ but it looks like this is a simple import error since six isn't installed system wide and there's no six.py in vendor/lexicon.

This is actually my first interaction with invoke so maybe I'm doing it wrong.

Here's some platform info:

% python --version
Python 2.7.2
% uname -a
Darwin nibbler 12.2.1 Darwin Kernel Version 12.2.1: Thu Oct 18 12:13:47 PDT 2012; root:xnu-2050.20.9~1/RELEASE_X86_64 x86_64

I.e. in the case where you had to name the task something silly to work around namespace collisions within the host Python module. See e.g. https://github.com/pyinvoke/invocations/blob/9c342190c5af1ea2ec74bbcc3b6fc1e935d3cfcd/invocations/docs.py where we had to rename what would've been def build to def _build, then build an explicit namespace mapping build=_build.

Would be marginally cleaner to be able to say @task(name='build') and then we can simply skip the extra Collection object.

This touches on the fact that in this case, where name-in-collection differs from name-in-Python-module, there's a mismatch between calling a task directly (_build()) and claiming it as a pre-task (@task(pre=['build'])). Not sure there's much to be done there though...

EDIT: Well, the obvious answer is to say that calling via an explicit Collection API, e.g. Collection.execute("name"), is preferred over using __call__. Feels like we could offer both?

At the very least, need a couple toggles on our Popen subclass capable of telling it not to print stdout/stderr bytes.

Ideally this would be handled via logging but I am 99% sure logging simply won't fit at the byte-by-byte level. See #15.

Like a moron, I wrote the new parsing stuff without remembering that the initial context needs to be able to influence the rest of the parse run. I.e. invoke --collection=foo task1 task2.

So we need things to look like this:

• Parse once, focusing solely on the initial context and returning/putting aside everything after what would be the first context switch.
• Use that result to discover + load the remainder of the tasks/contexts into a new parser
• Run the unparsed result saved above, in a new parser primed with the newly loaded contexts.

Since the "initial" context will no longer ever be followed by non-initial contexts, the API should probably change around a bit.

Also consider a 'rolling' approach where, instead of explicitly/externally running this 2-step parse, we allow contexts to alter the parser itself when they run (as per Fabric #391).

E.g. the initial context actually "runs" instead of just being a container for core options, and as part of that execution, loads + adds the task collection(s)' contexts to the parser.

Then regular tasks/contexts are able to similarly manipulate the list of tasks being executed (though note that this should be distinct from altering the argv being parsed -- objects > lists of strings.)

This approach suffers from a lot of problems (ties together things that really ought to be separate, confuses the act of parsing with the act of execution of tasks, etc) but is at least worth recording.

Invoke could also be usable for projects that are trying to write their own CLI interfaces. Therefor, it should be possible to call a method that parses the arguments and dispatches them to the right task.
More customization should be allowed in that script.

Find a decent way to indicate, on a task or automatically, that e.g. invoke taskname --val=5 should result in a Python value 5 instead of "5", and of course similar for bools, lists etc.

Possibilities:

• Explicit on the CLI level:

  invoke mytask --wants-an-int int:5 --wants-a-list=list:"a;b;c"
  

• Explicit on the task declaration level (off the cuff API, but you get the gist):

  @task(argspec={'wants_an_int': int, 'wants_a_list': lambda x: x.split(';')})
  def mytask(wants_an_int, wants_a_list):
      # ...

• Implicit via conventions, e.g. try/except casting to int and/or float, anything containing semicolons gets split()'d, etc

  • Original ticket said to cast y/yes/etc to booleans, but now that we're using --real --flags booleans happen for free.
  • Would obviously need to be easily toggled, and think about what the default should be

• Explicit via helper functions -- obviously int/float already exist, so maybe a helper def listarg(x): return x.split(';')

  • Feels kinda dumb unless we can come up with a larger body of useful subroutines worth supporting.

Right now, I'm leaning towards the explicit-at-task-declaration level, with possible implicit-via-conventions. Very torn on whether the implicit should be on by default -- fits the "get stuff done fast" use case but clashes with "don't be surprising".

Was Fabric #69

Scenario A: somebody, somehow, accidentally ends up trying to Collection.add_task('foo', foofunc) 2x in a row (with foofunc being identical, or not, doesn't really matter.) Should raise an error by default but probably doesn't now. Add a test.

Scenario B: somebody explicitly wants to override an existing task, e.g. they're dealing with an upstream tasks module and want to customize behavior. They may want to do a not-quite-inheritance method of overriding by defining a function referencing the original task, and have their own function replace the original inside the collection.

(Yes, having true inheritance based class stuff is more ideal, but it feels like users should be capable of replacing individual tasks within a collection if they flip some option/arg enabling this kind of power-user feature.)

General needs (should tie into Collection):

• Organize tasks in a logical namespace tree instead of a flat namespace
  • Emphasis on logical - while module level scanning for Tasks should be an option it should not be the primary interface.
• Refer to them with dot syntax (foo.bar) both on CLI and in e.g. pre/post
• Aliasing (having a task or even a subtree show up in >1 location in the tree)
  • Including at the same level, of course
• Moving/renaming (like aliasing, but masking out the "original" location)
  • Hopefully not required anymore with an explicit API, like it would have been with Fab 1 style implicit-module stuff?
  • I.e. users who need this can simply alter the calls building the tree.
• Flexible display in --list
  • I.e. invoke --list --root foo.bar to display only the subtree beginning with foo.bar
  • Again, maybe not required if we can give users enough control, e.g. when the use case is to give non-admin users a different view, could define a real top level list task that does e.g. show_tasks('user') or whatever?

Right now the following task declaration "skips ahead" to avoid conflicts, which means it's tied to argument order, which means it can change sort-of unexpectedly if you have 2nd thoughts about your task function's signature:

@task
def mytask(foo, bar, biz): pass

As-is, that aliases -f to --foo, -b to --bar and -i to --biz (since -b is already taken.)

If one switched the task to become e.g. mytask(foo, biz, bar) the shortflags would change, with -b pointing to --biz not --bar, and with -i disappearing, replaced by -a. While conscientious users should be aware of this possibility, it feels like an easy foot-shooting.

Ways to avoid this:

• Refuse to create ambiguous shortflags. So above the only shortflags generated would be -f.
  • Saves foot-shooting at the expense of convenience (users then must explicitly set up short flags if desired).
  • Also means that addition of conflicting args where none existed prior, makes pre-existing shortflags disappear. Confusing.
• Don't change existing shortflag behavior, but disable it by default, making it a "power user, know what you're getting into" feature.
  • Including a warning in the docs about the potential for change if arg order switches.
  • Not having shortflags on by default is less convenient, unfortunately. Which is more important, safety or convenience?
• Don't change existing shortflag behavior, but ensure that e.g. #34/#35 are implemented well, so users have an easy way to override the automatic shortflags or to fill in omitted ones (if we go with the 1st bullet point above.)

I.e. instead of just inv --list spitting out the current default flat list, allow things like inv --list nested for a nested/indented display, inv --list=foo to just list the tasks within namespace foo, etc ad infinitum.

LATER EDITS: so a lot of strongly related tickets have popped up and there seem to be two major currents:

• Formatting, i.e. flat with dots vs nested vs machine-readable
• Content, i.e. all tasks vs just namespaces vs just tasks within a given namespace

Seems unwise/impossible to shoehorn all of that under a single optional value to --list so I'm sure we'll grow some additional flags either way, with the main question being which (if any) such option deserves to live as the optional --listvalue.

Brainstorm:

• Format could be a single flag with string option, e.g. --list-format (flat|nested|machine) (what term do other tools use for "machine-readable" anyways? Can't remember offhand...LATER EDIT: actually these days we often just use JSON for that and just call it eg --list-format=json) or it could be a handful of booleans like --list-nested, --list-machine.
  • Former feels easier to implement, harder to fuck up by giving >1 at a time, and easier for users to grasp (i.e. fewer options to think about) so it seems like a no-brainer.
• Content needs levers for 'root' (though this needs to not clash with the related-but-distinct --root flag which changes where tasks files are sought from; but perhaps we should be, well, namespacing those as well so they are renamed to eg --discovery-root) and 'depth` and I think that's it, since those together can enable just about anything:
  • "Show me everything": root is the top of the namespace, depth is None
  • "Show me just the namespaces": root is top of namespace, depth is 1
  • "Show me just tasks under namespace foo"
• Might actually need another lever for the confluence of those two, i.e. "I am listing some subset of tasks and I want to tweak whether they are shown as full paths or sub-paths"?
  • Though at this point I wonder if users who want a non-full-path display (i.e. inv --namespace foo --list spitting out task names like bar and baz instead of foo.bar and foo.baz) should be relying on "each Python module can be its own standalone namespace" and doing things like inv --collection=namespace --list" instead (see Collection loading.)
    • Since, if one truly wants to 'focus' on a given sub-namespace, giving it to --collection automatically ensures that everything - display, invocation, etc - instantly works as you'd expect.
  • The main argument I can think of why one would not want to go that route is if one wanted to do nontrivial slice n dice of namespace contents within the root tasks file (such that the namespace does not reflect Python module or package structure.)
  • However it's not necessarily out of the running, we could either implement things to be depth-aware and default to stripping out parent levels on display/invocation – or (probably better) just make it possible to say "hey I know you loaded my root tasks namespace, but I'd like you to act as if I gave you foo.bar as an argument to --collection - even though, between you and me, foo.bar isn't visible on the Python import paths."
  • Rereading this later, my gut says "ugh we're making this too complex, just add another flag if people REALLY want to see non-full paths for some stupid reason". And besides, what happened to the 'nested' idea? That's another way to show the 'short' names while clearly presenting depth/parentage.
• Minor quirk: do we bother showing the root namespace when we are being asked to "show me the namespaces"? I'm leaning towards no, but might be nice to be explicit...? e.g. "your namespaces are: (root), foo and bar"

E.g. obtaining a task from a Collection by its name; given the existing emphasis on dict-like syntaxes elsewhere, it feels like it'd make sense to take this step.

🚨 TODO: copy in stuff from the Fabric ticket 🚨

Might be nice to have the API for "I depend on these other tasks running first" be *args in @task, e.g. @task(other1, other2, kwargs=here). Make-like, simple, *args aren't sensible otherwise (e.g. to call the existing things intended to be kwargs). Props to @mattrobenolt for the idea.

Like set +x or how Fabric does [hostname] run: command here lines. Probably off by default?

• Originally (#1) decided positional args were a no-go due to ambiguity between post-task-name positional args and potential "next task name".
• However, at the time I believe I was thinking along the lines of optional and/or n-ary positional args.
• My current, albeit rusty, knowledge of the parsing situation implies that if we make positional args required -- as in, if a task somehow declares N positional args, N positional args must be given on the CLI -- then the parser can handle it fine, because it will know that the next non-flag, non-flag-value token it reads needs to fill in a positional arg and thus cannot be a task name.
• So, this will fill needs for folks who want a "classic" subcommand-like API for some tasks, while not causing parser ambiguity by trying to support arbitrary numbers of positional args.

Experiment with this -- we have enough tests for the current setup in place and passing that it should be obvious whether it will work or not.

Re #41

I didn't tackle this right off so as not to overcomplicate v1.0, but I could see some additional tweaks/options/flavors to how we handle deduplication:

• Right now we basically build a list of all to-be-invoked tasks and naively dedupe. No fuss no muss.
• However, task source could matter: invoked due to pre/post is not necessarily the same "priority level" as invoked directly by hand. Users might expect pre/post "dependencies" to get deduped but for directly-invoked things to always run.
  • For example, $ invoke foo foo being deduped might be confusing.
  • Make itself, however, does do this level of deduping; make foo foo will run foo once and then say foo is up to date.

https://github.com/fabric/fabric/issues?labels=Invoke&page=1&state=open

Left mostly-working help system in a branch. First obvious thing that still needs work: a useful column formatter. Probably vendorize clint or similar?

Using default values on anything other than 'bool' type arguments causes a ParseMachine error.

Context argument:

Argument(
    names=('setup', 'S'),
    default='~/.stitches',
    help="Setup 'DIR' as stitches root, defaults to '~/.stitches'."
)

Output:

$ stch -S
DEBUG:stitches:Base argv from sys: ['-S']
DEBUG:stitches:Parsing initial context (core args)
DEBUG:invoke:Initialized with context: <Context: {'root': <Argument: root (r)>, 'setup': <Argument: setup (S)>, 'version': <Argument: version (V)>, 'list': <Argument: list (l)>, 'help': <Argument: help (h)>}>
DEBUG:invoke:Available contexts: {}
DEBUG:invoke:Wrapping up context None
DEBUG:invoke:Starting argv: ['-S']
DEBUG:invoke:Handling token: '-S'
DEBUG:invoke:Saw flag '-S'
DEBUG:invoke:Moving to flag <Argument: setup (S)>
DEBUG:invoke:ParseMachine: 'context' => 'end'
Flag <Argument: setup (S)> needed value and was not given one!

This led me to:

invoke/parser/init.py:200

def complete_flag(self):
    if self.flag and self.flag.takes_value and self.flag.raw_value is None:
        self.error("Flag %r needed value and was not given one!" % self.flag)

My quick fix:

def complete_flag(self):
    if self.flag and self.flag.takes_value and self.flag.raw_value is None and self.flag.default is None:
        self.error("Flag %r needed value and was not given one!" % self.flag)

After that it passes but there is no easy way to tell (that I can see) if a flag value was set or not from the command line or if the default was used. Where that becomes a problem is if I am trying to use for example:

if args.setup.value:
    # do stuff

As the argument always has a value (as it falls back to the 'default' attribute and the 'args' Lexicon lists all the context flags regardless if they were passed or not). Maybe a 'was_passed' attribute on the argument itself?

For now I can check back against the originally passed args.

Thanks!

Right now disabling positionals requires @task(positional=[]) which is pretty fugly.

Slightly better would be None but that's the default implying to use auto-positionals. (Could maybe make a legit sentinel for the default, like we've done elsewhere, then use None to mean disabling...)

Or we could add another kwarg like no_positionals, but then there's the usual "what happens if both are given" problem.

Or could rely on global setting added in #27 but I'm kind of sick of that being the only way to configure things in my software.

So far that first option actually sounds cleanest.

Re #39, re #27

Existing examples: Fabric's _AttributeDict (object attribute access on top of regular dict behavior) and _AliasDict (allow N-1 relationships from keys=>values, i.e. aliasing.)

Need in Invoke: probably for config objects, similar to how Fabric uses _AttributeDict now, and to handle task/argument/etc aliasing (as is used in _AliasDict now.) Would be best to merge both features and make it public.

Fabric needs on top of what's there now:

• Apply this rich dict to roledefs
• Add aliasing to the env AttributeDict

Ideally, find prior art and use that instead. Possible candidates:

• Bunch - reasonably new, only fills in for AttributeDict, would need to extend as an AliasDict
• easydict - pretty much the same as Bunch? Check the github, compare the codebases or something.

Also maybe related:

• nested_dict - not attribute or alias, but easy recursively nested dicts. No source link but if quality's OK could "fork" onto our github maybe.
  • This would be mostly useful for the config-object use case; nesting of parser contexts & task namespaces is likely too complex/specific to be treated as simple nested dicts.
• infinitedict - ditto, but on GH, somewhat newish.

It apparently uses 2to3 in setup.py; we need a sixified version that I can re-vendor

There's just too many levels of tests now, I tend to end up writing one for each level of the stack and that's just too much repetition both in terms of actual tests & setup for tests.

E.g. I have at least 3-4 spots where I set up a collection w/ top level tasks + subcollection + aliases + default tasks, all in the same way (or different ways! like in-code vs loading a _support module) and then make the same assertions in e.g. cli parsing, Collection.to_contexts, Collection.task_names, etc.

In that example we could at least consolidate all the setup bits into the same module, and then remove e.g. the to_contexts or task_names tests (task_names is really just an impl detail at this point).

TDD is great and all but once the tests are done I have to remember to go remove some, otherwise any changes to behavior require altering far too many tests.

There appears to be an issue surrounding the variable names I use to define base and namespaced collections. The following code in 'tasks.py':

from invoke import Collection, task, run

@task
def release():
    run("ls")

@task
def example():
    run("ls")

base = Collection()
base.add_task(release)

ns = Collection('name')
ns.add_task(example)

base.add_collection(ns)

Only outputs the 'name' collection, and uses it as the base:

$ invoke --list
DEBUG:invoke:Base argv from sys: ['--list']
DEBUG:invoke:Parsing initial context (core args)
DEBUG:invoke:Initialized with context: <Context: {'help': <Argument: help (h)>, 'list': <Argument: list (l)>, 'collection': <Argument: collection (c)>, 'version': <Argument: version (V)>, 'root': <Argument: root (r)>, 'no-dedupe': <Argument: no-dedupe>}>
DEBUG:invoke:Available contexts: {}
DEBUG:invoke:Wrapping up context None
DEBUG:invoke:Starting argv: ['--list']
DEBUG:invoke:Handling token: '--list'
DEBUG:invoke:Saw flag '--list'
DEBUG:invoke:Moving to flag <Argument: list (l)>
DEBUG:invoke:Marking seen flag <Argument: list (l)> as True
DEBUG:invoke:ParseMachine: 'context' => 'end'
DEBUG:invoke:Wrapping up context None
DEBUG:invoke:After core-args pass, leftover argv: []
DEBUG:invoke:No collection given, loading from None
DEBUG:invoke:Adding <Context 'example'>
DEBUG:invoke:Parsing actual tasks against collection <invoke.collection.Collection object at 0x10dd592d0>
DEBUG:invoke:Initialized with context: None
DEBUG:invoke:Available contexts: {'example': <Context 'example'>}
DEBUG:invoke:Wrapping up context None
DEBUG:invoke:Starting argv: []
DEBUG:invoke:ParseMachine: 'context' => 'end'
DEBUG:invoke:Wrapping up context None
Available tasks:

    example

But when I swap the variable names around (ns <--> base) like so:

from invoke import Collection, task, run

@task
def release():
    run("ls")

@task
def example():
    run("ls")

ns = Collection()
ns.add_task(release)

base = Collection('name')
base.add_task(example)

ns.add_collection(base)

I get what I expect on output:

$ invoke --list
DEBUG:invoke:Base argv from sys: ['--list']
DEBUG:invoke:Parsing initial context (core args)
DEBUG:invoke:Initialized with context: <Context: {'help': <Argument: help (h)>, 'list': <Argument: list (l)>, 'collection': <Argument: collection (c)>, 'version': <Argument: version (V)>, 'root': <Argument: root (r)>, 'no-dedupe': <Argument: no-dedupe>}>
DEBUG:invoke:Available contexts: {}
DEBUG:invoke:Wrapping up context None
DEBUG:invoke:Starting argv: ['--list']
DEBUG:invoke:Handling token: '--list'
DEBUG:invoke:Saw flag '--list'
DEBUG:invoke:Moving to flag <Argument: list (l)>
DEBUG:invoke:Marking seen flag <Argument: list (l)> as True
DEBUG:invoke:ParseMachine: 'context' => 'end'
DEBUG:invoke:Wrapping up context None
DEBUG:invoke:After core-args pass, leftover argv: []
DEBUG:invoke:No collection given, loading from None
DEBUG:invoke:Adding <Context 'release'>
DEBUG:invoke:Adding <Context 'name.example'>
DEBUG:invoke:Parsing actual tasks against collection <invoke.collection.Collection object at 0x10ddab210>
DEBUG:invoke:Initialized with context: None
DEBUG:invoke:Available contexts: {'release': <Context 'release'>, 'name.example': <Context 'name.example'>}
DEBUG:invoke:Wrapping up context None
DEBUG:invoke:Starting argv: []
DEBUG:invoke:ParseMachine: 'context' => 'end'
DEBUG:invoke:Wrapping up context None
Available tasks:

    release
    name.example

EDIT: Added output with debug on

Most of the API is showing up in the autodoc now, but there are probably missing bits. Make sure anything that should be public API is displayed; make sure any impl details are hidden; etc.

https://github.com/fabric/fabric/issues?labels=Invoke&page=1&state=open

Dislike using .value all over parse code; decide if it's worth the magic to implement __str__ and __nonzero__ and etc so one can simply say eg args.name vs args.name.value (for e.g. def mytask(name=xxx).)

This allows us to place stuff in __init__.py for a compact API import, while both letting setup.py access version stuff, and the installed package access version with a normal import.

Could also just move to a pure data format like JSON (given it's stdlib in 2.6) but that cuts out any possibility of doing algorithmic things in the module. Which I'd like to avoid anyway, but still.

(Was: Deal with Fluidity dependency)

This desc is out of date, see this comment below for new game plan.

Original desc follows.

We're using Fluidity for state machine fun, but their packaging is FUBARed right now: version 0.2.1 has a useful-for-debugging hook not in 0.2.0, but 0.2.1 is not actually tagged or on PyPI so it's not possible (AFAIK) to obtain it via a regular pip install.

I'll ping the devs there but if they are unable to comply, options are:

• Vendorize it. Meh. Always annoying re: syncing with upstream. OTOH it's not a big or complex package nor is it heavily developed, so.
• Publish our own copy of it, either via Github tag-url or another name on PyPI. Also meh, but useful if we ever needed to fork it for real sometime (unlikely.)

Scenario:

• tasks.py in my project
• It does from invocations.docs import * to obtain those tasks as top level tasks in my own project's collection.
• It also defines custom tasks.
• It does not do any custom namespace creation.

Expected:

• All tasks showed up equally valid

Found:

• Only invocations.docs tasks appeared; mine were ignored.

At some point invocations.docs grew a custom top level namespace, mostly so it could control what it exported to importers who use it in a custom Collection (e.g. Collection(stuff, here, also, docs))

However, this means that the star import added that ns object to the local tasks module -- and then became "the" local custom namespace. Thus ignoring my local tasks.

Solutions:

• Perform module location test to ensure a given Collection.load_module(x) call only loads namespace collections defined in x and not imported into x. Obvious/naive option.
• Do nothing, ensure users are informed, chalk it up to PEBCAK. (Minor side benefit: advanced users can import a default namespace from outside if they somehow find that useful. I can't think of when this would be useful though.)

• pty-related stuff using pexpect works fine by hand on OS X
• But fails on travis-CI with OSError and regular test Failures
• Also fails -- but only about 50-75% of the time -- when executed by hand on Linux (Debian), with the same errors for the most part.
• Earlier assumed it was due to running w/o a pty and tried things like expect-dev's unbuffer, socat and coreutils' (>=7.5) stdbuf, but none of those help on Travis (though some make things "pass" by masking the return value...)
• But since the errors pop up on a non-headless VPS, may be something else
• This page mentions stuff about pexpect.spawn().interact() dying when trying to handle an EOF.
• Printing out various things about the spawn() object (exit code, signal code, isalive, etc) seems sporadic and not super helpful
• Tho we could possibly get away with ignoring OSErrors which contain the Input/output error text and when exitcode is non-None? But see above -- sometimes we get the errors even when exitcode is non-None too :(
• The actual test run itself (when invoked as inv test vs spec) also exhibits the same errors -- it's not just the test suite.
• Confirmed that minimal test case is simply a task containing run("echo whatever", pty=True)

When running run("python"):

• >>> print 'hm' prints the next line's prompt + "hm" on the same line.
  • Implies mixing of stdout/err poorly still; using print >>sys.stderr, 'hm' correctly deals w/ newlines
• Trying to hit up-arrow for shell history prints ^[[A (control chars) instead of interacting with readline.

Neither of these problems exist when pty=True. I feel like this is the same as in Fabric and any other solution where a pty is not the default behavior. At the very least we should make this explicit in the docs, probably by making an FAQ.

and this makes me mad.

What used to be here is now over in #57...

Would be nice to globally/temporarily set certain default-but-configurable behaviors to specific values.

E.g. in our own test suite, when #23 landed I had to add a bunch of positional=[] args to @task. Would be nice to be able to set that at the test suite level or something.

Main hurdle is simply that we want to do away with module level bullshit so this needs to be effected some other way; possibly nothing we support ourselves besides "well, just do this:"

from functools import partial
from invoke.task import task

task = partial(task, positional=[])

In which case this would just become a "apply that technique to our own tests."

Turning this into a general "how to do logging and useful output controls for invoke and fabric 2" ticket.

Use cases

• Display the task or tasks being invoked so users can follow the "flow" of nontrivial execution
  • plus parameters
  • plus dependency chain info
    • even skipped/satisfied dependencies
• Display subprocess commands being called so users can tell what's actually being done on their behalf
  • plus which task is currently in play (optionally) in case the per-task info has already scrolled by
• Display some sort of 'prefix', on a line-by-line basis, prepended to subprocess stdout/err - such as the command, task or task parameter that spawned it
  • Command itself less likely (esp since most command strings tend to be long) but task or task param is useful for things like parallel execution (#63) or Fabric per-host exec.
• Probably more besides...
• Must be able to turn a lot of this off or on at will
• May require an entirely different stdout/err mirroring/display mode when prefixes are enabled, since otherwise they'd get in the way of bytewise output (think Unicode decoding, interactive sessions, etc)

Older thoughts

Thoughts for now, jumping off of the description of fabric/fabric#57 & current state of things in Invoke master:

• Re: log lib to use:
  • May not do anything besides stdlib logging after all, it's still the most-used option, and the core problem of "what to log, when, and how" is not going to change drastically even if we use another lib. It's Not That Bad, Really™.
  • Structlog came around after that original ticket started, and actually looks like the best option if we want to add anything on top of stdlib (in part as it's the only major one that supports 2.6-3.x, has lots of good features, actually runs on top of stdlib or whatever you want, and I am mildly biased as I know the author :3)
  • Twiggy is 2.x only, so nope.
  • Logbook is 2.7-3.x, so until we're comfortable dropping 2.6 (not for a while...) it's also a no-go. Still a decent looking option otherwise.
• Log a lot of things by default, mostly at DEBUG for, well debugging. E.g. logic paths used, etc - good for troubleshooting.
  • Invoke does this already; it simply offers invoke.utils.debug (w/ setup to add other levels easily) that logs to a package-level logger. Could probably be expanded some, it's mostly used in the parsing and config modules.
• Log 'operational' level things as INFO or similar, sans any 'large' fields - i.e. log when run is called, on which host (in Fabric), with which command (tho this may want truncating...), run-time, etc - but not the stdout/stderr contents (though sizes of those, similar to HTTP log formats, is probably good).
  • Basically, the same data we expose in Result objects.
  • Re: this ticket's original aegis, DO we want to be logging the stdout/stderr ever, at any point? Or should we rely on users' ability to redirect those streams where they need?
  • They could, for example, just point both at their process' stdout/err, which would mix log output with command stdout/stderr well enough...
• I was wondering whether to log both the initiation of, and completion of, actions, or just completion (as above). Seems like best is to log initiation as DEBUG or something else below INFO (since you mostly care when troubleshooting) and completion as INFO.
• When I started fabric/fabric#57 way back, I was assuming we'd want ALL output to use the logging system, with an open question about how to handle bytewise output (as it's not line-based).
  • I no longer think this makes sense - simply log a lot of stuff, as above, and don't forward it to stdout/stderr by default.
  • Allow users to determine where their logs go (usually to a file).
  • When run via the CLI, default to a "Fabric 1-like" mode where basic info is printed to stdout/stderr - what task is run, what command is run, hook up its stdout/err, file transfer info, etc.
  • When NOT run via the CLI, probably default to no output at all - expect users to supply desired file-like objects to e.g. run's stream arguments, and to configure logging to taste.
    • Users wanting a 'hybrid' setup would do what I mentioned above, point both the runners' streams AND the logging framework to system pipes.
• Regarding Fabric specifically, how to handle multi-host invocation?
  • Having line prefixes is still useful for some use cases (especially, but not only, for parallel execution), which also means, we probably do want to retain a line-buffer at some level
  • Having 'headers' is another useful mode I've wanted at times, where I don't want the clutter of line prefixes, but do want to be able to read output sequentially and tell what happened on which host.

Original/outdated description

Right now stdout/stderr is printed in our Popen subclass byte-by-byte to allow interaction etc. However, eventually we'll want real logging, meaning an ability to throw things line-by-line into a log library, meaning an extra buffer at the Popen level that does additional Things with lines as they are tallied up.

I suppose there should be an option to turn off the bytewise in favor of just linewise, but that could be a 2nd tier thing or saved for when we start layering Fab 2 on top of this.

Many tests that are testing specific behavior of e.g run() take the easy route and call e.g. run('true', xxx), then introspect the result. This of course runs an actual subprocess and makes for a slower test suite ("only" ~4s right now, but it will keep growing.) Some even spawn a subprocess that spawns another subprocess! (E.g. the end of #21's new tests do this to get around pty stuff.)

Only the highest level sanity/integration tests really need to do this; the rest should be able to work with a dependency-injected mock object that behaves as needed w/o spawning a real subprocess.

Required:

• Choose a mocking lib if we aren't already using one.
• Change run to accept a runner argument & factor out the pexpect/subprocess using chunk of current function body.
• Update most tests to do run(runner=<mock>).

Ken's Envoy project has a similar run() to what Invoke will have, and somebody raised the point about making it easy for users to be secure against injection/quoting attacks, by adding a DB-API like parameterization format.

The link: not-kennethreitz/envoy#16

I agree with Ken's points therein that the point of these tools is ease of use, not correctness, but also with the "let's offer users the option of correctness if they want it" corollary.

So, check out that pypi'd package linked at the end of the discussion and see if it makes sense to integrate.

EDIT: cut to chase, link is http://shell-command.readthedocs.org/en/latest/ . Untouched since initial release, so now ~3 years old.

Strongly related here is the idea of exposing Popen's alternate list-of-parameters format for its initial argument, i.e. not this:

run("foo --the-bars plz")

but instead, this:

run(['foo', '--the-bars', 'plz'])

Some users find this easier than trying to handle/escape complex all-in-one command strings.

See also: Fabric #69

Problem

Need to allow parameterization of tasks as invoked via the CLI (tasks invoked via Python are simply passing regular args/kwargs or their list/dict counterparts.) Sub-problem: translating strings/flag existence into Python types such as lists, booleans etc.

Current implementation in Fabric is fab taskname:positional_arg_value,keyword_arg=kwarg_value,.... Some other tools use "regular" CLI flags, e.g. paver taskname -F value.

"Custom"/"attached" args

Pluses:

• Works great for multi-task invokes, zero ambiguity about which param values go to which tasks
• Supports positional arguments well

Minuses:

• Requires custom parsing, which may lead to bugs and at the very least increases code maintenance burden
• Slight documentation burden -- requires users to learn how that custom parsing works, re: need for escaping (esp. how that meshes with normal shell escapes) and etc.

"Regular" flag args

Pluses:

• Can (probably) just use argparse
• Many users already familiar with it

Minuses:

• Doesn't mesh well with multi-task invokes unless we:
  • enforce BSD-style "order matters" (not even sure argparse can do this? EDIT: perhaps with subcommands, if >1 subcommand can be invoked at a time)
  • or use an ugly "task delimiter" syntax like invoke task1 --arg1 --next task2 --arg2 (again, can argparse do this?)
  • or hope/pray that all simultaneously-invoked tasks have zero overlap in their kwargs and no positional args are ever needed.
    • How common is this? If we assume flags nix any possibility of positional args and that this is OK, it could be reasonable to assume two or three tasks called together would all have different names for their kwargs. Maybe.
• Namespace clashes with core (non task related) flags such as -c/--collection
  • This also requires order-matters, or simply limiting the user-allowed namespace
  • Limiting the namespace also presents backwards compatibility problems -- if version 1.1 lacks a --foo flag and a user declares mytask(foo='bar'), and then we unknowingly add --foo to core in 1.2, that user's code will break.
• May require more setup on the user side, re: explicitly declaring each task's args/kwargs via decorators or whatnot
  • Though we could introspect the function signature and automatically create some, if needed
• Doesn't allow for positional arguments unless we go the delimiter route.

Potential solutions

• Continue using Fabric's "custom parsing" solution pretty much as-is
  • See above re: pluses/minuses
• Try coming up with a "better" custom parsing solution/syntax, if there are any
  • What do other tools like Make, Rake, Thor etc use?
    • Thor uses regular flags + positional arguments, seems to simply ignore the "multiple tasks vs one task + positional args" problem, and uses "pass all flags to all tasks" to handle multi-task invocation.
      • Don't like this very much -- it "works" but has the "all tasks invoked must have different args" requirement mentioned above.
    • make uses shell environment variables, which is similar to how Thor treats flags -- every task is going to get the same values for any overlapping names.
    • Rake uses custom parsing like we do, but uses taskname[args, here] instead of taskname:args,here. I find this actually worse than the colon we use, because some shells like zsh treat square-brackets specially and need extra escaping/quoting.
• Allow custom parsing everywhere, but also allow use of flag args in single-task invocations
  • Plus: gives flag lovers what they want
  • Minus: "more than one way to do it" and its problems (newbie confusion, more code/bugs, more docs to write.)
• Use flags 100% of the time, relying on ordering or delimiting flags to handle multi-task invocations
  • See above re: pluses/minuses
  • Possibly use argparse subcommands, but AFAIK those are again targeted at using a single subcommand per invoke.

AKA def foo(blah_whatever): should be invocable as inv foo --blah-whatever. Probably controllable.

In the same way that one can rename tasks to have a different name (eg to work around builtin shadowing for function names), it should probably be possible to do the same for arguments.

This requires something like @task(arg_names={'realname': 'desiredname'}) or similar.

Ties in strongly with #34 re: adding extra names/aliases to kwargs. Maybe require some holistic "arg spec" option (though that then requires some duplication if you don't want to rename everything.)

Doesn't hurt the "be really concise" use case, but more intuitive for folks who want to be more explicit or who simply think "I wanna hide 'stdout'".

Keep getting tripped up by the "non-defaulted function args do not become valid flags" implication of #23. Even if/when #27 is implemented, I can see it tripping users up (if it trips up the author, it's gonna trip up the users!)

I am not 100% sure we can't have + eat our cake; we still cannot have arbitrary numbers of positional args, but within the "we know whether or not the next non-flag thing is a posarg or a task name" framework, it should be possible to "chip away" at positional args by giving them as flags.

For example, imagine this task:

@task(foo, bar, biz='baz'):
    pass

Currently, as of #23, the Context created for this task looks as follows:

• foo is a positional argument, and is in .args but not .flags
• Ditto for bar
• biz is a nonpositional argument and is in both .args and .flags

When parsing occurs, the parser knows it cannot skip to the next task boundary until both foo and bar have been filled in, by virtue of Context.needs_positional_arg. That's the only actual test.

Right now we explicitly test to make sure positionals are not available as flags (i.e. they're not added to the flags data structure and thus --a-positional-arg value is actually invalid). I do not remember why this is, but hopefully tweaking it and running the test suite will remind me.

Assuming it was just tripping up something dumb in lazy tests, I'll probably just remove that blockade and ensure that positional args given as flags are removed from the positional arg "queue". Thus, in the above example, if we parsed invoke mytask --foo=foovalue barvalue it would remove foo from the positionals list, then when it saw barvalue it would correctly associate it with bar.

Finally: this does add conceptual complexity to the parser, but I think it's worth it to avoid the more irritating pitfall of "I didn't give this arg a default value, why can't I give it as a flag now?"

E.g. the return value of @task should be the original function with a new attribute added to it, not a Task object. This preserves ability to call tasks from other tasks without jumping through hoops.

Alternately, implement __call__?

[Ed: was 'invoke doesn't run commands on Windows']

runner.py imports pty which (through tty -> termios) isn't available on Windows.

• Largely already implemented by use of Lexicon everywhere, but make sure it's well tested
• Should get its own decorator for >1-alias use: @aliases('foo', 'bar')
• Should remain available via @task for simpler use; possibly just singular? (@task(alias='foo'))