Taskcluster CLI
License: Mozilla Public License 2.0
TaskCluster CLI is a command-line client offering control and access to taskcluster from the comfort of your command-line. It provides utilities ranging from direct calls to the specific API endpoints to more complex and practical tasks like listing and cancelling scheduled runs.
This repository has been merged into the Taskcluster repo; see clients/client-shell.
In summary, this will add taskcluster version command. It can be just a template with a fake command version until we decide how we store and update the version.
It's quite possible that users will write scripts that start a task, then wait for it to complete. Something like:
taskcluster api queue createTask $TASKID <task.json
taskcluster task await-completion $TASKID
This need came out last Friday, when releng had several releases in flight and all of them were claiming the same workers. One way to solve this issue is to handle priorities in a different manner. Nonetheless, having a manual way to cancel/rerun a certain type of tasks, would be great to manually help a graph to claim workers.
For example:
taskcluster group cancel --worker-type 'signing-worker-v1' $TASK_GROUP_IDwill:
rerun may also follow the same workflow.
What do you guys think?
As of now, when we need to add a command, we create an implementation of the CommandProvider interface, and register it. We have already added a few commands using this:
taskcluster helptaskcluster from-now [...]However, there are commands that have their own subcommands. taskcluster api [...] is an example of this. Another example:
taskcluster slugid v4taskcluster slugid nicetaskcluster slugid decode <slug>taskcluster slugid encode <uuid>In this case, the best option has been to implement all four subcommands manually as part of the slugid command. It would be preferable to be able to have one implementation of CommandProvider for each subcommand and ideally be able to register it under the slugid command.
It seems docopt is able to parse subcommands, so this is good.
This will be pretty much essential to all the taskcluster api subcommands.
In taskcluster we use scopes for authorization, see:
https://docs.taskcluster.net/presentations/scopes/
(formal definition and examples)
The taskcluster-auth service maintains roles... a role is effectively just a mapping from a scope on the form assume:... to a set of scopes that the role grants.
Example:
assume:project:taskcluster:tutorial grants scopes listed here:
https://tools.taskcluster.net/auth/roles/#project:taskcluster:tutorial
A command like this:
$ taskcluster expand-scopes assume:project:taskcluster:tutorial
assume:project:taskcluster:tutorial
queue:create-task:aws-provisioner-v1/tutorial
secrets:get:garbage/*
secrets:set:garbage/*Would be pretty nice to have. The API has a method for expanding scopes here:
https://docs.taskcluster.net/reference/platform/auth/api-docs#expandScopes
So this is really just parsing arguments, call API and print the result line by line.
Get rid of the timestamp error that shows up in Task Creator for an out-of-date task, once the timestamps have been updated. The warning remains even though the task is valid to be re-run.
Pulling this out of #92, there's a few things that can be added to taskcluster.yml at some point.
gometalinter things:
--enable=varcheck --enable=aligncheck --enable=errcheck --enable=ineffassign --enable=unconvert -enable=goimports --enable=unused
go fmt
Now we have .taskcluster.yml going (see #49), we want to be able to run some of the extra things we had for travis:
When I try to watch or fetch the logs for a task that was just created (pending, unscheduled, or otherwise), I'm getting an ugly error message:
{
"code": "ResourceNotFound",
"message": "Artifact not found\n----\nmethod: getLatestArtifact\nerrorCode: ResourceNotFound\nstatusCode: 404\ntime: 2017-03-10T15:46:56.672Z",
"requestInfo": {
"method": "getLatestArtifact",
"params": {
"0": "public/logs/live.log",
"taskId": "8aATqJZQSfihCp0cI0axdA",
"name": "public/logs/live.log"
},
"payload": {},
"time": "2017-03-10T15:46:56.672Z"
}
}
Error: Received unexpected response code 404
The task log should catch that error (logs don't exists), and I think that it should first check the status of the task, so that it can display better error messages if the task doesn't exist or the logs haven't been created yet.
When we create a new Github release of taskcluster-cli, tc-github should automatically build and upload binaries for the various platforms we support, and make them available for users.
Taskcluster uses Hawk for authentication under the hood. The credentials module provides an interface for taskcluster authentication. Unfortunately, it doesn't have any unit test case. Implementing a test case for follows these steps:
Credentials object with ClientID = tester and AccessToken = no-secretSignRequestWe have a fancy one-click-loaner option that is accessible via https://tools.taskcluster.net, but the terminal emulation is TERRIBLE and you can't copy/paste or do any of the normal terminal things.
Much better would be to connect my existing terminal to a one-click loaner:
dustin@lamport ~ $ taskcluster terminal $TASKID
# <--- shell prompt from loaner
This should support both websocket protocols
taskcluster task-log <taskId>
should output the logs for the given taskId as they are generated, and exit when the task completes.
A previous attempt at a command line client in javascript exists (see gregarndt/taskcluster-cli).
This client should be able to replace that javascript implementation.
taskcluster task create to replace taskcluster run: creates and schedules a tasktaskcluster task run to replace taskcluster run-task: creates, schedules, and watches a task (streaming logs from public/logs/live.log by default)taskcluster task await to wait the completion of a given task (by taskId)There was no specific issue for this. I'm currently on it.
See branch test-apis for details and current progress.
The Python, Node, and Go TaskCluster clients have certain conventions for getting credentials. In theory they are all the same conventions, but that should be verified.
Then, those conventions should be written down in the docs.
Then, ensure that taskcluster uses the same conventions, and that simply running taskcluster signin will set credentials that all of these things can use.
We have this neat thing where, if running in a task, you can connect to e.g., http://taskcluster/queue/v1/task/<taskId> This request is proxied to the queue (in this case) with credentials matching the task's scopes attached.
taskcluster-cli should support using this proxy when available.
An open question is, how to tell when the proxy is available, and how to calculate the appropriate URL.
Using auth.awsS3Credentials takes taskcluster credentials and returns temporary STS credentials for S3.
It would be cool if we had a command like taskcluster s3 copy ./myfile.txt s3://<bucket>/<prefix>
that could get temp creds from auth and use those to upload a file to S3.
Obivously, it would have to do retries with exponential back-off, ideally also multi-part upload for large files.
Detect mimetype based on file extension... Provide content-md5 for S3 to verify.
See: https://docs.taskcluster.net/reference/platform/auth/api-docs#awsS3Credentials
Further steps could include:
x-amz-meta-content-sha256 for integrity checks on downloadsIt's not glamorous, but most of taskcluster-cli is currently un-tested, and we need to fix that. So we need to write tests for all (or at least most) of the existing code!
This will probably involve refactoring some of that code to make it more testable, in a go-friendly way.
An artifact is a result of of task. It can be a binary file, a test report, a log, etc. We need to implement a command to download an artifact:
taskcluster download <taskId> <runId> <artifact>
taskcluster download <taskId> <artifact>
taskcluster download --index <indexNamespace> <artifact>When <runId> is omitted, it should download artifacts from latest runId.
Please check https://godoc.org/github.com/taskcluster/taskcluster-client-go/queue#Queue.GetArtifact_SignedURL
If is omitted, it should list all artifacts.
We have taskcluster-lib-docs for JS services, which can upload a tarball containing documentation. Let's build a similar thing for Go (but omitting the API references and exchanges, since we don't implement TaskCluster HTTP APIs in Go), so that we can upload documentation about all of the tc-cli options to taskcluster-docs, too.
This can't work because here we are generating URL auth, and URL Auth doesn't set a nonce (since bewits are intentionally not protected from replay attack, hence why they should only be used for GET requests).
Instead, we need to use Request auth for signing a request, not URL auth... :-)
Otherwise, it would probably go unnoticed (client may operate without problems) but we'd have a gaping security hole...
dustin@lamport ~/go/src/github.com/taskcluster/taskcluster-cli [master] $ ./taskcluster signin
Starting
Listening for a callback on: http://localhost:37706
..and it hangs there. What do I do? Some better documentation for the user would be helpful :)
./taskcluster auth --help
currently shows the description of the auth API (the top of https://docs.taskcluster.net/reference/platform/auth/references/api) However,
./taskcluster auth listClients --help
does not show the listClients description. It should! It should also show the payload and response schemas.
Users will want to be able to hit tab to complete a command, or to see a list of command options if there is more than one available.
We want to enable gometalinter's misspell check in .taskcluster.yml. The warnings we're getting just need to be fixed, and then this can be enabled again.
taskcluster-cli from-nowExample:
taskcluster-cli from-now '2 days'Travis is a Continuous Integration system aimed for small and medium size projects. taskcluster-ci doesn't run any tests or linter on pushes and pull request, which is sad. We would like to use travis to run:
go test -v -racego fmt -s -w and then check if we have uncommited changesgometalinter --disable-all --enable=gotype --enable=golint --enable=deadcode --enable=staticcheck --enable=deadcode --enable=misspell --enable=vet --enable=vetshadow --enable=gosimple --enable=varcheck --enable=aligncheck --enable=errcheck --enable=ineffassign --enable=unconvert --enable=staticcheck --enable=goimports --enable=unusedYou can see how we run go test and go fmt in taskcluster-worker.
This should run on Linux, MacOSX and Windows. Ideally we should run under go 1.7, but Travis doesn't support 1.7 version, so it is ok to use 1.6 for now. But if a keen to make 1.7 working, you can see how we do it in taskcluster-worker and do something like that.
You can start fixing errors reported by gometalinter and then add travis.
Currently, all errors message go to os.Stdout. We should use fmt.Fprintf to redirect them to os.Stderr.
It would be nice for users to always have the latest and greatest taskcluster client. We could do this a few ways:
taskcluster api would always be up-to-date, but the other subcommands would only get updates when the user manually downloads a new version.There are some serious security concerns with options 1 and 3, of course!
In #57 we have a plan to implement file-transfer by running file-browser-go and talking to it over stdin/stdout through the websocket protocols implemented in #54.
A cool addition would be the ability to mount the entire root file-system of the task container locally using FUSE. There is a golang library here https://godoc.org/bazil.org/fuse.
I'm not sure how easy/hard it would be, also I doubt we need to support all FUSE features, like file locking and what not. But just supporting the platform independent parts would be cool.
This would obviously be a command that only work on linux... but, as always we could look for other virtual file-system implementations to support Windows or OS X. It looks like there already is a Windows thing for golang https://github.com/dokan-dev/dokany, though it might not be as easy to build.
@ckousik, this is another use-case of the file-browser-go, perhaps this is a more efficient way of doing things compared to re-implementing rsync. As this way we would only transfer things as we read/write them.
Doing the signin subcommand works fine and generates the configuration file in .config/taskcluster.yml.
However, running any api command after that fails with the following error message:
Failed to sign request, error: Failed to parse certificate, error: json: Unmarshal(nil *client.certificate)
We have a basic .travis.yml set up in #46, but we should be using taskcluster via taskcluster-github instead of travis. And doing releases (binary builds) when we push a new tag.
This would be an extension of #54 to also support transferring files to/from the loaner.
There is no scp or anything like that on the loaner, so this will require figuring out how to handle this on the loaner side. Maybe Perhaps use: https://github.com/taskcluster/file-browser-go (author is planning to have included in docker-worker, and writing a JS API for using it)
In taskcluster we use scopes for authorization, see:
https://docs.taskcluster.net/presentations/scopes/
(formal definition and examples)
It would nice if we had a command like:
$ taskcluster scope-check scopeA scopeB... satisfies scopeC scopeD...Which will then determine if scopes scopeA, scopeB, ... satisfies scopeC, scopeD, ..., the argument satisfies would be a keyword in this command.
Example:s
$ taskcluster scope-check queue:ping satisfies queue:ping # exit code 0
YES
$ taskcluster scope-check queue:ping satisfies queue:test # exit code 1
NO, missing:
- queue:test
$ taskcluster scope-check 'queue:*' test:scope other-scope satisfies queue:test # exit code 0
YES
$ taskcluster scope-check 'queue:*' satisfies other-scope # exit code 1
NO, missing:
- queue:*
$ taskcluster scope-check 'queue:*' satisfies 'queue:create-ta*' # exit code 0
YES Maybe the output messages could be formatted better...
Note: Step one is to just do simple string comparison. Next step would then be to take the left-hand-side scopes and expand them using:
https://docs.taskcluster.net/reference/platform/auth/api-docs#expandScopes
Any scope on the form assume:... may be expanded. On the right handside expansion is unnecessary since if assume:... is satisfied, then any expansion on the left-hand-side would also include scopes assume:... expanded to.
While going over config/serialization I realized that there might be issues when some options are defined both as environment variables and as config file values.
I assume that when an option is tied to an environment variable, we will need to edit that environment variable to change the value of that option. But what happens if I do taskcluster config set OPTION VALUE? Which value then takes precedence?
We can delete these lines, when shibukawa/configdir#3 is merged upstream.
Perform tests(/write a script) to make sure ./taskcluster is behaving properly (breakout of #47)
This command would call (maybe in parallel?) the ping endpoint for every service that has one, printing an "OK!" line for each service for which it returns successfully.
Users would find this helpful when something goes wrong and they wonder, as they always do, "is taskcluster down?" There's also https://status.taskcluster.net that shows about the same data.
We are using github.com/shibukawa/configdir in the status command, for locating the cache directory, but could also use it for locating the config directory in ´config/serialization.go´.
For command-line completion (#64), we need information on the available command-line options.
Docopt, as-written, doesn't support that: its Parse method just interprets the usage strings, then returns the result of parsing the command line. It doesn't give any information from its interpretation of the usage strings.
Options:
All or most of our API services have a status end-point...
Based on JSON from: https://github.com/taskcluster/taskcluster-cli/tree/master/apis/definitions
We should write a command that uses this JSON to for each service print:
service status
queue up
auth up
...
By calling the status endpoint to verify that the services are up and running.
The service definitions are fetched from the schema server and turned into go code through go generate .
Currently this is a manual process, with the results checked in when we remember to do so. We have a few ideas regarding how to make this better.
The first idea is to have a scheduled task in taskcluster to run go generate and generate a PR if there are changes.
The other option is to run go generate only as part of the release process, the generated binaries would have the latest definitions, but the repository would have old versions checked-in.
The help currently looks like this:
auth Operate on the auth service
authEvents Operate on the authEvents service
awsProvisioner Operate on the awsProvisioner service
awsProvisionerEvents Operate on the awsProvisionerEvents service
config Get/set taskcluster CLI configuration options
github Operate on the github service
githubEvents Operate on the githubEvents service
help Prints help for a command.
hooks Operate on the hooks service
index Operate on the index service
login Operate on the login service
notify Operate on the notify service
pulse Operate on the pulse service
purgeCache Operate on the purgeCache service
purgeCacheEvents Operate on the purgeCacheEvents service
queue Operate on the queue service
queueEvents Operate on the queueEvents service
scheduler Operate on the scheduler service
schedulerEvents Operate on the schedulerEvents service
secrets Operate on the secrets service
signin Sign-in to get temporary credentials
treeherderEvents Operate on the treeherderEvents service
version Prints the TaskCluster version.
and that makes it hard to spot the few options that aren't about a service. I'd prefer
api Perform TaskCluster API operations
config Get/set taskcluster CLI configuration options
help Prints help for a command.
secrets Operate on the secrets service
signin Sign-in to get temporary credentials
version Prints the TaskCluster version.
Users will want to know the best way to do some common things with this tool:
Some really clear, well-thought-out documentation guiding them through this process would be great, especially if added to the tutorial at https://docs.taskcluster.net.
I'm not 100% sure how to do this.. There is probably some docs to read :)
But it is possible to do auto-completion for a custom command, I suspect that we can auto-generate such auto-completion logic from the docopt strings and generate a command that can install the auto-complete script in /etc/bash_completion.d/taskcluster that would be pretty cool.
A command taskcluster pulse-publish [--format yaml|json] [--message <message>] <exchange> <routingkey>
That if no --message is given opens a prompt that you can enter the message into...
Also it should declare the exchange if it doesn't exist.
Read more about pulse here: https://wiki.mozilla.org/Auto-tools/Projects/Pulse
Similar to #87 we can store the username/password in the config system for taskcluster-cli... These should also default to PULSE_USERNAME and PULSE_PASSWORD env vars, if not config values are entered...
Notice: this should primarily be based on the stuff in config/...
Ideally, we can reuse the same config options so you don't have to set the password twice...
Maybe for this to work it as to be taskcluster pulse with two subcommands publish and listen, either way let's start by implementing one of these.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.