arnikz / xenon-cli

Perform files and jobs operations with Xenon library from command line

Xenon Command Line Interface

Command line interface which uses the Xenon library to perform job and file operations.

Install

Goto https://github.com/NLeSC/xenon-cli/releases and download a tarball (or zipfile). The tarball can be installed with:

tar -xf build/distributions/xenon*.tar
xenon*/bin/xenon --help

Add xenon*/bin to your PATH environment variable for easy usage.

Usage

# List files on local filesystem
xenon filesystem file list /etc
# List files on remote filesystem using sftp
xenon filesystem sftp --location localhost list /etc
# Copy local file to remote filesystem
xenon filesystem sftp --location localhost upload /etc/passwd /tmp/copy-of-passwd
# Execute a program remotely using ssh
xenon scheduler ssh --location localhost exec /bin/hostname
# Pipe to a remote file
echo "sleep 30;echo Hello" | xenon sftp --location localhost upload - /tmp/myjob.sh
# Submit to a remote Slurm batch scheduler
xenon scheduler slurm --location ssh://localhost submit /bin/sh /tmp/myjob.sh

The above commands use your current username and keys from ~/.ssh.

To keep password or passphrase invisible in process list put the password in a text file (eg. 'password.txt') and then use '@password.txt' as argument. For example:

xenon filesystem sftp --location localhost --username $USER --password @password.txt list $PWD/src

Build

./gradlew build

Generates application tar/zip in build/distributions/ directory.

Tests

Requirements for the integration tests:

• docker, v1.13 or greater
• docker-compose, v1.10 or greater

The unit and integration tests can be run with:

./gradlew check

Release

1. Bump version in build.gradle, add version to CHANGELOG.md and commit/push
2. Create a new GitHub release
3. Upload the files in build/distributions/ directory to that release
4. Publish release
5. Edit Zenodo entry, correct license, add Xenon doi as is referenced by this upload.

Docker

Run Xenon CLI as a Docker container.

The Docker image can be build with

./gradlew docker

Generates a nlesc/xenon-cli Docker image.

To use local files use volume mounting (watch out as the path should be relative to mount point):

docker run -ti --rm nlesc/xenon-cli --user $USER -v $PWD:/work --adaptor ssh upload --source /work/somefile.txt --location localhost --path /tmp/copy-of-somefile.txt 

Common Workflow Language

Run Xenon CLI using a cwl-runner or as a tool in a Common Workflow Language workflow.

Requires nlesc/xenon-cli Docker image to be available locally.

Example to list contents of /etc directory via a ssh to localhost connection with cwl-runner:

./xenon-ls.cwl --adaptor sftp --location [email protected] --certfile ~/.ssh/id_rsa --path /etc
# Copy file from localhost to working directory inside Docker container
./xenon-upload.cwl --adaptor sftp --certfile ~/.ssh/id_rsa --location [email protected] --source $PWD/README.md --target /tmp/copy-of-README.md
# Copy file inside Docker container to localhost
./xenon-download.cwl --adaptor sftp --certfile ~/.ssh/id_rsa --location [email protected] --source /etc/passwd --target $PWD/copy-of-passwd

(Replace <user>@<host> with actual username and hostname + expects docker with default network range)