GithubHelp home page GithubHelp logo

devcmd's Introduction

DevCmd - Development Commands in Node.js and TypeScript

What is DevCmd?

  • Automation: Improve your development life, speed up recurring tasks, and reduce errors by automating your development tasks like building, running tests, or bumping versions.
  • Library: DevCmd gives you the tools to make automation simpler, such as easily running external programs (in series or in parallel) or composing smaller commands into more powerful ones.
  • Launcher: With the included devcmd launcher, you can easily start your commands with yarn devcmd or npx devcmd. Additionally, you can globally install devcmd-cli to use the launcher directly and from anywhere.
  • TypeScript & JavaScript: Benefit from the power of the npm ecosystem. Use the safety and abstraction of TypeScript where it helps you. Drop to plain JavaScript when you want to.

Usage

Getting Started

  • Install the devcmd package in your workspace:

    $ yarn add -D devcmd
    # - or -
    $ npm install -D devcmd
  • If you want to write commands in TypeScript, you also need typescript and ts-node. If you don't already have these installed, do so now:

    $ yarn add -D typescript ts-node
    # - or -
    $ npm install -D typescript ts-node
  • Create a directory named "dev_cmds" in your workspace (this name is required).

  • Create your commands in this "dev_cmds" directory.

    • The file name (without the extension) is the command name.

    • Each command is run as a standalone script, so top-level statements are permissible. When you want to use Promises (async/await), you need to take care of top-level Promise rejections as well.

    • For example, to create a "build" command, add a file "dev_cmds/build.ts" with the following content (or "dev_cmds/build.js" and drop the type definitions):

      // TypeScript
      import { execPiped, execPipedParallel, runAsyncMain } from "devcmd";
      
      export async function example() {
        console.log("Example command for single-package-json example");
      
        await execPiped({
          command: "node",
          args: ["-v"],
        });
      
        await execPipedParallel({
          nodeVersion: {
            command: "node",
            args: ["-v"],
          },
          npmVersion: {
            command: "npm",
            args: ["--version"],
          },
        });
      }
      
      runAsyncMain(example);
  • You can now run your commands with the locally installed devcmd CLI. We recommend using yarn or npx to invoke it:

    $ yarn devcmd <script name> [<args of you scripts>...]
    # - or -
    $ npx devcmd <script name> [<args of you scripts>...]
    • For example, to run the "build" command from above:

      $ yarn devcmd build
      # - or -
      $ npx devcmd build
  • If you don't want to type yarn/npx every time, you can globally install devcmd-cli (see below).

Using the Global Launcher: devcmd-cli

devcmd-cli is a command line utility to launch your dev commands that is intended to be installed globally:

$ yarn global add devcmd-cli
# - or -
$ npm install --global devcmd-cli

Once installed, running devcmd looks for the closest "dev_cmds" directory and starts command you entered with the locally installed devcmd package there. This way, you don't have to synchronize the versions of DevCmd you are using in different projects, and you can use the global launcher with a wide range of devcmd versions.

Going back to the example "build" command from above, you can now run it from the project directory or any subdirectories with:

$ devcmd build

Contribute

Setting up a local development setup

It is recommended to use the Remote-Containers extension for Visual Studio Code for this. Once the extension is installed, click Reopen in Container within the popup in the bottom right or open the command palette and run the command Remote-Containers: Reopen in Container.

Since both the devcmd and the devcmd-cli packages use local installations within node_modules folders, developing and especially testing devcmd locally can be challenging. In order to setup a local development environment, you can use the devcmd task setup-dev, which intializes a dev-folder, that contains the required devcmd-package as a yarn symlink.

$ yarn devcmd setup-dev
# - or -
$ npx devcmd setup-dev

You can now change the devcmd and the devcmd-cli packages. In order to test your changes, build the packages and run devcmd in the generated dev folder.

# Build all packages
$ yarn devcmd build-all
# - or -
$ npx devcmd build-all

# You can also build a package seperately
$ yarn workspace <package> build

# Run devcmd in the dev environment using the utility script in the package.json
$ yarn dev [PARAMS]
# - or -
$ npm run dev [PARAMS]

# Alternatively you can run devcmd from within the dev folder
$ cd dev
/dev $ yarn devcmd [PARAMS]
# - or -
/dev $ npx devcmd [PARAMS]

To properly cleanup the folder and the symlinks, use the devcmd task clean-dev.

$ yarn devcmd clean-dev
# - or -
$ npx devcmd clean-dev

Release Process

  • First of all you should ensure that everything is working. Check this with running the integration tests:
$ yarn devcmd run-integration-tests
# - or -
$ npx devcmd run-integration-tests
# - or with global launcher -
$ devcmd run-integration-tests
  • Depending on whether a change has been made to the devcmd-cli or devcmd, the steps to be carried out must be distinguished in the following.

Release of changes in devcmd-cli

  • Bump the version of the devcmd-cli. You should use the task bump-version and select devcmd-cli in the prompt:
$ yarn devcmd bump-version
# - or -
$ npx devcmd bump-version
# - or with global launcher -
$ devcmd bump-version
  • But this command also updates devcmd, so you need to update devcmd as well (see below).
  • Build the devcmd-cli package. You can use the task build-all:
$ yarn devcmd build-all
# - or -
$ npx devcmd build-all
# - or with global launcher -
$ devcmd build-all
  • Publish to npm.

Release of changes in devcmd

  • Bump the version of the devcmd-cli. You should use the task bump-version and select devcmd in the prompt:
$ yarn devcmd bump-version
# - or -
$ npx devcmd bump-version
# - or with global launcher -
$ devcmd bump-version
  • Build the devcmd package. You can use the task build-all:
$ yarn devcmd build-all
# - or -
$ npx devcmd build-all
# - or with global launcher -
$ devcmd build-all
  • Publish to npm.
  • Update the version of devcmd in the dev_cmds/package.json in this repository to use the new release.

devcmd's People

Contributors

corux avatar do-see avatar flo-planfox avatar lehnerpat avatar lenaxus avatar lieberlois avatar patricklehnerxi avatar schierla avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

devcmd's Issues

Renaming main branch from "master" to "main"

@lenaxus @lieberlois just wanted to let you know that I'm renaming the main branch of this repository from master to main.

This is a general trend in the Git community, and GitHub has changed the default branch name to "main" by now. You can read more about the background here: https://github.com/github/renaming

For you, this means mostly that you need to update your local clones to refer to the renamed remote branch, as described here in the GitHub docs. Feel free to rename your local branch accordingly :)

I'm closing this issue in a minute when I've done the rename, just wanted to have you notified. I you have any questions or problems, feel free to reply here or contact me directly.

Allow ignoring of error codes in execToString

I want to call docker inspect myContainer and use the returned JSON result.

This works fine.
When the container does not exist I get an empty array but also an errorcode != 0.

I would like to have an option "ignoreErrorCode" or "returnErrorCode". So that I can handle the error code on my own (or not handle it). Example can be seen here in Jenkins: https://www.jenkins.io/doc/pipeline/steps/workflow-durable-task-step/#sh-shell-script

I think it needs to be added here:

I would also prefer, if I can decide whether the output is printed to the console or not.

Add integration tests for passing arguments to commands

The issues found and fixed in #5 showed that passing arguments to commands isn't trivial, and especially may behave differently between OSs.

We should add some test coverage (at least in the integration tests, but maybe also unit tests?) to make sure that running commands with arguments does work, esp. the non-trivial cases (such as arguments containing spaces, containing quotes, containing backslashes).

Devcmd runner does not report exit code when underlying script fails

When a devcmd script fails with a non-zero exit code, the corresponding "yarn devcmd" still returns exit code 0.

Steps to reproduce:

  1. Create a devcmd test.ts containing process.exit(1)
  2. Run yarn devcmd test && echo This should not be shown if the command fails

Observed behavior:

This should not be shown if the command fails is written to the console.

Expected behavior:

No output, because the devcmd failed.

Likely cause

The return value of spwanSync (including the status code) is ignored in devcmd/from-cli

spawnSync(withCmdOnWin(launcher), [scriptFilepath, ...commandArgs], { stdio: "inherit" });

Bug in 0.0.10: ts-node is not found although available (yarn workspaces?)

Hi!

I wanted to upgrade vom 0.0.8 to 0.0.10 but I now get this error:
No script runner for TypeScript devcmds found. Did you forget to install ts-node?

We are using yarn workspaces. ts-node is available in the node_modules folder on workspace level.

If I execute npm ls ts-node (which the checkPackageAvailable does) I get

PS E:\Projects\folder> npm ls ts-node
pos@ E:\Projects\folder
`-- [email protected]  extraneous

npm ERR! extraneous: [email protected] E:\Projects\folder\node_modules\ts-node
PS E:\Projects\folder> 

Add a dev_cmd(s) to bump the package versions

So far, we've been versioning devcmd and devcmd-cli in lockstep because our early issues and changes always affected both packages.
In the future, devcmd-cli should ideally get less frequent releases than devcmd.

Bumping the package versions has some pitfalls, so we should add dev_cmd(s) to automate this process.

  • When bumping devcmd-cli's version, the dependency in devcmd must be updated accordingly.
  • When bumping devcmd's version, the dependencies in the example projects and in our own dev_cmds must be updated accordingly.
  • In both cases, an appropriate git tag must be created (deciding a tag naming pattern for non-lockstep versioning is part of this ticket)

"help" / "list" feature to see all devcmd tasks

There currently exists no "help" / "list" feature to see all devcmd tasks. To see all tasks, you have to navigate to the dev_cms folde.

The following approach would be desirable:

  • Type an command like this: yarn devcmd help
  • Get an helpful output with the task names and maybe a short description. This can look for example like this:
    image

Change working directory to project root instead of dev_cmds directory

The working directory seems to be set to the dev_cmds/ directory, not to the project root. This doesn't seem very useful, since I usually want to interact with other project files, and comparatively rarely with other devcmds.
In particular, this behavior is different from yarn/npm scripts and other task runners.

Note: This feedback was provided internally at XITASO, so I'm creating this issue for tracking.

Using yarn over npx results in much more verbose error logs

When using devcmd with invalid arguments with yarn, for example yarn devcmd, the error log looks something like this:

yarn run v1.22.10
$ cd dev_cmds && yarn devcmd
$ /home/luis/dev/devcmd/dev_cmds/node_modules/.bin/devcmd
devcmd v0.0.8
No script specified.
Command failed: node -e require('devcmd/from-cli').run(...process.argv.slice(1))
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

However, when using npx, the error log is much simpler and way easier to read, something like this:

devcmd v0.0.8
No script specified.

Write basic project and package README files

devcmd is approaching some basic usability now, so we should add some basic documentation to the README files.

This includes both the project README (shown on the repo front page) and the package READMEs for devcmd and devcmd-cli, which are shown on the npm repository pages.

Add documentation for public API

There currently exists basically no documentation of the API that devcmd exposes.

We should write some basic documentation of the publicly exposed functionality, at least in the code.

If possible, we should also generate easily-viewable pages from this code doc (e.g. markdown files or so), though we might split this into a separate issue.

Add documentation of release process

The release process for DevCmd isn't currently documented (since only I have been doing it until now).

We recently discussed the process (e.g. bumping package versions, running tests, releasing to NPM) internally in the XITASO team, but we should document it here in the repository as well.

Evaluate whether we can use a standard testing framework for integration tests

I know your PR isn't really the problem here, but maybe we should think about extracting this logging logic into a util function, since it is copied in almost every error case in each integration test 🤔 I can imagine something like this

function logStdoutFailure(actual: string, expected: string)

would reduce code duplication in this project. Maybe it would even be a good idea to extract line 78 to 88 entirely ^^

Originally posted by @lieberlois in #49 (comment)

on the one hand, I agree we shouldn't let the duplication here increase too much.
on the other hand, I'd also prefer not to build our own testing framework from scratch :D

I think with the exit code handling in #48 and the still-to-do "suppressNotices" option (that I explained here: #47 (comment)), we should re-evaluate whether we could use a regular testing framework (like Jest) for our integration tests instead.

Originally posted by @lehnerpat in #49 (comment)

DevCmd's exec* methods should be documented (in .d.ts)

The process execution functions (execPiped etc.) exposed by the devcmd package aren't currently publicly documented. This should be added to make it easier to understand their usage and behavior, esp. the differences between the various functions we have.

Note: This feedback was provided internally at XITASO, so I'm creating this issue for tracking.

Idea: offer to `yarn install` for dev_cmds if necessary when using devcmd launcher

I just set up devcmd for use in a new project, and a fellow developer (@davidpomerenke) suggested the following:
it would be nice if the devcmd launcher would suggest to directly do yarn install in the dev_cmds dir if necessary.

There are a lot of detailed questions about how to determine when to offer this, and where/how to run yarn install if we do, but from a (first-time) user's point of view, I found this a valid feature request.

Use a different script runner than ts-node for better performance

Would it be possible to use a different script runner than ts-node? Especially for short build scripts, ts-node is quite slow -- and I don't need to type-check my build scripts all the time. In our TS code base, we're using babel-node as our runner, which only transpiles and is therefore significantly faster.
Additionally, we recently found out that ts-node can be configured to use swc for transpilation. If this works for devcmd, this issue might be solvable by configuration.

Note: This feedback was provided internally at XITASO, so I'm creating this issue for tracking.

Improve examples about usage of runAsyncMain

Some of the examples don't currently use the runAsyncMain function, while others do. This was a bit confusing at first, and doesn't clearly show how DevCmd is intended to be used.

Note: This feedback was provided internally at XITASO, so I'm creating this issue for tracking.

Namespacing in command_names

At the moment it is only possible to run commands in the self-created "dev_cmds" directory.

When running a task with devmd it is only searching in the "dev_cmds"-directory, not in subdirectories. It could therefore be possible that with many self-created tasks it is difficult to structure the tasks.

This issue should adress this problem with the expansion of the namespace of the commands. The easiest way should be this suggestion:

  • It should be possible to save commands in subdirectories. With this approach every user can structure his commands and gets the overview.
  • When running a task with devcmd the script-name is no longer a single file-name. It is a path to the file starting in the "dev_cmds"-directory. This path can be build with slashes or colons. So for example it can look like this when I am trying to run the task foo in a subdirectory tutorial: 'yarn devcmd tutorial/foo'.

Clarify how DevCmd should augment PATH when multiple node_modules exist in the directory hierarchy

This issue relates to the augmentation of the PATH environment variable to allow easy use of bin scripts provided by npm packages, i.e. for use inside dev commands.
It does not affect the resolution of the right dev_cmds directory.

I had previously assumed that npm, npx, and yarn would search for bin scripts in the .bin directories of all node_modules directories up the path hierarchy. This is not the case.
Instead, only the node_modules/.bin directory next to the current package.json file (i.e. the closest package.json going up the path hierarchy) is considered. (Or some more, if the CWD is inside nested node_modules directories, but that's not very relevant for DevCmd).

This is most clearly recognizable (though still not obvious IMO) in the npm-path package, which has extracted npm's lookup/path augmentation algorithm. See getPath in timoxley/npm-path#src/index.js

We should keep this behavior in mind, because it differs significantly to how plain node modules (import/require) are resolved.
And we should consider well whether we want to stick to this behavior or make custom additions to $PATH.
And either way, document the behavior.


To illustrate, consider the following setup (which is a situation we want to support according to our design goals):

|
+-- dev_cmds/
| +-- node_modules/
| | +-- .bin/
| | \-- ...
| +-- build.ts
| \-- package.json   <-- (A)
+-- node_modules/
| +-- .bin/
|   \-- ...
+-- ...
\-- package.json     <-- (B)

Let's talk about running a dev command, i.e. starting a script in dev_cmds/ with the CWD also set to dev_cmds/.

The npm PATH augmentation (as implemented by npm and npm-path, and implemented sufficiently similarly by yarn, when running scripts) produces an augmented NPM_PATH as follows, where $PATH is the non-augmented (i.e. OS-provided) PATH:

  • If the inner package.json (A) exists, we get roughly export NPM_PATH = $(realpath dev_cmds/node_modules/.bin):$PATH. The outer bin dir (node_modules/.bin/) is not on the augmented PATH.

  • If the inner package.json (A) does not exist, we get roughly export NPM_PATH = $(realpath node_modules/.bin):$PATH. The inner bin dir (dev_cmd/node_modules/.bin/) is not on the augmented PATH.

This has the following effects:

  • If (A) does not exist and all dependencies (devcmd, ts-node, etc.) are included in (B), everything works fine.
  • If (A) does exist, and all all dependencies (devcmd, ts-node, etc.) are included in (A), anything available in (A) works fine (e.g. running other dev commands). Bin-dependencies included only in (B) cannot be directly executed.
  • If (A) does exist and includes the devcmd dependency, but the ts-node dependency is included only in (B) (e.g. because the project itself uses it), dev commands cannot be run. (At the moment, it fails silently, see #17, but even if an error is emitted, this failure seems unexpected to me.)

Yarn 3 script output not showing

Hi Xitasonians,

Nice to see that devcmd is used more and more!

Don't know if it is actually an issue with devcmd. I noticed after upgrading yarn that the output from the scripts is not shown in the console anymore. Running the example command from the README, the only output I get is:

❯ yarn devcmd build
devcmd v0.0.8: cmd "build"

Did someone came across this yet? I hope I did not miss something super obvious...

Best
Chris

Versions:

❯ yarn --version
3.1.0
❯ node --version
v14.17.6
❯ npm --version
6.14.15

Provide shell command completion support (for Bash)

Most shells offer tab completion to automatically complete commands and arguments based on what the user has entered on the prompt so far.

These completions usually automatically include things like all executables on the PATH, or local files and directories.
Additionally, they can usually be extended by users and third-party programs, e.g. to add (more or less intelligent) completion of sub-commands and command-line options of programs. A popular example is Git, which offers brings tab-completion for the Bash shell, with which you can complete Git sub-commands, branch names, etc.

It would be a great convenience feature for DevCmd to offer completion integration with popular shells, so that the locally available devcmd tasks are automatically completed.

The integration is usually shell-specific and therefore needs to be done individually; I suggest that we start with Bash, since this is still a very popular shell.

The tab-completion for devcmd can probably make good use of the "list" option (#30), or a variant for machine consumption.

Improve detection and reporting of missing TS launcher (ts-node)

For script launchers that aren't node itself (currently only ts-node for TS commands), we use npm ls to look up whether the launcher package is installed/available, and notify the user about its absence otherwise.

However, #55 revealed that this has two problems:

  1. npm ls returns non-zero not only when the package is missing, but also when it is "extraneous" (present in node_modules but not in package.json).
  2. Apparently it is also noticably slow in certain situations (probably depending on size of node_modules and disk performance)

For problem 1., we should consider whether we want to keep this behavior -- having extraneous packages is an inconsistent state, hence npm ls showing it as an error, but technically we could still try to execute our command.
If we keep it, we should improve the output because we're currently always reporting the "missing" case even if the package is extraneously present. If we want to allow extraneously present ts-node for DevCmd, we should use a different method of detection.

Taking problem 2. into account, we should also consider whether we always want to check for ts-node presence pre-emptively, or if it's enough to check it when spawning it fails.


For reference, the code using npm ls is:

export async function checkPackageAvailable(packageName: string, directory: string): Promise<boolean> {
return new Promise<boolean>((res) => {
// `npm ls <package_name>` only exits with status code 0 when <package_name> is available
const childProcess = spawn(withCmdOnWin("npm"), ["ls", packageName], { cwd: directory });
childProcess.on("error", (): void => res(false));
childProcess.on("close", (code: number): void => res(code === 0));
});
}

And it is invoked from here (line 144):

async function tryRunScript(
devCmdsDir: string,
scriptFilepath: string,
commandArgs: string[],
extension: string,
launcher: string
): Promise<boolean> {
if (await isFile(scriptFilepath)) {
if (extension === "ts" && !(await checkPackageAvailable(launcher, devCmdsDir))) {
throw new Error(`No script runner for TypeScript devcmds found. Did you forget to install ${bold(launcher)}?`);
}
// TODO: use spawn or so instead
const { status } = spawnSync(withCmdOnWin(launcher), [scriptFilepath, ...commandArgs], {
stdio: "inherit",
cwd: devCmdsDir,
});
if (status !== null && status != 0) throw new Error(`Process failed with exit code ${status}`);
return true;
}
return false;
}

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.