GithubHelp home page GithubHelp logo

guybedford / chomp Goto Github PK

View Code? Open in Web Editor NEW
136.0 6.0 7.0 14.45 MB

'JS Make' - parallel task runner for the frontend ecosystem with a JS extension system.

Home Page: https://chompbuild.com

License: Apache License 2.0

Rust 99.48% JavaScript 0.52%

chomp's Introduction

CHOMP

Crates.io Discord

Chomp is a frontend task runner with advanced features focused on ease-of-use and not getting in the way!

  1. An advanced task runner with a single command!
  2. Easily adapt existing projects / task systems - no need for a rewrite.
  3. You enable and manage advanced task runner features with single-line updates.

Chomp is a great option for frontend projects where the goal is getting advanced task runner features (like smart caching) without complexity and overhead.

One-line migration from npm scripts

Chomp can import a project's established package.json scripts without breaking them, as it supports the same features:

chomp --init --import-scripts

Now you can run your npm scripts using Chomp!

i.e npm run <task> becomes chomp <task> and behaves the same, and you can opt in to further features as needed.

The only difference is, with Chomp — it's faster. And, with a few more tweaks, you can enable smart caching, parallelism, and more!

What features does Chomp provide?

Chomp is an advanced task runner. It provides features similar to turbo and nx but focuses on ease of use, *not monorepos. It's based on the same principles as traditional make files.

Parallelism

Chomp runs tasks in parallel, based on an extecuted task's dependencies!

Watch/Serve

Chomp watches any task by including a --watch or --serve option! Read more about the power of --watch and --serve.

A JS extension system

Chomp has a JS extension system that allows you to extend Chomp with your own custom tasks

Smart caching

Chomp caches tasks based on task dependencies like other tasks or updated files. You don't have to worry about it!

*Chomp works for monrepos but it's architected for ease of use and not getting in the way first.

Install

If you use Cargo, run:

cargo install chompbuild

If you don't use Cargo, run:

npm install -g chomp

Note: npm scripts add over 100ms to the script run time.

Common platform binaries are also available for all releases.

To quickly set up Chomp in a GitHub Actions CI workflow, see the Chomp GitHub Action.

Documentation

Getting Started

Migrating from npm Scripts

To convert an existing project using npm "scripts" to Chomp, run:

$ chomp --init --import-scripts
√ chompfile.toml created with 2 package.json script tasks imported.

or the shorter version:

$ chomp -Ii
√ chompfile.toml created with 2 package.json script tasks imported.

Then use chomp <name> instead of npm run <name>, and enjoy the new features of task dependence, incremental builds, and parallelism!

Hello World

chomp works against a chompfile.toml TOML configuration in the same directory as the chomp command is run.

Chomp builds up tasks as trees of files which depend on other files, then runs those tasks with maximum parallelism.

For example, here's a task called hello which builds hello.txt based on the contents of name.txt, which itself is built by another command:

chompfile.toml

version = 0.1

[[task]]
target = 'name.txt'
run = '''
  echo "No name.txt, writing one."
  echo "World" > name.txt
'''

[[task]]
name = 'hello'
target = 'hello.txt'
dep = 'name.txt'
run = '''
  echo "Hello $(cat name.txt)" > hello.txt
'''

with this file saved, the hello command will run all dependency commands before executing its own command:

$ chomp hello

🞂 name.txt
No name.txt, writing one.
√ name.txt [4.4739ms]
🞂 hello.txt
√ hello.txt [5.8352ms]

$ cat hello.txt
Hello World

Finally it populates the hello.txt file with the combined output.

Subsequent runs use the mtime of the target files to determine what needs to be rerun.

Rerunning the hello command will see that the hello.txt target is defined, and that the name.txt dependency didn't change, so it will skip running the command again:

chomp hello

● name.txt [cached]
● hello.txt [cached]

Changing the contents of name.txt will then invalidate the hello.txt target only, not rerunning the name.txt command:

$ echo "Chomp" > name.txt
$ chomp hello

● name.txt [cached]
  hello.txt invalidated by name.txt
🞂 hello.txt
√ hello.txt [5.7243ms]

$ cat hello.txt
Hello Chomp

The deps array can be defined for targets, whose targets will then be run first with invalidation based on target / deps mtime comparisons per the standard Makefile approach.

Powershell is used on Windows, while Bash is used on POSIX systems. Since both echo and > are defined on both systems, the examples above work cross-platform (Powershell is automatically put into UTF-8 mode for > to work similarly).

Note that && and || are not supported in Powershell, so multiline scripts and ; are preferred instead.

JS Tasks

Alternatively we can use engine = 'node' or engine = 'deno' to write JavaScript in the run function instead:

chompfile.toml

version = 0.1

[[task]]
target = 'name.txt'
engine = 'node'
run = '''
  import { writeFile } from 'fs/promises';
  console.log("No name.txt, writing one.");
  await writeFile(process.env.TARGET, 'World');
'''

[[task]]
name = 'hello'
target = 'hello.txt'
deps = ['name.txt']
engine = 'node'
run = '''
  import { readFile, writeFile } from 'fs/promises';
  const name = (await readFile(process.env.DEP, 'utf8')).trim();
  await writeFile(process.env.TARGET, `Hello ${name}`);
'''

Tasks are run with maximum parallelism as permitted by the task graph, which can be controlled via the -j flag to limit the number of simultaneous executions.

Using the --watch flag watches all dependencies and applies incremental rebuilds over invalidations only.

Or, using chomp hello --serve runs a static file server with watched rebuilds.

See the task documentation for further details.

Monorepos

There is no first-class monorepo support in chomp, but some simple techniques can achieve the same result.

For example, consider a monorepo where packages/[pkgname]/chompfile.toml defines per-package tasks.

A base-level chompfile.toml could run the test task of all the sub-packages with the following chompfile.toml:

[[task]]
name = 'test'
dep = 'packages/#/chompfile.toml'
run = 'chomp -c $DEP test'

chomp test will then use task interpolation to run the multiple sub-package test tasks in parallel. A similar approach can also be used for a basic unit testing.

By adding serial = 'true', the interpolation can be made to run in series rather than in parallel.

Cross-project dependencies are not currently supported. Instead, if packages/a/chompfile.toml's build task depends on packages/b/chompfile.toml's build task to run first, then packages/a/chompfile.toml might look like:

[[task]]
name = 'build'
run = 'cargo build'
dep = 'build:deps'

[[task]]
name = 'build:deps'
run = 'chomp -c ../a build'

This would still be fast, so long as packages/a/chompfile.toml's build task has its targets and dependencies properly configured to do zero work if the all target mtimes are greater than their dependencies.

Extensions

Extensions are able to register task templates for use in Chompfiles.

Extensions are loaded using the extensions list, which can be any local or remote JS file:

version = 0.1
extensions = [
  "./local.js",
  "https://remote.com/extension.js"
]

A core extensions library is provided with useful templates for the JS ecosystem, with the short protocol chomp:ext, a shorthand for the @chompbuild/extensions package contents.

A simple example is included below.

See the @chompbuild/extensions package for extension descriptions and examples.

Example: TypeScript with SWC

To compile TypeScript with the SWC template:

version = 0.1
extensions = ['[email protected]:swc']

[[task]]
name = 'build:typescript'
template = 'swc'
target = 'lib/##.js'
deps = ['src/##.ts']

In the above, all src/**/*.ts files will be globbed, have SWC run on them, and output into lib/[file].js along with their source maps.

The ## and # interpolation syntax is special because, unlike glob dependencies (which are also supported), there must be a 1:1 relationship between a dependency and its target.

Only non-existent files, or files whose src mtimes are invalidated will be rebuilt. If SWC itself is updated, all files that depend on it will be re-built.

Specific files or patterns can be built directly by name as well, skipping all other build work:

chomp lib/main.js lib/dep.js

🞂 lib/dep.js
🞂 lib/app.js
√ lib/dep.js [317.2838ms]
√ lib/app.js [310.0831ms]

Patterns are also supported for building tasks by name or filename (the below two commands are equivalent):

$ chomp lib/*.js
$ chomp :build:*

To remove the template magic, run chomp --eject to convert the chompfile.toml into its untemplated form:

$ chomp --eject

√ chompfile.toml template tasks ejected

Resulting in the updated chompfile.toml:

version = 0.1

[[task]]
name = 'build:typescript'
target = 'lib/##.js'
dep = 'src/##.ts'
stdio = 'stderr-only'
run = 'node ./node_modules/@swc/cli/bin/swc.js $DEP -o $TARGET --no-swcrc --source-maps -C jsc.parser.syntax=typescript -C jsc.parser.importAssertions=true -C jsc.parser.topLevelAwait=true -C jsc.parser.importMeta=true -C jsc.parser.privateMethod=true -C jsc.parser.dynamicImport=true -C jsc.target=es2016 -C jsc.experimental.keepImportAssertions=true'

License

Apache-2.0

chomp's People

Contributors

aslemammad avatar canadaduane avatar easrng avatar guybedford avatar jake-danton avatar yowainwright avatar zachsa avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

chomp's Issues

Templates

Templates would allow defining a way to generate parameterized tasks. For example, a Svelte template might look like (based on using engines #4):

[[template]]
  name = "svelte"
  [[template.option]]
    name = "indir"
    prompt = "Enter input directory"
    default = "src"
    type = "string"
  [[template.option]]
    name = "outdir"
    prompt = "Enter output directory"
    default = "lib"
    type = "string"
  [template]
    generate = """({ outdir, indir }) => [{
      name: 'build:svelte',
      target: `${outdir}/#.svelte.js`,
      deps: [`${indir}`],
      engine: 'node',
      run: `
        import { readFileSync, writeFileSync } from 'fs';
        import { compile } from 'svelte/compiler';

        const source = fs.readFileSync(process.env.dep, "utf-8");
        const result = svelte.compile(source, {
          filename: process.env.DEP,
          css: false,
        });

        writeFileSync(process.env.TARGET, result.js.code);
        writeFileSync(process.env.TARGET + ".map", JSON.stringify(result.js.map));

        const cssFile = process.env.TARGET.replace(/\\.js$/, ".css");
        writeFileSync(cssFile, result.css.code);
        writeFileSync(cssFile + ".map", JSON.stringify(result.css.map));
      `
    }]
    """

With the template "installed", one could then write a task like:

[[task]]
  name = "build:svelte"
  template = "svelte"
  [task.args]
    indir = "src"
    outdir = "lib"
    dev = true

where the target and dependency configuration would be automatically managed based on the provided template input.

Internally, the template is expanded based on the options provided, with the template generation itself running as an embedded JS engine like QuickJS or similar.

The inner "run" via engines still runs in Node.js though!

This means the template can be very quickly expanded without needing to spawn an external process and internally turned into its expanded task form.

When initializing the template to begin with, the prompt definitions allow for nice standard CLI prompts for setting template options. Possibly even via a visual tool in future.

Directory targets should invalidate subtargets

When a target is a directory, and that target completes, any subtargets of that target should have their mtimes updated based on the task that created the directory. As a result this should effectively stop the subtargets from applying if they've been defined by the directory target.

The use case here is npm installs where there is the npm install operation and an npm install pkg@v operation, where the npm install target is node_modules and package-lock.json and the npm install pkg@v target is package-lock.json and node_modules/pkg. If the npm install is running and the npm install pkg@v depends on it, then the npm install pkg@v should be able to be preemptively stopped by the npm install task.

Another way to implement this might be to simply lazily compute mtimes after dependencies have finished processing.

Watcher to halt running processes

If the watcher invalidates a task whose execution is currently "in progress", then we should terminate the running process. This will enable eg Deno to be restarted by Chomp.

Task arguments

It would be useful to define arguments for tasks to allow eg chp build "custom data" style executions.

One approach would be to have special types of tasks which explicitly take arguments, defining their arguments (task-defined).

Alternatively we could simply entirely rely on the process model of argv, so that all tasks can have any number of arguments provided (caller-defined).

Invocation in task-defined would be automatic - chp custom-task custom-data

Invocation in caller-defined would require something like -- - chp custom-task -- custom-data.

The last is already familiar to npm scripts users, so may actually be ok.

Note that only tasks that do not define targets can be permitted arguments.

Suggestion: use 'chomp' as CLI

I like brevity, but I also appreciate being able to learn what's happening when seeing others' code or commands for the first time. I think using the full name "chomp" will lead to better outcomes wrt sharing and teaching.

FWIW I see "chp" and want to pronounce it "chip".

edit to add: I considered aliasing chomp to chp so I could see the full name and find familiarity with the project & name, but then thought that was silly and made this suggestion instead :)

Multiple target soundness

We need to ensure that when multiple tasks point to the same target that which task is applied is well-defined.

Templates "fighting" to install dependencies

I believe what I'm seeing is multiple templates starting pnpm simultaneously(?) and fighting to modify package.json.

My chompfile.toml:

version = 0.1

[default-template-opts.npm]
  auto-install = true
  pkg-manager = "pnpm"

[[task]]
  name = "build"
  deps = [":generate", ":css"]

[[task]]
  name = "generate"
  template = "jspm"
  target = "public/index.html"
  deps = ["src/index.html", "svelte", "swc"]
  [task.template-opts]
    env = ["browser", "production"]

[[task]]
  name = "svelte"
  template = "svelte"
  target = "public/build/#.js"
  deps = ["src/#.svelte"]

[[task]]
  name = "swc"
  template = "swc"
  target = "public/build/#.js"
  deps = ["src/#.ts"]

[[task]]
  name = "css"
  target = "public/#.css"
  deps = ["src/#.css"]
  run = "cp $DEP $TARGET"

[[task]]
  name = "clean"
  run = "rm -rf public/*"

My package.json (before chomp build):

{
  "name": "tendril-art",
  "version": "1.0.0",
  "private": true,
  "scripts": {
    "build": "chomp",
    "dev": "chomp -s"
  },
  "devDependencies": {
    "@tsconfig/svelte": "^2.0.0"
  },
  "dependencies": {
    "@smui/form-field": "^5.0.0-beta.7",
    "@smui/select": "^5.0.0-beta.7",
    "@smui/switch": "^5.0.0-beta.7",
    "@smui/textfield": "^5.0.0-beta.7",
    "@types/three": "^0.133.0",
    "sirv-cli": "^1.0.0",
    "svelte-range-slider-pips": "^1.8.0",
    "three": "0.131.3",
    "troika-three-text": "0.43.0",
    "troika-three-utils": "0.43.0",
    "troika-worker-utils": "^0.43.0"
  }
}

During the package install phase, there are many references to my version of pnpm being out of date (which clued me into what was happening). Usually pnpm shows the out-of-date warning just once. In addition, many packages are being "reused", and seem to be installing the same thing over and over again:

Progress: resolved 46, reused 22, downloaded 19, added 086 MB, done
Progress: resolved 62, reused 28, downloaded 19, added 086 MB
Progress: resolved 63, reused 29, downloaded 20, added 0.2.128: 1.53 MB/13.8 MB
Downloading registry.npmjs.org/@swc/core-linux-x64-gnu/1.2.128: 13.8 MB/13.8 MB, donee
Downloading registry.npmjs.org/@swc/core-linux-x64-gnu/1.2.128: 13.8 MB/13.8 MB, done
DoDownloading registry.npmjs.org/@swc/core-linux-x64-musl/1.2.128: 8.39 MB/13.9 MB

(It's a bit hard to see, because pnpm likes to erase the line and write over top of the line to show the next package download).

Finally, the end state of my package.json looks like it only installed one of the templates' dependencies (swc in this case):

{
  "name": "tendril-art",
  "version": "1.0.0",
  "private": true,
  "scripts": {
    "build": "chomp",
    "dev": "chomp -s"
  },
  "devDependencies": {
    "@swc/cli": "0.1",
    "@swc/core": "1",
    "@tsconfig/svelte": "^2.0.0"
  },
  "dependencies": {
    "@smui/form-field": "^5.0.0-beta.7",
    "@smui/select": "^5.0.0-beta.7",
    "@smui/switch": "^5.0.0-beta.7",
    "@smui/textfield": "^5.0.0-beta.7",
    "@types/three": "^0.133.0",
    "sirv-cli": "^1.0.0",
    "svelte-range-slider-pips": "^1.8.0",
    "three": "0.131.3",
    "troika-three-text": "0.43.0",
    "troika-three-utils": "0.43.0",
    "troika-worker-utils": "^0.43.0"
  }
}

Global template options

Global environment passing to templates was recently added. This works as a sort of global template configuration space.

But for a feature like autoInstall which is an option on every template, it could be useful to explicitly treat this as global template configuration instead of an environment variable.

The suggestion would be to not pass env into templates, but instead pass globalTemplateConfig, where this is configured like:

[global_config]
  pkgManager = 'yarn'
  autoInstall = true

Deno engine

Just like we support a Node.js engine, it would be nice to have a Deno engine similarly. For the Deno engine it may be beneficial to also allow passing an import map alongside the run string.

Docs

Pad out the CLI documentation, ready for website work.

Named environments

Currently we support a global [env] which permits global env options (alongside local env options per-task).

It would be beneficial to support global named environments like [env.default], [env.development] and [env.production] then to have a --env=development flag that can be passed to vary the global environment.

This means replacing the global [env] with [env.default]. Per-task we may want the ability to split on environment as well in which case the same should also happen to tasks.

[env.default] should always be treated as extended by the current environment I think.

Server says "Hello World" independent of code

I wrote a little svelte app that said "hello world" (lower case) and was totally confused when the app showed "Hello World" (capitalized).

I believe there is an internal "Hello World" web page being sent by the chomp server?

Could it instead show instructions about how to create a public 'index.html' page somewhere in the project directory, or explain how to configure chomp to find it?

Changing the chompfile.toml while watching via `chomp -w` does not invalidate

I modified the chompfile while using chomp -ws and I was expecting it to re-build targets affected by the task whose options I modified:

[[task]]
  name = "build:html"
  template = "jspm"
  target = "lib/index.html"
  deps = ["src/index.html", ":build:svelte", ":build:typescript"]
  [task.template-options]
    env = ["browser", "production"]
    preload = true
    integrity = true

I added whitespace = false to the template-options, expecting index.html's import map to be formatted differently.

Engines

It would be useful to have an engines field that can permit different runtime engines, with the default effectively being Bash / Powershell.

For example a Node.js engine would allow directly writing Node.js code in tasks:

[[task]]
  target = "copy-file.js"
  deps = ["file.js"]
  engine = "nodejs"
  run = """
    import { writeFileSync } from 'fs';
    writeFileSync('./file.js', readFileSync(process.env.DEP));
  """

Support package managers other than npm in install template

Ideally, the install template should use a lockfile or other detection to determine the package manager to use, then apply it.

Effectively this is task branching of a sort.

One mechanism to do this is via multiple targets as the package locks, based on the different package locks used by the different package managers, where the target check takes the first found, resulting in that dependency path being taken. This will fully handle the use case, down to the package lock file being necessarily located otherwise it would default to npm.

Alternatively more advanced task predicates could be constructed, but I'd prefer not to open that box anytime soon.

We could also look at a corepack integration.

License

I've decided to ship this project with the GPL3 license to start. The closest project to this is GNU Make, which is licensed with this same license and as far as I'm aware that's never been too much of a restriction for that project.

The extensions will remain MIT licensed though.

This license is quite rare in the JS ecosystem currently, so if there is interest in the project there may well be questions about this.

Creating this issue to handle any feedback. The licensing decision can be changed at any point in future. By starting with MIT the converse isn't true though.

npm install operation queue

CI is now failing due to npm install operations bottlenecking since we aren't implementing a proper install queue system, it is only serial at the parent level.

We need to either have some way of coalescing the install tasks into a serial graph or to indicate they should be queued with only one install operation running at a time.

Multiple targets

We should support an alternative to target that is targets supporting an array of targets. This would be useful for builds that emit multiple files (eg JS + CSS).

In addition, both target and targets could be made to support globs, although I'd prefer to delay that extension until really needed further.

Glob dependencies

It should be possible to define dependencies via globs for jobs, separate to interpolation.

`chp svelte`: Error: EISDIR: illegal operation on a directory, read

I have the following chompfile.toml in an empty directory (or, with src and dist folders--same result):

version = 0.1

[[task]]
  name = "svelte"
  template = "chomp:svelte"
  [task.args]
    indir = "src"
    outdir = "dist"

When I type chp svelte I get the following error (and I don't understand what's going on):

- package.json [- 303.066µs]
- node_modules/svelte [- 702.144µs]
- node_modules/mkdirp [- 700.336µs]
○ :svelte
node:internal/process/esm_loader:94
    internalBinding('errors').triggerUncaughtException(
                              ^

[Error: EISDIR: illegal operation on a directory, read] {
  errno: -21,
  code: 'EISDIR',
  syscall: 'read'
}

Discussion: what are expectations wrt config files existing in the project root dir?

The Big Question

Given that Chomp creates, reads, modifies or ignores (depending on the context, see examples below) config files in the project root, what should a developer expect from Chomp? How can we make the rules clear, the problems easy to explain (if any), and minimize negative surprise?

Example 1: package.json

With an empty project folder and a chompfile.toml below, running chomp svelte will create a package.json file with "svelte": "^3.45.0" as a devDependency. This procedure seems to assume certain things about the project--for example that the package.json does not exist, or is valid JSON, or has sane configuration.

Interesting Edge Cases:

  1. If a pre-existing package.json file exists with an incompatible version of svelte specified (e.g. "2.0.0"), what should happen? (Currently, "ERR_UNSUPPORTED_DIR_IMPORT" is thrown by ESMLoader).
  2. What should happen if the package.json file is invalid? For example, if it has comments in it and npm cannot process it. (Currently, an "EJSONPARSE" error is produced).
  3. What should happen if the package.json file has changed in a way that makes the compile step valid or invalid? For example, changing a package.json file with "svelte": "2.0.0" to "svelte": "^3.45.0" (with 2.0.0 installed in node_modules) will not install svelte 3.45.0, and will fail to compile with ERR_UNSUPPORTED_DIR_IMPORT.

Example 2: svelte.config.js

In a "regular" svelte project, there is a svelte.config.js file that configures svelte compiler and preprocessor options. Currently, Chomp ignores this file and directly transforms (compiles) svelte files into js and css via a template specific to svelte.

Questions:

  1. If a developer expects the svelte.config.js file to do something (for example, accept "sass" format for <style> tags), but the file is ignored, how can we signal to the developer that it is ignored?
  2. How would they go about configuring svelte if it is ignored?
  3. If it should not be ignored, how does that affect our current chomp:svelte template? And how can we best balance simplicity and configurability?

Sample chompfile.toml:

version = 0.1

[[task]]
  name = "svelte"
  template = "chomp:svelte"
  target = "lib/#.js"
  deps = ["src/#.svelte"]

Template options validation

I had pkg-manager set in my chompfile, but the name of the setting has since changed to package-manager. It took a minute to figure out why ${packageManager} was still equal to "npm".

Lazy mtime computations

When one task depends on another, and the task target is defined by the first task, we should let the mtime be computed after the dependency job completion cause the next task to not run per the invalidation rules.

Command logging

When writing shell scripts, it could be useful to have a traditional Makefile style command log output.

One way to achieve this could be to just duplicate each line of the script as an associated echo, then have a special option for this in the command engine like shell-log = true or similar.

A global option could also be available with per-task configurations, effectively replacing the current debug.

Conflict in target extension causes dependency fighting

I have two problems:

  1. I've forgotten how to tell chomp how to compile both svelte (.svelte) and typescript (.ts) to javascript (.js)
  2. When I naively ran chomp build using the below chompfile, chomp tried installing swc multiple times
version = 0.1
extensions = ['[email protected]:swc', '[email protected]:svelte']

[env]
PACKAGE_MANAGER = "pnpm"

[template-options.npm]
auto-install = true

[[task]]
name = 'build'
deps = ['build:typescript', 'build:svelte']

[[task]]
name = 'build:typescript'
template = 'swc'
target = 'lib/#.js'
deps = ['src/#.ts']

[[task]]
name = 'build:svelte'
template = 'svelte'
target = 'lib/#.js'
deps = ['src/#.svelte']

Here is the output:

devDependencies:
+ @swc/cli 0.1.55
+ @swc/core 1.2.155
+ mkdirp 1.0.4
Downloading registry.npmjs.org/@swc/core-linux-x64-musl/1.2.155: 21.9 MB/21.9 MB, done
Downloading registry.npmjs.org/@swc/core-linux-x64-gnu/1.2.155: 20.7 MB/20.7 MB, done
Downloading registry.npmjs.org/@swc/core-linux-x64-gnu/1.2.155: 19.2 MB/20.7 MB
  lib/main.js invalidated by node_modules/@swc/core
Downloading registry.npmjs.org/@swc/core-linux-x64-gnu/1.2.155: 20.1 MB/20.7 MB
Downloading registry.npmjs.org/@swc/core-linux-x64-gnu/1.2.155: 20.7 MB/20.7 MB, done
Progress: resolved 38, reused 24, downloaded 3, added 3, done
  lib/App.js invalidated by node_modules/svelte
🞂 lib/App.js
node:internal/errors:464
    ErrorCaptureStackTrace(err);
    ^

TypeError [ERR_INVALID_ARG_TYPE]: The "data" argument must be of type string or an instance of Buffer, TypedArray, or DataView. Received null
    at writeFile (node:internal/fs/promises:760:5)
    at file:///home/duane/Relm/art-studio/[cm]:21:9 {
  code: 'ERR_INVALID_ARG_TYPE'
}
x lib/App.js [90.167254ms]
x :build:svelte
x :build
Unable to complete all tasks.

I think this is similar to #33

$ chomp --version
Chomp 0.1.12

Deps analysis

Currently deps must always be direct deps. It could be useful to define a form of dependency that means depend on this file and everything it statically imports via eg JS analysis / CSS import analysis etc.

We have es-module-lexer in Rust to use for this.

One way to do this might be to define deps = ["^lib/app.js"] where eg the ^ leading symbol implies deps analysis to form the deps expansion into a flat deps = ["lib/app.js", "lib/dep.js"] underneath.

Custom check functions

Currently the target_check field can be used to switch between "mtime" and "exists" checks for targets.

It might be useful to permit some kind of check scripting, for example, something like:

[[task]]
  deps = ['file.txt']
  target_check = "script"
  check_script = """([...deps], target) => {
    return deps.every(dep.mtime > X && dep.source.matches(/some text/));
  }"""

This would be sufficient to implement eg package.json version checks for npm installs.

Run limits

We should implement a proper run pool with a maximum number of run invocations. If we can link run invocations to their templates we could also limit by template which would be an easy way to solve the npm install contention issue.

Add Windows CI

It's important to have Windows CI alongside the current tests to avoid cross-platform regressions.

Template library

Once templates are implemented via #5, we can expose an internal template registry / library. This could just be a simple manifest that maps string names to template definitions. The registry itself would be synced from a main URL or git repo that anyone could send a PR to to add a template.

Another way around this could be to have a chp template install github:some/pkg style workflow. Or possibly a chp template --update where it syncs the template registry.

This is the most important UX to get right to enable a real ecosystem of composable build primitives.

Discussion: visibility of template dependencies and args

From #31 (comment):

Templates are functions that output a list of tasks, where those tasks in
turn can use templates. The jspm template will output the call to the
necessary npm template for JSPM to work, with the global option making auto
install configurable.

I think what's a bit tricky from a dev standpoint is that there is some "magic" happening inside that is difficult to reason about. What templates depend on what other templates? What args or options are available if I use this template vs. that template?

I'm trying to think of a way to make this easier to see inside. Possibly a chomp command line option that reads the chompfile and lists templates, its dependencies, and the options for all of these? I don't love it, but it's better than trying to find the template sources and learning about internals, I suppose.

Would calling it global-options.npm seem clearer? It can be useful to eg
configure a global Babel config this way for all Babel templates to avoid
repetition so that’s why I was making sure that connection was clear first
and foremost.

I like the goal here, I'm just not sure about how to make invisible things more visible.

Template test system

A proper template test system needs to be set up to ensure the internal templates can easily accept new PRs with standard protocols and without regressions

Error handling

Currently all errors are just panics - we need to implement proper error messages.

Stop multiline scripts on failure

It could be useful to have an option to stop multiline scripts if any given line fails.

In powershell this is done with $ErrorActionPreference = "Stop";

In bash, with set -eu.

One problem is that native executions in powershell still pass, but perhaps that is an allowable compat subset to at least get close to the desired behaviour.

This could either be an environment variable like CHOMP_CMD_STOP_ON_ERR = '1' or a direct field on the task.

Template task reductions

The npm install task currently initiates a new install operation per package being installed. Ideally we could merge all of these install tasks into one so there is one npm install call despite it being generated from many different templates.

This is a kind of stateful reduction of a template over many invocations. One model to handle this could be to maintain a shared state object as the second argument to all hooks of a given template, with its state maintained successively over each call. Then a new template hook - completion would be able to output a list of tasks on completion with this object as its argument. This way the template could compile shared state as it runs through each template invocation, then finally output a single reduction task in completion (the combined npm install operation in this case).

No args "thread 'main' panicked"

I have a chompfile.toml in an empty directory like this:

version = 0.1

[[task]]
  name = "svelte"
  template = "chomp:svelte"
  [task.args]
    indir = "src"
    outdir = "dist"

When I type "chp" on the command line (no args), I get the following panic:

thread 'main' panicked at 'called `Option::unwrap()` on a `None` value', src/task.rs:1452:44
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

With RUST_BACKTRACE=1 I get the following:

thread 'main' panicked at 'called `Option::unwrap()` on a `None` value', src/task.rs:1452:44
stack backtrace:
   0: rust_begin_unwind
             at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/panicking.rs:517:5
   1: core::panicking::panic_fmt
             at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/core/src/panicking.rs:101:14
   2: core::panicking::panic
             at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/core/src/panicking.rs:50:5
   3: core::option::Option<T>::unwrap
             at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/core/src/option.rs:735:21
   4: chp::task::run::{{closure}}
             at /home/duane/tmp/chomp/src/task.rs:1452:20
   5: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/core/src/future/mod.rs:80:19
   6: chp::main::{{closure}}
             at /home/duane/tmp/chomp/src/main.rs:105:14
   7: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/core/src/future/mod.rs:80:19
   8: tokio::park::thread::CachedParkThread::block_on::{{closure}}
             at /home/duane/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.14.0/src/park/thread.rs:263:54
   9: tokio::coop::with_budget::{{closure}}
             at /home/duane/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.14.0/src/coop.rs:106:9
  10: std::thread::local::LocalKey<T>::try_with
             at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/thread/local.rs:399:16
  11: std::thread::local::LocalKey<T>::with
             at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/thread/local.rs:375:9
  12: tokio::coop::with_budget
             at /home/duane/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.14.0/src/coop.rs:99:5
  13: tokio::coop::budget
             at /home/duane/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.14.0/src/coop.rs:76:5
  14: tokio::park::thread::CachedParkThread::block_on
             at /home/duane/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.14.0/src/park/thread.rs:263:31
  15: tokio::runtime::enter::Enter::block_on
             at /home/duane/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.14.0/src/runtime/enter.rs:151:13
  16: tokio::runtime::thread_pool::ThreadPool::block_on
             at /home/duane/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.14.0/src/runtime/thread_pool/mod.rs:77:9
  17: tokio::runtime::Runtime::block_on
             at /home/duane/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.14.0/src/runtime/mod.rs:463:43
  18: chp::main
             at /home/duane/tmp/chomp/src/main.rs:113:5
  19: core::ops::function::FnOnce::call_once
             at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/core/src/ops/function.rs:227:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

Hot-reloading workflows via websockets

It should be possible to achieve hot-reloading when using chomp --serve by creating a websocket connection that publishes the watcher events, and then having a plugin for es-module-shims that supports in-browser hot-reloading of all module resources.

Makes for some good social media.

Install instructions don't take `package-manager` setting into account

When running chomp without auto-install, it provides warning messages showing how to install the necessary dependencies, e.g.:

Chomp: Some packages are missing. Please run npm install @jspm/generator mkdirp -D

I'm using pnpm and (stupidly) copy/pasted the npm command. Would be nice if it took package-manager into account, i.e.:

Chomp: Some packages are missing. Please run pnpm install @jspm/generator mkdirp -D

JSX compilation broken

Is there a way to enable jsx compilation? Seems to be broken after update. Probably due to this #51 some configuration needs to be changed.

Wildcard targets

Now that we have multiple target support, it should be possible to define wildcard targets the target an entire glob pattern. The pattern should have no conflicts or crossover with other targets. Then it can be fully iterated and built like any other target.

Multiple target builds

If there are multiple ways to build the same target, rather than throwing we can instead handle validation as any of the build paths being valid.

Then when invalidated, the closest build path (the one with the least work) can be used.

This effectively enables a sort of mini conditional build approach without needing a predicate system just yet.

Re-running `chomp build` always tries to install packages

When I run chomp build the first time, it makes sense that targets with packages marked autoInstall = true install packages via pnpm.

However, when I re-run chomp build, I expect the package management to be done and to not run again.

This is the output after running chomp build the 2nd time:

$ chomp
- lib/assets/royal_esplanade_1k.hdr [- 405.656µs]
- lib/assets/Default_normal.jpg [- 551.638µs]
- package.json [- 217.317µs]
  node_modules/svelte invalidated by package.json.
○ node_modules/svelte
  node_modules/@swc/core invalidated by package.json.
○ node_modules/@swc/core
- lib/assets/Default_emissive.jpg [- 2.882522ms]
  node_modules/@jspm/generator invalidated by package.json.
○ node_modules/@jspm/generator
- lib/assets/Default_albedo.jpg [- 3.799818ms]
- lib/assets/Default_metalRoughness.jpg [- 3.855873ms]
- lib/assets/DamagedHelmet.gltf [- 4.125918ms]
- lib/assets/Default_AO.jpg [- 4.169401ms]
- lib/assets/DamagedHelmet.bin [- 4.250695ms]
- :copy [- 4.394801ms]
Progress: resolved 1, reused 0, downloaded 0, added 0
Progress: resolved 1, reused 0, downloaded 0, added 0
 WARN  @jspm/generator > ipfs-client > ipfs-grpc-client: @improbable-eng/[email protected] requires a peer of google-protobuf@^3.14.0 but none was installed.
 WARN  @jspm/generator > ipfs-client > ipfs-grpc-client > ipfs-core-types > multiaddr > dns-over-http-resolver: [email protected] requires a peer of node-fetch@* but none was installed.
 WARN  @jspm/generator > ipfs-client > ipfs-grpc-client: @improbable-eng/[email protected] requires a peer of google-protobuf@^3.14.0 but none was installed.
 WARN  @jspm/generator > ipfs-client > ipfs-grpc-client > ipfs-core-types > multiaddr > dns-over-http-resolver: [email protected] requires a peer of node-fetch@* but none was installed.
Already up-to-date
 WARN  @jspm/generator > ipfs-client > ipfs-grpc-client: @improbable-eng/[email protected] requires a peer of google-protobuf@^3.14.0 but none was installed.
 WARN  @jspm/generator > ipfs-client > ipfs-grpc-client > ipfs-core-types > multiaddr > dns-over-http-resolver: [email protected] requires a peer of node-fetch@* but none was installed.
Already up-to-date
Already up-to-date
Progress: resolved 150, reused 139, downloaded 0, added 0, done
√ node_modules/@swc/core [573.1474ms 575.99461ms]
- lib/helmet.js [- 576.042333ms]
- lib/main.js [- 576.049968ms]
- lib/setup.js [- 576.059158ms]
Progress: resolved 150, reused 139, downloaded 0, added 0, done
√ node_modules/svelte [612.150277ms 614.15801ms]
- lib/App.js [- 614.149749ms]
- :svelte [- 614.219651ms]
√ node_modules/@jspm/generator [615.134442ms 619.007555ms]
  lib/index.html invalidated by :svelte.
○ lib/index.html
√ lib/index.html [174.482914ms 794.79709ms]
- :build [- 794.900256ms]

Edit: Adding my chompfile:


[env]
PKG_MANAGER = "pnpm"

[[task]]
  name = "build"
  deps = [":generate"]

[[task]]
  name = "svelte"
  template = "svelte"
  target = "lib/#.js"
  deps = ["src/#.svelte"]
  [task.args]
    autoInstall = true

[[task]]
  name = "swc"
  template = "swc"
  target = "lib/#.js"
  deps = ["src/#.ts"]
  [task.args]
    autoInstall = true

[[task]]
  name = "generate"
  template = "jspm:generate"
  target = "lib/index.html"
  deps = ["public/index.html", "svelte", "swc", "copy"]
  [task.args]
    autoInstall = true

[[task]]
  name = "copy"
  target = "lib/assets/#"
  deps = ["public/assets/#"]
  run = "cp $DEP $TARGET"

[[task]]
  name = "clean"
  run = "rm -rf lib/assets/; rm lib/*"```

main panicked

I pulled the latest commit and recompiled, but chomp svelte now panics:

- package.json [- 182.396µs]
thread 'main' panicked at 'called `Option::unwrap()` on a `None` value', src/task.rs:1004:58
stack backtrace:
   0: rust_begin_unwind
             at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/panicking.rs:517:5
   1: core::panicking::panic_fmt
             at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/core/src/panicking.rs:101:14
   2: core::panicking::panic
             at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/core/src/panicking.rs:50:5
   3: core::option::Option<T>::unwrap
             at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/core/src/option.rs:735:21
   4: chomp::task::Runner::drive_completion
             at /home/duane/tmp/chomp/src/task.rs:1004:34
   5: chomp::task::Runner::drive_all
             at /home/duane/tmp/chomp/src/task.rs:887:37
   6: chomp::task::Runner::drive_completion
             at /home/duane/tmp/chomp/src/task.rs:998:17
   7: chomp::task::Runner::drive_targets::{{closure}}
             at /home/duane/tmp/chomp/src/task.rs:1328:13
   8: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/core/src/future/mod.rs:80:19
   9: chomp::task::run::{{closure}}
             at /home/duane/tmp/chomp/src/task.rs:1434:5
  10: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/core/src/future/mod.rs:80:19
  11: chomp::main::{{closure}}
             at /home/duane/tmp/chomp/src/main.rs:105:14
  12: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/core/src/future/mod.rs:80:19
  13: tokio::park::thread::CachedParkThread::block_on::{{closure}}
             at /home/duane/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.14.0/src/park/thread.rs:263:54
  14: tokio::coop::with_budget::{{closure}}
             at /home/duane/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.14.0/src/coop.rs:106:9
  15: std::thread::local::LocalKey<T>::try_with
             at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/thread/local.rs:399:16
  16: std::thread::local::LocalKey<T>::with
             at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/thread/local.rs:375:9
  17: tokio::coop::with_budget
             at /home/duane/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.14.0/src/coop.rs:99:5
  18: tokio::coop::budget
             at /home/duane/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.14.0/src/coop.rs:76:5
  19: tokio::park::thread::CachedParkThread::block_on
             at /home/duane/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.14.0/src/park/thread.rs:263:31
  20: tokio::runtime::enter::Enter::block_on
             at /home/duane/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.14.0/src/runtime/enter.rs:151:13
  21: tokio::runtime::thread_pool::ThreadPool::block_on
             at /home/duane/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.14.0/src/runtime/thread_pool/mod.rs:77:9
  22: tokio::runtime::Runtime::block_on
             at /home/duane/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.14.0/src/runtime/mod.rs:463:43
  23: chomp::main
             at /home/duane/tmp/chomp/src/main.rs:113:5
  24: core::ops::function::FnOnce::call_once
             at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/core/src/ops/function.rs:227:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

chompfile.toml

version = 0.1

[[task]]
  name = "svelte"
  template = "chomp:svelte"
  target = "lib/#.js"
  deps = ["src/#.svelte"]

Note that the above chompfile and setup seems to work fine prior to 465ed94

npm install version verification

The npm install task check currently only checks the package exists in node_modules. We should update this to check the package exists and is the right version range.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.