steelbrain / pundle

👾 peaceful bundles - js bundler, built from the ground up for speed and extensibility

Pundle

modern javascript bundler

Pundle is a next generation module bundler. It's written with extensibility and performance in mind.

Getting Started

Pundle is still in beta but it's pretty stable. Here's how you can try it today!

For Yarn users

# For a react app with babel + hmr preconfigured
yarn create pundle-react-app my-app
# For a JS app with babel preconfigured
yarn create pundle-babel-app my-app
# For a pure bare bones app
yarn create pundle-app my-app

If you have Node v8+

# For a react app with babel + hmr preconfigured
npm init pundle-react-app my-app
# For a JS app with babel preconfigured
npm init pundle-babel-app my-app
# For a pure bare bones app
npm init pundle-app my-app

Top Priority

• docs

Note to self

• "ModuleScopePlugin" to limit direct import of files
• ESlint to lint files
• enforce case sensitivity?

Post-initial demo goals

• visualize dependency file size
• Chunk minifers for HTML
• Make tool to analyze cost of imports in app
• Fork browser field resolver, it's so wasteful of fs

pundle's Issues

Handle non-root imports properly in sourceMap

Currently full path is shown for imports outside of root

Introduce new events on modules class to help with HMR

A diff event would be nice, general cleanup of transform is also necessary

Use lerna to manage mono-repo arch

https://github.com/lerna/lerna

Show line number and column in MODULE_NOT_FOUND errors

Just like the title says, currently the stack is like

Error: Module not found 'some-random-module'
  at $root/test.js:0:0

We should include the correct source when possible

NPM Installer Module

It should automatically install npm modules as we require

Remove special cases for $root everywhere

Currently we're injecting $root and there's special cases for it everywhere, would be nice if we could remove those special cases

Decide on name for config file

Follow up to #75

Errors don't show exact file it comes from

In motion:

[Error: Unable to resolve './views/icon' from '$root/views' at /app/node_modules/pundle/lib/path.js:59:15]

The actual file that caused the error was segments.js in $root/views

Requiring JSON files doesn't work very well

Just like the title says module.exports remains empty, it shouldn't

pundle-cli can't find its entry

${ project root } replaces with my actual path for privacy

> yarn add [email protected]

> $(yarn bin)/pundle
Error Cannot find module '${ project root }/src/mount.tsx'

> cat ${ project root }/src/mount.tsx
import * as React from 'react';
import * as ReactDOM from 'react-dom';

import Main from './components/Main';

ReactDOM.render(
  <Main />,
  document.getElementById('root')
);

if (module.hot) {
  module.hot.accept()
}

> cat Pundleconfig.js 
module.exports = {
  entry: [require.resolve('./src/mount.tsx')],
  pathType: 'filePath',
  rootDirectory: __dirname,
  replaceVariables: {
    'process.env.NODE_ENV': 'development',
  },
}

Garbage collection not triggered in some cases

If the number of imports remains the same, even when the imports have changed, it does not trigger a garbage collection

pundle-typescript doesn't report type errors

According to microsoft/TypeScript#4864, the simple transpileModule function in the TypeScript compiler only reports syntax errors (which is what babylon gives you anyway). For real errors, we'd need something more sophisticated, likely using language services to watch the filesystem. pundle is already using chokidar for watching.

TBD:

• Is there a way to replace pundle's watcher with TypeScript's when using pundle-typescript?
• What's the performance impact on letting both do the watching (pundle for bundling and TypeScript for error reporting)?
• How much work it would be to write a Language Services-based pundle plugin?

Let `.loadPlugins` return pundle instance

This is a discussion about api: what about let .loadPlugins(and .compile(), .generate(), etc.) return promise(which resolves a pundle instance), so we can use it like this:

import Pundle from 'pundle'

// create pundle instance
new Pundle({
  entry: ['index.js'],
  rootDirectory: process.cwd(),
  pathType: 'filePath',
  moduleDirectories: ['node_modules'],
})

// load plugins
.loadPlugins([
  [require.resolve('babel-pundle'), {
    config: {
      presets: ['steelbrain']
    }
  }],
  require.resolve('pundle-some-magical-plugin'),
])

// compile
.then(function(pundle) {
  return pundle.compile()
})

// config loaders
.then(function(pundle) {
  pundle.loadLoaders([
    { extensions: ['.coffee'], loader: require('pundle-coffee') },
    { extensions: ['.less'], loader: require('pundle-less') },
  ])
  return pundle.generate({ sourceMap: true })
})

// output
.then(function(generated) {
  FS.writeFileSync('./bundle.js', `${generated.contents}\n//# sourceMappingURL=bundle.js.map`)
  FS.writeFileSync('./bundle.js.map', generated.sourceMap)
}).catch(function(error) {
  console.error('error', error)
})

This might looks a bit more simpler & consistent than the original example IMHO, what do you think? @steelbrain

HMR presumes index.js in current folder

  Pundle:Error Error: ENOENT: no such file or directory, stat '~/Projects/pundle/packages/test/index.js'

if I run ./devServer.js --entry --hmr ./src/index.js. cding into src first fixes it.

merge source maps in babel transpiler

Currently sourcemaps are not working very well after being babel transpiled

Add specs everywhere

To make sure we don't break stuff as we progress

Implement static content loader

Not sure how great an idea this is, but a string loader at least would help me with a couple things. Specifically, we are using svg's and I want to share between a few different apps by putting in a package. So, naturally this would help.

For now the process is manual: copy and paste into a string inside a js file.

Installer doesn't work on docker

⠋ Installing sb-pluralize[consistent-env] Unable to determine environment TypeError: Path must be a string. Received undefined
    at assertPath (path.js:7:11)
    at Object.basename (path.js:1355:5)
    at /app/node_modules/consistent-env/lib/index.js:39:40
    at new Promise (/app/node_modules/core-js/modules/es6.promise.js:193:7)
    at module.exports.async (/app/node_modules/consistent-env/lib/index.js:33:10)
    at /app/node_modules/sb-exec/lib/helpers.js:13:51
    at next (native)
    at step (/app/node_modules/sb-exec/lib/helpers.js:59:191)
    at /app/node_modules/sb-exec/lib/helpers.js:59:451
    at new Promise (/app/node_modules/core-js/modules/es6.promise.js:193:7)
Install sb-pluralize ✗
[Error: spawn npm ENOENT at exports._errnoException (util.js:953:11)]
[Error: Unable to resolve 'sb-pluralize' from '$root/helpers' at /app/node_modules/pundle/lib/path.js:59:15]

Source maps are a bit off

An easy to repro thing would be debugger put somewhere in the app

Fixing not found file error fails

[Error: Cannot find module './logo' at $root/node_modules/macro-ui/lib/index.js:120:13]

# after fixing:

[Error: Module not found 'index.js' at getAllImports (/Users/nw/company/macro/apps/macro/node_modules/pundle/lib/helpers.js:314:15)]
[Error: Module not found 'index.js' at getAllImports (/Users/nw/company/macro/apps/macro/node_modules/pundle/lib/helpers.js:314:15)]

Allow the use of a pundle config file.

Right now, although powerful, having to manually go through the boilerplate steps of calling pundle.compile, pundle.generate, and fs.writeFileSync all in different promise blocks is a hassle. I know it's not that big of a deal, but I feel like since every other alternative offers a config file, the programmatic approach can scare newcomers away.

pundle-dev kinda solves this problem with it's server config, but that doesn't work with just pundle by itself.

It'd be nice to have something like:

export default {
  entry: ['index.js'],
  rootDirectory: './src',
  pathType: 'filePath',
  moduleDirectories: ['node_modules'],
  plugins: [
    ['babel-pundle', { config: { presets: ['steelbrain'] } }],
    'pundle-some-magical-plugin'
  ],
  loaders: [
    { extensions: ['.coffee'], loader: 'pundle-coffee' },
    { extensions: ['.less'], loader: 'pundle-less' }
  ],
  sourceMap: true,
  outputDirectory: 'lib'
}

The API for the config file would probably be different than this, but you get the idea. You could also add a server block if you instead to use this config with a server. It also carries the benefit of being more readable.

"Sending HMR 0" if `--hmr` not sent

It's easy for a new user to think that HMR is on, even if they forgot the --hmr flag, because the logs still mention it

Pundle:Server Sending HMR of 0 file(s) to 0 connection(s) +43s
Pundle:Watcher File Changed :: ~/pundle/packages/test/src/index.js +21s

Should HMR ever be off in the Dev Server? It should probably default to on. If it is off, HMR disabled is a better message than Sending HMR 0 file(s).

Hide initial loading of node_modules

It would be good to not lots of these on every start of motion

Babel plugin doesn't run with dev-server.

I can't get the babel plugin to run with the dev-server. It seems to load fine, but the code is never transpiled. I'm assuming it's a config error? I've tried looking through the source code but I don't quite understand where the plugins are supposed to be applied.

My config is here: https://github.com/eldh/jour/blob/master/dev-server.js. And I guess cloning that repo and doing npm i && npm start would reproduce the issue.

devServer onboarding experience needs love

../devServer.js --entry --hmr ./index.js

In the browser console, when I hit save:

bundle.js:1 Uncaught Error: Unable to apply HMR because some modules didnt accept it

How do I accept HMR modules? Why doesn't the dev server do this by default?

Handle errors in push imports properly

Currently after a module is set and it's dependency has an error, it's dependencies won't be processed at all. We should be handling this

Push compile errors to browser

This would help for when you miss it in your linter/console.

Using npm package `electron`

[Error: Unable to resolve './lib-cov/electron' from '$root/node_modules/electron' at /Users/nw/motion/packages/motion/node_modules/pundle/lib/path.js:59:15]

Listen to deleted event

We should be deleting files from our cache once they are deleted from the FS, this should be for node_modules too

Issues with babel-pundle.

1. From my understanding of conventions created by Webpack, loaders are String -> String transformations of the contents of a file, and plugins modify the behavior of the bundler itself. For example, CSS -> JS (style-loader) is a loader, and CommonChunksPlugin, which does static analysis of your project as a whole is a plugin.

Given this definition, since babel does ES6/7 -> ES5, which boils down to String -> String, shouldn't it be classified as a loader?

---

2. What's the convention for naming pundle packages in the future going to be?

Currently you have pundle, babel-pundle, and pundle-dev, and then for the loaders (which don't exist?) you have pundle-less and pundle-css.

It seems like babel-pundle is the odd one out. Are you going to change the name to pundle-babel to reflect that?

What do you think of adopting a pundle-plugin-* and pundle-loader-* approach?

NPM Installer should only try to install root packages

If we have a deep require, the base package exists but deep doesn't. It tries to install it, that is not the expected behavior

Create pundle-errors package

All plugins and loaders will throw PundleError then and we'd know if we should show just the error message or a stack trace based on that

Separate the concept of a plugin from the concept of a file transformation.

Why

As per my comment here #62 (comment), I think the separate concepts of a plugin and a transform are conflated, and pundle can benefit by making the concepts distinct.

• It would make pundle easier to understand in my opinion since it'd map 1-to-1 with the concept of Extract-Tranform-Load.
  • Extract = require()
  • Transform = transforms/transformer <- new addition to replace plugin
  • Load = loaders
• This frees up the word plugin to mean side-effects of the ETL process.
  • An eslint plugin to log errors
  • A plugin to install npm packages if they're found in the code.
  • Anything else someone wants to do by hooking into pundle
• In my experience / in my opinion, separating impure/pure code makes it easier to maintain pundle, and it would make it easier for users to see where their problems are occurring.

How

Add a new loadTransformers() function that puts each of the transformers into an array called this.transformers or something like that. The after the line here where you emit the before-process event do something like

let result = event.contents
for (const transformer of this.transformers) {
  result = transformer(this, filePath, result, event.sourceMap)
}
result = loader(this, filePath, event.contents, event.sourceMap)

Thoughts?

Allow hiding file paths in output

Just like the title says

pundle-dev: default to index.html

Firebase hosting has a neat feature where you can direct all missed URLs to a single file (index.html). They specifically advertise it for single-page apps (where the routing is handled by something like React Router).

Can I configure Pundle do this too? I'd love to have my local and hosted apps behave as identically as possibly. It's also nice not to get a 404 if you reload on a client-side route.

Notes for v2

Just writing these down so I don't forget

Main Components

These are going to be the main components of the v2 architecture

Resolvers

Resolvers will be responsible for resolving the path to a file based on relativity. They should be able to accept multiple module dirs and support live fs changes.

Loaders

Loaders will have the responsibility to process the raw contents of the files, even css or woff or something, and convert it into a js source, that automatically sets the contents to module.exports.

Loaders will also have the responsibility to tell pundle what others modules were required from it (only the ones where we can do it dynamic, like for js or coffee sources but NOT for less sources) as well as swap values in require to be usable in runtime

Transformers

Transformers will work in a way that they accept javascript and output javascript. Babel is an example use case of transformers.

Post-Transformers

Post-Transformers will work just like transformers, except after all transformers have been executed. UglifyJS is an example use case of transformers. Unlike transformers, these will not be executed on individual files but on the generated content.

Plugins

Everything else that does not modify the contents of the generated output will be a plugin. ESLint, NPM Installer and others could be counted as plugins.

Generators

Generators will be given the processed contents of each files and will be responsible for making output bundles. The default generator generates CommonJS style output.

Reporters

Reporters will be responsible for keeping the user aware of the status of the builds. For example, a reporter in pundle-dev could show the messages in browser, while the pundle CLI would show them in the terminal like this post

Other Notes to self

• Make sure to split CLI into a separate package pundle-cli or something and make pundle an API only base package
• Make sure to implement advance resolve config like aliases
• Every component should have a this context that has a report() method, useful in cases like eslint
• When this.resolve(path, ?from, ?cached) is called from a component, and the component is a resolver, make sure NOT to invoke that one again for that request until the existing request is fulfilled

TODO: Write more notes here

config.entry should be an Array or string error if TypeScript not installed

I think the config.entry error is too greedy in Pundle 2. I get this error if the TypeScript Transformer can't find TypeScript:

"config.entry should be an Array or string"

transforming sister modules

Consider a project that includes many modules in a monorepo:

packages/module-1/devServer.js
packages/module-1/src/index.ts
packages/module-1/dist/index.js

packages/module-1/node_modules/my-project-module-2 points at:

packages/module-2/src/index.ts
packages/module-2/dist/index.js

Is there a way to ask pundle to watch/transform module-2 from the module-1 dev server? It would have to

1. know that node_modules that start with my-project are fair-game for watching/transforming, and
2. know to look in src instead of dist for those dependencies

As I understand it now, I have to rebuild module-2/dist whenever I make changes to module-2/src, otherwise, pundle won't find the changes.

Roadmap

• Ability to subscribe to errors specific to a compilation
• Ability to watch directories from the core
• Ability to add pre-compile hook and add support for babel
• Ability to add post-compile hook and add support for uglify
• Ability to generate sources and maps of selected modules in format of HMR or Bundle
• Ability to generate sources and maps for all modules in format of Bundle
• Middleware for express to serve latest compiled files
• Ability to pass HMR callback on client side that determines how updates are applied
• Ability to replace variables like process.env in the app
• Ability to require node modules or their shims into the browser
• Add a CLI package similar to webpack-dev-server

Removed Items

• Ability to generate sources and maps of individual modules in format of HMR

Installer doesn't quit loop if module is invalid

If module has an invalid main property and the resolver doesn't work after install, the installer just reinstalls again. It should quit the loop instead and show an error to the user

HMR is not applied for Top-Level only changes

v1 of HMR implementation has a bug where it only updates the module accepting HMR if any of it's children are updated. It should also update when it itself is updated

Exit command not exiting

When you type exit in motion you get this error.

Uncaught Exception TypeError: Cannot read property 'dispose' of undefined
    at /Users/macbook/Desktop/motion/motion/packages/motion/lib/index.js:53:12
    at Emitter.emit (/Users/macbook/Desktop/motion/motion/packages/motion/node_modules/sb-event-kit/lib/emitter.js:78:28)
    at CLI.<anonymous> (/Users/macbook/Desktop/motion/motion/packages/motion/lib/cli/index.js:81:21)
    at CommandInstance.<anonymous> (/Users/macbook/Desktop/motion/motion/packages/motion/lib/cli/cli.js:54:29)
    at EventEmitter.session.execCommandSet (/Users/macbook/Desktop/motion/motion/packages/motion/node_modules/vorpal/dist/session.js:470:20)
    at EventEmitter.vorpal._exec (/Users/macbook/Desktop/motion/motion/packages/motion/node_modules/vorpal/dist/vorpal.js:956:18)
    at EventEmitter.vorpal._execQueueItem (/Users/macbook/Desktop/motion/motion/packages/motion/node_modules/vorpal/dist/vorpal.js:747:17)
    at EventEmitter.vorpal._queueHandler (/Users/macbook/Desktop/motion/motion/packages/motion/node_modules/vorpal/dist/vorpal.js:731:10)
    at EventEmitter.vorpal.exec (/Users/macbook/Desktop/motion/motion/packages/motion/node_modules/vorpal/dist/vorpal.js:678:10)
    at /Users/macbook/Desktop/motion/motion/packages/motion/node_modules/vorpal/dist/vorpal.js:621:10

Support watching of node modules

We ignore source modules but if some is a linked one then someone is possibly working in it, it should be an option disabled by default

Syntax Errors not being reported

Just noticed this in motion, Thanks @natew

Modules run in strict mode?

I updated pundle-dev from 1.0.1 to 1.1.4 and got errors that seems to be caused by modules running in strict mode (basically this: http://stackoverflow.com/questions/31221357/webpack-firebase-disable-parsing-of-firebase). Is this a bug or something that can be fixed with config?

Add a prefix to core modules

So they can be ignored, something like $core/fs

Can't require files with same name as sibling directory

When running our current stack I get:

[Error: Unable to resolve 'babel-runtime/core-js/symbol' from '$root/node_modules/babel-runtime/helpers' at /Users/nw/motion/packages/motion/node_modules/pundle/lib/path.js:59:15]

Modules are installed twice if required twice at the same time

We should have a lock or something in npm installer

typescript-pundle missing lib/index.js

typescript-pundle is missing its exports!

> yarn add typescript-pundle
> ls node_modules/typescript-pundle

README.md
node_modules
package.json
spec

Maybe you forgot to run bootstrap before publishing?

Make the requests wait properly if there's a compilation in progress

Currently only the first request waits, every other request throws an error. That shouldn't be happening