trivago / parallel-webpack Goto Github PK

Hi!

Thanks for this tool!! It has sped up our compilations for webpack immensely.

We are using Angular, and the amount of webpack work Angular does is huge. We do inmemory compilation of TypeScript and NGC, and other optimizations, and it gets up past 4GB of compilation.

We specify --max-old-space-size to node to complete the compilation. I assume this is passed along in the Node workers, correct?

Also, we are hitting this error on JSON.stringify, when stats: true is set on the run() method.

internal/child_process.js:626
    var string = JSON.stringify(message) + '\n';
                      ^

RangeError: Invalid string length
    at Object.stringify (native)
    at process.target._send (internal/child_process.js:626:23)
    at process.target.send (internal/child_process.js:538:19)

I am pretty sure this is due to a huge stats object being passed in. Thoughts on how to get this to not crash?

Thanks!
Joe

I'm running four webpack builds in parallel and occasionally have to terminate early by pressing CTRL-C.

When I press CTRL-C, I see the output about forcefully shutting down the child processes, but the main process stays alive and hangs without exiting. I need to press CTRL-C 1-2 more times to kill everything. I can verify that --max-retries=0 is getting correctly passed in. FWIW, one of my builds finishes rather quickly and the other three are longer running, so usually the SIGINT is sent after the first one is complete. Looks like if I CTRL-C quick enough before any builds have finished, the processes all terminate as expected.

Here is some example output from CTRL-C'ing later in the build:

^C[WEBPACK] Forcefully shutting down

(node:38167) Warning: a promise was rejected with a non-error: [object Object]
[WEBPACK] Build failed after 1.783 seconds
(node:38167) Warning: a promise was rejected with a non-error: [object Object]
(node:38167) Warning: a promise was rejected with a non-error: [object Object]
[WEBPACK] Forcefully shut down [name].js
CSPEER:(myBranch ⚡)$ events.js:160
      throw er; // Unhandled 'error' event
      ^

Error: channel closed
    at process.target.send (internal/child_process.js:540:16)
    at callback (/Users/cspeer/repos/my-repo/node_modules/worker-farm/lib/child/index.js:29:17)
    at process.shutdownCallback (/Users/cspeer/repos/my-repo/node_modules/parallel-webpack/src/webpackWorker.js:96:17)
    at emitNone (events.js:86:13)
    at process.emit (events.js:185:7)
    at Signal.wrap.onsignal (internal/process.js:199:44)

CSPEER:(myBranch ⚡)$ events.js:160
      throw er; // Unhandled 'error' event
      ^

Error: channel closed
    at process.target.send (internal/child_process.js:540:16)
    at callback (/Users/cspeer/repos/my-repo/node_modules/worker-farm/lib/child/index.js:29:17)
    at process.shutdownCallback (/Users/cspeer/repos/my-repo/node_modules/parallel-webpack/src/webpackWorker.js:96:17)
    at emitNone (events.js:86:13)
    at process.emit (events.js:185:7)
    at Signal.wrap.onsignal (internal/process.js:199:44)
events.js:160
      throw er; // Unhandled 'error' event
      ^

Error: channel closed
    at process.target.send (internal/child_process.js:540:16)
    at callback (/Users/cspeer/repos/my-repo/node_modules/worker-farm/lib/child/index.js:29:17)
    at process.shutdownCallback (/Users/cspeer/repos/my-repo/node_modules/parallel-webpack/src/webpackWorker.js:96:17)
    at emitNone (events.js:86:13)
    at process.emit (events.js:185:7)
    at Signal.wrap.onsignal (internal/process.js:199:44)

This is more of an annoyance than anything else most of the time, but sometimes results in stray processes left running. Not sure if this is an issue with worker farm or parallel-webpack's own worker management.

As discovered in #3 the documentation is unclear about what the baseOptions parameter for the createVariants utility is doing.
That effect is increased by using a valid Webpack configuration option (devtool) as the option name and then not actually using it within the configuration creation callback.

We should adjust the documentation to explain more clearly what that parameter is intended to be used for and also use a different option name (i.e. preferredDevTool) which ideally should be read from the environment and then we also need to make sure that the option is actually used within the callback.

The max-retries option can easily cause confusion and leftover processes when combined with webpacks watch mode.

As max-retries is rarely useful, it makes sense to set its default value to 0.

Please create a new release with the updated peerDepedencies

Is there any way to determine when building has been completed? I'm launching the app using OpenBrowserPlugin but I don't know when it's completed all parallel builds.

Assume I have an array of unrelated, handcrafted webpack configurations. I would like to run them in parallel using parallel-webpack node.js API. For example if run function took an array of configs instead of config paths, it would somewhat work for me.

My proposal is to have a lower level API for running array of configs that is reused by the actual run function that takes config path.

Explain the problem

When using webpack you can return promise from webpack config function. If you want multiple configurations you need to return a promise that returns an array of configs. parallel-webpack instead works only if you return array of promises.
This breaks compatibility with webpack and webpack-dev-server

Expected Behaviour

parallel-webpack should work same as webpack and correctly handle promise that returns array of configs.

Actual Behaviour

parallel-webpack sees only one target and then crashes because of error:
Unhandled rejection TypeError: Cannot read property 'filename' of undefined

Steps to reproduce

Create webpack config that returns promise that resolves to array instead of array of promises.

Provide your webpack config

module.exports = (env) => {
  return getTranslations().then(allTranslations => langs
    .map(lang => translations: allTranslations[lang])
    .map(translations => {
      return {
        entry: ...
        ...
      }
  }))
}

Provide your Environment details

• Node version: 9.4.0
• Operating System: Windows 10
• webpack version: 3.6.0
• parallel-webpack version: 2.2.0

For now webpack workers build and start watch only for first maxConcurrentCallsPerWorker * maxConcurrentWorkers variants. Rest variants never builds until we press Ctrl+C and stop watching first variants.

I don't know how to fix this.

Is it possible to pass configuration array to the 'run' method. Currently it only takes configuration file name(as string).
I have a scenario where I need to dynamically create configurations and kickstart webpack bundling.

I get a huge amount EventEmitter warnings when running with either Webpack 1.x or 2.x:

(node:10856) Warning: Possible EventEmitter memory leak detected. 11 SIGINT listeners added. Use emitter.setMaxListeners() to increase limit
(node:10867) Warning: Possible EventEmitter memory leak detected. 11 SIGINT listeners added. Use emitter.setMaxListeners() to increase limit
(node:10856) Warning: Possible EventEmitter memory leak detected. 11 exit listeners added. Use emitter.setMaxListeners() to increase limit
(node:10867) Warning: Possible EventEmitter memory leak detected. 11 exit listeners added. Use emitter.setMaxListeners() to increase limit
(node:10856) Warning: Possible EventEmitter memory leak detected. 11 exit listeners added. Use emitter.setMaxListeners() to increase limit
(node:10855) Warning: Possible EventEmitter memory leak detected. 11 SIGINT listeners added. Use emitter.setMaxListeners() to increase limit
(node:10867) Warning: Possible EventEmitter memory leak detected. 11 exit listeners added. Use emitter.setMaxListeners() to increase limit
(node:10855) Warning: Possible EventEmitter memory leak detected. 11 exit listeners added. Use emitter.setMaxListeners() to increase limit
(node:10856) Warning: Possible EventEmitter memory leak detected. 11 exit listeners added. Use emitter.setMaxListeners() to increase limit
(node:10849) Warning: Possible EventEmitter memory leak detected. 11 SIGINT listeners added. Use emitter.setMaxListeners() to increase limit
(node:10849) Warning: Possible EventEmitter memory leak detected. 11 exit listeners added. Use emitter.setMaxListeners() to increase limit
(node:10867) Warning: Possible EventEmitter memory leak detected. 11 exit listeners added. Use emitter.setMaxListeners() to increase limit
(node:10855) Warning: Possible EventEmitter memory leak detected. 11 exit listeners added. Use emitter.setMaxListeners() to increase limit
(node:10849) Warning: Possible EventEmitter memory leak detected. 11 exit listeners added. Use emitter.setMaxListeners() to increase limit
(node:10855) Warning: Possible EventEmitter memory leak detected. 11 exit listeners added. Use emitter.setMaxListeners() to increase limit
(node:10849) Warning: Possible EventEmitter memory leak detected. 11 exit listeners added. Use emitter.setMaxListeners() to increase limit

I'm using NodeJS 6.8.1 on Ubuntu 16.04 with parallel-webpack 1.5.0. I do not get any warnings running webpack normally.

I imagine that these items may be the cause:

• https://github.com/trivago/parallel-webpack/blob/master/src/webpackWorker.js#L135

• parallel-webpack/index.js

  Line 81 in cd78da5

  process.on('SIGINT', function() {

I suspect that there's no real issue here, but the warnings are unsightly. Very happy that it works with Webpack2.

It recompiles only the first task with --watch on Ubuntu. Windows & MacOS are good.

I use a webpack.config.babel.js as my webpack config file, and now I'm trying to use parallel-webpack with this project.

Webpack supports config files those named webpack.config.babel.js or webpack.config.coffee. As I see, this nice feature comes from js-interpret. Could parallel-webpack add interpretation for these too?

Hi there,

would be nice if I can see the filename.

current webpack config

current log

should be:

thx!

greetings
crazyx13th

Question

HELLO! How are you doing?

Provide your Environment details

• Node version: 10

• Operating System:
  Linux / Win

• webpack version:
  4

• parallel-webpack version:
  Question, just with TypeScript it start to slow down (7700k server ok, but for 3770k too slow is not good anymore for development)

My current build requires these two flags provided by the webpack binary. Is there some way I can get the same effect from parallel-webpack? Here is the full command:

$(npm bin)/webpack --colors --config config/webpack/production.config.js --json --bail 2>&1

I tried but failed to minimize my files

parallel-webpack -p=1 --optimize-minimize --progress --colors --watch -- --env.mode=build

&

parallel-webpack -p=1 --progress --colors --watch -- --optimize-minimize --env.mode=build

Any solution?

nodejs api
the ```run`` function support to use webpack.config object as well as config path.

Hey everyone,

We are in process of rewriting the workers with jest-worker in favor of simplifying the code base and getting rid of the zombie leftover process issues.

You can try and give feedback on our latest alpha release currently: 3.0.0-alpha.2. Keep in mind it is still in progress a lot of things are not working as expected.

We will drop support for some features initially. Either because of we don't have any internal usages left for it or some features that were coming from node-worker-farm does not exist on jest-worker. Some of them we may re-introduce in time. If you are relying on that features you can still use 2.x versions. PR's and feedback are very much appreciated as always.

Cheers,

Branch

Doing this would allow for a lot more freedom when it comes to configurations, for example depending on when mine is run my config slightly changes meaning that it needs to be a function that can be executed rather than simply required in which causes problems when coming to use this package.

Will try to find a way to reproduce, but often running parallel-webpack leaves behind 8 or so node processes that eat up a ton of CPU/memory.

In webpack we can specify production mode by adding a "-p" as explained in this link - https://webpack.js.org/guides/production/. What is the equivalent for it in parallel-webpack?

Explain the problem

I use karma-webpack for testing, and there is no karma-parallel-webpack plugin. My code gets built three times -- during build (using parallel-webpack), and twice more for two test suites. I can't (easily) write a karma-parallel-webpack plugin, because parallel-webpack doesn't expose the compiler/compiler.plugin API that karma-webpack relies on.

What is involved in exposing this API? It seems like it would involve lots of message-passing.

What the title says :)

parallel-webpack version 1.2.0 added support for using process.argv within the webpack.config.js by forwarding it consistently. This should serve the same purpose as the fork that @ascbfgwewe created (input welcome!) but it is currently still lacking documentation.

Hi,

Today my webpack config has around 10 entries and I use the CommonsChunkPlugin to generate a common.js file. Is it possible to use CommonsChunkPlugin with parallel-webpack?

Do I have to change my webpack config from a object with many entries to an array like you do in your example?

Best regards,
Kenneth

Pretty please?

Hi!

I use the parallel-webpack watch feature, but it seems to watch all files in my filesystem. Also, it doesn't seem to respect the setting: watchOptions.ignored in my webpack config when set.

Is there a way to get this in parallel-webpack as well? I can consider a PR when I find some time. Maybe it already supports this and I am not setting it up correctly?

Thanks!

I've seen the other two issues about webpack 2 but this one is slightly more specific;

You can now export a function as your webpack config file which is their recommended way of doing so if you want to use custom command arguements.

module.exports = function(env) {
  var customStuff = env.customStuff;
  /* ... */
  return config;
};

I believe parallel-webpack expects an object so this isn't supported. Is this something you would consider? Without knowing the code inside out I believe it's a straightforward amend.

I have an environment where I build 3 entry points and use '[name]' for output, so I set up the parallel with variants. It's working great, build time went down and the config was easily shared. Good stuff! Only downside is that in the console the processes are all logged as '[name] started ...'.

I can't really use an extra variant property (ex. name) because I'm only interested in those 3 builds, the others would only provide overhead, or am I missing something crucial here? If not, I think it would be fairly easy to implement some sort of name resolver / fallback, something like:

// webpackWorker.js
function getAppName(webpackConfig) {
    var entryName = webpackConfig.name || webpackConfig.output.filename;
    return entryName.indexOf('[name]') === -1 ? entryName : entryName.replace('[name]', Object.keys(webpackConfig.entry));
}

I am wondering if this tool allows using various kinds of module loaders such as babel-loader, I tried to use that with babel-loader, but it seems skip the loader and not understand ES6 in the build.

We can probably utilize Listr (https://www.npmjs.com/package/listr) to generate a much cleaner UI with a --verbose flag that allows switching back to the current UI.

While the information that we print is useful in some cases, most commonly the only interesting part is knowing when a build starts/is finished.

Although #1 is closed, I might believe there is a follow-up use case for a "common" chunk across all variants. So: When a given module is the same in all variants, it gets extracted to a "common" chunk. Opposed to that, the current behaviour creates one "common" chunk for each variant, dropping potential re-use of chunks across multivariant environments quite a bit.

Think of our trivago use case: A user on trivago.ae switching from the LTR to the RTL layout of that page would have to re-download each of the chunks, although only relatively little may actually be altered.

And no: I have not the slightest idea how to achieve this given the current code.

in a current project of mine, i have 12 builds running in watch mode. every now and then, when i quit watch mode (ctrl+c), some of the child processes keep staying alive, leading to a couple of them running at the end of the day.

now that i am aware of this, i kill them manually each time i stop watch mode, but obviously, that should not be the way to handle that.

can i change anything in my setup to prevent this in the first place? are there any known bugs unfortunately i cannot share any project details...

Edit: Using latest version 1.6.1

When using 'npm start' it seems that the statistics for build times are adding up on the start time of the watcher when it was initially fired up.

This leads to
[WEBPACK] Finished building es5.ltr within 787.011 seconds
[WEBPACK] Finished building es5.ltr within 811.579 seconds
even though the builds just took a few mseconds.

In dev mode, webpack dev server or webpack dev middleware delivers great performance by not writing assets to disk. However, it would be awesome if it would also compile the entry points in parallel(I have about 3 of them). Does parallel-webpack work with these 2 ? Could it be made to work with them if not ?

while running on circleci the process completely stops on parallel-webpack

it never continues . I have to cancel the build. Is there anything i am missing?

I'm really interested in using this package for bundling our client/server scripts in parallel, however, I am struggling with how to go about orchestrating running the builds, with watch enabled, and starting our server.

ex. Webpack Bundles -> Server Startup -> Watcher Start

Do you have any thoughts on this topic?

Is this built in such a way that you can only supply a path to a Webpack config file and not an object containing the configuration? Glancing at the code, it seems that way, but I wanted to ask in case I was missing something. Thanks!

right now it's not possible to use parallel-webpack with multiple function configuration,
(https://webpack.js.org/configuration/configuration-types/#exporting-multiple-configurations)
which you now need to use to pass command line arguments to webpack.
(webpack/webpack#2254 (comment)).

current behavior is throwing of Unhandled rejection TypeError.

Explain the problem

Sometimes a developer wants to access webpacks internal command line options in his config.

Expected Behaviour

The config function should pass two params into the config function:

• An environment as the first parameter. See the environment options CLI documentation for syntax examples.
• An options map (argv) as the second parameter. This describes the options passed to webpack, with keys such as output-filename and optimize-minimize.

source

Actual Behaviour

parallel-webpack omits the second param (argv).

Provide your webpack config

module.exports = (env, argv) => {
     console.log('isEnvSet', !!env)
     console.log('isArgvSet', !!argv)
     return {}
}

Steps to reproduce

Use the config above and run:

parallel-webpack --config=webpack.config.js

Provide your Environment details

• Node version: v9.1.0
• Operating System: MacOS 10.13.3
• webpack version: 3.11.0
• parallel-webpack version: 2.3.0

After upgrading to 1.6.0 from 1.3.1 my build (with --watch arg) restarts in the background when I ctrl+c the currently running one. BTW I'm on webpack 2.2.0-rc.2.

It would be nice, if you could also add an time stamp for the current build to the console output for the continously rebuilds via the watcher. If you running the watcher for a long time, then it is easy to loose the overview of your rebuilds.
Sometimes I am not sure when I am touching some code and the watcher is turned on, if the latest changes are already built or are still in building.

Best regards,
Chris

Hi!

Thanks for this package, it seems quite nice and we've sometimes used it in our projects.

It would help a lot in upgrading if you had a changelog I could view. You can generate one with https://github.com/skywinder/github-changelog-generator to provide some baseline which wouldn't take much time.

When a module is missing:

 $ Unhandled rejection (<{"message":"\u001b[31m[WEBPACK]\u001b[...>, no stack trace)

Hi,I have test this on my company production jenkins env,But it seems that --bail option do not work,So errors happen and build not break and react as success.T T

You subscribe on SIGINT event in the main run function and inside every worker, but never call removeListener.

At my organization, we currently parallelize four configs. It has resulted in console outputs multiplied by 4 when building with webpack or starting a watch task.
I managed to make most of the logs silent with the json option. But, when shutting down the process with Ctrl+C (SIGINT), I still get 4 log messages that bleed out in the next prompt.

Worker Farm: Received message for unknown child. This is likely as a result of premature child death, the operation will have been re-queued.
Worker Farm: Received message for unknown child. This is likely as a result of premature child death, the operation will have been re-queued.
Worker Farm: Received message for unknown child. This is likely as a result of premature child death, the operation will have been re-queued.
➜  my-app git:(yg/better-logs) ✗ Worker Farm: Received message for unknown child. This is likely as a result of premature child death, the operation will have been re-queued.

Would it be possible to make those silent as well?