Named Modules about systemjs

[guybedford]

I initially attempted an implementation of named defines, but it turned out to be a harder problem than I had hoped. I think it is still possible, but it will be a lot of work to get there.

Ideally bundling happens at the protocol level itself using SPDY push. This allows caching of resources that formerly needed to be inlined. I do believe that approaches like this are far better as repeat visitors to an app only need to download the changed modules if they have this fine-grained caching.

In terms of supporting bundling like it is done today, I see two ways of handling this:

1. Create successive Loader.set / Loader.get statements:

  System.set('first-module', (function(__exports) {
    var dep1 = System.get('dependency');

    __exports.firstExport = function() {}

    return new Module(__exports);
  }({});

  System.set('second-module', ...

1. Use the Loader.define method:

  System.define('my-module', "var p = 4; exports.q = 4");
  System.define('amd-module', "define(function() { return {}; })";
  System.define('es6-module', "import 'amd-module'; export class q {}");

I think the Loader.define method will probably become the most successful way of doing this, although string sources may seem ugly at face value, ultimately it offers the most flexibility.

[EisenbergEffect]

I tested the System.set/System.get approach. It only works if all your modules are defined this way. In my case I want to define several modules up front but allow other modules to use AMD,, CJS, and ES6 formats to require them. In those cases, it breaks as it tries to make a network request for modules that it already has.

I haven't yet tested this with the System.define method...as that's a real pain to try out for me at the moment. I'll look into it in another week or so...

It seems to me that the loader should first check to see if a module exists for the non-normalized name, before it makes any transformations. If it does exists, then don't do anything. Would something like that work?

[guybedford]

What name are you defining, and what name are you requiring that this isn't working?

The approach definitely works -

  jspm.set('jquery', { hello: 'world' })
  jspm.import('jquery', function($)   { console.log($) } )

Note that we need to use jspm.set for the jspm loader as it is a separate loader from the System loader.

[probins]

As I understand the ES6 spec, you bundle modules in one file with:

module "foo" {
  export ...
}
module "bar" {
  export ...
}

which is equivalent to html (static):

<module id="foo">...
<module id="bar">...

and to (dynamic):

loader.define("foo",...
loader.define("bar",...

and to current named/bundled modules in RequireJS/AMD.

If your module declarations are in a file foobar.js, foobar could itself be a module, in which case you would import it, or you could just load it in a script tag like any other bundle. If you import it, then the module registry will have entries for foobar as well as foo and bar; this is also the same as RequireJS at the moment.

However, it looks like Traceur doesn't support named modules at the moment either, so we can't use this ES6 syntax.

[guybedford]

The module syntax of:

  module "foo" {
    export ...
  }

Is no longer in the specification, and the loader.define method has replaced this now.

As for loading a module that contains define statements, this should work fine as long as the module does not try to define its own name. There may be room for conflict there, and I'm not sure this has been considered in the spec. Instead of defining its own name, the bundle could just use an export statement to indicate its own export where appropriate though.

[guybedford]

Note the define function is yet to be supported in the ES6 Module Loader polyfill. I'm working on this soon.

[probins]

no longer in the specification

that may be why Traceur doesn't support it :-)
But I'm not sure that's a good idea. If you put defines in the bundle, then you are specifying the loader and limiting yourself to that loader, aren't you? e.g. System.define or jspm.define. If you use jspm.import on a bundle that contains System.define, your bundle would be put in a different registry from the defined modules.
Having a neutral module "foo" seems a better solution to me.

As for loading a module that contains define statements, this should work fine as long as the module does not try to define its own name. There may be room for conflict there, and I'm not sure this has been considered in the spec. Instead of defining its own name, the bundle could just use an export statement to indicate its own export where appropriate though.

but it wouldn't normally need it, would it? The bundler would have to add a define wrapper to each module, but the bundle itself would not export anything, would it? You could just load it with a script tag (unless you need/want dynamic loading).

[guybedford]

The nature of custom loaders is that custom loaders won't necessarily share the same canonical normalized names.

For example, in jspm the canonical name is always detemined by the endpoint, making the defines of the form:

  jspm.define('github:some/[email protected]', "...");

I think the bundling tool can be shared taking the server version of the loader as the argument (jspm can run in NodeJS allowing for full normalization and module traces). Then the bundling tool would create the bundle for the loader given the input module and bundling optimization options.

I think that's a good compromise.

Yes you are right that we can potentially end up with an ecosystem of loaders here. Personally I think that would be a good thing (within reason), and that is why I've tried to extend the package.json configuration instead of having jspm-only configuration to potentially share this.

For the bundling, sometimes it can be useful to bundle the dependencies of a module into the module itself, making the bundle itself a module. In this scenario, the bundle might need to export, but yes it is an edge case.

[EisenbergEffect]

Bingo. Using jspm.set/jspm.get I was able to get it working. I had a couple of ridiculous bugs on my end which didn't help the situation ;) This is looking good for my needs in terms of framework implementation. However, we'll definitely need a way to write a simple optimizer that can inline arbitrary modules into a single file. It sounds like jspm.define will be the way to go once it's implemented.

[guybedford]

Great. To give a timescale to things a little - I'm knee deep in testing libraries for jspm at the moment. I expect the new polyfill rewrite to land early Jan, and then I'm keen to work on seeing if this bundling optimization can be done within the Traceur project including AMD transpilation. So the earliest I see it happening is in Feb unfortunately (unless someone else gets to it sooner).

In the mean time, all we have is SPDY push through the CDN, which is working ok for us mostly (it's on the jspm website and also http://getkickstrap.com/).

If there is anything else I can do in the mean time to help with your project just let me know.

[EisenbergEffect]

That timeline certainly works great for me. Can you elaborate on the "new"
pollyfill and how it differs from the current one?

On Sun, Dec 1, 2013 at 1:06 PM, Guy Bedford [email protected]:

Great. To give a timescale to things a little - I'm knee deep in testing
libraries for jspm at the moment. I expect the new polyfill rewrite to land
early Jan, and then I'm keen to work on seeing if this bundling
optimization can be done within the Traceur project including AMD
transpilation. So the earliest I see it happening is in Feb unfortunately
(unless someone else gets to it sooner).

In the mean time, all we have is SPDY push through the CDN, which is
working ok for us mostly (it's on the jspm website and also
http://getkickstrap.com/).

If there is anything else I can do in the mean time to help with your
project just let me know.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/31#issuecomment-29578983
.

Rob Eisenberg,
President - Blue Spire
www.durandaljs.com

[guybedford]

The polyfill is currently built to this spec - http://wiki.ecmascript.org/doku.php?id=harmony:modules

And needs to be updated to this spec - https://people.mozilla.org/~jorendorff/js-loaders/Loader.html

It will be quite a change, but the jspm API will remain the same.

[EisenbergEffect]

Cool. I had stumbled across the new spec recently but hadn't realized it
was actually different from what is on the ecmascript wiki, so I didn't pay
close attention at the time. After having a quick look at it, it seems to
be looking very good. I'm glad that it seems to be embracing promises. I
have a couple of quick questions:

1. When the new implementation is working (Feb-ish) will that include the
   most up to date promises implementation as linked in the loader spec?
2. Is the new Loader.prototype.instantiate ( load ) hook capable of doing
   tracing, tagging, etc. that we discussed in another issue?

On Mon, Dec 2, 2013 at 4:32 AM, Guy Bedford [email protected]:

The polyfill is currently built to this spec -
http://wiki.ecmascript.org/doku.php?id=harmony:modules

And needs to be updated to this spec -
https://people.mozilla.org/~jorendorff/js-loaders/Loader.html

It will be quite a change, but the jspm API will remain the same.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/31#issuecomment-29604244
.

Rob Eisenberg,
President - Blue Spire
www.durandaljs.com

[guybedford]

Thanks for the points - keeps me on my toes!

1. I have no idea what to do about promises - ideally we'd need a polyfill for that as well. My hope is to see if it is possible to partially implement promises in a way that is compatible with the spec, but without being a full implementation. Not sure how this will work out yet.
2. There will be no tracing or post-execution hooks possible in the ES6 loader. Only pre-execution hooks are possible. ES6 loads are "declarative", while AMD and CommonJS loads are "dynamic". The main reason for this underneath is that circular dependency binding is supported by the loader, so the execution function is abstracted away pretty deeply in the ES6 scenario. Unfortunately the previous issue about exposing it would be such a big change to this part of the implementation that it looks certain not to happen.

[EisenbergEffect]

Can you explain what the instantiate hook is then? I saw that in the spec
and didn't remember it being there before. I was hoping that would give me
access to the "created" module and its metadata. That's what it looked
like, but I wasn't sure.

On Tue, Dec 3, 2013 at 5:21 AM, Guy Bedford [email protected]:

Thanks for the points - keeps me on my toes!

1. I have no idea what to do about promises - ideally we'd need a
   polyfill for that as well. My hope is to see if it is possible to partially
   implement promises in a way that is compatible with the spec, but without
   being a full implementation. Not sure how this will work out yet.
2. There will be no tracing or post-execution hooks possible in the
   ES6 loader. Only pre-execution hooks are possible. ES6 loads are
   "declarative", while AMD and CommonJS loads are "dynamic". The main reason
   for this underneath is that circular dependency binding is supported by the
   loader, so the execution function is abstracted away pretty deeply in the
   ES6 scenario. Unfortunately the point about exposing it would be such a big
   change to this part of the implementation that it looks certain not to
   happen.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/31#issuecomment-29698046
.

Rob Eisenberg,
President - Blue Spire
www.durandaljs.com

[guybedford]

The instantiate hook allows the standard execution of modules to be overridden specifically to support AMD and CommonJS modules.

This is done by returning an object:

return {
  imports: ['some', 'modules'],
  execute: function() {
    // we have full control of execution here
    eval(source);
  }
}:

This is the "dynamic" linking.

If it returns undefined, then "declarative" linking is used, which is the case for ES6 modules.

In this scenario we have no control over the execution function as in the dynamic scenario. Since this is the last hook, no further information / meta can be added.

My suggestion was to expose the ES6 linking just like the dynamic linking, but because ES6 supports circular references, it isn't as straightforward as the dynamic linking case.

[EisenbergEffect]

Hmmm. Lack of a final hook that exposes constructed modules is really going
to hurt my project...as in kill one of it's major unique features. I think
it's worth investigating adding another hook that just exposes the result
of a module instantiation regardless of the path by which it is
instantiated. This seems super trivial to do...just call an activated or
buildFinished hook before or immediately after adding it to the internal
registry.

I can manage to work around this is I have a way to obtain a module name,
given a module instance (for my specific scenarios). I thought I saw some
mechanisms for enumerating the internal registry of the loader that would
allow this. Can you confirm that?

On Tue, Dec 3, 2013 at 8:28 AM, Guy Bedford [email protected]:

The instantiate hook allows the standard execution of modules to be
overridden specifically to support AMD and CommonJS modules.

This is done by returning an object:

return {
imports: ['some', 'modules'],
execute: function() {
// we have full control of execution here
eval(source);
}}:

This is the "dynamic" linking.

If it returns undefined, then "declarative" linking is used, which is the
case for ES6 modules.

In this scenario we have no control over the execution function as in the
dynamic scenario. Since this is the last hook, no further information /
meta can be added.

My suggestion was to expose the ES6 linking just like the dynamic linking,
but because ES6 supports circular references, it isn't as straightforward
as the dynamic linking case.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/31#issuecomment-29708998
.

Rob Eisenberg,
President - Blue Spire
www.durandaljs.com

[guybedford]

Yes the module registry does provide iteration through Loader.prototype.entries, Loader.prototype.keys and Loader.prototype.values.

A final hook would be trivial to add, and may also solve the tracing problem if it were of the form:

  post_instantiate_hook: function(name, imports, module) {

  }

That would allow both tracing and adjustments.

[EisenbergEffect]

I can work with the iteration. That would be fine. However, I know I'm not
the only one who would see value in a post_instantiate_hook. I really
think it's worth serious consideration. Thanks for hearing me out either
way :) I'm thankful you and others have been so receptive.

On Tue, Dec 3, 2013 at 8:48 AM, Guy Bedford [email protected]:

Yes the module registry does provide iteration through
Loader.prototype.entries, Loader.prototype.keys and
Loader.prototype.values.

A final hook would be trivial to add, and may also solve the tracing
problem if it were of the form:

post_instantiate_hook: function(name, imports, module) {

}

That would allow both tracing and adjustments.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/31#issuecomment-29710393
.

Rob Eisenberg,
President - Blue Spire
www.durandaljs.com

[probins]

I have no idea what to do about promises - ideally we'd need a polyfill for that as well

do you know Q? https://github.com/kriskowal/q

[guybedford]

Thanks, I actually ended up going with this library
https://github.com/RubenVerborgh/promiscuous as it is small enough to embed
in the code base.

On 14 December 2013 18:01, Peter Robins [email protected] wrote:

I have no idea what to do about promises - ideally we'd need a polyfill
for that as well

do you know Q? https://github.com/kriskowal/q

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/31#issuecomment-30577220
.

[guybedford]

This is now supported in the new rebranding of the project.