Proposed overhaul to be more URL-based about import-maps

[domenic]

Significant progress on the rewritten document today, as a baseline. Still more work to do before I'd feel comfortable merging into master and re-triaging all the issues based on that, but for the insiders subscribed to this thread, feel free to take a look.

[domenic]

Yeah, tentatively "module import maps" instead of "package name maps" since it's now focused more on controlling imports in general and deemphasizing the package concept.

[robpalme]

Scope

I don't mind expanding the scope of PNMs to include fallbacks. It helps to have all resolution control in one place. It's good to have things that improve runtime efficiency by making more information available early, and help tooling understand whats going on by centralizing the information.

Observability

In a module loaded via a fallback, I assume the importee module can discover its resolved/physical identity via import.meta.url. Will there be a builtin way for the importer module to know what they loaded? Given we're introducing non-determinism in loaded modules when using an array of mappings, we'll need debugging/telemetry to remotely report on what modules your app truly loaded for a particular run.

Bare specifier definition

the import: part will get auto-appended for you, when you use a bare import specifier.

The examples show mappings with absolute URLs on the left-hand side, e.g. "/node_modules/als-polyfill". Is a leading forward-slash still considered a bare import specifier?

Extensions

On the file extension drawback, I have a mild personal preference to lean in and drop _class-of-URLs_mappings. It promotes deep imports into arbitrary files within external packages, that may or may not be considered implementation details. I think it best to surface inter-package dependencies in the PNM to aid static analysis. My expectation is that PNMs will primarily be tool-generated (e.g. as a pre-deployment step), so I'm not worried about extra lines.

Misc

Two typos:

• auto-appended -> auto-prepended
• right-right -> right-

[ljharb]

So you can default to your own implementation but use module maps to gracefully replace it with the builtin when available, is my understanding.

[kinu]

The proposal looks exciting. Let me list some of the comments/questions I had (some might have been already discussed or explained).

• This says that it should be able to used to fetch any resources, which seems to imply that this is actually more about 'fetch resource map' rather than import map, is my interpretation right?
• It's a little unclear what should happen if the import URL is used for navigations or worker / service worker instantiations.
• Related, how the origin concept should be handled with this URL?
• It's already mentioned but it's unclear how importmap should be specified for modules in/with workers, and then for service workers.
• What should happen if a resource loading triggered in a context where a importmap is set up redirects to an import: URL?
• How should the import: URL loading interact with Service Workers?
• How should it work with preloads/prefetches that come with link headers?

[guybedford]

Very exciting to see these directions being embraced.

arrays of the above, which result in trying each URL in sequence and falling back on network error or non-ok fetching status.

This is absolutely amazing, and much nicer than the previous fallbacks proposal.

Now for my concerns :)

The two arguments for URL mapping seem to be (1) standard module are URLs and (2) For use case (C) we want to support URL mapping.

I don't think this is an adequate justification for moving to the more complex URL -> URL mapping.

Recursion gets really complex fast, and cycles are a real problem here too, so we need to have a really good reason to embrace this direction.

1. Standard modules are URLs

You're introducing an indirection anyway here as well with an index for the standard modules to distinguish anyway.

If users just use a package map to map into the standard module URL instead of writing a URL (say import @std/async-local-storage), then the package name map can solve use case B fine.

2. URL mapping for browsers without package map support (use case C)

This feels to me like trying to do too much in one go. The web traditionally lets the ecosystem handle compatibility problems, and instead focuses on the future workflows. If you let go of this constraint, then I think you will find the complexities of the proposal can be avoided.

[domenic]

Thanks both for the comments!

@robpalme

Observability

An interesting question. I think the platform has some things that help with this, such as the resource timing API. Ideally I'd like to reuse those.

The examples show mappings with absolute URLs on the left-hand side, e.g. "/node_modules/als-polyfill". Is a leading forward-slash still considered a bare import specifier?

No. In this scheme, you import URLs, always. And package name map left-hand sides are URLs. So "/node_modules/als-polyfill" is a URL, just like "import:lodash" is. Bare import specifiers are no longer really a thing, except that there's that auto-prepending sugar which gives you the same effect.

Extensions

Interesting viewpoint. On reflection, I think I agree. That makes this all a lot simpler, which helps address some of @guybedford's concerns. We can just have string left hand sides, and pure URL right-hand sides.

Two typos:

Fixed, thanks!

@guybedford

This is absolutely amazing, and much nicer than the previous fallbacks proposal.

Glad to hear it! Do you have opinions on whether x: [x, y] is too confusing, and maybe we should do a dedicated fallbacks section instead?

Recursion gets really complex fast, and cycles are a real problem here too, so we need to have a really good reason to embrace this direction.

I don't think this is a big of a deal as you say. The algorithm is fairly straightforward, from some initial prototyping; I hope I can work on a pull request that shows that more concretely.

Standard modules are URLs

I don't think this is an accurate phrasing of the argument. The issue is that every module that is eventually requested is a URL. If we say that standard modules are special, then we can no longer use a general mechanism---like package name maps---for manipulating them.

URL mapping for browsers without package map support (use case C) ... If you let go of this constraint, then I think you will find the complexities of the proposal can be avoided.

Unfortunately this is a hard constraint we must meet. So I think it's best we work on a way to meet it while avoiding the complexity you're worried about. @robpalme's simplification might help in that regard... I'll start writing it up.

[guybedford]

Glad to hear it! Do you have opinions on whether x: [x, y] is too confusing, and maybe we should do a dedicated fallbacks section instead?

I really like this form, and I think the way it expands is quite predictable and intutitive.

I don't think this is a big of a deal as you say. The algorithm is fairly straightforward, from some initial prototyping; I hope I can work on a pull request that shows that more concretely.

Do you allow infinite loops? Eg:

{
  packages: {
    "/x": {
      "path": "/x/x"
    }
  }
}

how do you know when to stop the recursion in this scenario?

Standard modules are URLs

I don't think this is an accurate phrasing of the argument. The issue is that every module that is eventually requested is a URL. If we say that standard modules are special, then we can no longer use a general mechanism---like package name maps---for manipulating them.

Why should we be able to map standard modules, when indirection is fine?

Unfortunately this is a hard constraint we must meet.

Who is "we"? Don't change the web to solve things it doesn't need to solve. Why not go the tried and tested polyfill route rather?

[domenic]

how do you know when to stop the recursion in this scenario?

The same way you usually do? Keeping track of where you've been?

Why should we be able to map standard modules, when indirection is fine?

I don't understand what "indirection" is in this instance, or why it is fine.

Who is "we"? Don't change the web to solve things it doesn't need to solve.

"We" is the proposal authors and implementers. We are not interested in working on package name maps if they don't also solve the important use cases for built-in modules. I understand you might not think this is important to solve, but from talking with our customers and partners we've found that without a solution for these cases, there's not enough interest in this feature to be worth our devleopment effort.

Why not go the tried and tested polyfill route rather?

I don't understand what this means, as this whole proposal is about enabling the polyfill route to even be possible.

[domenic]

Writing up the proposal in more detail I've run into one fly in the ointment. It's weird that package name maps allow remapping of all URLs, but only the remapping of import: URLs works outside of import statements and import() expression contexts. Trying to explain this is quite weird, because I talk about the benefits of import: URLs and how they are usable everywhere, and then I talk about how you configure them using package name maps, but note that you can also configure other URLs using package name maps, and those aren't usable everywhere...

Not sure what to do with this. When the namespaces were separate, import: meant "use module specifier resolution", and that story was pretty easy. Now that there's just the URL namespace, but applied unevenly, it's more troubling. Might need to back up a few steps... Sigh.

Edit: current potential solution I'm thinking of is to say that (a) the package name map only changes import: URL resolution, even in import and import(); but (b) import: always gets auto-prepended to import and import() statements. So, contrary to what I said to Rob above, import "/node_modules/als-polyfill/index.mjs" is actually equivalent to import "import:/node_modules/als-polyfill/index.mjs". We'll see what that looks like. In this version you wouldn't use import: prefixes in the package name map.

[domenic]

Started on a writeup, but ran out of time for today. Still, should give the basic idea, for those very actively engaged: https://github.com/domenic/package-name-maps/blob/url-based/README.md

[littledan]

I'm a big fan of just about everything in this iteration of the proposal, and agree with @domenic that polyfilling built-in modules is very important. This proposal seems to work well for both the case of an entirely missing built-in module, and the ability to wrap a built-in module in another module (e.g., in case a function is added, and you want to polyfill it in old browsers; see tc39/proposal-built-in-modules#2).

Potential solution: lean into it, and get rid of the class-of-URLs-to-class of URLs mapping entirely? I.e. make everything 1:1, so you'd need to enumerate the submodules of each package, both for LAPIs and non-LAPIs.

I'm wondering whether there's something halfway to that point: Could we say that, if you use a string, you're indicating just a single mapping, and then the object can be used to permit submodules as well, if we want that feature?

[daKmoR]

To be honest, when this first started I was a little hesitant as the "current" implementation with path_prefix, packages, scopes, main and everything looked really complicated.

However, this new flatter approach is sooo much easier to understand, generate and I hope applying it would also be not too complex.

So, contrary to what I said to Rob above, import "/node_modules/als-polyfill/index.mjs" is actually equivalent to import "import:/node_modules/als-polyfill/index.mjs"

So especially in this usecase the "import:" would do nothing right? as it's followed by a /. If it where
import:foo/bar then it would actually look up foo in the map. Just wanna be sure about my understanding.

@guybedford I'm curious if you would consider this "new" approch in your shim maybe in a branch? I would like to play around with a generator of this map... most likely on top of yarns "beta" plug and play implementation

[MylesBorins]

This looks great! Any expectation for a name change?

…

On Tue, Oct 2, 2018, 8:48 PM Domenic Denicola ***@***.***> wrote: Significant progress on the rewritten document today, as a baseline. Still more work to do before I'd feel comfortable merging into master and re-triaging all the issues based on that, but for the insiders subscribed to this thread, feel free to take a look <https://github.com/domenic/package-name-maps/blob/336ccbcc2000235733ce3fd93b98934b47882c8e/README.md> . — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#53 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAecV9H7CO-T8u4LF8oiFcnzZBaZmbleks5uhAlbgaJpZM4WEvVh> .

[ljharb]

@domenic This looks great! Not sure where to provide feedback:

• re https://github.com/domenic/package-name-maps/blob/url-based/README.md#denying-access-to-a-built-in-module, since import 'nonexistent/path'; wouldn't throw with @std/blank, it seems like @std/thrower is a better default implementation - or perhaps, @std/parseError?
• re https://github.com/domenic/package-name-maps/blob/url-based/README.md#installation, what happens if i try to dynamically add a map to the page with JS later? I'd hope/expect that would throw.
• re extension lookup, it's quite apparent that browsers can't do dynamic lookups, but in the same vein as "ends with a /", is there something we could do like a "foo/bar" scope under which all paths get a .js added? Certainly this can be handled by build tooling, but it might be nice to allow for (static, predetermined, hardcoded) extension/index resolution.
• re import:, none of the examples seem to make it explicit - can you import 'import:blah' the same as import 'blah'? Is the duality desirable, or would there be a way to prevent import: from being present in JS contexts?

[michael-ciniawsky]

What's the reasoning behind import:./x, import:../y, import:/z? Can't those not just use a 'bare specifier' aswell and left as is otherwise?

url('bare/logo')
<link href="bare/style">
import module from 'bare'

{
   modules: {
      bare: {
         '/': '/path/to/bare', // Path (Context)
         index: './lib/main.js',
         style: './lib/css/main.css',
         logo: './lib/assets/logo.svg',
         ...
      }
   }
}

[domenic]

@michael-ciniawsky I don't really understand your example, but the reasoning is to support https://github.com/domenic/package-name-maps/blob/url-based/README.md#for-built-in-modules-in-browsers-without-module-import-maps

[michael-ciniawsky]

What is unclear can you elaborate?

[michael-ciniawsky]

Why would pollyfilling need support for ./, ../, / in particular?

{
  "modules": {
    bare: {
      index: [
        "import:@std/buildin",
        "/node_modules/polyfill/index.js",
        "https://cdn.fallbacks.com/polyfill/index.js"
      ]
    }
  }
}

[robpalme]

This new version is good. The trailing / convention is neat.

• Typo: import:query -> import:jquery
• Defining the right-hand side to be non-bare seems like a good idea to add clarity, i.e. forcing a prefix of ./ ../ / or import: Support for bare rvalues can always be introduced later.
• None of the JS code examples show the use of explicitly prepending import: to a specifier. I assume this is legal and provides identical resolution to not specifying it. I suggest adding an example.
• The existence of @std/blank seems a bit arbitrary. Blacklisting (denying) a specifier is a first-class use-case so deserves first-class syntax in my opinion. An empty string would work, right?
• you'd better be using HTTP/2 push to get that thing to us - maybe say "the client" rather that "us".
• Typo: succesfully
• Typos: othre
• I'd like to see some statement in the document on how to programatically reflect on the results of the fallback-array non-determinism. You've previously stated we might be able to reuse existing techniques.

[dcleao]

I like the uniformity and power of the proposed import: scheme, as it would be usable in all contexts where a url can be specified.

• If you load a module from a CDN that uses bare import specifiers, you'll need to know ahead of time what bare import specifiers that module adds to your app, and include them in your application's module import map. (That is, you need to know what all of your application's transitive dependencies are.) It's important that control of which URLs are use for each package stay in control of the application author, so they can holistically manage versioning and sharing of modules.

  While it is great to be able to have control, I believe it should only be an option. I should not need to know what are the libraries that a library that I depend on uses for its own implementation.

• I don't have a preference, but I wonder what are the gains of the"/" syntax when comparing it with the "main" property alternative. It does get more verbose.

[littledan]

The newest version looks even better. Two small comments:

• Interesting example for an initial self-reference of the name when expressing fallbacks for built-in modules. I'm wondering, do you have an idea for the logic to prevent an infinite loop when resolving this case? (I can think of more than one possibility.)
• It sounds like you've gone back towards remapping arbitrary URLs, not just "bare" specifiers. I'm happy with the design decisions this and other proposals have made in this space, but curious about if this opens up other possibilities, e.g.,
  • (Although I personally prefer @std/) Would it work out to use std: as the prefix, and permit remapping of this pseudo-scheme as well?
  • Should bare specifiers like 'bare.mjs' in module imports actually have the semantics of'./bare.mjs', if the two are equally remappable?

[nyaxt]

The latest draft lgtm.

FYI: @yutakahirano

I think we need to clarify:

• How the extended import URL scheme maps to SecurityOrigin.
• The scope resolution algorithm: Given the map and a context script URL, resolve scope.
  • IIRC, the advantage of the explicit package based mapping was that scope resolution behavior is intuitive.
  • Giving it an another thought, may be it wouldn't be a blocker as long as the algorithm is deterministic.
• "sensitivity to the current script file"
  • Is this the referencing script's base URL == the script's response URL?

Would it make sense to implement this by having import: URLs issue redirect to the resolved URLs?

[RangerMauve]

How would this new import: scheme work with the work being done on changing navigator.registerProtocolHandler to use a blocklist instead of an allow list?

I assume import: would need to be added to the blocklist.