whatwg / fetch Goto Github PK

We have an internal implementation of this in Gecko, but it would be nice if we could expose that to script.

I'm proposing to add support for (federated) Token Binding to fetch.

I've written a requirements/problem-description document for a Token Binding DOM API: https://docs.google.com/document/d/1x_3KI_v35NQFEsAbGGjNcyTw3Waoy1mL3YfTaQbyyuM/view?rm=minimal&pli=1

While I have included a strawman proposal in the doc, I'd appreciate help from the fetch editors/implementers to figure out what the right approach is.

current fetch api spec says

it is currently lacking when it comes to reporting progression.

I think this solves when response will supports stream api of whatwg.
but there is no spec about that, only returns promise and resolve only once.

do you currently have any idea about this ?
or any point of discussion about this ?

The trailer semantics in RFC 7230 (and also since 2616) provides an ability to send/receive a meta-data out of and after the body part.

HTTP2 also supports it though the chunked TE has been deprecated.

gRPC (https://github.com/grpc/grpc) utilizes it and other applications may also benefit from it.

For XHR2, there was some discussion about trailer support in 2010: https://lists.w3.org/Archives/Public/public-webapps/2010OctDec/thread.html#msg163. @mnot well described that together with observation such as increasing use of the trailers.

Anne declined it at that time https://lists.w3.org/Archives/Public/public-webapps/2010OctDec/0233.html. But maybe it's good to revisit this needs in the context of the Fetch API standardization. Introduction of interfaces to access the trailer may affect streams integration design. So, we should start thinking how it would be like asap if we support it.

dictionary RequestInit {
  ...
  Promise<HeadersInit> trailer;
  ...
};

interface Request {
  ...
  readonly attribute Promise<Headers> trailer;
  ...
};

dictionary ResponseInit {
  ...
  Promise<HeadersInit> trailer;
  ...
};

interface Response {
  ...
  readonly attribute Promise<Headers> trailer;
  ...
};

Since XMLHttpRequest.prototype.responseXML used to exist, I guess response.xml() should exist too, right?

Florent

We should stipulate in the specification what happens there.

Browsers have default "busy indicators" that provide feedback to users that the page is loading, eg, the tab icon, status bar, and reload icon. These busy indicators are triggered in some situations (eg, clicking a link), but not others (eg, issuing an XHR). (See http://www.stevesouders.com/blog/2013/06/16/browser-busy-indicators/.)

In the absence of these busy indicators, users are uncertain and thus anxious about the status of pages loading. If the XHR response is slow to arrive and its content is critical to the user experience, the user has no idea that anything is happening. This problem grows as more pages adopt XHRs for core functionality.

A good example is this article from eBay about how their XHR-based single-page-web-app was perceived as slow until they added a progress bar (http://calendar.perfplanet.com/2014/the-power-of-perceived-performance/). That article also mentions Twitter's approach of adding a throbber to the page when XHRs are fetching tweets.

While requiring developers to add custom progress indicators is a workaround to this problem, it's not ideal because 1) it's more work for developers and 2) each progress indicator is a custom implementation and thus there is not a typical experience for the user that they are trained to look for and expect.

Since users are "trained" to look for the typical busy indicators, these indicators should be triggered when an XHR is issued for content the user is waiting on. However, XHRs are also used for background requests where the busy indicators should NOT be triggered. I suggest we add an option to XMLHttpRequest, eg, showBusy. Setting showBusy=true would explicitly tell the browser to trigger the busy indicators. This would allow for the current behavior (no busy indicators) to be preserved, but also allows an easier and more standard way for developers to use XHR while giving progress feedback to users.

https://fetch.spec.whatwg.org/#http-fetch

1. Let response be the result of performing a CORS preflight fetch using request.
2. If response is a network error, return response.

response is already used for a variable introduced in step 1, which I think this variable is supposed to shadow. It would be clearer to use "preflight response", say.

If you manage all network activity yourself you still need a way to inject the results into existing APIs that consume them. There's two approaches here that I know of:

<img>.srcObject = responseInstance
<img>.src = convertResponseToURL(responseInstance)

The first is more elegant, the second invokes Fetch twice (but the convertResponseToURL() URL should be a way shorter trip). If we go with the second, which scales better due to the many APIs in existence, we need to avoid the flaws from blob URLs and make sure the URL can be used only once and consumes the underlying stream (you cannot use responseInstance again, if you wanted to do that you'd have to clone it first).

Currently JSON.stringify() returns simply {} for Request and Response objects. It might be nice to define serializers for these classes to help developers debug their code.

http://heycam.github.io/webidl/#idl-serializers

In Chrome Canary, if searching for a non-existent pattern via https://domenic.github.io/streams-demo/, memory usage increases from ~65MB to ~155MB (as measured by the Task Manager on OS X). That is, the Chrome process seems to grow by about 90MB for each 1000MB downloaded.

(Perhaps this is because the standard method of processing "infinite" streams via a recursive calls seems to require at least two (?) stack frames per read() chunk. Is there any TCO?)

Current list of contexts contains prefetch but omits prerender and preload, which need to be added to the list. I'm thinking...

• preload -> connect-src (see w3c/preload#17 (comment))
• prefetch -> no CSP policy? Is there a reason? Should it be connect-src as well?
• prerender -> intuitively, seems that it should be the same as prefetch.

/cc @yoavweiss @mikewest

Currently all fetches will tee the request body due to these three reasons:

1. Redirects
2. HTTP authentication
3. HTTP proxy authentication

1 can be disabled through redirect: "error". 2 is already disabled for fetch() (handling it automatically and prompting the user that is). 3 cannot be disabled at the moment.

The question is whether it would be acceptable to provide an option to error on 3 as well and thereby opening up the ability to make a client-side decision not to tee request's body.

... or leave as is. So this is now not a off-topic. Please restore all our comments here.

It seems to some extent the new cache mode feature can be emulated using timing information. However, it is unclear whether that is sufficient justification to expose the information with more certainty.

Sharing whether the user has an entry in its HTTP cache does allow for fingerprinting and figuring out what sites a user visits. (Note that CORS doesn't really change this as far as I can tell.)

CC: @slightlyoff

The current specification for an opaque filtered response requires that the status always be set to 0. Could this be relaxed?

I understand that the restriction is in place to prevent leaking information that wouldn't otherwise be available for non-CORS requests/responses, but exposing either the actual HTTP response code or a boolean success/error would be very useful, and may not(*) leak more information that could already be inferred via the onerror/onload handlers for elements like <img> and <link>.

As it's currently implemented, when fetch()ing a non-CORS request from the context of a Service Worker, it's not possible to make an informed decisions about whether the filtered opaque response should be cached or not. Blindly caching a (possibly transient) error response isn't ideal, but the other extreme of not caching any filtered opaque responses runs counter to the utility of Service Workers.

(*It's entirely possible that I'm missing some important privacy implications.)

Goal

Provide developers with a method to abort something initiated with fetch() in a way that is not overly complicated.

Previous discussion

• #20
• w3c/ServiceWorker#592
• w3c/ServiceWorker#625
• whatwg/streams#297
• tc39/proposal-cancelable-promises#4 (security issues)

Viable solutions

We have two contenders. Either fetch() returns an object that is more than a promise going forward or fetch() is passed something, either an object or a callback that gets handed an object.

A promise-subclass

In order to not clash with cancelable promises (if they ever materialize) we should pick a somewhat unique method name for abortion. I think terminate() would fit that bill.

var f = fetch(url)
f.terminate()

Note: the Twitter-sphere seemed somewhat confused about the capabilities of this method. It would most certainly terminate any ongoing stream activity as well. It's not limited to the "lifetime" of the promise.

A controller

The limited discussion on es-discuss https://esdiscuss.org/topic/cancelable-promises seemed to favor a controller. There are two flavors that keep coming back. Upfront construction:

var c = new FetchController
fetch(url, {controller: c})
c.abort()

Revealing constructor pattern:

fetch(url, {controller: c => c.abort()})

Open issues

• What is the effect on the promise? Both forever-pending and explicit rejection have reasonable arguments. We could offer the choice to the developer, but what should be the default?
• What is the effect on the stream? I suspect the Streams Standard is already conclusive on this.
• What syntax of the above two-three solutions do we favor?

When getting a response back from the service worker, need to operate on internal response.

It could be added since XHR2 support PATCH method
https://fetch.spec.whatwg.org/#methods

Since we are rejecting with a TypeError when the Response is type error (aka network error per spec), what's your suggestion on how to distinguish between unparseable url, broken location header, network timeout, redirect limit reached.

Is checking error message the suggested way?

This might be less of a problem on client side, but when implemented on server side, once I allowed custom timeout and maximum redirect count, that error is no longer solely caused by remote resource, it can also be related to request config. This is a concern for debugging and logging purposes.

for server-side implementation, see node-fetch, also related to #20.

The fetch spec doesn't define what a user agent should do when the document unloads. I expect the fetch request be aborted. This expectation seems reasonable, since it is similar to what the SW spec states: https://slightlyoff.github.io/ServiceWorker/spec/service_worker/index.html#on-fetch-request-algorithm:

18 . ...
If task is discarded or the script has been aborted by the termination of activeWorker, set handleFetchFailed to true.
...
21 . If handleFetchFailed is true, then:

1. Return a network error and run the following substeps in parallel.

22 . Else:

1. Return a response represented by response and run the following substeps in parallel.

Chrome's implementation already follows this behavior, Firefox does not (see https://bugzilla.mozilla.org/show_bug.cgi?id=1165237).

It would be much more useful if it did, or if there were an option for it to.

Should the patch verb be included in the spec? (https://tools.ietf.org/html/rfc5789)

here -
https://fetch.spec.whatwg.org/#concept-method-normalize

and
https://fetch.spec.whatwg.org/#http-network-or-cache-fetch

If request's body is null and request's method is `HEAD`, `POST`, or `PUT`, set contentLengthValue to `0`.

Research:

• https://lists.w3.org/Archives/Public/public-webappsec/2015Apr/0004.html
• https://lists.w3.org/Archives/Public/public-webappsec/2015Apr/0005.html

Todo:

• Study various other contexts.
• Fix tests to match the web-platform-tests framework.

Tentative plan:

• Define the X-Content-Type-Options header.
• Add a hook in #concept-fetch that checks a response for "nosniff" violations. If there are violations, return a network error instead.
• Write a section that defines the check for "nosniff" violations. Checking the request context against the MIME type if a "nosniff" header is around.

Open issue:

• Might need to annotate the response with a "nosniff" flag of sorts so further checks can be done by special components. E.g. an image decoder might want to reject a GIF with a image/png MIME type (IE11 does this).

If you do the following:

headers.append("accept", "text/plain")
headers.append("Accept", "application/x-www-form-urlencoded")

I assume all of these should return an array of two items:

headers.getAll("accept");
headers.getAll("Accept");
headers.getAll("ACCEPT");

But if we have to serialize out, does it matter what form of the header case should be used?

I propose allowing the browser to select a single case for output serialization, iteration, etc. Preserving case seems relatively unimportant and would require memory/complexity.

From @wanderview:

Looking at the gecko code it appears that there are many cases where the content is explicitly ignored regardless of what the server does:

• HEAD method
• CONNECT method
• Responses with status codes 101, 204, 205, 304

See mNoContent in the code here:

https://dxr.mozilla.org/mozilla-central/source/netwerk/protocol/http/nsHttpTransaction.cpp#1523

See also w3c/ServiceWorker#655 and web-platform-tests/wpt#1697

@mayhemer it's not entirely clear to me how to deal with partial content. Does us handling that mean we also pay special attention to a 206 response? What's the logic here, if we notice we have a partial cache entry, we set If-Range, then if there's a 206 response, we combine some stuff and hand out the combined response as a 200 to content? Does this only happen for media elements basically?

As discussed a bit in mikewest/internetdrafts@d2e4cd8, the First-Party-Only cookie attribute requires tracking the originator of a request across navigations. That doesn't seem possible at the moment via Fetch's current definition.

window.fetch is a great progress to make asynchronous requests, thanks for that!
I am not sure whether that's the right place for doing so, I also propose that we can parse the response as HTML content.

Something like response.html([context]) where context is the parent node of where the nodes would be inserted. It would return Promise<DocumentFragment>.

A polyfill of them would be:

response.html = function(context) {
  return this.text().then((text) => {
    var range = document.createRange();
    range.selectNodeContents(context);
    return range.createContextualRange(text);
  };
};

Use case: having directly parsed elements to possibly manipulate their content before inserting.

What do you think?

Florent

Over in yutakahirano/fetch-with-streams#37 (comment) @yutakahirano noticed that this code may not do what is expected:

var req = new Request(url, {body: 'hello'});
fetch(req);
fetch(req);

I think we would expect the second fetch() to reject on a drained Request, but it won't. The first fetch() sets the request's used flag to true and its body to null. The second fetch(), however, only looks at the used flag if the body is not null.

What do you think @annevk?

Throughout the document, I have not found anything that clearly states whether the when/then API of fetch() is just defined as such for readability, or if fetch() requests are promises. I was also expecting to see the request class extend some sort of thenable interface, but it looks like it's a standalone object.

I'm not qualified to comment on if it is appropriate or not to do something like:

// serviceworker.js
onfetch = function(e) {
  e.respondWith(fetch(e.request).then(function(response) {
    response.headers.append("Reply-from-SW", "yes");
    return response;
  })
}

but from talking with @annevk on IRC, it seems this is not ok.

When execute fetch() on origin http://a.com, the spec seems to allow us to fetch or redirect to a URL with username/password, if the all URLs in the redirect chain before that URL are on the origin http://a.com.
Particularly, if other conditions are OK (e.g. Access-Control-Allow-Origin is properly set),
fetch('http://user:[email protected]/', {mode: 'cors'}): OK
fetch('http://user:[email protected]/', {mode: 'cors'}): OK
fetch('http://a.com/', {mode: 'cors'}), which is redirected to http://user:[email protected]/: OK
fetch('http://a.com/', {mode: 'cors'}), which is redirected to http://user:[email protected]/: OK
fetch('http://b.com/', {mode: 'cors'}), which is redirected to http://user:[email protected]/: Network Error
fetch('http://b.com/', {mode: 'cors'}), which is redirected to http://user:[email protected]/: Network Error

Is my understanding is correct and is this behavior intentional?
(e.g. the spec doesn't intend to reject all cross-origin requests/redirects to URLs with username/password, right?)

In a discussion on .NET MVC Framework, aspnet/HttpAbstractions#245 there's some talk about what's the future of the old mvc version feature: Request.IsAjaxRequest() Method.

Right now, the only way we developers checks for a request that's initiated via javascript we check the used X-Requested-With custom header.

I was wondering if there's some plan or intention to have a 'standard' header or realiable mechanism for a server to know if the request originates from a fetch request or we should use custom headers for this kind of scenario.

Thanks and sorry if this has already been addresses, couldn't find it on current issues.

The spec does not document any way to specify GET query parameters (e.g. search?q=unicorns&limit=10&foobar) for a request. GET is the most common case but in practice, query parameters may be used with any HTTP method.

As I understand, the way to do this is using URLSearchParams from the URL spec. It would be good to provide a clearer view of how these two APIs fit together, especially as query strings are often supported directly by the high-level XHR APIs that fetch is meant to replace (e.g. jQuery.ajax or reqwest), often as a data option for example.

In general, it feels like it would be really great to have a boilerplate-free way to use only native web APIs to do such common things as passing URI parameters.

See also this Twitter thread.

IIUC we can create a GET request with a non-null body:

var req = new Request('https://www.example.com/', {method: 'POST', body: 'hello'});
var req2 = new Request(req, {method: 'GET'});

Is this OK?

The manifest spec currently lacks a "context", as none of the ones listed appear to be suitable. This is a feature request to add a suitable context for manifest.

According to the CSP spec
https://w3c.github.io/webappsec/specs/content-security-policy/#csp-request-header

If the user agent monitors or enforces a policy that contains a directive that contains a source list, then the user agent MUST set a CSP Request Header when requesting cross-origin resources, as described in §3.4 The CSP HTTP Request Header.

But "CSP" is not a simple header
https://fetch.spec.whatwg.org/#simple-header

A simple header is a header whose name is either one of Accept, Accept-Language, and Content-Language, or whose name is Content-Type and value, once parsed, has a MIME type (ignoring parameters) that is one of application/x-www-form-urlencoded, multipart/form-data, and text/plain.

So when the user agent requests a cross-origin resource which CSP is set, it must send a CORS preflight fetch.

This means when we use CSP, we can't use CDN which doesn't support CORS preflight.

Is this my understanding correct?

After trying to stub fetch() in some unit tests in this project, I came across that bodyUsed wasn't what I expected. I thought it was a boolean showing that the response contained a body, instead of a value telling that the body has already been used somewhere in your own code.

As the specs are currently not very explanatory on that particular subject:

The bodyUsed attribute's getter must return true if the used flag is set, and false otherwise.

I'd suggest to rename bodyUsed to bodyAlreadyUsed, to make the intent clearer.

What do you all think?

The use case here is to allow a service worker to make a fetch just like <img> can with equivalent effects.

This requires an opaque response that can only be used by features that can legitimately set the same context as otherwise CSP could be circumvented. E.g. if a page were only allowed to fetch images this could be used to fetch "an image" and instead use the result for something else.

I think do this properly I need @mikewest to finish the CSP rewrite in terms of Fetch. And we need to fix the SW and CSP issue: w3c/webappsec#227

Currently new Request() will get a context of "fetch" even though it may not be used with the fetch() function. Perhaps we should distinguish this case by using a different default context value. Maybe empty string or "synthesized".

This is related to "what should we do with context" in Cache:

w3c/ServiceWorker#664

As I read the Body Mixins section, options.body can be introspected for certain types, which will cause the Content-Type header to be set appropriately.

Can we have it check for plain-ol'-JS-objects (or any other stringifyable type), run JSON.stringify(options.body), and set options.headers["Content-Type"] = "application/json"?

We want to introduce fromCache member on Response objects. However, this in part depends on how we tackle #38 and also #39 of course. If a Response is partially formed from a cached entry, should it say "partial"?

When initiating a fetch request for a resource, the user agent initializes the fetch with content/context specific settings. For example, when fetching an <img>, the UA applies the appropriate image-src CSP checks; may advertise different set of request headers (e.g. advertise support for some image formats via Accept header, and may emit additional "hint" headers); may set different transport options, such as request priority, dependencies, etc.

How do we enable the same functionality with fetch() API? Without the above, in the worst case, an image resource fetch initiated via fetch() may violate CSP, will return an incorrect/suboptimal asset due to missing headers, and may be fetched with incorrect priority. Of course, images are just one example; same logic applies to all other content types. Related discussions:

• I want to make an image request: w3c/ServiceWorker#279 (comment)
• Initializing fetch settings via "as": w3c/preload#17 (comment)
• Initializing fetch settings via markup attribute: https://www.w3.org/Bugs/Public/show_bug.cgi?id=26533

Intuitively, the difference between using fetch() vs. say, and <img>, is that the latter knows its context and is thus able to initialize all the right checks + properties. Perhaps we can address this whole issue by providing "treat this fetch as if it was an 'image'" setting or signal? Handwaving...

fetch(url, {as: 'image', headers: {...}, otherOpt: {}, ...})

• Use as (or whatever fits best) as an explicit context signal that initializes appropriate defaults?
• headers and other fetch options are then applied on top of initialized defaults as overrides
• As a last step, UA initializes any missing / not specified headers? E.g. User-Agent, etc.

The mental model here is very similar to what @domenic proposed in #37 (comment):

fetchSettings = mergeMultimaps(asDefaults, optSpecifiedSettings, uaDefaultSettings);

Assuming above sounds plausible, we could then also enable this in other parts of the platform:

• <img src="photo.jpg" fetch-settings="{...}"> -- implicit: as == image, and fetch-settings can be used to override other UA + fetch defaults.
• <link rel=preload as="image" href="photo.jpg" fetch-settings="{...}"> -- as is declared explicitly, and options passed in through same mechanism.

WDYT, does it sound crazy?

Most ajax libraries and xhr itself offer a convenient way to set a timeout for a request. Adding timeout to the properties accepted in the options object therefore seems like a sensible addition to fetch.

JakeChampion/fetch#68

Should fetch() set user-agent by default? Allow appending bytes? Allow replacing it? Allow it to be omitted?

See w3c/ServiceWorker#348 (comment) for context.

And why is it on the forbidden header list? It has been since forever, but is there strong rationale?

Is there any way I can attach a progress listener to the xhr.upload object used in fetch?

xhr.upload.addEventListener('progress', function(){ ... })

since I use fetch to upload files (with FormData), I would find this very useful.

I initially reported this at JakeChampion/fetch#89, and was suggested to have it reported here

We could avoid teeing the request body with this protocol feature: http://httpwg.github.io/specs/rfc7231.html#header.expect

We would need to decide how much of the protocol is implemented in Fetch and how much we leave up to developers.

I'm implementing Request class in Chromium.
While I'm writing test codes like the following, I noticed a problem.

var baseRequest = new Request('test.html', {headers: {'X-Test': 'test'}});
var request = new Request(baseRequest, {mode: 'no-cors'});
console.log(request.headers.get('X-Test'))  // -> 'test'

This request.headers should not have 'X-Test' header, because the mode is no-cors and 'X-Test' is not a simple header.

http://fetch.spec.whatwg.org/#request-class
The Request(input, init) constructor must run these steps:
...
If r's request's mode is no CORS, run these substeps:

1. If r's request's method is not a simple method, throw a TypeError.
2. Set r's Headers object's guard to request-no-CORS.

I think we should remove non simple headers here.

Is my understanding correct?

The spec currently has three examples under "5 Fetch API", using ES6 syntax. Since the document is pretty long, it would help a lot to have some usage examples at the start of the spec, for example as part of the "Goals" section, similar to the examples in the Web Audio spec. I also suggest using at most ES5 syntax for those examples.

What would those examples accomplish? A lot of specs, including this one, are mostly written for implementers. Getting early feedback from users is generally desirable, but often hindered by the lack of usage examples. Adding those, or in this case, making them easier to find, would make the spec a lot more accessible.