Comments (43)
If User-Agent
is allowed to be assigned, I think the expected usage would be to pass it in with the rest of the headers for the request.
fetch('/users', {
headers: {
'User-Agent': 'Web/2.0',
'Accept': 'application/json'
}
})
from fetch.
(It might be worth noting that GitHub itself recommends User-Agent set to your GH user name in API requests - this is of course trivial if you write Python/Node/whatever clients, but not currently possible from an API consumer running in a browser. So there's a big and important use case right here.)
from fetch.
Right, so why is userAgent: null !== userAgent: "null"
? This seems kind of WTFWebPlatform-ish to me.
It looks like from my reading of the spec there's no actual way to tell what User-Agent
header gets sent then (currently)? E.g. const r = new Request(...); r.headers.get("User-Agent")
will return undefined
? That seems worrying.
One last point before I go to sleep: I think what's a bit tricky for me here is that the most straightforward mental model for the programmer is that in the constructor, there's some kind of
this.headers = mergeMultimaps(defaultHeaders, passedHeaders);
That is, just naively looking at the shape of the API, and knowing the bit of extra information that there are more headers sent in the request/response than you set, I think most programmers will guess that there's a set of default headers and you can set new headers or override old ones by using the headers
option. So, the deviations we make from that model need to be done cautiously.
For example in that model I think if passedHeaders
contained "User-Agent": undefined
, that would override the default "User-Agent": "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:37.0) Gecko/20100101 Firefox/37.0"
.
It seems like that is not the model though, as evidenced by the r.headers.get("User-Agent") === undefined
. Rather, r.headers
is a straight copy of the passed-in headers, and then the actual fetch uses a hidden set of headers which are compiled based on r.headers
+ other knowledge (possibly from RequestInit, also from the cache, also from the user-agent string, etc.)
from fetch.
If the browser is confused about same-origin that would be a much bigger problems than setting User-Agent
.
from fetch.
@steike you already ceded that same-origin-or-CORS helps and your counter argument to that was bogus. If a browser is confused about same-origin that would be a high priority security bug.
from fetch.
.02 - it sure would be nice if UA could be appended to, so you can add "mywidget/1.0" or "(test 3)" for example, rather than blowing the entire thing away.
Syntax here:
http://httpwg.github.io/specs/rfc7231.html#header.user-agent
from fetch.
navigator.userAgent + 'my string'
from fetch.
@sicking I don't understand why omitting the header altogether is a cause for concern. It wasn't when we stopped sending Referer
.
from fetch.
I think we 'inherited' User-Agent from a list of headers Flash disallowed changing in its implementation. There was some speculation that some services might exist that only allowed specific customized browsers to access them, using UA string as a sort of auth token, but I don't think anybody ever found a real-life case of such a system. (Obviously it would be astonishingly poor design).
Personally, I think we can and should drop User-Agent from the list of headers you're not allowed to set.
from fetch.
If we allow setting it to any value and effectively replace anything the user agent would set we should probably make it a parameter of RequestInit
as the supplied headers
argument combines with existing headers (and never replaces). So e.g.
fetch(url, {userAgent: "Hi"})
from fetch.
If we allow setting it to any value and effectively replace anything the user agent would set we should probably make it a parameter of RequestInit as the supplied headers argument combines with existing headers (and never replaces). So e.g.
Hmm, I'm not sure I necessarily agree. It seems a straightforward extension to say that headers provided by the RequestInit overwrite default headers. What do you think would be confusing about allowing User-Agent to be set through the normal mechanisms?
from fetch.
GitHub itself recommends User-Agent set to your GH user name in API requests
https://developer.github.com/v3/#user-agent-required
This helps us contact application owners when there are problems, like a rogue script making lots of API calls. However, many API resources are only available to authenticated sessions, so the user account is already known. And when accessing the API through a browser, the CORS request includes the Origin
header.
The User-Agent
header guidance in the API is helpful, but it's not necessarily an example for or against this capability being added to fetch
.
from fetch.
I guess that might be okay as well. We could have a step in https://fetch.spec.whatwg.org/#http-network-fetch that appends it if it is not already present in HTTPRequest's header list.
from fetch.
So actually, the reason you want it to be an argument rather than part of headers
is so you can force it to be omitted.
from fetch.
Probable rationale for why it is currently forbidden is in this email by @sicking: https://lists.w3.org/Archives/Public/public-webapi/2008May/0456.html
from fetch.
So actually, the reason you want it to be an argument rather than part of headers is so you can force it to be omitted.
Can you expand on this? I don't quite understand why one location or the other would matter for this, or why you would force it to be omitted.
from fetch.
If we offer customization, I think it would make sense to also offer omitting it altogether e.g. to reduce the size of the request or debug a server. The location matters since by default it is included and the Headers
class is not an instruction set but rather a list of headers included in the request.
from fetch.
Oh, I see!
Would it be conceivable not to set the header by default at all? Hmm, probably not very good, I'm being too Node-influenced here...
So how would you omit it? omitUserAgent: true
? userAgent: null
?
from fetch.
null
was my idea, since undefined
already means doing the default, which is to include it and seems highly unlikely to change.
from fetch.
Why would userAgent: null
be good, but 'User-Agent': null
not be good?
Related question: when you do const h = new Headers()
, does h.get("User-Agent")
return something? Why or why not?
from fetch.
"User-Agent": null === "User-Agent": "null"
. new Headers()
creates an empty multimap, it is not populated in any way.
from fetch.
Basically the model is that the network layer appends a set of headers that are not in control of the developer. And that headers.set()
stringifies its arguments and validates them against the HTTP header syntax seems totally natural and not at all weird.
from fetch.
FWIW, if that email from me is really the reason that we don't allow setting the user-agent, then I believe that that's fixable.
If we simply treat user-agent as a "custom" header, then things seem fine with me. I.e. we can let it be set completely by the page for same-origin requests. For cross-origin requests we can allow it to be set, but it'd cause a preflight, and would require the server to send a "access-control-allow-headers: user-agent" header in the response.
I don't have opinions about how to deal with getting the default value or removing the header completely.
from fetch.
Given the requirements and existing constraints I see a couple of options:
User-Agent
is passed in as part ofheaders
. We provide a distinctomitUserAgent
boolean option for removing anyUser-Agent
headers from the request in the network layer.omitUserAgent
is also put onRequest.prototype
. The network layer only appendsUser-Agent
header with a default value if noUser-Agent
header is present.userAgent
is an option that is undefined, null, or a header value. undefined indicates the network layer inserts aUser-Agent
header with a default value. null means the network layer does nothing. A header value means the network layer insertsUser-Agent
header with that value.
2 seems a little cleaner to me, but I don't care strongly.
from fetch.
Will XSS via
User-Agent
string become a reality thanks tofetch()
?
from fetch.
Despite the discussion having progressed a lot already, do you consider XSS or CRSF-to-XSS via UA string in scope for the spec? Or would that be something, the implementers have to take care of on their own?
P.S. @mathiasbynens You are faster than light :D
from fetch.
One note about my context maybe:
Right now, for penetration tests, we use malformed UA strings to aim for persistent XSS or Intranet XSS. Giving an attacker control over the UA string via fetch()
opens the door to abuse that in a CSRF scenario and beyondd. Not sure if that is a good idea. Thus my question, if you consider that to be in scope or not.
from fetch.
The header can only be set for fetches that are same-origin or subject to CORS (with a preflight where the server needs to opt into User-Agent
), same as all developer-set headers. If that still enables attacks I would love to see an example.
from fetch.
@annevk So, to quote @sicking here:
For cross-origin requests we can allow it to be set, but it'd cause a preflight, and would require the server to send a "access-control-allow-headers"
If that means, the request will not even be sent with the modified UA in case of cross-origin requests and a failed CORS preflight, then spec-wise this should be fine.
from fetch.
@annevk Alright, I had a closer look at how fetch()
is implemented in Blink at the moment.
If the custom UA string header is in fact being implemented the same way as any other custom header - meaning that the particular header has to be permitted via CORS for cross-origin requests, then this should indeed be safe! No objections from my side so far.
from fetch.
There is value in having a reliable user-agent header. Historically we've had some browser bugs where it was possible to protect the user server-side, but where the fix would be too expensive (in terms of cost, perf or user annoyance) to apply to all users.
More commonly, a new browser feature might allow some feature to be reimplemented in a more secure way; with a reliable user-agent header, it's easy to disable the "unsafe" back-compat implementation in decent browsers.
If a malicious script can lie about the browser version, protecting against such attacks becomes a lot harder.
(The fact that this is same-origin-or-CORS-only helps a lot, of course, but not in the case where the particular browser bug is that the browser is confused about what counts as the same origin...)
Would a solution where the user data is either appended or prepended to the 'responsible' user agent be an acceptable compromise?
from fetch.
Wait, I think you got that backwards ("if the zombies attack, you have bigger problems than your broken shotgun"). I'm not worried about the user agent as an attack vector; I want to keep using it for defense.
from fetch.
@steike I believe, with this feature, you have even more possibilities to use the UA string header as a defensive feature. Differently, yes - but more powerful too.
from fetch.
Yes, a high-pri bug. That doesn't mean it wouldn't take the vendor months to fix it, or that users would upgrade instantly once the fix was out.
Let's say that 10% of users have browsers that are vulnerable to a bug. Let's say when this particular vulnerability is exploited, there is a detectable header anomaly. Let's say 1% of all users happen to be behind various crappy firewalls that introduce the same anomaly for legitimate requests.
With a working user-agent header, we can block the attack by telling 0.1% of users that they must upgrade their browser before they can use the site. Without, the choice would be to force-upgrade 10% of users, outright block 1% of users, or hope no one finds the bug.
...
It's not the end of the world, of course. We can add a few bits to some cookie; it'll just be one more layer of web cruft to carry around. You asked for a rationale for keeping a working UA header. What I have is "those of us who need it will have to reimplement the feature if you take it away". To be fair there's probably not that many of us. If the benefit of User-Agent
over X-Requested-With
is great, we can live with it.
from fetch.
@steike I don't understand your attack scenario. The User-Agent
retains its default value for the majority of fetches. It is only fetches that go through fetch()
that are potentially impacted and those are under control of the site.
And cross-origin resources need to explicitly opt-in to allowing User-Agent
headers with non-default values, but your attack does not seem to concern those.
from fetch.
userAgent is an option that is undefined, null, or a header value. undefined indicates the network layer inserts a User-Agent header with a default value. null means the network layer does nothing. A header value means the network layer inserts User-Agent header with that value.
+1 to this route. I agree with @domenic's comment on "mergeMultimaps" mental model (#37 (comment)) as being developer friendly and most intuitive.
from fetch.
Whatever syntax we end up using, we should make it very explicit that "removing" the user-agent header should from a CORS point of view be equivalent to setting it. So it would still require server opt-in. Adding a note to this effect might increase the chances that the browser actually tests for this case.
from fetch.
I'm inclined to define 1 in #37 (comment) but leave the omitUserAgent
feature out for now since that is harder to tackle (unless @sicking is wrong and it should not require explicit opt-in to omit it).
from fetch.
If we need omitUserAgent
please file a new issue. Ideally reach out to user agents for their security requirements, since they're not entirely clear to me.
from fetch.
gecko bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1188932
from fetch.
chromium bug: https://crbug.com/571722
from fetch.
Is User-Agent still forbidden after dab09b0 ?
from fetch.
It's not as per this change.
from fetch.
Related Issues (20)
- When there is an issue with the URL parameter, fetch does not have any errors
- add a method to get a Uint8Array to Body HOT 4
- Fetch with Flask or Microdot HOT 1
- Service worker "handle fetch" seems to assume it's called in parallel HOT 2
- Redirect loop handling not discussed
- Add compression dictionary negotiation and decoding to the fetch processing model HOT 6
- Reconsider default Accept values for images
- Fetch support for explicit resource management
- Consider renaming or replacing http3only? HOT 1
- [render-blocking] The links of "render-blocking" in fetch are vague HOT 2
- Define what consequences NULL bytes (0x00) or other invalid values in header names have HOT 1
- 0x00 byte (%00) in a data: URL
- fetch does not allow caching requests with a null client HOT 1
- Question about stream handling around fetch requests with integrity metadata HOT 3
- Allow `new Response` with 101 status code HOT 6
- Question regarding preflight requests using the original request destination HOT 1
- Expected behavior of Response.redirect(null)? HOT 1
- Spec unexpectedly requires caching 30x responses themselves — rather than caching the result of running HTTP-redirect fetch to follow the 30x redirects HOT 9
- Spec WWW-Authenticate and Proxy-Authenticate handling in HTTP-network-or-cache fetch
- Add Range to no-cors safelisted headers HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from fetch.