Trailers about http-core

[mnot]

I wonder whether it would be best to add something like:

A header field MUST NOT appear in trailers unless its definition allows it. Receiving implementations MUST ignore header fields appearing in trailers when their definitions do not allow it.

... with an appropriate column in the header field registry to record this. I don't think this would break any existing implementations, because trailers are so seldom used right now, and I think there's a strong argument that this is the case anyway; the consuming application has to expect something in trailers to act upon it.

[annevk]

Can we differentiate between what is processed and what is exposed to APIs? I think we should expose the headers to APIs still and let consumers of those make their own choice.

[mnot]

Yep, I was thinking that too. It's really hard for a generic implementation to know the semantics of individual headers. For ones that the implementation does look at, however (e.g., caching headers), it should pay attention.

This is really just clarifying current behaviour, I think.

[royfielding]

I don't think that is helpful. First, nobody reads header field definitions. Second, servers will not differentiate the processing of extension fields according to their extension semantics, since they don't know the semantics.

In any case, we can't assume the client is a browser and the server is a website; that's not the scope of HTTP. When a use case is more limited, an API, media type, or specific application of HTTP may choose to exclude or otherwise narrow the use of trailers. This doesn't prevent them from being used in other contexts.

[mnot]

Roy, what we're trying to avoid is the assumption that headers and trailers are semantically equivalent (which 7230 heavily implies), mostly for the benefit of senders. The requirements proposed above aren't intended to be targeted at generic HTTP software.

How about:

When a message is generated, a header field MUST NOT be included in trailers unless explicitly allowed by its definition. When consuming messages, recipients MUST ignore header fields appearing in trailers if their definitions do not allow it. Note that these requirements apply to the party actually responsible for generating or consuming the header in the implementation; it does not imply that a generic implementation is responsible for filtering trailers.

[reschke]

WFM.

[royfielding]

How about:

When a message is generated, a header field MUST NOT be included in trailers unless explicitly allowed by its definition. When consuming messages, recipients MUST ignore header fields appearing in trailers if their definitions do not allow it. Note that these requirements apply to the party actually responsible for generating or consuming the header in the implementation; it does not imply that a generic implementation is responsible for filtering trailers.

The phrasing is important and shouldn't diverge from our existing forms.

A sender MUST NOT generate a trailer field having the same name as a header field that is not defined as being allowed in trailers. A recipient that processes a trailer field for conversion to a corresponding header field MUST ignore and discard any such fields that are not known by the recipient to be allowed in trailers (by definition). Hence, extension fields that are not understood by the recipient are discarded rather than risk trailer fields being abused to bypass filtering or routing by header field values.

[annevk]

What is a recipient?

In browsers at least there's the low-level HTTP library that's the initial recipient, then there's something akin to what Fetch describes, which is what most features would use, and then there's various APIs such as <img> or fetch() which is what web applications would end up using.

Currently fetch() is described in such a way as to expose trailers as a separate field on Response objects, assuming none of them get discarded at a lower layer. Nobody has thus far implemented that so we will likely drop it, but I'd still like to know what the architecture should be if it ever came back.

[carl-mastrangelo]

Nobody has thus far implemented that so we will likely drop it, but I'd still like to know what the architecture should be if it ever came back.

This is really a shame, and was one of the features I was hoping to use. I can't speak for what the standard should say, but my own usage (gRPC) would treat trailers as a key-value list not treated specially by the browser (or any other intermediaries). That means they only have meaning (from the browser POV) to the JS applications, not anything else.

[royfielding]

A recipient is the role associated with anyone/anything receiving the message (as opposed to sending a message). Which API is involved cannot matter for a messaging protocol. However, the reason I narrow the scope to the convoluted "that processes a trailer field for conversion to a corresponding header field" is because this requirement only matters for code that takes a trailer field and merges it with a set of header fields. It would not apply to code that keeps headers and trailers as separate tables.

[mnot]

A sender MUST NOT generate a trailer field having the same name as a header field that is not defined as being allowed in trailers.

What's the import of "having the same name as a header field"? It seems to imply that there are two separate name spaces for headers and trailers, which is surprising to me.

[spaceone]

@mnot there are just some headers which can't be send as a trailer, e.g Transfer-Encoding, Content-Length and Trailer.

[mnot]

Sure. I just found that turn of phrase very confusing. I think we can simplify it.

[spaceone]

Okay :)
Btw, if it's not already answered, I would like that it's explained what a component should do if e.g. the server tells that it has Trailer: xyz but doesn't send them then. Should the whole response then be dropped as invalid. Does it influence security?

[annevk]

@spaceone please file a new issue for that.

@royfielding thanks, I guess I'm still a bit confused. Generally I would expect trailers not to be merged with headers. They might perhaps end up overriding some functionality, but the initial message parse would just pass data on to a streaming API, which most naturally would expose these as separate fields and not expose trailers as updates to the headers field.

[royfielding]

@annevk My original expectation regarding trailers was an attempt to supply the same information as header fields for data that wasn't known at the message start. Making them the same space was both an editorial simplification (we didn't have to define a new space and could reuse the field definitions) and a byproduct of how we expected shared caches to store them for later use. In other words, I expected a shared cache to merge the fields when storing it, since then it could supply that data as normal header fields on later hits that might serve both HTTP/1.0 and HTTP/1.1 clients.

With 20 years of hindsight and better awareness of the security issues, I agree we should have just defined them as two different spaces. I am not sure if we can undo that now.

[PiotrSikora]

@royfielding I'm not sure if there is too much undoing to do, really, because:

1. Existing implementations don't merge trailers with headers. AFAIK, none of the browsers support trailers. Apache HTTPD used to merge trailers with headers, but switched to treating them separately a few years ago due to CVE-2013-5704. NGINX only added support for them in the middle of last year (and only for gRPC, not when proxying HTTP/1.1 responses), and it always passes them to the clients as trailers, without any merging.

2. This is mostly unused feature. The only real-world use case for trailers, that I'm aware of, is gRPC, which sends grpc-status trailing status after completed response body.

Therefore, I think that we should sync with the real-world and split headers and trailers into two different spaces.

[mcmanus]

server-timings also uses http trailers successfully.

Like Piotr mentions, the http trailer is used to populate the server-timing information but is not merged with the header information for security and sanity reasons.

[mnot]

Discussed in Bangkok; result was "improve wording."

[mnot]

Friendly edit to Roy's suggestion:

A sender MUST NOT generate a trailer field when that field is not explicitly defined as being allowed in trailers. A recipient converting a message's trailer fields into header fields MUST ignore and discard any such fields that are not known by them to be explicitly allowed in trailers (by definition). Hence, trailer fields that are not understood by the recipient are discarded, thereby avoiding the risk of their being a means of bypassing controls on header fields.

[annevk]

Unless I'm missing something that still leaves the issue unaddressed. Are implementations allowed to maintain trailers separately from headers? (Note that per @mcmanus's comment they already do.) Can something "downstream" from HTTP require that?

[mnot]

I think implementations are free to consider them separate, but they have to understand that somewhere in the chain, an intermediary may have collapsed trailers into headers -- so if something depends on a particular field showing up in headers, they're going to be disappointed.

[annevk]

Hmm, I guess as long as that's true it's better to not support them in clients at all, to prevent that kind of thing from messing with application security.

[mnot]

How do you get there, Anne? How would this "mess with application security"?

[annevk]

Right, for what I'm worried about it doesn't really matter if browsers implement it or not, it's mostly intermediaries that might do such combining naïvely. E.g., if they put a Content-Type in a trailer block in the header block. Still, browsers not supporting them might make it more likely that this feature goes away.

[mnot]

If someone designs a field that has to be in the trailer, and is somehow insecure if it's promoted into a header by an intermediary, they're going to have a bad time (because intermediaries are allowed to fold all trailers into headers).

More generally, if someone designs a field that is allowed to be in a trailer and they depend on it being available there end-to-end, they're going to have a bad time. TE might give them some help here, though (see #18).

If someone puts a header that doesn't explicitly say it's allowed in trailers into trailers, they're going to have a bad time (because if it arrives at the other end, someone may not be looking for it there -- i.e., trailers are not always automatically folded into headers).

AIUI Firefox already supports Server-Timing in trailers, and that works well for its use case. It doesn't have any of the bad interactions above.

There are other specific cases where a browser could make a decision about whether or not it supports trailers for a header that it processes -- e.g., ETag is asked for once in a while (so that a server can compute an identifier based upon the body without buffering it).

I think that browsers can also expose trailers in an API safely, provided that they're not automatically collapsed into headers. That said, the constraints above still apply to the people using them -- and we definitely need to do a better job documenting them.

[annevk]

Thanks, I think given this the API is rather tricky to expose, as you cannot depend on there not being intermediaries in the wild, so your application might end up failing for a small number of users without that being very visible, but I'll follow-up in #18.

[royfielding]

Going back to the core issue of trailers overriding headers, we should first decide whether we continue to recommend/allow trailer fields to be merged with headers. If not, we should actually specify that! None of the changes in #219 are necessary if merging is not allowed.

[annevk]

It'd be amazing if we could agree that merging is not allowed and they're essentially a separate member of the response to be processed independently. I think that would increase robustness of the whole setup quite a bit.

[mnot]

I think we'd still want to say:

A sender &MUST-NOT; generate a trailer field when that field is not explicitly defined as being allowed in trailers.

... and probably add that older implementations might still merge them, so recipients SHOULD disregard headers that are only defined to appear in trailers.

As to whether we should do it -- I doubt that merging is widely implemented in pure intermediaries, since it would require buffering the message. However, I wonder whether it is in proxy/caches; I'll do some testing.