Consider flattening headers and cookies about apm-server

[felixbarny]

IIRC, we didn't always index all headers. When and why did we change that? Or did we always do that?

[axw]

The change was made in #12102. The idea was to align with how we would deal with OpenTelemetry semantic conventions in the future, which would involve dynamically mapping headers.

We could revert to disabling indexing for headers, or switch to flattening them. Either way it's a special case, and not a general solution to avoid hitting field limits.

[felixbarny]

Interestingly, that change used the flattened field type for HTTP headers

http.{request,response}.headers have been changed from object to flattened.

Did this change to dynamically indexing as keywords in the context of the apm-data plugin in Elasticsearch?

[axw]

@felixbarny sorry, yes, I should have mentioned earlier: I ended up removing that particular use of flattened and switched it to being dynamically mapped to keywords, to match what we would end up doing with OTel.

[felixbarny]

Both for OTel and for non-OTel, I think we should use flattened for HTTP headers.

[axw]

I think that should be fine. OTOH we've found that using flattened for stack traces is causing problems for ingesting exceptions with huge amounts of source context or deeply nested vars, so we'll probably revert to disabling there.

[axw]

elastic/elasticsearch#108227

[axw]

We'll also need to bring those changes across to the integration package for 8.15.0. We should be pretty close to switching to apm-data, but I think it may be safer to wait until 8.16.0.