Add support for runtime fields in packages about package-spec

[ruflin]

@ycombinator for the spec
@skh @neptunian For awareness for the template / index pattern creation
@ph For awareness
@andresrc We should figure out if and where we want to use this in packages in the future.

[ruflin]

We would really like to see this prioritised. The way I would propose this to be implemented is as following:

    - name: foo.bar
      type: long
      runtime: true

This is slightly different from the way it looks in Elasticsearch where a separate block for runtime fields has to be specified. Having it directly as a boolean in the fields definition allows a very simple way to enable / disable it for devs. Fleet still has to create the additional block.

[mrodm]

Maybe we can try to remove a field in a package, and add it as a runtime field in the index_template configuration of its manifest. Then check what system tests do, and what sample documents they generate.

I've tested this approach, and what I observed is that runtime fields are just present in the fields in the documents instead of _source. As it is indicated here:

Use the fields parameter on the _search API to retrieve the values of runtime fields. Runtime fields won’t display in _source, but the fields API works for all fields, even those that were not sent as part of the original _source.

In that case, elastic-package should also take into account the fields from the documents retrieved from Elasticsearch to validate the fields defined in the package.

Created issue to support runtime fields in elastic-package elastic/elastic-package#1229

Also tested that if nowadays a field with the runtime key is added into a package, this is not going to fail and it will just add a new mapping with the given type. Because of that, it is also needed to add a new validation rule in package-spec to ensure a minimum Kibana version (8.8 ? 8.9?)

[jlind23]

@mrodm moving this issue out of our current sprint as they are is nothing to do here from a package-spec perspective. We will close this one later as ruflin stated as soon as elastic/kibana#155255 is fixed.

[ruflin]

Adding to this, Elasticsearch supports to set the dynamic: runtime: https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic-field-mapping.html A package dev should be able to select this option as the default for dynamic fields. APM asked for this in the context of the apm package defaults (@simitt ).

[simitt]

@elastic/apm-server feel free to pick this up for 7.15

[amitkanfer]

@jlind23 putting this on your radar.

[joshdover]

A couple things to consider here. Runtime fields is mostly optimized for extracting new fields out of other indexed fields that are available in the "doc values" and the runtime field's script has to be written differently depend on whether or not the field that we're extracting from is being indexed. So there's really 2 use cases here:

• Runtime fields that have supply a custom script to extract data from another indexed field
• Runtime fields that simply populate a value from _source that is not indexed
  • This is the use case that is covered by @ruflin's comment in #39 (comment)

When Fleet creates the index templates for runtime fields, the script field for a runtime field will need to differ based on whether or not the field is indexed. If index: false is specified or the field is not mapped at all, the field will not be available in doc['foo'] but will be available in params._source['foo'].

We can probably support a design like this for both use cases:

- name: foo.bar
  type: long
  runtime: true
- name: foo.baz
  type: long
  runtime: |
    doc['message'].value().doSomething()

I think the implementation could work something like this:

• If runtime is a boolean true, then create a runtime script with params._source['<field>']
• If runtime is a string, just use that for the script directly

[mrodm]

The approach mentioned by @joshdover in #39 (comment) LGTM

For adding this new field, according to the docs it should be checked that runtime is just added/defined when some of the types mentioned there are used for that specific field:

• NOTE: currently, package-spec does not support all the types mentioned there for field types.
• Those checks look like could be done using JSON schema conditionals like in this PR (#500)

I'll try to create a proposal PR in package-spec to include that field.

Fleet should also add support to create those runtime fields, @kpollich do you foresee any issue having that field being boolean or string for fleet?

In elastic-package there could be some improvements to be added:

• Add into the generated READMEs some indication that the field is a Runtime field (example apache docs) ?
  • README files should not be changed if there are no runtime fields.
• To be checked once the fleet support is added to include runtime fields when the package is installed, whether or not system tests need to be updated to give special support for runtime fields (could it be checked previously @jsoriano ?)

[jsoriano]

Plan sounds good to me too.

If runtime is a boolean true, then create a runtime script with params._source['<field>']

This seems to be already the behaviour of Elasticsearch when no script is set (see here). In principle it seems enough to add the field as runtime field, without any script, and it does the equivalent to params._source['<field>'].

Fleet should also add support to create those runtime fields

@mrodm could you please check if there is already an open issue for that, and create one otherwise?

do you foresee any issue having that field being boolean or string for fleet?

It seems possible to have variables with multiple types in typescript, I think this will need more special handling in Go if we need to do something with this field 🙂

• could it be checked previously @jsoriano ?

Maybe we can try to remove a field in a package, and add it as a runtime field in the index_template configuration of its manifest. Then check what system tests do, and what sample documents they generate.

[kpollich]

Fleet should also add support to create those runtime fields, @kpollich do you foresee any issue having that field being boolean or string for fleet?

I don't foresee this being an issue on Fleet's end. Just a small piece of logic for how the field is used when generating mappings.

[jen-huang]

Thanks @mrodm for the work here so far. I'm going to move this issue to Sprint 12 but of course it could get closed out by #505 if that merges soon. Follow up issues elastic/elastic-package#1229, elastic/kibana#155255) will be moved to Sprint 12 as well.

[jlind23]

@mrodm @jsoriano what is missing here as we have merged #505 ?

[mrodm]

@mrodm @jsoriano what is missing here as we have merged #505 ?

The support for package-spec has been added through that PR, but runtime fields need to be supported also in Kibana/Fleet. There is another issue for that elastic/kibana#155255. According to the comments in the PR, it should be for 8.9.0 https://github.com/elastic/package-spec/pull/505/files#r1173839069

Should we close this one and just let the other issue in kibana open ? @jlind23

[ruflin]

We should not close this issue. This is about adding support fo runtime fields in packages. As long as Fleet does not support it, it is from my perspective not supported in packages even though users could add it.