Nested Object Arrays and Kibana about ecs

[ruflin]

What I recommend in such cases is that each "nested doc" is sent as it's own even but contains enough meta information / identifiers to connect it to the parent event and do the above query. Would this work in your case?

[larryrodrigues]

Thanks for the reply. Sorry, but I do not know how that would work. Could you post an example mapping?

[webmat]

I'm not sure I understand what you're asking, but I'll try to answer.

a document which contains 5 of those type records

I take this to mean you are using ip_rec_obj a few times on a given event, with detailed information about each IPs.

I want to build a dashboard of counts by type across a time period

I take this to mean you're unable to have Kibana easily count the amount of IP structs on a given event, to classify your stream into events that have 2, 3, 4, 5 and + IPs.

If this is indeed the problem you're facing, you are right that Kibana cannot count in this way. You have a few options on how to deal with this:

• You can calculate this count in your pipeline, and store it in a separate field on each event.
  • The advantage of this approach is that using this count field will be very performant for aggregations (counting/visualizing all those with value 2 vs 3 vs 4 vs 5 etc).
  • If you already have a big history of events you'd like to have this on, you will have to reindex them to add this field, however.
• You can create a scripted field in your Kibana index, that will use the Painless language to count the number of items in your nested field.
  • The advantage of this approach is that you don't need to reindex. You can look at your history right after adding this scripted field and see the calculated values.
  • This will however be much heavier on every single request, as the calculation will have to be done every single time, for every event. I'm not 100% sure if you can make aggregations/visualizations around a scripted field, but if so, the load generated by this would also be prohibitive.

I'd like to point out that if I understood your question correctly, a better place for having this discussion is in our Discuss forums, at https://discuss.elastic.co, likely in the Kibana section. This GitHub repository is not about Kibana, but rather about building a standard for a common event schema for storing time-based events in ElasticSearch :-)

If I misunderstood your question, feel free to clarify here, however.

[webmat]

@larryrodrigues Did this help answer your question? My understanding is that this issue doesn't really belong in ECS. Please let me know if I misunderstood.

[webmat]

Please let us know if we misunderstood your request, and if you think this should be an issue on ECS.

Thanks for getting in touch!

[larryrodrigues]

Thank you for the message. I am still trying to digest the information.

…

On Mon, Aug 20, 2018 at 12:44 PM, Mathieu Martin ***@***.***> wrote: Please let us know if we misunderstood your request, and if you think this should be an issue on ECS. Thanks for getting in touch! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#78 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ALra1X_FXHup51KLHnZ-ayS5cVsjOHpPks5uSueFgaJpZM4V63AP> .