Comments (6)
ignore_above
can be seen as a safety feature. It only prevents the keyword (exact match search + aggregations) indexing to take into account values that are longer than 1024. Or more precisely, it stops caring at 1024 chars.
But I don't think it changes anything performance or storage wise on fields where values are all reasonably sized (e.g. 20 to 100 chars).
from ecs.
For the example template see here: https://github.com/elastic/ecs/blob/master/template.json#L481 I hope this has the details you are looking for.
Good question about the ignore_above
and I must confess I haven't really though about it yet. The current default we use is 1024
as the same template generator is used as we use for beats and that is the default there.
I would expect there is a very small subset of ecs fields where we can expect very long values. There we should probably come up with a good default for ignore_above value that we also recommend to be used. To you have some specific fields in mind?
@willemdh Please let us know if you find fields which are missing in ECS but should be there for your project.
from ecs.
@ruflin The example template indeed clears things up. Untill now I tried to set ignore_above
as small as possible, but 1024 seems like a nice default. I wonder if setting ignore_above
smaller for certain fields has any worthwile advantages. Looking for example at agent.version => This should never be larger then let's say 25 characters? Would it prevent us problems setting ignore_above
to 25 or would it just make things more complex?
If I ever encounter something that requires an ignore_above
higher then 1024 I'll let you know.
About missing fields related to F5, most of them seem pretty specific to F5, I guess it's not worth it creating an ECS field for every possible value? Or would you advise making a separate f5 object in ECS containing all F5 related fields?
Just another question about template usage, do you think it's possible to use https://github.com/elastic/ecs/blob/master/template.json as a default template for all indices containing data which can leverage ECS? Maybe with a high order, making sure it applies? Or would this mean too much unneeded fields are added to each index?
Or do you advise I only use the fields I need from ECS and integrate this into my existing templates, eg https://github.com/OutsideIT/logstash_filter_f5/blob/master/f5.template?
from ecs.
The main reason we have ignore_above
to make sure no field can become massive and would have an affect on Elasticsearch potentially. But in the ECS case I don't think we have such a potential field yet. In general I would prefer not to have the ignore_above
but place it only on the values that need it. I think for values that are only expected to contain 25 chars I would not set an ignore_above at all.
For the F5 fields: I don't think we should have a namespace for it in ECS but on your end I would recommend you to put it under f5.*
to make sure you don't have any conflicts. Long term if many people will need f5 fields I could see that we have something in an extended version of ECS that covers this.
Elasticsearch 6.x has become much better in handling sparse documents. So I'm not too worried about having more fields defined then actually needed. I definitively like your idea about having ECS there for all indices but must confess I haven't tested this enough yet. But when I created the template I had exactly something like this in mind. Let me know how this goes.
from ecs.
@ruflin 1024 seems like a good default, i'll revert my smaller ignore_above's to that.
I'll create an f5.* object one of the following months
About the ecs template as default for all indices, let me know if you find the time to test this. It could be a start to use this officially for Beats indices one of the following major release.
For now I'll integrate the used fields into my existing templates.
from ecs.
I think all my questions in this issue got answered. Tx all. Closing up.
from ecs.
Related Issues (20)
- Typo in changelog for 8.8.0 HOT 1
- Tune container.image.* fields to follow OCI spec HOT 3
- log.offset not ECS compliant? HOT 16
- Multiple nested fields are missing from generated yml HOT 1
- Generator does not add "ignore_above" property for field type: flattened
- [Discuss] `ignore_above` on `flattened` fields
- [ci] Pipeline should fail if schema contains invalid field types
- field collisions with organization.id & elastic maintained integrations
- [RFC] Support id in threat.indicator for STIX 2.1 HOT 6
- Create equivalent of ecs-dotnet and ecs-typescript for java
- new field set: for log documents themselves
- Clarify the type of container disk and network metrics HOT 1
- ECS Vulnerability Published field
- Add threat.indicator.tags field
- [Proposal] Make event.kind a list HOT 3
- Incorrect generated/beats/fields.ecs.yml, not accounting on top_level: false HOT 1
- [ECS] Addition - http.request.header.bytes & http.response.header.bytes HOT 2
- Add lowercase normaliser to ECS fields which support security incident response process
- Mark experimental fields as `beta` in generated files
- Elastic-Agent Integrations Use of Legacy Mapping Types Impacts .fleet_globals & prevents agents from being upgraded
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ecs.