Comments (13)
@ruflin I had another thought, that perhaps field aliases (assuming they are going to land in Elasticsearch) might be another way to solve this.
My ECS transform is done in Go right now before ingestion, for example, and I copy things as-is without modifying the values (to change types from string/date/number/etc). Instead, with field aliases, we could provide this to Elasticsearch and let it do the mappings (This assumes the field values are usable as-is without modification, though?)
For Okta, for example, I transform actor.alternateId
to be ECS' user.id
. If Elasticsearch gets field aliases, then an ingest author doesn't need to do this transform because we can tell Elasticsearch "The field user.id
comes from the actor.alternateId
field". This may be the best of both worlds where we can allow search/aggs by ECS but we don't actually modify the source data, so subject matter experts would see Okta System Logs (for example) in their native format and we can still build dashboards against the ECS schema.
from ecs.
Hi @ruflin the ECS repo now has tooling which allows for users to add their own fields in conjunction with ECS
from ecs.
For things that need to live along side a standard/specification but are not part of it, I have two examples:
- HTTP headers. Non-standard headers historically were prefixed with
X-
such asX-Forwarded-For
. This practice was deprecated though because it made standardizing them difficult (globally renaming a headerX-Forwarded-For
toForwarded-For
is basically impossible given the thousands of http clients, servers, and devices). - CSS browser prefixes. Non-standard css properties are prefixed with the vendor/browser, such as
-moz-user-select
. This probably has similar challenges as HTTP to standardization because now CSS authors now have to write both-moz-user-select
anduser-select
in order to support both older and newer browser versions.
I wonder if we make some recommendations for namespacing vendor/device-specific fields.
Something like vendor.product.field
For example, Okta's System Log has a top-level object:
"debugContext": {
"debugData": {
"requestUri": "/login/do-login"
}
},
This is likely uncommon and specific to Okta's System Log. Using the above proposal, maybe this can be stored as:
okta.system_log.debug_context: {
"debugData": {
"requestUri": "/login/do-login"
}
},
My goals are:
- Namespace things to prevent field type conflicts across multiple products
- Keep uncommon things in the same structure as the original log (if it is structured).
from ecs.
My above comment assumes we want non-ECS data to mix with ECS data in the same documents. Is this assumption true?
from ecs.
@jordansissel Yes, it's expected we mix non-ECS and ECS data in one event. As an example Metricbeat has lots of fields which are very specific to some service and will not make sense in ECS but still the "foundation" of the event will be based on ECS.
from ecs.
BTW: I think the two topics here are a bit different. One is guidance on what our recommendation is on how to extend events outside ECS. The other part is how we let people share ideas on how they mapped the fields to ECS and structured their events around it with others for inspiration. I wonder if we should open a separate issue to discuss the "recommendations"?
from ecs.
I have a similar aproach in mind for Beats just the other way around. The "original" will be the ECS one and the okta specific name in your case will be the field alias, so you still need to transformation. The reason is that a field alias can only point to one field and multiple aliases can point to the same field.
Lets assume you have a.hostname
, b.host_name
and both should be queried by host.name
. You can have a.hostname
and b.host_name
as alias point to host.name
but you can't have host.name
being an alias and point to the other two. If you have only a 1-1 mapping case, then your approach works.
from ecs.
the other part is people might get confused on what is part of ECS and what is not.
@ruflin Maybe it's an idea to create a subfolder in the ecs repository, for example named custom
, pre-ecs
or non-ecs
, where custom objects could be created and built over time for specific use cases, such as (in my case):
- f5.yml
- infoblox.yml
- openshift.yml
- mcafee.yml
Once a custom object is mature enough, it could be migrated to ecs.
from ecs.
@willemdh WDYT about my idea here: #55 ? I would expect a f5.yml
to also contain ECS fields but not only.
from ecs.
@ruflin Well my idea was to work with 2 templates and that f5.yml would only contain the dedicated f5 object fields. ECS fields would come from the template.json, which should have a higher template order in that case. As I'm personnally not interested in fields that have no use for me (and I can imagine fields.yml could grow fast), Î was thinking aggregating all 'maybe-ecs' fields into 1 fields.yml would make it bloated and 95 % unusable..
On the other side, as you suggest above, If I don't want to use more then 2 templates, this approach means I would have to group everything which is not already in ECS in f5.yml while there are definitely use cases where f5 could contain other root level data which is also not in ECS, but could also deserve their own root object.
Seeing as you did quite a bit of work already in #55 I defintely see the added value in your approach. As for now I'm integrating the ecs fields into my f5 template, I see no problem in testing your method and re-evaluating if necessary. Not sure if your pr means these mappings will be added to template.json too?
from ecs.
The PR #55 is only for sharing fields and was not thinking yet of the implication of template. It also has no effect on the template.json that is generated, this is ECS only. The idea would be to have there a f5.yml
file which contains the f5
fields and the ECS fields you need (not all of them). That seems also to be what you were planning to do.
from ecs.
That seems also to be what you were planning to do.
Correct.
Fyi, I have a lot more use cases then only F5. The F5 grok patterns are just some sort of a hobby project which I manage in my private time..
from ecs.
@willemdh Good to hear. As soon as #55 is merge feel free to open PR's against use cases. I think the more use cases we have and see, the better we understand what needs to be in ECS (and potentially also what not).
from ecs.
Related Issues (20)
- Clarify the type of container disk and network metrics HOT 1
- ECS Vulnerability Published field
- Add threat.indicator.tags field
- [Proposal] Make event.kind a list HOT 3
- Incorrect generated/beats/fields.ecs.yml, not accounting on top_level: false HOT 1
- [ECS] Addition - http.request.header.bytes & http.response.header.bytes HOT 2
- Add lowercase normaliser to ECS fields which support security incident response process
- Mark experimental fields as `beta` in generated files
- Elastic-Agent Integrations Use of Legacy Mapping Types Impacts .fleet_globals & prevents agents from being upgraded
- Add `related.url` field
- Add event.zone and event.environment fields
- Addition of additional allowed values for event.type
- Support cloud events in schema HOT 1
- Better abstraction of the type event.kind: alert
- ECS can no longer map all components out of the box HOT 13
- Define a standard way to identify prevention and detection security alerts HOT 5
- Support multi-key fields from SemConv HOT 5
- Allow risk object to be nested under network.
- Add a multi-field user.id.text to the user.id field.
- [Discuss] Add `agent.group` and `host.group` field
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ecs.