sekoia-io / intake-formats

The intake formats supported by SEKOIA.IO

Intake Parsers

Welcome to the Sekoia.io Intakes repository!

This repository contains all the community parsers of Sekoia.io. Parsers are fundamental parts of a cyber analysis and detection process. They extract useful information conveyed by events and make them understandable to decision-making processes. A poor quality parser does not allow informed decision making. Conversely, a good quality parser extracts and normalizes all the useful information present in an event to maximizes its decision making.

We have created this space to ensure the quality of our parsers and allow our users to participate in their development. Contact [email protected] for questions and feedback.

Example

name: my-intake
pipeline:
  - name: parsed_event
    external:
      name: json.parseJSON
  - name: network
    filter: '{{parsed_event.message.log_type == "network"}}'
  - name: file
    filter: '{{parsed_event.message.log_type == "file"}}'
stages:
  network:
    actions:
      - set:
          destination.ip: "{{parsed_event.message.traffic.target}}"
          source.ip: "{{parsed_event.message.traffic.source}}"
  file:
    actions:
      - set:
          file.name: "{{parsed_event.message.file_name | lower}}"

Documentation

Documentation along with tutorials and examples are available in the doc directory.

Organization

Intakes are organized in modules. Each module is associated with a category or a product vendor. An intake is made up of three directories:

1. the _meta directory contains all the intake metadata such as its description or the icons used for its integration into the public catalog of Sekoia.io,
2. the ingest directory contains the definition of the parser,
3. the events directory is optional and contains the definitions of smart-descriptions and lookup tables which adapt the processing and events display within Sekoia.io interfaces.

Contributing

You can click the Fork button in the upper-right area of the screen to create a copy of this repository in your GitHub account. This copy is called a fork. Make any changes you want in your fork, and when you are ready to send those changes to us, go to your fork and create a new pull request to let us know about it.

Once your pull request is created, a Sekoia.io reviewer will take responsibility for providing clear, actionable feedback. Once merged, your changes will be included in the next release to be deployed on Sekoia.io.

intake-formats's Issues

vade: fix parsing of email.message_id (remove brackets)

Notes:

• some message ids don't have brackets
• some events seem to have multiple messages (message is an array), not sure why

Add a test for Smart Descriptions

CI testing for smart descriptions

• Create python test unit for smart description
• Add CI check for smart description

Cloudflare Access Requests - parsing error: `set:set_cloudflare_fields:cloudflare - isn't defined in taxonomy`

Hi!

This event (anonymized from a real one) causes a parsing error in the Cloudflare Access Requests intake, with the message
set:set_cloudflare_fields:cloudflare - isn't defined in taxonomy.

{"Action":"sso","Allowed":true,"AppDomain":"foo-bar-baz.xyz","AppUUID":"f1c47079-d821-3e7a-8f67-625c1e4fb6a4","Connection":"onetimepin","Country":"fr","CreatedAt":"2023-04-27T14:40:10Z","Email":"[email protected]","IPAddress":"78.101.123.45","PurposeJustificationPrompt":"why?","PurposeJustificationResponse":"because!","RayID":"7bf00f0ea9e9f858","TemporaryAccessApprovers":["[email protected]"],"TemporaryAccessDuration":900,"UserUID":"03251629-69f1-5ba0-a154-463be885554b"}

office365: see if InternetMessageId can be parsed in AirInvestigation events

CrowdStrike IDP timestamp are parsed incorrectly

In CrowdStrike IDP events, some timestamp are not in the UNIX format but in the LDAP format.

They are currently not converted correctly and events are inserted several years in the past because of this.

For an example, look for events with crowdstrike.event_type: IdpDetectionSummaryEvent.

If useful, here's how the official Elastic parser handles the conversoin:
https://github.com/elastic/integrations/blob/main/packages/crowdstrike/data_stream/falcon/elasticsearch/ingest_pipeline/default.yml#L38-L80

Infoblox DDI: DNS query logs are not parsed for some alternate logging configuration

On intake 0d1009c7-8c78-476a-8c1c-2439ddb57462, the DNS query logs are not parsed correctly.

The problem seems to be that the syslog message starts with queries: client , e.g.:

queries: client @0x7f62b80115d0 192.168.0.2#55473 (docs.sekoia.io): query: docs.sekoia.io IN AAAA + (192.168.0.1)

whereas the parser expects the message to start with client :

intake-formats/Infoblox/ddi/ingest/parser.yml

Lines 8 to 19 in 59d3514

	pattern: "%{CLIENT}"
	custom_patterns:
	QUERY_FLAGS: "%{QUERY_FLAGS_RD:flags_rd}%{QUERY_FLAGS_EDNS:flags_edns}?%{QUERY_FLAGS_TCP:flags_tcp}?%{QUERY_FLAGS_DNSSEC:flags_dnssec}?%{QUERY_FLAGS_CD:flags_cd}?%{QUERY_FLAGS_DNS_SERVER_COOKIE}?%{QUERY_FLAGS_DNS_SERVER_COOKIE_WITHOUT_VALID_SERVER}?"
	QUERY_FLAGS_RD: '[\+\-]'
	QUERY_FLAGS_SIGNED: "S?"
	QUERY_FLAGS_EDNS: 'E(\(%{INT}\))?'
	QUERY_FLAGS_TCP: "T"
	QUERY_FLAGS_DNSSEC: "D"
	QUERY_FLAGS_CD: "C"
	QUERY_FLAGS_DNS_SERVER_COOKIE: "V"
	QUERY_FLAGS_DNS_SERVER_COOKIE_WITHOUT_VALID_SERVER: "K"
	CLIENT: 'client ?(%{DATA}) %{IP:src}#%{INT:spt} (%{DATA}): query: %{IPORHOST:dns_question_name} %{WORD:dns_question_class} %{WORD:dns_question_type} %{QUERY_FLAGS} \(%{IP}\)'

According to the Infoblox docs, the name of the log category (here queries: ) can be prefixed or not to the syslog messages depending on the configuration.

Logging Category: Select one of the following logging categories:

• Send all: [...] the syslog messages are not prefixed when you select this option.
• Send selected categories: [...] The syslog messages are prefixed with a category name to which it belongs. [...]

The integration documentation doesn't currently specify either option.

I think that we should modify the parser to handle both cases.

Here's how the official Elastic integration seems to parse the logs:

^(%{NOTSPACE:infoblox_nios.log.dns.category}:)?\\s*client %{DATA} %{IP:client.ip}...