Mongo Aggregations or MapReduce queries about stix-shifter

[JasonKeirstead]

@StephenOTT Can you go into more detail as to what you're looking for here?

[StephenOTT]

Shifter provides the ability to convert from the stix pattern into another format, such as going from the stix pattern to the elastic search query. Would be great if same could be done but with mongo aggregation query. So you can go from stix pattern to mongo aggregation query. (Mongo aggregation query is just another json object)

[JasonKeirstead]

@StephenOTT First the data format and layout for the security data living in Mongo that we're trying to go against would need to be defined. Shifter doesn't work if it doesn't understand the data... since a Mongo database can contain "anything", this is a problem.

[StephenOTT]

I think that would be fine. You are basically doing the same for elastic indexes?

[JasonKeirstead]

@StephenOTT Kind of, except with Elastic we have some standard schemas to target. MITRE has defined a translation to their CAR schema, and we will also be developing a translation to the ECS standard schema. The goal of Shifter is to work "out of the box" for most security products.

If there is some kind of standard schema for Mongo you have in mind that is in use in a product we would definitely look at this.

[pcoccoli]

I don't know anything about MongoDB, but some data sources have aggregations. For example in QRadar AQL you can GROUP BY and then use an aggregation function (IIUC). In STIX Observations, there is first_observed, last_observed, and number_observed, so it seems like we should be able to handle simple "count" aggregations, at least for data sources that support it.

Supporting such aggregations gives us a way to "push" some of the computational burden down the stack, and reduce the amount of data transmitted.