opencybersecurityalliance / stix-shifter

This project consists of an open source library allowing software to connect to data repositories using STIX Patterning, and return results as STIX Observations.

Introduction

STIX-shifter is an open source python library allowing software to connect to products that house data repositories by using STIX Patterning, and return results as STIX Observations.

This library takes in STIX 2 Patterns as input, and "finds" data that matches the patterns inside various products that house repositories of cybersecurity data. Examples of such products include SIEM systems, endpoint management systems, threat intelligence platforms, orchestration platforms, network control points, data lakes, and more.

In addition to "finding" the data by using these patterns, STIX-Shifter also transforms the output into STIX 2 Observations. Why would we do that you ask? To put it simply - so that all of the security data, regardless of the source, mostly looks and behaves the same.

Project Documenation

For general information about STIX, this project, and the command line utilities, see the STIX-shifter Documenation

Installation

The recommended method for installing stix-shifter is via pip. Two prerequisite packages needs to be installed inlcuding the package of stix-shifter connector module to complete a stix-shifter connector installation. Run the below commands to install all the packages:

1. Main stix-shifter package: pip install stix-shifter

2. Stix-shifter Utility package: pip install stix-shifter-utils

3. Desired stix-shifter connector module package: pip install stix-shifter-modules-<module name> Example: pip install stix-shifter-modules-qradar

Dependencies

STIX-shifter requries Python 3.8 or greater. See the requirements file for library dependencies.

Usage

STIX-Shifter can use used the following ways:

As a command line utility

The STIX-Shifter comes with a bundled script which you can use to translate STIX Pattern to a native datasource query. It can also be used to translate a JSON data source query result to a STIX bundle of observable objects. You can also send query to a datasource by using a transmission option.

More details of the command line option can be found here

$ stix-shifter translate <MODULE NAME> query "<STIX IDENTITY OBJECT>" "<STIX PATTERN>" "<OPTIONS>"

Example:

$ stix-shifter translate qradar query {} "[ipv4-addr:value = '127.0.0.1']" {}

In order to build stix-shifter packages from source follow the below prerequisite steps:

1. Go to the stix-shifter parent directory
2. Optionally, you can create a Python 3 virtual environemnt: virtualenv -p python3 virtualenv && source virtualenv/bin/activate
3. Run setup: python3 setup.py install

Running from the source

You may also use the python3 main.py script. All the options are the same as the command line utility described above.

Example:

python3 main.py translate qradar query {} "[ipv4-addr:value = '127.0.0.1']" {}

In order to run python3 main.py from the source follow the below prerequisite steps:

1. Go to the stix-shifter parent directory
2. Optionally, you can create a Python 3 virtual environemnt: virtualenv -p python3 virtualenv && source virtualenv/bin/activate
3. Run setup to install dependancies: INSTALL_REQUIREMENTS_ONLY=1 python3 setup.py install.

Note: setup.py only installs dependencies when INSTALL_REQUIREMENTS_ONLY=1 directive is used. This option is similar to python3 generate_requirements.py && pip install -r requirements.txt

As a library

You can also use this library to integrate STIX Shifter into your own tools. You can translate a STIX Pattern:

from stix_shifter.stix_translation import stix_translation

translation = stix_translation.StixTranslation()
response = translation.translate('<MODULE NAME>', 'query', '{}', '<STIX PATTERN>', '<OPTIONS>')

print(response)

Use of custom mappings

If a connector has been installed using pip, the process for editing the STIX mappings is different than if you have pulled-down the project. When working locally, you can edit the mapping files directly. See the mapping files for the MySQL connector as an example. Editing the mapping files won't work if the connector has been installed with pip; the setup script of the stix-shifter package includes the mapppings inside config.json. This allows stix-shifter to injest custom mappings as part of the connector's configuration.

Refer to Use of custom mappings for more details on how to edit the mappings in the configuration.

Contributing

We are thrilled you are considering contributing! We welcome all contributors. Please read our guidelines for contributing.

Connector Developer Guide

CLI tools and Connector Development Labs

Licensing

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

More Resources

Join us on Slack!

Click here and fill out the form to receive an invite to the Open Cybersecurity Alliance slack instance, then join the #stix-shifter channel, to meet and discuss usage with the team.

Introduction Webinar!

Click here to view an introduction webinar on STIX Shifter and the use cases it solves for.

Changelog

• Changelog

stix-shifter's Issues

`network-traffic:src/dst_ref.value` of domain-name failed for QRadar

Describe the bug
Currently network-traffic:src_ref.value and network-traffic:dst_ref.value are mapped in AQL to IP address or MAC address, however, domain-name is also a possibility for this field according to STIX 2.1 doc for example [network-traffic:dst_ref.value = 'example.com'] however, the STIX Shifter will return a mapping error for the translation, even though the mapping exists for network-traffic:dst_ref.value (if I change the value to '127.0.0.1', the translation is okay).

example at 9.8 - https://docs.oasis-open.org/cti/stix/v2.1/csprd01/stix-v2.1-csprd01.html#9.8

On the full example of the pattern they use a type field as well, [network-traffic:dst_ref.type = 'domain-name' AND network-traffic:dst_ref.value = 'example.com'] but this type field is not mapped at the STIX -> AQL mapping, thus return error as well.

To Reproduce
Steps to reproduce the behavior:

1. Try to translate for QRadar: translate qradar query "{}" "[network-traffic:dst_ref.value = 'example.com']" "{}"

Strip out unmapped STIX comparison operators

We currently use the UnmappedAttributeStripper class in stix-shifter to remove comparison statements (from the antlr parsing of the STIX pattern) before translating into a native data source query. This allows us to still construct a query if logically possible (ie. comparison statements joined by OR). We want to do something similar with unmapped operators. Not every data source query language supports the full collection of STIX operators (ie. matches).

The operators are defined in a dictionary in each module's query_constructor. They should probably get moved out into a mapping json file for better visibility. We may be able to just add to the attribute stripper logic.

STIX to AQL blows up on START STOP when using a leading t in timestamp

The leading t in the timestamp, as shown in this example taken from the stix documentation

[file:name = 'foo.dll'] START t'2016-06-01T00:00:00Z' STOP t'2016-07-01T00:00:00Z'

is blowing up the translation. However the leading t also causes this pattern to fail STIX validation so there may be a problem in either the documentation of the validation module from https://github.com/oasis-open/cti-stix-validator

Update ECS mappings to cover all fields

Describe the bug
The current mapping covers a small set of ECS fields that match with STIX objects and properties. There are many that are still missing. Any fields that don't fit with an official STIX object should be associated with a custom object (using the format of x-ecs-<object name>) or a custom property on an official STIX object (using the format of x_ecs_<property name>)

The ECS field reference can be found at https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html.

Map Carbon Black Response `last_update` process field to `last_observed`

Describe the bug
Missing support for Carbon Black Response last_update process fields.

To Reproduce
N/A

Expected behavior
Currently, the mapping for the start field in the Carbon Black Response to_stix_map.json translation file is as follows:

"start": [
    {
      "key": "process.created",
      "object": "process"
    },
    {
      "key": "first_observed",
      "cybox": false
    },
    {
      "key": "last_observed",
      "cybox": false
    }
  ]

However, Carbon Black Response process object provide a last_update field that designates when a running process was last updated. This field would map better to last_observed.

Screenshots
Please see: https://github.com/opencybersecurityalliance/stix-shifter/blob/develop/stix_shifter_modules/carbonblack/stix_translation/json/to_stix_map.json

Desktop (please complete the following information):
N/A

Smartphone (please complete the following information):
N/A

Incorrect QRadar translation of pattern with grouping via parens

Describe the bug
When 2 or more observation expressions are ANDed or ORed together, wrapped in parentheses, and followed by a START/STOP qualifier, the qualifier should apply to ALL the observations expressions in the parens. The AQL queries produced are not doing that.

To Reproduce
Steps to reproduce the behavior:

$ stix-shifter translate qradar query {} "([ipv4-addr:value = '192.168.1.2'] OR [url:value LIKE '%.example.com']) START t'2020-09-11T13:00:52.000Z' STOP t'2020-09-11T13:59:04.000Z'" {}
{
    "queries": [
        "SELECT QIDNAME(qid) as qidname, qid, CATEGORYNAME(category) as categoryname, category as categoryid, CATEGORYNAME(highlevelcategory) as high_level_category_name, highlevelcategory as high_level_category_id, logsourceid, devicetype, LOGSOURCETYPENAME(devicetype) as logsourcetypename, LOGSOURCENAME(logsourceid) as logsourcename, starttime, endtime, devicetime, sourceaddress as sourceip, sourceport, sourcemac, destinationaddress as destinationip, destinationport, destinationmac, username, eventdirection as direction, identityip, identityhostname, eventcount, PROTOCOLNAME(protocolid) as protocol, UTF8(payload) as eventpayload, URL as url, magnitude, Filename as filename, \"File Hash\" as filehash, \"File Path\" as filepath, severity, credibility, relevance, geographiclocation as geographic, \"CRE Name\" as crename, \"CRE Description\" as credescription, creeventlist, rulename(creeventlist) as rulenames, domainid, DOMAINNAME(domainid) as domainname FROM events WHERE url LIKE '%%.example.com%' limit 10000 START 1599829252000 STOP 1599832744000",
        "SELECT QIDNAME(qid) as qidname, qid, CATEGORYNAME(category) as categoryname, category as categoryid, CATEGORYNAME(highlevelcategory) as high_level_category_name, highlevelcategory as high_level_category_id, logsourceid, devicetype, LOGSOURCETYPENAME(devicetype) as logsourcetypename, LOGSOURCENAME(logsourceid) as logsourcename, starttime, endtime, devicetime, sourceaddress as sourceip, sourceport, sourcemac, destinationaddress as destinationip, destinationport, destinationmac, username, eventdirection as direction, identityip, identityhostname, eventcount, PROTOCOLNAME(protocolid) as protocol, UTF8(payload) as eventpayload, URL as url, magnitude, Filename as filename, \"File Hash\" as filehash, \"File Path\" as filepath, severity, credibility, relevance, geographiclocation as geographic, \"CRE Name\" as crename, \"CRE Description\" as credescription, creeventlist, rulename(creeventlist) as rulenames, domainid, DOMAINNAME(domainid) as domainname FROM events WHERE (sourceip = '192.168.1.2' OR destinationip = '192.168.1.2' OR identityip = '192.168.1.2') OR limit 10000 last 5 minutes",
        "SELECT applicationid, APPLICATIONNAME(applicationid) as applicationname, CATEGORYNAME(category) as categoryname, credibility, destinationasn, destinationbytes, destinationdscp, destinationflags, destinationip, destinationifindex, destinationpackets, destinationport, destinationprecedence, destinationv6, domainid, fullmatchlist, firstpackettime, flowbias, flowdirection as direction, flowinterfaceid, flowsource, flowtype, geographic, hasoffense, icmpcode, icmptype, flowinterface, intervalid, isduplicate, lastpackettime, partialmatchlist, PROTOCOLNAME(protocolid) as protocol, QIDNAME(qid) as qidname, qid, processorid, relevance, retentionbucket, severity, sourceasn, sourcebytes, sourcedscp, sourceflags, sourceip, sourceifindex, sourcepackets, sourceport, sourceprecedence, sourcev6, starttime, endtime, UTF8(sourcepayload) as flowsourcepayload, UTF8(destinationpayload) as flowdestinationpayload FROM flows WHERE (sourceip = '192.168.1.2' OR destinationip = '192.168.1.2') limit 10000 START 1599829252000 STOP 1599832744000"
    ]
}

Three queries are produced; one for the ipv4-addr from events, one for the url from events, and one for the ipv4-addr from flows. The query for the ipv4-addr from events is not using the qualifier; it instead is using the default "last 5 minutes". Also, it seems to have a stray OR before the limit clause, though I am not an AQL expert.

Expected behavior
All three queries should use the START/STOP qualifier.

Application identified in the flow data should appear in search results

Describe the bug
QRadar flow data identifies applications based on default mapping or user defined application mapping. When returning the search results from Flow data, the application must be transformed to equivalent STIX object. Likewise, support for "application" in the STIX query should include Flow data in the results.

Multiple values of `artifcat:payload_bin` translation for QRadar

QRadar module keywords (text in event payload) - STIX Shifter currently supports payload text search with "lucene" TEXT SEARCH. Some STIX patterns contains multiple "keywords", for example:

"[artifact:payload_bin LIKE '%Set-ItemProperty%' AND(OR) artifact:payload_bin LIKE '%New-Item%']"

which translated to:
"FROM events WHERE TEXT SEARCH '%Set-ItemProperty%' AND(OR) TEXT SEARCH '%New-Item%'"

and is invalid AQL, using "lucene" TEXT SEARCH should be like:
TEXT SEARCH '%Set-ItemProperty% AND(OR) %New-Item%'

Moreover, you also cannot use any other field in the WHERE clause when using TEXT SEARCH, for example:
[ipv4-addr:value = '127.0.0.1' AND artifact:payload_bin LIKE 'x'] will cause an invalid AQL

Another way to make it work out-of-the-box is using UTF8(payload) LIKE '...' without "lucene".

Elastic ECS translation and LIKE operator

Describe the bug
A query translation for elastic_ecs datasource that contains LIKE operator with a wildcard is behaving incorrectly when there is whitespace between the terms, for example the query "[(x-event:action LIKE 'Registry %')]" which should look for an event action that starts with Registry , will look for matching Registry OR % individually instead - "event.action : Registry * AND (@timestamp:[\"2020-08-23T13:02:04.184Z\" TO \"2020-08-23T13:07:04.184Z\"])",
and then you'll get many unrelated results.

the fix would be wrapping with quotes when doing LIKE query,
"event.action : \"Registry *\" AND (@timestamp:[\"2020-08-23T13:03:08.041Z\" TO \"2020-08-23T13:08:08.041Z\"])"

query should result in event.aciton : "Registry *" and not event.aciton : Registry *

I am attaching a pull request to fix the issue #414

Documentation issue - reference to data_mapper.py

Describe the bug
https://github.com/opencybersecurityalliance/stix-shifter/blob/master/adapter-guide/develop-translation-module.md
instructs the developer to make sure that their stix_translation subdirectory contains data_mapper.py

This appears to be obsolete and should be removed.

QRadar adapter needs to be smarter about data mappings (needs to take advantage of category and possibly device type)

Currently the QRadar adapter's mapping methodology is based on what fields are present in the event. If the right fields are present, then they get mapped and a resultant STIX object is created.

This can result in sub-optimal mappings, because in order to know what fields should be mapped and how, you really need to look at the event category.

As an example - almost all records in QRadar have a source and destination IP address assigned. This however, in and of itself, is not enough evidence to ensure that an event is communicating a network traffic event, for a few reasons.

The adapter should look at the category to help decide what STIX objects to create. It possibly should also look at the device type. We probably need another configuration file for this adapter that maps these things.

Add missing fields to carbon black STIX-to-binary query mapping

There are some fields defined in the CB Response API doc that aren't yet mapped. Some fields are represented in both process and binary queries, in which case the query defaults to binary (though I'm not sure why it was built that way).

There is an issue however with having patterns that mixes fields that are only represented in process or binary but not both. For example, if I have a pattern like [process_observation OR binary_observation], I will get a translated query looking for both of the mapped fields in a binary query; I should instead get two queries, one for process and one for binary only looking for the fields relevant to each.

missing `registry` and `dns` in query to ecs

There are mapping to STIX of fields like registry and dns in the mapping file, but if you take a look into the api_client you can notice that those keys registry and dns are not being query from elastic at all.

fixed at #410

Mongo Aggregations or MapReduce queries

Would be great (unless i missed it in the docs) for a Mongo Aggregation and/or mapreduce query generation.

stix-shifter is breaking on stix2-patterns 1.2.0

Describe the bug
The CTI Pattern Validator project was updated a few days ago from 1.1.0 to 1.2.0.
https://github.com/oasis-open/cti-pattern-validator/releases/tag/v1.2.0
This is causing the following error
ModuleNotFoundError: No module named 'stix2patterns.pattern'
This can be reproduced by running a translation command from the CLI or by just running pytest

To Reproduce
Steps to reproduce the behavior:

1. Create a python virtual environment from the current requirements.txt
2. Run a translation command from the CLI, for instance python main.py translate msatp query '{}' "[ipv4-addr:value = '127.0.0.1']"
3. See error

Expected behavior
There shouldn't be an error and the pattern should be successfully translated.

Additional context
As a quick fix, we may need to just lock the stix2-patterns version to 1.1.0 until we can figure out what's going on.

Python 3.7 will cause some Unit Test Failures

=============================================================================================================== FAILURES ================================================================================================================
/Users/ben.craigca.ibm.com/PycharmProjects/stix-shifter/tests/patterns/test_analytic_translator.py:21: AssertionError: '|whe[54 chars]tch(tag, "dm-process-.*") AND NOT (match(exe, "^%\\.exe$"))))' != '|whe[54 chars]tch(tag, "dm-process-.*") AND NOT (match(exe, "^.*\\.exe$"))))'
/Users/ben.craigca.ibm.com/PycharmProjects/stix-shifter/tests/patterns/test_analytic_translator.py:21: AssertionError: '|whe[30 chars]4) AND (tag="process" AND NOT (match(process, "^%\\.exe$"))))' != '|whe[30 chars]4) AND (tag="process" AND NOT (match(process, "^.*\\.exe$"))))'
/Users/ben.craigca.ibm.com/PycharmProjects/stix-shifter/tests/patterns/test_analytic_translator.py:21: AssertionError: '|where (match(tag, "dm-file-.*") AND match(file_name, "^%\\.exe$"))' != '|where (match(tag, "dm-file-.*") AND match(file_name, "^.*\\.exe$"))'
/Users/ben.craigca.ibm.com/PycharmProjects/stix-shifter/tests/patterns/test_analytic_translator.py:21: AssertionError: '|where (match(tag, "dm-file-.*") AND NOT (match(file_name, "^%\\.exe$")))' != '|where (match(tag, "dm-file-.*") AND NOT (match(file_name, "^.*\\.exe$")))'
/Users/ben.craigca.ibm.com/PycharmProjects/stix-shifter/tests/patterns/test_web_api.py:38: AssertionError: '|whe[54 chars]tch(tag, "dm-process-.*") AND NOT (match(exe, "^%\\.exe$"))))' != '|whe[54 chars]tch(tag, "dm-process-.*") AND NOT (match(exe, "^.*\\.exe$"))))'
/Users/ben.craigca.ibm.com/PycharmProjects/stix-shifter/tests/patterns/test_web_api.py:38: AssertionError: '|where (match(tag, "dm-file-.*") AND match(file_name, "^%\\.exe$"))' != '|where (match(tag, "dm-file-.*") AND match(file_name, "^.*\\.exe$"))'
/Users/ben.craigca.ibm.com/PycharmProjects/stix-shifter/tests/patterns/test_web_api.py:38: AssertionError: '|where (match(tag, "dm-file-.*") AND NOT (match(file_name, "^%\\.exe$")))' != '|where (match(tag, "dm-file-.*") AND NOT (match(file_name, "^.*\\.exe$")))'

Switching to python 3.6 fixes the issue

Conversion of ISSUPERET operator to AQL query is not supported

STIX Pattern:

ipv4-addr:value ISSUPERET '1.1.1.0/24'
Expected AQL query is:

SELECT * FROM events WHERE INCIDR(sourceip, '1.1.1.0/24') OR INCIDR(destinationip, '1.1.1.0/24') OR INCIDR(identityip, '1.1.1.0/24')

But running above AQL query throws error because AQL built in function INCIDR() doesn't validate CIDR in the second column/parameter which is INCIDR(sourceip, '1.1.1.0/24').

A bug has been logged for the QRadar AQL team. Once that is fixed, this support should be added in QRadar module.

Failed to use "translate" from CLI when value includes backslashes (QRadar)

Describe the bug
Running translate command from CLI when value includes backslashes returns an error and failed to translate for QRadar AQL.

To Reproduce
Steps to reproduce the behavior:

1. Open terminal
2. Type: python main.py translate qradar query "{}" "[(x-ibm-ariel:event_name = 'event\\example')]" "{}" which is valid STIX pattern for QRadar
3. Results in an error, however when using the same pattern through the Data Explorer it does work properly.

Expected behavior
AQL query: ... FROM events WHERE qidname = 'event\\example'

Screenshots

Desktop (please complete the following information):

• OS: OSX terminal

Error when creating reference object during to-stix translation

Describe the bug
I was adding references in Qradar and had added this mapping in the to_stix_map.json

 "filepath": [
     {
       "key": "directory.path",
       "object": "dir"
     },
     {
       "key": "file.parent_directory_ref",
       "object": "fl",
       "references": "dir"
     }
   ]

It was discovered that if the data source returned a null file hash and file path, the translation would blow up. This will take a bit of investigation, but I think this is because the value is attempted to reference the non-existent directory object group. json-to-stix will need to be updated to make sure reference objects actually exist before connecting to them.

To Reproduce
Steps to reproduce the behavior:

1. Add in a mapping like listed above where data creates both a STIX object and a reference from a second object that points to the first.
2. Use test data for the mapping where the value comes in as None/nil.
3. An error should be thrown during the to-STIX translation.

Expected behavior
References should only get created and point to an object that actually exists.

Order of observations in STIX pattern effects resulting AQL query.

The following pattern:

"[network-traffic:src_port = 37020 AND network-traffic:dst_port = 635] START '2018-08-01T00:00:00Z' STOP '2018-08-01T01:11:11Z' OR [domain-name:value = 'example.com' and mac-addr:value = '00-00-5E-00-53-00'] OR [ipv4-addr:value = '333.333.333.0'] START '2018-08-01T00:00:00Z' STOP '2018-08-01T01:11:11Z'"

generated only two queries where a domain name, Mac address, and ip were all treated as one query.
I would expect the mac address and domain name to be in their own query since the start stop at the end of the pattern should only apply to the ipv4 observation.

This bug is dependant on the ordering of the observations in the pattern. If an observation without the START STOP qualifier is put at the end, it is correctly put into it's own AQL query.

Requirements state antlr4-python3-runtime==4.7 but installation fails requiring 4.8

Steps to reproduce the behavior:

1. Go to https://github.com/opencybersecurityalliance/stix-shifter

2. Install the requirements
   $ pip install git+git://github.com/oasis-open/[email protected]#egg=stix2-matcher

3. instructions state: antlr4-python3-runtime==4.7

4. but installation requires 4.8
   error: ERROR: stix2-patterns 1.3.0 has requirement antlr4-python3-runtime~=4.8.0; python_version >= "3", but you'll have antlr4-python3-runtime 4.7 which is incompatible.

(venv-3.8.2) user1:stix-shifter user1 $ pip install git+git://github.com/oasis-open/[email protected]#egg=stix2-matcher
Collecting stix2-matcher from git+git://github.com/oasis-open/[email protected]#egg=stix2-matcher
  Cloning git://github.com/oasis-open/cti-pattern-matcher.git (to revision v0.1.0) to /private/var/folders/26/1tdp9y855zd3xct_24w3fxm80000gn/T/pip-install-uz1bwjag/stix2-matcher
  Running command git clone -q git://github.com/oasis-open/cti-pattern-matcher.git /private/var/folders/26/1tdp9y855zd3xct_24w3fxm80000gn/T/pip-install-uz1bwjag/stix2-matcher
  Running command git checkout -q fbe7b5010ac42299344f702034e8970e0f5b8e79
Requirement already satisfied: python-dateutil in /Users/user1/.pyenv/versions/3.8.2/envs/venv-3.8.2/lib/python3.8/site-packages (from stix2-matcher) (2.8.1)
Requirement already satisfied: six in /Users/user1/.pyenv/versions/3.8.2/envs/venv-3.8.2/lib/python3.8/site-packages (from stix2-matcher) (1.14.0)
Requirement already satisfied: stix2-patterns>=1.0.0 in /Users/user1/.pyenv/versions/3.8.2/envs/venv-3.8.2/lib/python3.8/site-packages (from stix2-matcher) (1.3.0)
Collecting antlr4-python3-runtime==4.7 (from stix2-matcher)
  Using cached https://files.pythonhosted.org/packages/0b/6b/30c5b84d203b62e1412d14622e3bae6273399d79d20f3a24c8145213f610/antlr4-python3-runtime-4.7.tar.gz
ERROR: stix2-patterns 1.3.0 has requirement antlr4-python3-runtime~=4.8.0; python_version >= "3", but you'll have antlr4-python3-runtime 4.7 which is incompatible.

Inconsistency in API contract for translate function

Describe the bug
There is an inconsistency in the API contract for the translate() function in STIX Translation module, because most translation modules return a list of queries, while the bundle module returns a single query string. This means the API does not have a solid contract and you have to check the type of the returned value in your code.

To Reproduce
dsl = translation.translate(query['translation_module'], 'query', self.identity_object, query['query'])

Expected behavior
All modules should return a list - if there is only one result it should be a single item list.

There should be a unit test that verifies the contract for modules

Store Qradar payload as base64 artifact

It's currently being run through a function to translate the results to UTF8. A STIX artifact object expects it to be base 64, which is what Qradar returns anyway.

CCB Minutes and Process Improvement

** CCB Tracking ***
Start Pattern of Minute Tracking in Wiki

** Define Template for Process Improvement **
(Label Process in Git)
(Template for Issues

Prioritization Process for PRs and Issues

proxy host translate results issue

There is an unneeded json.dumps in translate_results in proxy_host.py. self.request_args["results"] is already a json

Steps to reproduce the behavior:
Run stix shifter in host mode, configure a proxy on cp4s, query data source using this host proxy. Results are malformed because of the unneeded json.dumps in translate_results in proxy_host.py

Can't install package - Mac OSX

Cloned repo, executed sudo python3 setup.py build, and then tried sudo python3 setup.py install

$ sudo python3 setup.py install

<SNIP>
copying build/lib/stix_shifter/src/modules/base/base_result_translator.py -> build/bdist.macosx-10.9-x86_64/egg/stix_shifter/src/modules/base
byte-compiling build/bdist.macosx-10.9-x86_64/egg/tests/patterns/test_web_api.py to test_web_api.cpython-35.pyc
byte-compiling build/bdist.macosx-10.9-x86_64/egg/tests/patterns/generate_test_case.py to generate_test_case.cpython-35.pyc
byte-compiling build/bdist.macosx-10.9-x86_64/egg/tests/patterns/__init__.py to __init__.cpython-35.pyc
byte-compiling build/bdist.macosx-10.9-x86_64/egg/tests/patterns/test_miscellaneous_tests.py to test_miscellaneous_tests.cpython-35.pyc
byte-compiling build/bdist.macosx-10.9-x86_64/egg/tests/patterns/test_analytic_translator.py to test_analytic_translator.cpython-35.pyc
byte-compiling build/bdist.macosx-10.9-x86_64/egg/tests/patterns/integration_tests.py to integration_tests.cpython-35.pyc
byte-compiling build/bdist.macosx-10.9-x86_64/egg/tests/patterns/helpers/connectors.py to connectors.cpython-35.pyc
byte-compiling build/bdist.macosx-10.9-x86_64/egg/tests/patterns/helpers/input_file_helpers.py to input_file_helpers.cpython-35.pyc
byte-compiling build/bdist.macosx-10.9-x86_64/egg/tests/qradar_stix_to_aql/test_class.py to test_class.cpython-35.pyc
byte-compiling build/bdist.macosx-10.9-x86_64/egg/tests/qradar_stix_to_aql/__init__.py to __init__.cpython-35.pyc
byte-compiling build/bdist.macosx-10.9-x86_64/egg/tests/qradar_json_to_stix/test_class.py to test_class.cpython-35.pyc
  File "build/bdist.macosx-10.9-x86_64/egg/tests/qradar_json_to_stix/test_class.py", line 88
    assert(ip_ref in objects), f"dst_ref with key {nt_object['dst_ref']} not found"
                                                                                  ^
SyntaxError: invalid syntax

byte-compiling build/bdist.macosx-10.9-x86_64/egg/tests/qradar_json_to_stix/__init__.py to __init__.cpython-35.pyc
byte-compiling build/bdist.macosx-10.9-x86_64/egg/tests/car_json_to_stix/test_class.py to test_class.cpython-35.pyc
  File "build/bdist.macosx-10.9-x86_64/egg/tests/car_json_to_stix/test_class.py", line 100
    assert(binary_ref in objects), f"binary_ref with key {binary_ref} not found"
                                                                               ^
SyntaxError: invalid syntax

byte-compiling build/bdist.macosx-10.9-x86_64/egg/tests/car_json_to_stix/__init__.py to __init__.cpython-35.pyc
byte-compiling build/bdist.macosx-10.9-x86_64/egg/stix_shifter/stix_shifter.py to stix_shifter.cpython-35.pyc
byte-compiling build/bdist.macosx-10.9-x86_64/egg/stix_shifter/__init__.py to __init__.cpython-35.pyc
byte-compiling build/bdist.macosx-10.9-x86_64/egg/stix_shifter/src/patterns/grammar/STIXPatternVisitor.py to STIXPatternVisitor.cpython-35.pyc
byte-compiling build/bdist.macosx-10.9-x86_64/egg/stix_shifter/src/patterns/grammar/STIXPatternListener.py to STIXPatternListener.cpython-35.pyc
byte-compiling build/bdist.macosx-10.9-x86_64/egg/stix_shifter/src/patterns/grammar/STIXPatternLexer.py to STIXPatternLexer.cpython-35.pyc
byte-compiling build/bdist.macosx-10.9-x86_64/egg/stix_shifter/src/patterns/grammar/STIXPatternParser.py to STIXPatternParser.cpython-35.pyc
byte-compiling build/bdist.macosx-10.9-x86_64/egg/stix_shifter/src/patterns/grammar/__init__.py to __init__.cpython-35.pyc
byte-compiling build/bdist.macosx-10.9-x86_64/egg/stix_shifter/src/patterns/__init__.py to __init__.cpython-35.pyc
byte-compiling build/bdist.macosx-10.9-x86_64/egg/stix_shifter/src/patterns/parser.py to parser.cpython-35.pyc
byte-compiling build/bdist.macosx-10.9-x86_64/egg/stix_shifter/src/patterns/pattern_objects.py to pattern_objects.cpython-35.pyc
byte-compiling build/bdist.macosx-10.9-x86_64/egg/stix_shifter/src/patterns/errors.py to errors.cpython-35.pyc
byte-compiling build/bdist.macosx-10.9-x86_64/egg/stix_shifter/src/patterns/translator.py to translator.cpython-35.pyc
byte-compiling build/bdist.macosx-10.9-x86_64/egg/stix_shifter/src/transformers.py to transformers.cpython-35.pyc
byte-compiling build/bdist.macosx-10.9-x86_64/egg/stix_shifter/src/json_to_stix/json_to_stix.py to json_to_stix.cpython-35.pyc
byte-compiling build/bdist.macosx-10.9-x86_64/egg/stix_shifter/src/json_to_stix/observable.py to observable.cpython-35.pyc
byte-compiling build/bdist.macosx-10.9-x86_64/egg/stix_shifter/src/json_to_stix/__init__.py to __init__.cpython-35.pyc
byte-compiling build/bdist.macosx-10.9-x86_64/egg/stix_shifter/src/json_to_stix/json_to_stix_translator.py to json_to_stix_translator.cpython-35.pyc
byte-compiling build/bdist.macosx-10.9-x86_64/egg/stix_shifter/src/__init__.py to __init__.cpython-35.pyc
byte-compiling build/bdist.macosx-10.9-x86_64/egg/stix_shifter/src/exceptions.py to exceptions.cpython-35.pyc
byte-compiling build/bdist.macosx-10.9-x86_64/egg/stix_shifter/src/modules/car/car_data_mapping.py to car_data_mapping.cpython-35.pyc
byte-compiling build/bdist.macosx-10.9-x86_64/egg/stix_shifter/src/modules/car/__init__.py to __init__.cpython-35.pyc
byte-compiling build/bdist.macosx-10.9-x86_64/egg/stix_shifter/src/modules/car/car_translator.py to car_translator.cpython-35.pyc
  File "build/bdist.macosx-10.9-x86_64/egg/stix_shifter/src/modules/car/car_translator.py", line 27
    obj[f"{typ}.{field}"] = fields[field]
                       ^
SyntaxError: invalid syntax

byte-compiling build/bdist.macosx-10.9-x86_64/egg/stix_shifter/src/modules/dummy/dummy_translator.py to dummy_translator.cpython-35.pyc
byte-compiling build/bdist.macosx-10.9-x86_64/egg/stix_shifter/src/modules/dummy/dummy_query_translator.py to dummy_query_translator.cpython-35.pyc
byte-compiling build/bdist.macosx-10.9-x86_64/egg/stix_shifter/src/modules/dummy/dummy_result_translator.py to dummy_result_translator.cpython-35.pyc
byte-compiling build/bdist.macosx-10.9-x86_64/egg/stix_shifter/src/modules/elastic/elastic_query_constructor.py to elastic_query_constructor.cpython-35.pyc
byte-compiling build/bdist.macosx-10.9-x86_64/egg/stix_shifter/src/modules/elastic/elastic_translator.py to elastic_translator.cpython-35.pyc
byte-compiling build/bdist.macosx-10.9-x86_64/egg/stix_shifter/src/modules/elastic/stix_to_elastic.py to stix_to_elastic.cpython-35.pyc
  File "build/bdist.macosx-10.9-x86_64/egg/stix_shifter/src/modules/elastic/stix_to_elastic.py", line 35
    raise NotImplementedError(f"Module {data_mapper_module_name} not implemented")
                                                                                ^
SyntaxError: invalid syntax

byte-compiling build/bdist.macosx-10.9-x86_64/egg/stix_shifter/src/modules/__init__.py to __init__.cpython-35.pyc
byte-compiling build/bdist.macosx-10.9-x86_64/egg/stix_shifter/src/modules/cim/cim_data_mapping.py to cim_data_mapping.cpython-35.pyc
byte-compiling build/bdist.macosx-10.9-x86_64/egg/stix_shifter/src/modules/qradar/__init__.py to __init__.cpython-35.pyc
byte-compiling build/bdist.macosx-10.9-x86_64/egg/stix_shifter/src/modules/qradar/qradar_data_mapping.py to qradar_data_mapping.cpython-35.pyc
byte-compiling build/bdist.macosx-10.9-x86_64/egg/stix_shifter/src/modules/qradar/aql_query_constructor.py to aql_query_constructor.cpython-35.pyc
byte-compiling build/bdist.macosx-10.9-x86_64/egg/stix_shifter/src/modules/qradar/stix_to_aql.py to stix_to_aql.cpython-35.pyc
byte-compiling build/bdist.macosx-10.9-x86_64/egg/stix_shifter/src/modules/qradar/qradar_translator.py to qradar_translator.cpython-35.pyc
byte-compiling build/bdist.macosx-10.9-x86_64/egg/stix_shifter/src/modules/splunk/stix_to_splunk.py to stix_to_splunk.cpython-35.pyc
  File "build/bdist.macosx-10.9-x86_64/egg/stix_shifter/src/modules/splunk/stix_to_splunk.py", line 35
    raise NotImplementedError(f"Module {data_mapper_module_name} not implemented")
                                                                                ^
SyntaxError: invalid syntax

byte-compiling build/bdist.macosx-10.9-x86_64/egg/stix_shifter/src/modules/splunk/splunk_query_constructor.py to splunk_query_constructor.cpython-35.pyc
byte-compiling build/bdist.macosx-10.9-x86_64/egg/stix_shifter/src/modules/splunk/encoders.py to encoders.cpython-35.pyc
byte-compiling build/bdist.macosx-10.9-x86_64/egg/stix_shifter/src/modules/splunk/object_scopers.py to object_scopers.cpython-35.pyc
byte-compiling build/bdist.macosx-10.9-x86_64/egg/stix_shifter/src/modules/splunk/splunk_translator.py to splunk_translator.cpython-35.pyc
byte-compiling build/bdist.macosx-10.9-x86_64/egg/stix_shifter/src/modules/base/base_translator.py to base_translator.cpython-35.pyc
byte-compiling build/bdist.macosx-10.9-x86_64/egg/stix_shifter/src/modules/base/base_query_translator.py to base_query_translator.cpython-35.pyc
byte-compiling build/bdist.macosx-10.9-x86_64/egg/stix_shifter/src/modules/base/__init__.py to __init__.cpython-35.pyc
byte-compiling build/bdist.macosx-10.9-x86_64/egg/stix_shifter/src/modules/base/base_result_translator.py to base_result_translator.cpython-35.pyc
creating build/bdist.macosx-10.9-x86_64/egg/EGG-INFO
copying stix_shifter.egg-info/PKG-INFO -> build/bdist.macosx-10.9-x86_64/egg/EGG-INFO
copying stix_shifter.egg-info/SOURCES.txt -> build/bdist.macosx-10.9-x86_64/egg/EGG-INFO
copying stix_shifter.egg-info/dependency_links.txt -> build/bdist.macosx-10.9-x86_64/egg/EGG-INFO
copying stix_shifter.egg-info/requires.txt -> build/bdist.macosx-10.9-x86_64/egg/EGG-INFO
copying stix_shifter.egg-info/top_level.txt -> build/bdist.macosx-10.9-x86_64/egg/EGG-INFO
zip_safe flag not set; analyzing archive contents...
stix_shifter.src.modules.qradar.__pycache__.qradar_data_mapping.cpython-35: module references __file__
stix_shifter.src.modules.qradar.__pycache__.qradar_translator.cpython-35: module references __file__
tests.patterns.__pycache__.generate_test_case.cpython-35: module references __file__
tests.patterns.__pycache__.test_web_api.cpython-35: module references __file__
tests.patterns.helpers.__pycache__.input_file_helpers.cpython-35: module references __file__
creating 'dist/stix_shifter-1.0.0-py3.5.egg' and adding 'build/bdist.macosx-10.9-x86_64/egg' to it
removing 'build/bdist.macosx-10.9-x86_64/egg' (and everything under it)
Processing stix_shifter-1.0.0-py3.5.egg
removing '/Users/pmaroney/anaconda3/lib/python3.5/site-packages/stix_shifter-1.0.0-py3.5.egg' (and everything under it)
creating /Users/pmaroney/anaconda3/lib/python3.5/site-packages/stix_shifter-1.0.0-py3.5.egg
Extracting stix_shifter-1.0.0-py3.5.egg to /Users/pmaroney/anaconda3/lib/python3.5/site-packages
  File "/Users/pmaroney/anaconda3/lib/python3.5/site-packages/stix_shifter-1.0.0-py3.5.egg/stix_shifter/src/modules/car/car_translator.py", line 27
    obj[f"{typ}.{field}"] = fields[field]
                       ^
SyntaxError: invalid syntax

  File "/Users/pmaroney/anaconda3/lib/python3.5/site-packages/stix_shifter-1.0.0-py3.5.egg/stix_shifter/src/modules/elastic/stix_to_elastic.py", line 35
    raise NotImplementedError(f"Module {data_mapper_module_name} not implemented")
                                                                                ^
SyntaxError: invalid syntax

  File "/Users/pmaroney/anaconda3/lib/python3.5/site-packages/stix_shifter-1.0.0-py3.5.egg/stix_shifter/src/modules/splunk/stix_to_splunk.py", line 35
    raise NotImplementedError(f"Module {data_mapper_module_name} not implemented")
                                                                                ^
SyntaxError: invalid syntax

  File "/Users/pmaroney/anaconda3/lib/python3.5/site-packages/stix_shifter-1.0.0-py3.5.egg/tests/car_json_to_stix/test_class.py", line 100
    assert(binary_ref in objects), f"binary_ref with key {binary_ref} not found"
                                                                               ^
SyntaxError: invalid syntax

  File "/Users/pmaroney/anaconda3/lib/python3.5/site-packages/stix_shifter-1.0.0-py3.5.egg/tests/qradar_json_to_stix/test_class.py", line 88
    assert(ip_ref in objects), f"dst_ref with key {nt_object['dst_ref']} not found"
                                                                                  ^
SyntaxError: invalid syntax

stix-shifter 1.0.0 is already the active version in easy-install.pth

Installed /Users/pmaroney/anaconda3/lib/python3.5/site-packages/stix_shifter-1.0.0-py3.5.egg
Processing dependencies for stix-shifter==1.0.0
Searching for stix2-validator==1.0.1
Reading https://pypi.python.org/simple/stix2-validator/
No local packages or working download links found for stix2-validator==1.0.1
error: Could not find suitable distribution for Requirement.parse('stix2-validator==1.0.1')

Invalid STIX bundles are able to be used with the Bundle connector, when it should reject them

Currently the bundle connector allows an invalid bundle to be used, relying on an error to happen later on during filtering. It should validate the bundle in advance and reject any invalid bundle.

Need documentation on how to develop / create a new module

Currently the documentation link from the main README.md, points to how to use a module, but does not actually go into any details as to how to develop a module.

We need some basic README that describes what is necessary to make a module, and also preferably a "stub" or "dummy" module that is no-op that one can use as an example.

Prevent references from creating useless STIX cybox objects

Using QRadar results as an example, if a filename came in as either null or an empty string, stix-shifter would not create that property. If there was no other properties for the file object (such as a file hash) then the cybox object wouldn't get created.

However, if the results also returned a filepath, that value would get written to STIX as directory.path and a file.parent_directory_ref would also get written pointing to the directory object. So now a file object has been created but with only a type ('file') and a parent_directory_ref property.

{{
{ '0': {'type': 'directory', 'path': 'some/path'}
, '1': {'type': 'file', 'parent_directory_ref': '7'} }}}

This is valid STIX, but the file object doesn't give any useful information and probably should not show up in the results.

Network traffic port numbers are strings when they should be integers

src_port and dst_port are getting stored in the network-traffic object as integers. The STIX specification requires them to be integers.

Support eBPF observability / container-optimized operating systems using Sysdig Secure captures

Container-optimized operating systems forensics need eBPF observability.
(https://sysdig.com/blog/introducing-container-observability-with-ebpf-and-sysdig/)

Sysdig Secure can automatically capture all system calls made by any process. These captures can easily be converted to JSON e.g.
sudo sysdig -r sysdig.scap -j > sysdig.json

It'd be great if we could provide an end-to-end example of using STIX-Shifter to translate Sysdig Secure JSON captures into STIX format.

Relevant links:
https://github.com/draios/sysdig/wiki/Sysdig-Examples
https://sysdig.com/blog/fishing-for-hackers/
https://sysdig.com/blog/fishing-for-hackers-part-2/

.travis.yml: The 'sudo' tag is now deprecated in Travis CI

Travis are now recommending removing the sudo tag.

"If you currently specify sudo: false in your .travis.yml, we recommend removing that configuration"

STIX-Shifter errors out on most of the STIX2.0 specification examples

I'm getting errors on the majority of the pattern examples from the STIX 2.0 specification document ([STIX™ Version 2.0. Part 5: STIX Patterning, Committee Specification 01, 19 July 2017 )

I pulled the examples out of Section 6, and ran them. My code and results attached. I would expect that most, if not all of the examples from the spec would work. Maybe I'm doing something wrong?

thanks,
Chris
stix-shifter-examples.pdf

Carbon Black Response connector is improperly creating network-traffic objects.

Describe the bug
It looks like it's creating a network-traffic object for every process using Cb's interface_ip as the source and comms_ip as the destination. The end result is useless noise, or worse, misleading. It looks like every process is opening a network connection.

I have an example Cb API response where a process has a netconn_count of 0 but the stix-shifted bundle still shows an opened_connection_ref. That network-traffic object goes from interface_ip to comms_ip.

To Reproduce
Steps to reproduce the behavior:

1. use the connector/adapter thingy

Expected behavior
network-traffic objects should only be added if the process itself created them (i.e. netconn_count > 1). I don't know the Cb API but interface_ip and comms_ip are not the right data values.

Screenshots
Can't share the data at this time.

Desktop (please complete the following information):

• OS: [e.g. iOS]
• Browser [e.g. chrome, safari]
• Version [e.g. 22]

Smartphone (please complete the following information):

• Device: [e.g. iPhone6]
• OS: [e.g. iOS8.1]
• Browser [e.g. stock browser, safari]
• Version [e.g. 22]

Additional context
Add any other context about the problem here.

Data type of `resolves_to_refs` property for IP addresses and Domain name needs to be a list

Describe the bug
Currently the data type of resolves_to_refs property for IP addresses are setting as an integer or string value. But according to STIX 2.0 specification, it needs to be a list.

To Reproduce
Steps to reproduce the behavior:

1. Go to Open a terminal to the stix-shifter root directory to use the CLI tools
2. Translate a datasource result which contains both IPv4/IPv6 address and mac-address or Domain name
3. Note the resulting object:

"objects": {
                "0": {
                    "resolves_to_refs": "1",
                    "type": "ipv4-addr",
                    "value": "192.168.0.1"
                },
               "1": {
                    "type": "mac-addr",
                    "value": "AA:BB:CC:DD:11:22"
                }
}

Expected behavior
The resulting object should look like-

"objects": {
                "0": {
                    "resolves_to_refs":  ["1"],
                    "type": "ipv4-addr",
                    "value": "192.168.0.1"
                },
               "1": {
                    "type": "mac-addr",
                    "value": "AA:BB:CC:DD:11:22"
                }
}

Webinar link results in "This site can't be reached" error

Describe the bug
Clicking on webinar link not only takes a long time to load (which I realize will be different as per internet access)... but the result is a "This site can’t be reached".

To Reproduce
Steps to reproduce the behavior:

1. Click on https://ibm.biz/BdzTyA with a browser (I used Chrome)
2. Wait up to 4 minutes
3. See error "This site can’t be reached".

Expected behavior
A webinar video.

Screenshots

Desktop (please complete the following information):

• OS: MacOS Catalina
• Browser: Chrome
• Version: Version 83.0.4103.61 (Official Build) (64-bit)

Smartphone (please complete the following information):
n/a

Additional context
None

Process Definition: Maintainer Role

Process Improvement.
Define Maintainers Roles and QoS.

Conversion to/from Sigma

Very promising project, we've been struggling a fair bit with how to make stix 2 patterns actionable, so this project definitely is a step in the right direction. Considering how most of the CERT community has adopted Sigma, would including a converter to/from Sigma be an option for future enhancements?

Thank you for all the work that's gone in to this!

`ipv4-addr:value` with LIKE operator for QRadar is invalid query

Describe the bug
ipv4-addr:value with LIKE operator for QRadar is invalid query, for example this pattern [ipv4-addr:value LIKE '127.%'] will translate to ...FROM events WHERE (sourceip LIKE '%127.%%' OR destinationip LIKE '%127.%%' OR identityip LIKE '%127.%%') limit 1000 last 5 minutes which is invalid AQL query.

To Reproduce
Steps to reproduce the behavior:

1. Translate this STIX pattern to QRadar AQL: [ipv4-addr:value LIKE '127.%']
2. Paste the AQL into QRadar advanced search in Log Activity and see the syntax error

Expected behavior
Retrieve the events where IPv4 starts with 127.*
as this AQL will do:
select * from events where str(sourceip) LIKE '127.%'

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: OSX
• Browser Chrome

Add scenario from connector to main introduction

Is your feature request related to a problem? Please describe.
I'm always frustrated when newbies can't figure out what is going on in an project and the overview jumps from 50,000 feet to below sea level on the intro page.
In particular I think https://github.com/opencybersecurityalliance/stix-shifter/blob/master/OVERVIEW.md#why-would-i-want-to-use-this needs some "for example".

Describe the solution you'd like
The https://github.com/opencybersecurityalliance/stix-shifter/blob/master/adapter-guide/develop-stix-adapter.md page has a good example and the first two sections (https://github.com/opencybersecurityalliance/stix-shifter/blob/master/adapter-guide/develop-stix-adapter.md#participants and https://github.com/opencybersecurityalliance/stix-shifter/blob/master/adapter-guide/develop-stix-adapter.md#problem-to-solve) could get added to https://github.com/opencybersecurityalliance/stix-shifter/blob/master/OVERVIEW.md#why-would-i-want-to-use-this. Things clicked much better for me once I read these two sections.

As a separate aside - you might want to make 'documentation' a peer to bug/feature in issues. I agree documentation is part of the product (so everything, including documentation, is a bug or a feature) but still might be better separate (eg was this issue a bug -newbie had trouble understanding - or a feature - improve documentation).

Domain name from QRadar not getting written to STIX object

Describe the bug
If Qradar passes in a domain name without the http:// or https://, a domain-name observed-data object will be written with an empty value.

To Reproduce
Steps to reproduce the behavior:

1. Open a terminal to the stix-shifter root directory to use the CLI tools
2. Run python main.py translate qradar results '{"type": "identity","id": "identity--f431f809-377b-45e0-aa1c-6a4751cae5ff","name": "QRadar","identity_class": "events"}' '[ { "domainname": "example.com" } ]'
3. Note the resulting object:

"objects": {
    "0": {
        "type": "domain-name",
         "value": ""
   }
}

Expected behavior
The value should contain "example.com"

Additional context
If the QRadar results were '[ { "domainname": "http://example.com" } ]' then the STIX object's value would be "http://example.com" The issue is most likely in the ToDomainName transformer.

stix-shifter does not install correctly via pip

Describe the bug
stix-shifter does not install correctly via the recommended approach with pip. Using it results in ModuleNotFoundError: No module named 'stix_shifter_utils'.

To Reproduce
Steps to reproduce the behavior:

1. Create new virtualenv: $ mkvirtualenv shifter --python=/usr/local/bin/python3.7
2. Install stix-shifter with pip: $ pip install stix-shifter
3. Execute example: $ stix-shifter translate qradar query {} "[ipv4-addr:value = '127.0.0.1']" {}
4. See error
5. Use as library: >>> from stix_shifter.stix_translation import stix_translation
6. See error

Expected behavior
Examples from the documentation should work without error.

Error Message

$ stix-shifter translate qradar query {} "[ipv4-addr:value = '127.0.0.1']" {}
Traceback (most recent call last):
  File "***/.virtualenvs/shifter/bin/stix-shifter", line 5, in <module>
    from stix_shifter.scripts.stix_shifter import main
  File "***/.virtualenvs/shifter/lib/python3.7/site-packages/stix_shifter/scripts/stix_shifter.py", line 3, in <module>
    from stix_shifter.stix_translation import stix_translation
  File "***/.virtualenvs/shifter/lib/python3.7/site-packages/stix_shifter/stix_translation/stix_translation.py", line 2, in <module>
    from stix_shifter_utils.stix_translation.src.patterns.parser import generate_query
ModuleNotFoundError: No module named 'stix_shifter_utils'

Empty strings can get written to STIX objects

Describe the bug
I noticed when doing some CLI tests on a new adapter that if the data source returns an empty string in one of the values, it's possible that it could get written to the STIX attribute.

"3": {
     "type": "user-account",
     "user_id": ""
  }

There should be a check during the json-to-stix flow to prevent this.

To Reproduce
Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: [e.g. iOS]
• Browser [e.g. chrome, safari]
• Version [e.g. 22]

Smartphone (please complete the following information):

• Device: [e.g. iPhone6]
• OS: [e.g. iOS8.1]
• Browser [e.g. stock browser, safari]
• Version [e.g. 22]

Additional context
Add any other context about the problem here.

Add qradar domain to the mapping

Is your feature request related to a problem? Please describe.
It is a very common use case to filter to domain name in qradar. Also, right now it is pretty annoying to see only the domainid in the resultset, which is not human readable.

Describe the solution you'd like
Map the DOMAINNAME(domainid) aql property to an existing attribute (like domainname) or a custom one.

Describe alternatives you've considered
I don't see any viable alternatives, this change is pretty straightforward (one more attribute definition).

Additional context

[QRadar] Escaping `(` and `)` and `"` in value changes AQL meaning

Describe the bug

given the following STIX pattern: [(process:command_line LIKE 'my command line is (here)')], when translating to AQL, the value is escaped and now is LIKE '%my command line is \(here\)%' which is different from 'my command line is (here)' in QRadar.

the AQL FROM events WHERE ProcessCommandLine LIKE '%my command line is \(here\)%' limit 1000 last 5 minutes is different than FROM events WHERE ProcessCommandLine LIKE '%my command line is (here)%' limit 1000 last 5 minutes and will return zero results.

Using value as LIKE '%my command line is (here)%' does seems to be correct with QRadar, I think the escaping inside the value is not needed and changes the AQL.

To Reproduce
Steps to reproduce the behavior:

1. Generate the following event
   <142>Jul 13 14:37:00 WIN-VM-BAR AgentDevice=WindowsLog AgentLogFile=Microsoft-Windows-Sysmon/Operational PluginVersion=7.2.9.105 Source=Microsoft-Windows-Sysmon Computer=MSEDGEWIN10 OriginatingComputer=MSEDGEWIN10 User=SYSTEM Domain=NT AUTHORITY EventID=1 EventIDCode=1 EventType=1 EventCategory=1 RecordNumber=6257 TimeGenerated=1593692483 TimeWritten=1593692483 Level=Informational Keywords=0x8000000000000000 Task=SysmonTask-SYSMON_CREATE_PROCESS Opcode=Info Message=IntegrityLevel: System CommandLine: my command line is (here) CurrentDirectory: x Image: example\powershell.exe FileVersion: x ParentImage: C:\Windows\System32\svchost.exe ParentCommandLine: x

2. Execute using STIX Shifter: [(process:command_line LIKE 'my command line is (here)')] you'll get zero results.

3. After removing from query_constructor.py line 88 to be return '{}'.format(value.replace('\"', '\\"')) and not return '{}'.format(value.replace('\"', '\\"').replace('(', '\\(').replace(')', '\\)')), it will work fine

Desktop (please complete the following information):

• QRadar 7.3.3 Community Edition

Missing escaping for ECS str in equality values

Missing escaping for ECS str in equality values for backslash
pattern

[(windows-registry-key:key = 'HKLM\\SYSTEM\\CurrentControlSet\\Control\\MiniNt' AND x-event:code = 1)] 

should be queried as

{
    "queries": [
        "event.code : \"1\" AND registry.key : \"HKLM\\\\SYSTEM\\\\CurrentControlSet\\\\Control\\\\MiniNt\" AND (@timestamp:[\"2020-08-25T11:55:00.000Z\" TO \"2020-08-26T06:05:00.000Z\"])"
    ]
}

and not as

{
    "queries": [
        "event.code : \"1\" AND registry.key : \"HKLM\\SYSTEM\\CurrentControlSet\\Control\\MiniNt\" AND (@timestamp:[\"2020-08-25T11:55:00.000Z\" TO \"2020-08-26T06:05:00.000Z\"])"
    ]
}

The if-cases will go to the _format_equality case, the _escape_value does not get called. it makes incorrect query.

for example key registry.key and value System\CurrentControlSet\Services\W32Time\Config\LastKnownGoodTime

pattern [(process:command_line = 'foo\\bar')] translated to "process.command_line : \"foo\\bar\"", using this query value format would return zero result -

but if you escape the value backslashes with double backslashes, it will retrieve results -

so pattern should be translated as [(process:command_line = 'foo\\bar')] translated to "process.command_line : \"foo\\\\bar\""

Document the Community Process for STiX Shifter

https://github.com/opencybersecurityalliance/stix-shifter/wiki/Process-Proposal

Flow support for QRadar

Is your feature request related to a problem? Please describe.
The QRadar translation adapter only supports events right now but I would expect it to also support flows.

Describe the solution you'd like
STIX-shifter is able to translate patterns into AQL flow queries and translate the results back into STIX objects.

Additional context
Very little work should be needed, the API calls used in the transmission methods will be the same. The mappings will change slightly from what's currently used for events.

Using backslashes ("\") in STIX values with QRadar backend

I am trying to execute the following query for cp4s for QRadar
select * FROM events WHERE Image LIKE '%\powershell.exe%' limit 1000 START 1593407880000 STOP 1593451080000

I added custom field at the STIX Shifter and connected to local QRadar, if running on QRadar I do get results, but cp4s won’t let me do the query as [(process:image_ref.name LIKE '%\powershell.exe')] START t'2020-06-29T05:18:00.000Z' STOP t'2020-06-29T17:18:00.000Z', so it pass validation when adding extra \ to be [(process:image_ref.name LIKE '%\\powershell.exe')] START t'2020-06-29T05:18:00.000Z' STOP t'2020-06-29T17:18:00.000Z' but then, I don’t get any results since it query for select * FROM events WHERE Image LIKE '%\\\\powershell.exe%' limit 1000 START 1593407880000 STOP 1593451080000 which does not match the single quote at the image name %\powershell.exe%

It worked fine only after adding un-escaping for the query before sending it to QRadar backend
query = query.replace('\\\\', '\\')