Converting YAML to Json about dettect

[marcusbakker]

@hawki999 could you please have a look at this comment from @rubinatorz. As your issues seem to be the same.

[hawki999]

Afternoon Thanks Marcus

Unfortunately i still get the same error after the proposed workaround

[!] Data source: 'Logon Session Creation' is MISSING from the YAML file
[!] Data source: 'Scheduled Job Modification' is MISSING from the YAML file
[!] Data source: 'Instance Stop' is MISSING from the YAML file
[!] Data source: 'Cloud Service Modification' is MISSING from the YAML file
[!] Data source: 'File Modification' is MISSING from the YAML file
[!] Data source: 'Logon Session Metadata' is MISSING from the YAML file
[!] Data source: 'Firewall Metadata' is MISSING from the YAML file
[!] Data source: 'Volume Modification' is MISSING from the YAML file
[!] Data source: 'Web Credential Usage' is MISSING from the YAML file
[!] Data source: 'Pod Enumeration' is MISSING from the YAML file
[!] Data source: 'File Deletion' is MISSING from the YAML file
[!] Data source: 'Container Start' is MISSING from the YAML file
[!] Data source: 'Cloud Service Metadata' is MISSING from the YAML file
[!] Data source: 'Container Creation' is MISSING from the YAML file
[!] Data source: 'User Account Modification' is MISSING from the YAML file
[!] Data source: 'Volume Metadata' is MISSING from the YAML file
[!] Data source: 'Active Directory Object Access' is MISSING from the YAML file
[!] Data source: 'Host Status' is MISSING from the YAML file
[!] Data source: 'Command Execution' is MISSING from the YAML file
[!] Data source: 'Pod Creation' is MISSING from the YAML file
[!] Data source: 'OS API Execution' is MISSING from the YAML file
[!] Data source: 'Image Deletion' is MISSING from the YAML file
[!] Data source: 'Driver Metadata' is MISSING from the YAML file
[!] Data source: 'Instance Enumeration' is MISSING from the YAML file
[!] Data source: 'Pod Modification' is MISSING from the YAML file
[!] Data source: 'Network Traffic Flow' is MISSING from the YAML file
[!] Data source: 'Active Directory Object Modification' is MISSING from the YAML file
[!] Data source: 'Process Metadata' is MISSING from the YAML file
[!] Data source: 'Firmware Modification' is MISSING from the YAML file
[!] Data source: 'User Account Metadata' is MISSING from the YAML file
[!] Data source: 'Snapshot Deletion' is MISSING from the YAML file
[!] Data source: 'Scheduled Job Creation' is MISSING from the YAML file
[!] Data source: 'Windows Registry Key Creation' is MISSING from the YAML file
[!] Data source: 'User Account Deletion' is MISSING from the YAML file
[!] Data source: 'File Creation' is MISSING from the YAML file
[!] Data source: 'Process Termination' is MISSING from the YAML file
[!] Data source: 'Instance Modification' is MISSING from the YAML file
[!] Data source: 'Service Modification' is MISSING from the YAML file
[!] Data source: 'Kernel Module Load' is MISSING from the YAML file
[!] Data source: 'User Account Authentication' is MISSING from the YAML file
[!] Data source: 'Container Enumeration' is MISSING from the YAML file
[!] Data source: 'File Content' is MISSING from the YAML file
[!] Data source: 'Image Modification' is MISSING from the YAML file
[!] Data source: 'Network Traffic Content' is MISSING from the YAML file
[!] Data source: 'Instance Metadata' is MISSING from the YAML file
[!] Data source: 'Process Creation' is MISSING from the YAML file
[!] Data source: 'Cloud Storage Access' is MISSING from the YAML file
[!] Data source: 'Module Load' is MISSING from the YAML file
[!] Data source: 'Web Credential Creation' is MISSING from the YAML file
[!] Data source: 'Pod Metadata' is MISSING from the YAML file
[!] Data source: 'File Metadata' is MISSING from the YAML file
[!] Data source: 'Instance Start' is MISSING from the YAML file
[!] Data source: 'Cluster Metadata' is MISSING from the YAML file
[!] Data source: 'Cloud Storage Metadata' is MISSING from the YAML file
[!] Data source: 'Windows Registry Key Deletion' is MISSING from the YAML file
[!] Data source: 'Windows Registry Key Modification' is MISSING from the YAML file
[!] Data source: 'Instance Deletion' is MISSING from the YAML file
[!] Data source: 'Active Directory Object Deletion' is MISSING from the YAML file
[!] Data source: 'Process Access' is MISSING from the YAML file
[!] Data source: 'Active Directory Credential Request' is MISSING from the YAML file
[!] Data source: 'Script Execution' is MISSING from the YAML file
[!] Data source: 'Snapshot Enumeration' is MISSING from the YAML file
[!] Data source: 'Cloud Storage Creation' is MISSING from the YAML file
[!] Data source: 'Windows Registry Key Access' is MISSING from the YAML file
[!] Data source: 'Service Metadata' is MISSING from the YAML file
[!] Data source: 'Service Creation' is MISSING from the YAML file
[!] Data source: 'Cloud Storage Modification' is MISSING from the YAML file
[!] Data source: 'Volume Enumeration' is MISSING from the YAML file
[!] Data source: 'Snapshot Metadata' is MISSING from the YAML file
[!] Data source: 'Volume Deletion' is MISSING from the YAML file
[!] Data source: 'WMI Creation' is MISSING from the YAML file
[!] Data source: 'Firewall Rule Modification' is MISSING from the YAML file
[!] Data source: 'Application Log Content' is MISSING from the YAML file
[!] Data source: 'Snapshot Creation' is MISSING from the YAML file
[!] Data source: 'Driver Load' is MISSING from the YAML file
[!] Data source: 'Active Directory Object Creation' is MISSING from the YAML file
[!] Data source: 'Network Connection Creation' is MISSING from the YAML file
[!] Data source: 'Drive Modification' is MISSING from the YAML file
[!] Data source: 'Volume Creation' is MISSING from the YAML file
[!] Data source: 'Firewall Enumeration' is MISSING from the YAML file
[!] Data source: 'Group Modification' is MISSING from the YAML file
[!] Data source: 'Network Share Access' is MISSING from the YAML file
[!] Data source: 'Group Metadata' is MISSING from the YAML file
[!] Data source: 'Container Metadata' is MISSING from the YAML file
[!] Data source: 'User Account Creation' is MISSING from the YAML file
[!] Data source: 'Snapshot Modification' is MISSING from the YAML file
[!] Data source: 'Drive Access' is MISSING from the YAML file
[!] Data source: 'Drive Creation' is MISSING from the YAML file
[!] Data source: 'File Access' is MISSING from the YAML file
[!] Data source: 'Scheduled Job Metadata' is MISSING from the YAML file
[!] Data source: 'Cloud Storage Enumeration' is MISSING from the YAML file
[!] Data source: 'Image Metadata' is MISSING from the YAML file
[!] Data source: 'Image Creation' is MISSING from the YAML file
[!] Data source: 'Cloud Service Disable' is MISSING from the YAML file
[!] Data source: 'Cloud Service Enumeration' is MISSING from the YAML file
[!] Data source: 'Instance Creation' is MISSING from the YAML file
[!] Data source: 'Group Enumeration' is MISSING from the YAML file
Traceback (most recent call last):
File "dettect.py", line 365, in
_menu(_init_menu())
File "dettect.py", line 254, in _menu
generate_data_sources_layer(file_ds, args.output_filename, args.layer_name, args.platform)
File "/home/craig/DeTTECT/data_source_mapping.py", line 24, in generate_data_sources_layer
my_techniques = _map_and_colorize_techniques(my_data_sources, platform, exceptions)
File "/home/craig/DeTTECT/data_source_mapping.py", line 261, in _map_and_colorize_techniques
total_ds_count = _count_applicable_data_sources(t, applicable_data_sources)
File "/home/craig/DeTTECT/data_source_mapping.py", line 240, in _count_applicable_data_sources
ds = ds.split(':')[1][1:]
IndexError: list index out of range

[rubinatorz]

hi @hawki999

How does your data source YAML file look like and what specifiek dettect.py command line are you using?

[hawki999]

python3 dettect.py ds -fd /mnt/c/Users/craig/Downloads/data-sources-new.yaml -l --health

version: 1
file_type: data-source-administration
name: example
platform:

• all
  data_sources:
• data_source_name: Firewall Disable
  date_registered: null
  date_connected: null
  products: []
  available_for_data_analytics: true
  comment: ''
  data_quality:
  device_completeness: 1
  data_field_completeness: 1
  timeliness: 1
  consistency: 1
  retention: 1
• data_source_name: Cloud Storage Deletion
  date_registered: null
  date_connected: null
  products: []
  available_for_data_analytics: true
  comment: ''
  data_quality:
  device_completeness: 1
  data_field_completeness: 1
  timeliness: 1
  consistency: 1
  retention: 1

[rubinatorz]

hi @hawki999

Thanks, I tried the exact same command with the exact same YAML file and get this:

python3 dettect.py ds -fd ds-error.yaml -l --health
[!] Data source: 'Cloud Storage Modification' is MISSING from the YAML file
...
[!] Data source: 'Network Traffic Content' is MISSING from the YAML file
File written:   output/data_sources_example.json

I removed a bunch of lines in this output to keep it short. So all working here...

I can't get my finger on it...

Can you run the "pip3 list" command and send me the output?

[hawki999]

Hi Ruben

please find the list below - best regards

Package Version

---

antlr4-python3-runtime 4.8
attackcti 0.3.4.3
attrs 19.3.0
Automat 0.8.0
blinker 1.4
certifi 2019.11.28
chardet 3.0.4
Click 7.0
cloud-init 21.2
colorama 0.4.3
command-not-found 0.3
configobj 5.0.6
constantly 15.1.0
cryptography 2.8
dbus-python 1.2.16
distro 1.4.0
distro-info 0.23ubuntu1
entrypoints 0.3
eql 0.9.9
httplib2 0.14.0
hyperlink 19.0.0
idna 2.8
importlib-metadata 1.5.0
incremental 16.10.1
Jinja2 2.10.1
jsonpatch 1.22
jsonpointer 2.0
jsonschema 3.2.0
keyring 18.0.1
language-selector 0.1
lark-parser 0.11.3
launchpadlib 1.10.13
lazr.restfulclient 0.14.2
lazr.uri 1.0.3
MarkupSafe 1.1.0
more-itertools 4.2.0
netifaces 0.10.4
numpy 1.20.3
oauthlib 3.1.0
pandas 1.2.4
pexpect 4.6.0
pip 20.0.2
plotly 5.0.0
pyasn1 0.4.2
pyasn1-modules 0.2.1
PyGObject 3.36.0
PyHamcrest 1.9.0
PyJWT 1.7.1
pymacaroons 0.13.0
PyNaCl 1.3.0
pyOpenSSL 19.0.0
pyrsistent 0.15.5
pyserial 3.4
python-apt 2.0.0+ubuntu0.20.4.5
python-dateutil 2.8.1
python-debian 0.1.36ubuntu1
pytz 2021.1
PyYAML 5.3.1
requests 2.22.0
requests-unixsocket 0.2.0
ruamel.yaml 0.17.9
ruamel.yaml.clib 0.2.2
SecretStorage 2.3.1
service-identity 18.1.0
setuptools 45.2.0
simplejson 3.16.0
six 1.14.0
sos 4.1
ssh-import-id 5.10
stix2 2.1.0
stix2-patterns 1.3.2
systemd-python 234
taxii2-client 2.3.0
tenacity 7.0.0
Twisted 18.9.0
ubuntu-advantage-tools 27.0
ufw 0.36
unattended-upgrades 0.1
urllib3 1.25.8
wadllib 1.3.3
wheel 0.34.2
XlsxWriter 1.4.3
zipp 1.0.0
zope.interface 4.7.1

[rubinatorz]

hi @hawki999

I noticed you are using attackcti==0.3.4.3, while in requirements.txt we set it to 0.3.3. Please try to install the python library versions as mentioned in the requirements.txt and try again. I tried with the 0.3.4.3 version and it gives me the same error, so pretty sure when you use the 0.3.3 version it should work.

We will look into the error for the 0.3.4.3 version for future releases.

[hawki999]

Ruben

me bad ,

top man works perfect thanks for all your help have a good weekend

[rubinatorz]

Hi @hawki999

Great! My pleasure.

Enjoy the weekend!