APT19 Group part of the data set about dettect

[rubinatorz]

Hi @DFIRmadness

I tried to reproduce the error but I couldn't. Can you try to clear the cache directory and try it again?

If that doesn't work, please let us know the DeTT&CT version and your python packages versions (pip list).

Regards,
Ruben

[DFIRmadness]

Hey Ruben,

Thanks for the help.

Here are the results:

root@f41cd8582e67:/opt/DeTTECT# rm cache/*
root@f41cd8582e67:/opt/DeTTECT# python dettect.py g -g G0073 -p all -of test.json
[taxii2client.common] [WARNING ] [2021-10-18 20:26:12,696] TAXII Server Response did not include 'Content-Range' header - results could be incomplete
[taxii2client.common] [WARNING ] [2021-10-18 20:26:12,915] TAXII Server Response did not include 'Content-Range' header - results could be incomplete
[taxii2client.common] [WARNING ] [2021-10-18 20:26:13,051] TAXII Server Response did not include 'Content-Range' header - results could be incomplete
[taxii2client.common] [WARNING ] [2021-10-18 20:26:14,811] TAXII Server Response did not include 'Content-Range' header - results could be incomplete
[taxii2client.common] [WARNING ] [2021-10-18 20:26:16,777] TAXII Server Response did not include 'Content-Range' header - results could be incomplete
[taxii2client.common] [WARNING ] [2021-10-18 20:26:17,295] TAXII Server Response did not include 'Content-Range' header - results could be incomplete
[taxii2client.common] [WARNING ] [2021-10-18 20:26:19,528] TAXII Server Response did not include 'Content-Range' header - results could be incomplete
[taxii2client.common] [WARNING ] [2021-10-18 20:26:20,240] TAXII Server Response did not include 'Content-Range' header - results could be incomplete
[taxii2client.common] [WARNING ] [2021-10-18 20:26:20,579] TAXII Server Response did not include 'Content-Range' header - results could be incomplete
[!] Group not part of the data set: g0073
root@f41cd8582e67:/opt/DeTTECT# python dettect.py --version
dettect.py 1.4.3

I downloaded this container on Friday, October 15th.

[rubinatorz]

hi @DFIRmadness

Thanks for your input. I'm now able reproduce the error on our latest docker image. It was a struggle to find out what the root cause was. I found out that the latest stix2 package (version 3.x) has some issues with the version of the attackcti library we are using. Anyways, the solution is to use a 2.x version of stix2. I updated the requirements.txt file and we built a new docker container. So please pull the latest docker container and that one should work.

cheers,
Ruben

[DFIRmadness]

Ruben,

Thanks! I am sure that was not easy to trace down. I downloaded the latest docker and confirmed it is now working as expected! Thanks!

[DFIRmadness]

All,

I was doing some further testing. I was trying to get overlays etc.

I removed all docker images. Re-installed on (Evening east coast time of the 20th). The following same bug occurred.

root@a608b6b9230c:/opt/DeTTECT# ls -l
total 408
-rw-rw-rw- 1 root root   597 Oct 20 11:28 Dockerfile
-rw-rw-rw- 1 root root 35149 Oct 20 11:28 LICENSE
-rw-rw-rw- 1 root root  4124 Oct 20 11:28 README.md
drwxr-xr-x 2 root root  4096 Oct 20 23:17 __pycache__
drwxr-xr-x 2 root root  4096 Oct 20 23:34 cache
-rw-rw-rw- 1 root root 18580 Oct 20 11:28 constants.py
-rw-rw-rw- 1 root root 29869 Oct 20 11:28 data_source_mapping.py
-rw-rw-rw- 1 root root 25881 Oct 20 11:28 dettect.py
drwxrwxrwx 6 root root  4096 Oct 20 11:28 editor
-rw-rw-rw- 1 root root  2169 Oct 20 11:28 editor.py
-rw-rw-rw- 1 root root 17553 Oct 20 11:28 eql_yaml.py
-rw-rw-rw- 1 root root 52768 Oct 20 11:28 generic.py
-rw-rw-rw- 1 root root 29665 Oct 20 11:28 group_mapping.py
-rw-rw-rw- 1 root root 19014 Oct 20 11:28 health.py
drwxr-xr-x 2 root root  4096 Oct 20 20:23 input
-rw-rw-rw- 1 root root 21754 Oct 20 11:28 interactive_menu.py
drwxrwxrwx 2 root root  4096 Oct 20 11:28 mitre-data
drwxr-xr-x 2 root root  4096 Oct 20 23:26 output
-rw-rw-rw- 1 root root   148 Oct 20 11:28 requirements.txt
drwxrwxrwx 2 root root  4096 Oct 20 11:28 sample-data
-rw-rw-rw- 1 root root 29726 Oct 20 11:28 scoring_table.xlsx
-rw-rw-rw- 1 root root 26977 Oct 20 11:28 technique_mapping.py
drwxrwxrwx 4 root root  4096 Oct 20 11:28 threat-actor-data
-rw-rw-rw- 1 root root 29050 Oct 20 11:28 upgrade.py
root@a608b6b9230c:/opt/DeTTECT# python dettect.py g -g G0025,G0026,G0073,G0045,G0116,G0027,G0016,G0096,G0117,G0083,G0065 -o sample-data/techniques-administration-endpoints.yaml -t visibility -of Test
[!] Group not part of the data set: g0025
root@a608b6b9230c:/opt/DeTTECT# rm cache/*
root@a608b6b9230c:/opt/DeTTECT# python dettect.py g -g G0025,G0026,G0073,G0045,G0116,G0027,G0016,G0096,G0117,G0083,G0065 -o sample-data/techniques-administration-endpoints.yaml -t visibility -of Test
[!] Group not part of the data set: g0025
root@a608b6b9230c:/opt/DeTTECT# python dettect.py g -g G0025 -o sample-data/techniques-administration-endpoints.yaml -t visibility -of Test
[!] Group not part of the data set: g0025
root@a608b6b9230c:/opt/DeTTECT# python dettect.py g -g g0025 -o sample-data/techniques-administration-endpoints.yaml -t visibility -of Test
[!] Group not part of the data set: g0025
root@a608b6b9230c:/opt/DeTTECT# python dettect.py g -g g0025
[!] Group not part of the data set: g0025
root@a608b6b9230c:/opt/DeTTECT# python dettect.py g -g g0026
File written:   output/attack_windows_g0026.json
root@a608b6b9230c:/opt/DeTTECT#

I tried on another computer all together and the bug was not present. I am very confused -any help would be appreciated. Thanks.

[rubinatorz]

hi @DFIRmadness,

Same error, different cause this time :-)

G0025/APT7 is a group that only has "techniques used" in the PRE platform:
https://attack.mitre.org/groups/G0025/

For the group mode of DeTT&CT, the default value for platform is Windows (check dettect.py g -h):

-p {all,PRE,Windows,macOS,Linux,Office 365,Azure AD,Google Workspace,IaaS,SaaS,Network,Containers}, --platform {all,PRE,Windows,macOS,Linux,Office 365,Azure AD,Google Workspace,IaaS,SaaS,Network,Containers}
                        specify the platform (default = Windows). Multiple platforms can be provided with extra '-p/--platform' arguments

And that results in G0025/APT17 not being recognised. So to resolve your issue, add "-p all" to your command and it will use all platforms.

[rubinatorz]

While thinking of this default behaviour and discussing it with Marcus... We are going to change the default platform for groups mode to "all". That makes much more sense.

Anyways, your quickfix is using "-p all" and soon we will make this the default.

[rubinatorz]

While thinking of this default behaviour and discussing it with Marcus... We are going to change the default platform for groups mode to "all". That makes much more sense.

Anyways, your quickfix is using "-p all" and soon we will make this the default.

'all' is default since recent commit ee0decb. Closing the issue, since both issues discussed are solved.