cert-w / certitude

The Seeker of IOC

CERTitude - The seeker of IOC

Description

CERTitude is a Python-based tool which aims at assessing the compromised perimeter during incident response assignments. It allows analysts to perform large scale scans of Windows-based information systems by searching for behavioural patterns described in IOC (Indicator Of Compromise) files.

Notable features:

• Ability to scan hosts in a way that prevents the target workstation from knowing what the investigator is searching for
• Ability to retrieve some pieces of data from the hosts
• Multiple scanner instances (for IOCs and/or hash scans) can be run at the same time for parallel scanning
• Built with security considerations in mind (protected database, secure communications with hosts using IPSec)

Documentation: https://github.com/CERT-W/certitude/wiki

Contributors

Developers

• Aurélien BAUD
• Adrien DEGRANGE
• Thomas LABADIE
• Jean MARSAULT
• Vincent NGUYEN
• Fabien SCHWEBEL
• Antoine VALLEE

External dependencies

• Plyara : https://github.com/8u1a/plyara/

Copyright © Wavestone 2017

certitude's Issues

Translate code and comments from french to english

Hello,

There's a lot of code and comments in english, for instance:

• Comments in config.py
• Code and comments in components/scanner/hashscan_queue.py: "Erreur : lancez le script depuis main.py et non directement", "demarrer_scanner"

This is important, please translate all stuff in english.

Cheers

Improve the readme

Hello,

The current README contains useful information but some info are missing and the order is quite confusing :

• Why the available features come at last ? (Available modules)
• Why the software requirements does not first ?
• Why there is no licence mentioned ?

It would be a good idea to post some screenshots.
Plus, there are some typo (for instance Runnning).

All in all, I think that you should follow the same order of wavecrack's README which is really nice ;)

Cheers.

"seeker" account does not follow the password policy enforced on the interface

Hello,

python main.py init leads to the creation of the seeker account: it is possible to put a weak password while adding users on the interface require to follow a specific password policy.
In order to be coherent a password policy should also be enforced when creating the seeker account.

Cheers.

Clean test code

Clean sensitive files dealing with cryptography of test code; the Python unit testing framework could be used instead

Getting TypeError once i have installed pycryptodome instead of pycrypto. Looking for help on this

Getting TypeError once i have installed pycryptodome instead of pycrypto. Looking for help on this:

TypeError

TypeError: Only byte strings can be passed to C code
Traceback (most recent call last)

File "/files0/home/kumarrd/certitude/lib/python2.7/site-packages/flask/app.py", line 1997, in __call__

return self.wsgi_app(environ, start_response)

File "/files0/home/kumarrd/certitude/lib/python2.7/site-packages/flask/app.py", line 1985, in wsgi_app

response = self.handle_exception(e)

File "/files0/home/kumarrd/certitude/lib/python2.7/site-packages/flask/app.py", line 1540, in handle_exception

reraise(exc_type, exc_value, tb)

File "/files0/home/kumarrd/certitude/lib/python2.7/site-packages/flask/app.py", line 1982, in wsgi_app

response = self.full_dispatch_request()

File "/files0/home/kumarrd/certitude/lib/python2.7/site-packages/flask/app.py", line 1614, in full_dispatch_request

rv = self.handle_user_exception(e)

File "/files0/home/kumarrd/certitude/lib/python2.7/site-packages/flask/app.py", line 1517, in handle_user_exception

reraise(exc_type, exc_value, tb)

File "/files0/home/kumarrd/certitude/lib/python2.7/site-packages/flask/app.py", line 1612, in full_dispatch_request

rv = self.dispatch_request()

File "/files0/home/kumarrd/certitude/lib/python2.7/site-packages/flask/app.py", line 1598, in dispatch_request

return self.view_functions[rule.endpoint](**req.view_args)

File "/files0/home/kumarrd/certitude/components/interface/web.py", line 124, in decorated

return f(*args, **kwargs)

File "/files0/home/kumarrd/certitude/components/interface/web.py", line 459, in wincredAdd

encrypted_account_password = crypto.encrypt(account_password, MASTER_KEY)

File "/files0/home/kumarrd/certitude/helpers/crypto.py", line 73, in encrypt

return base64.b64encode(iv + cipher.encrypt(__pad(m, AES.block_size)))

File "/files0/home/kumarrd/certitude/lib/python2.7/site-packages/Crypto/Cipher/_mode_cbc.py", line 160, in encrypt

expect_byte_string(plaintext)

File "/files0/home/kumarrd/certitude/lib/python2.7/site-packages/Crypto/Util/_raw_api.py", line 200, in expect_byte_string

raise TypeError("Only byte strings can be passed to C code")

TypeError: Only byte strings can be passed to C code

The debugger caught an exception in your WSGI application. You can now look at the traceback which led to the error.

To switch between the interactive traceback and the plaintext one, you can click on the "Traceback" headline. From the text traceback you can also create a paste of it. For code execution mouse-over the frame you want to debug and click on the console icon on the right side.

You can execute arbitrary Python code in the stack frames and there are some extra helpers available for introspection:

dump() shows all variables in the frame
dump(obj) dumps all that's known about the object

Improve library compatibility

Enhance compatibility:

• Fix impacked issues
• Fix lxml issues
• Move to pycryptodome
• etc.

KeyError while creating a configuration profile with no yara rule list

Hello,

I encountered a bug while following these steps:

1. Add an IOC file (interface /config)
2. Create a new scan profile with a name and selecting the created IOC, but with no yara rule list. It is accepted by the server.
3. Go to /config and see the following error:

KeyError: ''
File "/opt/certitude/components/interface/web.py", line 350, in config
> yaradesclist[cp.id] = '||'.join([yararef[str(id)] for id in cp.yara_list.split(',')])

Cheers.

Une DOC des erreurs

Une petite documentation afin d'identifier les codes d'erreur de l'IOCScan serait intéressant.

Rework the usage

Hello,

It would be great to rework the current usage options:

$ certitude -h
usage: certitude [-h] [-c COMPONENT] [-b BATCH_NAME] command

CERTitude, the modular Python scanner, network mapper and IOC Seeker

positional arguments:
  command               command to run ('init', 'run')

optional arguments:
  -h, --help            show this help message and exit
  -c COMPONENT, --component COMPONENT
                        component to run ('interface', 'iocscan')
  -b BATCH_NAME, --batch-name BATCH_NAME
                        [iocscan] Specify batch name

With this current usage options, I naively understand that:

• Without specific option I could type:

  • certitude init: which really exists, for the first launch
  • and certitude run: which does not exist

• With specific options I could type:

  • certitude -c interface run: which exists
  • and certitude -c interface init: which does not exists
  • and certitude -c iocscan init: which does not exist
  • and certitude -c iocscan run: which exists
  • and certitude -b <batchname> init: which does not exist
  • and certitude -b <batchname> run: which does exist

So, the init command is really misleading so you should make it a specific option (better, mandatory for the first launch), not positional argument.
IHMO you should go
from

$ certitude -h
usage: certitude [-h] [-c COMPONENT] [-b BATCH_NAME] command

CERTitude, the modular Python scanner, network mapper and IOC Seeker

positional arguments:
  command               command to run ('init', 'run')

optional arguments:
  -h, --help            show this help message and exit
  -c COMPONENT, --component COMPONENT
                        component to run ('interface', 'iocscan')
  -b BATCH_NAME, --batch-name BATCH_NAME
                        [iocscan] Specify batch name

to

$ certitude -h
usage: certitude [-h] [-c COMPONENT] [-b BATCH_NAME] command

CERTitude, the modular Python scanner, network mapper and IOC Seeker

positional arguments:
  command               command to run ('run')

optional arguments:
  -h, --help            show this help message and exit
  -i, --init            init the database and create user accounts (for the first launch)
  -c COMPONENT, --component COMPONENT
                        component to run ('interface', 'iocscan')
  -b BATCH_NAME, --batch-name BATCH_NAME
                        [iocscan] Specify batch name

Cheers.

IOC selector has hardcoded list of supported items

See available_modules.py to fix

No authentication required to access results and iocs

Hello,
Why no authentication is required to access some data from the web interface such as:

• ioc
• scan result

And if I do understand the code, no authentication is also enforced while accessing the bokeh server.
These data might be sensitive during DFIR, please enforce authentication to access them.

Cheers.

sqlalchemy.exc.ProgrammingError

I have a programming error after login. It says:
ProgrammingError: (sqlite3.ProgrammingError) SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 16912 and this is thread id 7636.
[SQL: SELECT batches.id AS batches_id, batches.name AS batches_name, batches.configuration_profile_id AS batches_configuration_profile_id, batches.windows_credential_id AS batches_windows_credential_id
FROM batches ORDER BY batches.name ASC]
[parameters: [{}]]
(Background on this error at: http://sqlalche.me/e/f405)

PEP8 the code

Hello,

As you might know some code sections are not really pythonic.

Please clean it up a bit, for instance there are 2 subprocess imports on the web interface...

Some good resources:

• PEP8
• Pylint
• Python code formatter

Cheers.

Erreur lors de la recherche d'iocs avec empreintes (md5, sha1,sha-256, etc.)

Bonjour

Ayant crée des iocs au format OpenIOC avec l'outil IOC Editor (de Mandian), j'ai de nombreuses erreurs lorsque j'essaye de rechercher des iocs sur un disque. exemple d'erreur :

iocscanner.3afa14 : INFO Searching for IOC mimikatz (id=2)
iocscanner.3afa14 : WARNING FlatEval files: Md5sum/is is not in evaluation list
iocscanner.3afa14 : WARNING FlatEval files: Sha1sum/is is not in evaluation list
iocscanner.3afa14 : WARNING FlatEval files: Sha256sum/is is not in evaluation list
iocscanner.3afa14 : WARNING FlatEval files: PEInfo/VersionInfoList/VersionInfoItem/InternalName/contains is not in evaluation list
iocscanner.3afa14 : WARNING FlatEval files: Sha256sum/is is not in evaluation list
iocscanner.3afa14 : WARNING FlatEval files: Sha256sum/is is not in evaluation list

Du coup toutes les empreintes md5, sha1, sha256, etc. sont ignorées lors de mes recherches. Par contre la recherche d'iocs par nom de fichier fonctionne bien.

Kesako ?

Bonne année :-)

Use a true KDF for AES

In certitude/helpers/crypto.py, it is said Create an AES key from text (password); Padding is used as a countermeasure to SHA2 rainbow tables
Padding before applying a SHA2 is not a secure KDF, see https://crypto.stackexchange.com/questions/9345/whats-the-most-secure-way-to-derive-a-key-from-a-password-repeatably
Furthermore, the use of an authenticated encryption mode seems to be preferable since the attacker would have a decryption oracle, see http://blog.cryptographyengineering.com/2012/05/how-to-choose-authenticated-encryption.html.

IOC deletion error

Update configuration profiles to remove IOC when deleted

ValueError: invalid literal for int() with base 10

Hello!

I'm running Certitude on a Win7 32bit machine (Python 2.7.10 and all requirements are configured) but when launching the scanner via "python main.py run -c iocscan" I get the error in the printscreen. Please let me know what is the issue in this case.

Regards,
George

Secure SSL parameters

When generating SSL certs in certitude/ssl/gen-cert-for-me.bat, no value in specified for the length of the key, therefore the application rely on insecure default, see https://www.openssl.org/docs/manmaster/apps/genrsa.html for values used and http://www.keylength.com for future-proof recommendations.

Pip package available

Hello,

I managed to package certitude:

• $ git clone https://github.com/maaaaz/certitude.git
• $ cd certitude
• $ pip install .
• $ certitude -h

I had to patch some hardcoded file paths:

• So make sure to use os.path.join(os.path.dirname(__file__), '<file>' as much as possible: perform some tests to see if haven't forgotten other hardcoded file paths
• You can check diffs here

For the next steps:

1. Register an account on Pypi (test and production)
2. Configure your .pypirc file with credentials
3. $ cd certitude
4. $ python setup.py sdist
5. $ pip install twine (for HTTP upload to Pypi)
6. $ twine upload dist/*
7. Profit

For further information: https://packaging.python.org/tutorials/distributing-packages/

Cheers

Map IndicatorItems category aliases

Some IOC writers use alternate versions of standard IndicatorItems names to address elements on the system. For example, you would find Network/DNS instead of DnsEntryItem/RecordData/Host.
The idea is to implement a system of aliases linked to a canonical item name in the OpenIOC standard, upon scan.

Specify the seeker password policy upon init

Hello,

Please specify the password policy upon init as users won't want to read the helpers.py verifyPassword function to know that they have to input at least 12 chars...

So you should move from

$ certitude init
[+] Generating Master Key...
[+] Creating "seeker" account...
Please enter "seeker" password: 

to

$ certitude init
[+] Generating Master Key...
[+] Creating "seeker" account...
Please enter "seeker" password (12 chars minimum among X classes blabla): 

Cheers.

Use a python decorator for user authentication on the web interface

Hello,

On the interface, every function starts with if 'logged_in' in session: else: structure.
Use a flask decorator such as the one in wavecrack (definition here and usage here) as it is more pythonic and gives a less ugly code :)

Cheers

Erreur lors de la création d'une target

Hi,

Impossible de créer une target avec l'IHM. J'obtient l'erreur suivante :

Internal Server Error

The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.

La sortie console :

127.0.0.1 - - [22/Jul/2015 14:38:40] "GET / HTTP/1.1" 302 -
127.0.0.1 - - [22/Jul/2015 14:38:40] "GET /login HTTP/1.1" 200 -
127.0.0.1 - - [22/Jul/2015 14:38:40] "GET /static/reset.css HTTP/1.1" 304 -
127.0.0.1 - - [22/Jul/2015 14:38:40] "GET /static/bootstrap.min.css HTTP/1.1" 30
4 -
127.0.0.1 - - [22/Jul/2015 14:38:40] "GET /static/style.css HTTP/1.1" 304 -
127.0.0.1 - - [22/Jul/2015 14:38:40] "GET /static/vis_files/ga.js HTTP/1.1" 304
-
127.0.0.1 - - [22/Jul/2015 14:38:40] "GET /static/vis_files/modernizr-2.0.6.min.
js HTTP/1.1" 304 -
127.0.0.1 - - [22/Jul/2015 14:40:34] "POST /login HTTP/1.1" 302 -
127.0.0.1 - - [22/Jul/2015 14:40:34] "GET / HTTP/1.1" 302 -
127.0.0.1 - - [22/Jul/2015 14:40:34] "GET /campaignplan HTTP/1.1" 200 -
127.0.0.1 - - [22/Jul/2015 14:40:34] "GET /static/reset.css HTTP/1.1" 304 -
127.0.0.1 - - [22/Jul/2015 14:40:34] "GET /static/bootstrap.min.css HTTP/1.1" 30
4 -
127.0.0.1 - - [22/Jul/2015 14:40:34] "GET /static/style.css HTTP/1.1" 304 -
127.0.0.1 - - [22/Jul/2015 14:40:34] "GET /static/vis_files/ga.js HTTP/1.1" 304
-
127.0.0.1 - - [22/Jul/2015 14:40:34] "GET /static/vis_files/modernizr-2.0.6.min.
js HTTP/1.1" 304 -
127.0.0.1 - - [22/Jul/2015 14:40:37] "GET /targetcreation HTTP/1.1" 200 -
127.0.0.1 - - [22/Jul/2015 14:40:37] "GET /static/reset.css HTTP/1.1" 304 -
127.0.0.1 - - [22/Jul/2015 14:40:37] "GET /static/style.css HTTP/1.1" 304 -
127.0.0.1 - - [22/Jul/2015 14:40:37] "GET /static/bootstrap.min.css HTTP/1.1" 30
4 -
127.0.0.1 - - [22/Jul/2015 14:40:37] "GET /static/vis_files/ga.js HTTP/1.1" 304
-
127.0.0.1 - - [22/Jul/2015 14:40:37] "GET /static/vis_files/modernizr-2.0.6.min.
js HTTP/1.1" 304 -
127.0.0.1 - - [22/Jul/2015 14:41:04] "POST /targetcreation HTTP/1.1" 500 -

Mon environnement :

Windows 7 Pro 32 bits

Stop using pycrypto, prefer pycryptodome

Hello,
As you may know PyCrypto developement stopped few years ago and some vulns will remained unpatched, for instance this one.
Pycryptodome is a good replacement.

Cheers.

Better handle bad IOC inputs

Hello,

I tried to scan with a badly formatted IOC file (a plain file with test inside) and I got non-blocking errors on the import.
I could launch the scan but obviously get it crashed with the following error:

File "/opt/certitude/components/interface/web.py", line 812, in resultscsv
    infos = getInfosFromXML(ioc.xml_content)
  File "/opt/certitude/components/interface/web.py", line 764, in getInfosFromXML
    xml = ET.fromstring(c)
  File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1311, in XML
    parser.feed(text)
  File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1653, in feed
    self._raiseerror(v)
  File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1517, in _raiseerror
    raise err
ParseError: syntax error: line 1, column 0

Cheers.

IOC support

Hello,

Just a quick question about STIX, YARA support.

Did you planned to support another IOC format ?

Thank you

Multiple XSRF vulnerabilities

The web interface is not secure enough, even if not published in the local network; an attacker can perform XSRF attacks on all fields and controllers that do not require the current user password to decipher the MASTER_KEY.
Exemple: <img src="1.2.3.4/config/profile/0/delete" /> when included in any webpage visited by an authenticated user, will delete the first user in the database. A local IP can also be used on older browser to perform the attack without the need to know the IP of certitude host.
This attack can also be used to delete IOCs, modify AD credentials so that certitude propagates them in the IS, but not to destroy evidence, as the web interface is read-only on this point.

impacket module is in requirements.txt but also packaged in repo

Hello,

The impacket module is mentioned in requirements.txt but there's also a pre-packaged version in the repo impacket folder.
Which one is used ? Mixing different versions will eventually break some stuff.

Cheers.

Agent sur les postes de travail/serveurs

Hello,

Ça serait top d'avoir un agent à déployer sur les postes. En effet une interrogation en PSexec sur les postes avec un compte qui est admin local est top pour le forensic mais pas dans le cadre de la sécu opérationnelle. Cette fonctionnalité pourrait permettre de déployer les agents et d'utiliser l'outil comme outil de supervision en plus d'un outil de forensic !

Bye,

Better handle access to the interface when no "seeker" account is created

Hello,

Trying to access the interface when no seeker account is created leads to the following error:

sqlalchemy.exc.OperationalError

OperationalError: (sqlite3.OperationalError) no such table: users [SQL: u'SELECT users.id AS users_id, users.username AS users_username, users.password AS users_password, users.email AS users_email, users.active AS users_active, users.encrypted_master_key AS users_encrypted_master_key, users.b64_kdf_salt AS users_b64_kdf_salt \nFROM users \nWHERE users.username = ?\n LIMIT ? OFFSET ?'] [parameters: (u'seeker', 1, 0)]

Please handle better that case providing a nice error message :)

Cheers.

Missing steps for installation

Hello,
I tried to install Certitude from scratch in a virtualenv named venvcertitude on an Ubuntu 16.10 virtual machine from osboxes.org.
Here are my feedbacks:

• requirements.txt needs pyopenssl module which needs cryptography module, which needs OpenSSL headers. Without them you have that error:

Building wheels for collected packages: cryptography
[...]
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-lMBuS3/python2.7-2.7.12=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/usr/include/python2.7 -c build/temp.linux-x86_64-2.7/_openssl.c -o build/temp.linux-x86_64-2.7/build/temp.linux-x86_64-2.7/_openssl.o
  build/temp.linux-x86_64-2.7/_openssl.c:434:30: fatal error: openssl/opensslv.h: No such file or directory
   #include <openssl/opensslv.h>
                                ^
  compilation terminated.
  error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
 ----------------------------------------
  Running setup.py clean for cryptography
Failed to build cryptography

=> so apt-get install libssl-dev on debian or yum install openssl-devel is a prerequisite

• gen-cert-for-me.bat: why there is not a Linux shell version of that script ?
• python main.py init leads to that error:

Traceback (most recent call last):
  File "main.py", line 81, in <module>
    main()
  File "main.py", line 62, in main
    from helpers import init
  File "/opt/certitude/helpers/init.py", line 36, in <module>
    import crypto
  File "/opt/certitude/helpers/crypto.py", line 26, in <module>
    from Crypto.Cipher import AES
ImportError: No module named Crypto.Cipher

=> so pip install pycrypto is needed. Add pycrypto in your requirements.txt

Cheers.

Bokeh is not configured to listen on LISTEN_ADDRESS or LISTEN_PORT

No description needed.

OSError: [Errno 2] No such file or directory

sh interface.sh
api : INFO Web interface starting
werkzeug : INFO * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
werkzeug : INFO * Restarting with stat
Traceback (most recent call last):
File "main.py", line 81, in
main()
File "main.py", line 65, in main
from components.interface import web
File "/root/certitude/components/interface/web.py", line 137, in
bokeh_process = subprocess.Popen(['bokeh', 'serve', 'crossbokeh.py'], stdout=subprocess.PIPE)
File "/usr/lib/python2.7/subprocess.py", line 711, in init
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1343, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory