jaegeral / firemisp

FireEye Alert json files to MISP Malware information sharing plattform (Alpha)

python script for interacting with misp

Inspired by: https://github.com/spcampbell/FireStic

Installation

update the values to your needs

modify the API Key:

cp config.example.cfg config.cfg
vi config.cfg
#change the values

Running

run the script (in that example 192.168.178 is the IP of MISP):

python firemisp/firemisp.py
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): 192.168.178.71
DEBUG:requests.packages.urllib3.connectionpool:"GET /servers/getVersion HTTP/1.1" 200 20
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): 10.50.12.71
DEBUG:requests.packages.urllib3.connectionpool:"GET /attributes/describeTypes.json HTTP/1.1" 200 4819
INFO:__main__:Starting HTTP server 127.0.0.1 8080

Testing

To test with real data put your *.json files in testing/real (they will be ignored by git)

    python testing/fmtest.py -d testing/real

To test with sample data:#

    python testing/fmtest.py -f testing/alert_details_fireeye_reducted.json

If you do not have a MISP instance, you can get a VM with MISP at https://www.circl.lu/services/misp-training-materials/ Once you have that MISP instance running and reachable from the system you are running FireMisp, get the API key at

$YOURIPOFMISP/users/view/me

And edit the config.cfg according to your needs.

To delete events that have been created for test purposes, uncomment the section in firemisp.py

   #clean the database for test purposes
    '''for i in range (200,1348,1):
        misp.delete_event(i)
    exit()
   '''

And adjust the id values to your need

Issues

There is no

DEBUG:requests.packages.urllib3.connectionpool:"GET /servers/getVersion HTTP/1.1" 200 20
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): 10.50.12.71

After starting FireMisp.py

Instead:

pymisp.api.PyMISPError: Unable to connect to MISP (http://192.168.178.71). Please make sure the API key and the URL are correct (http/https is required): HTTPConnectionPool(host='192.168.178.71', port=80): Max retries exceeded with url: /servers/getVersion (Caused by <class 'socket.error'>: [Errno 110] Connection timed out)

That means the connection to the MISP instance.

Example:

To be done

Roadmap

There are obviously some things to be done in the future:

• improve current mappings (pyFireEyeAlert.py)
• make the mapping more robust (pyFireEyeAlert.py)
• introduce new mappings (FireMisp.py + pyFireEyeAlert.py)
• improve correlation (Feedback welcome)
• test it with high volume of alerts
• Python3 support
• unittests
• make LDAP a config part

firemisp's Issues

pyFireEyeAlert missing

need to upload the pyFireEyeAlert file.

Weird Reset

Im getting a weird reset from the server when doing a test or accessing the port. Its establishing a connection (See tcpdump below) but its resting the connection after the post. I'm guessing its something with the http server. Thanks for the help, really want to get this working.

strace
poll([{fd=3, events=POLLIN}], 1, 500) = 1 ([{fd=3, revents=POLLIN}])
accept4(3, {sa_family=AF_INET, sin_port=htons(48042), sin_addr=inet_addr("172.31.74.93")}, [16], SOCK_CLOEXEC) = 4
getsockopt(4, SOL_SOCKET, SO_TYPE, [1], [4]) = 0
ioctl(4, FIONBIO, [0]) = 0
getpeername(4, {sa_family=AF_INET, sin_port=htons(48042), sin_addr=inet_addr("172.31.74.93")}, [16]) = 0
read(4, "POST / HTTP", 11) = 11
ioctl(4, FIONBIO, [0]) = 0
close(4) = 0
clock_gettime(CLOCK_MONOTONIC, {6477550, 132108430}) = 0

Requirement already satisfied (use --upgrade to upgrade): simplejson>=3.6.5 in /usr/local/lib/python3.5/dist-packages (from -r requirements.txt (line 1))
Requirement already satisfied (use --upgrade to upgrade): pymisp>=2.4.62 in /usr/local/lib/python3.5/dist-packages (from -r requirements.txt (line 2))
Requirement already satisfied (use --upgrade to upgrade): httplib2>=0.8 in /usr/local/lib/python3.5/dist-packages (from -r requirements.txt (line 3))
Requirement already satisfied (use --upgrade to upgrade): configparser in /usr/local/lib/python3.5/dist-packages (from -r requirements.txt (line 4))
Requirement already satisfied (use --upgrade to upgrade): urllib3 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 5))
Requirement already satisfied (use --upgrade to upgrade): six in /usr/lib/python3/dist-packages (from pymisp>=2.4.62->-r requirements.txt (line 2))
Requirement already satisfied (use --upgrade to upgrade): requests in /usr/lib/python3/dist-packages (from pymisp>=2.4.62->-r requirements.txt (line 2))
Requirement already satisfied (use --upgrade to upgrade): python-dateutil in /usr/local/lib/python3.5/dist-packages (from pymisp>=2.4.62->-r requirements.txt (line 2))
Requirement already satisfied (use --upgrade to upgrade): jsonschema in /usr/local/lib/python3.5/dist-packages (from pymisp>=2.4.62->-r requirements.txt (line 2))

tcpdump -Anni lo port 8080
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on lo, link-type EN10MB (Ethernet), capture size 262144 bytes
17:29:31.716449 IP 172.31.74.93.47976 > 172.31.74.93.8080: Flags [S], seq 657993979, win 43690, options [mss 65495,sackOK,TS val 1618276462 ecr 0,nop,wscale 7], length 0
E..<..@[email protected]]..J].h..'80..........'.........
t.n........ 17:29:31.716461 IP 172.31.74.93.8080 > 172.31.74.93.47976: Flags [S.], seq 1921625788, ack 657993980, win 43690, options [mss 65495,sackOK,TS val 1618276462 ecr 1618276462,nop,wscale 7], length 0 E..<..@[email protected]]..J]...hr...'80......'......... t.nt.n.... 17:29:31.716469 IP 172.31.74.93.47976 > 172.31.74.93.8080: Flags [.], ack 1, win 342, options [nop,nop,TS val 1618276462 ecr 1618276462], length 0 E..4..@[email protected]]..J].h..'80.r......V....... t.nt.n 17:29:31.716672 IP 172.31.74.93.47976 > 172.31.74.93.8080: Flags [P.], seq 1:203, ack 1, win 342, options [nop,nop,TS val 1618276463 ecr 1618276462], length 202: HTTP: POST / HTTP/1.1 E.....@[email protected]]..J].h..'80.r......V....... t.o`t.nPOST / HTTP/1.1
Host: 172.31.74.93:8080
Accept: /
User-Agent: python-requests/2.9.1
Accept-Encoding: gzip, deflate
content-type: application/json
Content-Length: 1595
Connection: keep-alive

17:29:31.716682 IP 172.31.74.93.8080 > 172.31.74.93.47976: Flags [.], ack 203, win 350, options [nop,nop,TS val 1618276463 ecr 1618276463], length 0
E..4^.@[email protected]]..J]...hr...'81....^.......
t.ot.o
17:29:31.716715 IP 172.31.74.93.8080 > 172.31.74.93.47976: Flags [R.], seq 1, ack 203, win 350, options [nop,nop,TS val 1618276463 ecr 1618276463], length 0
E..4^.@[email protected]]..J]...hr...'81....^.......
t.ot.o

python3 fmtest.py -f alert-details.json -u 172.31.74.93 -p 8080
"{"msg": "extended", "product": "Web MPS", "version": "7.7.0.123456", "appliance": "fireeye.foo.bar", "appliance-id": "00:11:11:11:11:11","alert": [{ "src": { "ip": "10.1.2.3", "host": "internalclient.intra.net", "vlan": "0", "mac": "00:24:aa:aa:aa:aa" }, "severity": "minr", "alert-url": "https://fireeye.foo.bar/event_stream/events_for_bot?ma_id=12345678", "explanation": { "malware-detected": { "malware": { "profile": "win7x64-sp1", "http-header": "POST http://malicious.com", "name": "Misc.Eicar-Test-File", "md5sum": "44d88612fea8a8f36de82e1278abb02f", "executed-at": "2016-01-19T08:30:21Z", "application": "Windows Explorer", "type": "exe", "original": "driver.exe", "stype": "24" } }, "protocol": "", "analysis": "binary", "cnc-services": { "cnc-service": [ { "protocol": "tcp", "port": "4143", "channel": "\\\\026\\\\003\\\\001", "address": "198.50.234.211" }, { "protocol": "tcp", "port": "9943", "channel": "\\\\026\\\\003\\\\001", "address": "80.96.150.201" }, { "protocol": "tcp", "port": "4493", "channel": "\\\\026\\\\003\\\\001", "address": "1.179.170.7" } ] }, "anomaly": "98816" }, "occurred": "2016-01-19 08:30:21+00", "id": "12345678", "action": "notified", "interface": { "mode": "tap" }, "dst": { "ip": "10.1.2.4", "mac": "00:24:bb:bb:bb:bb" }, "name": "malware-object"}]}"
COMMUNICATION ERROR : ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

describe how to put the firemisp weberver behind an apache

Would like to get ideas / input about use cases for it if encryption is an issue, not a big deal I guess to put it behind an apache as an reverse proxy.

No Module named ldap_query

Hey there. Just did a git pull and am trying to run the script. Getting this error and can't find a corresponding module to go with it.

Traceback (most recent call last):
  File "firemisp.py", line 22, in <module>
    from ldap_query import search_host_and_fqdn, search_userprinciplename, search_for_mail
ImportError: No module named ldap_query

Any help would be appreciated.

Thanks!

problem in integration misp and fire-eye

i have problem in integration misp and fire-eye
i install fire-misp from this link :
https://github.com/deralexxx/FireMISP
i set the config.cfg and the fire-eye notifications .
when i run the script firemisp.py i get the error :
jsonschema.exceptions.validationerror: 'Event is a required property
failed validating 'required' in schema
attach screen shot .


ubuntu 16.04

Need more json files

TO make the import and mapping more accurate and complete, we rely on the community to provide sample files that can be used to test the scripts.