joshuasmeda / mimecast_log_collector

Collect SIEM logs from the Mimecast API using Python. Currently supports multiple endpoints. Use this to be able to easily and quickly forward to events for security logging, such as a SIEM.

📈 GitHub Stats

mimecast_log_collector's Issues

High CPU usage

The log collector seems to be utilising more CPU than it should today.
I think this may be because the ttp and audit logs are continually checking and then checking the hashes (I'm getting many hash already exists messages within milliseconds of each other)
Maybe this should be rate limited to only check every thirty seconds?

Issue with parsing JSON from Mimecast TTP and Audit endpoints

Hello,

I'm using this script to send log events to Fluentd. I noticed Fluentd kicking out a lot of records from Mimecast, so I investigated, and this is what I found -

Events coming into Fluent from this script were not being escaped properly.

I fixed it with this patch:

--- ttp_events_orig.py	2022-07-28 10:23:04.884010316 -0500
+++ ttp_events.py	2022-07-28 10:27:00.860478268 -0500
@@ -45,7 +45,7 @@
 
             # Forward each event individually
             for row in resp_body:
-                row = str(row).replace("'", '"') # Convert audit event to valid JSON
+                row = json.dumps(json.loads(row))
 
                 try:
                     if configuration.syslog_details['syslog_output'] is True:
--- audit_events_orig.py	2022-07-28 10:23:21.752349268 -0500
+++ audit_events.py	2022-07-28 10:27:23.544881198 -0500
@@ -45,7 +45,7 @@
 
             # Forward each event individually
             for row in resp_body:
-                row = str(row).replace("'", '"') # Convert audit event to valid JSON
+                row = json.dumps(json.loads(row))
 
                 try:
                     if configuration.syslog_details['syslog_output'] is True:

ttp and audit events sending the same entry repeatedly

I tried your new version with the ttp and audit events enabled, but the most recent log entry for ttp and audit events appears to be continually sent to the syslog server which is just filling it up with thousands of duplicate messages.

(I'm not sure if it is only the most recent entry or all of them being re-sent, looks like it may be all)

Mimecast increased rate limiting on their API.

Mimecast headers applying stricter API restrictions.

I have a workaround. Replace rate limit exceptions (if r.status_code == 429):
log.warn('Rate limit hit. Sleeping for ' + str(r.headers['X-RateLimit-Reset']))
rate_limit = (int(r.headers['X-RateLimit-Reset'])/1000)%60
time.sleep(rate_limit * 20)
r = requests.post(url='https://' + base_url + uri, data=json.dumps(post_body), headers=headers)

When handling errors from the API. Specify the run_script function to restart the script. For example:

if r.status_code != 200:
log.error('Request to ' + uri + ' with, request id: ' + request_id + ' returned with status code: ' + str(r.status_code) + ', response body: ' + r.text)
log.error('API error. Restarting script via run_script()')
run_script()

Python2 -> Python3

To be implemented:

• Convert to python 3,
• Add support for additional sources TTP URLS, Audit events,
• Implement OOP
• Code testing - * leaving for later

SSL issues with communicating with Mimecast API

Refer to this Stackoverflow post:

https://stackoverflow.com/questions/56858075/pulling-mimecast-logs-with-python/58444619#58444619

Invalid syntax to siem_events.py

Hi

I just migrated this to a different server and the normal events aren't coming through. I'm getting this error:

Process Process-1:
Traceback (most recent call last):
  File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/mimecast_log_collector/run.py", line 26, in <lambda>
    p = multiprocessing.Process(target=lambda: __import__(i))
  File "/opt/mimecast_log_collector/siem_events.py", line 41
    log.warn('Rate limit hit. Sleeping for %s % str(resp_headers('X-RateLimitReset']))
                                                                  ^
SyntaxError: invalid syntax

This appears to be changed since I last updated. That log.warn line and the section it is in appears to have the wrong brackets in the wrong places - but I'm not a Python expert so my attempts to fix it didn't help.
Certainly the resp_headers( should be resp_headers[, but the single quotes don't match either!