misp / docker-misp Goto Github PK

Automated Docker MISP container - Malware Information Sharing Platform and Threat Sharing

License: BSD 3-Clause "New" or "Revised" License

Shell 3.31% Dockerfile 96.69%

misp dockerhub security information-security threat-sharing malware malware-analysis threat-intelligence

docker-misp's Introduction

MISP - Threat Intelligence Sharing Platform

MISP is an open source software solution for collecting, storing, distributing and sharing cyber security indicators and threats about cyber security incidents analysis and malware analysis. MISP is designed by and for incident analysts, security and ICT professionals or malware reversers to support their day-to-day operations to share structured information efficiently.

The objective of MISP is to foster the sharing of structured information within the security community and abroad. MISP provides functionalities to support the exchange of information but also the consumption of said information by Network Intrusion Detection Systems (NIDS), LIDS but also log analysis tools, SIEMs.

  ●  Core functions   ●  Website / Support   ●  PHP and MISP
  ●  Installation   ●  Documentation   ●  Contributing
  ●  License

Latest Release
CI
Gitter
Mastodon
Twitter
Localization
Contributors
License

Core functions

An efficient IOC and indicators database, allowing to store technical and non-technical information about malware samples, incidents, attackers and intelligence.
Automatic correlation finding relationships between attributes and indicators from malware, attack campaigns or analysis. The correlation engine includes correlation between attributes and more advanced correlations like Fuzzy hashing correlation (e.g. ssdeep) or CIDR block matching. Correlation can also be enabled or event disabled per attribute.
A flexible data model where complex objects can be expressed and linked together to express threat intelligence, incidents or connected elements.
Built-in sharing functionality to ease data sharing using different model of distributions. MISP can automatically synchronize events and attributes among different MISP instances. Advanced filtering functionalities can be used to meet each organization's sharing policy including a flexible sharing group capacity and an attribute level distribution mechanisms.
An intuitive user-interface for end-users to create, update and collaborate on events and attributes/indicators. A graphical interface to navigate seamlessly between events and their correlations. An event graph functionality to create and view relationships between objects and attributes. Advanced filtering functionalities and warning lists to help the analysts to contribute events and attributes and limit the risk of false-positives.
storing data in a structured format (allowing automated use of the database for various purposes) with an extensive support of cyber security indicators along fraud indicators as in the financial sector.
export: generating IDS, OpenIOC, plain text, CSV, MISP XML or JSON output to integrate with other systems (network IDS, host IDS, custom tools), Cache format (used for forensic tools), STIX (XML and JSON) 1 and 2, NIDS export (Suricata, Snort and Bro/Zeek) or RPZ zone. Many other formats can be easily added via the misp-modules.
import: bulk-import, batch-import, import from OpenIOC, GFI sandbox, ThreatConnect CSV, MISP standard format or STIX 1.1/2.0. Many other formats easily added via the misp-modules.
Flexible free text import tool to ease the integration of unstructured reports into MISP.
A user-friendly system to collaborate on events and attributes allowing MISP users to propose changes or updates to attributes/indicators.
data-sharing: automatically exchange and synchronize with other parties and trust-groups using MISP.
delegating of sharing: allows for a simple, pseudo-anonymous mechanism to delegate publication of event/indicators to another organization.
Flexible API to integrate MISP with your own solutions. MISP is bundled with PyMISP which is a flexible Python Library to fetch, add or update events attributes, handle malware samples or search for attributes. An exhaustive restSearch API to easily search for indicators in MISP and exports those in all the format supported by MISP.
Adjustable taxonomy to classify and tag events following your own classification schemes or existing classification. The taxonomy can be local to your MISP but also shareable among MISP instances.
Intelligence vocabularies called MISP galaxy and bundled with existing threat actors, malware, RAT, ransomware or MITRE ATT&CK which can be easily linked with events and attributes in MISP.
Expansion modules in Python to expand MISP with your own services or activate already available misp-modules.
Sighting support to get observations from organizations concerning shared indicators and attributes. Sighting can be contributed via MISP user-interface, API as MISP document or STIX sighting documents.
STIX support: import and export data in the STIX version 1 and version 2 format.
Integrated encryption and signing of the notifications via GnuPG and/or S/MIME depending on the user's preferences.
Real-time publish-subscribe channel within MISP to automatically get all changes (e.g. new events, indicators, sightings or tagging) in ZMQ (e.g. misp-dashboard) or Kafka publishing.

Exchanging info results in faster detection of targeted attacks and improves the detection ratio while reducing the false positives. We also avoid reversing similar malware as we know very fast that other teams or organizations have already analyzed a specific malware.

A sample event encoded in MISP:

Website / Support

Checkout the website for more information about MISP software, standards, tools and communities.

Information, news and updates are also regularly posted on the MISP project Mastodon account, twitter account and news page.

PHP and MISP

MISP currently requires PHP 7.4, an end-of-life version of PHP. Because of this it is recommended that you only run MISP on distributions or PHP installs that you know will get security fixes backported, like Red Hat or Debian and derratives.

MISP 3.x, currently in development will support PHP 8.x.

Installation

For test- og production installations we recommend you check out the possible options on misp-project.org/download.

Documentation

MISP user-guide (MISP-book) is available online or as PDF or as EPUB or as MOBI/Kindle.

It is also recommended to read the FAQ

Contributing

If you are interested to contribute to the MISP project, review our contributing page. There are many ways to contribute and participate to the project.

Please see our Code of conduct.

Feel free to fork the code, play with it, make some patches and send us the pull requests via the issues.

Feel free to contact us, create issues, if you have questions, remarks or bug reports.

There is one main branch:

2.4 (current stable version): what we consider as stable with frequent updates as hot-fixes.

and features are developed in separated branches and then regularly merged into the 2.4 stable branch.

License

This software is licensed under GNU Affero General Public License version 3

For more information, the list of authors and contributors is available.

docker-misp's People

Contributors

Stargazers

Watchers

docker-misp's Issues

MISP Error Using Docker-Compose

It runs fine with the docker run command using the directory for mysql-db but gives following error on docker-compose.

Error:

misp_test | 2019-04-22 05:55:57,701 CRIT Supervisor running as root (no user in config file) misp_test | 2019-04-22 05:55:57,712 INFO supervisord started with pid 1 misp_test | 2019-04-22 05:55:58,714 INFO spawned: 'master' with pid 8 misp_test | 2019-04-22 05:55:58,715 INFO spawned: 'workers' with pid 9 misp_test | 2019-04-22 05:55:58,717 INFO spawned: 'syslog-ng' with pid 10 misp_test | 2019-04-22 05:55:58,718 INFO spawned: 'apache2' with pid 11 misp_test | 2019-04-22 05:55:58,719 INFO spawned: 'cron' with pid 12 misp_test | 2019-04-22 05:55:58,720 INFO spawned: 'misp-modules' with pid 13 misp_test | 2019-04-22 05:55:58,721 INFO spawned: 'mysqld_safe' with pid 14 misp_test | 2019-04-22 05:55:58,722 INFO spawned: 'redis-server' with pid 15 misp_test | 2019-04-22 05:55:58,769 INFO success: misp-modules entered RUNNING state, process has stayed up for > than 0 seconds (startsecs) misp_test | 2019-04-22 05:55:59,132 INFO exited: mysqld_safe (exit status 0; not expected) misp_test | 2019-04-22 05:55:59,770 INFO success: master entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) misp_test | 2019-04-22 05:55:59,770 INFO success: workers entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) misp_test | 2019-04-22 05:55:59,770 INFO success: syslog-ng entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) misp_test | 2019-04-22 05:55:59,770 INFO success: apache2 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) misp_test | 2019-04-22 05:55:59,770 INFO success: cron entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) misp_test | 2019-04-22 05:55:59,770 INFO success: redis-server entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) misp_test | 2019-04-22 05:56:00,217 INFO spawned: 'mysqld_safe' with pid 279 misp_test | 2019-04-22 05:56:00,570 INFO exited: workers (exit status 0; expected) misp_test | 2019-04-22 05:56:00,597 INFO exited: mysqld_safe (exit status 0; not expected) misp_test | 2019-04-22 05:56:02,600 INFO spawned: 'mysqld_safe' with pid 454 misp_test | 2019-04-22 05:56:02,976 INFO exited: mysqld_safe (exit status 0; not expected) misp_test | 2019-04-22 05:56:05,980 INFO spawned: 'mysqld_safe' with pid 611 misp_test | 2019-04-22 05:56:06,364 INFO exited: mysqld_safe (exit status 0; not expected) misp_test | 2019-04-22 05:56:07,365 INFO gave up: mysqld_safe entered FATAL state, too many start retries too quickly


services:
  MISP:
#    build: web
    container_name: misp_test
    image: vairavtech/misp:v1.1
    restart: unless-stopped
    ports:
       - "80:80"
       - "443:443"
       - "3306:3306"
       - "6379:6379"
    volumes:
#        - /dev/urandom:/dev/random
#        - /root/misp/docker-misp/data/misp/web:/var/www/MISP
       - /root/misp/docker-misp/test/misp-db:/var/lib/mysql
#        - /root/misp/docker-misp/certs:/etc/ssl/private

Docker image not starting

I'm running the ./build.sh successfully on a Ubuntu box. I am able to run the container. mysql and redis will not start. In my configuration of build.sh, I've changed the FQDN to be 127.0.0.1. I can hit the website on 443, but get a page stating:

An Internal Error Has Occurred.
Error: An Internal Error Has Occurred.

When i pull up a bash in the docker image, I don't see mysql or redis running and see the following output in the supervisord output when the container starts:

2018-07-07 21:07:53,926 INFO supervisord started with pid 1
2018-07-07 21:07:54,928 INFO spawned: 'master' with pid 7
2018-07-07 21:07:54,930 INFO spawned: 'workers' with pid 8
2018-07-07 21:07:54,931 INFO spawned: 'syslog-ng' with pid 9
2018-07-07 21:07:54,932 INFO spawned: 'apache2' with pid 10
2018-07-07 21:07:54,939 INFO spawned: 'cron' with pid 11
2018-07-07 21:07:54,940 INFO spawned: 'misp-modules' with pid 14
2018-07-07 21:07:54,944 INFO spawned: 'mysqld_safe' with pid 17
2018-07-07 21:07:54,946 INFO spawned: 'redis-server' with pid 19
2018-07-07 21:07:54,964 INFO success: misp-modules entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
2018-07-07 21:07:54,965 INFO exited: redis-server (exit status 0; not expected)
2018-07-07 21:07:55,425 INFO exited: mysqld_safe (exit status 0; not expected)
2018-07-07 21:07:55,964 INFO success: master entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-07-07 21:07:55,964 INFO success: workers entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-07-07 21:07:55,964 INFO success: syslog-ng entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-07-07 21:07:55,964 INFO success: apache2 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-07-07 21:07:55,964 INFO success: cron entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-07-07 21:07:56,065 INFO spawned: 'redis-server' with pid 250
2018-07-07 21:07:56,070 INFO exited: redis-server (exit status 0; not expected)
2018-07-07 21:07:56,071 INFO reaped unknown pid 255
2018-07-07 21:07:56,437 INFO spawned: 'mysqld_safe' with pid 281
2018-07-07 21:07:56,744 INFO exited: workers (exit status 0; expected)
2018-07-07 21:07:56,764 INFO exited: mysqld_safe (exit status 0; not expected)
2018-07-07 21:07:58,768 INFO spawned: 'mysqld_safe' with pid 445
2018-07-07 21:07:58,769 INFO spawned: 'redis-server' with pid 446
2018-07-07 21:07:58,775 INFO exited: redis-server (exit status 0; not expected)
2018-07-07 21:07:58,776 INFO reaped unknown pid 457
2018-07-07 21:07:59,104 INFO exited: mysqld_safe (exit status 0; not expected)
2018-07-07 21:08:02,109 INFO spawned: 'mysqld_safe' with pid 604
2018-07-07 21:08:02,111 INFO spawned: 'redis-server' with pid 605
2018-07-07 21:08:02,116 INFO exited: redis-server (exit status 0; not expected)
2018-07-07 21:08:02,117 INFO gave up: redis-server entered FATAL state, too many start retries too quickly
2018-07-07 21:08:02,118 INFO reaped unknown pid 615
2018-07-07 21:08:02,493 INFO exited: mysqld_safe (exit status 0; not expected)
2018-07-07 21:08:03,495 INFO gave up: mysqld_safe entered FATAL state, too many start retries too quickly

Putting SQL files inside docker

Hello

I would like to run MISP inside docker and push it to Amazon using Kubernetes.

However, this is using a VOLUME and the DB files are at the host.

How can i modify the file to have everything inside docker?

I tried to remove the VOLUME directive and then:
RUN /init-db;

WORKDIR /etc/logrotate.d...

But i am failing to run the container.

Any idea what is happening?

Thanks

[docker-misp]Proxy doesn't work

Thank you for your awesome work on harvarditsecurity/misp.
I have clone it from github and built my own container with the modification of build.sh.

Unfortunately, I have a problem when fetch feed through proxy.
I can see the proxy information in app/Config/config.php.
When I check servers/serverSettings/diagnostics, it shows:

Proxy
This tool tests whether your HTTP proxy settings are correct.
Proxy settings…OK

I try to capture the packets with tcpdump, but there has no traffic sent to proxy(all packets are sent to the feed server directly).
It seems that the proxy configuration didn't work.

Have you ever have this problem?
Do you know how to fix it?

Thank you again and waiting for your feedback.

build err

Collecting socketio-client==0.5.6
Downloading socketIO-client-0.5.6.tar.gz (12 kB)
ERROR: Command errored out with exit status 1:
command: /usr/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-_c41rtbv/socketio-client/setup.py'"'"'; file='"'"'/tmp/pip-install-_c41rtbv/socketio-client/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-install-_c41rtbv/socketio-client/pip-egg-info
cwd: /tmp/pip-install-_c41rtbv/socketio-client/
Complete output (7 lines):
Traceback (most recent call last):
File "", line 1, in
File "/tmp/pip-install-_c41rtbv/socketio-client/setup.py", line 6, in
README = open(os.path.join(here, 'README.rst')).read()
File "/usr/lib/python3.6/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 5786: ordinal not in range(128)
----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

README BEFORE OPENING NEW ISSUES

The MISP Docker project is created and maintained by:

Harvard University's Security Architecture and Engineering team
(https://github.com/harvard-itsecurity/docker-misp)

Please OPEN ALL NEW ISSUES, on the Harvard IT Security github page.

Docker Build Error - UnicodeDecodeError

When attempting to build on OSX I get this error at step 26, any recommendations?

Step 26/45 : RUN sudo pip3 install .
---> Running in 5c095762a0d6
Processing /var/www/MISP/app/files/scripts/python-maec
ERROR: Command errored out with exit status 1:
command: /usr/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-e8c6zs2_/setup.py'"'"'; file='"'"'/tmp/pip-req-build-e8c6zs2_/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' egg_info --egg-base pip-egg-info
cwd: /tmp/pip-req-build-e8c6zs2_/
Complete output (9 lines):
Traceback (most recent call last):
File "", line 1, in
File "/tmp/pip-req-build-e8c6zs2_/setup.py", line 54, in
long_description=get_long_description(),
File "/tmp/pip-req-build-e8c6zs2_/setup.py", line 27, in get_long_description
return f.read()
File "/usr/lib/python3.6/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 141: ordinal not in range(128)
----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
The command '/bin/sh -c sudo pip3 install .' returned a non-zero code: 1
bash-3.2$

Custom SSL certs

Original reported by @cellango:

Docker container works great without certs.

#!/bin/bash
docker run -it -d 
-p 443:443 
-p 80:80 
-p 3306:3306 
-v /home/centos/docker/docker-misp/misp-db:/var/lib/mysql 
-v /home/centos/docker/docker-misp/certs:/etc/ssl/private 
harvarditsecurity/misp

misp.crt and misp.key file in /home/centos/docker/docker-misp/certs. Also put the org master and intermediate in /etc/pki/ca-trust/source/anchors. Is there a way to debug what is going on?

@cellango - When I test with generating custom certs:

openssl genrsa -out misp.key 2048
openssl req -new -x509 -key misp.key -out misp.crt -days 3650 -subj /CN=test-ssl-cert-for-misp

Which produces:

% ls
misp.crt  misp.key

Then running docker with:

docker run -it -d \
-p 443:443 -p 80:80 -p 3306:3306 \
-v /docker/misp/certs:/etc/ssl/private \
-v /docker/misp:/var/lib/mysql \
harvarditsecurity/misp

I do see the cert:

What problem are you seeing?

MISP Server Configuration resets if container stop and restart

Hello,

I set several custom parameters in the MISP server configuration. However, after restarting the container, all settings were lost and the default (initial) configuration appeared again.

Thanks,
Francesco

The application log will not show using docker logs, only supervisord logs are showing

When MISP is started, application logs by default are located at /var/www/MISP/app/tmp/logs, I have changed bootstrap.php to use Console log instead of file log. However it could be because of supervisord, the docker logs will only show supervisord logs.

Here is what's in your supervisord configuration.
command=/bin/bash -c "source /etc/apache2/envvars && exec /usr/sbin/apache2 -DFOREGROUND"

I know docker will only collect logs from current console. So I have added the following based on this document.
http://veithen.io/2015/01/08/supervisord-redirecting-stdout.html
But it's still not working.
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0

I think the reason might be because the command starts a new shell.

Could you find out why?

Thanks
Tao

Fully Container MISP (DB inside the container)

Hello

Is it possible to have the the $docker-root inside the container?

I would like to put this on AWS using Kubernetes but there is this external set of files.

Would you be so kind to help me with this?

Thanks

Non persistent files and missing MySQL root password

Hello!

I have some question about your Docker Container:

1.) There is no volume in your documentation for the downloaded attachments under
/var/www/MISP/app/files
All attachments like malware examples, screenshots or other IOC´s are saved here. They are non-persistent in your docker configuration and will be earse by a rebuild ?!

2.) The deployment for your MySQL Server comes without a MySQL root password.
So the root password is blank. Also you forward the MySQL Port 3306 directly to the MySQL Server. Shouldn´t be that optional in that case ?

Is your Docker Container that one that we be also maintained in the future ?
Because there is a "docker-misp" and a "misp-docker" repository here which is very confusing.

Thanks for your help!

Can not start docker

I getting this error.
[admc1jac@apdstvmmips01 container]$ sudo docker run -it -d \

-p 443:443 \
-p 80:80 \
-p 3306:3306 \
-p 6666:6666 \
-v $docker-root/misp-db:/var/lib/mysql \
harvarditsecurity/misp

docker: Error response from daemon: create -root/misp-db: "-root/misp-db" includes invalid characters for a local volume name, only "[a-zA-Z0-9][a-zA-Z0-9_.-]" are allowed. If you intended to pass a host directory, use absolute path.
See 'docker run --help'.
[admc1jac@apdstvmmips01 container]$ sudo docker run
"docker run" requires at least 1 argument.
See 'docker run --help'.

Usage: docker run [OPTIONS] IMAGE [COMMAND] [ARG...]

Run a command in a new container
[admc1jac@apdstvmmips01 container]$ ls
Dockerfile supervisord.conf
[admc1jac@apdstvmmips01 container]$ cd /var/lib
[admc1jac@apdstvmmips01 lib]$ ls
AccountsService authconfig chrony colord dhclient flatpak fwupdate geoclue initramfs lldpad misc NetworkManager PackageKit pulse rpcbind rsyslog setroubleshoot systemd udisks2 upower xkb
alsa bluetooth clamav containerd dnsmasq fprint games gssproxy iscsi logrotate mlocate nfs polkit-1 rasdaemon rpm samba stateless tpm unbound uptrack yum
alternatives boltd color dbus docker fwupd gdm hyperv libvirt machines net-snmp os-prober postfix rhsm rpm-state selinux supportinfo tuned up2date vmware
[admc1jac@apdstvmmips01 lib]$ sudo docker run -it --rm \

-v $docker-root/misp-db:/var/lib/mysql \
harvarditsecurity/misp /init-db