flyballlabs / threatdetectionservice

Threat Management Platform with Apache Metron as the core engine

Threat Management Platform (formally Threat Detection Service)

The purpose of the Threat Management Platform (TMP) is to provide a platform to protect K-12 schools from threats. The threats can be Cyberattacks, Active Shooter, Child Predretors and etc. Hence, the core modules of TMP is CyberSecurity, School Lockdown Management and Suspicious Surveillance Alerts. The following diagram depicts the core modules:

This project was started in Detroit with a focus on delivering a solution to Detroit Charter Schools, but the project can leveraged for any school.

The first release of this software is focused on cybersecurity. We find that a portion of the Internet bandwidth for Detroit Charter Schools are being utilzied by hackers trying to take control of machines on the network to commit cyberattacks against other organizations. Also, there are computer viruses that install as ransomware where they pose as legitimate anti-virus software, but really they are just trying to obtain credit card numbers by having the user purchase fake virus software. Lastly, we want to protect the identity of the students and prevent unauthorizied access to their private information, which could be used to hijack the identity of the student. The theft of an identity causes major issues for an adult, but can cause similar issues or worst issues for a student down the road.

Our Threat Management Platform is using Apache Metron as the core engine. In order to detect Child Predretors we plan to use facial recognition software developed by Kairos

The architecture for TMP is shown below:

TMP API Server

The TMP API Server can be used to provide an API on top of Metron. Metron has alot of raw functionality and it's evolving each day. Our API provides a high level interface into Metron and adds concepts such as a User, Asset and Agents. We have developed the API to be easy to install and configure. Also, it comes with some sample data. The instructions to configure the API can be found here

Getting the Software Running

1. git clone https://github.com/flyballlabs/threatdetectionservice.git
2. cd threatdetectionservice
3. Start the API by following the instructions here
4. Start the GUI by following the instructions here
5. TODO: Simulate some threats

Note, the API can be run without the GUI. The API docs will be produced soon

threatdetectionservice's Issues

Agent Authentication

Users are authenticated by the api but agents should have verification as well.
At a minimum, when an agent calls to the api, it should be checked against the database, if their is a match then we know an authenticated user created that agent.
This will alleviate any script injection or zombie agent usage.

Metron-Setup Script

This script is responsible for configuring a Metron installation to work with the Threat Management Platform (TMP). The first release is focused on making it work with the Metron Quickstart VM.

Adding Notification setting for a user

A user should be able to specify how they want notification data to be sent. Per the following ways"

Email:

-provide an email field
-provide a score level. For example, only send messages when the score is 5 or less

SMS:

• provide a phone number
• provide a score level in which to be notified. For example, only send SMS messages when the score is 5 or more

Style the homepage and login page

In preparation for the upcoming release we need to style the homepage and login page

Add Notifications API

The Notifications API will be used to notify a SOC User that a Threat was received with a particular Threat Level. The following components will need to be changed:

Metron Threat Intel Configs:

• parser/enrichment/triage_configs

Please use the issue # to commit any changes. This will allow us to track the commits related to this Issue

Agent update script

Needs to have cmds expanded to accept 'provision' as a cmd.
API needs to have this functionality added as well.

Expose Agent API functions

Agents need to be searchable through gui.
Need to be able to add agents using gui.
Should be searchable by company and per single agent as well.
Should be able to add agent to database.

Login Errors

Catch Login errors that happen because the API has a failure other then the API Server not being available.

Metron won't enrich threat intel

Can not get metron to enrich threat intel via threat intel bulk loader.
When I try the threat intel bulk load cmd zookeeper hangs at connection, see output below:

2017-01-16 02:38:57,725 INFO [main] zookeeper.ZooKeeper: Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2017-01-16 02:38:57,725 INFO [main] zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
2017-01-16 02:38:57,725 INFO [main] zookeeper.ZooKeeper: Client environment:java.compiler=
2017-01-16 02:38:57,725 INFO [main] zookeeper.ZooKeeper: Client environment:os.name=Linux
2017-01-16 02:38:57,725 INFO [main] zookeeper.ZooKeeper: Client environment:os.arch=amd64
2017-01-16 02:38:57,725 INFO [main] zookeeper.ZooKeeper: Client environment:os.version=2.6.32-642.6.2.el6.x86_64
2017-01-16 02:38:57,725 INFO [main] zookeeper.ZooKeeper: Client environment:user.name=root
2017-01-16 02:38:57,725 INFO [main] zookeeper.ZooKeeper: Client environment:user.home=/root
2017-01-16 02:38:57,725 INFO [main] zookeeper.ZooKeeper: Client environment:user.dir=/assets-hosts
2017-01-16 02:38:57,726 INFO [main] zookeeper.ZooKeeper: Initiating client connection, connectString=node1:2181 sessionTimeout=90000 watcher=hconnection-0x4671115f0x0, quorum=node1:2181, baseZNode=/hbase-unsecure
2017-01-16 02:38:57,747 INFO [main-SendThread(node1:2181)] zookeeper.ClientCnxn: Opening socket connection to server node1/192.168.66.121:2181. Will not attempt to authenticate using SASL (unknown error)
2017-01-16 02:38:57,754 INFO [main-SendThread(node1:2181)] zookeeper.ClientCnxn: Socket connection established to node1/192.168.66.121:2181, initiating session
2017-01-16 02:38:57,756 INFO [main-SendThread(node1:2181)] zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2017-01-16 02:38:57,861 WARN [main] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=node1:2181, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase-unsecure/hbaseid
2017-01-16 02:38:59,233 INFO [main-SendThread(node1:2181)] zookeeper.ClientCnxn: Opening socket connection to server node1/192.168.66.121:2181. Will not attempt to authenticate using SASL (unknown error)
2017-01-16 02:38:59,234 INFO [main-SendThread(node1:2181)] zookeeper.ClientCnxn: Socket connection established to node1/192.168.66.121:2181, initiating session
2017-01-16 02:38:59,234 INFO [main-SendThread(node1:2181)] zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2017-01-16 02:38:59,334 WARN [main] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=node1:2181, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase-unsecure/hbaseid
2017-01-16 02:39:01,153 INFO [main-SendThread(node1:2181)] zookeeper.ClientCnxn: Opening socket connection to server node1/192.168.66.121:2181. Will not attempt to authenticate using SASL (unknown error)
2017-01-16 02:39:01,153 INFO [main-SendThread(node1:2181)] zookeeper.ClientCnxn: Socket connection established to node1/192.168.66.121:2181, initiating session
2017-01-16 02:39:01,155 INFO [main-SendThread(node1:2181)] zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect

After replacing the ojdbc.jar file mentioned in the error zookeeper client is unable to connect. See below:

2017-01-16 02:38:57,725 INFO [main] zookeeper.ZooKeeper: Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2017-01-16 02:38:57,725 INFO [main] zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
2017-01-16 02:38:57,725 INFO [main] zookeeper.ZooKeeper: Client environment:java.compiler=
2017-01-16 02:38:57,725 INFO [main] zookeeper.ZooKeeper: Client environment:os.name=Linux
2017-01-16 02:38:57,725 INFO [main] zookeeper.ZooKeeper: Client environment:os.arch=amd64
2017-01-16 02:38:57,725 INFO [main] zookeeper.ZooKeeper: Client environment:os.version=2.6.32-642.6.2.el6.x86_64
2017-01-16 02:38:57,725 INFO [main] zookeeper.ZooKeeper: Client environment:user.name=root
2017-01-16 02:38:57,725 INFO [main] zookeeper.ZooKeeper: Client environment:user.home=/root
2017-01-16 02:38:57,725 INFO [main] zookeeper.ZooKeeper: Client environment:user.dir=/assets-hosts
2017-01-16 02:38:57,726 INFO [main] zookeeper.ZooKeeper: Initiating client connection, connectString=node1:2181 sessionTimeout=90000 watcher=hconnection-0x4671115f0x0, quorum=node1:2181, baseZNode=/hbase-unsecure
2017-01-16 02:38:57,747 INFO [main-SendThread(node1:2181)] zookeeper.ClientCnxn: Opening socket connection to server node1/192.168.66.121:2181. Will not attempt to authenticate using SASL (unknown error)
2017-01-16 02:38:57,754 INFO [main-SendThread(node1:2181)] zookeeper.ClientCnxn: Socket connection established to node1/192.168.66.121:2181, initiating session
2017-01-16 02:38:57,756 INFO [main-SendThread(node1:2181)] zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2017-01-16 02:38:57,861 WARN [main] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=node1:2181, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase-unsecure/hbaseid

Create central configuration files for the TMP API and GUI

The configuration scripts should be Python based with a name of config.py. There should be a script for api and gui components such as api/config.py and gui/config.py. The scripts should provide a place to put central configuration info.

Expose Agent Configuration Parameters via GUI

This includes but not limited to adding, updating and deleting Agent configuration data. Once the data is added we should be able to call the agent API and return that data.

Database Schema Should be Automatically Updated

We need to have an approach that will automatically update the database schema when changes occur at the DB level

We need to implement the User and Agent Entity Based on Standards

I've created a standard approach based on the flask documentation that all of our entities within our API should implement. Please apply this approach to the User and Agent entities.

The following is in the "Adding an API" section of this document:

https://github.com/flyballlabs/threatdetectionservice/blob/master/CONTRIBUTING.md

Add an agent search and update

Add the ability to search and update agents. We are going to add the MAC Address

Add Facial Recognition

This will allow a user to copy and paste and email into the system and then perform facial recognition on a pre-defined set of mugshots