Software Defined Network Situational Awareness
Poseidon began as a joint effort between two of the IQT Labs: Cyber Reboot and Lab41. The project's goal is to explore approaches to better identify what nodes are on a given (computer) network and understand what they are doing. The project utilizes Software Defined Networking and machine learning to automatically capture network traffic, extract relevant features from that traffic, perform classifications through trained models, convey results, and provide mechanisms to take further action. While the project works best leveraging modern SDNs, parts of it can still be used with little more than packet capture (pcap) files.
- Background
- Prerequisites
- Installing Poseidon
- SDN Controller Configuration
- Usage
- Related Components
- Additional Info
The Poseidon project originally began as an experiment to test the merits of leveraging SDN and machine learning techniques to detect abnormal network behavior. (Please read our blogs posts linked below for several years of background) While that long-term goal remains, the unfortunate reality is that the state of rich, labelled, public, and MODERN network data sets for ML training is pretty poor. Our lab is working on improving the availability of network training sets, but in the near term the project remains focused on 1) improving the accuracy of identifying what a node IS (based on captured IP header data) and 2) developing Poseidon into a "harness" of sorts to house machine learning techniques for additional use cases. (Read: Not just ours!)
- A dedicated Linux System or Virtual Machine (A Debian-based distribution is preferred - Ubuntu 16.x is ideal)
- Currently supported versions for the .DEB install are:
- Ubuntu 14.04
- Ubuntu 16.04
- Ubuntu 18.04
- Ubuntu 18.10
- Currently supported versions for the .DEB install are:
- Docker - Poseidon and related components run on top of Docker, so understanding the fundamentals will be useful for troubleshooting as well. A Good Ubuntu Docker Quick-Start
- ~10GB of free disk space
- An SDN Controller - specifically BigSwitch Cloud Fabric or Faucet - if you want full functionality (this is now optional, if you simply want to replay previously created captures of a network, or don't have an SDN environment availabble).
Note: Installation on
OS X
is possible but not supported, see the./helpers/run
file (above) as a starting point.
On Ubuntu, this will download and install our .deb
package from Cloudsmith.
sudo usermod -aG docker $USER
sudo apt-get install -y apt-transport-https curl
curl -sLf "https://dl.cloudsmith.io/public/cyberreboot/poseidon/cfg/gpg/gpg.F9E23875C98A1F72.key" | sudo apt-key add -
sudo add-apt-repository "deb [arch=amd64] https://dl.cloudsmith.io/public/cyberreboot/poseidon/deb/ubuntu $(lsb_release -cs) main"
sudo apt-get update
sudo apt-get install poseidon
If you prefer a lightweight package that downloads the images from the latest build at install time, you can alternatively install: sudo apt-get install poseidon-net
Note: The installer has a
Demo
option in the installation wizard that will deploy and configure the full Poseidon package, the Faucet SDN contoller (and related components like Grafana and Prometheus), mininet, and openvswitch. We suggest the demo install as a starting point if much of this is new to you.
If you opt to do a full install (NOT the demo mode), you need to first identify one of the two supported controllers (BigSwitch Cloud Fabric or Faucet). The controller needs to be running and accessible (via network API) by the Poseidon system. We recommend making sure the SDN portion is configured BEFORE the above Poseidon installation, but it's not a hard requirement.
You will need to access the API of your Big Cloud Fabric (BCF) controller using authorized credentials (user/pass) and you will need to add support in your BCF controller for moving mirrored endpoint network data around your network. In BCF, allowing port mirroring for Poseidon is done using 1) the "span-fabric" feature and 2) identifying a switch interface to send the captured ("spanned") traffic out of. The BigSwitch config will need an admin to add:- span-fabric: you need to define a fabric-wide port mirroring mechanism and give it a name (e.g. 'poseidon')
- interface-group: you need to identify which port the mirrored traffic is going to egress from, and name it (e.g. 'ig1')
Replace <name>
with the name of your span-fabric and <interface-group>
with the name of your interface-group.
! span-fabric
span-fabric <name>
active
destination interface-group <interface-group>
priority 1
Replace <interface-group>
with the name of your interface-group. Additionally fill in the YOUR_LEAF_SWITCH
and YOUR_INTERFACE_WHERE_VENT_WILL_RECORD_TRAFFIC_FROM
.
! interface-group
interface-group <interface-group>
description 'packets get mirrored here to be processed'
mode span-fabric
member switch YOUR_LEAF_SWITCH interface YOUR_INTERFACE_WHERE_VENT_WILL_RECORD_TRAFFIC_FROM
Poseidon will connect to the BCF controller using its REST API, so you will also need the BCF API hostname or IP address and credentials for the controller. If your controller is an HA pair and has a virtual IP address, we recommend using that virtual address. Also, because Poseidon will be making dynamic filter
rule changes we will need an account that has administrative priveleges. (Poseidon only modifies the filter rules of the defined span-fabric, but until BigSwitch has more granular access control options this means admin privs!) Bringing the above configuration requirements together, below is an example of what the relevant parts of your BCF configuration could look like where the span-fabric is called 'poseidon', the user 'poseidon' is defined for API access, and the egress interface is interface '48' on switch 'leaf04' and labelled as interface group 'ig1':
! user
user poseidon
hashed-password method=PBKDF2WithHmacSHA512,salt=M4534fcV1Ksg_fNm2pGQ,rounds=25000,ph=true,eWNHYUPVAUYosBVRguJnkmAzM
! group
group admin
associate user poseidon
! interface-group
interface-group ig1
description 'Mirroring for Poseidon'
mode span-fabric
member switch leaf04 interface ethernet48
! span-fabric
span-fabric poseidon
active
destination interface-group ig1
priority 1
BCF is now configured and ready for use with Poseidon.
Poseidon requires at least Faucet version 1.8.6 or higher.Unless Poseidon and Faucet are running on the same host, Poseidon will connect to Faucet using SSH. So you'll need to create an account that can SSH to the machine running Faucet and that has rights to modify the configuration file faucet.yaml
(currently Poseidon expects it to be in the default /etc/faucet/faucet.yaml
location and dps
must all be defined in faucet.yaml
for Poseidon to update the network posture correctly).
Faucet needs to be started with the following environment variables set:
export FAUCET_EVENT_SOCK=1
export FAUCET_CONFIG_STAT_RELOAD=1
If using the RabbitMQ adapter for Faucet (recommended) make sure to also export FA_RABBIT_HOST
to the IP address of the host where Poseidon is running.
Faucet is now configured and ready for use with Poseidon.
NEW: If you have used the .DEB installer previously, it is worth noting that Poseidon is now packaged as a standard Linux service, and ties in nicely to both systemctl and journalctl.
After installation you'll have a new command poseidon
available for looking at the status, logs, changing the configuration, or stopping and starting the service.
$ poseidon help
Poseidon 0.5.7, an application that leverages software defined networks (SDN) to acquire and then feed network traffic to a number of machine learning techniques. For more info visit: https://github.com/CyberReboot/poseidon
Usage: poseidon [option]
Options:
-a, api get url to the Poseidon API
-c, config display current configuration info
-e, shell enter into the Poseidon shell, requires Poseidon to already be running
-h, help print this help
-i, info/status display current status of the Poseidon service
-l, logs display the information logs about what Poseidon is doing
-L, system-logs display the system logs related to Poseidon
-p, pcap process a PCAP file or directory of PCAPs (uses sudo)
-R, reconfig reconfigures all settings (uses sudo, will restart the Poseidon service)
-r, restart restart the Poseidon service (uses sudo)
-s, start start the Poseidon service (uses sudo)
-S, stop stop the Poseidon service (uses sudo)
-v, viz/visualize get url to visualize Poseidon with CRviz
-V, version display the version of Poseidon and exit
-Z, reset reset the configuration (uses sudo)
Poseidon makes heavy use of a sister project, vent. With a successful installation you should minimally see a combination of Poseidon and Vent components, to include:
- The following 14 containers with a "(healthy)" STATUS listed (NOTE: this is truncated output):
# docker ps
CONTAINER ID IMAGE COMMAND STATUS
8c07adf421fb cyberreboot/poseidon:master "/bin/sh -c '(flask …" Up 2 hours (healthy)
0a4f947f299b cyberreboot/vent-file-drop:master "/bin/sh -c '(flask …" Up 2 hours (healthy)
511f90c6ddd3 cyberreboot/crviz:master "serve -s build -l 5…" Up 2 hours (healthy)
fb250044ff17 cyberreboot/poseidon-api:master "/bin/sh -c '(flask …" Up 2 hours (healthy)
8e898fd68c08 cyberreboot/vent-network-tap:master "/bin/sh -c '(flask …" Up 2 hours (healthy)
552f65d7a982 cyberreboot/vent-rq-worker:master "/bin/sh -c '(flask …" Up 2 hours (healthy)
8dbabe78d1b9 cyberreboot/vent-rq-worker:master "/bin/sh -c '(flask …" Up 2 hours (healthy)
e076452c1515 cyberreboot/vent-rq-worker:master "/bin/sh -c '(flask …" Up 2 hours (healthy)
d0f406f240b1 cyberreboot/vent-rq-worker:master "/bin/sh -c '(flask …" Up 2 hours (healthy)
6229e46723a9 cyberreboot/vent-rq-dashboard:master "/bin/sh -c '(flask …" Up 2 hours (healthy)
5c695040603b cyberreboot/vent-redis:master "docker-entrypoint.s…" Up 2 hours (healthy)
004e5fdde96e cyberreboot/vent-syslog:master "/usr/sbin/syslog-ng…" Up 2 hours (healthy)
c0cd7c1f881c cyberreboot/vent-rabbitmq:master "docker-entrypoint.s…" Up 2 hours (healthy)
d81f5509628c cyberreboot/vent "/bin/sh -c '(flask …" Up 2 hours (healthy)
(NOTE: On our test systems it usually takes 1-2 minutes for all containers to spin up)
If you performed the demo installation, you should also see the following Faucet-related containers running (NOTE: Faucet has not yet implemented docker-friendly health checks, so the "(healthy)" reference will not be shown):
196f53632485 grafana/grafana:5.2.1 "/run.sh" Up 2 hours
cfb7d68b66e0 prom/prometheus:v2.3.1 "/bin/prometheus --c…" Up 2 hours
86f4188d67f6 faucet/gauge:latest "/usr/local/bin/entr…" Up 2 hours
2e186573532e faucet/event-adapter-rabbitmq "/usr/local/bin/entr…" Up 2 hours
c0652a6ccd44 influxdb:1.6-alpine "/entrypoint.sh infl…" Up 2 hours
2a949b5b1687 faucet/faucet:latest "/usr/local/bin/entr…" Up 2 hours
- You should see "Poseidon successfully started, capturing logs..." in your syslog output:
# journalctl -u poseidon | grep capturing
Jul 25 15:42:20 PoseidonHost poseidon[4273]: Poseidon successfully started, capturing logs...
To continue to test (assuming demo installation), please see /opt/poseidon/docs/demo.txt
, also referenced in the repo above.
If installed as described above, poseidon's codebase will be at /opt/poseidon
. At this location, a .vent_startup.yml
file can be edited to point to a fork of the original repository. Develop and commit changes on the poseidon fork and use poseidon -r
to reload and see your changes.
You can verify that it's building against your fork by doing a docker ps
and the poseidon container will be named off of your fork.
To make changes to anything outside of the poseidon
subdirectory you will need to build a new .deb
and reinstall.
git clone <YOUR-POSEIDON-FORK>
cd poseidon
make build_debian
sudo dpkg -i dist/poseidon*.deb
- Authors
- Blog posts:
- Running Poseidon on a 100G Netowork
- Using machine learning to classify devices on your network
- CRviz: Initial Release
- CRviz: Scalable design for network visualization
- A better way to visualize what’s on our networks?
- TCPDump, and the care and feeding of an intelligent SDN
- The Case for Detecting Lateral Movement
- Poseidon with FAUCET SDN Controller
- Building a Software-Defined Network with Raspberry Pis and a Zodiac FX switch
- Thanks to FAUCET, Poseidon Now Supports Switches Running OpenFlow 1.3
- Deep Session Learning for Cyber Security
- Introducing Vent
- SDN and the need for more (security) verbs
- See the latest changes here.
- Code of Conduct
- Want to contribute? Awesome! Issue a pull request or see more details here.
- Developer Guide
- License
- Maintainers
- Releases
- Tests
- Version