rocknsm / rock Goto Github PK
View Code? Open in Web Editor NEWAutomated deployment scripts for the RockNSM network hunting distribution.
Home Page: http://rocknsm.io
License: Apache License 2.0
Automated deployment scripts for the RockNSM network hunting distribution.
Home Page: http://rocknsm.io
License: Apache License 2.0
After the first Bro log rollover, the unified2 reader seems to stop picking up snort alerts.
User Story: As a power user, I want a web UI to manage advanced Elasticsearch queries so that I can access raw data and aid in development of new features.
Sense is now a Kibana plugin. This should be added to the download / offline install procedures.
https://www.elastic.co/guide/en/sense/current/installing.html
The duration field is being indexed and valued, but as an individual "string" such as 0.0000986, 0.0000987, etc.
This is crippling because you can't visualize strings over time due to obvious reasons.
Side Note: Using Offline ISO Installation Build.
Installation of SimpleRock failed with non-vagrant VM method. Used:
Beginning of installation failure as follows:
+++++++++++++++++++++++++++++++++++++++++++
Error executing action `run` on resource 'execute[start_bro]'
Mixlib::ShellOut::ShellCommandFailed
+++++++++++++++++++++++++++++++++++++++
Attached is the full transcript:
For prettier, sexier graphs, we should install timelion, which eats timeseries data for breakfast (lunch is zebras).
For connected systems, the way to do this is as follows:
./bin/kibana plugin -i kibana/timelion
Need to research offline install methods.
Temp workaround is (possibly) to add a "-k" to the curl syntax. Cert handling/generation upstream may need adjusted to permanently fix.
CentOS-CR.repo doesn't exist until yum update -y && yum upgrade -y
, might want to add this to the prep or the recipe.
Also I think yum install -y kernel-headers kernel-devel && reboot
should be part of the prep or part of the recipe.
plugin:marvel Marvel version 2.2.0 is not supported with Kibana 4.3.1
error on current build of the ROCK
Users can benefit from being able to generate alerting conditions based on frequency, volume, and content based (keyword and diff) searches that are scheduled to run periodically and when they meet a defined condition they report to another application (like a chat client, email, tracker, etc...). It looks like Yelp has built a nice framework for this in ElastAlert that could be a suitable candidate for inclusion in SimpleROCK.
Apparently, the install fails with Bro (full transcript attached).
Super low priority request but I noticed that a couple of the diagrams under https://github.com/CyberAnalyticDevTeam/SimpleRock/tree/gh-pages/images are using outdated logos for Elasticsearch, Logstash, and Kibana. The most recent product logos are available at https://www.elastic.co/brand. Thanks!
I'm having an issue running Vagrant.
SimpleRock$ vagrant up
/opt/vagrant/embedded/gems/gems/vagrant-1.8.1/lib/vagrant/pre-rubygems.rb:31: warning: Insecure world writable dir /opt in PATH, mode 040777
/opt/vagrant/embedded/gems/gems/bundler-1.10.6/lib/bundler/shared_helpers.rb:78: warning: Insecure world writable dir /opt in PATH, mode 040777
Bringing machine 'default' up with 'virtualbox' provider...
==> default: Checking if box 'relativkreativ/centos-7-minimal' is up to date...
==> default: Clearing any previously set forwarded ports...
==> default: Clearing any previously set network interfaces...
==> default: Preparing network interfaces based on configuration...
default: Adapter 1: nat
==> default: Forwarding ports...
default: 5601 (guest) => 5601 (host) (adapter 1)
default: 9200 (guest) => 9200 (host) (adapter 1)
default: 22 (guest) => 2222 (host) (adapter 1)
==> default: Running 'pre-boot' VM customizations...
==> default: Booting VM...
==> default: Waiting for machine to boot. This may take a few minutes...
The guest machine entered an invalid state while waiting for it
to boot. Valid states are 'starting, running'. The machine is in the
'poweroff' state. Please verify everything is configured
properly and try again.
If the provider you're using has a GUI that comes with it,
it is often helpful to open that and watch the machine, since the
GUI often has more helpful error messages than Vagrant can retrieve.
For example, if you're using VirtualBox, run `vagrant up` while the
VirtualBox GUI is open.
The primary issue for this error is that the provider you're using
is not properly configured. This is very rarely a Vagrant issue.
I opened VirtualBox and tried to start it manually and it said that there was a network interface issue.
Nonexistent host networking interface, name 'vboxnet0' (VERR_INTERNAL_ERROR).
Result Code:
NS_ERROR_FAILURE (0x80004005)
Component:
ConsoleWrap
Interface:
IConsole {872da645-4a9b-1727-bee2-5585105b9eed}
I need to be able to specify the Ethernet monitoring interface. Short-term fix is to manually modify the Chef code to specify the specific interface that monitoring needs to take place on, but not sure how to do that.
The bro-cut command is not located in $PATH
echo $PATH output currently shows
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/kafka/bin:/root/bin
bro-cut works by using /opt/bro/bin/bro-cut currently.
I find it helpful to create a symlink from the "normal" directories to the modified directories. This reduces unfamiliarity to new users so they can just get the job done.
To free space on ROCK without without permanently losing data, include or document a method to export indexes from elasticsearch. Should also be able to import previously exported data.
Added bonus: This would allow Kibana dashboards to be exported and imported.
From Derek:
How To Update the Ethernet Monitoring Interface AFTER Installation
Modify the following:
By default, Kafka dumps topic data to /tmp/kafka-logs
. This is....not exactly good. Should consider moving to /data/kafka
to be consistent and make it easier to allocate data space.
This is set in /opt/kafka/config/server.properties
.
Propose change:
log.dirs=/tmp/kafka-logs
to
log.dirs=/data/kafka
This should be relatively easy to implement with Logstash aggregating, but I've had a couple people ask lately for an ability to setup a clustered mode similar to that of Security Onion. Just dropping here for idea recording.
Need a script to configure kibana to load indexes upon starting of kibana
Create a systemd service that runs idstool to generate JSON events for snort. This then needs to be ingested by logstash as is currently done with suricata eve.json
See https://blog.jasonish.org/2014/04/16/snort-logstash-elastic-search-and-kibana/
Upon setup, would like to have the option to monitor 2-4 Ethernet interfaces instead of the default single interface. How would this be done after installation?
For suricata eve.json, the project provides a recommended configuration
/data/suricata/*.log /data/suricata/*.json /data/suricata/unified2*.alert
{
rotate 3
missingok
nocompress
create
sharedscripts
postrotate
/bin/kill -HUP $(cat /var/run/suricata.pid)
endscript
}
Maybe something like:
Need option for optimized settings for higher-speed monitoring interface if used for sniffing - 10GB Ethernet.
If this is best done/tweaked AFTER installation, then we should document in the Wiki.
With the exception of wget, opsode runs a pretty nice, and well maintained base set of boxes for various hypervisors.
Here's the kickstart.
https://github.com/chef/bento/blob/master/http%2Fcentos-7.2%2Fks.cfg
Setting module parameters should occur with a .conf file in /etc/modprobe.d as detailed in the RHEL documentation.
/etc/pf_ring/pf_ring.conf
doesn't actually do anything unless you use the PF_RING init script. This isn't necessary unless you're using the ZeroCopy commercial drivers, as it requires a daemon for management.
Some helpful items:
Need to upgrade Kibana to 4.2 and create bats test for status to be green
See https://www.elastic.co/guide/en/kibana/current/status.html
Help us Obi-wan @dcode, you're our only hope!
Simple Documentation, no change requested.
Decided to do a little tinkering with SimpleRock from a Windows 7 Machine with Virtualbox 5.0.12 and ran into a slight problem with the Virtualbox provider configs for --hostonlyadapter2. From the virtualbox logs I found this useful error message:
00:00:00.492329 VMSetError: Nonexistent host networking interface, name 'vboxnet0'
Switching vboxnet0 with the actual host only adapter name pulled from ipconfig /all did the trick.
Heads-up. Checked out the Snort setup for SimpleRock with a friend and it looks like the new build with Snort has a misconfiguration with Share Object Rules in snort.conf and pulledpork.
http://blog.talosintel.com/2009/01/using-vrt-certified-shared-object-rules.html
More info on Shared Object rules:
http://blog.snort.org/2011/02/snort-shared-object-rules.html
However, some may be finding it difficult to use the rules, so let me point you to a couple guides. One guide is here, on Snort.org, at the bottom of the "platform" list. The VRT also has a blog post that can help you install the Shared Object rules.But, by far, the easiest way to use Shared Object rules reliably is through the configuration and use of a tool called PulledPork, which JJ Cummings of Sourcefire is the primary author of. After the configuration of PulledPork, the tool will generate the Shared Object rule stubs for you, and place everything in the correct directories for ease of use. This is amongst the many features of PulledPork (including flowbit dependency resolving) which are useful.
Update Wiki (or readme) with enabling tcpreplay procedure for testing:
vi /etc/pf_ring/pf_ring.conf
change enable_tx_capture to "1"
Restart system
It appears that pf_ring is preventing the monitor interface from starting (or staying up). The install appeared to work properly, all the other services are working.
Troubleshooting:
rock_stop
rock_start
Output from rock_status
[root@simplerockbuild ~]# rock_status
✗ Check each monitor interface is live
(in test file /usr/local/bin/rock_status, line 24)
`[ $packets -gt 0 ]' failed
eno33554960 had 0 packets in 5 secs.
✓ Check for interface errors
✓ Check monitor interface for tx packets
✓ Check PF_RING settings
✓ Check that broctl is running
✓ Check for bro-detected packet loss
✓ Check that zookeeper is running
✓ Check that zookeeper is listening
✓ Check that client can connect to zookeeper
✓ Check that kafka is running
✓ Check that kafka is connected to zookeeper
✓ Check that logstash is running
✓ Check that elasticsearch is running
✓ Check that kibana is running
14 tests, 1 failure
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.