Error when configurating Minerva about minerva

[rc1405]

The latest push should have fixed the problem. When connecting to an already configured Minerva db, I missed some of the initial configuration. Let me know if that resolves the problem

[samiux]

I need to delete /opt/minerva before re-configure the Minerva. The latest version can configure Minerva successfully.

However, there is no way to enter username and password of Webserver. Meanwhile, when I start agent.py and receiver.py, the following error displayed.

samiux@smp:~$ sudo python /opt/minerva/bin/agent.py
Traceback (most recent call last):
File "/opt/minerva/bin/agent.py", line 160, in
main()
File "/opt/minerva/bin/agent.py", line 132, in main
cur_config = core.MinervaConfigs(conf=os.path.join(os.path.abspath(os.pardir), 'etc/minerva.yaml')).conf['Agent_forwarder']
File "/usr/lib/python2.7/Minerva/core.py", line 34, in init
raise "Config File not found"
TypeError: exceptions must be old-style classes or derived from BaseException, not str

samiux@smp:~$ sudo python /opt/minerva/bin/receiver.py
Traceback (most recent call last):
File "/opt/minerva/bin/receiver.py", line 129, in
main()
File "/opt/minerva/bin/receiver.py", line 82, in main
minerva_core = core.MinervaConfigs(conf=os.path.join(os.path.abspath(os.pardir), 'etc/minerva.yaml'))
File "/usr/lib/python2.7/Minerva/core.py", line 34, in init
raise "Config File not found"
TypeError: exceptions must be old-style classes or derived from BaseException, not str

samiux@smp:~/Minerva/bin$ sudo python setup.py install
Please choose an install method:
1. StandAlone (Server, Agent and Receiver)
2. Server/Receiver
3. WebServer only
4. Receiver Only
5. Agent Only
6. Database Only

1
Enter installation Directory: /opt/minerva
Connect to existing database? [y/n] y
Setting Up Receiver DB connection
Please enter database ip: [127.0.0.1] 127.0.0.1
Please enter database port: [27017] 27017
Use db authentication? Y/N [N] N
Enter number of minutes until each console session times out: 10
Setting up the web server

Enter hostname for webserver: minerva_web
Enter IP Address to bind to: 192.168.20.251
Enter Port for webserver to run on: [443] 9443
Enter number of threads to respond to web requests: [8] 8
Enter full path of webcertificate to use (Will create one if none exists) [/var/lib/minerva/webserver/server.pem] /var/lib/minerva/webserver/server.pem
Enter full path of web server's private key: /var/lib/minerva/webserver/private.pem
Enter # of logon attempts before user is locked out: [3] 3
Enter minimum length for user passwords: [8] 8
Enter # of days a password is valid before needed to be changed: [90] 90
Enter # of seconds to wait on a pcap request: [300] 300
Enter # of results to show in the console at a time: [5000](15000 max) 5000
Enter minimum # of lower case letters in a password: [2] 2
Enter minimum # of upper case letters in a password: [2] 2
Enter minimum # of numbers in a password: [2] 2
Enter minimum # of special characters in a password: [2] 2
Setting up the event receiver

Enter IP Address to listen on: 192.168.20.251
Enter port to listen on: 5688
Do you want to add more ports? [y/n] n
How many threads do you want to process events? 8
Do you want to add another IP? [y/n] n
Enter number of seconds to timeout on a single receive thread: [20] 20
Enter number of processes you want to insert alerts: [4] 4
Enter max number of events to insert at a time: [500] 500
Enter max seconds to wait before inserting events: [20] 20
Enter full path of certificate to use (will create in this lcoation if it doenst exist): [/var/lib/minerva/receiver/server.pem] /var/lib/minerva/receiver/server.pem
Enter full path of private key to use w/ the certificate above: /var/lib/minerva/receiver/private.pem
Enter IP Address to listen for pcap requests from the webserver: 192.168.20.251
Enter Port of Receiver to list for pcap requests for: [10009] 10009
Enter number of threads to process pcap requests: [4] 4
Enter number of seconds to wait for a pcap request, Should be the same as webserver value: [300] 300
Setting up the agent

Enter name of sensor: sensor
Enter full pathname of sensor certificate (One will be created if it doesn't exist): [/var/lib/minerva/agent/agent.pem] /var/lib/minerva/agent/agent.pem
Enter full pathname of sensor private key for the certificate above: /var/lib/minerva/agent/private.pem
Enter full pathname of log file to send in: /var/log/minerva
Enter alert type of log file: (suricata_eve, snort_alert): suricata_eve
Enter full pathname of position file: /var/log/suricata
Do you want to add more log files? [y/n] n
Enter full pathname of where to save server cert: [/var/lib/minerva/agent/server.pem] /var/lib/minerva/agent/server.pem
Enter IP address of receiver to send to: 192.168.20.251
Enter destination port to send to: 6666
Enter max # of events to send at once: [500] 500
Enter max # of seconds to wait to send events (Will send earlier if max events is reached): [10] 10
Configuring Agent PCAP Requests
Enter max # of packets to return per request: [10000] 10000
Enter Max size(mb) of pcap files to return per reqeust: [20] 20
Enter Max # of pcap files to search through per request: [10] 10
Enter max time in seconds past an event to grab packets for: [300] 300
Enter prefix for pcap files: []
Enter suffix for pcap files: [.pcap] .pcap
Enter complete path to base directory for pcap files: /var/log/mineria/pcap
Enter complete path of temp storage for pcap requests: /var/log/mineria/pcap/temp
Enter ip address to listen for requests on: 192.168.20.251
Enter port to listen for requests on: [10009] 10009
Enter number of threads to process requests: [4] 4
samiux@smp:~/Minerva/bin$

[rc1405]

Enter n when prompted at "Connect to existing database? [y/n]". Y assumes the db and users are already configured and that you just want to add an additional process watching it. This is more targeting additional receivers or an upgrade to existing instance.

For the config error, cd /opt/minerva/bin and then execute python receiver.py. I'll add a bit more logic to finding it here shortly.

[samiux]

What can I set to (1) "Enter IP address of receiver to send to:" and (2) "Enter destination port to send to:" ?

Webserver port is 9443
MongoDB port is 27017
Receiver pcap port is 10009
Receiver listen on port 10008

Setting up the agent

Enter name of sensor: sensor
Enter full pathname of sensor certificate (One will be created if it doesn't exist): [/var/lib/minerva/agent/agent.pem] /var/lib/minerva/agent/agent.pem
Enter full pathname of sensor private key for the certificate above: /var/lib/minerva/agent/private.pem
Enter full pathname of log file to send in: /var/log/minerva
Enter alert type of log file: (suricata_eve, snort_alert): suricata_eve
Enter full pathname of position file: /var/log/suricata
Do you want to add more log files? [y/n] n
Enter full pathname of where to save server cert: [/var/lib/minerva/agent/server.pem] /var/lib/minerva/agent/server.pem
Enter IP address of receiver to send to: 192.168.20.251
Enter destination port to send to:

[rc1405]

Setting up the event receiver

Enter IP Address to listen on: 192.168.20.251
Enter port to listen on: 5688

The destination for the agent is the same information that you entered at the receiver setup above

[samiux]

When I start the browser and point to https://192.168.20.251:9443, the following error displayed.

Unrecoverable error in the server.
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/cherrypy/_cprequest.py", line 589, in run
self.respond(pi)
File "/usr/local/lib/python2.7/dist-packages/cherrypy/_cprequest.py", line 690, in respond
self.handle_error()
File "/usr/local/lib/python2.7/dist-packages/cherrypy/_cprequest.py", line 767, in handle_error
self.error_response()
File "/opt/minerva/bin/webserver.py", line 877, in handleError
cherrypy.response.body = [open(os.path.join(os.getcwd(), 'static/html/500.html'),'r').read()]
IOError: [Errno 2] No such file or directory: '/static/html/500.html'

[rc1405]

cp -r Minerva/bin/static/html /opt/minerva/bin/static. Where Minerva is the cloned directory. I'll have to push a fix to the install to correct it.

[rc1405]

Actually the whole static directory should have been copied. What directory are yourunning that out of?

[rc1405]

What os and version?

[samiux]

Ubuntu Server 64-bit 14.04.3

I copied sudo cp -pR ~/Minerva/bin/static/* /opt/minerva/bin/

I will reinstall it again.

[samiux]

Meanwhile, the system will be hang when it is reboot.

[rc1405]

commit 061918b should clean up the issues. I changed how it was determining location of folders by the scripts location instead of your cwd.

[samiux]

I cannot reinstall as there is no more "crypt" at https://pypi.python.org/simple/crypt/

[rc1405]

requirement removed, no longer needed

[samiux]

I set the directory to "/opt/minerva". However, the installation will copy to "/opt/minerva/minerva"

[samiux]

I finally setup Minerva. The settings are as the following. I am sure that Suricata is working fine and eve.json has entries. I have one network interface and the IP address is 192.168.20.251 and it set to offloading. However, there is no sensor on the web interface of Minerva and there is no alert too.

By the way, can I just press "enter" to accept the default settings when configuring Minerva?

Cleaning up...
Please choose an install method:
1. StandAlone (Server, Agent and Receiver)
2. Server/Receiver
3. WebServer only
4. Receiver Only
5. Agent Only
6. Database Only

1
Enter installation Directory: /opt/
Connect to existing database? [y/n] n
Setting up the Database
Please enter database ip: [127.0.0.1] 127.0.0.1
Please enter database port: [27017] 27017
_IF AUTHENTICATION METHOD IS CHOSEN, IT MUST BE SETUP PRIOR TO RUNNING SETUP_*
Use db authentication? Y/N [N] N
Enter number of days to keep alerts: 90
Enter number of days to keep flow data: 90
Enter number of minutes until each console session times out: 10
Enter username of admin user to create: samiux
Enter password:
Password:
Re-enter password:
Password:
Setting up the web server

Enter hostname for webserver: smp
Enter IP Address to bind to: 192.168.20.251
Enter Port for webserver to run on: [443] 443
Enter number of threads to respond to web requests: [8] 8
Enter full path of webcertificate to use (Will create one if none exists) [/var/lib/minerva/webserver/server.pem] /var/lib/minerva/webserver/server.pem
Enter full path of web server's private key: /var/lib/minerva/webserver/private.pem
Enter # of logon attempts before user is locked out: [3] 3
Enter minimum length for user passwords: [8] 8
Enter # of days a password is valid before needed to be changed: [90] 999999
Enter # of seconds to wait on a pcap request: [300] 300
Enter # of results to show in the console at a time: [5000](15000 max) 5000
Enter minimum # of lower case letters in a password: [2] 2
Enter minimum # of upper case letters in a password: [2] 2
Enter minimum # of numbers in a password: [2] 2
Enter minimum # of special characters in a password: [2] 2
Setting up the event receiver

Enter IP Address to listen on: 192.168.20.251
Enter port to listen on: 10008
Do you want to add more ports? [y/n] n
How many threads do you want to process events? 8
Do you want to add another IP? [y/n] n
Enter number of seconds to timeout on a single receive thread: [20] 20
Enter number of processes you want to insert alerts: [4] 4
Enter max number of events to insert at a time: [500] 500
Enter max seconds to wait before inserting events: [20] 20
Enter full path of certificate to use (will create in this lcoation if it doenst exist): [/var/lib/minerva/receiver/server.pem] /var/lib/minerva/receiver/server.pem
Enter full path of private key to use w/ the certificate above: /var/lib/minerva/receiver/private.pem
Enter IP Address to listen for pcap requests from the webserver: 192.168.20.251
Enter Port of Receiver to list for pcap requests for: [10009] 10009
Enter number of threads to process pcap requests: [4] 4
Enter number of seconds to wait for a pcap request, Should be the same as webserver value: [300] 300
Setting up the agent

Enter name of sensor: minerva
Enter full pathname of sensor certificate (One will be created if it doesn't exist): [/var/lib/minerva/agent/agent.pem] /var/lib/minerva/agent/agent.pem
Enter full pathname of sensor private key for the certificate above: /var/lib/minerva/agent/private.pem
Enter full pathname of log file to send in: /var/log/minerva
Enter alert type of log file: (suricata_eve, snort_alert): suricata_eve
Enter full pathname of position file: /var/log/suricata/eve.json
Do you want to add more log files? [y/n] n
Enter full pathname of where to save server cert: [/var/lib/minerva/agent/server.pem] /var/lib/minerva/agent/server.pem
Enter IP address of receiver to send to: 192.168.20.251
Enter destination port to send to: 10008
Enter max # of events to send at once: [500] 500
Enter max # of seconds to wait to send events (Will send earlier if max events is reached): [10] 10
Configuring Agent PCAP Requests
Enter max # of packets to return per request: [10000] 10000
Enter Max size(mb) of pcap files to return per reqeust: [20] 20
Enter Max # of pcap files to search through per request: [10] 10
Enter max time in seconds past an event to grab packets for: [300] 300
Enter prefix for pcap files: []
Enter suffix for pcap files: [.pcap] .pcap
Enter complete path to base directory for pcap files: /var/log/suricata/pcap
Enter complete path of temp storage for pcap requests: /var/log/suricata/pcap/temp
Enter ip address to listen for requests on: 192.168.20.251
Enter port to listen for requests on: [10009] 10009
Enter number of threads to process requests: [4] 4

[samiux]

sudo netstat -anpt

samiux@smp:/var/log$ sudo netstat -anpt
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 127.0.0.1:27017 0.0.0.0:* LISTEN 966/mongod
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 943/sshd
tcp 0 0 192.168.20.251:10008 0.0.0.0:* LISTEN 1119/python
tcp 0 0 192.168.20.251:10009 0.0.0.0:* LISTEN 1112/python
tcp 0 0 192.168.20.251:443 0.0.0.0:* LISTEN 1015/python
tcp 0 0 127.0.0.1:27017 127.0.0.1:43040 ESTABLISHED 966/mongod
tcp 0 0 127.0.0.1:43044 127.0.0.1:27017 ESTABLISHED 1119/python
tcp 0 0 127.0.0.1:27017 127.0.0.1:43038 ESTABLISHED 966/mongod
tcp 0 168 192.168.20.251:22 192.168.20.190:65338 ESTABLISHED 1209/sshd: samiux [
tcp 0 0 127.0.0.1:27017 127.0.0.1:43039 ESTABLISHED 966/mongod
tcp 0 0 127.0.0.1:43038 127.0.0.1:27017 ESTABLISHED 1112/python
tcp 0 0 127.0.0.1:43040 127.0.0.1:27017 ESTABLISHED 1115/python
tcp 0 0 127.0.0.1:27017 127.0.0.1:43041 ESTABLISHED 966/mongod
tcp 0 0 127.0.0.1:43041 127.0.0.1:27017 ESTABLISHED 1114/python
tcp 0 0 127.0.0.1:27017 127.0.0.1:43042 ESTABLISHED 966/mongod
tcp 0 0 127.0.0.1:27017 127.0.0.1:43043 ESTABLISHED 966/mongod
tcp 0 0 127.0.0.1:43043 127.0.0.1:27017 ESTABLISHED 1116/python
tcp 0 0 127.0.0.1:43039 127.0.0.1:27017 ESTABLISHED 1113/python
tcp 0 0 127.0.0.1:27017 127.0.0.1:43044 ESTABLISHED 966/mongod
tcp 0 0 127.0.0.1:43042 127.0.0.1:27017 ESTABLISHED 1112/python
tcp6 0 0 :::22 :::* LISTEN 943/sshd

[rc1405]

any prompt during setup with brackets at the end is the default, you can hit enter and it'll choose what is inside the brackets. The exception is [y/n] which indicates the options available. The sensor will get inserted when it first tries to send an event. At that point you will need to accept it on the sensor web page.

There is a conflict of ports in the setup. No need to rerun the setup, just change it in the yaml config file. One of the 10009 ports needs to be changed. I'll change it in the setup so there isn't a conflict in a standalone configuration.

What version of suricata?

[rc1405]

I found the problem, Pull down the latest push and cp ~/Minerva/bin/agent.py /opt/minerva/bin/. No need to run setup again.

[rc1405]

bb8a0a9 should make the install process a whole lot smoother now. I added more checks and validations and added additional descriptions based on this open issue.

[samiux]

I cloned the latest version. However, I am confusing when setting it up. Does the port (no default value) are the same port when I asked to enter? I have no entries and no sensor at the moment.

samiux@smp:~/Minerva/bin$ sudo python setup.py install

---

•                Minerva-IDS Setup                   *
  

•                                                    *
  

• Please choose an install method: *
• 1. StandAlone (Server, Agent and Receiver) *
• 2. Server/Receiver *
• 3. WebServer only *
• 4. Receiver Only *
• 5. Agent Only *
• 6. Database Only *

---

*> 1
Enter installation Directory: [/opt/minerva] /opt/minerva

---

• Only use if you have an already configured minerva database in mongodb *

  ---

  Connect to existing minerva database? [y/n] n

  ---

•           Setting up the Database                  *
  

  ---

  Please enter database ip: [127.0.0.1] 127.0.0.1
  Please enter database port: [27017] 27017
  _IF AUTHENTICATION METHOD IS CHOSEN, IT MUST BE SETUP PRIOR TO RUNNING SETUP_*
  Use db authentication? Y/N [N] N
  Database already exists, do you want to keep it? [N]N
  Enter number of days to keep alerts: 90
  Enter number of days to keep flow data: 90
  Enter number of minutes until each console session times out: 10
  Enter username of admin user to create: samiux
  Enter password:
  Password:
  Re-enter password:
  Password:

  ---

•           Setting up the web server                *
  

  ---

  Enter hostname for webserver: minerva
  Enter IP Address to bind to: 192.168.20.251
  Enter Port for webserver to run on: [443] 443
  Enter number of threads to respond to web requests: [8] 8
  Enter full path of webcertificate to use (Will create one if none exists) [/var/lib/minerva/webserver/server.pem] /var/lib/minerva/webserver/server.pem
  Enter full path of web server's private key: /var/lib/minerva/webserver/private.pem
  Enter # of logon attempts before user is locked out: [3] 3
  Enter minimum length for user passwords: [8] 8
  Enter # of days a password is valid before needed to be changed: [90] 9999999999
  Enter # of seconds to wait on a pcap request: [300] 300
  Enter # of results to show in the console at a time: [5000](15000 max) 5000
  Enter minimum # of lower case letters in a password: [2] 0
  Enter minimum # of upper case letters in a password: [2] 0
  Enter minimum # of numbers in a password: [2] 2
  Enter minimum # of special characters in a password: [2] 0

  ---

•          Setting up the event receiver             *
  

  ---

  ---

• The next IP and Port is what will listen for events *
• This is the combination that will be required for *
• setting up Agent Forwarders. *

---

Enter IP Address to listen on: 192.168.20.251
Enter port to listen on: 10008 <------------------------------------ Does it require the same?
Do you want to add more ports? [y/n] n
How many threads do you want to process events? 8
Do you want to add another IP? [y/n] n
Enter number of seconds to timeout on a single receive thread: [20] 20
Enter number of processes you want to insert alerts: [4] 4
Enter max number of events to insert at a time: [500] 500
Enter max seconds to wait before inserting events: [20] 20
Enter full path of certificate to use (will create in this lcoation if it doenst exist): [/var/lib/minerva/receiver/server.pem] /var/lib/minerva/receiver/server.pem
Enter full path of private key to use w/ the certificate above: /var/lib/minerva/receiver/private.pem

---

• The next IP/port will be used to process pcap requests *
• from the webserver. This will only be used for *
• webserver communications *

---

Enter IP Address to listen for pcap requests from the webserver: 192.168.20.251
Enter Port of Receiver to list for pcap requests for: [10009] 10009
Enter number of threads to process pcap requests: [4] 4
Enter number of seconds to wait for a pcap request, Should be the same as webserver value: [300] 300

---

•              Setting up the agent                  *
  

  ---

  Enter name of sensor: minerva
  Enter full pathname of sensor certificate (One will be created if it doesn't exist): [/var/lib/minerva/agent/agent.pem] /var/lib/minerva/agent/agent.pem
  Enter full pathname of sensor private key for the certificate above: /var/lib/minerva/agent/private.pem
  Enter full pathname of log file to send in: /var/log/minerva/minerva.log
  File /var/log/minerva/minerva.log does not exist, add it anyways? [y/n] y
  Enter alert type of log file: (suricata_eve, snort_alert): suricata_eve
  Enter full pathname of position file: /var/log/suricata/eve.json
  Do you want to add more log files? [y/n] n
  Enter full pathname of where to save server cert: [/var/lib/minerva/agent/server.pem] /var/lib/minerva/agent/server.pem

  ---

• The Receiver IP And Port is where events will be forwarded to *

  ---

  Enter IP address of receiver to send to: 192.168.20.251
  Enter destination port to send to: 10008 <------------------------------------ Does it require the same?
  Enter max # of events to send at once: [500] 500
  Enter max # of seconds to wait to send events (Will send earlier if max events is reached): [10] 10

  ---

•      Configuring Agent PCAP Requests               *
  

  ---

  Enter max # of packets to return per request: [10000] 10000
  Enter Max size(mb) of pcap files to return per reqeust: [20] 20
  Enter Max # of pcap files to search through per request: [10] 10
  Enter max time in seconds past an event to grab packets for: [300] 300
  Enter prefix for pcap files: []
  Enter suffix for pcap files: [.pcap] .pcap
  Enter complete path to base directory for pcap files: /var/log/minerva/pcap
  Enter complete path of temp storage for pcap requests: /var/log/minerva/pcap/temp

  ---

• This next IP and port is the what the agent will listen to *
• for pcap requests from the receiver *

---

Enter ip address to listen for requests on: 192.168.20.251
Enter port to listen for requests on: [10010] 10010
Enter number of threads to process requests: [4] 4
samiux@smp:~/Minerva/bin$

[rc1405]

You are correct. 192.168.20.251 port 10008 will show up twice. Once to receiver data and the second as a destination for the agent to send it.

[samiux]

However, I do not have entries in the web interface. Meanwhile, the netstat does not show port 10009. port 10008 and 10010 have no connection.

[rc1405]

Enter full pathname of log file to send in: /var/log/minerva/minerva.log
File /var/log/minerva/minerva.log does not exist, add it anyways? [y/n] y this should be /var/log/suricata/eve.json
Enter alert type of log file: (suricata_eve, snort_alert): suricata_eve
Enter full pathname of position file: /var/log/suricata/eve.json this file is how the agent will keep track of where it is at in the log from above. Input could be something like /var/log/eve.pos
Do you want to add more log files? [y/n] n

[rc1405]

minerva_setup.txt
minerva_start.txt

I ran through on Ubuntu LTS 14.04. See above for reference.

[rc1405]

Can you upload your minerva.yaml file from /opt/minerva/etc?

[samiux]

I can get the sensor on the web interface but it still has no traffic. Moreover, the port 10009 is missing.

Webserver:
db:
url: 127.0.0.1
port: 27017
useAuth: False
AuthType:
username:
password:
PW_Mechanism:
auth_cert:
auth_ca:
SESSION_KEY: "bb251f97554f4cf0a7518682cfc818fc"
web:
hostname: minerva
bindIp: 192.168.20.251
port: 443
threads: 8
pcap_timeout: 300
certs:
webserver_cert: /var/lib/minerva/webserver/server.pem
webserver_key: /var/lib/minerva/webserver/private.pem
session_timeout: 10
password_requirements:
password_tries: 3
password_min_length: 8
password_max_age: 90
lower_count: 0
upper_count: 0
digit_count: 2
special_count: 0
events:
maxResults: 5000
max_age: 90
flow_max_age: 90
Event_Receiver:
listen_ip:
192.168.20.251:
ports:
- 10008
receive_threads: 8
listener_timeout: 20
insertion_threads: 4
insertion_batch: 500
insertion_wait: 20
certs:
server_cert: /var/lib/minerva/receiver/server.pem
private_key: /var/lib/minerva/receiver/private.pem
PCAP:
ip: 192.168.20.251
port: 10009
threads: 4
timeout: 300
Agent_forwarder:
sensor_name: minerva
client_cert: /var/lib/minerva/agent/agent.pem

client_private: /var/lib/minerva/agent/private.pem
logfiles:
/var/log/suricata/eve.json:
type: suricata_eve
position_file: /var/log/minerva/eve.pos
target_addr:
server_cert: /var/lib/minerva/agent/server.pem
destination: 192.168.20.251
port: 10008
send_batch: 500
send_wait: 10
pcap:
max_packets: 10000
max_size: 20
max_files: 10
thres_time: 300
prefix:
suffix: .pcap
pcap_directory: /var/log/minerva/pcap
temp_directory: /var/log/minerva/pcap/temp
listener:
ip: 192.168.20.251
port: 10010
threads: 4

[samiux]

All of sudden, I see there are some entries when I boot it up again.

[rc1405]

ok, good so you have the sensor in the webpage, which means it is sending something. Can you run the commands below?

mongo
use minerva
db.alerts.count()
db.flow.count()

What version of suricata are you using?

[samiux]

I am using the latest version from github (2.1dev) suricata. It is looking like that it is a very long time to get the pcap from the web interface. Meanwhile, the latest alert is at the last of the list. If it can display on the first line should be better.

samiux@smp:/opt/minerva/etc$ mongo
MongoDB shell version: 3.0.7
connecting to: test
Welcome to the MongoDB shell.
For interactive help, type "help".
For more comprehensive documentation, see
http://docs.mongodb.org/
Questions? Try the support group
http://groups.google.com/group/mongodb-user
Server has startup warnings:
2015-11-15T22:46:49.570+0800 I CONTROL [initandlisten]
2015-11-15T22:46:49.570+0800 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
2015-11-15T22:46:49.570+0800 I CONTROL [initandlisten] ** We suggest setting it to 'never'
2015-11-15T22:46:49.570+0800 I CONTROL [initandlisten]
2015-11-15T22:46:49.570+0800 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
2015-11-15T22:46:49.570+0800 I CONTROL [initandlisten] ** We suggest setting it to 'never'
2015-11-15T22:46:49.570+0800 I CONTROL [initandlisten]

use minerva
switched to db minerva
db.alerts.count()
1874
db.flow.count()
0

[rc1405]

ok, now try
mongo
use minerva
db.alerts.findOne()

You can filter out all of the data, I just need to see the fields and then the value for MINERVA_STATUS

[samiux]

It cannot display the latest alerts so far but the counter at the bottom right hand side is increasing.

db.alerts.findOne()
{
"_id" : ObjectId("5647b064b3de2904474093dd"),
"src_port" : 41653,
"event_type" : "alert",
"proto" : "TCP",
"orig_timestamp" : "2015-11-15T05:49:13.276655+0800",
"timestamp" : ISODate("2015-11-14T21:49:13.276Z"),
"in_iface" : "eth0",
"alert" : {
"category" : "",
"severity" : 3,
"rev" : 1,
"gid" : 1,
"signature" : "SURICATA TCPv4 invalid checksum",
"action" : "allowed",
"signature_id" : 2200074
},
"src_ip" : "192.168.20.190",
"logType" : "alert",
"epoch" : 1447537753,
"ssh" : {
"client" : {
"proto_version" : "2.0",
"software_version" : "OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.3"
},
"server" : {
"proto_version" : "2.0",
"software_version" : "OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.3"
}
},
"flow_id" : 41650016,
"dest_port" : 22,
"sensor" : "minerva",
"dest_ip" : "192.168.20.251",
"MINERVA_STATUS" : "OPEN"
}

[samiux]

I make a mistake. It shows the current alerts but they are at the bottom of the list.

[rc1405]

Ok great. Then you are getting events? There is no current refresh mechanism, so it has to be manually refreshed at the moment. The sorting of events goes by alert severity and then by age, oldest first.

[samiux]

The performance is quite poor on my virtualbox vm. pcap seems do not work.

[rc1405]

what resources do you have allocated to it? For the PCAP what is the error message that is being displayed?

[samiux]

I assigned 4 CPU and 4GB RAM to VirtualBox VM. There is no error display before I quit it. I am waiting for a very long time.

[rc1405]

Pop up blocker? Change pcap_timeout: to a lower value if you're waiting too long, but you should be getting some sort of response, even on timeout.

[rc1405]

For performance can you run:
mongo
use minerva
db.alerts.getIndexes()
db.flow.getIndexes()

[rc1405]

what are you using for PCAP collection?

Right now I have a standalone setup with the latest Suricata 2.1 beta on Ubuntu 14.04 hosted on an AWS micro instance. What is the lag time that you're experiencing w/ pulling alerts? Rough estimate of seconds click to events?

[samiux]

db.alerts.getIndexes()
[
{
"v" : 1,
"key" : {
"_id" : 1
},
"name" : "id",
"ns" : "minerva.alerts"
},
{
"v" : 1,
"key" : {
"MINERVA_STATUS" : 1,
"epoch" : 1,
"alert.severity" : -1,
"src_ip" : 1,
"src_port" : 1,
"dest_ip" : 1,
"dest_port" : 1,
"proto" : 1,
"alert.signature" : 1,
"alert.category" : 1,
"alert.signature_id" : 1,
"alert.rev" : 1,
"alert.gid" : 1,
"sensor" : 1
},
"name" : "alert-search-index",
"ns" : "minerva.alerts"
},
{
"v" : 1,
"key" : {
"timestamp" : 1
},
"name" : "timestamp_1",
"ns" : "minerva.alerts",
"expireAfterSeconds" : 7776000
}

]

db.flow.getIndexes()
[
{
"v" : 1,
"key" : {
"_id" : 1
},
"name" : "id",
"ns" : "minerva.flow"
},
{
"v" : 1,
"key" : {
"src_ip" : 1,
"src_port" : 1,
"dest_ip" : 1,
"dest_port" : 1,
"proto" : 1,
"netflow.start_epoch" : 1,
"netflow.stop_epoch" : 1,
"sensor" : 1
},
"name" : "src_ip_1_src_port_1_dest_ip_1_dest_port_1_proto_1_netflow.start_epoch_1_netflow.stop_epoch_1_sensor_1",
"ns" : "minerva.flow"
},
{
"v" : 1,
"key" : {
"timestamp" : 1
},
"name" : "timestamp_1",
"ns" : "minerva.flow",
"expireAfterSeconds" : 7776000
}
]

[rc1405]

I found a pymongo issue with multi-threading on the distribution. I'll need to restructure receiver.py a bit to accommodate. I'll get a fix pushed out later today

[rc1405]

Several updates pushed. Two relating to performance, one was a bug with the users page.

[rc1405]

Closing issue as initial configuration has been worked through. If you continue to see performance issues open up a new thread with any details. Thanks for the input!