jmpsec / osctrl Goto Github PK

The handler function that gets called after a user logs in always redirects users to the system-wide defaultEnv on the Osctrl server.

Unfortunately, if the user does not have permissions to view/access the defaultEnv, they are sent to a blank page. This happens both if a user has access to no environments or any number of other, non-defaultEnvs.

If they do not have access to any environments, they should have an error or message printed.

If they have access to $n other environments (none of which are the Osctrl server's defaultEnv), they should be redirected to https://IP_ADDR:8443/environment/ENV_NAME_THEY_CAN_ACCESS/active so they can view their environments.

Ability to display data security estimation per enrolled node within a dashboard in osctrl-admin

Problem

Currently osctrl only support environments and platforms to group nodes. Whereas environments can be created and nodes are enrolled to them, platform comes from the enrolled node and can not be changed. There is a need to group machines so different configuration can be distributed, or query packs, and also on-demand queries / file carves can be run against.

Solution

Create tags so enrolled nodes can be associated. One node can have multiple tags.

Insecure server validation when establishing TLS connections may lead to MITM:

https://github.com/jmpsec/osctrl/blob/master/utils/http-utils.go

InsecureSkipVerify should not be used.

make docker_all fails with:

writing new private key to 'docker/certs/osctrl.key'
docker/certs/osctrl.key: No such file or directory
140048378546064:error:02001002:system library:fopen:No such file or directory:bss_file.c:402:fopen('docker/certs/osctrl.key','w')
140048378546064:error:20074002:BIO routines:FILE_CTRL:system lib:bss_file.c:404:
make: *** [docker_all] Error 1

Attempting to manually start:

docker-compose up
...
ERROR: Cannot locate specified Dockerfile: docker/tls/Dockerfile

docker-compose.yml is now located inside the docker directory, it seems the path to the Dockerfile should be just tls/Dockerfile.

After editing paths, docker-compose up fails with:

ERROR: Service 'osctrl-tls' failed to build: COPY failed: stat /var/lib/docker/tmp/docker-builder567593261/cmd/tls/scripts: no such file or directory

This appears to be because it's looking for the ~/osctrl/cmd/tls/scripts directory that is one directory up from the docker-compose.yml.

Steps to reproduce

In the admin UI, viewing the nodes table:
1 - Input some text to filter nodes
2 - Use the select all elements of the table
3 - Apply an action to selected nodes (delete, query...)

Expected result

Only the nodes filtered should get that action applied.

Actual result

All the nodes of the table get the action applied.

Issue

Using osctrl v0.1.5, using auth = "none" in the service configuration (admin.json) for osctrl-admin, when attempting to browse the admin UI, server throws a panic and browser shows a nginx 502 Bad Gateway error.

Steps to reproduce

1 - Deploy locally, using vagrant up for example,
2 - Change auth = "db" for auth = "none" in /opt/osctrl/config/admin.json
3 - Run make install to rebuild and deploy.
4 - Crash, panic and nginx 502 Bad Gateway. Also this stacktrace:

Aug 27 22:56:12 osctrl-Dev osctrl-admin[21166]: server.go:3012: http: panic serving 127.0.0.1:41128: interface conversion: interface {} is nil, not main.contextValue
Aug 27 22:56:12 osctrl-Dev osctrl-admin[21166]: goroutine 690 [running]:
Aug 27 22:56:12 osctrl-Dev osctrl-admin[21166]: net/http.(*conn).serve.func1(0xc000083cc0)
Aug 27 22:56:12 osctrl-Dev osctrl-admin[21166]:         /usr/local/go/src/net/http/server.go:1769 +0x139
Aug 27 22:56:12 osctrl-Dev osctrl-admin[21166]: panic(0xa3a900, 0xc000341980)
Aug 27 22:56:12 osctrl-Dev osctrl-admin[21166]:         /usr/local/go/src/runtime/panic.go:522 +0x1b5
Aug 27 22:56:12 osctrl-Dev osctrl-admin[21166]: main.environmentHandler(0xbcf240, 0xc0001bc700, 0xc000176400)
Aug 27 22:56:12 osctrl-Dev osctrl-admin[21166]:         /vagrant/cmd/admin/handlers-get.go:121 +0xec6
Aug 27 22:56:12 osctrl-Dev osctrl-admin[21166]: net/http.HandlerFunc.ServeHTTP(0xb01fd0, 0xbcf240, 0xc0001bc700, 0xc000176400)
Aug 27 22:56:12 osctrl-Dev osctrl-admin[21166]:         /usr/local/go/src/net/http/server.go:1995 +0x44
Aug 27 22:56:12 osctrl-Dev osctrl-admin[21166]: main.handlerAuthCheck.func1(0xbcf240, 0xc0001bc700, 0xc000176400)
Aug 27 22:56:12 osctrl-Dev osctrl-admin[21166]:         /vagrant/cmd/admin/auth.go:17 +0xc2
Aug 27 22:56:12 osctrl-Dev osctrl-admin[21166]: net/http.HandlerFunc.ServeHTTP(0xc0003c3dc0, 0xbcf240, 0xc0001bc700, 0xc000176400)
Aug 27 22:56:12 osctrl-Dev osctrl-admin[21166]:         /usr/local/go/src/net/http/server.go:1995 +0x44
Aug 27 22:56:12 osctrl-Dev osctrl-admin[21166]: github.com/gorilla/mux.(*Router).ServeHTTP(0xc0000de9c0, 0xbcf240, 0xc0001bc700, 0xc000176200)
Aug 27 22:56:12 osctrl-Dev osctrl-admin[21166]:         /home/vagrant/go/pkg/mod/github.com/gorilla/[email protected]/mux.go:212 +0xe3
Aug 27 22:56:12 osctrl-Dev osctrl-admin[21166]: net/http.serverHandler.ServeHTTP(0xc0004c3c70, 0xbcf240, 0xc0001bc700, 0xc000176200)
Aug 27 22:56:12 osctrl-Dev osctrl-admin[21166]:         /usr/local/go/src/net/http/server.go:2774 +0xa8
Aug 27 22:56:12 osctrl-Dev osctrl-admin[21166]: net/http.(*conn).serve(0xc000083cc0, 0xbd0b40, 0xc0001790c0)
Aug 27 22:56:12 osctrl-Dev osctrl-admin[21166]:         /usr/local/go/src/net/http/server.go:1878 +0x851
Aug 27 22:56:12 osctrl-Dev osctrl-admin[21166]: created by net/http.(*Server).Serve
Aug 27 22:56:12 osctrl-Dev osctrl-admin[21166]:         /usr/local/go/src/net/http/server.go:2884 +0x2f4

Details

Currently osctrl does not have a CONTRIBUTING.md file that outlines details for contributions. Let's create one!

Overview

The utility provision.sh allows to deploy all components of osctrl in a standalone linux system. It would be useful to expand that to cloud providers and deploy using terraform.
First just in EC2, same as a standard deployment but further improvements could be deploy each component separately, for example if using AWS:

• osctrl-tls, osctrl-admin and osctrl-api -> lambda functions.
• osctrl-db -> RDS
• osctrl-nginx -> EBS/ABS

Issue

Using osctrl v0.1.4, when running an on-demand query or carving a file that will have as target all environments or platforms, the target isn't set and there are no targets selected.

Steps to reproduce

1 - Login to the admin console
2 - Go to Run Query or Carve File
3 - Select all environments or all platforms as target, and submit it with any query or file.

Expected result: The on-demand query or carve will run for all environments or platforms.
Actual result: No target is selected, therefore the query or carve fails to get delivered to any target.

Display enrolled date in the main table per environment, the same way as Last Seen is displayed.

One of the most common use cases of osctrl is to forward logs to SIEM (Splunk, Graylog, ELK) solutions or long term storage such as Kafka or s3. Since it allows to store a local copy of those logs in the backend it may be helpful having a process that expires or deletes those entries when they are older than a specific threshold: 1 Day, 1 week, 1 month...

Overview

Currently there is only a docker-compose.yml to run osctrl in a docker environment. It would be good to have an actual deploy for k8s, so it can be used within kubernetes.

Using kompose the docker-compose.yml can be converted directly, so I will start exploring that option first.

Overview

When a new node is enrolled, tags are applied based on environment (the tags are created when an environment is created). Tags should be created based on platform when a node is enrolled. Perhaps even more granular tags.

Issue

Using osctrl v0.1.8, using logging = "graylog" in the service configuration (tls.json) for osctrl-tls, using Graylog's HTTP GELF receiver, messages that are group in JSON dictionaries are not accepted.

Steps to reproduce

1 - Set logging to graylog, logging = "graylog" in tls.json
2 - Create graylog HTTP GELF input.
3 - Generate logs

GELF seems to expect individual log statements in JSON {GELF_OBJ}. The osctrl graylog plugin is grouping logs together, and the final document being sent to graylog is [{GELF_OBJ},{GELF_OBJ},{GELF_OBJ},...], which evidently is not cool. This was validated via test GELF messages and tcpdump to view the raw texts to graylog - it dumps them silently otherwise.

To resolve this, I removed the grouping components of the plugin, and send a request for each log.

I haven't used Graylog much, so maybe I've missed something, but the message won't even make it to an index - it gets dropped.

Data security can be measured in each enrolled node by extracting information with queries. With a query pack this data can be aggregated and used for metrics.

Hi Javier,
I have deployed osctrl using, make docker_all. I have created a new environment (and copied the certificates of the dev environment) using the admin interface. I was attempting to enroll an Ubuntu 18.04.3 LTS using the quick-enroll method. On running the enrollment link using curl, agent (enroll.sh) does not download and install. I have attached the logs generated by each container as well as the curl on the remote machine. I would like to add that the system is on the local network and accessible as the logs show. Please help me with this.
osctrl-enroll-file-download-fail-issue.txt

I would love to see some docs on API endpoints. At a minimum a list of the urls, a list of parameters and a summary of the responses would be great. Right now it's pretty difficult to try and use the api as I don't write go, so digging into the code for the api is 😬

Overview

When visiting Manage Users in the admin UI, some fields are not editable, such as email or full name. It would be nice to allow the edition of those fields, same as the settings are edited.

A malicious XSS payload can be injected in the environment name, inside the function statsRefresh:

https://github.com/jmpsec/osctrl/blob/master/admin/static/js/stats.js

Although exploitation would be hard, is better to follow best practices and consider this as a dangerous vulnerability.

I 'm looking for documentation to install osctrl on ubuntu directly [not using docker]

So, we are looking for better tools then we currently have (Wazuh, works nice.. but its yugh in my opinion) and then i ran into osquery.. Seems nice :) doesnt seem yugh like Wazuh :)

Now i wonder, could osctrl be be used for a small MSSP with multi tenants and alerting? (in some form, or another)
in the end we would also like reporting.. but first thing's first ;-)

Currently an existing environment only allows to edit the hostname using osctrl-cli, it would be better if that can be done from the admin UI.

OSQuery is per definition the ideal tool to gather information from a host. maybe write an extention to allow this information to be used to manage the host as well. this makes OSCTRL in league with tools like SCCM, DesktopCentral, Pulseway etc.

For Windows this means the ability to execute powershell commands and/or modify the registry. for Linux this means modifying config files and/or execute bash commands.

Overview

With tags created by environment and platform, we can run queries based on tags and simplify how those queries are delivered.

Overview

Add the ability to create query packs within osctrl-admin. Those packs can be added to tags so they will be delivered to specific enrolled nodes.

This may be superfluous with #19 coming down the pipeline. Nevertheless, including a configurable link to the actual logging system with the query name included would be convenient. This could be achieved with simple variable substitution in a URL (http://logs.example/search?q=name:${query_name}).

1. Determine if logging is set to something other than DB.
2. Add a button to /query/list that goes directly to a the search URL
3. Add a button or link somewhere on /query/logs/{query_id}

Stretch goal: include some absolute time variables based around the time that the query was initially kicked off so that we can easily get to older queries. Graylog uses ISO 8601 in the URL, so the url template would be: https://graylog.example/streams/000000000000000/search?rangetype=absolute&from=${date_start}&to=${date_plus_1hour}&q=name:${query_name}

Currently osctrl-tls updates the metadata for each node when requests are received, before sending logs to the configured logger (splunk, graylog...). This may interfere in the ability of osctrl-tls to deliver those logs no matter what.

Failures should be logged and metrics emitted but logs must always be delivered.

Problem

When adding a non-admin user to osctrl-admin, and granting access to a specific environment. If the default environment (setting for admin is default_env) is not the granted one, then when the user logs in, there will be a blank page.

Solution

Default environment should fall back to the one that the user has permissions to see.

Currently the permission to run on-demand queries is granted with the scope of all-or-none environments. It would be a good enhancement in RBAC to be able to isolate which environments to query.

According to https://osctrl.net/components/backend/ it wouldn't be much work to support backends other than postgres. I'd love to see mysql support as that's what my database admins support.

Overview

With tags created by environment and platform, we can carve files based on tags and simplify how those carves are delivered.

Overview

When using osctrl-cli to make a change, for example, updating the hostname for an environment, the command runs but there is no confirmation of the success or failure of the operation. It can be a bit confusing.

Overview

The code to enroll new clients automatically makes the UUID uppercase. In certain environments, for example docker, the UUID is extracted and submitted as lowercase, so the function getNodeByUUID can not find the node when trying to update the metadata or log status/result logs. This is the error seen in the logs:

osquery-ubuntu18    | I0710 17:30:03.995153    26 tls.cpp:253] TLS/HTTPS POST request to URI: https://osctrl-nginx/dev/log
osctrl-nginx        | 172.25.0.3 - - [10/Jul/2020:17:30:04 +0000] "POST /dev/log HTTP/1.1" 200 22 "-" "osquery/4.4.0" "-"
osctrl-tls          | dispatch.go:15: error updating metadata getNodeByUUID record not found
osctrl-tls          | dispatch.go:31: error refreshing last result getNodeByUUID record not found

Proposed solution

Every time the UUID is looked up it will be uppercase'd so consistency will be achieved and this will not affect metadata updates or logging ability.

OS security can be measured in each enrolled node by extracting information with queries. With a query pack this data can be aggregated and used for metrics.

Hello,
I am using the latest master branch for deploying osctrl. I am deploying using the dockerize.sh script and face the following issues on deployment.

1. osctrl-api | main.go:163: Error loading config/jwt.json - While parsing config: unexpected end of JSON input
2. osctrl-api exited with code 1

I find that osctrl-api container is in an exited state. After the script runs I see the following.

$ docker ps -a | head
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
540aa20eaa4d osctrl_osquery-ubuntu18 "/bin/sh wait.sh" About an hour ago Up About an hour osquery-ubuntu18
1d14cad30249 nginx:1.13.5 "nginx -g 'daemon of…" About an hour ago Up About an hour 0.0.0.0:443->443/tcp, 80/tcp, 0.0.0.0:8443->8443/tcp osctrl-nginx
d40d37391fbb osctrl_osctrl-tls "/osctrl-tls/bin/osc…" About an hour ago Up About an hour 0.0.0.0:9000->9000/tcp osctrl-tls
11298d44696b osctrl_osctrl-api "/osctrl-api/bin/osc…" About an hour ago Exited (1) About an hour ago osctrl-api
b09756739c1a osctrl_osctrl-admin "/bin/sh /osctrl-adm…" About an hour ago Exited (1) About an hour ago osctrl-admin
85e04607fd8d postgres:10-alpine "docker-entrypoint.s…" About an hour ago Up About an hour 0.0.0.0:5432->5432/tcp osctrl-db

I would like to add that this issue I face with osctrl-admin as well.
osctrl-admin | Created admin user
osctrl-admin | main.go:159: Loading config/admin.json
osctrl-admin | jwt.go:18: Loading config/jwt.json
osctrl-admin | main.go:228: Error loading config/jwt.json - While parsing config: unexpected end of JSON input
osctrl-admin exited with code 1

I find osctrl-api and osctrl-admin in the exited state after the script runs.

I also find these messages in the log.

osctrl-admin | 2020/05/07 01:51:01 Failed to load DB - dial tcp 172.18.0.2:5432: connect: connection refused
osctrl-db | creating subdirectories ... ok
osctrl-db | selecting default max_connections ... 100
osctrl-db | selecting default shared_buffers ... 128MB
osctrl-db | selecting default timezone ... UTC
osctrl-db | selecting dynamic shared memory implementation ... posix
osctrl-db | creating configuration files ... ok
osctrl-db | running bootstrap script ... ok
osctrl-db | performing post-bootstrap initialization ... sh: locale: not found
osctrl-db | 2020-05-07 01:50:52.870 UTC [29] WARNING: no usable system locales were found
osctrl-db | ok
osctrl-db | syncing data to disk ...
osctrl-db | WARNING: enabling "trust" authentication for local connections
osctrl-db | You can change this by editing pg_hba.conf or using the option -A, or
osctrl-db | --auth-local and --auth-host, the next time you run initdb.
osctrl-db | ok
osctrl-db |
osctrl-db | Success. You can now start the database server using:
osctrl-db |
osctrl-db | pg_ctl -D /var/lib/postgresql/data -l logfile start
osctrl-db |
osctrl-db | waiting for server to start....2020-05-07 01:51:01.424 UTC [34] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
osctrl-db | 2020-05-07 01:51:01.568 UTC [35] LOG: database system was shut down at 2020-05-07 01:50:57 UTC
osctrl-db | 2020-05-07 01:51:01.601 UTC [34] LOG: database system is ready to accept connections
osctrl-db | done
osctrl-db | server started
osctrl-db | CREATE DATABASE
osctrl-db |
osctrl-db |
osctrl-db | /usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
osctrl-db |
osctrl-db | waiting for server to shut down....2020-05-07 01:51:02.910 UTC [34] LOG: received fast shutdown request
osctrl-db | 2020-05-07 01:51:03.016 UTC [34] LOG: aborting any active transactions
osctrl-db | 2020-05-07 01:51:03.017 UTC [34] LOG: worker process: logical replication launcher (PID 41) exited with exit code 1
osctrl-db | 2020-05-07 01:51:03.017 UTC [36] LOG: shutting down
osctrl-db | 2020-05-07 01:51:03.513 UTC [34] LOG: database system is shut down
osctrl-db | done
osctrl-db | server stopped
osctrl-db |
osctrl-db | PostgreSQL init process complete; ready for start up.
osctrl-db |
osctrl-db | 2020-05-07 01:51:03.893 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
osctrl-db | 2020-05-07 01:51:03.893 UTC [1] LOG: listening on IPv6 address "::", port 5432
osctrl-db | 2020-05-07 01:51:04.002 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
osctrl-db | 2020-05-07 01:51:04.330 UTC [45] LOG: database system was shut down at 2020-05-07 01:51:03 UTC
osctrl-db | 2020-05-07 01:51:04.360 UTC [46] FATAL: the database system is starting up
osctrl-db | 2020-05-07 01:51:04.456 UTC [1] LOG: database system is ready to accept connections
osctrl-admin | Postgres is unavailable - Waiting...

Please help me with fixing this issue in deployment.

Thank you,
Avinash.

App security can be measured in each enrolled node by extracting information with queries. With a query pack this data can be aggregated and used for metrics.

Overview

Regardless the logging system that we have configured, it would be a great enhancement to keep the last 24 hours of status logs for each node locally, so it would be displayed in the admin UI. There would be two main components to add/modify:

• Always log status logs locally, if the logging configuration is different than db.
• Make sure that only the last 24 hours of logs are kept, so the database is not overloaded.

The 24 hours mark could be configurable and also would be interesting to be able to chose whether to discard data as it expires, or archive it somewhere else.

All osctrl-admin users should be able to change their own password and edit other fields such as fullname or email. The admins of osctrl-admin can edit the fullname and the email but not change password so that would be also be a good addition.

Just for context this is already possible but only using osctrl-cli.

Overview

When enrolling a new OSX node (platform darwin) using the quick-enroll oneliner, if osqueryd is already running, the process is not stopped and there is the following error:

[+] Starting com.facebook.osqueryd
/Library/LaunchDaemons/com.facebook.osqueryd.plist: service already loaded

This seems because the current script uses unload instead of stop && unload for the launchd service.

Thanks @ewaher for finding this!

An unprivileged user can see all environments in osctrl. When trying to browse to one of them an error is thrown because of the lack of permissions but still the environments names are leaked.

Ability to display app security estimation per enrolled node within a dashboard in osctrl-admin

When querying a specific platform, the results dont show any host name. this makes it hard to figure out which system is sending back which result when you are querying multiple systems.

Ability to display OS security estimation per enrolled node within a dashboard in osctrl-admin

Overview

When using osctrl-cli to update the hostname for a given environment, the flags are not updated so they will reflect the old hostname. Flags need to be generated again.

Thanks @ewaher for spotting it!

Pretty easy, just need to switch the base box and change the provider configs a little bit.

# -*- mode: ruby -*-
# vi: set ft=ruby :

VAGRANTFILE_API_VERSION = "2"

IP_ADDRESS = "10.10.10.5"

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
  config.vm.box = "hashicorp/bionic64"
  config.vm.network "private_network", ip: IP_ADDRESS
  # If we want to enroll nodes in the same network
  #config.vm.network "forwarded_port", guest: 443, host: 443
  config.vm.hostname = "osctrl-dev"
  config.ssh.shell = "bash -c 'BASH_ENV=/etc/profile exec bash'"
  config.vm.provision "shell" do |s|
    s.path = "deploy/provision.sh"
    s.args = [
      "--nginx", "--postgres", "-E", "--metrics", "--all-hostname",
      IP_ADDRESS, "--password", "admin"
    ]
    privileged = false
  end
  ["vmware_desktop", "virtualbox", "hyperv"].each do |provider|
    config.vm.provider provider do |v, override|
      v.memory = "1024"
      v.cpus = 1
    end
    config.vm.provider "virtualbox" do |v|
      v.customize ["modifyvm", :id, "--natdnshostresolver1", "on"]
    end
    config.vm.provider :vmware_desktop do |vmware|
      vmware.vmx["ethernet0.pcislotnumber"] = "32"
    end
  end
end

Overview

Currently only one logger is allowed when shipping logs collected by osctrl-tls. It can be Splunk or Graylog, but only one, although db logger is always enabled, to keep a small window of logs in the backend (postgres).

It would be a good feature to enable the forwarding to multiple loggers, so logs can be replicated easily as they arrive.

What happens:

Node Status logs and Result logs do not display until refresh counter triggers

Repro:
Using admin console, click into an environment, select a node.

Click Status Log or Result Logs tab - observe no data appears.

Wait 60 sec (default timer) and tables will fill.

Ability to carve files should be added to osctrl-api to increase the automation capabilities.

Overview

There is one admin setting called inactive_hours that defines how nodes will be classified as active or inactive. For some environments it may be a good idea to show machines as inactive after a short period of time, and the other way around.
Currently the setting applies to the admin component only, it would be great if it can be applied by environment.