abguthrie / goquery Goto Github PK

This is an interesting issue because we need to consider if we prefer code duplication, more complex dependencies, or a refactor which would be quite odd.

Option One: Code Duplication
We copy paste all the code from querySuggest to scheduleSuggest.

This is certainly the easiest to understand but means that when we have to update that code (we will), we'll end up right back here where something is supported in one but not the other.

Option Two: More Complex Dependencies
We have scheduleSuggest call directly into querySuggest.

This is the most expeditious thing to do, and it has no code duplication. The issue here is that now commands can rely on other commands in hard to reason about ways. I can't think of a time where editing querySuggest would cause problems with scheduleSuggest but the fact that a developer might not realize that is sub optimal.

Option Three: A Refactor
We take our query suggestion code and refactor it out into another file or module.

This solves both the above problems which is good, but creates a new question: where would that code live? Where ever it is, it needs access to the hosts module so there might be some reason to put it there. Utils is much lower level and would probably create a circular dependency between utils and hosts. We could create a command utils, but it seems such a function would be useful even outside of commands, along with conceptually not fitting.

I tend to think that Option Two is the best one here but I think that Option Three also needs to be strongly considered. Option One is definitely out though.

Commands like cd must issue a command to the host to verify directory existence which means they end up in the query history. There should be an API that allows you to mark a query and not wanting to appear in the history (though it would be nice if there was a way to request the full history and have them shown)

If you're using goquery in a vacuum (such as using mock server), it may be hard to find what host to connect to.

A new command and new API to pull hosts from the backend that you can connect to would be useful in these circumstances, especially depending on how searching works (can you search by username/hostname for example)

Right now we print all columns in alphabetical order which works decently but isn't very flexible. We should improve the printing code so defined order can be passed in and printed out that way.

This will allow us to reorder .hosts fields in pretty and line print as well as eventually use the ordering that osquery defaults do (for example if path is present, it is almost always first)

If you submit a query with a " then mock server doesn't escape it and it breaks the json it pushes down to the host.

GoQueryAPI should be of type interface.
We can't rely on init function call order, so main should grab the config, and call an initialize function in the API module that creates an internal auth object based on the config. The internal api object will have the CheckHost, ScheduleQuery, etc functions bound to it.

Title, this is broken and needs to be fixed before public release

Commands should be stored in a user's home directory in ~/.goquery/history

We need integration tests

The project needs a licence and I'm not quite sure which we should use.

Likely options are:
MIT
Apache 2.0
GPL 2.0

osquery has two printing modes we need: line and pretty.

Both of these are implemented in C++ so the code needs to be rewritten in go.

Printing Modes Implemented:

• Pretty (#55)
• Line (#42)
• JSON
• CSV (#48)
• TSV

ATC is one of the most powerful functions of osquery and is currently unsupported by goquery.

We should definitely support targeted ATC on hosts. For goquery this will require a new command (which will pair very nicely with the alias system) but mock server will need a few updates as well.

• Implement New API Handler for ATC in Mock Server
• Implement New Command in goquery taking: Name, Query, Path, Duration
• Implement Mock Server Configuration Generation
• Implement New Command in goquery: Refresh Host (to pull new ATC table into suggestions)

I forgot to change it from the defaults of alice/hunter2.

It should be changed to goquery/goquery to make it easier to step into the project and bring more consistency.

New users to goquery should have an help command that lists all commands along with their help strings to help them get acquainted with the shell.

In future it should probably also list loaded aliases.

Right now if a host if offline you're screwed. Ctrl-C should interrupt the wait and return you to the REPL.

Sometimes you typo your password, this shouldn't mean we give up and back out.

We should put the user back at the shell and they can try again if they wish.

Right now CD doesn't handle .. or . properly. This needs to be handled to allow efficient navigation of the filesystem.

We need a docker environment set up that contains at least 3 but preferably 4 hosts.

• One osquery host running ubuntu
• One osquery host running macOS
• One server running our mock server (which also needs a better name because it doesn't mock)
• One server running goquery.

We could get away with a single server for goquery and the mock server but more will sell the point better.

We need to append all queries to the host query history structure for auditing and replay.

You should be able to put in a query name and fetch results again from the server.

We need to use a proper REPL library so we can get things like arrow keys, tab complete, and history

The command should print a simple table containing the host name, uuid, number of queries in history, pending queries, and (maybe) connected status. Connected status will require some sort of persistence for which hosts have been connected to previously.

There are many backends available for osquery and while we support osctrl out of the box (though you need to write your own authentication code), there isn't a lot of support that explains how to do it.

This should be filled out so people can integrate goquery with their backends more easily and with fewer mistakes.

Aliasing allows you to abstract long and complicated goquery commands into your own short aliases. This helps reduce errors and speed up usage.

Aliases should be loaded from the .goquery folder on start up and should be capable of taking parameters as well. Possible usage could look like:

goquery > .alias .all .query select * from {}
goquery > .alias .quit .exit
goquery > .all system_info
Running: select * from system_info
.
.
.
goquery > .quit
Goodbye!

We can assert that aliases may contain no spaces which means parsing the command to process the new alias name and content should be straight forward. It should also be the case that command code is completely separate from the alias code (alias replacement and processing happens before calling into command/ code)

Although it doesn't matter we should try to not get caught up in github repo skimmers that scrape private keys. It'll also provide more incentive for people not to use them.

There isn't a way to do this right now, but if a query is scheduled for a host (via UUID) but that host is not the current host, it will go into the current host (if there is one) or crash otherwise.

We track all queries for each host but we need a way to list them all out so they can have their results fetched again, or just get a history for what's happened.

An osctrl token is valid for over a day by default so we can get away with storing it, checking if it's valid, and if not, then running the authentication flow. This will make it much easier to jump in and out goquery when using the osctrl backend.

Sometimes you just wanna .clear.

Right now in line mode each row is in a different order making it difficult to look at multiple rows and determine what's happening. The JSON printer solves this by sorting by keys first so line should do something similar OR preserve the order that is given to it by the backend.

goserver should be accelerating hosts because that's how real infra would work. Right now we're just hacking around it by setting the checkin time to something absurdly low.

Things need unit tests

Placeholder for linking assets in README

Add a syntax to allow piping the result of query to a file, saving on disk.
For example: .query select * from processes |> ./queries/out.txt --json

This will allow the larger queries to potentially be easily parsed with external tools, and avoid printing a huge amount of data.
This feature could support flags for the output print mode, for example --json and --csv

Overview

When using goquery the default user agent is the one inherited from go, which is Go-http-client/1.1. It would be cool to customize it to be custom with version or something like that.

Great project! kudos!

That's what it is and will make prototypes much simpler

As much as I don't want to, it needs it, preferably with Zstd support as well.

The correct implementation for service protected by SSO or SAML auth is with a reverse proxy. The fact that goserver implements SAML on its own is at best inelegant.

This should be refactored into an nginx reverse proxy that handles saml and will pass along calls when authentication is successful. This would also mean another docker container for this service.

We need a module to hold the config state for debug, print mode, and any other persistent configuration.

Debug flag parsing should happen in main, but be set in config.go

Right now there is code duplication between the REPL code and the commands package. The list of commands should only be stored in the commands package so there is a single source of truth.

Implement a --debug flag to enable verbose logging within command functions

The ability to pull files from remote hosts is one of the most useful features of osquery. goquery should support it, recognize valid paths, track file requests and pull them down to the local host.

UUIDs are long and annoying. All connected hosts should show up as tab complete options with their hostname to make it easier to move between hosts

osctrl by default uses it's own database to handle authentication and is just a series of post requests. This is likely what other smaller deployments use so we should natively support that in goquery.

To switch between hosts you use the .connect command which means that it issues the validation query to pull hosts from the remote machine again.

Ideally this shouldn't happen as we already know what they are.

Right now we only pass UUID and ComputerName. We also need to pass platform, version, etx.

This is due to the way we do command parsing. We need to support escapes, quoting, or ideally both.

Commands shouldn't be created in main.go like they are. They should be in their own file where ideally the map would be stored as well.

This will clean up main to focus on REPL improvements.

Kolide's Fleet is pretty popular and it would make sense to have native support for it if possible.