getgort / gort Goto Github PK

When testing for a possible race condition, I was unable to run tests with --count greater than 1.

Example:

go test --race --count 100 ./config
--- FAIL: TestStandardizeDatabaseConfigNone (0.00s)
    config_test.go:124: 
                Error Trace:    config_test.go:124
                Error:          Should be empty, but was someRandomPassword
                Test:           TestStandardizeDatabaseConfigNone

For clarity, we should describe the rule syntax in Backus–Naur form in the documentation.

The gort bundle list command should include the following flags/behaviors (for parity with cogctl)

(From docker run -it operable/cogctl:alpine-1.0.1 cogctl bundle --help)

Options:
  -e, --enabled   List only enabled bundles
  -d, --disabled  List only disabled bundles
  -v, --verbose   Display additional bundle details
  --help          Show this message and exit.

The gort group remove command should allow the addition of multiple users (for parity with cogctl)

(From docker run -it operable/cogctl:alpine-1.0.1 cogctl group remove --help)

Usage: cogctl group remove [OPTIONS] GROUP USERNAMES...

  Remove one or more users from a group.

Options:
  --yes   Confirm the action without prompting.
  --help  Show this message and exit.

The gort group grant command should allow the addition of multiple roles (for parity with cogctl)

(From docker run -it operable/cogctl:alpine-1.0.1 cogctl group grant --help)

Usage: cogctl group grant [OPTIONS] GROUP ROLES...

  Grant one or more roles to a group.

Options:
  --help  Show this message and exit.

Currently, a bundle command only accepts a single executable argument, so if a command binary has any subcommands, they MUST be subcommands of the bundle command too. This makes building rules harder.

commands:
  gort:
    description: "Bringing the power of the command line to chat."
    executable: "/bin/gort"
    rules:
      - allow
      - with arg[0] == "bundle" must have gort:manage_commands
      - with arg[0] == "group" must have gort:manage_groups
      - with arg[0] == "role" must have gort:manage_roles
      - with arg[0] == "user" must have gort:manage_users

If commands allowed multiple words (as Docker's ENTRYPOINT supports), we could do something like the following (perhaps):

commands:
  bundle:
    description: "Bringing the power of the command line to chat."
    executable: [ "/bin/gort", "bundle" ]
    rules:
      - must have gort:manage_commands

  user:
    description: "Bringing the power of the command line to chat."
    executable: [ "/bin/gort", "bundle" ]
    rules:
      - must have gort:manage_users

  version:
    description: "Bringing the power of the command line to chat."
    executable: [ "/bin/gort", "version" ]
    rules:
      - allow

Example:

One execution of gort user --help returns the entire help output:

$ gort user --help

GortAPP  9:56 AM
Allows you to perform user administration.

Usage:
  gort user [command]

Available Commands:
  create      Create a new user
  delete      Delete an existing user
  info        Retrieve information about an existing user
  list        List all existing users
  update      Update an existing user

Flags:
  -h, --help   help for user

Global Flags:
  -P, --profile string   Gort profile to use

Use "gort user [command] --help" for more information about a command.

While the next execution (with nothing else having changed) returns only the first line:

$ gort group --help

GortAPP  9:57 AM
Allows you to perform group administration.

Current the config automatically reloads when the underlying config file changes. This has led to race conditions and has made testing more difficult.

Instead, config should behave like Prometheus and reload when it receives SIGHUP or request to a /-/reload endpoint.

Uninstalling a bundle of one type (such as "test") disables all versions of that bundle.

$ go run . bundle list -v                        
BUNDLE   ENABLED   INSTALLED VERSIONS    
gort     0.0.1     0.0.1                 
test     0.0.2     0.0.1, 0.0.2, 0.0.3

$ go run . bundle uninstall test 0.0.3
Bundle test 0.0.3 uninstalled.

$go run . bundle list -v             
BUNDLE   ENABLED   INSTALLED VERSIONS   
gort     0.0.1     0.0.1                
test     -         0.0.1, 0.0.2 

$ go run . bundle info test
Name: test
Versions: 0.0.1, 0.0.2
Status: Disabled

The gort role info command should exist (for parity with cogctl)

(From docker run -it operable/cogctl:alpine-1.0.1 cogctl role info --help)

cogctl role info --help 
Usage: cogctl role info [OPTIONS] ROLE

  Show role details

Options:
  --help  Show this message and exit.

The gort bundle uninstall command should include the following flags/behaviors (for parity with cogctl)

(From docker run -it operable/cogctl:alpine-1.0.1 cogctl bundle uninstall --help)

Options:
  -c, --clean         Uninstall all disabled bundle versions
  -x, --incompatible  Uninstall all incompatible versions of the bundle
  -a, --all           Uninstall all versions of the bundle
  --help              Show this message and exit.

Commands often require access to runtime variables, particularly when interacting with external services, ranging from mundane values (like the URL of a downstream resource) to highly sensitive information (like database passwords or access tokens).

Currently, providing these values to a command requires baking them into the bundle’s container image, which greatly limits bundle reusability and introduces a significant security risk.

Dynamic configuration would runtime command configuration by injecting appropriate configuration information into worker containers at runtime in a way that’s secure and (internally) accessible.

An early design doc draft is available here. Feel free to comment.

A rule tester would be helpful for rule development. Maybe something like:

$ gort evaluate --user admin \
  --rule 'with arg[0] == "user" must have gort:manage_users' 
  --command 'gort user list'

Could respond with something like:

{
  "result": true
}

or, alternatively:

{
  "result": false,
  "error": "failed to parse rule"
}

Per title.

Listing bundles seems to work as expected:

$ gort bundle list        
BUNDLE      VERSION     TYPE        STATUS
curl        -           Default     Enabled
echo        -           Default     Enabled
gort        0.0.1       Explicit    Enabled

And getting the info of the explicitly installed bundle works okay:

$ gort bundle info gort
Name: gort
Versions: 0.0.1
Status: Enabled
Enabled Version: 0.0.1
Commands: gort
Permissions: manage_commands, manage_groups, manage_roles, manage_users

But not so with the default curl bundle:

$ gort bundle info curl
Error: No Content

Oops! :)

The gort group add command should allow the addition of multiple users (for parity with cogctl)

(From docker run -it operable/cogctl:alpine-1.0.1 cogctl group add --help)

Usage: cogctl group add [OPTIONS] GROUP USERNAMES...

  Add one or more users to a group.

Options:
  --help  Show this message and exit.

The documentation is incomplete, and what's there is often outdated. It needs to be brought to a minimally usable state by documenting (or updating the documentation of):

1. "Quick start" and "getting started" guides (for both Docker and standard binary execution)
2. Architecture
3. Bootstrapping
4. Server configuration (including installing TLS certs)
5. Commands/bundles (what they are, how to manage them, how to install them, default vs. installed bundles)
6. Permissions and rules
7. User management
8. Audit log (where to find it, how to read it, what it contains)
9. Planned/in-development features

I have two profiles: localhost and matt. The first is an admin user, the second isn't, as follows:

defaults:
    profile: localhost
localhost:
    url: http://localhost:4000
    password: <nope>
    user: admin
    allow_insecure: true
matt:
    url: localhost:4000
    password: <nah>
    user: matthew.titmus
    allow_insecure: true

Doing: gort user list on the command line as admin works, and the controller log provides this output:

INFO   [0956] REST service event                request="GET /v2/users HTTP/1.1" size=159 status=200 user=admin

Switching profiles:

$ go run . profile default matt
Profile 'matt' set to default.

$ go run . profile list        
Name         User              URL                      Default
localhost    admin             http://localhost:4000    
matt         matthew.titmus    http://localhost:4000       *

matt is now the default, however:

go $ run . user list
USERNAME  FULL NAME           EMAIL ADDRESS
admin     Gort Administrator  admin@gort
matthew.titmusmatt                [email protected]

(Note: formatting here is also broken)

Controller log still thinks it's admin

INFO   [1073] REST service event                request="GET /v2/users HTTP/1.1" size=159 status=200 user=admin

Same deal with using the -P flag:

go run . -P matt user list   
USERNAME  FULL NAME           EMAIL ADDRESS
admin     Gort Administrator  admin@gort
matthew.titmusmatt                [email protected]

Controller:

INFO   [1153] REST service event                request="GET /v2/users HTTP/1.1" size=159 status=200 user=admin

EDIT: Interestingly, if I stop and restart the server, the set default (matt) takes effect, but I can't change it back!

INFO   [0076] REST service event                 request="GET /v2/bundles HTTP/1.1" size=13 status=401 user=matthew.titmus

SECOND EDIT: Re-setting the default doesn't work. Perhaps it's just taking the last profile in the file?

Running make test I see the following error on the github.com/clockworksoul/gort/dataaccess/postgres package.

     base_test.go:53: 
         	Error Trace:	base_test.go:53
         	Error:      	Received unexpected error:
         	            	Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
         	Test:       	TestMain
         	Messages:   	failed to start database container

I'm running Docker Desktop 3.3.3 on MacOS (docker engine version 20.10.6).

Currently, the output of executed is just dumped into a fixed-width code box in the chat. We need something better.

Command response formatting should include:

• The ability to embed formatting directives in command output.
• Defaults to fixed width (current format)
• Cog used “Greenbar” templates (here and here)
• We’d more likely allow output to be embedded in command responses
• Maybe we can support “themes” that are defined outside of the command output to handle un-formatted output?

The data access layer uses a singleton representation, which has led to race conditions and increased complexity due to the need for locking.

The DAL should return a new DataAccess instance for each request.

We should support the following profile value overrides in the gort command (copy/pasted from cogctl)

• -c, --config-file PATH Path to an INI-formatted configuration file
• -u, --url TEXT Override API URL root to use, e.g. 'https://127.0.0.0:4000'
• -U, --user TEXT Override account to authenticate against the API
• -P, --password TEXT Override password to authenticate against the API

These were taken from cogctl, which is available by running the following:

$ docker run -it operable/cogctl:alpine-1.0.1 cogctl
Usage: cogctl [OPTIONS] COMMAND [ARGS]...

  Manage Cog via its REST API on the command line.

Options:
  -c, --config-file PATH  Path to an INI-formatted configuration file
                          [default: ~/.cogctl]
  -p, --profile TEXT      The profile within the config file to use
  -u, --url TEXT          Override API URL root to use, e.g.
                          'https://127.0.0.0:4000'
  -U, --user TEXT         Override account to authenticate against the API
  -P, --password TEXT     Override password to authenticate against the API
  -v, --verbose           Be verbose
  --help                  Show this message and exit.

Currently Gort only responds to explicit commands, formatted like structured commands. There should be some kind of ability to respond to non-command text, sort of like Slack "triggers" (but not like Cog "triggers", which are very different).

Triggers would be configured with a pattern to match and a command, so when Gort sees text matching the defined pattern, it executes the command.

• Open question: How will it behave wrt to trigger text + command inputs?

  • Should input be passed through to the command parser? Maybe set as an envvar?

• What if a command also matches a trigger? Which takes precedence?

  • Execute triggers only if there’s no matching command?
  • Execute trigger and stop processing (so command never executes)?
  • Either of the above/configurable per trigger?

• What about timed triggers (maybe use cron definitions)?

The default gort bundle uses a long_description field in the commands' YAML definitions, but those fields are currently ignored. We want to include them so they can be used in help output.

1. Add a LongDescription field to the data.BundleCommand struct
2. Update the dataaccess.postgres.createBundlesTables function to add a long_description to the bundle_commands table.
3. Update the dataaccess.postgres.doBundleCommandsDataGet and doBundleInsertCommands accordingly.
4. Make cli.detailCommand function read the description from the command instead of the bundle.

Successfully using the gort bootstrap command overwrites your ~/.gort/profile file with a new file with one new profile. This is unexpected, especially if a user wants to bootstrap multiple Gort deployments.

The command should work like gort profile create by adding a new profile to any existing configuration.

Also a thought: gort bootstrap uses a default profile name. What if that name already exists? Maybe add a flag to override the default profile name?

Currently Gort users map to Slack users by way of their email address. This isn't especially secure, and may not even be supported by other chat providers.

• Use each chat provider’s user ID as an identifier instead of the user’s email.
  • Future chats might not provide or even use user email
  • This is more pressing now that Slack socket mode doesn’t make user email easily accessible.
• This means we’ll also need a user map command or something to explicitly map gort users to chat users.
• Will have consequences for auto-generating users.

We could use a gort role info command, which displays information about roles,. including (but not necessarily limited to):

• Role name
• Permissions granted
• Groups added to the role, maybe?

There's no way to manage profiles from the command line (aside from manually editing the profile file).

We should add the following subcommands:

• gort profile list -- List existing profiles
• gort profile create -- Adds a new profile
• gort profile default -- Sets the default profile
• gort profile delete -- Deletes an existing profile

This may require making the client.loadClientProfile and client. loadClientProfile functions exported.

From cogctl:

$ docker run -it operable/cogctl:alpine-1.0.1 cogctl profile --help
Usage: cogctl profile [OPTIONS] COMMAND [ARGS]...

  Manage Cog profiles.

  If invoked without a subcommand, lists all the profiles in the config
  file.

Options:
  --help  Show this message and exit.

Commands:
  create   Add a new profile to a the configuration...
  default  Sets the default profile in the configuration...

The gort bundle install command should accept stdin input, similar to how cogctl bundle install did.

Per cogctl bundle install --help:

When installing from a file, you may either give the path to the file, as in:

  cogctl bundle install /path/to/my/bundle/config.yaml

or you may give the path as `-`, in which case standard input is used:

  cat config.yaml | cogctl bundle install -

The instructions documented in the Quickstart page will likely be the first experience many users have with Gort. As such, it is important that it works.

To ensure this, we can add an acceptance test in a GitHub Action that will execute the command-line instructions from this page and ensure they succeed.

I've been working on creating an acceptance test for the quickstart (#83), but I've hit some issues with a race condition when starting with docker-compose. If the database isn't fully up and running before gort starts, the DAL won't retry the connection.

Beyond what's in the PR, I've been working on creating a unit test that verifies the retry logic, but I've hit a number of issues with the race detector. After trying a few approaches, I've only managed to create solutions that work locally, but not when running in a container.

I'd like to recommend that we remove the auto-reloading of config to simplify the logic involved in maintaining the singletons. This would make it easier for us to test and improve the rest of Gort's functionality.

The gort bundle install command should include the following flags/behaviors (for parity with cogctl)

(From docker run -it operable/cogctl:alpine-1.0.1 cogctl bundle install --help)

Options:
  -e, --enable               Automatically enable a bundle after installing?
                             [default: False]
  -f, --force                Install even if a bundle with the same version is
                             already installed. Applies only to bundles
                             installed from a file, and not from the Warehouse
                             bundle registry. Use this to shorten iteration
                             cycles in bundle development.  [default: False]

Currently, typing !gort:help provides a list of the commands that Gort knows.

This functionality should be extended so that typing !gort:help COMMAND provides some basic information about the command (even if it's just the bundle+command name, its "long description", and a suggestion to type !COMMAND --help for command-specific help)

Open question: should this include the command's rules? Is that a security probelm?

The Slack adapter uses Slack's deprecated Real Time Messaging API (which was the right way to do it back when it was built...).

It looks like we want to use socket mode instead.

As described in the documentation, we should have the following commands:

• gort permission info
• gort permission list

Edit: permissions --> permission

The REST API currently doesn't do any authentication. It needs to.

Currently commands can only be executed in Docker via a local Docker socket.

Add support for executing commands via the K8S API.

The gort group revoke command should allow the addition of multiple roles (for parity with cogctl)

(From docker run -it operable/cogctl:alpine-1.0.1 cogctl group revoke --help)

Usage: cogctl group revoke [OPTIONS] GROUP ROLES...

  Revoke one or more roles from a group.

Options:
  --yes   Confirm the action without prompting.
  --help  Show this message and exit.

This command is included in the documentation and should be implemented.

$ docker run -it operable/cogctl:alpine-1.0.1 cogctl bundle versions --help
Usage: cogctl bundle versions [OPTIONS] NAME

  List installed bundle versions.

  If no bundle name is given, all versions of all bundles are listed, along
  with their status ("Enabled", "Disabled", "Incompatible")

  Alternatively, if a bundle name is supplied, only versions of that bundle
  are shown.

  In either case, supplying the `--incompatible` option restricts the
  listing to only incompatible versions.

Options:
  -x, --incompatible  List only incompatible bundle versions
  --help              Show this message and exit.