squarespace / pgbedrock

Manage a Postgres cluster's roles, role memberships, schema ownership, and privileges

pgbedrock

pgbedrock is an application for managing the roles, memberships, ownerships, and most importantly the permissions for tables, sequences, and schemas in a Postgres database.

Given the parameters to connect to a Postgres database (i.e. host, port, etc.) and a YAML file (a "spec") representing the desired database configuration, pgbedrock makes sure that the configuration of that database matches the spec. If there are differences, it will alter the database to make it match the spec.

It can be run as a docker container (via docker run quay.io/squarespace/pgbedrock) or as a local command-line utility (via pip install pgbedrock).

Detailed information can be found in the documentation.

Example

As an example, the definition for the jdoe role in the spec might look like this:

jdoe:
    can_login: yes
    is_superuser: no
    attributes:
        - PASSWORD "{{ env['JDOE_PASSWORD'] }}"
    member_of:
        - analyst
    owns:
        schemas:
            - finance_reports
        tables:
            - finance_reports.Q2_revenue
            - finance_reports.Q2_margin
    privileges:
        schemas:
            read:
                - finance
                - marketing
            write:
                - reports
        tables:
            read:
                - finance.*
                - marketing.ad_spend
                - marketing.impressions
            write:
                - reports.*
            except:
                - reports.Q2_fixed_assets
        sequences:
            write:
                - reports.*

When pgbedrock is run, it would make sure that:

• The role jdoe exists
• jdoe can log in
• jdoe is not a superuser
• jdoe's password is the same as what is in the $JDOE_PASSWORD environment variable
• All other role attributes for jdoe are the Postgres defaults (as defined by pg_authid).
• jdoe is a member of the analyst role
• jdoe is a member of no other roles
• jdoe owns the finance_reports schema
• jdoe owns the finance_reports.Q2_revenue and finance_reports.Q2_margin tables
• jdoe has read-level schema access (in Postgres terms: USAGE) for the finance and marketing schemas
• jdoe has write-level schema access (CREATE) for the reports schema
• jdoe has read-level access (SELECT) to all tables in the finance schema and to the marketing.ad_spend and marketing.impressions tables
• jdoe has default privileges to read from all future tables created in the finance schema
• jdoe has write-level access (SELECT, INSERT, UPDATE, DELETE, TRUNCATE, REFERENCES, and TRIGGER) to all tables in the reports schema except for the Q2_fixed_assets table
• jdoe has default privileges to write to all future tables created in the reports schema
• jdoe has write-level access (SELECT, USAGE, UPDATE) to all sequences in the reports schema
• jdoe has default privileges to write to all future sequences created in the reports schema
• jdoe does not have any access other than that listed above (except whatever it inherits from the analyst role that jdoe is a member of)

Quickstart

Using pgbedrock requires three steps: generating a spec for a database, reviewing that spec, and configuring the database using that spec. Below we will do this using the pgbedrock docker image, but these steps can also be done with the pip-installed version of the tool.

1. Generate a spec for a database. Specify the connection parameters below (host, port, database, username, and user password) as well as the place to output the tentative spec. Note that the user passed with -U must be a superuser.

   docker run -it \
       quay.io/squarespace/pgbedrock generate \
       -h myhost.mynetwork.net \
       -p 5432 \
       -d mydatabase \
       -U mysuperuser \
       -w supersecret > path/to/spec.yml

2. Review the spec. pgbedrock is not quite as flexible as Postgres's permissioning, and as a result the generated spec may differ slightly from the current state of your database. For more information on these potential simplifications, see the Notable Functionality And Caveats section in the docs. As a result, it is recommended to run pgbedrock configure in check mode the first time you use it to see what changes it would introduce to your current setup. This looks similar to the command above, but requires us to also pass in the passwords for any roles whose passwords are managed within Postgres itself. These can be identified in the spec file as roles with a line that looks like PASSWORD "{{ env['MYROLE_PASSWORD'] }}" (if you forget to pass in these passwords pgbedrock will just throw an error and refuse to run). Note that you must run pgbedrock configure against the Postgres primary. To run pgbedrock in check mode we do the following:

   docker run -it \
       -e "JDOE_PASSWORD=${JDOE_PASSWORD}" \
       -e "JSMITH_PASSWORD=${JSMITH_PASSWORD}" \
       -v /path/to/spec.yml:/opt/spec.yml \
       quay.io/squarespace/pgbedrock configure spec.yml \
       -h myhost.mynetwork.net \
       -p 5432 \
       -d mydatabase \
       -U mysuperuser \
       -w supersecret \
       --check

   Note that --check is actually the default behavior, so we could also omit that.

3. Configure the database using the spec. Once you feel comfortable with the changes pgbedrock would introduce, run the above command again using --live instead of --check. Changes will now be made real. To make future changes, modify the spec file and run the above command.

For further information, see the documentation.

License

Copyright 2018 Squarespace, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at:

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

pgbedrock's Issues

Add the possibility to set configuration parameters at the user level.

Hi! My team and I have been trying to figure out how we could set configuration parameters for specific roles and/or users using pgbedrock.

Given the documentation, and the code base, our take is that it is only possible to have pgbedrock run ALTER ROLE "{}" WITH "{}"; statements, not ALTER ROLE "{}" SET "{}"; ones.

Would the possibility to do this make sense or is there a better way to do this that doesn't involve manually connecting to our PG instances and manually configuring our users?

Need:

• the possibility to set configuration parameters for roles and/or users.

Example use-case:

Setting different statement_timeout values for different users using pgbedrock.

Manage schema owner's access to schema

Summary
pgbedrock currently ignores the access that a schema owner has to the schema it owns. If zmarine owns a schema zmtest, then pgbedrock will take no action whether or not zmarine has CREATE or USAGE on that schema, regardless if they should have those permissions or not.

pgbedrock deliberately doesn't automatically grant CREATE or USAGE just because a role owns a schema since it's possible that the owner isn't meant to be able to modify things, but this 1) shouldn't be the case for personal schemas, where we definitely want the owner to have CREATE and USAGE and 2) if we explicitly ask for a permission for that schema pgbedrock shouldn't ignore our request.

Verification of this buggy behavior is below:

-- Executed as another (super) user
CREATE SCHEMA zmtest AUTHORIZATION zmarine;
REVOKE CREATE ON SCHEMA zmtest FROM zmarine;
REVOKE USAGE ON SCHEMA zmtest FROM zmarine;
CREATE TABLE zmtest.foo AS (SELECT 1+1);
GRANT SELECT ON TABLE zmtest.foo TO zmarine;

-- Schema owner does not get USAGE automatically
SET ROLE zmarine;
SELECT * FROM zmtest.foo;  -- Permission denied
RESET ROLE;

GRANT USAGE ON SCHEMA zmtest TO zmarine;

-- Schema owner does not get CREATE automatically either
SET ROLE zmarine;
CREATE TABLE zmtest.foobar AS (SELECT 2);  -- Permission denied
RESET ROLE;

-- Clean up
DROP TABLE zmtest.foo;
DROP SCHEMA zmtest;



/*
Moreover, pgbedrock doesn't care about schema access for the owner. Given a spec like:
    zmarine:
        can_login: true
        owns:
            schemas:
                - zmtest

And access revoked with:
    CREATE SCHEMA zmtest AUTHORIZATION zmarine;
    REVOKE CREATE ON SCHEMA zmtest FROM zmarine;
    REVOKE USAGE ON SCHEMA zmtest FROM zmarine;

pgbedrock will say no changes are needed. That is correct: we don't know for sure that the owner
should be able to access things. However, the following spec will _also_ say no changes are needed:
    zmarine:
        can_login: true
        owns:
            schemas:
                - zmtest
        privileges:
            schemas:
                write:
                    - zmtest

That is a bug. This same behavior can be observed for personal schemas, where the owner
definitely _should_ have CREATE and USAGE. If we have a spec like:
    zmarine:
        can_login: true
        has_personal_schema: true

And access revoked with:
    REVOKE CREATE ON SCHEMA zmarine FROM zmarine;
    REVOKE USAGE ON SCHEMA zmarine FROM zmarine;

pgbedrock will again say that no changes are needed.
*/

Key Actions

• In privileges.py, if the role has has_personal_schema == True then add that write-level schema access to the desired privileges for that role.
• Don't ignore schema access requests just because the role being configured is the schema owner.

Setting password commented out

I tried to create a role with a password.

student:
  can_login: yes
  is_superuser: no
  attributes:
    - PASSWORD "{{ env['STUDENT_PASSWORD'] }}"
  privileges:
    schemas:
      read:
        - challenges
    tables:
      read:
        - challenges.*

I provided STUDENT_PASSWORD in the environment. This is the generated config. The password setting was skipped for some reason:

--------------------------------
--- Configuring attributes -----
--------------------------------

CREATE ROLE "student";
ALTER ROLE "student" WITH LOGIN;
--ALTER ROLE "student" WITH ENCRYPTED PASSWORD '******';

How can I get pgbedrock to actually set the password for my new user?

'ERROR: permission denied for relation pg_authid' when running against RDS

I created an empty DB in RDS using Postgres 9.6.6 owned by user postgres. I installed pgbedrock as a library (not as a container). Created an empty DB create database bedrock with encoding='UTF8' LD_COLLATE='en_US.UTF-8' LC_CTYPE='en_US.UTF-8' OWNER='postgres' ;. Ran the following command to generate the base spec:

pgbedrock generate \ -h MYINSTANCE.us-east-1.rds.amazonaws.com \ -p 5432 \ -d bedrock \ -U postgres \ -w <POSTGRES-PASSWORD> > spec.yml

When I look inside spec.yml it contains:

Failed to execute query " SELECT rolbypassrls, rolcanlogin, rolconnlimit, rolcreatedb, rolcreaterole, rolinherit, rolname, rolpassword, rolreplication, rolsuper, rolvaliduntil FROM pg_authid WHERE rolname != 'pg_signal_backend' ; ": permission denied for relation pg_authid

Add documentation about how to use pgbedrock in a docker compose setup

I have

version: '3'
services:
  postgres:
    container_name: postgres
    image: postgres
    restart: always
    ports:
      - "5432:5432"
    environment:
      POSTGRES_DB: DB
      POSTGRES_USER: user
      POSTGRES_PASSWORD: password
    volumes:
      - "./pgdata:/var/lib/postgresql/data"

How do I properly add pgbedrock to my compose setup?

How about 'brew'ing some snake oil?

On my Mac ...
$ pip install --upgrade pip
Collecting pip
....
Successfully installed pip-9.0.3

$ pip install wheel
Requirement already satisfied: wheel in /usr/local/lib/python2.7/site-packages

$ which wheel
/usr/local/bin/wheel

$ wheel version
wheel 0.30.0

$ wheel install pgbedrock
No match for requirement pgbedrock

$ which pgbedrock
$

No Joy in MuddVille ...

documentation assumes 'latest' tag on Quay exists

Unable to find image 'quay.io/squarespace/pgbedrock:latest' locally
docker: Error response from daemon: manifest for quay.io/squarespace/pgbedrock:latest not found.

Using 0.1.2 at least gets the docker image to run...

Support Google Cloud SQL

Similar to #9 I'd be very interested in using this to manage users on our Cloud SQL instance. Unfortunately, per the docs "any features that require SUPERUSER privileges" are not supported.

I'm working on testing this using the default postgres user, so I don't know exactly what any errors might be. But, this would still be very useful as a validation check against existing privileges.

Is there a route to rework the tool so that it could a) validate roles against a specified YAML role, b) write out the SQL that would have to be run to make any adjustments and c) all while using a non superuser role?

Basically, I'd see this as part of our test suite that would create an error if something was off, but wouldn't take action on it.

Provide multi-database support

Summary

At present pgbedrock is intended to configure a single database per spec, but this is kind of a strange compromise: roles and role memberships are at the Postgres instance level, meaning that if one Postgres instance has multiple databases in it then there is a need to have multiple spec.yml files which all have to be kept in sync with regards to the roles and role memberships in them. This can be done (especially by just doing pgbedrock generate before making any changes to a database), but it is awkward.

To solve this, pgbedrock should manage all databases in a Postgres instance. This would mean that the CLI would take a database name, but then would look up all databases via Postgres's pg_database table. The attributes.py and memberships.py submodules would run on any one database (which one doesn't really matter), but the ownerships.py and privileges.py submodules would run once for each database that exists.

In addition, objects in the spec would need to database-qualified, e.g. mydatabase.myschema.myobject. Then when the ownerships.py and privileges.py submodules run they should traverse the existing spec and pull out the objects that are relevant to that database. Then those submodules wouldn't need to really be changed at all. To do that, we'd ideally capture this 'pull things out of the spec.yaml' process in a new submodule (e.g. spec_inspector.py or something) so that we can keep all functionality together that pulls desired state out of a spec, as there is already some stuff that walks through a spec and get desired state out of it.

Key actions

• Add functionality to identify all databases in a Postgres instance
• Run ownerships.py and privileges.py for each database in the Postgres instance; attributes.py and memberships.py should just run on any one database
• Before running pgbedrock configure, ensure all objects in the spec.yml file are database-qualified
• Add a submodule that, among existing spec-inspection functionalities, also takes a database name and returns the portions of the spec that relate to objects from that database
• Modify pgbedrock generate to traverse through all databases and add database-qualified objects to its output spec
• Likely the best way to internally represent objects in this world would be as a namedtuple of (db, schema, objname), then it is only when we are going to output a spec that we convert it into a quoted, database-qualified object. This would allow us to avoid worrying about making sure all object are properly-quoted as we work with them internally.

support Redshift

Redshift is based on Postgres 8.0.2, so a few things are different. Maybe this issue should be a list of known issues?

pg_authid does not exist

Failed to execute query "SELECT * FROM pg_authid WHERE rolname != 'pg_signal_backend';": relation "pg_authid" does not exist

It looks like it was introduced in 8.1:

This change also replaces pg_shadow and pg_group by new role-capable catalogs pg_authid and pg_auth_members.

Add functions to ownership list

Functions, like tables and schemas, have an owner, which should also be managed by pgbedrock.

For example

jdoe:
    ...
    owns:
        schemas:
            - finance_reports
        tables:
            - finance_reports.Q2_revenue
            - finance_reports.Q2_margin
        functions:
            - f_cast_string_as_number

Stepping down as maintainer

Hey all,

I left Squarespace and the data engineering world almost a year ago and since then haven't used Postgres much. As a result, I haven't been very focused on this project or improvements to it.

If anyone is interested in taking over maintenance, feel free to respond here or reach out to @cboline or @jbeluch, as they'll have a better idea of whether this is something Squarespace wants to continue to be involved in or if they're happy handing over the reins.

I'll leave myself as owner for the next bit to see if any conversation emerges here, but in the near future will remove myself from the ownership role.

Best,
Zach

Exclude tables from access

Say we had a schema, the_data with three tables, a, b, and c. We have some users that have read/write access to all the tables in the_data, but after a while we decide that we do not like table b so we'd like to deprecate it by first restricting access to it for a while and then dropping the table properly later.

I'm open to other suggestions like "how about you just move table b to an off-limits schema", but I wanted to discuss if this is a pattern that's useful and if so what would that look like in the spec def?

Annotate all methods with Arg types and Return types

As other devs being to add features to pgbedrock, it would be helpful to have more detailed docstrings, similar to datasheets: show the arg types and their return types. If we wanted, we could even add type checking, though I think (?) we'd have to drop Python 2 support to do that. Still, it might be worth it.

Manage access to information schema and sys catalogs

I used pgbedrock to create a user which I don't want to allow access to the system catalogs and information schema. Is there a way to forbid this access in my YAML?

Ensure objects are not referenced multiple times within a spec

Summary
If a user referenced an object multiple times in a spec (e.g. schema ownership) then pgbedrock will just make the change at each spot (e.g. change ownership to role1 and then later change ownership to role2). While this isn't strictly wrong, it makes it too easy to make a mistake. Before running any of our analyzers we should verify that no object appears multiple times in the following cases:

• schema is not under owns for multiple roles or is in the has_personal_schema for a role and in the owns for another role
• verify that a role isn't defined multiple times
• within one role, no object occurs in both the read and write sections. This isn't really wrong since pgbedrock grants read as well if you desire write, but it'd be good to clarify this behavior to users by ensuring they can only put things in the expected way; if they could put things in write and either put things in read or not then they may assume that that means that read can not occur when write is granted.

I don't think there are any other areas at present where an object occurring multiple times would cause unexpected behavior, though it'd be worth doing a bit more deliberating on this to see if I've missed anything.

Key Actions

• In the spec_inspector.py referenced in the Provide multi-database support issue verify that each schema only shows up in one owns or has_personal_schema spot and never more than one spot.
• In that same spec_inspector.py, verify that no object occurs in both the read and write sections.
• Run the above verification before any submodule's analyzer functionality happens
• Make sure there aren't other areas where duplicate object references could cause unexpected behavior. If table and sequence ownership has been provided (via the Support table and sequence ownership issue), then those would definitely be something we'd want to verify have no duplicates as well.

Feature request: insert/update permissions without delete

Hi Zach,

We're building an app where we have some audit/compliance needs that require rows never be deleted -- only updated. It'd be great if pgbedrock had the ability to have more fine-grain permissions other than read/write. I did see the note in documentation about being open to a pull request with that functionality. Before starting in on that, do you have any particular constraints around a pull request that you'd accept?

I also noticed on a quick skim of the read/write code that there's a comment at https://github.com/Squarespace/pgbedrock/blob/master/pgbedrock/privileges.py#L62: "If a write privilege is desired then read access is as well" -- we actually have this case too, where we have a 3rd party system that needs permission to insert incoming data into our system but cannot have read access to that table (just insert, no select). Updating the permissions to be more fine-grain may require revisiting that?

Thanks,
Jeff

Support regex patterns

Currently we support the myschema.* pattern, which expands to all objects in myschema of the relevant object kind (e.g. tables or sequences). A common request is to support arbitrary regex patterns.

This is definitely doable, but will require modifying the spec validation, adding support for taking a regex pattern and expanding it, and of course adding associated testing.

Support table and sequence ownership

Summary
At present pgbedrock manages ownership of schemas, but it does not support management of table or sequence ownership (with the exception of objects in personal schemas). This is strange and leaves ownership of these objects in kind of a no-man's land. Ideally pgbedrock should be able to support ownerships of all objects it configures permissions on.

One wrinkle to take into account is what happens if someone wants to own an object that is in a personal schema. We should disallow that.

Key Actions

• Add tables and sequences keywords under the owns keyword
• Determine whether it makes more sense to modify the SchemaAnalyzer class or just create a new class (NonSchemaAnalyzer?) given that if a non-schema object doesn't exist we can't create it + the functionality for determining ownership of schemas uses different queries than for other objects ownership. My gut feeling is that this should all be one class, but it's worth verifying
• If the decision for the above is to make one class, then we should combine the Q_GET_ALL_NONSCHEMA_OBJECTS_AND_OWNERS, Q_GET_ALL_OBJECT_OWNERS, Q_GET_ALL_PERSONAL_SCHEMAS, and Q_GET_ALL_SCHEMAS_AND_OWNERS queries because let's be honest: it's weird that there's 4 different ways to get at very similar information (object + object owner).
• Before processing, identify all desired personal schemas (likely using the spec_inspector.py discussed in the Provide multi-database support issue) and make sure that no desired object ownership is for an object in one of the personal schemas.
• Modify pgbedrock generate to add ownership for tables and sequences to its output spec

Is this repository still maintained?

Hey @dmoore1989 @zcmarine @cpdean -- Sorry if you folks are the wrong ones to ping, but ya'll are some of the most recent, prolific contributors to this project. Is it still being used / maintained by Squarespace? Is there any chance support will pick up? Particularly supporting PG 11+ looks like an important piece and there is an open PR here for that: #67

Appreciate any response / help in this regard! A company I contract with is potentially using this tool, so I'd like to know if that is a bad choice or not.

Support schema names with dashes

Summary
Postgres doesn't like dashes in identifiers. To solve this, we can double-quote identifiers. We currently do this for tables and rolenames, but we don't do it for schemas. As a result, a spec like the below will cause an error:

zmarine:
    can_login: true
    owns:
        schemas:
            - schema-with-dash
    privileges:
        tables:
            read:
                - schema-with-dash.*

Postgres will complain that there is a syntax error at or near "-". To resolve this, we should double-quote all schemas as well.

Key Actions

• Identify all places where schemas are used and ensure they are double-quoted. This will likely just be modifying the queries within context.py to make sure they all double-quote schemas, but there may be downstream places that do comparisons / lookups of a schema and don't double-quote it.
• In addition, the submodules that use schemas (ownerships.py and privileges.py) will need to quote schemas that they are working with to make sure lookups with context.py work properly.
• This is true for all other issues as well, but we definitely want a test here to verify that this behavior works as expected.

Support account time limits as a first class feature

During hack week at Squarespace it is common to receive requests for access to the data warehouse. These special case accounts require cleaning up. It would be lovely if pgbedrock supported a time limit on roles.

I'm imagining an expires_at attribute like so:

jdoe:
    can_login: yes
    is_superuser: no
    expires_at: 2018-04-13

Pgbedrock can add a VALID UNTIL clause on the role when this attribute is set. It can also issue an error or warning when executed after the 13th to remind people to clean up the config.

Internally translate has_personal_schema

Currently personal schemas are supported as a first-class citizen throughout all of pgbedrock's code base. Once table and sequence ownership are supported, we can instead just translate the user's spec when we begin processing and thus get rid of all the special functionality internally that deals with personal schemas.

To be more specific, when we see has_personal_schema: True we would convert that to say that this role owns a schema of its own name and all tables and sequences within that schema, i.e. this role definition:

myrole:
    has_personal_schema: True

would be translated internally to this:

myrole:
    owns:
        schemas:
            - myrole
        tables:
            - myrole.*
        sequences:
            - myrole.*

We would then add myrole to a list of personal_schemas, so when we later saw personal_schemas.* somewhere we would know how to translate that, e.g. if we have found myrole0, myrole1, and myrole2 all had has_personal_schema: True, then we would take the following role definition:

another_role:
    privileges:
        tables:
            - personal_schemas.*

and translate that into:

another_role:
    privileges:
        tables:
            - myrole0.*
            - myrole1.*
            - myrole2.*

Conveniently, we already do this second part.

This would reduce a lot of head scratching that goes into making sure personal schemas are properly supported since everything after the initial loading of the spec could disregard them as a concept.

Password envvars created by pgbedrock generate should convert dashes to underscores

Various tools like Bamboo do not like dashes used in variable names. As a result, in those use cases the variable for the envvar has to be manually changed in the spec.yml to use these tools.

While it's feels lousy to name the envvar differently from the role it represents, it feels like the better option for usability of the tool.

This should be a pretty easy fix: basically just adding a .replace('-', '_') to this line and verifying that there is test coverage of that change.

`pgbedrock` errors on usernames with hyphens if that user also has a personal schema

Given a user:

captain-planet:
    has_personal_schema: true

The following error is returned by Postgres:

18-Jul-2018 19:57:19 | -- Skipping privilege configuration for user "captain-planet"": syntax error at or near "-"
-- | --
18-Jul-2018 19:57:19 | LINE 3:     ALTER DEFAULT PRIVILEGES IN SCHEMA captain-planet GRANT...

pgbedrock should be fully quoting captain-planet, which it does many places, but not, alas, enough places.