ramontayag / storey

Manage PostgreSQL's schemas for a multi-tenant Rails app with ease

Storey

Storey is used to manage multiple schemas in your multi-tenant Rails application.

Heavily inspired by the Apartment gem, Storey simplifies the implementation of managing a multi-tenant application. This simplifies things by doing away with the other implementations that Apartment has - like MysqlAdapter and managing multiple databases (instead of managing multiple schemas) which complicated development and testing.

Configuration

Typically set in an initializer: config/initializer/storey.rb

Storey.configure do |c|
  # Defines the tables that should stay available to all (ie in the public schema)
  # Note that there's currently no way to exclude tables that aren't linked to models
  # If you have any ideas on how to do this I'm open to suggestions
  c.excluded_models = %w(User Company Role Permission)

  # If set, all schemas are created with the suffix.
  # Used for obscuring the schema name - which is important when performing schema duplication.
  # c.suffix = "_suffix"

  # Defines schemas that should always stay in the search path, apart from the one you switched to.
  # c.persistent_schemas = %w(hstore)

  # If you use a connection string, set it here. When nil, it falls back to the configuration in database.yml
  c.database_url = ENV["DATABASE_URL"]
end

Switching in your Rack app

To switch in your application, you can just opt to call Storey.switch somewhere in your app. For example, in a Rails ApplicationController it would look like:

class ApplicationController
  before_action :switch_to_tenant

  def switch_to_tenant
    subdomain = request.subdomain
    Storey.switch(subdomain) if Website.exists?(subdomain: subdomain)
  end
end

There are some instances where you need to switch schemas before you get to your application. A good example is Devise. Devise uses Warden to authenticate users. Warden is inserted as a Rack application and checks to see if the user attempting to access the page signed in.

To do this, it must check the database. If your users live on separate schemas, there's a big chance that Warden will think the user does not exist. Especially in this scenario, use the Rack app found in this gem. In Rails, insert this somewhere in your application.rb or in an initializer:

Rails.application.config.middleware.
  insert_before Warden::Manager, Storey::RackSwitch

You must also define how to determine the schema to switch to. To do that, set switch_processor:

Storey.configure do |c|
  c.switch_processor = ->(env) do
    # find the schema name based on something in the env
    subdomain = find_in_env(env)
    return subdomain if Website.exists?(subdomain: schema)
  end

  # You can pass any object that responds to call and accepts one arg: the env.
  c.switch_processor = MyStoreySwitchProcessor
end

class MyStoreySwitchProcessor
  def self.call(env)
    subdomain = find_in_env(env)
    # ...
  end
end

When the result of switch_processor is a string, Storey.switch('the-string-it-returns') is called. If nil, no switching happens.

Methods

schema

Return the current schema.

Accepts options

array: true # return the schemas as an array

Usage:

Storey.schema # "\"$user\", public"
Storey.schema(array: true) ["\"$user\"", "public"]

schemas

Returns all postgres' schemas.

Accepts options:

:public => true

Usage:

Storey.schemas # defaults to :public => true
Storey.schemas(:public => true)

default_schema?

Returns true if the current schema is the default schema. Returns false otherwise. Useful for running migrations only for the public schema.

Usage:

Storey.default_schema?

create

Accepts:

String - name of schema

Usage:

Storey.create "schema_name"

drop

Accepts

String - name of schema

Usage:

Storey.drop "schema_name"

switch

Accepts

String - optional - schema name
Block - optional

If a block is passed, Storey will execute the block in the specified schema name. Then, it will switch back to the schema it was previously in.

Usage:

Storey.switch "some_other_schema"
Post.create "My new post"
Storey.switch # switch back to the original schema

Storey.switch "some_other_schema" do
  Post.create "My new post"
end

duplicate!(origin, copy)

Accepts

origin - name of old schema to copy
copy - name of new schema

Copies a schema with all data under a new name. Best used in conjunction with Storey.configuration.suffix set.

Usage:

Storey.duplicate!("original_schema", "new_schema")

Rake tasks

storey:hstore:install

Run rake storey:hstore:install to install hstore extension into the hstore schema. Ensure that 'hstore' is one of the persistent schemas.

Development

In the storey directory, after installing the gems:

• docker-compose up db
• `cp spec/dummy/config/database.yml{.sample,}
• rspec spec

storey's Issues

Have a Storey.default_schema? method to check if the default schema is currently being used

Will be helpful for migrations that only need to be done in the default schema:

def change
  if Storey.default_schema?
    User.create(...)
  end
end

@corroded, what do you think?

When no migrations have been run, storey should not blow up when getting schema migrations

When no migrations have been run, it seems that the schema_migrations table does not exist. Storey blows up when you run rake storey:migrate. The solution I have been doing was to run rake db:migrate first, then rake storey:migrate. Storey should work without having to run migrations the normal way.

When duplicating a schema with an invalid schema name, PG locks up

#21 should help prevent it, but if/when this happens, Storey should be able to handle it anyway.

@corroded, can you please paste that stuck process?

Switching to a non-existent schema does not raise exception anymore

pg gem does not seem to be raising error any longer, like it used to. We should check for the existence of the schema without relying on the exception being raised.

Reset and rollback database schema

Any suggestion if I want to reset (or rollback) all data in specific schema.

Thanks.

Fix failing tests

Haven't run the specs in a couple of months -- there are 3 failing specs

rspec ./spec/storey/duplicate_spec.rb:48 # Storey#duplicate! should clear the PGPASSWORD environment variable
rspec ./spec/storey/duplicate_spec.rb:17 # Storey#duplicate! when there's no suffix set should create a schema with the same data under a new name
rspec ./spec/storey/duplicate_spec.rb:62 # Storey#duplicate! when a suffix is set should create a schema with the suffix

Add ability for permanent schemas to be in search path

Need arose because of hstore. It is installed in 1 schema that always has to be accessible when creating a table that has hstore as the datatype.

Dump a schema

It would be great if there's a command to dump the sql of a certain schema. This is great for creating per-schema back ups (perhaps your users can download this and keep it for themselves).

Sample usage:

Storey.schema("user_23").dump
Storey.dump_schema("user_23") # this might be better

Problem copying schema migration versions

Creating a schema in production (tested in Rails) causes no schema migrations to be copied over. The problem is that the schema migration versions query is cached:

CACHE  (0.0ms)  SELECT "schema_migrations"."version" FROM "schema_migrations" ORDER BY "schema_migrations"."version" ASC

Thus, when the schema migrations are supposed to be copied over, they are not because it looks like it exists in the new schema being created. In reality, ActiveRecord just returned a cached result.

Issue with integer password

If you are set your password to integer like 123456 in database.yml then you'll have exception "can't convert Fixnum into String". But if you type '123456'(with quotes) then everything works fine

Rake storey:migrate does not update db/schema.rb

What subject says :p zeus may be a factor to this (I usually do zeus rake storey:migrate)

Sequel gem

Does this gem support the sequel gem, if not would you please consider it?

Developer should be able to install hstore with a rake task

Should live on the hstore schema.

Clean up console when running specs

It's quite noisy right now with a lot of PG related output:

CREATE SCHEMA
SET
SET
SET
SET
SET
SET
SET
SET
SET
CREATE TABLE
ALTER TABLE
CREATE SEQUENCE
ALTER TABLE
ALTER SEQUENCE
CREATE TABLE
ALTER TABLE
CREATE SEQUENCE
ALTER TABLE
ALTER SEQUENCE
CREATE TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
CREATE INDEX
REVOKE
REVOKE
GRANT
GRANT

How do you hide this?

Override `db:migrate` to run `storey:migrate`

I've gotten reports of some users accidentally running db:migrate which bring their apps down. Should we override what db:migrate and just do what storey:migrate does?

Will there be a case where one would want to just run the migrations on the public schema but not the rest?

This is up for discussion.

Production ready ?

Hi,

I would like to know if it is in production ?

How easy it is compared to Apartment in terms of understanding and operationally ?

Thanks.

Refactor dumper into two classes: sql dumper and ruby dumper

Promote SRP!

Developer can invoke a rake task that renames schemas to implement suffix

It would be nice to have a built-in rake task to implement schema suffixes.

For example: I have a system that doesn't use suffixes, but I want to be able to duplicate schemas and be sure that Storey does not accidentally rename a text in the SQL file that doesn't actually refer to the schema name. I can then set Storey.suffix = '_nakam' and run rake storey:rename_schemas. This will rename all schemas except public.

Also add ability to change suffix even if you already have set one. For example, if I have schemas with the suffix '_nakam' and I want to make them '_pucha', then I can:

1. Change the Storey.suffix = '_pucha'
2. Run rake storey_rename_schemas PREV_SUFFIX=_nakam

dash in schema name

Hi i have multi domain site with apartment gem,i which a sub domain can be dash-test.com,when a switch a schema with this gem it omit dash from name and make a schema with first name?
Is there any way do this?

Error checking missing for schema loading

storey/lib/storey/duplicator.rb

Line 89 in 4c6a59c

Open3.capture3(psql_load_command)

This does not do any checks on whether the schema loading actually worked.

I noticed this due to the dump failing to load by not finding a needed function.

The schema should be loaded with the following additional options and then the exit status checked:

psql -v ON_ERROR_STOP=true --single-transaction

`rake db:create` blows up in Rails 5.2

Story.init attempts to connect to the database, and this is added to to_prepare in the Rails initialization process. This breaks tasks like rake db:create.

License missing from gemspec

Some companies will only use gems with a certain license.
The canonical and easy way to check is via the gemspec
via e.g.

spec.license = 'MIT'
# or
spec.licenses = ['MIT', 'GPL-2']

There is even a License Finder to help companies ensure all gems they use
meet their licensing needs. This tool depends on license information being available in the gemspec.
Including a license in your gemspec is a good practice, in any case.

How did I find you?

I'm using a script to collect stats on gems, originally looking for download data, but decided to collect licenses too,
and make issues for missing ones as a public service :)
https://gist.github.com/bf4/5952053#file-license_issue-rb-L13 So far it's going pretty well

Hstore columns not being created with the table

There's a migration that creates other normal columns alongside an hstore column, but the hstore column doesn't get created in the public schema.

I think the problem is that rake storey:migrate migrates the public schema via rake db:migrate (which doesn't have the hstore in the search path), then the rest of the schemas via Storey::Migrator.migrate. The rest of the schemas have the column.

Fix: don't use db:migrate. Use Storey::Migrator.migrate to migrate the public schema as well. Which means Storey has to worry about dumping the db structure itself.

rake storey:migrate VERSION=xxxxxxx seems to ignore version

Developer sees exception when switching while in a database transaction

This usually just occurs when running acceptance tests with transactions on -- it would give pretty strange errors complaining about tables not existing. Storey should fail early and give the error that it can't switch while in a database transaction. Transactions aren't valid across switching of PG schemas anyway.

storey:rollback in Rails 5.x breaks

# Gemfile.lock
    activerecord (5.2.0)
      activemodel (= 5.2.0)

$ rails storey:rollback
NoMethodError: undefined method `rollback’ for ActiveRecord::Migrator:Class

This is because storey uses ActiveRecord::Migrator.rollback in

storey/lib/storey/migrator.rb

Line 36 in 68cab0c

::ActiveRecord::Migrator.rollback(

which doesn't exist in the ActiveRecord::Migrate class of active-record 5.2.0.

https://www.rubydoc.info/gems/activerecord/ActiveRecord/5.2.0/Migrator

Proposal

We can create a RollbackStrategy class/multiplexer that executes the correct rollback implementation based on the Rails version.

Apartment gem users also experienced this and implemented a similar fix influitive/apartment@06d80f0

Should be able to dump sql as instead of the ruby schema.rb to properly represent the structure of the db

• Set the schema dump format to :sql in Rails
• Run the migration. It should update structure.sql

Storey should clean up after itself, when it is done duplicating

A bunch of dumps are left in tmp/schema_dumps -- not good!

Move away from autoload

I heard there was a bunch of issues regarding this. Let's move away from it before we run into these.

When suffix is on and user creates schema without db structure, the schema created does not have the suffix

Storey should raise exception if attempting to duplicate from a nil schema

Developer can create new schemas reflecting the public schema that has column types that are not native to ActiveRecord

If there are data types that are not native types in ActiveRecord, then the schema.rb does not reflect the definitive structure any longer. This happened when I created a table with a column of the data type hstore. The schema.rb couldn't export that particular table.

Since Storey uses the schema.rb file to load into any new schemas created, this is obviously will be a problem. What we can do instead of use the existing duplicate functionality, but always copy from the public schema, and only copy the structure.

rollback passes the public schema twice

This rolls the public schema back twice instead of just once.

Probably because you first do rake db:rollback, then when you go through each schema, public is included in the array so it gets rolled back again. (just a hunch)

relation "some_table_name" does not exist at character 13

I have not gotten to the heart of the matter, but I get this error when running a spec. However, this spec only fails when one before it that uses the same schema is created.

• Rails 3.2.3

   ActiveRecord::StatementInvalid: PG::Error: ERROR:  relation "asset_groups" does not exist at character 13
: INSERT INTO "asset_groups" ("created_at", "name", "updated_at") VALUES ($1, $2, $3) RETURNING "id"
from /Users/ramon/.rvm/gems/ruby-1.9.2-p290/gems/activerecord-3.2.3/lib/active_record/connection_adapters/postgresql_adapter.rb:1152:in `get_last_result'

For documentation's sake, here are some guesses as to why this happens:

• Cached statement does not properly take into account search_path. Because the 2nd spec just truncates the schema, and runs the test on that schema, then perhaps there's some funkiness. This is a wild idea though. I looked at the Rails 3.2.3 code and poked around but did not get far.
• The schema truncation within a transaction does some funky stuff to the state of the db.

If the public schema is duplicated into a new schema while in a transaction, loading sql complains schema does not exist

It seems that this bug appeared with later versions of PostgreSQL installed (9.2). See this for hints.

Perhaps if loading the structure occurs while in a transaction, we should just commit the transaction and then drop down to the command line and call psql.

Validate the schema name

Can't seem to find documentation on what schemas are valid, but what we know so far:

• no spaces
• does not start with a number
• no underscores
• no dashes

Questions:

• Is there a length limit?

@corroded, is this correct?