Partition by ForeignKey? about pg_party

[rkrage]

Okay, yeah this should totally work. They introduced the ability to define PKs, foreign keys, and indexes on partitioned tables in PG 11. One small note about the above code sample. There are actually helper methods in partitioned model classes that will let you create partitions without directly using ActiveRecord::Base.connection. The helpers imply the partition key and figures out what underlying method to call (create_list_partition_of vs create_range_partition_of). This should probably be documented better.

SomeListRecords.create_partition(name: "some_unique_table_name", values: [some_foreign.id])

[rkrage]

That's a great question and I'll have to get back to you. Something like that might be supported in PG 12+

[mrlevitas]

I tried this setup and it seems to work (I am running postgres 12.3 though), but I'm not sure if there are hidden problems that can occur in regular usage.

Given the above setup, this spec passes:

describe SomeListRecord do
  describe 'partitions by some_foreign' do
    let!(:some_foreign_1) { create :some_foreign }
    let!(:some_foreign_2) { create :some_foreign }

    before do
      [some_foreign_1, some_foreign_2].each do |some_foreign|
        ActiveRecord::Base
          .connection
          .create_list_partition_of(
            :some_list_records, name: some_foreign.name, partition_key: :some_foreign_id, values: [some_foreign.id]
          )
       end
    end

    let!(:some_list_record_1)  { create :some_list_record, some_foreign: some_foreign_1 }
    let!(:some_list_record_2) { create :some_list_record, some_foreign: some_foreign_1 }
    
    let!(:some_list_record_3) { create :some_list_record, some_foreign: some_foreign_2 }

    it 'returns some_foreign_1 records' do
      expect(SomeListRecord.in_partition(some_foreign_2.name).all).to contain_exactly(some_list_record_1, some_list_record_2)
    end

    it 'returns some_foreign_2 record' do
      expect(SomeListRecord.in_partition(some_foreign_2.name).all).to contain_exactly(some_list_record_3)
    end

    it 'has 2 corresponding partitions' do
      expect(SomeListRecord.partitions).to contain_exactly(some_foreign_1.name, some_foreign_2.name)
    end
  end
end

[mrlevitas]

awesome, thanks for the response.

[mrlevitas]

just fyi, SomeListRecords.create_partition(name: some_foreign.name, values: [some_foreign.id]) instead of

ActiveRecord::Base
          .connection
          .create_list_partition_of(
            :some_list_records, name: some_foreign.name, partition_key: :some_foreign_id, values: [some_foreign.id]
          )

fails the test above.

Factorybot creates instances with id: nil, even though .errors is empty and .persisted?/.valid? are true.

[rkrage]

Huh, that's interesting. What does your SomeListRecords class look like?

[mrlevitas]

these are the models:

class SomeListRecords < ActiveRecord::Base
  list_partition_by :some_foreign_id

  belongs_to :some_foreign
  belongs_to :another_foreign
end

class SomeForeign < ActiveRecord::Base
  has_many :messages
  has_many :some_list_records
end

---

these are the migrations:

    create_list_partition :some_list_records, partition_key: :some_foreign_id do |t|
      t.string     :uuid, null: false
      t.references :some_foreign, foreign_key: true, index: false
      t.references :another_foreign, foreign_key: true
      t.timestamps null: false
    end



    add_index :some_list_records_template, [:uuid, :some_foreign_id]
    add_index :some_list_records_template, [:some_foreign_id, :created_at]

    create_table :some_foreigns do |t|
      t.string :name, null: false, unique: true
      t.timestamps null: false
    end

---

and these are my factories:

FactoryBot.define do
  factory :some_list_record, class: SomeListRecord do
    sequence(:uuid) { |n| 'uuid_' + n.to_s }
    association :some_foreign, factory: :some_foreign
    association :another_foreign, factory: :another_foreign
  end
end

FactoryBot.define do
  factory :some_foreign, class: SomeForeign do
    sequence(:name) { |n| 'some_foreign' + n.to_s }
  end
end

anything pop out?

[mrlevitas]

created a sample app that demonstrates this:
https://github.com/mrlevitas/partition-example

hope it helps debugging.

[rkrage]

Thanks for putting this together. I have a pretty busy week but can hopefully look into it soon

[rkrage]

Sorry it took me so long to look into this. I think this is actually an rspec issue. By default, it starts a transaction for each spec and rolls it back afterwards. This is generally fine in most cases where you're just performing inserts, updates, deletes. But when you introduce DDL operations things start to break down.

So by running ActiveRecord::Base.connection.create_list_partition_of, I think that was being executed in a separate connection outside of the transaction.

If you'd like to perform DDL in your tests you might want to consider using database_cleaner with the truncation strategy: https://github.com/rkrage/pg_party/blob/master/spec/spec_helper.rb#L59-L69

It's also possible that truncation is natively supported in rspec as well and I'm just out of the loop.

[rkrage]

Gonna close this for now, but feel free to open it again if you're still experiencing issues!