rkrage / pg_party Goto Github PK
View Code? Open in Web Editor NEWActiveRecord PostgreSQL Partitioning
License: MIT License
ActiveRecord PostgreSQL Partitioning
License: MIT License
I was just wondering if there's been any talks/RFCs/proposals to inline some pg_party functionality into Rails core?
In particular, it feels kinda bad that you have to do all this work essentially just to scope a class/ActiveRelation to a particular child table.
In particular, in my app, we have Inventory objects, and each inventory has its own set of partitioned relations such as inventory_items
, departments
, etc, and due to some issues with locking parent partitioned tables, when doing mass imports into these tables, we need to reference the partitioned child table directly, which means using in_partition
and caching this child-table-specific class in the PgParty cache.
That means if we have X inventories and Y partitioned relations, the size of this cache will grow to X*Y, and no our app probably hasn't hit a point where we should worry, but it still feels like a bad leak (especially if we add any more partitioned-by-inventory resources).
So I was thinking it'd be nice if Rails had something like Model.with_table_name('foo')
or something like that that could be created adhoc to override the table name for queries (and any other pg_party needs). Does something like that exist? Have there been any talks?
Seeing stuff like this:
Using `bin/rails db:structure:dump` is deprecated and will be removed in Rails 6.2. Configure the format using `config.active_record.schema_format = :sql` to use `structure.sql` and run `bin/rails db:schema:dump` instead
Merging ("uuid_string_ranges"."some_string" >= 'a') and ("uuid_string_ranges"."some_string" < 'z') no longer maintain both conditions, and will be replaced by the latter in Rails 6.2. To migrate to Rails 6.2's behavior, use `relation.merge(other, rewhere: true)`
Are you planning to change specification in Gemfile to allow activerecord >= 6?
To allow for partition tables like this:
CREATE TABLE measurement_year_month (
logdate date not null,
peaktemp int,
unitsales int
) PARTITION BY RANGE (EXTRACT(YEAR FROM logdate), EXTRACT(MONTH FROM logdate));
It will be pretty difficult to convert something like this to Arel for use in the model scopes. Maybe for the first iteration we could just refrain from injecting scope methods for complex keys?
I want to take a large table I have and split it per user.
Is it possible to:
Might be able to accomplish this by querying the pg_inherits
table: https://www.postgresql.org/docs/10/static/catalog-pg-inherits.html
This will require a custom implementation of the bigserial
column type that only exists in Rails 5.x
Hi, I hope you are doing well. I'm have some performance issues with huge table (millions of records) so I'm interested in convert this table to a partitioned table because it fits perfectly with what we need. ¿Is there a way to alter this table to make it partitioned or is required create another table?
Update readme examples for the new features on master
The 2.7 builds have a ton of deprecation warnings like:
Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
This is due to the work to separate positional and keyword args: https://www.ruby-lang.org/en/news/2019/12/12/separation-of-positional-and-keyword-arguments-in-ruby-3-0/
Just found this gem, and it looks like an excellent start to help manage partitions under PGSQL 10.
There are a couple of suggestions that I have. I have decided to use pg_partman to help manage my native child partitions, and this extension automatically creates a pool of new partitions before I need them.
The partitions are named "master_table_p0", "master_table_p1" etc based on the single value in my range that I am using as the partition key.
It would be nice in pg_party if I could provide the name of the partition when creating child partitions so that I can keep the same convention. The name convention will also help me manage old partitions that I no longer need.
Also, creating a child partition does not seem to allow for the use of a template table so that the indexes can be copied and built on the new partition.
It might make sense to add functionality to create_range_partition_of
and create_list_partition_of
to create child partitions from template tables. This would be useful to copy over indexes and constraints.
There's no changelog entry for version 1.5.0 (which we're currently on making it rather moot) but at some point we will want to upgrade to any new version (I'm upgrading gems for the Ruby 3.0 keyword change and found pg_party-1.5.0/lib/pg_party/adapter/postgresql_methods.rb:62: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
). A changelog (and preferably a ommitment to semantic versioning) would help with the upgrade process by making it clearer when there are safe/dangerous changes we need to handle.
Version: Rails 6, PostgreSQL 12
# config/initializers/pg_party.rb
PgParty.configure do |c|
c.create_with_primary_key = false
end
rails db:migrate
class CreateOrdersWithTablePartitioning < ActiveRecord::Migration[6.1]
def up
create_range_partition :orders, partition_key: :created_at do |t|
t.string :product_name
t.decimal :price
t.integer :status
t.references :product
t.timestamps
end
execute "ALTER TABLE public.orders ADD PRIMARY KEY (id, created_at);"
create_range_partition_of(
:orders,
name: "orders_#{Date.today.year}_#{Date.today.month}",
start_range: Date.today.beginning_of_month,
end_range: Date.today.next_month.beginning_of_month
)
end
def down
drop_table :orders
end
end
Add model bypassing migration
rails g model Order --skip-migration
Remove the warning of "WARNING: Active Record does not support composite primary key."
# /app/models/order.rb
class Order < ApplicationRecord
self.primary_key = :id
end
Thanks for author's awesome work.
Add some notes about configuration and caching behavior.
I noticed that unless I switch from schema.rb
to structure.sql
, I get:
PG::InvalidObjectDefinition: ERROR: table "x" is not partitioned
when calling create_list_partition_of
in rspec.
Is there any way to run tests on partitioned tables without having to switch to structure.sql
?
Hey there, thanks for creating and maintaining this gem!
I was wondering what we need in order to also support the recently released Rails 7.2 version. From what I can tell, it's been running fine on 7.2.0.beta3
but is now being held back by the ActiveRecord < 7.2
requirement in the gemspec.
Is there anything we need to update or something I can help with to make pg_party
work with Rails 7.2+?
Thank you!
Hi there,
this question may be a bit too "broad" but i wonder if these "partition tables" can be used for "personal report data".
That means big tables split up in maybe hundreds of "partitioned tables" which belong to several certain users.
(Yearly report for user X, monthly report for user Y and so on)
Are there known drawbacks if a user has, for example 10 partitioned tables?
Is this a use case for "partitioned tables" in postgres?
Thanks in advance!
Niklas
Having to manually create partition tables based on new criteria (dates for range partitions) or new list-related records leaves a lot of "figure it out for yourself!" up to the gem user. Would be really nice to have something in place to mitigate the looming question of lifetime partition maintenance.
I'm on Rails 5.2 and I see this when I instantiate a record from a partitioned table:
WARNING: Active Record does not support composite primary key.
partitioned_inventory_items has composite primary key. Composite primary key is ignored.
I've heard that AR doesn't support composite primary keys, but yet pg_party seems to suggest in the README that you use composite primary keys? I'm also wondering how this hasn't already come up in the issues...
To implement this, it might make sense to delegate migration methods to a class that wraps the postgres adapter
For better code quality / maintainability
When migrations are run and a new schema.sql file is generated, it contains the user-created partition tables for any partitions in the database. This is expected, but also seems wrong since in a multi-dev environment, there will be constant differences in this file since everyone will have different partitions.
Perhaps there's a way to instruct the schema generator to only dump the master partition record.
At first, thanks for the gem!
I have couple ideas, that I hope could make using the gem little easier.
Model.partitions(start_date: Date.today) or Model.partitions(end_date: Date.today)
Example: I need to find the last record, but not later than today.
Model.partitions(order: :asc) or Model.partitions(order: :desc)
Hello! We're in the process of upgrading to Ruby 3, and we're running into an issue with create_range_partition
:
# How we're using it in our DB migration
def up
create_range_partition(:table_name, partition_key: :created_at, primary_key: [:id, :created_at]) do |t|
...
end
end
# Error that occurs due to the Ruby 3 keyword arguments change
wrong number of arguments (given 2, expected 1; required keyword: partition_key)
Looking through the stacktrace, the issue seems to come from this module in pg_party
:
module PgParty
module Adapter
module PostgreSQLMethods
ruby2_keywords def create_range_partition(*args, &blk)
PgParty::AdapterDecorator.new(self).create_range_partition(*args, &blk)
end
...
end
end
The arguments are coming in as:
["table_name", {:partition_key=>:created_at, :primary_key=>[:id, :created_at]}]
So that when it bubbles up to the AdapterDecorator
, it can't parse out the partition key properly:
def create_range_partition(table_name, partition_key:, **options, &blk)
create_partition(table_name, :range, partition_key, **options, &blk)
end
At least, that's what I understand it to be doing? Are we using create_range_partition
incorrectly, or should there be a fix? Thanks so much!
Hi,
I just wonder how to check if a partition table for a given range exists.
The Readme describes the SomeRangeRecord.partitions functions, which gives just cryptic table names, but not telling for which ranges they were created.
What is the recommended way to insert some value if it is unknown whether a partition table exists for that range? Catch an exception, create a partition, and retry?
If schema_exclude_partitions = false
, how do ensure there is a writable partition in development/test environments?
I've caught an ArgumentError when I call model_name
method for my partition table.
class TMainDatum < ApplicationRecord
range_partition_by :date
end
For the parent class it's ok:
TMainDatum.model_name
=> #<ActiveModel::Name:0x007ffd671791b8
@collection="t_main_data",
@element="t_main_datum",
@human="T main datum",
@i18n_key=:t_main_datum,
@klass=TMainDatum (call 'TMainDatum.connection' to establish a connection),
@name="TMainDatum",
@param_key="t_main_datum",
@plural="t_main_data",
@route_key="t_main_data",
@singular="t_main_datum",
@singular_route_key="t_main_datum">
and for the child class it raises an error:
TMainDatum.in_partition('t_main_data_2017_34').model_name
ArgumentError: Class name cannot be blank. You need to supply a name argument when anonymous class given
from /ruby/gems/2.3.0/gems/activemodel-5.1.4/lib/active_model/naming.rb:149:in `initialize
Maybe you could simply to add self.name
to ModelDecorator, couldn't you?
def child_class(table_name) do
Class.new(__getobj__) do
...
def self.name
superclass.name
end
end
end
The real usage of model_name
dug into activerecord-import: https://github.com/zdennis/activerecord-import/blob/master/lib/activerecord-import/import.rb#L650
Is it possible to call create_list_partition_of
in a rake task?
I've added require 'pg_party'
at the top of the file but haven't had any luck with things like:
create_list_partition_of()
PgParty.create_list_partition_of()
Is there a different way I should be accessing this method outside of migration context?
Since the partitioned tables will/can get created outside the scope of a migration, it makes sense to add those helper methods (add_index) to the model helpers.
This would compliment the existing helpers to manage creating partitioned tables, like "create_partition".
Hello,
It would be nice to create a simple tutorial or whole sample app.
Currently, it's not clear how to use work with it. Plus it can bring more people to use this gem
Thanks
Good day.
Does anybody know how to migrate from https://github.com/fiksu/partitioned to pg_party
?
We have a super old project and try to update.
The partitioned
gem uses other schema for partitioned tables, e.g.:
p201511
, schemaname: my_table_name_partitions
my_table_name
, schemaname: public
Second, the partitioned
gem doesn't use native PG table partitioning :(
We use partitioning by date.
Thanks,
Aleksei.
Hi,
I'm just trying and playing around with it, and I found, that the migrations creating partitions are missing in db/schema.rb, thus the database cannot be recreated with the common rails tools.
I have partitioned by range tables in a separate database schema. When I try to run db:migrate
I get an exception:
pg_dump: error: cross-database references are not implemented: *.time_tracking.entries_y2022_m12_date_idx
rake aborted!
failed to execute:
pg_dump --schema-only --no-privileges --no-owner --file /Users/igor/workspace/onetribe/db/structure.sql onetribe_development -T *.time_tracking.entries_y2022_m12_date_idx -T *.time_tracking.entries_y2022_m11 -T *.time_tracking.entries_y2022_m12_pkey -T *.time_tracking.daily_entries_y2022_m12_member_id_task_uuid_date_idx -T *.time_tracking.daily_entries_y2022_m11_pkey -T *.time_tracking.daily_entries_y2022_m12_pkey -T *.time_tracking.daily_entries_y2022_m11_company_id_idx -T *.time_tracking.entries_y2023_m1 -T *.time_tracking.daily_entries_y2023_m1_company_id_idx -T *.time_tracking.daily_entries_y2022_m12 -T *.time_tracking.entries_y2023_m1_pkey -T *.time_tracking.entries_y2022_m11_pkey -T *.time_tracking.daily_entries_y2023_m1_pkey -T *.time_tracking.daily_entries_y2022_m11 -T *.time_tracking.entries_y2022_m11_date_idx -T *.time_tracking.entries_y2023_m1_date_idx -T *.time_tracking.entries_y2022_m12_company_id_member_id_task_uuid_idx -T *.time_tracking.entries_y2023_m1_company_id_member_id_task_uuid_idx -T *.time_tracking.entries_y2022_m11_company_id_member_id_task_uuid_idx -T *.time_tracking.daily_entries_y2022_m12_company_id_idx -T *.time_tracking.daily_entries_y2023_m1_member_id_task_uuid_date_idx -T *.time_tracking.entries_y2022_m12 -T *.time_tracking.daily_entries_y2023_m1 -T *.time_tracking.daily_entries_y2022_m11_member_id_task_uuid_date_idx
Please check the output above for any errors and make sure that `pg_dump` is installed in your PATH and has proper permissions.
Tasks: TOP => db:schema:dump
(See full trace by running task with --trace)
Hello,
Currently, I'm trying to define a 3 key range partition, but I can't make it work (in terms of PostgreSQL query).
First of all the code:
`
create_range_partition_of \
:scores,
name: "scores_2007".to_sym,
partition_key: ->{ "recent, deleted, played_at" },
start_range: [false, false, "1980-01-01"],
end_range: [false, false, "2008-01-01"]
`
It all works, the partitions get created and I can see them in PGAdmin, but when running a query that is expected to select a certain partition it does not work.
explain select * from scores where recent = false and deleted = false and played_at = DATE '2006-03-03'
- It checks on all partitions.
However, if I only do a 2 key range partition all works fine.
`
create_range_partition_of \
:scores,
name: "scores_2007".to_sym,
partition_key: ->{ "recent, played_at" },
start_range: [false, "1980-01-01"],
end_range: [false, "2008-01-01"]
`
explain select * from scores where recent = false and deleted = false and played_at = DATE '2006-03-03'
- Checks only scores_2007 partition.
Do you have any Idea if this is doable, or there are certain constraints that don't allow it to be done with 3 keys?
Thank you!
See edit below, this isn't even possible for us :(
Hey there, this isn't an issue or bug, more of a question. I'm running on a pretty old version of pg_party (which isn't even in your official tag list lol), this is on an old Rails app running Rails 5 & Ruby 2.2. We also have a new app running Rails 7 & Ruby 3.2.
Both of these apps share the same database, postgres 14.7. Back when we first started using pg_party we were running postgres 10 and we set things up to create the partition indexes after a partition is created. It looks like since pg_party 1.0, we no longer need to manually create the partition indexes, is that correct?
We want to create a new nested partition table (partition by list, then partitioned by range), which only exists in new versions of pg_party.
My thinking is we should be able to upgrade to 1.6.0 on our Rails 5 app, remove the code that creates indexes on partitions after they're created, and everything else can stay as-is. Or maybe I can just leave all that code as-is and it'll continue to create the indexes manually?
Then I can create my new partitioned table with the subpartions using the features from the latest gem version and we're off to the races.
Does that track with you @rkrage?
EDIT
Due to Ruby & Rails versions, the highest we can go with pg_party is 1.0.1, which is not too far off from what 0.7.3 offers. I think we're going to have to resort to some hackery to accomplish our goals in created the partitioned and subpartitioned tables.
Final Edit:
We're going to run the main migration in our legacy app (running pg_party 1.0) to create the list-based table partition, then use our modern app (pg_party 1.6.0) to create the sub-partitions on the fly as needed
Maybe create anonymous class in the model decorator. Something like:
def in_partition(name)
Class.new(__getobj__) do
self.table_name = name
end
end
Could you clarify if partitioning by foreign key is supported?
I see this example in the ReadMe:
create_list_partition :some_list_records, partition_key: :id do |t|
t.integer :some_foreign_id
t.text :some_value
t.timestamps
end
But is it possible to use partition_key: :some_foreign_id
like so:
create_list_partition :some_list_records, partition_key: :some_foreign_id do |t|
t.references :some_foreign, foreign_key: true
t.string :name
t.timestamps
end
?
So that if I want a partition per :some_foreign_id
, could I do something like:
SomeForeign.all.map do |foreign|
create_list_partition_of :some_list_records, name: foreign.name, partition_key: :some_foreign_id, values: [foreign.id]
end
with each partition getting the respective name
of each foreign record?
Since ActiveRecord does some crazy aliasing when joins are involved, I think we'll need to do some subquery magic for partition_key_eq
and partition_key_in
to be chainable.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.