Comments (10)
That would be cool. If you point me to where I could get started, I could try to implement it.
I guess the following steps are required:
- Define how sharding keys should be specified in
pgcat.toml
. Possible solutions could be a list of entries like["data.id","dependent.data_id"]
or a map like{ data = id, dependent = data_id }
- Adopt serde to serialize the entries into a hashmap that maps tables to sharding keys.
- Adopt
sqlparser
to check for all sharding keys in Hashmap
I guess I know what I should do to solve the first two steps. But I'm not really sure how to do the last one.
from pgcat.
Thanks! I will look into it. I will hopefully find some time in the next couple of days.
from pgcat.
Hi there,
Thank you for filing an issue. Could you give me a couple examples when the custom SQL syntax breaks compatibility with existing systems?
Thanks!
from pgcat.
Hi,
First of all, thanks for all your contributions, this is an awesome project.
I would like to use pgcat with postgrest. It currently doesn't support the custom SQL from pgcat.
And I would expect that ORM systems also don't support the custom SQL syntax.
from pgcat.
Ah yes, this is expected.
For this reason, we've introduced a couple more ways to handle sharding:
- using comments and parsing them with regex, e.g.
/* sharding_key: 5 */ SELECT * FROM table_x WHERE id = 5;
Seeregex_*
settings inpgcat.toml
. - automatically by parsing the SQL with
sqlparser
crate, that can be configured withautomatic_sharding_key
setting inpgcat.toml
What I usually see people do is extend their ORMs to allow for working with pgcat and sharding. Of course that's not possible or desirable for everyone so automatic sharding is the north star for us. If you could test it on your end and tell us how it's working for you, that would be great. We also welcome PRs if you find any areas of improvement.
Your proposal could also work, although I'm not entirely sure why you can't issue a SET SHARD TO
query instead of the query you've proposed. Is that something inherent to postgrest that limits you to only writing queries against tables?
Let me know if any of those work for you.
from pgcat.
Thank you for your suggestions. I will have look at them. Especially the automatic sharding sounds interesting. Somehow I missed that feature before.
I will report my findings.
Postgrest maps the CRUD operations on a table to the http methods: POST, GET, PATCH, DELETE on a specific route. That's why the capabilities apart from the table operations is somewhat limited.
from pgcat.
The automatic sharding looks great and it almost solves my use case. To entirely solve my problem I would need to define automatic sharding keys for multiple tables, but as I understood this is not supported.
I have one table with the actual sharding key and then dependent tables referencing the sharding key as a foreign key. I would need to also shard the dependent tables according to the foreign key.
from pgcat.
We can add support for multiple sharding keys. I was already thinking about that, because foreign keys is a very common use case.
from pgcat.
For 3, take a look at query_router.rs
, specifically: https://github.com/levkk/pgcat/blob/master/src/query_router.rs#L568
Convert this to a for each sharding key loop and match all of them. We dedup on shard values later and only allow to query one shard per query.
from pgcat.
I finally had time to work on the sharding keys. I created a PR that should enable multiple automatic sharding keys. If I'm not completely mistaken it was surprisingly simple to implement.
from pgcat.
Related Issues (20)
- Error with unclear message HOT 2
- support sql hint just execute query to primary node
- errors on insert with prepared statements HOT 2
- Health check command HOT 2
- request:release plan for pgcat in PostgreSQL 16 HOT 1
- Support Service annotations in the helm chart HOT 1
- Add support for negotiating the protocol version with the client
- Errors in some libraries when using PreparedStatements
- Fails to compile with space in path
- The docs don't link to the APT Repository for pgcat HOT 2
- PGCat not using idle connections, instead spawns new ones. HOT 3
- ERROR pgcat::server: Unknown code: N HOT 1
- PGcat detection from code HOT 1
- Feature request: per-statement-type counters
- Error when prepared Statement in transaction mode
- Ubuntu 24.04 Packages
- support sql admin pool and server
- Invalid SQL Statement in rust integrration test
- "No pool configured for database" when trying to connect to pgcat instance HOT 2
- SELECT ... FOR UPDATE is routed to REPLICA HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pgcat.