Comments (11)
One thing we need to pay attention to on our end is that each call to COPY creates a separate stripe. We'd need to implement this in a way that integrates nicely with the Insert APIs for fdws. I see two alternatives here:
1/ We implement BeginForeignModify / ExecForeignInsert / EndForeignModify proper. One thing we haven't yet discussed is the transactional semantics. If the user is inserting multiple entries and the operation fails, how do we roll it back? This requires some thought.
2/ We use foreign table inheritance (coming in 9.4?) and Create Table cstore_table_child1 As (Select columns From regular_table);. We already intercept the utility hook for the Copy command and the Create Table As uses the same machinery. So this should be relatively easy to integrate.
The question is, does option 2/ provide any benefit to the user? @jberkus, what do you think?
from cstore_fdw.
I don't understand option (2).
The reason I filed this issue is that the primary use of cstore_fdw for me today involves transferring large blocks of data from a Postgres table to a cstore table. Currently, I have to dump that data to csv and re-import it using COPY to get it into the cstore table. If Postgres had a COPY table TO other_table, then this wouldn't be a limitation.
BTW, if COPY fails in the middle, it's supposed to roll back too.
from cstore_fdw.
BTW, if we need to add a new stripe every time anyone does a bulk load, we're going to need a utility to rebuild cstore files eventually.
from cstore_fdw.
COPY rolls back if it fails in the middle. If you explicitly start a transaction however, issue COPY and a second COPY, and the second COPY fails in the middle, today, the first COPY still goes through. That's something we need to fix.
The Begin / Insert / End APIs for foreign data wrappers make transactional semantics more explicit, that's why we figured we'd think about them when we got to it. Once we define the semantics for Insert here, we'd then use them for Update / Delete too.
The second option I mentioned involves the following:
a/ The user creates a parent cstore table.
b/ The user creates a child cstore table using the upcoming (?) foreign table inheritance feature. When doing this, the user further uses Create Table cstore_table_child1 As (Select columns From regular table);
The difference here is that cstore_fdw can see the Create Table As command as part of the utility hook that it intercepted (so it's similar to Copy in that sense). We then need to direct the read tuples in a manner that's similar to what we do with Copy.
This naturally isn't as nice as implementing fdw's Begin / Insert / End. I'm just trying to explore our option space here.
from cstore_fdw.
Just wanted to note that as an interim solution until we implement "INSERT INTO ... SELECT ...", users can use unix pipes to achieve a similar thing:
psql -d postgres -c "COPY (SELECT * FROM lineitem) TO STDOUT" | psql -d postgres -c "COPY lineitem_cstore FROM STDIN"
from cstore_fdw.
As we had a lot of questions about this feature in the past two months, I'm posting a quick update here.
It looks like the best way to implement this feature is through PostgreSQL FDW Insert APIs. One decision we deferred making early on was whether or not to provide transactional semantics. The Insert APIs provide transaction related info, but getting transactions right with foreign tables is significantly harder.
What we heard from the users that we talked to was that they needed the "INSERT INTO .. SELECT" feature much more than they needed transactions.
If you're a user who is interested in this thread, and have comments about transactions, please feel free to share them with us. We are compiling feedback and we'll make a decision on this next month.
from cstore_fdw.
I need this feature, like, yesterday.
While I would like it to be transaction-safe, that's pretty far down my list of priorities. And the issue of transaction-safety for cstore_fdw is a fairly thorny one anyway; there doesn't seem to be any good way to implement it. You'd need to make a full file copy every time you wanted to start a transaction.
from cstore_fdw.
+1 need table copy is more important than transactional symantics
from cstore_fdw.
Support for "INSERT INTO cstore_table SELECT ..." was added to the develop branch. We will merge it into master branch in 2 months.
from cstore_fdw.
Wow, awesome! For me this is what makes cstore useful in production.
from cstore_fdw.
Merged into master. Closing.
from cstore_fdw.
Related Issues (20)
- select count(1) from table where datecolum = '2020-02-01' slow HOT 4
- Missing dependencies in makefile targets
- Cant ./configure in CentOS 07
- How to make use of parallelization with cstore_fdw? HOT 2
- Tests not running on Travis, and missing CircleCI configs
- Support for Heroku Postgres / Citus Data addon? HOT 1
- Is cstore_fdw still supported by developers? HOT 3
- segfault during copy into foreign table HOT 2
- hll build error against v13 HOT 5
- Does cstore_fdw Support Parallel COPY into the Same Table?
- Integer overflow in cstore_clean_table_resources()
- How to insert load / into cstore foreign tables in order
- Impact of number of partitions on cstore tables
- Skip indexes are not in use HOT 2
- make installcheck failed, "ERROR: could not open file ... for writing: Permission denied" HOT 1
- Support for PostgreSQL15 HOT 1
- v1.7.0 download files does not support PostgreSQL 13 HOT 1
- 目录不存在
- Unable to upgrade from cstore_fdw to citus
- pg14 is support?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cstore_fdw.