copyTable function for Command like renameTable about db

[dinhtrung]

It is quite convenient for some big data tables where we rotate data by monthly basis. For e.g, I usually have to set the following commands to run in monthly basis for big tables (several million rows):

CREATE TABLE log_events201609 LIKE log_events;
INSERT INTO log_events201609 SELECT * FROM log_events WHERE event_date < '2016-09-01';
DELETE FROM log_events WHERE event_date < '2016-09-01';

I know there is table partitioning in all DB engine, but it quite dangerous to group all the important logging data into one table. Also, after soft-partitioning the data like that, we can easily export the whole monthly report to flat files for reference later.

[zuoRambo]

backup history data before data migration

[SilverFire]

Might be implemented.

[nineinchnick]

I usually do something like CREATE TABLE copy AS SELECT * FROM original but that doesn't include table constraints. If you add LIMIT 0 you only copy the structure, without the data.

[samdark]

What's the use case?

[zuoRambo]

@nineinchnick CREATE TABLE copy AS SELECT * FROM original that's useful on mysql platform but not well on mssql

[samdark]

Useful for what?

[zuoRambo]

@samdark copyTable

[cebe]

The question here is: Why would you ever want to copy a table?

[dynasource]

The question here is: Why would you ever want to copy a table?

As both copying & tables are quite generic, you are ignoring users creativity in general with this question.

In fact, these are so generic that there will be infinite usecases.

Therefore 'why' is not the question, but 'how'.

[vakalaka]

Here is the usecase: I've got a huge table to be updated from the external source (for example: geo blocks), it may be significantly changed so to actually update is unnecessary hard.

So I should do:

1. copy table structure (without index keys)
2. fill the new temp table with actual data
3. create index keys
4. drop the old table
5. rename the new table

Everything but the first is perfectly done with createCommand(). And it is rather complicated, because I should check if the temp table already exists, truncate it or drop and create. And I'm forced to either store the table definition in two places (migration and data update) or use a cheat like CREATE TABLE ... AS SELECT * FROM ... LIMIT 0

Actually the copyTable() method should not be that hard to implement. We've got the yii\db\Connection::getTableSchema() method and yii\db\TableSchema::$columns property, so the main question is how to force createTable() to use the array of yii\db\ColumnSchema and that will do.

[samdark]

Note that table schema is usually cached.

I'm OK having a method but this way of updating data doesn't sound familiar to me. What are pros compared to simple update in this case? Also I guess original table should be locked during this process else you're going to lose data written into original table during it.

[vakalaka]

In my case it is the geo blocks table (about 3M of rows) and the key is start_ip-finish_ip. There may be (and there are) different keys in the update data. The IP ranges may match, intersect, separate or differ in other ways. It is possible to find and update (actually delete and insert I think) ranges but it is a bit harder then to create new table and replace the old one after data collection.

The second moment (in this particular case) is that the update/insert operations on the table with 3M rows of data and indices is very slow (in terms of DB). It is much faster to insert in empty table without indices and then to create indices. It is very useful approach for the dictionary-like data tables but you are to be sure about the input data (duplicates etc.). The faster update, instantly replaced table so no locks, no read lags.

And in my case the table is "read-only" after the update so I do not need to lock it as there are no any writes to the table. It is a rare case, but it happens...

[samdark]

Have you tried SET foreign_key_checks = 0;, SET unique_checks = 0; etc.?

[vakalaka]

Nope. But even if it fixes the data insertion timing problem, you still have to fight the search/replace ranges problem...

In other words everything is possible but is it worth it?!

Anyway it is just a usecase (for copyTable()), I think one of many...

[samdark]

OK.