fljdin / mysql_migrator Goto Github PK

View Code? Open in Web Editor NEW

3.0 3.0 0.0 116 KB

Tools for MySQL to PostgreSQL migration

License: PostgreSQL License

Makefile 2.04% PLpgSQL 96.68% Dockerfile 1.28%

mysql_migrator's People

Contributors

Stargazers

Watchers

mysql_migrator's Issues

Changed db_migrator API for partitioning

My commit cybertec-postgresql/db_migrator@1d85415 made some changes that affect this project:

the column expression in partitions and subpartitions has been renamed to key
for hash partitioning, values now only has to contain the remainder (I figure out the modulus from the size of the result set)

Sorry for the inconvenience.

Convert SET type to TEXT[]

A SET type represents a list of string in MySQL and should be translated to TEXT[] in PostgreSQL

MySQL

SELECT special_features FROM film WHERE special_features IS NOT NULL LIMIT 1;

+----------------------------------+
| special_features                 |
+----------------------------------+
| Deleted Scenes,Behind the Scenes |
+----------------------------------+
1 row in set (0,001 sec)

PostgreSQL

SELECT special_features FROM film WHERE special_features IS NOT NULL LIMIT 1;

            special_features            
----------------------------------------
 {"Deleted Scenes","Behind the Scenes"}
(1 row)

Replace ENUM type by check constraints

Given a table with a enum type defined as follow:

MySQL [sakila]> desc film;
+----------------------+------------------------------------+------+-----+-------------------+
| Field                | Type                               | Null | Key | Default           |
+----------------------+------------------------------------+------+-----+-------------------+
| ...                  |                                    |      |     |                   |
| rating               | enum('G','PG','PG-13','R','NC-17') | YES  |     | G                 |
| ...                  |                                    |      |     |                   |
+----------------------+------------------------------------+------+-----+-------------------+

Should be transform like this, with a table-related check constraint:

ALTER TABLE sakila.film ADD CONSTRAINT film_rating_chk CHECK (rating IN ('G','PG','PG-13','R','NC-17'));

                                      Table "sakila.film"
        Column        |            Type             | Collation | Nullable |      Default      
----------------------+-----------------------------+-----------+----------+-------------------
...
 rating               | text                        |           |          | 'G'::text
...
Check constraints:
    "film_rating_chk" CHECK (rating = 
      ANY (ARRAY['G'::text, 'PG'::text, 'PG-13'::text, 'R'::text, 'NC-17'::text]))

Attach sequence to column table

At this time, sequences are built from auto_increment attribute.

Each column with an auto_increment must have a default value defined as nextval('table_seq').

Migration of MySQL sakila database got errors

When migrating sakila database, call to db_migrate ends with 27 errors.

Create server objects

CREATE SERVER sakiladb FOREIGN DATA WRAPPER mysql_fdw OPTIONS ( host '127.0.0.1', fetch_size '1000' );
CREATE USER MAPPING FOR florent SERVER sakiladb OPTIONS (username 'sakila', password 'readme');

Run db_migrate()

SET client_min_messages = log;
SELECT db_migrate(plugin => 'mysql_migrator', server => 'sakiladb');

NOTICE:  Creating staging schemas "fdw_stage" and "pgsql_stage" ...
NOTICE:  Creating foreign metadata views in schema "fdw_stage" ...
NOTICE:  Creating schemas ...
NOTICE:  Creating sequences ...
WARNING:  Error creating sequence sakila.actor_seq
DETAIL:  syntax error at or near "START": 
WARNING:  Error creating sequence sakila.address_seq
DETAIL:  syntax error at or near "START": 
WARNING:  Error creating sequence sakila.category_seq
DETAIL:  syntax error at or near "START": 
WARNING:  Error creating sequence sakila.city_seq
DETAIL:  syntax error at or near "START": 
WARNING:  Error creating sequence sakila.country_seq
DETAIL:  syntax error at or near "START": 
WARNING:  Error creating sequence sakila.customer_seq
DETAIL:  syntax error at or near "START": 
WARNING:  Error creating sequence sakila.film_seq
DETAIL:  syntax error at or near "START": 
WARNING:  Error creating sequence sakila.inventory_seq
DETAIL:  syntax error at or near "START": 
WARNING:  Error creating sequence sakila.language_seq
DETAIL:  syntax error at or near "START": 
WARNING:  Error creating sequence sakila.payment_seq
DETAIL:  syntax error at or near "START": 
WARNING:  Error creating sequence sakila.rental_seq
DETAIL:  syntax error at or near "START": 
WARNING:  Error creating sequence sakila.staff_seq
DETAIL:  syntax error at or near "START": 
WARNING:  Error creating sequence sakila.store_seq
DETAIL:  syntax error at or near "START": 
NOTICE:  Creating foreign tables ...
NOTICE:  Migrating table sakila.actor ...
NOTICE:  Migrating table sakila.address ...
NOTICE:  Migrating table sakila.category ...
NOTICE:  Migrating table sakila.city ...
NOTICE:  Migrating table sakila.country ...
NOTICE:  Migrating table sakila.customer ...
NOTICE:  Migrating table sakila.film ...
NOTICE:  Migrating table sakila.film_actor ...
NOTICE:  Migrating table sakila.film_category ...
NOTICE:  Migrating table sakila.film_text ...
NOTICE:  Migrating table sakila.inventory ...
NOTICE:  Migrating table sakila.language ...
NOTICE:  Migrating table sakila.payment ...
NOTICE:  Migrating table sakila.rental ...
NOTICE:  Migrating table sakila.staff ...
NOTICE:  Migrating table sakila.store ...
WARNING:  Error creating view sakila.actor_info
DETAIL:  syntax error at end of input: 
WARNING:  Error creating view sakila.customer_list
DETAIL:  syntax error at end of input: 
WARNING:  Error creating view sakila.film_list
DETAIL:  syntax error at end of input: 
WARNING:  Error creating view sakila.nicer_but_slower_film_list
DETAIL:  syntax error at end of input: 
WARNING:  Error creating view sakila.sales_by_film_category
DETAIL:  syntax error at end of input: 
WARNING:  Error creating view sakila.sales_by_store
DETAIL:  syntax error at end of input: 
WARNING:  Error creating view sakila.staff_list
DETAIL:  syntax error at end of input: 
NOTICE:  Creating UNIQUE and PRIMARY KEY constraints ...
WARNING:  Error creating primary key or unique constraint on table sakila.film_actor
DETAIL:  could not create unique index "film_actor_actor_id_pkey": Key (actor_id)=(1) is duplicated.
WARNING:  Error creating primary key or unique constraint on table sakila.film_actor
DETAIL:  could not create unique index "film_actor_film_id_pkey": Key (film_id)=(515) is duplicated.
WARNING:  Error creating primary key or unique constraint on table sakila.film_category
DETAIL:  could not create unique index "film_category_category_id_pkey": Key (category_id)=(15) is duplicated.
NOTICE:  Creating FOREIGN KEY constraints ...
WARNING:  Error creating foreign key constraint on table sakila.film_actor
DETAIL:  insert or update on table "film_actor" violates foreign key constraint "fk_film_actor_film": Key (film_id)=(1) is not present in table "film".
WARNING:  Error creating foreign key constraint on table sakila.film_category
DETAIL:  insert or update on table "film_category" violates foreign key constraint "fk_film_category_film": Key (film_id)=(1) is not present in table "film".
WARNING:  Error creating foreign key constraint on table sakila.inventory
DETAIL:  insert or update on table "inventory" violates foreign key constraint "fk_inventory_film": Key (film_id)=(1) is not present in table "film".
NOTICE:  Creating CHECK constraints ...
NOTICE:  Creating indexes ...
NOTICE:  Setting column default values ...
WARNING:  Error setting default value on rating of table sakila.film to "G"
DETAIL:  cannot use column reference in DEFAULT expression: 
NOTICE:  Dropping staging schemas ...
NOTICE:  Migration completed with 27 errors.
 db_migrate 
------------
         27
(1 row)

Support declarative partitioning

Have fun

Support blob data migration

At this time blob migration does not working with mysql_fdw

Given this foreign table:

MySQL [sakila]> desc sakila.staff;
+-------------+-------------------+------+-----+-------------------+
| Field       | Type              | Null | Key | Default           |
+-------------+-------------------+------+-----+-------------------+
| staff_id    | tinyint unsigned  | NO   | PRI | NULL              |
| first_name  | varchar(45)       | NO   |     | NULL              |
| last_name   | varchar(45)       | NO   |     | NULL              |
| address_id  | smallint unsigned | NO   | MUL | NULL              |
| picture     | blob              | YES  |     | NULL              |
| email       | varchar(50)       | YES  |     | NULL              |
| store_id    | tinyint unsigned  | NO   | MUL | NULL              |
| active      | tinyint(1)        | NO   |     | 1                 |
| username    | varchar(16)       | NO   |     | NULL              |
| password    | varchar(40)       | YES  |     | NULL              |
| last_update | timestamp         | NO   |     | CURRENT_TIMESTAMP |
+-------------+-------------------+------+-----+-------------------+

CREATE FOREIGN TABLE sakila.staff_picture (staff_id smallint, picture bytea)
  SERVER mysql OPTIONS (dbname 'sakila', table_name 'staff');

Convert binary data to bytea in PostgreSQL does not return correct rows:

SELECT staff_id FROM sakila.staff_picture ;

 staff_id 
----------
        1
        2
(2 rows)

SELECT staff_id, picture FROM sakila.staff_picture ;

 staff_id | picture 
----------+---------
(0 rows)

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.

Jobs

Jooble