memgraph / mgmigrate

mgmigrate is a tool for migrating data from MySQL or PostgreSQL to Memgraph and between Memgraph instances.

mgmigrate

Notice:

The project has been archived. It's based on some really interesting research papers, however unfortunately the code has not kept up with changes so we've decided to archive it. We might revisit this research and resurrect this project in the future. If you would like to contribute, have feedback or suggestions please visit our Discord server

Welcome to the mgmigrate tool repository. This tool can help you migrate data from a PostgreSQL or MySQL database to Memgraph. It can also be used to migrate data between Memgraph instances.

⚙️ Installation guide

You can install mgmigrate on the following systems:

• Linux instructions
• Windows instructions
• macOS instructions

Linux

To install compile dependencies on Debian / Ubuntu run:

apt-get install -y git cmake make gcc g++ python3 libpq-dev postgresql-server-dev-all
pip3 install pymgclient psycopg2 mysql-connector-python

On RedHat / CentOS / Fedora run:

yum install -y git cmake make gcc gcc-c++ python3 libpqxx-devel
pip3 install pymgclient psycopg2 mysql-connector-python

Once all the requirements are in place, create a build directory inside the source directory and configure the build by running CMake from it as follows:

mkdir build
cd build
cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=. ..

Continue with the Building and installing step.

Windows

Before you can install mgmigrate on Windows, you will need to install the needed dependencies. On Windows, you need to install the MSYS2. Just follow the instructions, up to step 6. In addition, OpenSSL must be installed. You can easily install it with an installer. The Win64 version is required, although the "Light" version is enough. Both EXE and MSI variants should work. Then, you'll need to install the dependencies using the MSYS2 MINGW64 terminal, which should be available from your Start menu. Just run the following command inside the MSYS2 MINGW64 terminal:

pacman -Syu --needed base-devel git mingw-w64-x86_64-toolchain mingw-w64-x86_64-cmake mingw-w64-x86_64-openssl mingw-w64-x86_64-postgresql

Once all the requirements are in place, create a build directory inside the source directory and configure the build by running CMake from it as follows:

mkdir build
cd build
cmake -DCMAKE_BUILD_TYPE=Release -G"MSYS Makefiles" -DCMAKE_INSTALL_PREFIX=. ..

Continue with the Building and installing step.

macOS

On macOS, first, make sure you have XCode and Homebrew installed. Then, in the terminal, run:

brew install git cmake make openssl postgresql

Once all the requirements are in place, create a build directory inside the source directory and configure the build by running CMake from it as follows:

mkdir build && cd build
cmake -DCMAKE_BUILD_TYPE=Release -DOPENSSL_ROOT_DIR="$(ls -rd -- /usr/local/Cellar/[email protected]/* | head -n 1)" -DCMAKE_INSTALL_PREFIX=. ..

Continue with the Building and installing step.

Building and installing

After running CMake, you should see a Makefile in the build directory. Then you can build the project by running:

make

Once all the requirements are in place, run:

make install

This will install mgmigrate in the default system installation directory. If you want to change this location, use -DCMAKE_INSTALL_PREFIX option when running CMake.

📋 Usage

MySQL

This is an example of using mgmigrate with a MySQL database:

build/src/mgmigrate --source-kind=mysql /
  --source-host 127.0.0.1 /
  --source-port 33060 /
  --source-username root /
  --source-password mysql /
  --source-database=exampledatabase /
  --destination-host 127.0.0.1 /
  --destination-port 7687 /
  --destination-use-ssl=false

PostgreSQL

Here is an example of how to use the mgmigrate tool with PostgreSQL:

build/src/mgmigrate --source-kind=postgresql /
  --source-host 127.0.0.1 /
  --source-port 5432 /
  --source-username postgres /
  --source-password postgres /
  --source-database=exampledatabase /
  --destination-host 127.0.0.1 /
  --destination-port 7687 /
  --destination-use-ssl=false

🔎 Arguments

The available arguments are:

Parameter	Description	Default
--source-kind	The kind of the given server. Supported options are: memgraph , mysql and postgresql.	memgraph
--source-host	Server address of the source database.	127.0.0.1
--source-port	Server port of the source database.	0
--source-username	Username for the source database.	-
--source-password	Password for the source database.	-
--source-database	Database name. Applicable to PostgreSQL and MySQL source.	-
--destination-use-ssl	Should the connection to the source database (if Memgraph) use SSL.	false
--destination-host	Server address of the destination database.	127.0.0.1
--destination-port	Server port number of the destination database.	7687
--destination-username	Username for the destination database.	-
--destination-password	Password for the destination database.	-
--destination-use-ssl	Should the connection to the destination database (if Memgraph) use SSL.	false

mgmigrate's Issues

Additional dependencies for Debian/Ubuntu install

I couldn't install on Ubuntu, and I was receiving this error:

CMake Error at /usr/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:146 (message):
  Could NOT find PostgreSQL (missing: PostgreSQL_TYPE_INCLUDE_DIR) (found
  version "14.1")
Call Stack (most recent call first):
  /usr/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:393 (_FPHSA_FAILURE_MESSAGE)
  /usr/share/cmake-3.16/Modules/FindPostgreSQL.cmake:247 (find_package_handle_standard_args)
  CMakeLists.txt:47 (find_package)

Then I looked at CMakeLists.txt, looked into what find_package(PostgreSQL REQUIRED) actually does (it runs a CMake built-in script to find Postgres) and encountered this StackOverflow post about the same problem with that script: https://stackoverflow.com/a/40027295.

In short, the user should also be asked to apt-get install libpq-dev postgresql-server-dev-all.

Can't find table in schema

Hi, I'm trying to use mgmigrate with PostreSQL but after running the src/mgmigrate command I'm getting a weird error:

F20230220 15:54:34.233558 4138902784 schema_info.cpp:16] Couldn't find table 'pg_aggregate' in schema 'pg_catalog'!

I'm pretty sure the pg_aggregate table and pg_catalog schema exist so not sure what's going on. Is there something I'm missing here?

Postgres found by cmake, but not by make (Mac Ventura)

attempting to build on M1 Mac, I was able to resolve some, but not all errors.

To make it find postgres on my machine, as well older version of OpenSSL which it seems to require:

cmake -DPostgreSQL_LIBRARY=/usr/local/opt/[email protected]/lib -DPostgreSQL_INCLUDE_DIR=/usr/local/opt/[email protected]/include -DCMAKE_BUILD_TYPE=Release -DOPENSSL_ROOT_DIR=/usr/local/opt/openssl -DCMAKE_INSTALL_PREFIX=. ..

To resolve warnings (below) and also find system includes on MacOS, I modified CMakeLists.txt to add the following lines right below the version check:

cmake_policy(SET CMP0114 NEW)
cmake_policy(SET CMP0135 NEW)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/")

cmake runs without error:

-- The C compiler identification is AppleClang 12.0.5.12050022
-- The CXX compiler identification is AppleClang 12.0.5.12050022
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- CMake build type: Release
-- Found OpenSSL: /usr/local/opt/openssl/lib/libcrypto.dylib (found version "3.0.7")  
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE  
-- Found PostgreSQL: /usr/local/opt/[email protected]/lib (found version "9.5.25")  
-- Configuring done
-- Generating done
-- Build files have been written to: .../mgmigrate/build

when I run make, I see this error:

CMake Error at /usr/local/Cellar/cmake/3.25.0/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
  Could NOT find PostgreSQL (missing: PostgreSQL_LIBRARY
  PostgreSQL_INCLUDE_DIR)
Call Stack (most recent call first):
  /usr/local/Cellar/cmake/3.25.0/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:600 (_FPHSA_FAILURE_MESSAGE)
  /usr/local/Cellar/cmake/3.25.0/share/cmake/Modules/FindPostgreSQL.cmake:272 (find_package_handle_standard_args)
  cmake/config.cmake:26 (find_package)
  CMakeLists.txt:27 (include)

then the make process continues until it halts at:

[ 64%] No install step for 'postgresql-proj'
[ 65%] Completed 'postgresql-proj'
[ 65%] Built target postgresql-proj
make: *** [all] Error 2

How do I make it so it finds Postgres in make?

warning (resolved with changes to CMakeLists.txt)

CMake Warning (dev) at /usr/local/Cellar/cmake/3.25.0/share/cmake/Modules/ExternalProject.cmake:3955 (message):
  Policy CMP0114 is not set to NEW.  In order to support the Xcode "new build
  system", this project must be updated to set policy CMP0114 to NEW.

  Since CMake is generating for the Xcode "new build system",
  ExternalProject_Add will use policy CMP0114's NEW behavior anyway, but the
  generated build system may not match what the project intends.
Call Stack (most recent call first):
  CMakeLists.txt:52 (ExternalProject_Add)
This warning is for project developers.  Use -Wno-dev to suppress it.

CMake Warning (dev) at /usr/local/Cellar/cmake/3.25.0/share/cmake/Modules/ExternalProject.cmake:3075 (message):
  The DOWNLOAD_EXTRACT_TIMESTAMP option was not given and policy CMP0135 is
  not set.  The policy's OLD behavior will be used.  When using a URL
  download, the timestamps of extracted files should preferably be that of
  the time of extraction, otherwise code that depends on the extracted
  contents might not be rebuilt if the URL changes.  The OLD behavior
  preserves the timestamps from the archive instead, but this is usually not
  what you want.  Update your project to the NEW behavior or specify the
  DOWNLOAD_EXTRACT_TIMESTAMP option with a value of true to avoid this
  robustness issue.
Call Stack (most recent call first):
  /usr/local/Cellar/cmake/3.25.0/share/cmake/Modules/ExternalProject.cmake:4185 (_ep_add_download_command)
  CMakeLists.txt:52 (ExternalProject_Add)
This warning is for project developers.  Use -Wno-dev to suppress it.

CMake Warning (dev) at /usr/local/Cellar/cmake/3.25.0/share/cmake/Modules/ExternalProject.cmake:3955 (message):
  Policy CMP0114 is not set to NEW.  In order to support the Xcode "new build
  system", this project must be updated to set policy CMP0114 to NEW.

  Since CMake is generating for the Xcode "new build system",
  ExternalProject_Add will use policy CMP0114's NEW behavior anyway, but the
  generated build system may not match what the project intends.
Call Stack (most recent call first):
  CMakeLists.txt:73 (ExternalProject_Add)
This warning is for project developers.  Use -Wno-dev to suppress it.

CMake Warning (dev) at /usr/local/Cellar/cmake/3.25.0/share/cmake/Modules/ExternalProject.cmake:3955 (message):
  Policy CMP0114 is not set to NEW.  In order to support the Xcode "new build
  system", this project must be updated to set policy CMP0114 to NEW.

  Since CMake is generating for the Xcode "new build system",
  ExternalProject_Add will use policy CMP0114's NEW behavior anyway, but the
  generated build system may not match what the project intends.
Call Stack (most recent call first):
  CMakeLists.txt:94 (ExternalProject_Add)
This warning is for project developers.  Use -Wno-dev to suppress it.

CMake Warning (dev) at /usr/local/Cellar/cmake/3.25.0/share/cmake/Modules/ExternalProject.cmake:3955 (message):
  Policy CMP0114 is not set to NEW.  In order to support the Xcode "new build
  system", this project must be updated to set policy CMP0114 to NEW.

  Since CMake is generating for the Xcode "new build system",
  ExternalProject_Add will use policy CMP0114's NEW behavior anyway, but the
  generated build system may not match what the project intends.
Call Stack (most recent call first):
  CMakeLists.txt:122 (ExternalProject_Add)
This warning is for project developers.  Use -Wno-dev to suppress it.

CMake Warning (dev) at /usr/local/Cellar/cmake/3.25.0/share/cmake/Modules/ExternalProject.cmake:3955 (message):
  Policy CMP0114 is not set to NEW.  In order to support the Xcode "new build
  system", this project must be updated to set policy CMP0114 to NEW.

  Since CMake is generating for the Xcode "new build system",
  ExternalProject_Add will use policy CMP0114's NEW behavior anyway, but the
  generated build system may not match what the project intends.
Call Stack (most recent call first):
  CMakeLists.txt:145 (ExternalProject_Add)
This warning is for project developers.  Use -Wno-dev to suppress it.

CMake Warning (dev) at /usr/local/Cellar/cmake/3.25.0/share/cmake/Modules/ExternalProject.cmake:3955 (message):
  Policy CMP0114 is not set to NEW.  In order to support the Xcode "new build
  system", this project must be updated to set policy CMP0114 to NEW.

  Since CMake is generating for the Xcode "new build system",
  ExternalProject_Add will use policy CMP0114's NEW behavior anyway, but the
  generated build system may not match what the project intends.
Call Stack (most recent call first):
  CMakeLists.txt:173 (ExternalProject_Add)
This warning is for project developers.  Use -Wno-dev to suppress it.

CMake Warning (dev) at /usr/local/Cellar/cmake/3.25.0/share/cmake/Modules/ExternalProject.cmake:3955 (message):
  Policy CMP0114 is not set to NEW.  In order to support the Xcode "new build
  system", this project must be updated to set policy CMP0114 to NEW.

  Since CMake is generating for the Xcode "new build system",
  ExternalProject_Add will use policy CMP0114's NEW behavior anyway, but the
  generated build system may not match what the project intends.
Call Stack (most recent call first):
  CMakeLists.txt:202 (ExternalProject_Add)
This warning is for project developers.  Use -Wno-dev to suppress it.

Compilation Error during 'make' Step on Linux System

Issue Description

I am encountering an issue while trying to install the migration script. The problem arises during the make step, where the code fails to compile correctly. Specifically, I'm experiencing errors when configuring mysqlconnector-proj. Any help regarding this issue is greatly appreciated.

Environment

Operating System: Pop!_OS version 22.04
Virtualization software: VirtualBox
Host operating system: Windows 10

Steps to Reproduce

0. Start the virtual machine with Pop!_OS version 22.04.

1. Open the terminal and navigate to the desktop using the following command:

cd ~/Desktop

2. Clone the mgmigrate repository from GitHub:

git clone https://github.com/memgraph/mgmigrate.git

3. Change into the mgmigrate directory:

cd mgmigrate

4. Install the necessary dependencies by running the following commands:

sudo apt-get install -y git cmake make gcc g++ python3 libpq-dev postgresql-server-dev-all
sudo apt install python3-pip
pip3 install pymgclient psycopg2 mysql-connector-python 

5. Create a build directory and navigate into it:

mkdir build
cd build

6. Configure the build using CMake:

cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=. ..

7. Install the libreadline development package to resolve an error encountered during the 'make' step:

sudo apt-get install libreadline-dev

8. Finally, run the 'make' command to compile the code:

make

Error Screenshot

Please refer to the screenshots below, which illustrates the error I encountered during the 'make' step.

The following screenshot shows the tail end of the error.

In the same error I received an Openssl version issued that is screenshotted below:

Fix --version

mgmigrate --version returns just mgmigrate. Return the proper version string.

Instructions for compiling on Windows don't work

I've followed the instructions from README file and I wasn't able to compile the app on Windows 11 64 bit.

I've tried using 32 and 64 bit version of MSYS2. I got an error that there was no cmake and yet the install process has listed it.

I've added cmake package one more time using pacman -Syu cmake. Now I stop receiving message that I'm missing cmake but I got a new error:

$ cmake -DCMAKE_BUILD_TYPE=Release -G"MSYS Makefiles" -DCMAKE_INSTALL_PREFIX=. ..
CMake Error: Could not create named generator MSYS Makefiles

Generators
* Unix Makefiles               = Generates standard UNIX makefiles.
  Ninja                        = Generates build.ninja files.
  Ninja Multi-Config           = Generates build-<Config>.ninja files.
  CodeBlocks - Ninja           = Generates CodeBlocks project files.
  CodeBlocks - Unix Makefiles  = Generates CodeBlocks project files.
  CodeLite - Ninja             = Generates CodeLite project files.
  CodeLite - Unix Makefiles    = Generates CodeLite project files.
  Eclipse CDT4 - Ninja         = Generates Eclipse CDT 4.0 project files.
  Eclipse CDT4 - Unix Makefiles= Generates Eclipse CDT 4.0 project files.
  Kate - Ninja                 = Generates Kate project files.
  Kate - Unix Makefiles        = Generates Kate project files.
  Sublime Text 2 - Ninja       = Generates Sublime Text 2 project files.
  Sublime Text 2 - Unix Makefiles = Generates Sublime Text 2 project files.

Maybe some option is missing. Can you please check?