ecomdev / sync-magento-2-migration

Release of rough proof of concept from 2018 that allows to import and export millions of products quickly

Simple Import / Export tool

A tool that allows to quickly export data from Magento 1 and Magento 2 store and import it back into Magento 2. Table data gets exported without the knowledge of the entity identifiers and delta imports get processed by related unique keys:

• Categories: id attribute (gets stored in map table)
• Products: SKU
• Customers: email + website

When you export data from your existing stores you can configure mapping and skipped row conditions by using configuration.json:

Here is an example of configuration.json that maps all manufacturer attribute code in product data into a brand and skips.

{
  "product_data.csv": {
    "map": {
      "attribute": {
         "manufacturer": "brand" 
      }
    },
    "skip": [
      {
        "store": ["pl", "sk"]
      }
    ]
  }
}

Also, you can add own mappers for exported files. Here is an example on mapping output of product attributes to create website level price instead of global during migration:

{
  "product": {
    "mappers": {
      "product_attributes": [
        {
          "class": "EcomDev\\MagentoMigration\\CustomMappers\\PriceMapper",
          "setup": [
            ["withStore", "us_en", 1.0],
            ["withStore", "uk_en", 0.76],
            ["withStore", "eu_en", 0.89]
          ]
        }
      ]
    }
  }
}

Tests

Right now the automation suite might not run, as tool is released by stripping of all customer specific data from tests and codebase. PRs to re-introduce test cases that has been removed are welcome.

sync-magento-2-migration's Issues

Mismatch between import and export CSV enclosure characters

Export uses a single quote ': https://github.com/EcomDev/sync-magento-2-migration/blob/main/src/CsvFactory.php#L56

Import uses double quotes ": https://github.com/EcomDev/sync-magento-2-migration/blob/main/src/CsvReader.php#L24

Bug introduced in this commit: 518807c

Error "enterprise_catalog_product_rewrite" table not found

There's a hard reference to Magento Enterprise tables here, causing issues with product export:

https://github.com/EcomDev/sync-magento-2-migration/blob/main/src/ProductInfo.php#L408

I managed to get around it by excluding product URL rewrites, and generating rewrites using this: https://github.com/elgentos/regenerate-catalog-urls

Should there be a check to see if that table exists and/or the current version is Enterprise?

Category import is broken due to missing `path` value in insert

Fatal error: Uncaught PDOException: SQLSTATE[HY000]: General error: 1364 Field 'path' doesn't have a default value in /var/www/html/sync-magento-2-migration/vendor/zendframework/zend-db/src/Adapter/Driver/Pdo/Statement.php:239
Stack trace:
#0 /var/www/html/sync-magento-2-migration/vendor/zendframework/zend-db/src/Adapter/Driver/Pdo/Statement.php(239): PDOStatement->execute()
#1 /var/www/html/sync-magento-2-migration/src/CategoryImport.php(199): Zend\Db\Adapter\Driver\Pdo\Statement->execute()
#2 /var/www/html/sync-magento-2-migration/src/CategoryImport.php(229): EcomDev\MagentoMigration\CategoryImport->allocateEntityIds(1)
#3 /var/www/html/sync-magento-2-migration/src/CategoryImport.php(76): EcomDev\MagentoMigration\CategoryImport->resolveExternalCategoryIds(Array)
#4 /var/www/html/sync-magento-2-migration/src/Import.php(61): EcomDev\MagentoMigration\CategoryImport->importCategories(Object(Generator))
#5 /var/www/html/sync-magento-2-migration/src/ImportApplication.php(60): EcomDev\MagentoMigration\Import->importCategories( in /var/www/html/sync-magento-2-migration/vendor/zendframework/zend-db/src/Adapter/Driver/Pdo/Statement.php on line 244

Because here we're adding just a parent_id for the ID reservation strategy, but it seems Magento 2.4.3-p1 also requires a path value (and a position and children_count).

@IvanChepurnyi I assume this can be an empty string because it's purely used for ID reservation. Is that correct?