shopify / shopify_transporter

The Transporter tool allows you to convert data from a third-party platform into a format that can be imported into Shopify.

THE TRANSPORTER APP HAS BEEN SUNSET AND THIS GEM IS NO LONGER MAINTAINED

Shopify Transporter

shopify_transporter is a command-line tool that offers capabilities to extract and convert data from third-party platforms into a Shopify-friendly format.

This format can then be imported into Shopify via the Transporter app.

shopify_transporter offers built-in support for migrations from Magento (version 1), as well as providing support for you to write your own transformations for other platforms.

Note: the Transporter app is available to Shopify Plus plans only.

Basic usage

Here's how you can get started migrating a store from a third-party platform onto Shopify using the Transporter suite of tools.

1. Follow the Getting started guide to install the necessary requirements.
2. Create a new project for the source store.
3. Extract data from your source store.
4. Convert the extracted data into a Shopify-formatted CSV file.
5. Use the Transporter app to import the Shopify-formatted CSV file into a Shopify store

Submitting Issues

Please open an issue here if you encounter a specific bug with this library or if something is documented incorrectly.

Before filing an issue, please ensure that:

• The issue is not already covered in another open issue
• The issue does not contain any confidential or personally identifiable information
• The issue is specifically regarding the shopify_transporter gem and not related to the Transporter App.

Contributing

If you'd love to see a particular feature as part of shopify_transporter, submit a pull request adding that feature!

Please read our contribution guidelines before submitting a pull request.

License

shopify_transporter is listed under the MIT license. See LICENSE.md for more details.

shopify_transporter's Issues

If there is a problem with the extraction, an extra comma is printed in the extracted json file

Problem:

If an export is incomplete because of a problem, the last JSON object has an extra column at the end:

  },

]

This is causing issues with the conversion step as this file cannot be read.

Issues with Magento 1.9.3.4

I tried to export customers from our old magento store. All permissions are set for the SOAP user, but I am getting a NoMethodError all the time. Is there anything I am missing?

Add more supports for line_item fields

Fields that map to Magento but haven't been implemented:

• total_discount
• requires_shipping
• taxable
• fulfillment_status
• fulfillable_quantity

Improve XML to JSON conversion to return consistent object formats

Currently our SOAP requests are converted from XML to JSON by Savon. When certain nested objects have only one element, the object is converted to a hash instead of an array, despite the XML metadata containing information that indicates the nested data is an array.

We should investigate if it's possible to improve the parsing. This will reduce/remove the need for a lot of object checks where we check to see if a nested object is a hash or an array.

Converting Magento v1.3 Orders only produces name and email fields.

Converting Magento orders produces only name and email fields. Our current Magento version is v1.3 and the export was done with SOAP v2 using the method name salesOrderList. I

Order Tax Rate is not being extracted correctly

Tax rates should be extracted as a ratio instead of a percentage, and these are showing up wrongly when imported in shopify:

The actual amount is being extracted correctly, but in this case the rate should be extracted as 0.00825 instead of 8.25.

No implicit conversion of String into Integer (TypeError) when converting orders

Conversion of simple products is missing price info

To reproduce:

• Create a simple product without any parent in Magento and assign price to it
• Extract and convert this product using CLI
• The price information is missing in the converted CSV

The root cause is the code will only assign price to variants. In this case the product itself is simple and therefore no variants are associated with it.

Improve extraction by making parallel soap calls

The SOAP extraction currently extracts additional information for Orders, Products and Customers sequentially. It would be ideal if there was an option to configure making multiple calls in parallel to speed up the process.

If the order contains a simple product, its parent product will occupy a row in the conversion CSV too

Simple product in order will occupy two lines as their parent product shows up too. Shopify will treat them as two products instead of one. The solution is to remove the parent product from CSV.

In Magento:

In CSV:

Server Side Exception: Allowed Memory Exhausted

Original comment:

In some cases, the savon is returning an out of memory error, as noticed by @reckerswartz : #8 (comment)

Running ruby orders.rb is raising:

/gems/savon-2.12.0/lib/savon/response.rb:85:in `raise_soap_and_http_errors!': (SOAP-ENV:Server) Allowed memory size of 2147483648 bytes exhausted (tried to allocate 83 bytes) (Savon::SOAPFault)
	from /home/aventum/.rvm/gems/ruby-2.4.0@transporter/gems/savon-2.12.0/lib/savon/response.rb:14:in `initialize'
	from /home/aventum/.rvm/gems/ruby-2.4.0@transporter/gems/savon-2.12.0/lib/savon/operation.rb:72:in `new'
	from /home/aventum/.rvm/gems/ruby-2.4.0@transporter/gems/savon-2.12.0/lib/savon/operation.rb:72:in `create_response'
	from /home/aventum/.rvm/gems/ruby-2.4.0@transporter/gems/savon-2.12.0/lib/savon/operation.rb:58:in `call'
	from /home/aventum/.rvm/gems/ruby-2.4.0@transporter/gems/savon-2.12.0/lib/savon/client.rb:36:in `call'
	from orders.rb:28:in `base_orders'
	from orders.rb:19:in `orders'
	from orders.rb:9:in `run'
	from orders.rb:48:in `<main>'

Support for Magento 2.x Enterprise

Will there be support added for Magento 2.x EE anytime soon?

shopify_transporter convert command outputs non-unique handles in its product CSV files.

Problem:

The shopify_transporter convert command outputs non-unique handles on different products in its product CSV files .

Steps to Reproduce:

1. Duplicate a product in the Magento admin and save (Magento 1.x, not sure about 2.x). There are now two products with the same Magento url_key.
2. Run "shopify_transporter extract [...]" including both products.
3. Run shopify_transporter convert on the resulting JSON file.

The resulting CSV file will have the same handle for both the products, and the handle will be the same string as shared url_key of the Magento products.

Magento does not enforce uniqueness on url_key. shopify_transporter cannot reliably use the url_key in unaltered form as a product handle in cases where different products share the same url_key.

[Ignore] Images for Wiki

This issue is used to upload images to use in the wiki.

Some top level attributes of orders are not converted in CSV file

Columns that are not mapped yet:

• [] Financial Status: Needs investigation - Not sure how to determine if payments are voided and authorized in Magento
• Fulfillment Status: Done
• Currency: Done
• Buyer Accepts Marketing: No equivalent in Magento
• Cancel Reason: No equivalent in Magento
• Cancelled At: Done
• Closed At: Done
• Tags: No equivalent in Magento
• Note: No equivalent in Magento
• Phone: Done
• Referring Site: No equivalent in Magento

Some address attributes are also missing:

• Shipping Company
• Shipping Name
• Billing Company
• Billing Name

When extracting data, skip batches that raise errors

When extracting data, skip batches that raise errors and continue with the remaining batches of data.

NoMethodError in Product Extraction

Getting NoMethodError error while doing product extraction.

shopify_transporter 2.4.2
Magento 1.9.3.6
WS-I compliance mode: No

Discounts should be converted to discount_code, amount and type instead of being output as total_discounts

Discounts from Magento should be extracted to the discount fields supported by the Transporter App - https://help.shopify.com/en/manual/migrating-to-shopify/transporter-app/csv-orders :

• discount_code
• discount_amount
• discount_type

Currently they are not being exported.

Unpaid Magento orders show up as being paid in Shopify

To reproduce, use Order # 100000081 in Magento. Extract, convert and load this order into Shopify.

In Magento, the status is unpaid

In Shopify, the order is fully paid

Add a command line command to extract data from Magento

Add an extract command to the gem that does the following:

Reads these parameters from the config file (or as passed to the command):

• Magento Hostname
• SOAP API Username
• SOAP API Key
• Start Date
• End Date
• Output file
• Type of object (orders/customers/products)

This should incorporate the requirements from #12 and #13:

• return a file that can be used for the convert step of the gem.
• should be resumable.
• accepts a start and end date for the data.

Investigate magento child_orders and see if they need to be extracted differently

Some orders have parent_ids in Magento. Currently these are extracted as separate orders, but they might need to be merged with a parent order instead.

Magento exports: 0% Tax showing up beside regular tax amount

When extracting orders that have some line items that are taxed and some that are not, there is a 0% tax showing up in Shopify apart from the actual tax:

I think this is because we extract tax as 0,0 for line items without tax. Perhaps setting this to empty instead of 0 will fix this:

Does not extract or convert product categories.

CAUTION: Do not attach any sensitive information or PII

Please be careful not to include sensitive information in the files because this is a public repository (for example, real customer names, emails and addresses).

Problem:

Describe the problem you are experiencing with the gem.\

I want to migrate my product categories to Shopify.

(Some discussion about this exists in the Shopify Plus Partner slack. Can't share more details on public repo. DM me on slack if you're a Shopifolk and want to know more.)

Steps to Reproduce:

List the steps we can take to reproduce the problem:

Orders: grand_total is unused and can be removed from TopLevelAttributes

Problem:

The code for the top level attributes in orders extracts grand_total, but this is not exported: https://github.com/Shopify/shopify_transporter/blob/master/lib/shopify_transporter/pipeline/magento/order/top_level_attributes.rb

This should be removed or added to the export

Issue with exporting customers with id > 99999 in the extraction-scripts-by-daterange branch

Problem:

When there are a lot of customer ids, the customer export script in the extraction-scripts-by-daterange is exiting early, for example at customer 99999.

Steps to Reproduce:

List the steps we can take to reproduce the problem:

1. Try exporting customer sets with customers ids greater than 99999.

Extract from Magento returns empty JSON file

Problem:

Running the transporter returns an empty JSON file that only consists of square brackets:
[ ]

No error messages are given. This happens when trying to extract Customers, Products, or Orders. I've tried changing the limits to 1000, 100, and 10 and it's the same result every time. Is there anywhere I can look for in-depth debug/log info?

Steps to Reproduce:

List the steps we can take to reproduce the problem:

1. Get config.yml info for client's site
2. Attempt to run Transporter.

Extract error: unallowed token at this point in JSON text (Yajl::ParseError)

Converting Orders

Problem:

versions/2.5.1/lib/ruby/gems/2.5.0/gems/yajl-ruby-1.4.1/lib/yajl.rb:44:in `parse'
/Users/ariellacheskisgold/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/yajl-ruby-1.4.1/lib/yajl.rb:44:in `parse': parse error: unallowed token at this point in JSON text (Yajl::ParseError)
          rEntity"       }     }   },  ]
                     (right here) ------^

Steps to Reproduce:

List the steps we can take to reproduce the problem:

Order CSV with cancelled_at/closed_at timestamp won't end up as cancelled/closed after being imported using app

The reason why this happens is cancelled_at and closed_at are read-only fields in Shopify Order API.

A potential solution is call APIs to explicitly cancel/close the order in CLI.

Extract and convert order shipping_lines

Shipping_line is part of the sample CSV in Transporter app doc but not supported yet by our CLI

Magento product/customer tax classes are not handled in CLI

Magento simple products seem to have a tax_class_id which corresponds to Taxable Goods, Shipping and Tax Exempt. It might be possible to extract these as the Variant Taxable field.

I'm not sure if this changes across Magento instances - if it does, we might need to fetch the values for the tax_class_id from the database.

Conversion does not take special_price in Magento into account.

CAUTION: Do not attach any sensitive information or PII

Please be careful not to include sensitive information in the files because this is a public repository (for example, real customer names, emails and addresses).

Problem:

Describe the problem you are experiencing with the gem.

Special_price is ignored.

Steps to Reproduce:

List the steps we can take to reproduce the problem:

1. Import a product with both price and special price
2. Special price is ignored, price in Shopify matches price in Magento

Add a sample script to export customers from Magento using SOAP Requests

Create a sample script to extract information about customers from Magento using the SOAP API.

This script should:

• return a file that can be used for the convert step of the gem.
• should be resumable.
• accepts a start and end date for the data.

Line Item Taxes are not being exported

Problem:

Line item taxes are not being exported from json to the csv output.

Steps to Reproduce:

List the steps we can take to reproduce the problem:

1. Attempt to convert the following order with the transporter app:

[
  {
    "increment_id": "increment_id-value",
    "created_at": "created_at-value",
    "customer_email": "customer_email-value",
    "customer_firstname": "customer_firstname-value",
    "customer_lastname": "customer_lastname-value",
    "subtotal": "subtotal-value",
    "tax_amount": "1",
    "grand_total": "11",
    "items": {
      "result": {
        "items": {
          "item": [
            {
              "qty_ordered": "qty_ordered-value",
              "sku": "sku-value",
              "name": "name-value",
              "price": "10",
              "tax_amount": "1",
              "tax_percent": "10"
            }
          ]
        },
        "shipping_address": {
          "firstname": "firstname-value",
          "lastname": "lastname-value",
          "telephone": "telephone-value",
          "street": "street-value",
          "city": "city-value",
          "region": "region-value",
          "postcode": "postcode-value",
          "country_id": "country_id-value"
        },
        "billing_address": {
          "firstname": "firstname-value",
          "lastname": "lastname-value",
          "telephone": "telephone-value",
          "street": "street-value",
          "city": "city-value",
          "region": "region-value",
          "postcode": "postcode-value",
          "country_id": "country_id-value"
        }
      }
    }
  }
]

2. The line item taxes do not show up in the output.

Extract error: Unable to obtain SOAP session ID from server

Problem:

Getting following output in .json file when doing extraction:
`[

]Extract error: Unable to obtain SOAP session ID from server.

Details:
{:login_response_param=>{:result=>"[redacted session ID]"}}`

Ruby: 2.5.3

I'm using the latest master branch.

Readme: Add information about gems and windows

Somewhere near the beginning of the Readme, we should list the technologies/languages/programs that users need to use the gem I.e., Ruby (link to https://www.ruby-lang.org/en/), and any OS requirements. These are not clear right now.

Add a sample script to export orders from Magento using SOAP Requests

Duplicate Ticket

Create a sample script to extract information about orders from Magento using the SOAP API.

This script should:

• return a file that can be used for the convert step of the gem.
• should be resumable.
• accepts a start and end date for the data.

Need to catch empty inventory numbers in Product export

Problem:

It's possible to retrieve products from the SOAP call that do no have inventory levels set, so line method inventory_quantity_for in product_exporter.rb will throw an exception on line 117 where [:result] is not set.

Steps to Reproduce:

List the steps we can take to reproduce the problem:

1. Run shopify_transporter on a Magento db where a product does not have have a SOAP response from the inventory endpoint
2. [] is called on null object - process terminates

What is the recommended way to export huge data (1Lac plus rows).

Problem:

What is the recommended way to export huge data (1 Lac plus rows).
As the data will be processing line by line, what is the best possible way to handle this.

Exporting magento customers using customer.rb gives a raise_invalid_response_error

when trying to export magento customers using the customers.rb script, I get the following error:

response.rb:90:in raise_invalid_response_error!': Unable to parse response body: (Savon::InvalidResponseError)

It seems like the response is coming in as an entire string. I see all the customer information but it is not parsed correctly. Here's a screenshot:

It continues to print through every customer until it hits a timeout error:

from /Users/chrismark/.rvm/gems/ruby-2.4.0/gems/savon-2.12.0/lib/savon/response.rb:72:in `find'
	from /Users/chrismark/.rvm/gems/ruby-2.4.0/gems/savon-2.12.0/lib/savon/response.rb:37:in `body'
	from customers.rb:82:in `customer_list'
	from customers.rb:67:in `block in customers_starting_with'
	from /Users/chrismark/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/timeout.rb:93:in `block in timeout'
	from /Users/chrismark/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/timeout.rb:103:in `timeout'
	from customers.rb:65:in `customers_starting_with'
	from customers.rb:59:in `base_customers'
	from customers.rb:24:in `print_customers'
	from customers.rb:12:in `run'
	from customers.rb:97:in `<main>'

orders.rb worked fine and authentication seems to work.

When extracting orders i am getting Savon::HTTPError.

Problem:

When extracting orders i am getting this error.
'raise_soap_and_http_errors!': HTTP error (500) (Savon::HTTPError)

Steps to Reproduce:

1.Checkout the master branch
2.Run ./shopify_transporter export --config=../config.yml --object=order > orders.json
3 See below error

Default address is not converted correctly in pipeline

The way our code works is that it always assume the first address in the addresses array is the default address. However, Magento API returns the addresses array in the order of address creation.

This means that our code assumes that the address being created the first is the default one, but this is not always the case.

To fix the wrong behaviour:

• Reorder the addresses array in the conversion pipeline so that the address with "is_default_billing": true shows up as the first element in the array
• Magento supports different default billing and shipping addresses. However, this is not a thing in Shopify. When this case happens we've decided to use the default shipping address as the default address in Shopify.

Add transaction support for orders

Order exports don't support transactions. This makes the amount paid by customer always 0 in Shopify. See below.

When extracting data, skip individual objects that raise errors and continue with the remaining objects

When extracting data, skip individual objects that raise errors and continue with the remaining objects.

We make additional soap calls for each object (order, customer, product). Sometimes, these additional calls fail. We should skip these objects and continue with the rest.

No implicit conversion of Hash into String (TypeError)

Error reported by someone converting orders with addresses:

/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/shopify_transporter-2.5.1/lib/shopify_transporter/pipeline/magento/order/addresses_attribute.rb:47:in `+':
no implicit conversion of Hash into String (TypeError)

We are investing this, please add details here if you are running into the same issue.

Using the order convert command with an invalid json file results in an unclear error.

Problem:

Using the order convert command with an invalid json file results in an unclear error.

Steps to Reproduce:

1. Create a new shopify_transporter project.
2. Create a json file with this data:

{
  "increment_id": "increment_id-value"
}

3. Try to convert this file using this command:

shopify_transporter convert --config=config.yml --object=order anonymized_order.json > shopify_order.csv

This results in this error:

.../gems/shopify_transporter-1.0.0/lib/shopify_transporter/record_builder.rb:46:in `key_of': no implicit conversion of String into Integer (TypeError)

NoMethodError when converting orders

When converting orders I am getting a NoMethodError.

I disabled the discount pipeline in the config because there was an error too.

Thanks.

Magento 1.9 customer address are not map to shopify customer address

Magento 1.9 customer address are not map to shopify customer address
First name , last name and email are added by default TopLevelAttributes and I have added phone number by my custom staging.

Is it possible to get data based on "website_id" rather that "store_id" in magento platform.

Problem:

I want to get data using website_id rather than store_id, is that possible?

Bundle products are imported without variant options, and with wrong price

Problem:

Bundle product pricing and variants inconsistent between Shopify and Magento.

Steps to Reproduce:

1. Create a bundle product in Magento, with complex variant options (see screenshot)
2. Import said product using the extract + convert + import process
3. Observe the newly created product has no variants, and has the price of one random child product

Side note: I understand that Shopify doesn't natively support complex bundling like Magento does, so I know this isn't an easy problem to solve, and might be outside the scope of what this tool is designed to do.

Metafields, issue.