spark-solutions / spree_shopify_importer

Migrate your Shopify store to Spree

Spree Shopify Importer

The Spree Shopify Importer gem allows you to easily import data from Shopify store to Spree application. It's compatible with Spree 4 and above.

Behind-the-scenes, this extension is using Shopify API gem.

Currently, it's in version 0.1.0. It has been tested with thousands of real life orders, but we welcome new pull requests!

Installation

1. Add this extension to your Gemfile with this:

gem 'spree_shopify_importer', github: 'spark-solutions/spree_shopify_importer'
gem 'spree_auth_devise'

2. Install the gem using Bundler:

bundle install

3. Copy & run migrations:

bundle exec rails g spree_shopify_importer:install

4. Restart your server:

If your server was running, restart it so that it can find the assets properly.

5. Setup credentials and change default values if needed.

Getting Started

We are recommending using sidekiq for background processing with this stack of gems:

gem 'sidekiq'
gem 'sidekiq-limit_fetch'
gem 'sidekiq-unique-jobs'

We also recommend having a limit 2 for import queue, due to API limit. Default queue name is default but it can be changed in Spree::AppConfiguration under shopify_import_queue key.

Default values

All default values are saved in Spree::AppConfiguration

• ShopifyAPI credentials - used for api authorization.
  • shopify_api_key - nil
  • shopify_password - nil
  • shopify_shop_domain - nil
  • shopify_token - nil
  • shopify_api_version - nil
• Import Rescue Limit - used for retrying API errors, for example API limit hit.
  • shopify_rescue_limit - 5
• Import Queue Name
  • shopify_import_queue - 'default'

Import

Currently, you need to have access to the console to start the import.

1. To start import, in console run:

With default values

 SpreeShopifyImporter::Invoker.new.import!

or with credentials.

 SpreeShopifyImporter::Invoker.new(credentials).import!

Where credentials could have two formats:

    {
      api_key: 'api_key', 
      password: 'password',
      shop_domain: 'shop_domain',
      api_version: 'api_version'
    }

or

    {
      token: 'token',
      shop_domain: 'shop_domain'
    }

Import Model

1. SpreeShopifyImporter::DataFeed - this model contains copy of JSON imported from shopify and association to spree object.

Import Services

Import services are divided into four main parts. Each of them could be customized.

1. Data Fetchers are services which are fetching products, users, orders and collections from Shopify.

2. Importers are services which are saving Shopify data feeds (as shadow copy of import), and starting a create or update action for spree object.

3. Data Savers are services which are saving spree objects, each of them has a parser method which can be overridden to change update/create attributes and associations.

4. Data Parsers are services which are changing Shopify data to spree data.

Send reset password links to users

If your application uses devise as authentication system you can run next worker:

SpreeShopifyImporter::Users::ResetPasswordJob.new.perform(Spree::User.all)

or find users that you need using where and then run worker:

users_to_reset = Spree::User.where(id: [1, 2, 3, 4])
SpreeShopifyImporter::Users::ResetPasswordJob.new.perform(users_to_reset)

It's important to pass relation not array into worker.

Testing

To run all the tests for the build, clone the repo and run:

bundle exec rake

This will generate a dummy app for testing, run rubocop style checker and rspec tests.

When testing your applications integration with this extension you may use its factories. Simply add this require statement to your spec_helper:

require 'spree_shopify_importer/factories'

Contributing

We welcome new pull requests!

License

Spree Shopify Importer is copyright © 2015-2019 Spark Solutions Sp. z o.o.. It is free software, and may be redistributed under the terms specified in the LICENSE file.

About Spark Solutions

Spree Shopify Importer is maintained and funded by Spark Solutions Sp. z o.o. The names and logos are trademarks of Spark Solutions Sp. z o.o.

We are passionate about open source software. We are available for hire.

spree_shopify_importer's Issues

Handling Orders from Shopify without Users

Set store currency

ShopifyAPI::ShippingZone.find(<id>) does not work

Similar requests for products or orders work, and this one worked before as well. We need it to tests, not for the feature itself.

connect imported stock items with the corresponding variants

import stock locations

Rspec problems

Investigate the problem with VCR in tests:

• integration_spec.rb
• delivery_profile_spec.rb

Handling Images passing for Newer Spree versions

Handling addresses without Phone

Order #1002 differences

Order #1002 is partially fulfilled and has 2 products in Shopify. In Spree Order view, that Order has only 1 product.

Import shop settings

User's Address is not moved to Spree

It is present in Order Details, but it's not present on the User page in Admin

Adjust importer to Spree 4.0

Fix taxes in imported orders

Before, taxes were just imported from order ("fake" tax_category and tax_rate were created during order import process). Since tax_category and tax_rate are imported from Shopify, we have to implement a connection between them and order taxes (for each line_item, not just for whole order - like it was).

Import shipments

1. Now shipments are imported as "Shopify fulfillments" since it is logic analog of shipment
2. In Shopify fulfillment is created after shipping, in Spree shipment is created on address phase of order (or similarly) and just has a suitable state (firstly pending, after payment -> ready, and so on), though.
3. That's why shipments in importer should be created apart of fulfillment (maybe based on line_items, or per order with line_items - to consider and verify) and just take the state from fulfillment.

Error when importing orders with pending payment

Retro steps:

1. open console of the shop (locally or on Heroku)
2. set credentials and connect with Shopify API:
   api_key = ... password = ... shop_domain = ... api_version = ...

credentials = { api_key: api_key, password: password, shop_domain: shop_domain, api_version: api_version }
SpreeShopifyImporter::Connections::Client.instance.get_connection(credentials)

3. Try to import an order with pending payment e.g. 1035:

resource = ShopifyAPI::Order.find(1995375247459) SpreeShopifyImporter::Importers::OrderImporter.new(resource.attributes.to_json).import!

4. observe the error:

ActiveRecord::RecordInvalid (Validation failed: State is invalid, Payment State is not included in the list)

Handling tax rates

Fix all # TODO:

Before implementing new features I think it's a time to fix problems of past

Handling shipping categories and methods

The explanation for different logic must be in README.
no conditions for shipping methods

Constraint issue when importing OptionValues

spree_shopify_importer_data_feeds → data_feed change to json type

for now we have data_feed as a text field, but as I see sometimes we have to search through info that located in this field, I think good idea is to change this data type to json or jsonb.

Add gem isolator to prevent background jobs in transactions

`bundle exec rake` breaks with missing 'sqlite3' gem

When I run: bundle exec rake, the task fails because there's no sqlite3 gem set as dependency, but sqlite3 is set as default gem.

@Argonus can you help me resolve this? You mentioned that sqlite is not fully supported? Should we use postgres (or mysql) as default or just add back sqlite3 as dev dependency?

Sequence and parallelism of jobs

Handling Users without Addresses

Handling addresses without State

Deadlock issue while importing SpreeVariant

Handling DeliveryProfiles

Collect constants in one place

Now, we have a mess with a constants in this gem, I think the best way is:

• create a folder app/constants
• create few modules inside an app
• move some strings to constants into this module
• in code reviews check that all strings are constants and located in correct place

Create variant on spree side when try to import order with deleted variant on Shopify

When we try to import the order that contains variant that is not present in product variants we get an error.
Possible solution is to create this variant from scratch on spree side.

Handling returns

Should we support ruby 2.2.2 in travis?

I've noticed that we currently don't have ruby 2.2.2 as our lowest ruby version for travis builds. I think we should change that.

Fix import of delivery_profile

Delivery_profiles should be imported for each variant or a product if there is no variant (when it is a master variant).

Please setup CI. Thanks

Handling tax zones

The explanation for different logic must be in README.

import stock_item

Error 500 on trying to access "Items" / Specific User

https://spree-shopify-importer-test.herokuapp.com/admin/users/2/items

User affected: [email protected]

Adjust importer to new API version

Rubocop

• add more rubocop rules (we can check .rubocop.yml of any of the big and great projects)
• start to fix files that are in .rubocop_todo.yml
• add rule RSpec/LetSetup

Handling temp_passwords for imported Users

We import users but we don’t import passwords. Users need to have the possibility to login to spree after import, so we have to send them temporary passwords after whole import is completed

Handling Users without Mail addresses

Imported Users do not have addresses in Spree

Orders states: shipment and payment

Orders should have statuses in accordance with ones from Shopify.
e.g order 1029 from Shopify is Fulfilled.
https://spree-products-importer.myshopify.com/admin/orders/1920328335459
so in Spree, it should be shipped.

Error while running Orders Import

One of the orders is not importing correctly because of variant.

Paginate by 250 instead of by 50

You could add limit: 250 to the default opts here: https://github.com/spark-solutions/spree_shopify_importer/blob/master/app/services/spree_shopify_importer/connections/base.rb#L21

It will speed up large stores import by almost 5 times by returning 250 items instead of 50 per page.

Imported Orders do not have addresses in Spree

Handling multiple addresses for Spree < 4.0

Fix product import - SKU should be imported

DEPRECATION WARNING: Class level methods will no longer inherit scoping from `create!` in Rails 6.1

We have to find the reason and fix this.

Why we store OptionType and OptionValue with lowercase name?

I just noticed that we always downcase option types and option values. On this way we're losing some information from Shopify and make the resulting options always presented/stored in downcase.

I think we should store the options in the same case as they come from Shopify.

/cc @Argonus

spree_shopify_importer/app/services/shopify_import/creators/product.rb

Line 41 in eaad16e

def create_option_types

spree_shopify_importer/app/services/shopify_import/data_parsers/products/base_data.rb

Line 25 in eaad16e

def option_types