autotelik / datashift Goto Github PK
View Code? Open in Web Editor NEWFull Excel/CSV Import/Export facilities for Rails
Home Page: http://www.autotelik.co.uk
License: Other
Full Excel/CSV Import/Export facilities for Rails
Home Page: http://www.autotelik.co.uk
License: Other
Hello again,
I am now working on variants and I would like to set specific images (and a specific price and SKU by the way) on each variant. I am not sure it can be done right now... Or could it be ?
Thanks,
Manu
ndj-mbp:imazen-store nathanael$ b thor datashift:export:csv -m=Spree::Product -r=./
Please supply array of records to export
/Users/nathanael/.rbenv/versions/2.0.0-p353/lib/ruby/gems/2.0.0/gems/datashift-0.13.0/lib/exporters/csv_exporter.rb:38:in `export'
/Users/nathanael/.rbenv/versions/2.0.0-p353/lib/ruby/gems/2.0.0/gems/datashift-0.13.0/lib/thor/export.thor:100:in `csv'
/Users/nathanael/.rbenv/versions/2.0.0-p353/lib/ruby/gems/2.0.0/gems/thor-0.19.1/lib/thor/command.rb:27:in `run'
/Users/nathanael/.rbenv/versions/2.0.0-p353/lib/ruby/gems/2.0.0/gems/thor-0.19.1/lib/thor/invocation.rb:126:in `invoke_command'
/Users/nathanael/.rbenv/versions/2.0.0-p353/lib/ruby/gems/2.0.0/gems/thor-0.19.1/lib/thor.rb:359:in `dispatch'
/Users/nathanael/.rbenv/versions/2.0.0-p353/lib/ruby/gems/2.0.0/gems/thor-0.19.1/lib/thor/base.rb:440:in `start'
/Users/nathanael/.rbenv/versions/2.0.0-p353/lib/ruby/gems/2.0.0/gems/thor-0.19.1/lib/thor/runner.rb:36:in `method_missing'
/Users/nathanael/.rbenv/versions/2.0.0-p353/lib/ruby/gems/2.0.0/gems/thor-0.19.1/lib/thor/command.rb:29:in `run'
/Users/nathanael/.rbenv/versions/2.0.0-p353/lib/ruby/gems/2.0.0/gems/thor-0.19.1/lib/thor/command.rb:126:in `run'
/Users/nathanael/.rbenv/versions/2.0.0-p353/lib/ruby/gems/2.0.0/gems/thor-0.19.1/lib/thor/invocation.rb:126:in `invoke_command'
/Users/nathanael/.rbenv/versions/2.0.0-p353/lib/ruby/gems/2.0.0/gems/thor-0.19.1/lib/thor.rb:359:in `dispatch'
/Users/nathanael/.rbenv/versions/2.0.0-p353/lib/ruby/gems/2.0.0/gems/thor-0.19.1/lib/thor/base.rb:440:in `start'
/Users/nathanael/.rbenv/versions/2.0.0-p353/lib/ruby/gems/2.0.0/gems/thor-0.19.1/bin/thor:6:in `<top (required)>'
/Users/nathanael/.rbenv/versions/2.0.0-p353/bin/thor:23:in `load'
/Users/nathanael/.rbenv/versions/2.0.0-p353/bin/thor:23:in `<main>'
Warning: Error during export, data may be incomplete
Using these versions of gems.
Version: current latest on RubyGems, 0.40.0
Script:
require 'thor'
require 'datashift'
Error Message:
Problem initializing gem #<NameError: uninitialized constant DataShift::LoaderBase::Forwardable>
Problem loading datashift file /Users/songyy/.rvm/gems/ruby-2.3.1/gems/datashift-0.40.0/lib/exporters/csv_exporter.rb - #<NameError: uninitialized constant DataShift::ExporterBase
Did you mean? DataShift::Exporters>
Problem loading datashift file /Users/songyy/.rvm/gems/ruby-2.3.1/gems/datashift-0.40.0/lib/exporters/excel_exporter.rb - #<NameError: uninitialized constant DataShift::ExporterBase
Did you mean? DataShift::Exporters>
I am slowly looking at implementing this.
Need to create a better loader object management layer and then refactor the perform.._load methods (e.g perform_excel_load) as these currently do too much.
Look for a good pattern for Object Management, move out of loaders/Populator, into separate area.
Enable loaders to reach into this to grab current object, split out from the generic loaders, and provide hooks in those generic Loaders to enable custom loaders to more easily manage the object lifecycle themselves. Currently far too rigid, i.e just creates a new object for each row processed.
I'm working on a Rails 4 app and needed the latest code from datashift
. As it's not released yet, my Gemfile
directly links to a commit from the repo.
Everything worked fine until I tried to deploy on Heroku:
The slug size exceeds 300MB, and the main reason is that the datashift
git repo needs more than 200MB.
I've tracked down the problem to this folder.
231M vendor/bundle/ruby/2.0.0/bundler/gems/datashift-bfada2fd1319/.git/objects/pack
I don't have currently the time to find which commit is responsible.
Is it possible to create a has_many record on import? Do i have to create a custom loader (like the Spree Product loader)?
My install is failing, using jruby 1.6.5., therubyrhino, rails 3.2.3 and Spree 1.1.1.
Bailing out on the installation of Spree with a
Invalid gemspec in [/home/jet/.rvm/gems/jruby-1.6.5@datashift/specifications/spree_auth-1.1.1.gemspec]: Illformed requirement ["#<YAML::Yecht::DefaultKey:0x1f62951> 1.1.1"]
Tried getting by this by running,
jruby --1.9 -S bundle install (which if my version is only compatible with rails v1.8.7.p320, makes sense?)
...but failed at:
NoMethodError: undefined method `read_nonblock' for #OpenSSL::SSL::SSLSocket:0x1942a3c
An error occured while installing rake (0.9.2.2).
So, what pieces are you using to get this working?
Cheers,
Jet
bundle execute thor datashift_spree:load:products -i ~/datashift_autotelik/spec/fixtures/SpreeProductsMandatoryOnly.xls
DataShift::Product starting upload from file: /root/datashift_autotelik/spec/fixtures/SpreeProductsMandatoryOnly.xls
Loading from Excel file: /root/datashift_autotelik/spec/fixtures/SpreeProductsMandatoryOnly.xls
["sku", "name", "price"]
["sku", "name", "description", "shipping_category", "price", "option_types"]
ERROR: Excel loading failed : #<NoMethodError: undefined method `pry' for #<Binding:0x3c99ea87>>
Processing Summary Report
>>>>>>>>>>>>>>>>>>>>>>>>>
Processed total of 0 inbound entry
0 database objects were successfully processed.
0 inbound rows were successfully processed.
There were NO failures.
NoMethodError: undefined method `pry' for #<Binding:0x3c99ea87>
perform_excel_load at /usr/local/rvm/gems/jruby-1.7.19@mystore20151008/bundler/gems/datashift-91b44de22c91/lib/loaders/excel_loader.rb:92
each_with_index at file:/usr/local/rvm/rubies/jruby-1.7.19/lib/jruby.jar!/jruby/java/java_ext/java.lang.rb:19
perform_excel_load at /usr/local/rvm/gems/jruby-1.7.19@mystore20151008/bundler/gems/datashift-91b44de22c91/lib/loaders/excel_loader.rb:80
transaction at /usr/local/rvm/gems/jruby-1.7.19@mystore20151008/gems/activerecord-4.1.11/lib/active_record/connection_adapters/abstract/database_statements.rb:201
within_new_transaction at /usr/local/rvm/gems/jruby-1.7.19@mystore20151008/gems/activerecord-4.1.11/lib/active_record/connection_adapters/abstract/database_statements.rb:209
transaction at /usr/local/rvm/gems/jruby-1.7.19@mystore20151008/gems/activerecord-4.1.11/lib/active_record/connection_adapters/abstract/database_statements.rb:201
transaction at /usr/local/rvm/gems/jruby-1.7.19@mystore20151008/gems/activerecord-4.1.11/lib/active_record/transactions.rb:208
perform_excel_load at /usr/local/rvm/gems/jruby-1.7.19@mystore20151008/bundler/gems/datashift-91b44de22c91/lib/loaders/excel_loader.rb:78
perform_load at /usr/local/rvm/gems/jruby-1.7.19@mystore20151008/bundler/gems/datashift-91b44de22c91/lib/loaders/loader_base.rb:118
perform_load at /usr/local/rvm/gems/jruby-1.7.19@mystore20151008/bundler/gems/datashift_spree-37f79a7bf671/lib/loaders/spree/spree_base_loader.rb:55
perform_load at /usr/local/rvm/gems/jruby-1.7.19@mystore20151008/bundler/gems/datashift_spree-37f79a7bf671/lib/loaders/spree/product_loader.rb:58
products at /usr/local/rvm/gems/jruby-1.7.19@mystore20151008/bundler/gems/datashift_spree-37f79a7bf671/lib/thor/spree/products_images.thor:69
run at /usr/local/rvm/gems/jruby-1.7.19@mystore20151008/gems/thor-0.19.1/lib/thor/command.rb:27
invoke_command at /usr/local/rvm/gems/jruby-1.7.19@mystore20151008/gems/thor-0.19.1/lib/thor/invocation.rb:126
dispatch at /usr/local/rvm/gems/jruby-1.7.19@mystore20151008/gems/thor-0.19.1/lib/thor.rb:359
start at /usr/local/rvm/gems/jruby-1.7.19@mystore20151008/gems/thor-0.19.1/lib/thor/base.rb:440
method_missing at /usr/local/rvm/gems/jruby-1.7.19@mystore20151008/gems/thor-0.19.1/lib/thor/runner.rb:36
run at /usr/local/rvm/gems/jruby-1.7.19@mystore20151008/gems/thor-0.19.1/lib/thor/command.rb:29
run at /usr/local/rvm/gems/jruby-1.7.19@mystore20151008/gems/thor-0.19.1/lib/thor/command.rb:126
invoke_command at /usr/local/rvm/gems/jruby-1.7.19@mystore20151008/gems/thor-0.19.1/lib/thor/invocation.rb:126
dispatch at /usr/local/rvm/gems/jruby-1.7.19@mystore20151008/gems/thor-0.19.1/lib/thor.rb:359
start at /usr/local/rvm/gems/jruby-1.7.19@mystore20151008/gems/thor-0.19.1/lib/thor/base.rb:440
(root) at /usr/local/rvm/gems/jruby-1.7.19@mystore20151008/gems/thor-0.19.1/bin/thor:6
load at org/jruby/RubyKernel.java:1087
(root) at /usr/local/rvm/gems/jruby-1.7.19@mystore20151008/bin/thor:23
I used a pry statement in lib/loaders/excel_loader.rb
# be thor datashift_spree:load:products -i ~/datashift_autotelik/spec/fixtures/SpreeProductsMandatoryOnly.xls
[WARNING] You are not setting Devise.secret_key within your application!
You must set this in config/initializers/devise.rb. Here's an example:
Devise.secret_key = "0d17b9c9239f700e20e4b526459e55257301cd84c46f0654e70821ddbff6b1da631f1d011f59bc2f270d2c5ae6d0169c6903"
Instance method "open" is already defined in Object, use generic helper instead or set StateMachine::Machine.ignore_method_conflicts = true.
DataShift::Product starting upload from file: /root/datashift_autotelik/spec/fixtures/SpreeProductsMandatoryOnly.xls
Loading from Excel file: /root/datashift_autotelik/spec/fixtures/SpreeProductsMandatoryOnly.xls
["sku", "name", "price"]
["sku", "name", "description", "shipping_category", "price", "option_types"]
From: /usr/local/rvm/gems/jruby-1.7.19@mystore20151008/bundler/gems/datashift-91b44de22c91/lib/loaders/excel_loader.rb @ line 92 DataShift::ExcelLoading#perform_excel_load:
87: # Excel num_rows seems to return all 'visible' rows, which appears to be greater than the actual data rows
88: # (TODO - write spec to process .xls with a huge number of rows)
89: #
90: # This is rubbish but currently manually detect when actual data ends, this isn't very smart but
91: # got no better idea than ending once we hit the first completely empty row
=> 92: binding.pry
93: break if(@current_row.nil? || @current_row.compact.empty?)
94:
95: logger.info "Processing Row #{current_row_idx} : #{@current_row}"
96:
97: @contains_data = false
[1] pry(#<DataShift::SpreeEcom::ProductLoader>)> @current_row
=> #<Java::OrgApachePoiHssfUsermodel::HSSFRow:0x762c1d96>
[2] pry(#<DataShift::SpreeEcom::ProductLoader>)> @current_row.nil?
=> false
[3] pry(#<DataShift::SpreeEcom::ProductLoader>)> @current_row.compact
NoMethodError: undefined method `compact' for #<Java::OrgApachePoiHssfUsermodel::HSSFRow:0x762c1d96>
from (pry):3:in `evaluate_ruby'
environment: jruby 1.7.19, java version "1.7.0_95", rails (4.1.11)
delegate_belongs_to fields do not appear in the template output
For example see Spree, the model Spree::Product has following definition
delegate_belongs_to :master, :sku, :price, :currency, :display_amount, :display_price, :weight, :height, :width, :depth,
:is_master, :has_default_price?, :cost_currency, :price_in, :amount_in, :cost_price, :images
But none of these appear when generating an Excel template,
thor datashift:generate:excel -m Spree::Product -r tmp/product_template.xls
Suspect these will appear at least as instance methods so we need to make sure the catalogue class method can be configured to add those if requested - for example binder does not currently pass any options
model_method_mgr = ModelMethods::Manager.catalog_class(klass)
Hello,
Thanks for datashift. The first contact with it is neat and efficient !
I guees I have come accros a bug when using several subcategories of same name. For instance
lucky_product : A>mysubcat
bad_luck : B>mysubcat
The bad_luck product is created with in B and somehow pushed into A>mysubcat.
The category B>mysubcat is not created at all...
Hope this is clear enough,
Manu
Hey thanks for providing this great stuff, I have query when i upload some products via xls. I got error
WARNING : Check logs : 1 rows contained errors
which is expected as it was failing spree validation. I am sending a mail as soon the background process completes.
But the issue is that warning is not raising any exception such that i can rescue it and send failure mail.
In simple term, i want to send success mail when product uploaded successfully and failure mail when the upload fails
As per the demo spreadsheets which were created manually, when a template is generated, see if we can dynamically write similar comments into the heading of each column with details of the association type and the supported syntax for specifying lookups etc
NoMethodError: undefined method
text_delim=' for DataShift::Delimiters:Module
lib/exporters/csv_exporter.rb:42:in export'
exporter = DataShift::CsvExporter.new(file_path)
exporter.export(Foo.all, text_delim: "\t")
Is this function working correctly?
I cannot seem to get it working. The "f" option gives me a lookup error.
spree 3
Better loading of lib files e.g
Gem.find_files("gem_name/**/*.rb").each { |path| require path }
In my Gemfile:
gem 'datashift', github: 'autotelik/datashift'
gem 'datashift_spree', github: 'autotelik/datashift_spree'
When running
bundle exec thor datashift_spree:load:attach_images --input=/vagrant/imageimport/import.xls
I receive the error:
/var/lib/gems/1.9.1/gems/datashift_spree-0.5.0/lib/loaders/spree/image_loader.rb:19:in <module:SpreeHelper>': uninitialized constant DataShift::SpreeHelper::SpreeBaseLoader (NameError) from /var/lib/gems/1.9.1/gems/datashift_spree-0.5.0/lib/loaders/spree/image_loader.rb:10:in
module:DataShift'
from /var/lib/gems/1.9.1/gems/datashift_spree-0.5.0/lib/loaders/spree/image_loader.rb:8:in <top (required)>' from /var/lib/gems/1.9.1/gems/polyglot-0.3.5/lib/polyglot.rb:65:in
require'
from /var/lib/gems/1.9.1/gems/polyglot-0.3.5/lib/polyglot.rb:65:in require' from /var/lib/gems/1.9.1/gems/datashift_spree-0.5.0/lib/thor/spree/products_images.thor:77:in
attach_images'
from /var/lib/gems/1.9.1/gems/thor-0.19.1/lib/thor/command.rb:27:in run' from /var/lib/gems/1.9.1/gems/thor-0.19.1/lib/thor/invocation.rb:126:in
invoke_command'
from /var/lib/gems/1.9.1/gems/thor-0.19.1/lib/thor.rb:359:in dispatch' from /var/lib/gems/1.9.1/gems/thor-0.19.1/lib/thor/base.rb:440:in
start'
from /var/lib/gems/1.9.1/gems/thor-0.19.1/lib/thor/runner.rb:36:in method_missing' from /var/lib/gems/1.9.1/gems/thor-0.19.1/lib/thor/command.rb:29:in
run'
from /var/lib/gems/1.9.1/gems/thor-0.19.1/lib/thor/command.rb:126:in run' from /var/lib/gems/1.9.1/gems/thor-0.19.1/lib/thor/invocation.rb:126:in
invoke_command'
from /var/lib/gems/1.9.1/gems/thor-0.19.1/lib/thor.rb:359:in dispatch' from /var/lib/gems/1.9.1/gems/thor-0.19.1/lib/thor/base.rb:440:in
start'
from /var/lib/gems/1.9.1/gems/thor-0.19.1/bin/thor:6:in <top (required)>' from /usr/local/bin/thor:23:in
load'
from /usr/local/bin/thor:23:in `
hi Tom
I just have installed datashift and datashift_spree from rubygems with Spree 2-1-stable.
and created mysite.thor as README.
require 'datashift'
require 'datashift_spree'
DataShift::load_commands
When command "bundle exec thor list datashift" , got uninitialized constant.
Is there a way to fix this?
bundle exec thor list datashift
WARNING: unable to load thorfile "/home/digitalm/public_html/railsapp/spreestore/mysite.thor": uninitialized constant DataShift::Logging
/home/digitalm/public_html/railsapp/spreestore/vendor/bundle/ruby/1.9.1/gems/datashift-0.14.0/lib/datashift/method_mapper.rb:22:in `<class:MethodMapper>'
No Thor commands available
In addition , github repo cannot be installed currently.
bundle install
Updating git://github.com/autotelik/datashift.git
There was a NameError while loading datashift.gemspec:
uninitialized constant DataShift::Logging from
/home/digitalm/public_html/railsapp/spreestore/vendor/bundle/ruby/1.9.1/bundler/gems/datashift-14ca51f5f51e/datashift.gemspec:7:in `<main>'
Just for information : Rails 4.0.2 / ruby 1.9.3p484
Best regards,
Mika
Sorry for noobish question.
But. how to attach images to product ?
I have file myapp/db/datashift/SpreeProductsWithImages.csv which I've copied from https://github.com/autotelik/datashift/blob/master/spec/fixtures/spree/SpreeProductsWithImages.csv
And folder with images, myapp/db/datashift/fixtures/images tooked from there
https://github.com/autotelik/datashift/tree/master/spec/fixtures/images
I run:
$> bundle exec thor datashift:spree:products -i db/datashift/SpreeProductsWithImages.csv
There I get products which migrated right but with no images.
DataShift::Product starting upload from file: db/datashift/SpreeProductsWithImages.csv
WARNING: Following column headings could not be mapped : ["Images"]
Loading from CSV file: db/datashift/SpreeProductsWithImages.csv
Processing 3 rows
Created missing OptionType #<Spree::OptionType id: 5, name: "mime_type", presentation: "Mime type", created_at: "2012-07-08 22:32:08", updated_at: "2012-07-08 22:32:08", position: 0>
Created missing OptionType #<Spree::OptionType id: 6, name: "print_type", presentation: "Print type", created_at: "2012-07-08 22:32:09", updated_at: "2012-07-08 22:32:09", position: 0>
Then i run
$> bundle exec thor datashift:spree:images -i db/datashift/fixtures/images
So, how to upload images using specs tamplates ?
In which folder images must be located and what's way to declare to each image?
Thanks for awesome gem and your work!
Hello.
Recently you had a plans in todo to make a gui for importing/exporting.
Could we know some news about it?
Thanks.
My thor file looks like this:
require 'datashift'
require 'datashift_spree'
DataShift::load_commands
DataShift::SpreeHelper::load_commands
My Gemfile looks like this
gem 'spree', github: 'spree/spree', branch: '3-0-stable'
gem 'spree_gateway', github: 'spree/spree_gateway', branch: '3-0-stable'
gem 'spree_auth_devise', github: 'spree/spree_auth_devise', branch: '3-0-stable'
gem 'datashift'
gem 'datashift_spree'
When I do bundle exec thor list datashift
I get
WARNING: unable to load thorfile "/home/sanjay/git/growchef/growchef.thor": uninitialized constant DataShift::Logging
/home/sanjay/.rvm/gems/ruby-2.2.1/gems/datashift-0.15.0/lib/datashift/method_mapper.rb:22:in `class:MethodMapper'
No Thor commands available
Please help.
Thanks
Sanjay
If the Price column is a currency type containing symbol get a very nasty bug where the Product saves and save returns true but there is no master Variant. This is particularly nasty as will crash the main site as Products found, but access to master causes no method errors on nil.
Presume the price field is set to something on Master, not nil, so Product saves but then the underlying Variant fails to save when checks done on validity of Price field.
Odd since if you leave the Price column completely blank then the Product itself will not save, validation error something like 'Price must be set'
While I'm trying to parse csv file I received #<NoMethodError: undefined method `type_cast_from_database' for #<ActiveRecord::ConnectionAdapters::Column:0x007fd36dc5f358>>
Probably it's problem with rails 5 where this method for column is removed. Do you know any simple solution to fix it? Do you plan to release new, rails 5 compatible, datashift version?
Getting this error when running datashift:generate:csv
> bundle exec thor datashift:generate:csv -m Spree::Product -r spree_export_template.csv -a
warning: parser/current is loading parser/ruby21, which recognizes
warning: 2.1.5-compliant syntax, but you are running 2.1.0.
uninitialized constant Thor::Sandbox::Datashift::Generate::ModelMapper
ERROR: No such Model [Spree::Product] found - check valid model supplied
Following https://github.com/autotelik/datashift/wiki/Walkthroughs
but keep getting errors in the datashift.log about Cannot Save Spree::Product : ["Price can't be blank"]
whereas the prices
column in the spreadsheet is set with a numeric value.
My fault. Sorry for posting.
My csv like
name,sku,price,images
Aka,EBK.3435,10.00,fixtures/images/DEMO_001_ror_bag.jpeg|fixtures/images/DEMO_002_ror_bag.jpeg
when i run
bundle exec thor datashift_spree:load:products -i, --input=Sample_image.csv
It gives error
Loading from CSV file: Sample.csv
Processing 1 rows
wrong number of arguments (2 for 1)
Processing Summary Report
Processed total of 1 entry
0 objects were succesfully processed.
WARNING : Check logs : 1 rows contained errors and 1 records NOT created.
Can you please tell me what is the issue here??
its urgent.
Thanks a lot
Hi Tom,
I'm a newbie to Rails and Datashift and am trying to use some of the excel commands but continue to get
'Apologies but DataShift Excel facilities currently need JRuby. Please switch to, or install JRuby'
However, I have JRuby 1.6.7.2 installed on my machine. Is there something that I'm missing to get Datashift to recognize this?
Also, alternatively is there a way to export tables into a csv format? I see there's a way using import but none for export.
Thanks so much for your help and this tool is great!
-Jason
I'm hoping to use this for a project where I have some rather complicated import/export logic, and am wondering if there are any examples of this nature (where you actually extend the generator/importer). If not, would it be possible to create an example, or point me in the right direction and I can maybe write one? :)
Consider the use case of a Post that belongs to many Sub Categories, with each Sub Category belonging to a Parent Category. In my case, the logic is such that a Post must belong to one (and only one) of each Parent Category's Sub Categories. Say that we have a "Department" Parent Category, and a "Project" Parent Category. Each Post must belong to one of "Department"'s Sub Categories, and one of "Project"'s Sub Categories. However, the data is manipulated by humans as part of the import/export process, so I'm hoping to have an excel sheet that looks something like the following:
post_id | other_field | category:Department:sub_category | category:Project:sub_category |
---|---|---|---|
1 | something | Accounting | Greenhouse |
2 | a value | Accounting | Bridge |
3 | value 2 | Engineering | Greenhouse |
4 | something | Engineering | Bridge |
I exported my list of products ( not spree but custom made application) that have all an ID, when I import the excel file again with my changes ( description ), he tells me that there is a duplicate entry with ID x.
Is it possible with datashift to map an existing record with the ID, so when the ID exists that he updates the row with that specific ID with the values? We want to use the excel file to keep the products up to date on the website because the client is working internaly with this excel sheet.
I've been having some problems similar to other users with loading images. I finally found and followed your recipe in Issue #6 and got it to load products and images from 'SpreeProductsWithImages.csv'. It doesn't seem to work right when I try to do the images separately.
My file arrangement is:
spreestore/ (rails root)
spreestore/db/datashift/SpreeProductsWithImages.csv
spreestore/fixtures/images/{the image files}
SpreeProductsWithImages.csv has paths that are: 'fixtures/images/filename.jpeg'
So I sorted out that the file path is relative to where thor is running from, rather than from the directory the csv is in.
When I try to do it by loading products first using 'SpreeProducts.csv' (in the same directory as 'SpreeProductsWithImages.csv') I can load the products just fine, but then when I try to load images with:
bundle exec thor datashift:spree:images -i fixtures/images -s
I get this set of messages and no images in the store:
Using SKU for lookup
CONFIG: {"split_file_name_on"=>"", "input"=>"fixtures/images", "sku"=>"sku", "load"=>false}
OPTIONS {"split_file_name_on"=>"", "input"=>"fixtures/images", "sku"=>"sku"}
Will scan image names splitting on delimiter : _
WARNING : 1 of 1 images could not be attached to a Product
Copying all images with MISSING Records to ./MissingRecords >>
Copy fixtures/images/DEMO_002_Powerstation.jpg to MissingRecords folder
I'm running Rails 3.2.6, Spree 1.1.2, and datashift 0.80 from git. I've tweaking the datashift docs a bit as I figure out how it works-- I'd like to extend it to use it for loading products and digital files for spree_digital.
Currently, the gem is dependent upon Rails 4.2
Any plan to support Rails 5? Thx :)
What is this? No docs? What's the right syntax to use please
Currently we have wiki: Options and Configuration, but it's quite incomplete.
I want to learn how to:
Not sure this is an issue or me missing a step/clue, but I don't seem to have any thor task that can export or import as CSV.
$ be thor list datashift
WARNING: unable to load thorfile "/Users/sbeam/.rvm/gems/jruby-1.6.7.2@global/gems/datashift-0.8.0/lib/thor/spree/bootstrap_cleanup.thor": admin_interface_log preference not defined
/Users/sbeam/.rvm/gems/jruby-1.6.7.2@global/gems/spree_core-1.1.1/app/models/spree/preferences/preferable.rb:60:in `has_preference!'
datashift
---------
thor datashift:export:excel -m, --model=MODEL -r, --result=RESULT # export...
thor datashift:generate:excel -m, --model=MODEL -r, --result=RESULT # genera...
thor datashift:import:excel -i, --input=INPUT -m, --model=MODEL # import...
thor datashift:reports:missing_images # Spree ...
thor datashift:spree:attach_images -i, --input=INPUT # Popula...
thor datashift:spree:images -i, --input=INPUT # Popula...
thor datashift:spree:products -i, --input=INPUT # Popula...
we actually don't have any requirement to use Excel files, so was hoping to use the CSV functionality (though it might be nice, but we were planning on deploying under MRI 1.9.3 so jruby compatibility might be an issue)
Currently, the file structure is like this:
/path/to/datashift/
▾ lib/
▸ applications/
▸ datashift/
▸ exporters/
▸ generators/
▸ java/
▸ loaders/
▸ tasks/
datashift.rb
▸ spec/
CHANGELOG.md
datashift.gemspec
datashift.thor
Gemfile
LICENSE
md_style.rb
Rakefile
README.md
As we can see, folders like applications
are under the lib root path directly. Following this post about best practices, we can see that the current approach would pollute the global load path:
suppose there's another library called loaders
, then the current list of files in loaders/files
would pollute the original java library.
I'd suggest putting all files under lib/datashift/
folder.
Hi there,
I'm running into an error every time I try to generate a template with bundle exec thor datashift:generate:excel --model Spree:Product --result /home/devs/excel.xls --assoc
.
devs@barakah:~/artilujos$ bundle exec thor datashift:generate:excel --model Spree:Product --result /home/devs/excel.xls --assoc
/home/devs/.rbenv/versions/2.1.3/lib/ruby/gems/2.1.0/gems/durable_decorator-0.2.2/lib/durable_decorator/util.rb:46:in `logger': uninitialized constant Logging (NameError)
from /home/devs/.rbenv/versions/2.1.3/lib/ruby/gems/2.1.0/gems/durable_decorator-0.2.2/lib/durable_decorator/validator.rb:34:in `handle_soft_fault'
from /home/devs/.rbenv/versions/2.1.3/lib/ruby/gems/2.1.0/gems/durable_decorator-0.2.2/lib/durable_decorator/validator.rb:22:in `validate_decoration_meta'
I've seen a few similar issues reported and tried many things: adding 'include DataShift::Logging' and 'require 'logging'' to the util.rb file, fiddling with different versions and making the changes suggested in this thread autotelik/datashift_spree#1
I've run out of ideas and so I'd appreciate if you could point me in the right direction.
Gems are listed like this, at the bottom of my gemfile:
gem 'datashift', :git => 'http://github.com/autotelik/datashift'
gem 'datashift_spree', :git => 'http://github.com/autotelik/datashift_spree'
Thanks!
way I've implemented options causes issues with thor help/usage
$ th datashift:import:csv
No value provided for required options '--model', '--input'
Hello Tom,
I gave a new try on my file with multiple sub-categories of same name.
I have the following error :
$ bundle exec thor datashift:spree:products -i Omega_checked.csv
/home/emb/.../bundler/gems/datashift-6b6bd8c89874/lib/loaders/paperclip/image_loader.rb:14:in `<module:ImageLoading>': uninitialized constant DataShift::Paperclip (NameError)
from /home/emb/.../bundler/gems/datashift-6b6bd8c89874/lib/loaders/paperclip/image_loader.rb:12:in `<module:DataShift>'
from /home/emb/.../bundler/gems/datashift-6b6bd8c89874/lib/loaders/paperclip/image_loader.rb:10:in `<top (required)>'
from /home/emb/.../gems/polyglot-0.3.3/lib/polyglot.rb:63:in `require'
from /home/emb/.../gems/polyglot-0.3.3/lib/polyglot.rb:63:in `require'
from /home/emb/.../bundler/gems/datashift-6b6bd8c89874/lib/loaders/spree/product_loader.rb:11:in `<top (required)>'
from /home/emb/.../gems/polyglot-0.3.3/lib/polyglot.rb:63:in `require'
from /home/emb/.../gems/polyglot-0.3.3/lib/polyglot.rb:63:in `require'
from /home/emb/.../bundler/gems/datashift-6b6bd8c89874/lib/thor/spree/products_images.thor:36:in `products'
from /home/emb/.../gems/thor-0.16.0/lib/thor/task.rb:27:in `run'
from /home/emb/.../gems/thor-0.16.0/lib/thor/invocation.rb:120:in `invoke_task'
from /home/emb/.../gems/thor-0.16.0/lib/thor.rb:275:in `dispatch'
from /home/emb/.../gems/thor-0.16.0/lib/thor/base.rb:425:in `start'
from /home/emb/.../gems/thor-0.16.0/lib/thor/runner.rb:36:in `method_missing'
from /home/emb/.../gems/thor-0.16.0/lib/thor/task.rb:29:in `run'
from /home/emb/.../gems/thor-0.16.0/lib/thor/task.rb:126:in `run'
from /home/emb/.../gems/thor-0.16.0/lib/thor/invocation.rb:120:in `invoke_task'
from /home/emb/.../gems/thor-0.16.0/lib/thor.rb:275:in `dispatch'
from /home/emb/.../gems/thor-0.16.0/lib/thor/base.rb:425:in `start'
from /home/emb/.../gems/thor-0.16.0/bin/thor:6:in `<top (required)>'
from /home/emb/.../bin/thor:19:in `load'
from /home/emb/.../bin/thor:19:in `<main>'
from /home/emb/.../bin/ruby_noexec_wrapper:14:in `eval'
from /home/emb/.../bin/ruby_noexec_wrapper:14:in `<main>'
The file was correctly imported with stable version.
Please, tell me which information you need, if you need any more.
Cheers,
Manu
Looks like specifying a default available_on for Product not being picked up in ProductLoader
.. pretty sure general setting defaulots from config specs pass though so maybe something specific to db dates ?
Could not find spec/fixtures/spree directory for browsing examples.
hi @autotelik
I've been recently giving datashift a shot with my spree and it is really nice it works, really nice job thx!
However I am struggling with my product import which has the following option_types:
-size:small,medium
-color:black,red
it creates one variant per option value as described in the wiki:
What if I want variants to look like this :
Thanks
Stephane.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.