pcreux / csv-importer

CSV Import for humans on Ruby / Ruby on Rails

Comments for articles posted on https://pcreux.com.

👉 https://github.com/pcreux/pcreux/issues

csv-importer's Issues

Config can be passed at runtime

Example:

ImportSubscriberCSV.new(model: newsletter.subscribers, file: file)

Demo?

Thanks for the CSV importer. Maybe asking for too much but is there a demo site for this? :)

Empty Row Column at beginning of csv gives an error.

When ever I have a file like in the image below, with an empty row above the headers and a empty column before the first one it breaks the importer with an error. i.e I get a 'email column is missing' is this a known issue? or am I missing something?

Using after_save in block change class after_save_blocks array

I use cscv-importer in conroller:

    import = Importers::ImportMemberCSV.new(file: params[:import_file]) do
      after_save { |member| list.subscribe(member) }
    end
    import.run!

Nonetheless I use after_save in block, it adds proc into Importers::ImportMemberCSV.config.after_save_blocks array

class Importers::ImportMemberCSV
  include CSVImporter

  model Member

  column :email, to: ->(email) { email.downcase }, required: true
  column :first_name
  column :last_name
  column :ip

  when_invalid :skip
end

Postgresql Array format

Hi,
I'm using an array of integers but csv-importer fails silently in this case. Here is an example sensor.csv with a name, date and an array of temperatures
sensor01,2017-01-28,14 17 19

Is there a way to specify that the last field is an array?

Edit: I know that I can reformat the csv file to something like
sensor01,2017-01-28,"{14, 17, 19}"
but maybe there's a clever way to do it in the class without modifing the csv file

Implement .new(file: IOStream)

Add i18n to messages

All messages, such as The following columns are required: should be put into locale files, so that could be replaced on the client side.

Report should have built in success / error message

Implement skip or abort

Current behavior is "skip".

We should wrap all imports in a transaction.

When selecting abort, we should call save!

Add mechanism to skip row

skip_if do |row|
  row.raw[/^#/] # skip rows starting with '#'
end

or ...

after_build do |model|
  throw :skip if model.draft?
end

Importer class never found

I couldn't get this to work in a reasonable amount of time, so I was wondering where you typically place your importer files in a Rails application.

I tried putting my ImportItemsCSV in the models directory but even then I end up getting NoMethodError - undefined methodnew' for nil:NilClass:` when I try to instantiate it in a controller.

add "after_save" callback

can be defined at the class level or passed at runtime

class ImportUserCSV

  # ...

  after_save do |user|
    user.imported!
  end
end

or

progress = ProgressBar.new
ImportUserCSV.new(file: da_file, after_save: -> { increment_progress_bar })

or

ImportUserCSV.new(file: da_file) do
  after_save do |user|
    user.update!(account: current_account)
  end
end

Give CSV columns a type to Coerce to

column :published, type: Boolean
column :published_at, type: DateTime
column :reviews, type: Integer

Get current_user into CSVImporter::Row for after_build

Is there a good way to get current_user into the context for an after_build or after_save?

Release gem v0.1

• add missing infos to gemspec
• add csv-importer.rb that requires csv_importer.rb

Lazy load if possible

When loading CSV from a Input Stream, don't read the entire file. Lazy load rows instead.

Rescue from CSV::MalformedCSVError

Empty cells should return empty strings instead of nil

This should declutter the code a bit:

column :email, to: ->(email) { email.downcase.strip if email }
column :name, to: ->(name) { name || "N/A" }

would become

column :email, to: ->(email) { email.downcase.strip }
column :name, to: ->(name) { name.presence || "N/A" }

Exception when model.save fails

Having a model validator like this:

   validate :check_price
   ...
   ...
   def check_price
      if price.nil? && Spree::Config[:require_master_price]
        raise 'No master variant found to infer price' unless product && product.master
        raise 'Must supply price for variant or master.price for product.' if self == product.master
        self.price = product.master.price
      end
      if price.present? && currency.nil?
        self.currency = Spree::Config[:currency]
      end
    end

persisted_rows! method in Runner class will fail trying to save the model instead of just adding a new failure tag.

Jruby support?

Is this gen compatible with jruby and if so is there syntax changes or setup requirements I'd need to be aware of?

strip, downcase, underscore columns

Headers "first name", "First name", "first_name" should all be converted to :first_name

Generate CSV file with errors

Once a csv file has been imported, the report object should provide a csv_file_with_errors that is identical to the original one (same headers, same order), with only the rows that errored and with an extra column that contains the validation error.

Error on importing with missing required field

6:  column :email, to: ->(email) { email.downcase }, required: true

When I run an import with a CSV , I get an error ...

NoMethodError: undefined method `downcase' for nil:NilClass
on line 6:

I would expect this line in the CSV to simply be skipped because it is required? I could guard against the exception but then it would create a record with a blank email.

Thanks..

UPDATE: Not raising an error works and doesn't create the record. Not ideal, I think the block shouldn't be run unless email is present but it's an acceptable workaround

6:  column :email, to: ->(email) { email && email.downcase }, required: true

We should support csvs with missing `identifier`

Say you want to update records via id. You should be able to import a csv file without an id column to create records.

Report should respond to success?

CSV::MalformedCSVError: Illegal quoting on a line like 1,Glk,Lev "The Black Spider" Yashin,USSR

irb example:

2.1.5 :023 > str = "1,Glk,Lev \"The Black Spider\" Yashin,USSR"
 => "1,Glk,Lev \"The Black Spider\" Yashin,USSR"
2.1.5 :024 > CSV.parse(str)
CSV::MalformedCSVError: Illegal quoting in line 1.
    from /Users/shvetsovdm/.rvm/rubies/ruby-2.1.5/lib/ruby/2.1.0/csv.rb:1857:in `block (2 levels) in shift'
    from /Users/shvetsovdm/.rvm/rubies/ruby-2.1.5/lib/ruby/2.1.0/csv.rb:1819:in `each'
    from /Users/shvetsovdm/.rvm/rubies/ruby-2.1.5/lib/ruby/2.1.0/csv.rb:1819:in `block in shift'
    from /Users/shvetsovdm/.rvm/rubies/ruby-2.1.5/lib/ruby/2.1.0/csv.rb:1779:in `loop'
    from /Users/shvetsovdm/.rvm/rubies/ruby-2.1.5/lib/ruby/2.1.0/csv.rb:1779:in `shift'
    from /Users/shvetsovdm/.rvm/rubies/ruby-2.1.5/lib/ruby/2.1.0/csv.rb:1721:in `each'
    from /Users/shvetsovdm/.rvm/rubies/ruby-2.1.5/lib/ruby/2.1.0/csv.rb:1735:in `to_a'
    from /Users/shvetsovdm/.rvm/rubies/ruby-2.1.5/lib/ruby/2.1.0/csv.rb:1735:in `read'
    from /Users/shvetsovdm/.rvm/rubies/ruby-2.1.5/lib/ruby/2.1.0/csv.rb:1293:in `parse'
    from (irb):24
    from /Users/shvetsovdm/.rvm/rubies/ruby-2.1.5/bin/irb:11:in `<main>'

when_invalid :abort only report the first invalid record

so, how to make it report all invalid records, thanks

CSV files with comments

Some of our users have decided to include "comments" at the top of the CSV files. I know, this sucks, and not allowed in CSV, but... is there any way we can exclude some lines, ideally I'd like to run some lambda to check if the line should be processed. Is this possible?

Thanks, I get this isn't really your fault and this would be invalid CSV but ... users.. :)

How can I handle special characters like á, ü, ñ etc...

Multiple models to be used for a single row

Our CSV's single row represents multiple models, which may or may not have an ActiveRelation amongst themselves. it would be awesome to have multiple models with columns mapping for each models.

identifier can take many columns

It can find or create using multiple columns

  # find or create by name AND website
  identifier :name, :website

add after_build callback

can be defined at the class level or passed at runtime

class ImportUserCSV

  # ...

  after_build do |user|
    user.active = true
  end
end

or

ImportUserCSV.new(file: da_file, after_build: ->(user) { user.account = current_account })

or

ImportUserCSV.new(file: da_file) do
  after_build do |user|
    user.account = current_account
  end
end

Upload file

Can I upload only CSV file in director and save it in my rails app if yes how I can only upload?

Not really an issue but ...

... I just wanted to say that I love you ❤️

Cheers guys, you're awesome :)

Timeouts

Whenever I try to use this feature, I get timeouts on production. Any suggestions on moving this to a background job (we're using AWS SQS)?

Duplicate call to after_build blocks

after_build blocks are executed when in :set_attributes method.
:set_attributes is called two times for each row (in :model and :find_model).
I consider this a bug because I'm expecting after_build being called only one time for each row.

not able to read Japanese and Chinese content

If the csv file contents Japanese or Chinese, then will return nil.

Errno::ENOENT (No such file or directory @ rb_sysopen" ... )

When I pass in the url S3 url, I get "No such file or directory @ rb_sysopen". Does the "path" not accept a full S3 url?

find separator

csv_options[:col_sep] = [";","\t",","].sort_by{|separator| csv_sample.count(separator)}.last

Memory bloat when processing large files

I run into memory bloat when processing large files (50K+ rows). This can quickly be an issue for containerized environments. It think there is optimization potential in CSVImporter ::Runner#persist_row! and CSVImporter::Row#model. Correct me if I'm wrong, but right now the importer holds onto an ActiveRecord Model for every imported row until the import is complete.

I'll dig a bit more into this when I have the time, I just want to get the discussion going.

Discover if columns are required from Model definition?

Given:

class User
  validates :email, presence: true
end

The importer should figure that email is a required column. 😸

Implement .new(file_path: "a path")

`Header#*_columns` methods should return `Column` objects

And the ReportMessage should take care of generating error messages

Combine multiple columns into one

Is it possible to combine several columns from the csv file into one model column? For example, I have: Vendor Name, Vendor Phone, Vendor Email, etc ... and want to merge into just one column Vendor, separate each by new lines instead.

allow columns with accents - match columns via regexp or array

  column :first_name, as: "Prénom"
  column :first_name, as: /name/
  column :first_name, as: /prénom/i # case insensitive
  column :first_name, as: ["First name", "first name", "firstname", "name"]

Strip CSV cells

on_error is not a valid option

It's present in the README but I can't find it in the source code. Maybe just a typo for when_invalid

Pass identifier in with params

Is there a way I could pass an identifier to csv-importer so that a user could choose the identifer(s)?

handling of model associations in an import?

I'm using cvs-importer to import hardware details.. lets say:

model Host

And that model has a has_many :mac_addresses relationship.

Is it possible to perform an update/create for the corresponding MacAddress entries from a column in the CSV?

Pass Column as parameter in "to" procs

I'd like to add a 3rd option to this case where the block passed uses the column (not column_definition) too. This allows to do custom conversion with metadata stored in the column name. For example the columns header is defined as "name, birthdate[%d/%m/%y], address". So the column name matches trough a regex and you can capture the metadata inside []

    case to_proc.arity
    when 1 # to: ->(email) { email.downcase }
      model.public_send("#{column_definition.name}=", to_proc.call(csv_value))
    when 2 # to: ->(published, post) { post.published_at = Time.now if published == "true" }
      to_proc.call(csv_value, model)
   when 3 [some code]
    else
      raise ArgumentError, "`to` proc can only have 1 or 2 arguments"