paulspringett / csv_shaper Goto Github PK

It's ideal for converting database backed models with attrbiutes into CSV output.

Should be attributes

csv_shaper is a nice library/DSL, but performance about an order of magnitude slower than creating my own CSV. My simple test with about 5,000 rows in the resulting CSV file took 6 seconds creating the CSV manually as below, and took 70 seconds using csv_shaper! I used the same code to generate the @Entities result set which was < 1 second. All remaining time was in rendering.

my controller method

 def entities
    @entities = Entity.all.includes(:state)
    @headers =  ['ID', 'State' ,'Name']
    @data = []
    @entities.each do |e|
      @data << [e.id, e.state.short_name, e.name]
    end
    content = CSV.generate { |csv| csv << @headers; @data.map { |x| csv << x }; }
    ...
  end

entities.csv.shaper

csv.headers :id, :state_id, :name

csv.rows @Entities do |csv, entity|
csv.cells :id, :name
csv.cell :state_id, entity.state.short_name
end

I'm sure I've got this set up correct, but is not returning an actual CSV file for download - what am I missing? I can inspect the csv_string and it has the values I expect in there, just the render csv command seems to be doing nothing?

  csv_string = CsvShaper.encode do |csv|
    csv.headers :user_id, :total_leave_requested
    csv.rows @results do |csv, result|
      csv.cell :user_id, result.user_id
      csv.cell :total_leave_requested, result.total_leave_requested
    end
  end   
  render csv: csv_string

Hi there, this seems like a pretty likely scenario:

csv.headers do |csv| 
  csv.columns :state, :building_name

end

csv.rows @warranties do |csv, warranty|
  csv.cells :state

  if warranty.building
    csv.cell :building_name, warranty.building.name
  end
end

At the moment I get a undefined methodbuilding_name' for #Warranty:0x007fbc0890ece8` error.

I guess it's because the code is calling the building_name method on the warranty even though we're not going to be using that method to get the actual value?

I have a user case for .csv footers. I'd be willing to try to work up a PR for this if it's something you think other might be interested in.

Have you considered this before?

setting @filename didn't work for me as specified in the doco.

I'm in a bit of a hurry so I just set the response header myself and didn't look into it further - just thought you would like to know.

I was generating a CSV of about 300,000 rows and seeing a lot of memory usage.

Probably the best approach is to support 'streaming' (writing one row at a time) for large files, rather than generating the whole document in memory and then writing it out. I may have a crack at this when I get time, and send a pull request.

In the meantime I was in a hurry, so I made a very hacky monkey-patch to reduce the memory footprint. It is pretty nasty, but I thought you might be interested.

module CsvShaper
  class Encoder    
    # Monkey Patch to reduce memory usage while generating the csv.
    # - CSV::Rows are converted to a string immediately as they have a large footprint
    # - single map! rather than nested calls to map (note: this has nasty side effect of making this instance un-reusable)
    # This is a quick and nasty solution. A better approach would be to write out the rows as they are generated
    # rather than build the whole CSV document in memory. 
    def to_csv(local_config = nil)
      csv_options = options.merge(local_options(local_config))
      cols = @header.mapped_columns
      @rows.map! do |row|
        padded_row = CSV::Row.new(row.cells.keys, row.cells.values).values_at(*@header.columns)
        CSV::Row.new(cols, padded_row, false).to_csv
      end
      cols.join(",") + "\n" + @rows.join
    end
  end
end

Hi,

What do you think abut using @csv_filename instead of @filename which is very generic and can be confusing ?

Currently getting a lot of noise in PRs from existing code

I am getting quotes in the CSV file name being generated from the app.
Rails 3.2.6. It happens whether or not I set @filename or not. In fact, if I set @filename to xyz.csv, the code produces the file (with quotes) 'xyz.csv'.csv.

Unless I'm completely blind, I cannot seem to find a way to render partials from a csv.shaper template. A simple example would have the parent template containing a line like

render "invoices/csv/calls_summary"

with a file in that directory called calls_summary.csv.shaper. However when I try to render, I get the error message

you must define some headers using csv.headers ...

Am I missing or misunderstanding something?