About Streaming Large CSVs in Ruby on Rails

Asked 5 months ago, Updated 5 months ago, 11 views

I wanted to manipulate the data in the database and then output it to CSV, so I implemented it with reference to the following.

If the size is small, it will be fine, but if there are thousands of SQL issues, there will be more SQL issues.
It will take some time for the download to be ready.
(After pressing the download button, it takes about 30 minutes for all SQL issues to finish.)
If the size is small, it can be downloaded, but if the size is large, it will not be downloaded to the browser after publishing SQL (nothing happens).

I thought that the client might have timed out, so I implemented it with the following information as a streaming distribution.

If you do that, SQL will not run even if you press Download, and the data will be empty.
Where is the problem?I'll put the code on it.I also use Apache+Passenger.

class OutputController <ApplicationController
  def download
    self.response.headers ["Content-Type"] ||= 'text/csv;charset=Shift_JIS'
    self.response.headers ["Content-Disposition" = "attachment; filename=export_#{Time.now.to_i}.csv"
    self.response.headers ["Content-Transfer-Encoding"] = "binary"
    self.response.headers ["Last-Modified"] = Time.now.ctime.to_s
    @input=Input.find (params[:id])

    self.response_body=Enumerator.new do | yelder|
      @ input.baseurls.find_each do | baseurl |

In this case, @baseurls contains 1000 pieces of data, and I would like to pass it to download.csv.ruby in View to process 1000 pieces at a time.CSV.generate in download.csv.ruby.

ruby-on-rails ruby

2022-09-30 14:26

1 Answers

Based on the first reference, we assume that this render will create a huge CSV.In that case, no body data will flow until the render itself is executed.Therefore, if CSV creation takes a long time, it may have timed out because the body data did not come, and the body was processed empty. Compare the yelder<(render:content_type=>'text/csv') part to yelder<<@baseurl.to_s.

As a solution, if you want stream delivery, you can't use CSV.generate to create a string like the one in the first reference article.Because the method of this article is to create CSV all at once.To stream properly, CSVs must flow one row at a time into yelder.The following article will help you with the actual method.


*It may be possible with CSV.new(yelder), but I haven't tried it.

Alternatively, I would like to create a CSV separately and then change it to a download system.Rails 6 or higher should be combined with Active Storage.

2022-09-30 14:26

If you have any answers or tips

© 2023 OneMinuteCode. All rights reserved.