Sure! Here's what we're trying to do. This is a bit simplified, but the issue with Rack vs. Rails responses still applies. about falcon

[ioquatix]

@pschrammel I'm adding a fully working example here: socketry/async-examples#14

[ioquatix]

Okay, it's now working:

class StreamingController < ApplicationController
  def download
    headers = {
      'content-type' => 'application/json',
      'cache-control' => 'no-cache',
      'last-modified' => Time.now.httpdate
    }
    
    body = lambda do |stream|
      chunks = Async::Queue.new
    
      request_task = Async do
        $stderr.puts "Downloading..."
        response = Faraday.get("https://httpbin.org/stream/100") do |request|
          request.options.on_data = proc do |chunk|
            $stderr.puts "Received chunk: #{chunk.bytesize}"
            chunks.enqueue(chunk)
          end
        end
        $stderr.puts "Downloaded: #{response}"
        chunks.enqueue(nil)
      ensure
        $stderr.puts "Request Exiting: #{$!}"
      end
      
      while chunk = chunks.dequeue
        stream.write(chunk)
      end
    ensure
      $stderr.puts "Response Exiting: #{$!}"
      request_task&.stop
      stream.close
    end
    
    self.response = Rack::Response[200, headers, body]
  end
end

Client output:

> curl http://localhost:3000/streaming/download
{"url": "https://httpbin.org/stream/100", "args": {}, "headers": {"Host": "httpbin.org", "X-Amzn-Trace-Id": "Root=1-66c904ae-4ecedec9459327b3148d9705", "User-Agent": "Faraday v2.10.1"}, "origin": "119.224.53.223", "id": 0}
{"url": "https://httpbin.org/stream/100", "args": {}, "headers": {"Host": "httpbin.org", "X-Amzn-Trace-Id": "Root=1-66c904ae-4ecedec9459327b3148d9705", "User-Agent": "Faraday v2.10.1"}, "origin": "119.224.53.223", "id": 1}
{"url": "https://httpbin.org/stream/100", "args": {}, "headers": {"Host": "httpbin.org", "X-Amzn-Trace-Id": "Root=1-66c904ae-4ecedec9459327b3148d9705", "User-Agent": "Faraday v2.10.1"}, "origin": "119.224.53.223", "id": 2}
{"url": "https://httpbin.org/stream/100", "args": {}, "headers": {"Host": "httpbin.org", "X-Amzn-Trace-Id": "Root=1-66c904ae-4ecedec9459327b3148d9705", "User-Agent": "Faraday v2.10.1"}, "origin": "119.224.53.223", "id": 3}
{"url": "https://httpbin.org/stream/100", "args": {}, "headers": {"Host": "httpbin.org", "X-Amzn-Trace-Id": "Root=1-66c904ae-4ecedec9459327b3148d9705", "User-Agent": "Faraday v2.10.1"}, "origin": "119.224.53.223", "id": 4}
{"url": "https://httpbin.org/stream/100", "args": {}, "headers": {"Host": "httpbin.org", "X-Amzn-Trace-Id": "Root=1-66c904ae-4ecedec9459327b3148d9705", "User-Agent": "Faraday v2.10.1"}, "origin": "119.224.53.223", "id": 5}
{"url": "https://httpbin.org/stream/100", "args": {}, "headers": {"Host": "httpbin.org", "X-Amzn-Trace-Id": "Root=1-66c904ae-4ecedec9459327b3148d9705", "User-Agent": "Faraday v2.10.1"}, "origin": "119.224.53.223", "id": 6}
... snip ...

[ioquatix]

In theory this should work, but there are some issues with the protocol-rack gem (in principle) that make it more difficult than it should be.

Let me investigate a bit more. We definitely want this to work.

[ioquatix]

The full working example is here: https://github.com/socketry/async-examples/tree/main/rails-streaming-download