Consumers stop requesting events if they process events too quickly about gen_stage

[jschneider1207]

After some more testing, it seems the sleep time required increases with the amount of parallel chains of:

producer_consumer -> producer_consumer -> consumer

I have subscribed to the one initial producer. With a parallelism of 10, the consumers need around 10-20ms sleep time (doesn't seem to be 100% consistant)

[josevalim]

@sschneider1207 are your producer_consumer filtering events? Because I have just found a bug that, if the producer_consumer emits less events than it receives, the situation described above would happen. If not, if you can provide a way to reproduce this issue, it would be appreciated.

[josevalim]

@sschneider1207 please try master as I have fixed the issue I mentioned above. If it is not enough, please try to include a mechanism to reproduce the error so I can further investigate it!

[josevalim]

@sschneider1207 sorry, I didn't see you actually provided a link. Let me give it a try. :D

[josevalim]

Actually, it seems I will need elastic search... so I won't be able to try it for now. Please give it a try and let me know.

[jschneider1207]

@josevalim you don't need the elasticsearch bit, I'm just using the to create the initial list of events. My producer_consumers are just applying a single mapping function and returning the same number of events so I don't think this is the same issue. Here's an example minimum script that I put together that is roughly similar to what I'm doing that locks up after 4 batches:

alias Experimental.GenStage

defmodule A do
  use GenStage

  def init(:ok) do
    {:producer, nil}
  end

  def handle_demand(demand, nil) do
    events = Enum.to_list(1..demand)
    {:noreply, events, nil}
  end
end

defmodule B do
  use GenStage

  def init(:ok) do
    {:producer_consumer, nil}
  end

  def handle_events(events, _from, nil) do
    events = Enum.map(events, fn x -> List.duplicate(x, x) end)
    {:noreply, events, nil}
  end
end

defmodule C do
  use GenStage

  def init(:ok) do
    {:producer_consumer, nil}
  end

  def handle_events(events, _from, nil) do
    events = Enum.map(events, &Enum.reduce(&1, 0, fn x, acc -> x + acc end))
    {:noreply, events, nil}
  end
end

defmodule D do
  use GenStage

  def init(:ok) do
    {:consumer, nil}
  end

  def handle_events(events, _from, nil) do
    IO.inspect self
    {:noreply, [], nil}
  end
end

{:ok, a} = GenStage.start_link(A, :ok)
{:ok, b} = GenStage.start_link(B, :ok)
{:ok, c} = GenStage.start_link(C, :ok)
{:ok, d} = GenStage.start_link(D, :ok)

GenStage.sync_subscribe(d, to: c)
GenStage.sync_subscribe(c, to: b)
GenStage.sync_subscribe(b, to: a)

Process.sleep(:infinity)

[josevalim]

@sschneider1207 this is awesome, thank you! I will investigate it tomorrow! :D

[jschneider1207]

I'm not super familiar with how the different stage types are implemented, but I decided to turn tracing on one by one to see if I could notice any patterns. What I found was that enabling tracing on the C stage fixes it outright, so it might be at the root cause of this.
I also noticed that it seems like some of the ask messages might not be fully getting through:

• Stage A receives an initial ask of 1000 items, followed by six asks for 500 items.
• Stage B receives
  • one ask of 1000 items
  • four asks of 500 items
  • one batch of 1000 events
  • six batches of 500 events (so everything A sent)
• Didn't record anything from stage C as tracing added enough delay (?) to fix issue
• Stage D receives four batches of 500 events and locks up

Hopefully that is helpful.

[josevalim]

This has been fixed in master. Can you please confirm you see it fixed too? :D

[jschneider1207]

@josevalim Checked out the most recent commit and it is fixed!