Flow.zip as counterpart to Stream.zip about gen_stage

[josevalim]

Hi @paulbalomiri,

So I have just realized zip is harder conceptually then we first thought. The issue is, imagine you are zipping two flows into one. The new flow will actually be made of four stages:

[A1] [A2] [A3] [A4]   [B1] [B2] [B3] [B4]

      ... all A connected to Z ...
      ... all B connected to Z ...

        [Z1]   [Z2]   [Z3]   [Z4]

The problem is, because flow is by default partitioned, every event happening in B, in this case, the ticker, will be delivered to a different partition per time. If the event is always of the same value, it will always be delivered to the same partition which means events routing to the other partitions will never be emitted.

One option is to set the stages in B to use the broadcast dispatcher. In your case it would work exactly as expected.

So implementation-wise this is not a problem but rather how we want to expose this problem in the API. One option is to have two different zip functions: partitioned_zip and broadcast_zip. On both cases A behaves the same but you can configure how B works. Another option is to provide a generic API, such as Flow.zip(left, left_opts, right, right_opts, options) that allows you to customize exactly how which side will behave.

Yet another option is to kip the Flow.zip(left, right, options) API and allow a tuple format to be given:

Flow.zip({left, left_options}, right, options)

This API could also be useful in the merge function, where you may want to specify different dispatchers per flow being merged. For those reasons I am inclined to go with the tuple format.

Any thoughts or questions before I move forward?

[paulbalomiri]

If the event is always of the same value, it will always be delivered to the same partition

I guess this means the :tick events need either

1. some form of identity in order to be partitioned (ticker event scarcity)
2. as you say a broadcaster, (just ratelimiting of the flow, which scales with the partition count)

(1) could be achieved by piping through |>Stream.with_index() for each flow before Flow.zip() merging.

I'm not sure I understand why partitioned_zip and broadcast_zip would be inherent to the zip functionality.

@josevalim I'm still thinking about this, and if it's not too late i'd like to add to this answer tomorrow, on the trip to sofia. I will not be the driver :D

[josevalim]

Oh, let's talk about it in Sofia then! Please say hi when you find me. :)

[josevalim]

After some discussion in private, we have identified another solution to @paulbalomiri's problem, so we are closing this although the work has been published to a branch. If another person shows interest in this feature, we can gladly reopen the discussion.

[safwank]

@josevalim @paulbalomiri Do you mind sharing the solution you came up with?

[josevalim]

@safwank there is a WIP implementation here: 67ad323

[paulbalomiri]

@safwank I ended up not using a zip, but a consumer using ask on what would be left and right in the zip arguments.

Ended up creating 2 libraries (not published as of now - see below), which i called

• table_stage, a generic postgres producer (fed from rows) and
• ticker_stage, a producer for ticks.

@josevalim Is it ok to call packages *_stage when publishing on hex.pm and the source on github?

It might take one more month until i publish the packages as I am busy working on a new production release for my company, which also includes polish for these libraries.

[josevalim]

@josevalim Is it ok to call packages *_stage when publishing on hex.pm and the source on github?

That sounds great actually. :)