antonmi / flowex Goto Github PK

Hey,
This looks like a great library! Quick question (and apologies if I missed it somewhere in the documentation)--what's the license on its use?

Hello,

I would like to use that lib into my project but by seeing that there's not any updates since 2 years at least, I'm asking my-self if it is still a viable project for production-usage. Does this project just dead ?

I think I should go for flow which is quite different but we can achieve quite the same thing.

Thanks,

Hi! I am using Flowex to organise my code, and noticed that my functions should return the %__MODULE__{} struct when the function succeeds.
However, how can I trigger the error pipe when a case do in my function fails?
Or maybe I didn't understand how flowex handles that.

Have a nice day!

Using the "Module Pipes" example:

engine_1     |     ** (EXIT) an exception was raised:
engine_1     |         ** (KeyError) key :a not found
engine_1     |             (spider_man) lib/pipeline/pipeline.ex:16: AddOne.init/1

Occurs here:

defmodule AddOne do
  def init(opts) do
    %{opts | a: :add_one}
  end

Hello @antonmi,

First of all, thank you very much for creating flowex, looks amazingly useful.

I overall like your API design, just one tiny thing I think could be improved. When defining the number of worker processes a particular component can have

  pipe :mult_by_two, 3

how about using a more idiomatic keyed option instead of just giving the number

  pipe :mult_by_two, workers: 3

The options advantage is we are being more explicit about what 3 means (at the cost of writing a bit more, but I believe many Elixir libraries prefer to be explicit about intent)

I could provide a PR if you are interested in this change.

Cheers, and thanks again for Flowex :)

I have a pipeline as following.

defmodule Helios.Pipeline.Delete do
use Flowex.Pipeline

pipe Helios.Flow.DeleteAsset, count: 1
pipe Helios.Flow.Finalize, count: 1
end

And I connect the pipeline to superviser (strategy: :one_for_one).

pipeline_delete = Helios.Pipeline.Delete.supervised_start(supervisor_pid)

After the application is started and the pipeline is up andruning, I killed the pipeline process, but I can see only Producer process is back, but not the others.

Sorry my english is not very good , but i am working on it.

When needing to more than once (or repeate ) to call some pipelines, it seem to pre compile to define counts of consumer is not good enough.

Suppose many agents connected to a chatroom or game AOI(Area of Interest) , and push/receive messages very frequently. We maybe use some frp pattern (Functional reactive programming) to change related states in one tick of the timewheel.
for eg. use this module, https://github.com/Qqwy/elixir_gen_frp/blob/master/lib/gen_frp/callback.ex#L34
to regist a interval callback ( a Flowex.Pipeline start function for eg.) at a certain timeout

It's time sensitive , so we need to limit the max execution time of one sequence computational task.

When get the option's interval of outside, we can to retry to spawn more processor in
Flowex.Pipeline， so It need some ways to dynamically change the number of specified pipe.

 def start(pipeline_module, opts) do
   if opts.muti and opts.sup_pid  do     
      do_change(sup_pid, pipeline_module, opts)
   else
       {:ok, sup_pid} = Flowex.Supervisor.start_link(pipeline_module)
       do_start(sup_pid, pipeline_module, opts)
   end
 end

pipe :add_one,  name: fn ->String.to_atom("Flowex_ModuleName_PrevToNextLayerOrIndexNumber_add_one") end
pipe :mult_by_two, name: fn ->String.to_atom("Flowex_ModuleName_LayerOrIndexNumber_mult_by_two") end, min_time:30, step_by: 2, max_number:10

Something like this interface, in every tick we calculate the time difference (in microseconds) of begin and end of a Pipeline, if less than limit, it is ok, if not, in next tick we add more number of specified Pipe by step. (we also need limit max number of a pipe to avoid too much process)

I am using Flowex 0.5.2.
I have a application uses Flowex.Client.call() but it doesn't execute the pipeline concurrently.

Flowex.Client.call() is called from separated processes which is created by Task.async.
I want to use Flowex.Client.call() function to receive final value.

Enum.each(requests, &( Task.async(__MODULE__, :do_task, [self(), client_pid, &1]) ))

And do_task() function calls Flowex.Client.call().

def do_task(owner, client_pid, request) do
result = Flowex.Client.call(pipeline_pid, request) do

send( owner, {:completed, result})
end

When I do this way, I found pipeline process only one data at a time.
Once I change the call to cast() it processes concurrently. With case() I cannot get the result of the pipeline without extra message sending.

Thank you.

For proper working on release built by distillery need to add gen_stage into application deps.
Without it application is failed with :failed_to_start_childern because it can't find GenStage module by Flowex.

{:error, {{:shutdown, {:failed_to_start_child, :"Flowex.Producer_Notifications.ProceedPipeline_#Reference<0.3448033567.1304952833.40974>", {:EXIT, {:undef, [{GenStage, :start_link, [Flowex.Producer, nil, [name: :"Flowex.Producer_Notifications.ProceedPipeline_#Reference<0.3448033567.1304952833.40974>"]], []},....