Use case: dask + redis + progress reporting about streamz

[kszucs]

I really don't know, but there are practices:

• http://streamparse.readthedocs.io/en/stable/topologies.html#dealing-with-errors
• http://python-rq.org/docs/exceptions/

[mrocklin]

What do you do currently?

…

On Sun, Apr 16, 2017 at 12:42 PM, Krisztián Szűcs ***@***.***> wrote: I have a couple of DAG workflows in dask. These workflows are triggered via user interfactions through a web API. I'd like to achieve the followings: 1. The web API puts the workflow to a redis queue (RabbitMQ would be nice too, but redis is easier to start with). 2. Run workers like the majority of job queue frameworks do (rq <https://github.com/nvie/rq>, arq <https://github.com/samuelcolvin/arq>, huey <https://github.com/coleifer/huey>) 3. Store the progress and current status of the workflows (with MultiProgress <https://github.com/dask/distributed/blob/master/distributed/diagnostics/progress.py#L137> I guess) as key/values in redis What would be the correct approach to implement it? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#6>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AASszEyb1SUtIvudEewp5dR8D6xWcDUXks5rwkTngaJpZM4M-q40> .

[kszucs]

Currently nothing :) I wait for your advise.

[mrocklin]

Have a client on the same machine as the scheduler? Have the client process read from the redis queue and take whatever action you deem appropriate?

Honestly I'm not sure how much help I can be here without diving into your particular problem more deeply (which is probably not likely to occur).

If you have more particular questions about how things work or thoughts on what features you would need then please let me know .

[kszucs]

The part I'm least aware of is the progress monitoring:

• How can I subscribe to futures prefixed by a key?
• Should I instantiate a scheduler plugin (like progressbar) per task submission, or per client?
• Should I prefer long living Clients?
• How can I nicely handle (non-critical) task failures? (the frameworks mentioned above put the failed tasks to a separate queue) - a minimal error handling would be nice in streams.

Actually I would like to use redis as a resilient/persistent layer just before distributed if the scheduler dies, network error etc.

[mrocklin]

• If you're looking for a cross-client subscription mechanism then maybe use channels? There are more opportunities for cross-client communication; channels aren't perfect but they can often get the job done. Of course if you can solve your problem without multiple clients then this is probably better.
• I tend to create a single scheduler plugin. I haven't yet found an application where I needed to create one per client. You may have something else in mind though.
• Starting a client does take some milliseconds. I guess that this would depend on your performance needs.
• Yeah, I agree that streams should handle error handling and possibly stop signals. This is on a TODO list somewhere :)

[mrocklin]

How do you think streams should handle exceptions?