Queue-Worker System about ray-llm

[sihanwang41]

Hi @AIApprentice101, We don't have the functionality in the ray-llm, you have to set it up by yourself.

For redis solution, do you see any issue or pain points? Or it is more about the integration effort.

[AIApprentice101]

@sihanwang41 Thank you for your reply. I saw there's a RFC related to integration of queuing system in Ray serve: ray-project/ray#32292. So I was wondering if that's something Ray-LLM would consider, especially given the inference of LLM usually takes pretty long to run.

In the meantime, we can set up the queuing system ourselves.