VLLM Ray Workers are being killed by GCS about ray-llm

[shrekris-anyscale]

Could you provide a minimal repro for your setup? Do the vLLM workers die randomly, or is there a pattern?

[rtwang1997]

Hi,

We are finding that the vLLM workers die after running for around 1 hour (give or take), consistently.

We tried with a very simple ray service, where a deployment creates an actor in it's init function and I get the same behaviour where the actor dies after ~1h because all reference to it get removed. Here's the sample code:

import ray
from ray import serve
from fastapi import FastAPI

app = FastAPI()


@ray.remote
class Actor:
    def __call__(self):
        return "hello world"


@serve.deployment
@serve.ingress(app)
class APIIngress:
    def __init__(self):
        self.actor = Actor.remote()

    @app.get("/health")
    async def healthcheck(self):
        """
        checks if server is up.
        """
        return {"status": "server up"}

    @app.get("/test")
    async def test(self):
        return await self.actor.remote()


SERVE_APP = APIIngress.bind()

[shrekris-anyscale]

Thanks for the repro! What Ray version are you using?

[rtwang1997]

Hi,

We are using Ray version 2.7.1

[shrekris-anyscale]

Are there any updates here @rtwang1997? I synced with one of your coworkers on Slack and proposed some approaches, but I'm not sure how it went.

[shrekris-anyscale]

@rtwang1997 I'll close this issue for now. Feel free to reopen if you're still running into the problem.