[Bug]: Scheduler fail with assertion on "meta-llama/Meta-Llama-3-70B-Instruct" when calling concurrently about vllm

[tsvisab]

Update: the same exact error happens when intializing the meta-llama/Meta-Llama-3-8B-Instruct with only a single GPU

• model is served with FastApi + uvicorn
• sequential calls work fine, call in parallel yields an assertion:

/lib/python3.10/site-packages/vllm/core/scheduler.py", line 756, in _schedule_default
assert len(running_scheduled.prefill_seq_groups) == 0

[tsvisab]

Ok, so figures i wasn't using vllm properly, loading model with LLM isn't supporting async requests, maybe the lib should be more clear about that, should load with AsyncLLMEngine