v0.4.3 Release Tracker about vllm

[robertgshaw2-neuralmagic]

Is there any particular PR that we're waiting for before cutting the release?

The model support for Phi and Deepseek

[robertgshaw2-neuralmagic]

I am going to try to get these in

• #4525
• #4846
• #4464

probably will not make it but tracking to v0.4.4:

• #4582
• #4442

[njhill]

Sounds like we may want to include #4894 @rkooo567?

[robertgshaw2-neuralmagic]

• Fix for mistral-v0.3: #5005

[sasha0552]

With the patch, like you, running fp16 models (Mistral 7B for example) with no issues.

Not only fp16, but AQLM works well too (#5058)

[robertgshaw2-neuralmagic]

re: #4409 --> I did not have any issues running an fp16 model on a P40 when I installed from source.

Hi @robertgshaw2-neuralmagic - was this without the patch? I couldn't get a source build to run on P100's without the patch of #4409. With the patch, like you, running fp16 models (Mistral 7B for example) with no issues.

P40 requires building with the patch.

[sasha0552]

Hi, is it possible to include the following PRs?

• #4167
• #4409
• #4638

[simon-mo]

Thanks for bring these up @sasha0552!

#4167 is unlikely to be finished in time.
#4409 might need a little bit more discussion given what features are supported for Pascal GPUs and whether building from source might be a better option.
#4638 can be included if it gets merged in time.

We do commit to biweekly release cadence so don't worry many of these will get into soon enough!

[robertgshaw2-neuralmagic]

Thanks for bring these up @sasha0552!

#4167 is unlikely to be finished in time. #4409 might need a little bit more discussion given what features are supported for Pascal GPUs and whether building from source might be a better option. #4638 can be included if it gets merged in time.

We do commit to biweekly release cadence so don't worry many of these will get into soon enough!

re: #4409 --> I did not have any issues running an fp16 model on a P40 when I installed from source.

[rkooo567]

Yeah +1 on that PR @njhill

[jasonacox]

re: #4409 --> I did not have any issues running an fp16 model on a P40 when I installed from source.

Hi @robertgshaw2-neuralmagic - was this without the patch? I couldn't get a source build to run on P100's without the patch of #4409. With the patch, like you, running fp16 models (Mistral 7B for example) with no issues.

[vrdn-23]

Is there any particular PR that we're waiting for before cutting the release?