Is this running in fp16 or fp32, or is it something different? about web-llm

[jinhongyii]

Thanks for your advice. We are testing fp16 correctness and speed internally and will make it public soon

[xzuyn]

Thanks for your advice. We are testing fp16 correctness and speed internally and will make it public soon

I'm wondering about what the web demo uses currently. The model size is similar to a q4_0 ggml model, so is it running 4bit? I couldn't find any specific info on what precision you guys are using.

[jinhongyii]

it is using 4bit quantization and fp32 for compute

[xzuyn]

it is using 4bit quantization and fp32 for compute

Thank you. Good luck with everything, I'm looking forward to seeing how this progresses.