WebCPU? about web-llm

[CharlieFRuan]

Hi! As of now, the WebLLM project only supports devices/browsers with WebGPU support (support can be checked via https://webgpureport.org/).

While we have not tried on Samsung Note20 (only tested on Samsung S23 and Pixel 7 Pro), I would suggest try upgrading Android version and updating Chrome to the latest version, as these may affect WebGPU support. Can be verified via https://webgpureport.org/ after upgrades.

If with most up-to-date versions, the device still does not support WebGPU, you can also check out MLC-LLM (WebLLM is a companion project of MLC-LLM), which has Android support: https://llm.mlc.ai/docs/deploy/android.html

[fire17]

Good advice!
Before:

After updating chrome:

I did manage
To squeeze a few tokens per second but
These tiny models are still far from production ready.

Thanks a lot for the advice, wonder if we will see capable 1-bit llms soon, or other tiny but coherent models.
I'll stay tuned... All the best! 🙏🧘‍♀️

[tqchen]

glad it works