nenkoru / serverless-rwkv Goto Github PK

For now the Dockerfile and in general the code heavily relies on torch. Which makes a base image quite huge(6GB +-).
As well as making more lightweight image a speed of tokenizer, sampler and inference could be increased as well.
I strongly encourage anyone who reads this to take a look into this project, contribute or even donate a little bit of funds.
I see a big potential in RWKV in general, the project below in particular and deployment to serverless GPUs exactly.
Imagine 14B inference for a prompt under a 0.001$. So like 100prompts a day = 0.1$, 0.1*30=3$ a month for your own model, that is not collecting any data and is quite large.

https://github.com/harrisonvanderbyl/rwkv-cpp-cuda

https://github.com/BlinkDL/RWKV-LM