ChatRWKV triggers segmentation fault when using streaming or split loading on 4-14B about rwkv-lm

[BlinkDL]

Currently it requires huge amt (probably 40G+?) of CPU RAM to load 14B. See BlinkDL/ChatRWKV#41

Will be fixed soon :)

[nanonomad]

Thank you. Sorry for troubling you with the post - searched but didn't come up with the linked reply. I think you're going to be getting this one a lot, because this is not onaly a great project, but its accessible on retail hardware.

Looking forward to any optimizations that could make this work on 32gb. I think my 2-slot board is maxed out, and it took a heatgun and a rubber mallet to get a 3080 into this HP Omen prebuilt, so the thing is never coming apart again lol

And thank you for all the work you've put in to this project.

[BlinkDL]

This will be fixed in BlinkDL/ChatRWKV#40 very soon :)

[BlinkDL]

Update ChatRWKV v2 & pip rwkv package (0.7.0):
Use v2/convert_model.py to convert a model for a strategy, for faster loading & saves CPU RAM.
@nanonomad