How to use for inference? about gptq-for-llama

[qwopqwop200]

Added simple inference code.
CUDA_VISIBLE_DEVICES=0 python llama_inference.py decapoda-research/llama-7b-hf --wbits 4 --load llama7b-4bit.pt --text "this is llama"

[qwopqwop200]

Implemented batch processing of cuda kernel.

[ItsLogic]

It actually does work with standard transformers inference code but the kernel is limiting the input to 1 token. If you run inference with a word/letter that only counts as 1 token it will run to completion as shown below with the word "ball" which is a single token.

The kernel needs to be updated before inference can work with multiple tokens so we need to wait on #6 to be solved before its usable

[MarkSchmidty]

I don't have time to test, but if you clone PR #9 you may be able to use more than one input token. At least that's what the PR says.

[qwopqwop200]

this method is not implemented in cuda.
I currently implemented batch processing with cuda kernel and testing it.