lapnd / gpt4all-experiments

Repo to train / inference different models with GPT4ALL dataset.

gpt4all-experiments

Repo to train / inference different models with GPT4ALL dataset.

This repo is very much a WIP

Currently the following are available:

• hacked together training code for GPT-J and potentially GPT-NeoX
• inference code for GPT-J
• Model LoRA weights available here: M-Chimiste/GPT-J-6B-GPT4ALL-SAMP-LORA-v1 (Current model is only trained on 50k samples of the GPT4ALL dataset as the full dataset would have taken over 5 days to train on my 2x 3090s. Hopefully we can do this when I next go on vacation).

TODO:

• Figure out how to merge lora weights with base model to get an all encompassing checkpoint.