zhengbw0324 / lc-rec

LC-Rec

This is the official PyTorch implementation for the paper:

Adapting Large Language Models by Integrating Collaborative Semantics for Recommendation

Overview

We propose LC-Rec, a new approach to integrate Language and Collaborative semantics for improving LLMs in Recommender systems. To tackle the large gap between the language semantics modeled by LLMs and collaborative semantics implied by recommender systems, we make two major contributions in two aspects. For item indexing, we design a learning-based vector quantization method with uniform semantic mapping, which can assign meaningful and non-conflicting IDs (called item indices) for items. For alignment tuning, we propose a series of specially designed tuning tasks to enhance the integration of collaborative semantics in LLMs. Our fine-tuning tasks enforce LLMs to deeply integrate language and collaborative semantics (characterized by the learned item indices), so as to achieve an effective adaptation to recommender systems.

Requirements

torch==1.13.1+cu117
accelerate
bitsandbytes
deepspeed
evaluate
peft
sentencepiece
tqdm
transformers

Model Checkpoint

The delta weights on the three datasets can be downloaded from huggingface hub (Instruments, Arts, Games). After downloading, you can add our deltas to the original LLaMA weights to obtain LC-Rec weights:

1. Get the original LLaMA weights.
2. Use the following scripts to get LC-Rec weights by applying our delta.

python -m convert/merge_delta.py \
    --base-model-path /path/to/llama-7b \
    --target-model-path /path/output/lc-rec \
    --delta-path bwzheng0324/lc-rec-games-delta

Dataset

We use three datasets in our paper, all of which have been uploaded to Google Drive

Train

The detailed scripts for all three datasets are in run.sh:

DATASET=Games
BASE_MODEL=huggyllama/llama-7b
DATA_PATH=./data
OUTPUT_DIR=./ckpt/$DATASET/

torchrun --nproc_per_node=8 --master_port=23324 finetune.py \
    --base_model $BASE_MODEL \
    --output_dir $OUTPUT_DIR \
    --dataset $DATASET \
    --data_path $DATA_PATH \
    --per_device_batch_size 8 \
    --gradient_accumulation_steps 2 \
    --learning_rate 5e-5 \
    --epochs 4 \
    --weight_decay 0.01 \
    --save_and_eval_strategy epoch \
    --deepspeed ./config/ds_z3_bf16.json \
    --bf16 \
    --only_train_response \
    --tasks seqrec,item2index,index2item,fusionseqrec,itemsearch,preferenceobtain \
    --train_prompt_sample_num 1,1,1,1,1,1 \
    --train_data_sample_num 0,0,0,100000,0,0 \
    --index_file .index.json


cd convert
nohup ./convert.sh $OUTPUT_DIR >convert.log 2>&1 &
cd ..

Test

Test with a single GPU:

DATASET=Games
DATA_PATH=./data
CKPT_PATH=./ckpt/$DATASET/
RESULTS_FILE=./results/$DATASET/result.json

python test.py \
    --gpu_id 0 \
    --ckpt_path $CKPT_PATH \
    --dataset $DATASET \
    --data_path $DATA_PATH \
    --results_file $RESULTS_FILE \
    --test_batch_size 1 \
    --num_beams 20 \
    --test_prompt_ids all \
    --index_file .index.json

Test with multiple GPUs:

DATASET=Games
DATA_PATH=./data
CKPT_PATH=./ckpt/$DATASET/
RESULTS_FILE=./results/$DATASET/result.json

torchrun --nproc_per_node=8 --master_port=23324 test_ddp.py \
    --ckpt_path $CKPT_PATH \
    --dataset $DATASET \
    --data_path $DATA_PATH \
    --results_file $RESULTS_FILE \
    --test_batch_size 1 \
    --num_beams 20 \
    --test_prompt_ids all \
    --index_file .index.json

Acknowledgement

The implementation is based on HuggingFace.

lc-rec's Issues

请问训练所需的显存大概是多少？

数据集的llama embedding好像没在谷歌云盘的zip包里

zip包里好像只有用户交互数据和文本相关数据，没有每个item的llama-embedding向量

关于amazon_text_emb.py的一些疑问

代码中似乎默认的使用的是单卡完成embedding，但是我在尝试过程中，A100-40G A40-48G都出现了OOM，请问作者是使用一张单卡40G完成这个任务的吗，还是说有一些细节我忽略了