hxttkl / graphadapter

Code of GraphAdapter

Can GNN be Good Adapter for LLMs?

This repository is an implementation of GraphAdapter - Can GNN be Good Adapter for LLMs? in WWW 2024.

Requirements

• python = 3.8
• numpy >= 1.19.5
• pytorch = 1 .10.2
• pyg = 2.3.1
• transformers >= 4.28.1

For the largest dataset Arxiv, 300G storage is required

How to use our code

The datasets this paper used can be downloaded from here, please download them and put them in datasets to unzip.

Step 1. Preprocess data for training

python3 preprocess.py --dataset_name instagram --gpu 0 --plm_path llama2_path --type pretrain

The preprocess.py will load the textual data of Instagram, and next transform them to token embedding by Llama 2, which will be saved into saving_path. The saved embeddings will used in the training of GraphAdapter.

Step 2. Training GraphAdapter

python3 pretrain.py --dataset_name instagram --hiddensize_gnn 64 --hiddensize_fusion 64 --learning_ratio 5e-4 --batch_size 32 --max_epoch 15 --save_path your_model_save_path

Step 3. Finetuning for downstream task

GraphAdapter requires prompt embedding for finetuning,

python3 preprocess.py --dataset_name instagram --gpu 0 --plm_path llama2_path --type prompt

After preprocessing the dataset, now you can finetune to downstream tasks.

python3 finetune.py --dataset_name instagram  --gpu 0  --metric roc --save_path your_model_save_path 

Note: keep your_model_save_path consistent in both pretrain.py and finetune.py.

Citation

If you find our work or dataset useful, please consider citing our work:

@article{huang2024can,
  title={Can GNN be Good Adapter for LLMs?},
  author={Huang, Xuanwen and Han, Kaiqiao and Yang, Yang and Bao, Dezheng and Tao, Quanjin and Chai, Ziwei and Zhu, Qi},
  journal={WWW},
  year={2024}
}

graphadapter's Issues

Code & Data

Great work!

I think GraphAdapter will have a significant impact on the development of Graph+LLM.

Can I kindly ask if you will release the code together with the data soon?

Best regards,
Yuhan

Inquiry about raw TAG data of Arxiv

Hi, thanks for your hard work and it inspires me a lot! I am wondering could you please share the raw TAG data of Arxiv? Since I just found instagram and reddit raw TAG data in the link you shared for data downloading.

There is an error in self.is_gnn = is_gnn on line 114 of graphadapter.py. The reason may be that the is_gnn variable is not declared in the __init__ method of the GATBlock class.