nineabyss / glbench

GLBench: A Comprehensive Benchmark for Graphs with Large Language Models

GLBench: A Comprehensive Benchmark for Graphs with Large Language Models

If you like our project, please give us a star ⭐ on GitHub for the latest update.

This is the official implementation of the following paper:

GLBench: A Comprehensive Benchmark for Graphs with Large Language Models [Paper]

Yuhan Li, Peisong Wang, Xiao Zhu, Aochuan Chen, Haiyun Jiang, Deng Cai, Victor Wai Kin Chan, Jia Li

Trend of Graph&LLM.

Environment Setup

Before you begin, ensure that you have Anaconda or Miniconda installed on your system. This guide assumes that you have a CUDA-enabled GPU. After create your conda environment (we recommend python==3.10), please run

pip install -r requirements.txt

to install python packages.

Datasets

All datasets in GLBench are available in this link. Please place them in the datasets folder.

Benchmarking

Supervised

Classical(GNN)

Benchmark the Classical GNNs (grid-search hyperparameters)

cd models/gnn
bash models/gnn/run.sh

LLM

Benchmark the LLMs(Sent-BERT, BERT, RoBERTa)

cd models/llm
bash 

Enhancer

• GIANT

Due to some package conflicts or version limitations, we recommend using docker to run GIANT. The docker file is in

models/enhancer/giant-xrt/dockerfile

After starting the Docker container, run

cd models/enhancer/giant-xrt/
bash run_all.sh

• TAPE

cd models/enhancer/TAPE/
bash run.sh

• OFA

cd models/enhancer/OneForAll/
bash run.sh

• ENGINE

Predictor

• InstructGLM

• GraphText

Due to some package conflicts or version limitations, we recommend using docker to run GraphText. The docker file is in

models/predictor/GraphText/dockerfile

After starting the Docker container, run

cd models/predictor/GraphText
bash run.sh

• GraphAdapter

cd models/predictor/GraphAdapter
bash run.sh

• LLaGA

Aignment

• GLEM

cd models/alignment/GLEM
bash run.sh

• Patton

bash run_pretrain.sh
bash nc_class_train.sh
bash nc_class_test.sh

We also provide seperate scripts for different datasets.

• Zero-shot

LLM

Benchmark the LLMs(LLaMA3, GPT-3.5-turbo, GPT-4o, DeepSeek-chat)

cd models/llm

You can use your own API key for OpenAI.

Enhancer

• OFA

cd models/enhancer/OneForAll/
bash run_zeroshot.sh

• ZeroG

cd models/enhancer/ZeroG/
bash run.sh

Predictor

• GraphGPT

cd models/predictor/GraphGPT
bash ./scripts/eval_script/graphgpt_eval.sh

FYI: our other works

🔥 A Survey of Graph Meets Large Language Model: Progress and Future Directions (IJCAI'24)

Github Repo | Paper

🔥 ZeroG: Investigating Cross-dataset Zero-shot Transferability in Graphs (KDD'24)

Github Repo | Paper

Acknowledgement

We are appreciated to all authors of works we cite for their solid work and clear code organization! The orginal version of the GraphLLM methods are listed as follows:

Alignment:

GLEM:

• (2022.10) [ICLR' 2023] Learning on Large-scale Text-attributed Graphs via Variational Inference [Paper | Code]

Patton:

• (2023.05) [ACL' 2023] PATTON : Language Model Pretraining on Text-Rich Networks [Paper | Code]

Enhancer:

ENGINE:

• (2024.01) [IJCAI' 2024] Efficient Tuning and Inference for Large Language Models on Textual Graphs [Paper]

GIANT:

• (2022.03) [ICLR' 2022] Node Feature Extraction by Self-Supervised Multi-scale Neighborhood Prediction [Paper | Code]

OFA:

• (2023.09) [ICLR' 2024] One for All: Towards Training One Graph Model for All Classification Tasks [Paper | Code]

TAPE:

• (2023.05) [ICLR' 2024] Harnessing Explanations: LLM-to-LM Interpreter for Enhanced Text-Attributed Graph Representation Learning [Paper | Code]

ZeroG:

• (2024.02) [KDD' 2024] ZeroG: Investigating Cross-dataset Zero-shot Transferability in Graphs [Paper] | Code]

Predictor:

GraphAdapter:

• (2024.02) [WWW' 2024] Can GNN be Good Adapter for LLMs? [Paper]

GraphGPT:

• (2023.10) [SIGIR' 2024] GraphGPT: Graph Instruction Tuning for Large Language Models [Paper | Code]

GraphText:

• (2023.10) [Arxiv' 2023] GraphText: Graph Reasoning in Text Space [Paper] | Code]

InstructGLM:

• (2023.08) [Arxiv' 2023] Natural Language is All a Graph Needs [Paper | Code]

LLaGA:

• (2024.02) [Arxiv' 2024] LLaGA: Large Language and Graph Assistant [Paper | Code]

Code Base Structure

$CODE_DIR
    ├── datasets
    └── models
        ├── alignment
        │   ├── GLEM
        │   └── Patton
        ├── enhancer
        │   ├── ENGINE
        │   ├── giant-xrt
        │   ├── OneForAll
        │   ├── TAPE
        │   └── ZeroG
        ├── gnn
        ├── llm
        │   ├── deepseek-chat
        │   ├── gpt-3.5-turbo
        │   ├── gpt-4o
        │   └── llama3-70b
        └── predictor
           ├── GraphAdapter
           ├── GraphGPT
           ├── GraphText
           ├── InstructGLM
           └── LLaGA

glbench's Issues

Is this just a collection of codes snippets from other repos?

I think this is not a benchmark or a library; it looks more like a collection of code snippets from other repos. It appears that there was no attempt made to adapt the code from the authors to your own framework. Instead, I would say it is a straightforward copy-and-paste without additional efforts.

It's comprehensive, but not "usable". Overclaimed, your paper.

Why didn’t you include links to other relevant code or projects?

Thank you for sharing your work with the community! I noticed that the project doesn’t include any references or links to other relevant code or projects.
Could you add links to related projects or codebases that might provide additional value to the users?
Thank you for your time!

Issues about commands

Thanks for your excellent work!
However, I find that some of the code cannot be executed correctly using your commands. For example, when I run OFA using the following commands, it raises several issues.

cd models/enhancer/OneForAll/
bash run.sh