acbull / ladies

Code for NeurIPS'19 "Layer-Dependent Importance Sampling for Training Deep and Large Graph Convolutional Networks"

Overview

LADIES (Layer-Dependent Importance Sampling for Training Deep and Large Graph Convolutional Networks) is a novel sampling algorithm for training GNN. It considers the previously sampled nodes for calculating layer-dependent sampling probability.

Based on the sampled nodes in the upper layer, LADIES selects their neighborhood nodes, compute the importance probability accordingly and samples a fixed number of nodes within them. We prove theoretically and experimentally, that our proposed sampling algorithm outperforms the previous sampling methods regarding time, memory and accuracy.

You can see our NeurIPS 2019 paper “Layer-Dependent Importance Sampling for Training Deep and Large Graph Convolutional Networks” for more details.

Setup

This implementation is based on Pytorch We assume that you're using Python 3 with pip installed. To run the code, you need the following dependencies:

• Pytorch 1.0
• gensim
• networkx
• tqdm
• numpy

We upload the three small benchmark node classification datasets, cora, pubmed and citeseer in /data for usage. You can upload any graph datasets as you want, and change the data loader function in /utils.py

Usage

Execute the following scripts to train and evaluate the model:

python3 pytorch_ladies.py --cuda 0 --dataset cora  # Train GCN with LADIES on cora.

There's also other hyperparameters to be tuned, which can be found in 'pytorch_ladies.py' for details.

The main function is ladies_sampler(), which sample a fixed number of nodes per layer. The sampling probability (importance) is computed adaptively according to the nodes sampled in the upper layer. We currently implement it using numpy sparse matrix multiplication. One can also implement it via pytorch multiplication or dictionary operation later.

Citation

Please consider citing the following paper when using our code for your application.

@inproceedings{ladies2019,
  title={Layer-Dependent Importance Sampling for Training Deep and Large Graph Convolutional Networks},
  author={Difan Zou and Ziniu Hu and Yewen Wang and Song Jiang and Yizhou Sun and Quanquan Gu},
  booktitle={Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems, NeurIPS},
  year={2019}
}

ladies's Issues

Entry level bugs in the main script, it cannot be run

Dear authors,

Thanks for sharing the code, after try to run it, I found multiple entry level bugs in the code. Now the main script is not even executable. Could you help to solve the issues?

For example:

1. The "row_normalize" function is not even implemented.
2. The "load_data" function version is not correct, it outputs "degrees" in the function, but never been used in "pytorch_ladies.py". The corresponding data holder name is "labels".
3. Function "F.cross_entropy" never takes a matrix containing one hot vectors as ground-truth, the correct ground-truth label should be saved in a int vector.

After I fixed all the issues in the code, the algorithm cannot converge in any of the databases. The training error goes to 0 after 2 epochs but the validation error keep increasing, the F-1 score always fluctuates around 0.7, and the first iteration has the lowest validation loss, which is very confusing.

Could you please fix the errors in the code? It may mislead the community. Thank you very much!

YH

Question about SuGCN

What's SuGCN for? I'm sorry that i can't got this code very well.
没太明白为什么要专门写了SuGCN，想请教一下。

Question about the sampling code

In line #L517, nodes of the lowest layer are treated as input nodes for GCN. This suggests the lowest layer contains all the nodes in the sampled sub-graph. However, it is not always true.
For example,

---

layer1: 4->2 5 -> 2
layer2: 2->1 3->1
layer3: 1->0

---

the lowest layer contains node (4, 5, 2), the middle layer contain nodes (2,3,1), the top layer contains nodes (1, 0). In your code, the features for nodes 1 and 3 are lost.

code for calculating the memory in Table 3.

Hi all, could you release the code for calculating the memory in Table 3?

run out GPU memory

When I validate this model on Reddit dataset, the model always run out of gpu memory, where the validation are conducted on a machine with Tesla V100-PCIE GPU (32GB memory). This is inconsistent with the results shown in table 3 in your paper. The detailed error is as follow: Traceback (most recent call last):
File "pytorch_ladies_.py", line 321, in
output = best_model.forward(feat_data[input_nodes], adjs)[output_nodes]
File "pytorch_ladies_.py", line 91, in forward
x = self.encoder(feat, adjs)
File "/root/anaconda3/envs/LADIES/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "pytorch_ladies_.py", line 81, in forward
x = self.dropout(self.gcs[idx](x, adjs[idx]))
File "/root/anaconda3/envs/LADIES/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "pytorch_ladies_.py", line 62, in forward
return F.elu(torch.spmm(adj, out))
RuntimeError: CUDA out of memory. Tried to allocate 1.71 GiB (GPU 0; 31.75 GiB total capacity; 28.63 GiB already allocated; 453.50 MiB free; 1.63 GiB cached)

Issues with reproducing the results in Table 3 of the paper

Hi,
I was trying to reproduce the same set of results with the same datasets with your code. Though the accuracy mean and variance for 10 experiments gives almost the same result, the total time and batch number do not match. I also wonder how these two results are quite different for batch size 64 and 512. Could you please upload the final version of your code where you calculate these results or give an insight on how can I reproduce these results?

Thanks!

paper

Could you please upload your paper first? Thanks.

fastgcn_sampler

Hi, bro,
in your code,

fastgcn_sampler:
adj = row_norm(U[: , after_nodes].multiply(1/p[after_nodes]))

However, other people say: we should row - select and col - select the lap_matrix (U).
More details can see: https://github.com/khanhhhh/ladies/blob/khanh/solution_random_block/sampler.py

I can not understand, can you explain more?

Questions about the Reddit dataset

Hi, I am trying to reproduce the result of LADIES on Reddit dataset. I am curious about where is the download link of Reddit dataset and the processing function of reddit dataset. Thank you for your help in advance.

The citation is not this paper

Hi, it seems the citation bibtex in your README.md is not LADIES.

Some problems for the code

Hi, bro,
Thanks for making the code public. But there are so many bugs in the code which can't be run directly. By the way, could you please report the detailed parameters you used for results in Table 3 on the four datasets ? I tuned a lot but can't reproduce the performance. Thanks very much!