edisonleeeee / greatx

A graph reliability toolbox based on PyTorch and PyTorch Geometric (PyG).

Hi there, I'm EdisonLeeeee! 👋

- 😎 About me

• Sun Yat-sen University, China
• Major in Software Engineering
• Learning Python, Machine Learning
• TensorFlow and PyTorch Enthusiast

- Research on

• Graph Representation Learning
• Trustworthy Graph Learning
• Graph Self-supervised Learning

- Project

• GraphGallery: Graph Gallery for benchmarking graph neural networks
• GreatX (ongoing): Graph reliability toolbox based on PyTorch Geometric
• Mooon (ongoing): Graph data augmentation library based on PyTorch Geometric

greatx's Issues

Benchmark Results of Attack Performance

Hi, thanks for sharing the awesome repo with us! I recently run the attack sample code but the resultpgd_attack.py and random_attack.py under examples/attack/untargeted, but the accuracies of both evasion and poison attack seem not to decrease.

I'm pretty confused by the attack results. For CV models, pgd attack easily decreases the accuracy to nearly random guesses, but the results of GreatX seem not to consent with CV models. Is it because the number of the perturbed edges is too small?

Here are the results of pgd_attack.py

Processing...
Done!
Training...
100/100 [==============================] - Total: 874.37ms - 8ms/step- loss: 0.0524 - acc: 0.996 - val_loss: 0.625 - val_acc: 0.815
Evaluating...
1/1 [==============================] - Total: 1.82ms - 1ms/step- loss: 0.597 - acc: 0.843
Before attack
 Objects in BunchDict:
╒═════════╤═══════════╕
│ Names   │   Objects │
╞═════════╪═══════════╡
│ loss    │  0.59718  │
├─────────┼───────────┤
│ acc     │  0.842555 │
╘═════════╧═══════════╛
PGD training...: 100%|███████████████████████████████████████████████████████████████████████| 200/200 [00:02<00:00, 69.74it/s]
Bernoulli sampling...: 100%|██████████████████████████████████████████████████████████████████| 20/20 [00:00<00:00, 804.86it/s]
Evaluating...
1/1 [==============================] - Total: 2.11ms - 2ms/step- loss: 0.603 - acc: 0.842
After evasion attack
 Objects in BunchDict:
╒═════════╤═══════════╕
│ Names   │   Objects │
╞═════════╪═══════════╡
│ loss    │  0.603293 │
├─────────┼───────────┤
│ acc     │  0.842052 │
╘═════════╧═══════════╛
Training...
100/100 [==============================] - Total: 535.83ms - 5ms/step- loss: 0.124 - acc: 0.976 - val_loss: 0.728 - val_acc: 0.779
Evaluating...
1/1 [==============================] - Total: 1.74ms - 1ms/step- loss: 0.766 - acc: 0.827
After poisoning attack
 Objects in BunchDict:
╒═════════╤═══════════╕
│ Names   │   Objects │
╞═════════╪═══════════╡
│ loss    │  0.76604  │
├─────────┼───────────┤
│ acc     │  0.826962 │
╘═════════╧═══════════╛

Here are the results of random_attack.py

Training...
100/100 [==============================] - Total: 600.92ms - 6ms/step- loss: 0.0615 - acc: 0.984 - val_loss: 0.626 - val_acc: 0.811
Evaluating...
1/1 [==============================] - Total: 1.93ms - 1ms/step- loss: 0.564 - acc: 0.832
Before attack
 Objects in BunchDict:
╒═════════╤═══════════╕
│ Names   │   Objects │
╞═════════╪═══════════╡
│ loss    │  0.564449 │
├─────────┼───────────┤
│ acc     │  0.832495 │
╘═════════╧═══════════╛
Peturbing graph...: 253it [00:00, 4588.44it/s]
Evaluating...
1/1 [==============================] - Total: 2.14ms - 2ms/step- loss: 0.585 - acc: 0.826
After evasion attack
 Objects in BunchDict:
╒═════════╤═══════════╕
│ Names   │   Objects │
╞═════════╪═══════════╡
│ loss    │  0.584646 │
├─────────┼───────────┤
│ acc     │  0.826459 │
╘═════════╧═══════════╛
Training...
100/100 [==============================] - Total: 530.04ms - 5ms/step- loss: 0.0767 - acc: 0.98 - val_loss: 0.574 - val_acc: 0.791
Evaluating...
1/1 [==============================] - Total: 1.77ms - 1ms/step- loss: 0.695 - acc: 0.813
After poisoning attack
 Objects in BunchDict:
╒═════════╤═══════════╕
│ Names   │   Objects │
╞═════════╪═══════════╡
│ loss    │  0.695349 │
├─────────┼───────────┤
│ acc     │  0.81338  │
╘═════════╧═══════════╛

Questions about PGDAttack

Hi! Thanks for this great repo.
I have some questions about the implementation of PGDAttack.

1. The learning rate of PGDAttack.

   GreatX/greatx/attack/untargeted/pgd_attack.py

   Line 42 in 49675e3

   lr = base_lr * num_budgets / math.sqrt(epoch + 1)

   In the original paper of PGDAttack and implementation of DeepRobust, the learning rate here seems to be
   lr = base_lr / math.sqrt(epoch + 1). I notice that the default value of "base_lr" is kept the same, so the final "lr" would be very different as num_budget is often large. Will this difference matter a lot?
2. The choice of learning rate in PGDAttack.
   As suggested by the authors, PGDAttack prefers different base_lr for different loss_type. I think it would be better if this difference is included.
3. In PGD Example, the same attacker is applied in both poison and evasion settings. In the original implementation, there is a poison version of PGDAttack specific for the poison setting (named as MinMax in the DeepRobust repo). Will this version be included as well?

By the way, it is great to see that the repo is more PyG styled. Does that mean we can attack more PyG models as surrogate or victim models? For example, we can attack GAT and APPNP using PGDAttack as long as they are written in a PyG message-passing framework through edge_index.

problem with metattack

Thanks for this wonderful repo. However, when I run the metattack example ,the result is not promising
Here is my result when attack Cora with metattack
Training...
100/100 [====================] - Total: 520.68ms - 5ms/step- loss: 0.0713 - acc: 0.996 - val_loss: 0.574 - val_acc: 0.847
Evaluating...
1/1 [====================] - Total: 2.01ms - 2ms/step- loss: 0.522 - acc: 0.847
Before attack
╒═════════╤═══════════╕
│ Names │ Objects │
╞═════════╪═══════════╡
│ loss │ 0.521524 │
├─────────┼───────────┤
│ acc │ 0.846579 │
╘═════════╧═══════════╛
Peturbing graph...: 100%|██████████| 253/253 [01:00<00:00, 4.17it/s]Evaluating...
1/1 [====================] - Total: 2.08ms - 2ms/step- loss: 0.528 - acc: 0.844
After evasion attack
╒═════════╤═══════════╕
│ Names │ Objects │
╞═════════╪═══════════╡
│ loss │ 0.528431 │
├─────────┼───────────┤
│ acc │ 0.844064 │
╘═════════╧═══════════╛
Training...
32/100 [=====>..............] - ETA: 0s- loss: 0.212 - acc: 0.956 - val_loss: 0.634 - val_acc: 0.807
100/100 [====================] - Total: 407.58ms - 4ms/step- loss: 0.0601 - acc: 0.996 - val_loss: 0.704 - val_acc: 0.787
Evaluating...
1/1 [====================] - Total: 1.66ms - 1ms/step- loss: 0.711 - acc: 0.819
After poisoning attack
╒═════════╤═══════════╕
│ Names │ Objects │
╞═════════╪═══════════╡
│ loss │ 0.710625 │
├─────────┼───────────┤
│ acc │ 0.818913 │
╘═════════╧═══════════╛

SG Attack example cannot run as expected on cuda

Hello,
I got some error when I run SG Attack's example code on cuda device:

Traceback (most recent call last):
  File "src/test.py", line 50, in <module>
    attacker.attack(target)
  File "/greatx/attack/targeted/sg_attack.py", line 212, in attack
    subgraph = self.get_subgraph(target, target_label, best_wrong_label)
  File "/greatx/attack/targeted/sg_attack.py", line 124, in get_subgraph
    self.label == best_wrong_label)[0].cpu().numpy()
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:3 and cpu!

I found self.label is on cuda device, but best_wrong_label is on cpu.

GreatX/greatx/attack/targeted/sg_attack.py

Lines 123 to 124 in 73eac35

	attacker_nodes = torch.where(
	self.label == best_wrong_label)[0].cpu().numpy()

I remove line94 .cpu(), everything is going well and no error report

GreatX/greatx/attack/targeted/sg_attack.py

Lines 94 to 96 in 73eac35

	self.logits = self.surrogate(self.feat, self.edge_index,
	self.edge_weight).cpu()

I found there is a commit that adds .cpu end of line 94, so I dont know it's a bug or something else🤨

TypeError

Hi.
I have tried to run the metattack.py file. But got this error.

TypeError: fit() got an unexpected keyword argument 'mask'