ds4dm / sparse-gcn

Sparse graph attention

Sparse graph convolution network

Attention model is based on:

Vaswani, Ashish, et al. "Attention is all you need." Advances in Neural Information Processing Systems. 2017.

Graph Attention Network:

Veličković, Petar, et al. "Graph Attention Networks." arXiv preprint arXiv:1710.10903 (2017).

sparse-gcn's Issues

BUG: sparse gradient and coalesce()

Seems specific to CUDA implementation.

import torch
from torch.nn import Parameter
from torch.autograd import grad

import sgat.sparse as sp

cuda = True

_torch = torch.cuda if cuda else torch

# node features
x = _torch.FloatTensor([[1], [2]])

# attention
i = _torch.LongTensor([[0, 0], [0, 1]])
s = (2, 2)
w = Parameter(_torch.FloatTensor([1, 1]))
a = sp.build(i, w.exp(), s)  # softmax

# convolution
x = sp.matmul(a, x)

# normalization
a_norm = sp.matmul(a, _torch.FloatTensor(a.size(0), 1).fill_(1))
a_norm = a_norm.clamp(min=1e-12)  # avoid divide by zero
x /= a_norm.expand_as(x)

print("Forward ok")

print(grad(x.sum(), w, retain_graph=True))

print("Backward ok")

RuntimeError: The size of tensor a (4) must match the size of tensor b (2) at non-singleton dimension 0