kaiyuyue / mgd Goto Github PK

I notice that there are BNs for student features in MGDistiller. Why need additional batch normalizations? What if calculating distillation loss by student features directly ?

def amp(self, i, t_feats, s_feats, margins):
    """
    Absolute Max Pooling for channels reduction.
    """
    b, sc, h, w = s_feats[i].shape
    _, tc, _, _ = t_feats[i].shape

    groups = tc // sc

    t = []
    m = []
    for c in range(0, tc, sc):
        if c == (tc // sc) * sc and self.shave:
            continue

        t.append(t_feats[i][:, self.guided_inds[i][c:c+sc].detach(), :, :])
        m.append(margins[:, self.guided_inds[i][c:c+sc].detach(), :, :])

    t = torch.stack(t, dim=2)
    m = torch.stack(m, dim=2)

    t = t.reshape(b, sc, groups, -1)
    m = m.reshape(1, sc, groups, -1)

    t_inds = torch.argmax(t, dim=2)

    t = t.gather(2, t_inds.unsqueeze(2))
    m = m.mean(dim=2)

    t = t.reshape(b, sc, h, w)
    m = m.reshape(1, sc, 1, 1)

for Absolute Max Pooling ,t may be abs ?
the line t_inds = torch.argmax(t, dim=2) should change to t_inds = torch.argmax(torch.abs(t), dim=2)

I run your code and occur this error message, the self.model is an MGDistiller or SMDistiller object, and both of them don't have 'module' attribute, and I replace the self.model.module to self.model, the code works. It's right ?

Hi author,

Thanks a lot for sharing the nice work!
For detection part: RetinaNet-R50->RetinaNet-R18, run the command:
An error show up.
In mge/builder.py [line61]( self.t_channels = self.t_net.get_channel_num()):

line61: self.t_channels = self.t_net.get_channel_num()
AttributeError: 'RetinaNet' object has no attribute 'get_channel_num'

Hi there,

Since densenet, not like resnet, has many cross-block connections, I am wondering whether it, as the teacher, is compatible with your method?

hi, thank you for your job.
Could you please provide software version which you used for this job, such as torch, torchvision, detectron2, python, ...

Thanks!