yifanxu74 / evo-vit Goto Github PK

Hi, I want to ask that the behavior of training stage and inference stage. Since the paper illustrate the behavior of training stage, however, the inference stage did not mentioned. Is the inference stage still perform token selection, slow-fast updating and global class attention evolution? Thanks!

Sorry for so many questions. Another problem I encounter is the training result does not match the paper's result. For example, my training result of Evo-ViT-S is acc 66.22%, which is different from paper's acc 79.4%. I didn't modify the train settings (e.g., 300 epochs) except that I only use single GPU for training. Could you give me some suggetions, thanks!

Hi, thanks for your fantastic work. I have a question regarding the compatibility of Evo-ViT with ViTs that utilize window attention. In this context, the number of informative tokens may vary across different windows. Unfortunately, it appears that directly employing the "easy_gather" function is not feasible since it would compromise the spatial structure of the tokens. Could you please advise on how we can adapt Evo-ViT in such a scenario?

Hi, nice work, and could you provide Top5-Acc for each model?

In the original paper, there is a special token named representative token, which is aggregated by the placeholder tokens.
However, there is no corresponding implementation in your code.

In fact, you simply use argsort and select the topk informative tokens, which is non-differentiable.

# topk for slow update
x = x_[:, :N_ + 1] # L438
# simply copy for fast update
x = torch.cat((x, x_[:, N_ + 1:]), dim=1) # L473

I'm curious about the performance of using aggregating tokens and differentiable topk used in other paper.
Hopefully for your reply.

``Hi~ @YifanXu74
I find an interesting figure (Figure 4 in the paper) in your work. But I have some questions about this figure:
(1) In the Figure(a) I observe that ALL tokens CKA values are higher than informative and placeholders tokens. This phenomenon confuses me, I think the CKA of informative tokens should also be higher than all tokens?
(2) For Figure(b), I compute the cosine similarity and PCC by sampling some images from Imagenet based on the original DeiT-T codebase, but I find the cosine similarity of top layers is lower than the bottom layers. Did I do something wrong?
Here is my code in VisionTransformer Block:

    def forward(self, x):
        cos_list = []
        for i in range (1, x.size()[1]): # not consider CLS
            for j in range(i+1, x.size()[1]):
                cos_tmp = cos(x[0][j], x[0][i])
                cos_list.append(cos_tmp)
        cos_sum = sum(cos_list) / len(cos_list)
        x = x + self.drop_path(self.attn(self.norm1(x)))
        x = x + self.drop_path(self.mlp(self.norm2(x)))
        return x

Here is a example of my results:

tensor(0.4797, device='cuda:0')
tensor(0.5382, device='cuda:0')
tensor(0.4463, device='cuda:0')
tensor(0.3758, device='cuda:0')
tensor(0.3314, device='cuda:0')
tensor(0.3138, device='cuda:0')
tensor(0.2635, device='cuda:0')
tensor(0.2867, device='cuda:0')
tensor(0.3507, device='cuda:0')
tensor(0.3842, device='cuda:0')
tensor(0.4197, device='cuda:0')
tensor(0.4028, device='cuda:0')

Awesome work ! But I can not run the project correctly yet. Please provide me some information, thanks !

Excuse me, I want to ask that:

(1)Does this method provide FLOPs drop ratio, cause in the paper you use throughput as criteria

(2)How does the method work at extreme prune ratio (e.g. prune 80% FLOPs), since some token-pruning methods will have severe acc drop at larger prune ratio

Thanks!