DPT implementation contains unused parameters about transformers

[ducha-aiki]

@qubvel good point about the backbone. Probably because I have trained with a frozen backbone, which is kind of common.
And about the backbone removing unused params there would probably required too much changes.
I will do a PR then, thanks.

[qubvel]

Hi @ducha-aiki, thanks for reporting!

You are right, it looks like we can safely delete layers[0].residual_layer1 from DPTFeatureFusionStage because its never used.

Would you mind sharing why this prevents DDP training?

[ducha-aiki]

@qubvel I believe I shared this in:

Parameters which did not receive grad for rank 3: neck.fusion_stage.layers.0.residual_layer1.convolution2.bias, neck.fusion_stage.layers.0.residual_layer1.convolution2.weight, neck.fusion_stage.layers.0.residual_layer1.convolution1.bias, neck.fusion_stage.layers.0.residual_layer1.convolution1.weight

That is a quote from the error crash message I am getting, when running with accelerate for multi-GPU, when I specify in Trainer ddp_find_unused_parameters=False.

[qubvel]

Thank you, I missed it 🙂 I am trying to understand why backbone unused weights are not blocking, while neck's block. Did you try training with a fix?
Anyway if this solves the issue it is worth a PR.