shaohuadong2021 / dplnet

DPLNet (Efficient Multimodal Semantic Segmentation via Dual-Prompt Learning)

Welcome to the official code repository for Efficient Multimodal Semantic Segmentation via Dual-Prompt Learning. We're excited to share our work with you, please bear with us as we prepare the code and demo. Stay tuned for the reveal!

Motivation

Previous multimodal methods often need to fully fine-tune the entire network, which are training-costly due to massive parameter updates in the feature extraction and fusion, and thus increases the deployment burden of multimodal semantic segmentation. In this paper, we propose a novel and simple yet effective dual-prompt learning paradigm, dubbed DPLNet, for training-efficient multimodal semantic segmentation.

Framework

Overview architecture of the proposed DPLNet, which adapts a frozen pre-trained model using two specially designed prompting learning modules, MPG for multimodal prompt generation and MFA for multimodal feature adaption, with only a few learnable parameters to achieve multimodal semantic segmentation in a training-efficient way.

Visualization

RGBD Semantic Segmentation Results

NYU-V2

SUN-RGBD

RGBT Semantic Segmentation Results

MFNet

PST900

RGB-D SOD Results

RGB-T SOD Results

RGB-T Video Semantic Segmentation Results

dplnet's Issues

Great work, has your paper been published yet? When will the code be open sourced?

Looking forward to your reply. Thanks

Hello! This is great work! When will the code be open sourced?

1

public code available

Hi,

I found your work to be very interesting and inspiring. I would love to see the code to better understand and appreciate the details of your project. Could you please share when you plan to release it?

Thank you!

RGB backbone pre-training

Thank you very much for your work, may I ask which data set is used for pre-training of the RGB backbone in your model?

please give me your code!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Effect of the number of learnable token and their dimension

Do you use the same dimension and number of learnable tokens in every stage? Did conduct any ablation study on how the variation of these hyperparameters affects overall performance?

Regarding questions about the parameters

Hello! Thank you for your excellent work.

I have some questions about the parameters reported in your paper. It seems that the parameter count does not include the frozen pre-trained backbone (MiT-B5). Could you further clarify the total number of parameters during network deployment (including both trainable and non-trainable parameters)? Additionally, could you provide more details on the network's FLOPs?

1111111111111111

Ask about the preprocessing of depth images

When training on NYUv2 dataset, did you use HHA encoding image as inputs or did you just use the colorization depth images? Furthermore, did you crop the white border of depth images?

When will you release the code?

source code

First of all, thank you for your work, I am very interested in your model, may I ask when you plan to release the source code?