How does the swin transformer deal with the input image with the size of 448 x 576 ? about multi-task-transformer

[prismformore]

I don't think window attention in Swin block requires square input. We can still use the pre-trained weights.
https://github.com/microsoft/Swin-Transformer/blob/968e6b5e428186dc99d19166996439c23dccc4d1/models/swin_transformer.py#L77

[Rockyman-star]

Okay! Thanks for your reminder. I guess you modified all the window_size to a tuple/list form in the original swin_transformer.py to an uneven height-width partition. But how did you use officially released pretrained parameters relative_position_bias_table as the shape changes? By bilinear interpolation? And what size is the window_size set to? (14, 18) to main the same receptive field? or (7, 9)?

[prismformore]

I didn't change the window size so relative_position_bias_table here is not changed. Window size is independent from the input size.

[prismformore]

You may check the swin transformer code here https://github.com/prismformore/Multi-Task-Transformer/blob/main/TaskPrompter/models/transformers/taskprompter_swin.py Get Outlook for Android<https://aka.ms/AAb9ysg>

…

________________________________ From: samuel-zhaoy ***@***.***> Sent: Wednesday, April 10, 2024 10:34:49 PM To: prismformore/Multi-Task-Transformer ***@***.***> Cc: Hanrong Ye ***@***.***>; Comment ***@***.***> Subject: Re: [prismformore/Multi-Task-Transformer] How does the swin transformer deal with the input image with the size of 448 x 576 ? (Issue #29) Okay! Thanks for your reminder. I guess you modified all the window_size to a tuple/list form in the original swin_transformer.py to an uneven height-width partition. But how did you use officially released pretrained parameters relative_position_bias_table<https://github.com/microsoft/Swin-Transformer/blob/968e6b5e428186dc99d19166996439c23dccc4d1/models/swin_transformer.py#L101> as the shape changes? By bilinear interpolation? And what size is the window_size set to? (14, 18) to main the same receptive field? or (7, 9)? — Reply to this email directly, view it on GitHub<#29 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ADLPY6XIHS3255SU2LAOJSLY4YOHTAVCNFSM6AAAAABF7XJOWOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANBYHE2TOMBTGE>. You are receiving this because you commented.Message ID: ***@***.***>

[Rockyman-star]

You may check the swin transformer code here https://github.com/prismformore/Multi-Task-Transformer/blob/main/TaskPrompter/models/transformers/taskprompter_swin.py Get Outlook for Androidhttps://aka.ms/AAb9ysg
…
________________________________ From: samuel-zhaoy @.> Sent: Wednesday, April 10, 2024 10:34:49 PM To: prismformore/Multi-Task-Transformer @.> Cc: Hanrong Ye @.>; Comment @.> Subject: Re: [prismformore/Multi-Task-Transformer] How does the swin transformer deal with the input image with the size of 448 x 576 ? (Issue #29) Okay! Thanks for your reminder. I guess you modified all the window_size to a tuple/list form in the original swin_transformer.py to an uneven height-width partition. But how did you use officially released pretrained parameters relative_position_bias_tablehttps://github.com/microsoft/Swin-Transformer/blob/968e6b5e428186dc99d19166996439c23dccc4d1/models/swin_transformer.py#L101 as the shape changes? By bilinear interpolation? And what size is the window_size set to? (14, 18) to main the same receptive field? or (7, 9)? — Reply to this email directly, view it on GitHub<#29 (comment)>, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ADLPY6XIHS3255SU2LAOJSLY4YOHTAVCNFSM6AAAAABF7XJOWOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANBYHE2TOMBTGE. You are receiving this because you commented.Message ID: @.***>

Ok, thanks. I find that you pad the image to the shape which can be divisible by window_size.