Inquiry about the audio augmentation and evaluation setting for audio-vidual mid-fusion about ssw60

[rui1996]

Thank you for your interest in our work! Regarding your questions:

1. It is true that for multimodal fusion, one video input can only interact with one audio spec input. However, we could adopt multiple views for one video and multiple views for one audio spec. We fuse a pair of them each time and finally average the logits. This is what we do and we consider it is comparable.
2. Since we are using transformer as the backbone, MBT or token concatenation would be the most straightforward way. In MBT paper, it shows concatenation is not as good as MBT, so we just adopt MBT.
3. Yes we will release the code. For your question, the code looks like this:

def freq_masking(self, img, freq_factor=1.0, mask_len=15):
		factor = np.random.RandomState().rand()
		freq_len = img.shape[0]
		if factor <= freq_factor:
			start = np.random.randint(0, freq_len - mask_len)
			interval = np.random.randint(0, mask_len)
			img[start : start + interval, :] = 0
		return img

def time_masking(self, img, time_factor=1.0, mask_len=15):
		factor = np.random.RandomState().rand()
		time_len = img.shape[1]
		if factor <= time_factor:
			start = np.random.randint(0, time_len - mask_len)
			interval = np.random.randint(0, mask_len)
			img[:, start : start + interval] = 0
		return img

[Rick-Xu315]

Thanks for your reply! I would also like to ask another question:
In your paper you mention in table 3 that you get a lower performance of audio resnet 18 after finetuning on video-audio. I find similar result after we finetune the concat-based av model composed of pretrained unimodels and linear probe the audio backbone. I would like to know your opinions why the audio backbone gets worse after finetuning. Great thanks!