yanchaoyang / fda Goto Github PK
View Code? Open in Web Editor NEWFourier Domain Adaptation for Semantic Segmentation
Fourier Domain Adaptation for Semantic Segmentation
Can not find the synthia_mapped_to_cityscapes(labels) in dataset. Can you help me?
Hi, thanks for your awesome work, it's really a simlpe but effective method for domain adaptation.
However, the FDA_source_to_target_np()
and FDA_source_to_target()
is a little slow for runing code, is this a normal phenomenon?
Hello and thank you for your great work :-)
I have a question about the performance boost of around 1,5% after the second SST round.
How do you achieve that? I tried it out twice and both times my model has a worse performance than after the first SST round.
Do I have to change something in the settings or do I just have to retry it a couple more times.
Hi, thanks for your work. Running train.py gives error :
/media/data/ObjectDetectionExperiments/Projects/3_SemanticSegment/FDA-master/my_env/lib/python3.5/site-packages/torch/nn/functional.py:52: UserWarning: size_average and reduce args will be deprecated, please use reduction='elementwise_mean' instead.
warnings.warn(warning.format(ret))
Traceback (most recent call last):
File "/media/data/ObjectDetectionExperiments/Projects/3_SemanticSegment/FDA-master/train.py", line 135, in
main()
File "/media/data/ObjectDetectionExperiments/Projects/3_SemanticSegment/FDA-master/train.py", line 94, in main
trg_seg_score = model(trg_img, lbl=trg_lbl, weight=class_weights, ita=args.ita) # forward pass
File "/media/data/ObjectDetectionExperiments/Projects/3_SemanticSegment/FDA-master/my_env/lib/python3.5/site-packages/torch/nn/modules/module.py", line 477, in call
result = self.forward(*input, **kwargs)
File "/media/data/ObjectDetectionExperiments/Projects/3_SemanticSegment/FDA-master/model/deeplab.py", line 181, in forward
self.loss_seg = self.CrossEntropy2d(x, lbl, weight=weight)
File "/media/data/ObjectDetectionExperiments/Projects/3_SemanticSegment/FDA-master/model/deeplab.py", line 237, in CrossEntropy2d
predict = predict[target_mask.view(n, h, w, 1).repeat(1, 1, 1, c)].view(-1, c)
RuntimeError: CUDA error: out of memory
If i use in data/init.py image size = (512, 256), train is running without error and take ~5Gb GPU. Why it happen?
作者您好,请问Source Set和Target Set的图像配对是随机的吗?我看代码中是这样做的,请问这样不会有归纳偏差吗?
hi, how to convert the format? do I need to preprocess the labels of synthia dataset?
Hi, thanks for sharing your work. However, I have some questions about your work.
Or you just introduce two types of self training ? entropy minimization and pseudo label retraining?
Thanks! looking forward to your reply!
Line 57 in b9a0cdf
Hi,
During the second and the third round of SSL training, are the models still be trained from scratch or restored from the end of the previous round? According to the instruction, it seems like it will be initialized from scratch again?
Thanks
When I run train.py with --LB=0.01 as the command you show, I get files about:
gta5_2500.pth~gta5_100000.pth, I guess you will test this file and rename the best as gta2city_LB_0_01.pth for multiple test. But I can I find the best model of single beta?
The remapping labels range from 0~18, and the num_classes is set into 19. The background and ignored semantic objects are missing? At the inference stage, the backgrounds and ignored objects are forcibly allocated to 0~18?
Great work! Could you share the model weights before the SST step (those getting around 44.6 mIoU on GTA5 to Cityscapes)? It might be interesting to see if other self-training methods might improve the results.
Hello, thank you for your outstanding work. I would like to know if you have compared the fda method and the random image augmentation method, because from the visual results, there are some similarities and fda may be a more theoretical image enhancement.
Thanks a lot for your excellent work!
I found the loss_seg_src are used in SStrain.py. But in other related work, the model is usually only fine-tuned training on pseudo-label samples. I would like to ask if there is a big difference between these two methods
Unable to reproduce good results of Synthia to Cityscapes. In LB=0.01, the miou is 44.09 before ssl. In LB=0.05, the miou is 43.3 before ssl. Can you please provide the hyperparameter settings on the Synthia dataset or your intermediate weight?Thanks a lot.
Is this Deeplabv2 pretrained on source data? Or is it the initialized model Deepla-resnet101 imagenet weights? Quite important if I would need to train deeplabv2 on my source data, or can I just start with these weights
Hello~As we know, SYNTHIA dataset only share 16 classes with Cityscapes. So when training the model on the Synthia dataset, do I need to change the model classification layer to 16 channels? Does this matter?
Hi, thank you for sharing your code!
I have noticed two errors in your implementation of FDA in
Lines 11 to 18 in b9a0cdf
Only the the top left and bottom left part of the image in the Fourier space needs to be changed.
This is because you use rfft (and not fft) and, as a result, only half of the image in the Fourier domain is returned (due to the symmetry). Otherwise, you are replacing high frequency which is not what is described in the paper.
The indices of the region mutated should be symmetrical around 0. Otherwise the output after inverse FFT will have a non-zero imaginary part (-b
should be -b+1
above).
Also you may want to multiply b
by 0.5 to get the max transform for L=1 instead of L=0.5 (see #36 (comment))
I propose to tackle those issues with this modified version fo your pytorch implementation:
def extract_ampl_phase(fft_im):
# fft_im: size should be b x 3 x h x w
fft_amp = torch.abs(fft_im)
fft_pha = torch.angle(fft_im)
return fft_amp, fft_pha
def low_freq_mutate(amp_src, amp_trg, L=0.1):
_, _, h, w = amp_src.size()
# multiply w by 2 because we have only half the space as rFFT is used
w *= 2
# multiply by 0.5 to have the maximum b for L=1 like in the paper
b = (np.floor(0.5 * np.amin((h, w)) * L)).astype(int) # get b
if b > 0:
# When rFFT is used only half of the space needs to be updated
# because of the symmetry along the last dimension
amp_src[:, :, 0:b, 0:b] = amp_trg[:, :, 0:b, 0:b] # top left
amp_src[:, :, h-b+1:h, 0:b] = amp_trg[:, :, h-b+1:h, 0:b] # bottom left
return amp_src
def FDA_source_to_target(src_img, trg_img, L=0.1):
# get fft of both source and target
fft_src = torch.fft.rfft2(src_img.clone(), dim=(-2, -1))
fft_trg = torch.fft.rfft2(trg_img.clone(), dim=(-2, -1))
# extract amplitude and phase of both ffts
amp_src, pha_src = extract_ampl_phase(fft_src.clone())
amp_trg, pha_trg = extract_ampl_phase(fft_trg.clone())
# replace the low frequency amplitude part of source with that from target
amp_src_ = low_freq_mutate(amp_src.clone(), amp_trg.clone(), L=L)
# recompose fft of source
real = torch.cos(pha_src.clone()) * amp_src_.clone()
imag = torch.sin(pha_src.clone()) * amp_src_.clone()
fft_src_ = torch.complex(real=real, imag=imag)
# get the recomposed image: source content, target style
_, _, imgH, imgW = src_img.size()
src_in_trg = torch.fft.irfft2(fft_src_, dim=(-2, -1), s=[imgH, imgW])
return src_in_trg
I have tested it with PyTorch v1.11.0.
Best wishes,
Lucas
Good day! I have read your paper and want to implement it on another 2 datasets. I want to ask you about adaptation - did you take different images from dataset you adopt to or you took only 1? I mean you take for every adopting image random target image or 1 for everything?
Why does VGG16 have different initialization weights for synthia and GTA5?
Hello @YanchaoYang, thanks for the great work. I have some questions about the weights initialization (--init-weights='DeepLab_init.pth'). What are they? I mean, if I initialize the network with those weights and infer in some Cityscapes images without any training, we already have some prediction results.
But when I try to train the model from the scratch (without any weight initialization), the loss get stuck. Also, I found that if we do not initialize the model with some weights, they are initialized with zeros (at least in VGG model, as in code bellow).
def _initialize_weights(self):
for m in self.modules():
if isinstance(m, nn.Conv2d):
m.weight.data.zero_()
if m.bias is not None:
m.bias.data.zero_()
if isinstance(m, nn.ConvTranspose2d):
assert m.kernel_size[0] == m.kernel_size[1]
initial_weight = self.get_upsampling_weight(
m.in_channels, m.out_channels, m.kernel_size[0])
m.weight.data.copy_(initial_weight)
My question is: What are the steps required to train the network from scratch? For example in another dataset that I don't have this initialization weights.
Thank you again!
In the paper it is mentioned, gpu GTX1080 Ti is used but nothing is mentioned about CPU ram. This can cause problems because the get_sudo_label command which is used for pseudo label generation requires a lot of CPU ram ( almost 36 GB to be precise). Also I felt the get_sudo_label file could have been coded better to decrease the CPU ram usage.
Hi,
I run the code " train.py" to reproduce "Sim2Real Adaptation Using FDA (single beta)" performance following the procedure explained in Usage 2 in READme.
Unfortunately, I failed to reproduce the results and got 42.69 42.81 41.32 for β=0.01,0.05,0.09 respectively.
Can you please help me to understand the potential reasons behind this performance drop?
Regards,
Yiting Cheng
Hi,
in the training phase, when calculating the loss_ent variable in line 173 there is a line as follows:
ent = ent / 2.9444 # chanage when classes is not 19
what does the number 2.9444 represent and how should we change it if we use a different number of classes than 19?
I did not find any reference to this in the paper.
Thanks.
Thanks for your work!
I have a publicly available CT segmentation dataset for kidney tumor, and a private CT dataset also for kidney tumor. And I have trained a segmentation model using the public dataset after simple FDA (β=0.01 or 0.09), but there is not any improvement when test on the private dataset using the trained model. I don't know what should I do for this matter.
Hi,
I trained a T=0 B=0.01 model (without self learning for now) and the final checkpoint weights of iteration 100,000 was ~41% mIoU.
When checking more checkpoints of earlier iterations I found better scores like ~43%.
My questions is as follows: How did you evaluate your score in the paper? for the same experiment you achieved a score of 44.61% and I was wondering if you used the last checkpoint or did you check all 40 iteration checkpoints until you found the best one.
Thanks.
Can you provide the GTA5 dataset
Hi,
Pretty cool work! Any updates on when the code would be released?
Thanks!
Z.
Hi again,
Does this file hold the weights of DeepLab trained on GTA5 dataset or is it some regular initialization (like Imagenet for example)?
Thanks
How to extend FDA to 3D?
Hi, @YanchaoYang , thanks for the nice work.
I have a question about the image transformation process.
In FDA_demo.py, the pixel values of source and target image are in range of 0-255. But after we transforming the source image to the target style by swapping the amplitude spectrum, the generated image are not in this range (for exmaple the pixel value is in range -119-181 when setting alpha as 0.001).
Would this cause any trouble for model training since the generated images and target images used for training are not in the same pixel value range?
Many thanks.
Hi,
Thanks for this contribution.
What are the requirements for training the model from scratch? being more specific:
I there anything crucial needed for reproducing the results of the paper? from my experience minor changes in the training environment can make a big difference.
Thanks =)
thank you for this good work. I have a question for this dataset. Now i have two own dataset. One is generated from my simulation environment and another is from real world without annotation. It is actually about airport runway segmentation. If i want to use your model FDA, how can i adjust my dataset structure to train this model? thank you very much in advance
Why in
self.id_to_trainid = {3: 0, 4: 1, 2: 2, 21: 3, 5: 4, 7: 5,
15: 6, 9: 7, 6: 8, 16: 9, 1: 10, 10: 11, 17: 12,
8: 13, 18: 14, 19: 15, 20: 16, 12: 17, 11: 18}
you dont convert "void" 0 to 255 ?
I guess you use code in FDA_demo to transform original images, but when run FDA_demo.py, nothing happen and the process over in several seconds. Does FDA_demo.py really work?
Hi,
Is there any reason why the resolution of cityscapes is 1344, 576 instead of 1024, 512 while testing?
Thanks a lot in advance. I'm looking forward to your reply.
Can you please share the environment you used to run FDA_demo.py ? I am unable to get the translation result as that of FDA_demo.py using the latest versions, and getting artefacts as follows:
The code I used is:
import numpy as np
from PIL import Image
def low_freq_mutate_np( amp_src, amp_trg, L=0.1 ):
a_src = np.fft.fftshift( amp_src, axes=(-2, -1) )
a_trg = np.fft.fftshift( amp_trg, axes=(-2, -1) )
_, h, w = a_src.shape
b = ( np.floor(np.amin((h,w))*L) ).astype(int)
c_h = np.floor(h/2.0).astype(int)
c_w = np.floor(w/2.0).astype(int)
h1 = c_h-b
h2 = c_h+b+1
w1 = c_w-b
w2 = c_w+b+1
a_src[:,h1:h2,w1:w2] = a_trg[:,h1:h2,w1:w2]
a_src = np.fft.ifftshift( a_src, axes=(-2, -1) )
return a_src
def FDA_source_to_target_np( src_img, trg_img, L=0.1 ):
# exchange magnitude
# input: src_img, trg_img
src_img_np = src_img #.cpu().numpy()
trg_img_np = trg_img #.cpu().numpy()
# get fft of both source and target
fft_src_np = np.fft.fft2( src_img_np, axes=(-2, -1) )
fft_trg_np = np.fft.fft2( trg_img_np, axes=(-2, -1) )
# extract amplitude and phase of both ffts
amp_src, pha_src = np.abs(fft_src_np), np.angle(fft_src_np)
amp_trg, pha_trg = np.abs(fft_trg_np), np.angle(fft_trg_np)
# mutate the amplitude part of source with target
amp_src_ = low_freq_mutate_np( amp_src, amp_trg, L=L )
# mutated fft of source
fft_src_ = amp_src_ * np.exp( 1j * pha_src )
# get the mutated image
src_in_trg = np.fft.ifft2( fft_src_, axes=(-2, -1) )
src_in_trg = np.real(src_in_trg)
return src_in_trg
im_src = Image.open("source.png").convert('RGB')
im_trg = Image.open("target.png").convert('RGB')
im_src = im_src.resize( (1024,512), Image.BICUBIC )
im_trg = im_trg.resize( (1024,512), Image.BICUBIC )
im_src = np.asarray(im_src, np.float32)
im_trg = np.asarray(im_trg, np.float32)
im_src = im_src.transpose((2, 0, 1))
im_trg = im_trg.transpose((2, 0, 1))
src_in_trg = FDA_source_to_target_np( im_src, im_trg, L=0.01 )
src_in_trg = src_in_trg.transpose((1,2,0))
# scipy.misc.toimage(src_in_trg, cmin=0.0, cmax=255.0).save('demo_images/src_in_tar.png')
src_in_trg_=Image.fromarray(np.uint8(src_in_trg))
src_in_trg_.save('gfg_dummy_pic.png')
I am not able to understand the use of this line in the entropy loss implementation. How was this derived? What is its relation to number of classes?
Line 51 in b9a0cdf
The border b
in FDA numpy implementation https://github.com/YanchaoYang/FDA/blob/master/utils/__init__.py#L64 is taken as min(height, width)*beta
which is not in line with the paper.
To quote the paper:
As we can see from Eq. (3),
$\beta = 0$ will render$x^{s→t}$ the same as the original source image$x^s$ . On the other hand, when$\beta = 1.0$ , the amplitude of$x^s$ will be replaced by that of$x^t$ .
But while setting the beta=1, we get almost the original source image back. Also, setting beta=1 should replace all amplitude not just along the smaller axis.
Steps to reproduce the behavior:
betas = [0.001, 0.1, 0.5, 0.8, 0.9, 1.0]
f, axes = plt.subplots(3, 2, figsize=(30, 20))
for beta,ax in zip(betas, axes.ravel()):
image = FDA_source_to_target_np(src, tar , L=beta)
image = np.clip(image, 0, 255).astype(np.uint8)
image = image.transpose(1,2,0)
ax.imshow(image)
ax.set_title(f"L: {beta}")
ax.set_axis_off()
f.tight_layout()
The "augmentation" should be maximum at beta=1 not at beta=0.5.
The readme notes that we should subtract the mean after the Fourier conversion.
However, I notice that the mean of the images are subtracted in the dataset class, method get_item.
Line 58 in b9a0cdf
And during the train loop, the mean is subtracted again:
Line 81 in b9a0cdf
May anyone shed some lights for this approach? Thank you for your help in advance.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.