Comments (10)
We use a trick to reverse the gradient before the gradient back-propagate from discriminator to the feature extractor. So we do not need to use -1 at the discriminator loss to train the feature extractor.
There is no input from the discriminator to the classification loss. According to the Pytorch auto-grad principle. There is no gradient from the classification loss even we back-propagate from the sum of the classification loss and the transfer loss.
from cdan.
Thanks for your answer, the trick is x.register_hook(grl_hook(coeff))
, is that right?
from cdan.
Yes
from cdan.
How about the tensorflow version? Where is the trick of reversing the gradient?
Thanks a lot!
from cdan.
How about the tensorflow version? Where is the trick of reversing the gradient?
Thanks a lot!
The Tensorflow version is under implementation.
The trick of gradient reversing is in pytorch/network.py line 388. The grl_hook adds a grl layer between the ResNet CNN and the Domain Discriminator, which enables the update of the two adversarial players in one feedforward and backward propagation.
from cdan.
In the DANN model, after inserting a GRL between the generator and the discriminator, the gradient of domain loss w.r.t the feature extractor F will be multiplied by -1. But in CDAN model, the input of discriminator is the tensor product between feature vector and the predicted probability vector. So during backward propagation, the domain loss would have gradient with regard to both feature extractor F and classifier G. May I know how your algorithm computes the gradient of domain loss w.r.t. the predicted probabilities output by classifier G? Will the grl_hook also reverse the gradient of domain loss w.r.t. the classifier G? Thanks a lot!
from cdan.
In the DANN model, after inserting a GRL between the generator and the discriminator, the gradient of domain loss w.r.t the feature extractor F will be multiplied by -1. But in CDAN model, the input of discriminator is the tensor product between feature vector and the predicted probability vector. So during backward propagation, the domain loss would have gradient with regard to both feature extractor F and classifier G. May I know how your algorithm computes the gradient of domain loss w.r.t. the predicted probabilities output by classifier G? Will the grl_hook also reverse the gradient of domain loss w.r.t. the classifier G? Thanks a lot!
In pytorch/loss.py line 22, softmax_output = input_list[1].detach()
This detaches G from the domain loss during back-propagation, so the domain loss will not be used to update classifier G.
from cdan.
I cannot undstand two things. I appreciate it if you can explain. (1)pytorch/loss.py line 33. entropy.register_hook(grl_hook(coeff))
, Why the entropy need this *-1 hook? The grads passed back from the domain discriminator to the feature extractor have been inverted by using x.register_hook(grl_hook(coeff))
, Registering a *-1 hook for the entropy confuses me. (2) I noticed that you use softmax_output=input_list[1].detach()
which blocks the grads from the discrininator to the classifier, but the entropy is obtained by loss_func.Entropy(softmax_output)
resulting to entropy.requires_grad=True. This means the grads can be back-propagated to the classifier through entropy (am I right?), What is this for?
from cdan.
I cannot undstand two things. I appreciate it if you can explain. (1)pytorch/loss.py line 33.
entropy.register_hook(grl_hook(coeff))
, Why the entropy need this *-1 hook? The grads passed back from the domain discriminator to the feature extractor have been inverted by usingx.register_hook(grl_hook(coeff))
, Registering a *-1 hook for the entropy confuses me. (2) I noticed that you usesoftmax_output=input_list[1].detach()
which blocks the grads from the discrininator to the classifier, but the entropy is obtained byloss_func.Entropy(softmax_output)
resulting to entropy.requires_grad=True. This means the grads can be back-propagated to the classifier through entropy (am I right?), What is this for?
I also feel strange about problem(1). Do you understand now?
from cdan.
In the DANN model, after inserting a GRL between the generator and the discriminator, the gradient of domain loss w.r.t the feature extractor F will be multiplied by -1. But in CDAN model, the input of discriminator is the tensor product between feature vector and the predicted probability vector. So during backward propagation, the domain loss would have gradient with regard to both feature extractor F and classifier G. May I know how your algorithm computes the gradient of domain loss w.r.t. the predicted probabilities output by classifier G? Will the grl_hook also reverse the gradient of domain loss w.r.t. the classifier G? Thanks a lot!
In pytorch/loss.py line 22, softmax_output = input_list[1].detach()
This detaches G from the domain loss during back-propagation, so the domain loss will not be used to update classifier G.
But generally speaking, the domain loss needs to optimize the feature extraction network G
from cdan.
Related Issues (20)
- path HOT 2
- AlexNet pre-trained model HOT 1
- SVHN -> MNIST Accuracy HOT 3
- Confusion about the data HOT 1
- DANN HOT 4
- do we need whole iterations 100000? HOT 1
- Questions about learning rate decay
- How to choose checkpoints?
- IWCV
- Cannot train pytorch version on multi-gpu HOT 1
- NotImplementedError HOT 1
- Office-31 Test Set And Val Set
- Different results with multi-GPUs HOT 1
- RuntimeError: size mismatch, m1: [64 x 640], m2: [1792 x 1792] at /opt/conda/conda-bld/pytorch_1550813258230/work/aten/src/THC/generic/THCTensorMathBlas.cu:266
- The parameters in the ad_net
- How can I get usps dataset
- Learning rate setting HOT 1
- How to use CDAN to train with my own dataset?
- how to set parameters when training office-home with version of pytorch HOT 1
- Access for ImageCLEF dataset
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cdan.