ikostrikov / pytorch-flows Goto Github PK
View Code? Open in Web Editor NEWPyTorch implementations of algorithms for density estimation
License: MIT License
PyTorch implementations of algorithms for density estimation
License: MIT License
In our application, we've found the need to provide the PDF for the latent variables (in our case, it is useful that the PDF depends on the conditional inputs). We've found a small fix to your code that let's you have this option, by just adding an init method to FlowSequential and changing log_probs accordingly. I'll share the different code here. Here is the added init method
def __init__(self, *args, log_probs = 'gaussian'):
super(FlowSequential,self).__init__(*args)
if log_probs = 'gaussian'
def __log_probs(x, *_):
return torch.sum(-0.5 * x ** 2 - 0.5 * math.log(2 * math.pi),
-1, keepdim=True)
self.log_probs = __log_probs
else:
self.log_probs = log_probs
and here is the modified log_probs.
def log_probs(self, inputs, cond_inputs = None):
u, log_jacob = self(inputs, cond_inputs)
log_probs = self.log_probs(u, cond_inputs)
return (log_probs + log_jacob).sum(-1, keepdim=True)
Then, to provide your own PDF, just do
model = FlowSequential(*modules,logprobs=your_own_PDF)
Does your code support MADEs with an arbitrary number of masked linear layers? My main question is really if the function get_mask works for hidden layers of arbitrary depth. Thanks!
I'm running into a problem when I call model.forward(..., mode='inverse')
before calling model.forward(..., mode='direct')
, in which case batch_mean
and batch_var
are not defined. What is a proper why to fix this problem? I need to use inverse mode first during the training stage in my application.
The code snippet looks like this. I've omitted some application specific codes.
block = [
fnn.MADE(num_inputs, num_hidden),
fnn.BatchNormFlow(num_inputs),
fnn.Reverse(num_inputs)
]
model = fnn.FlowSequential(*blocks)
model.train()
model.forward(..., mode='inverse')
AttributeError: 'BatchNormFlow' object has no attribute 'batch_mean'
The readme states:
For the moment, I removed MNIST and CIFAR10 because I have plans to add pixel-based models later.
what do you mean by that? What's missing?
I might be able to implement some stuff.
To reproduce, use:
python main.py --dataset GAS --flow glow
Printed message for first epoch:
Warning: Results for GLOW are not as good as for MAF yet.
Warning: Results for GLOW are not as good as for MAF yet.
Warning: Results for GLOW are not as good as for MAF yet.
Warning: Results for GLOW are not as good as for MAF yet.
Warning: Results for GLOW are not as good as for MAF yet.
Train Epoch: 0 [0/852174 (0%)] Loss: 6.648179
Train Epoch: 0 [100000/852174 (12%)] Loss: -2.193499
Train Epoch: 0 [200000/852174 (23%)] Loss: -2.497202
Train Epoch: 0 [300000/852174 (35%)] Loss: -1.122860
Train Epoch: 0 [400000/852174 (47%)] Loss: -2.892694
Train Epoch: 0 [500000/852174 (59%)] Loss: -2.268665
Train Epoch: 0 [600000/852174 (70%)] Loss: -3.088916
Train Epoch: 0 [700000/852174 (82%)] Loss: -3.798649
Train Epoch: 0 [800000/852174 (94%)] Loss: -2.554942
Validation set: Average loss: nan
Best validation at epoch 0: Average loss: inf
I'm trying to replicate the MNIST and CIFAR-10 experiments used in the paper. If I want to modify the code to add Conditional Masked Autoregressive Flow, which part of the neural network model should I modify?
Thanks for your help!!
I found that constructing masks using the original MADE paper https://arxiv.org/abs/1502.03509 (nodes are randomly assigned ids) works better than assigning an id deterministically as in
Line 19 in 5520ebe
(The reason for this is likely simple, that if the output size is not an exact multiple of the number of hidden units, some input features will be assigned more hidden nodes than others)
Line 114 in 34a34a0
This currently matches
Line 130 in 34a34a0
The forward computation is on
Line 113 in 34a34a0
u = (inputs - m) * torch.exp(-a)
The inverse of this is
inputs = u * torch.exp(a) + m
So the log determinant jacobian of the inverse is a.sum(-1, keepdim=True)
not -a.sum(-1, keepdim=True)
.
Might be missing something here!
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.