ikostrikov / pytorch-flows Goto Github PK

In our application, we've found the need to provide the PDF for the latent variables (in our case, it is useful that the PDF depends on the conditional inputs). We've found a small fix to your code that let's you have this option, by just adding an init method to FlowSequential and changing log_probs accordingly. I'll share the different code here. Here is the added init method

    def __init__(self, *args, log_probs = 'gaussian'):
        super(FlowSequential,self).__init__(*args)
        if log_probs = 'gaussian'
            def __log_probs(x, *_):
                return torch.sum(-0.5 * x ** 2 - 0.5 * math.log(2 * math.pi),
                                                            -1, keepdim=True)
            self.log_probs = __log_probs
        else:
            self.log_probs = log_probs

and here is the modified log_probs.

    def log_probs(self, inputs, cond_inputs = None):
        u, log_jacob = self(inputs, cond_inputs)
        log_probs = self.log_probs(u, cond_inputs)
        return (log_probs + log_jacob).sum(-1, keepdim=True)

Then, to provide your own PDF, just do
model = FlowSequential(*modules,logprobs=your_own_PDF)

Does your code support MADEs with an arbitrary number of masked linear layers? My main question is really if the function get_mask works for hidden layers of arbitrary depth. Thanks!

I'm running into a problem when I call model.forward(..., mode='inverse') before calling model.forward(..., mode='direct'), in which case batch_mean and batch_var are not defined. What is a proper why to fix this problem? I need to use inverse mode first during the training stage in my application.

The code snippet looks like this. I've omitted some application specific codes.

block = [
fnn.MADE(num_inputs, num_hidden),
fnn.BatchNormFlow(num_inputs),
fnn.Reverse(num_inputs)
]
model = fnn.FlowSequential(*blocks)
model.train()
model.forward(..., mode='inverse')

AttributeError: 'BatchNormFlow' object has no attribute 'batch_mean'

The readme states:

For the moment, I removed MNIST and CIFAR10 because I have plans to add pixel-based models later.

what do you mean by that? What's missing?
I might be able to implement some stuff.

To reproduce, use:

python main.py --dataset GAS --flow glow

Printed message for first epoch:

Warning: Results for GLOW are not as good as for MAF yet.
Warning: Results for GLOW are not as good as for MAF yet.
Warning: Results for GLOW are not as good as for MAF yet.
Warning: Results for GLOW are not as good as for MAF yet.
Warning: Results for GLOW are not as good as for MAF yet.
Train Epoch: 0 [0/852174 (0%)]	Loss: 6.648179
Train Epoch: 0 [100000/852174 (12%)]	Loss: -2.193499
Train Epoch: 0 [200000/852174 (23%)]	Loss: -2.497202
Train Epoch: 0 [300000/852174 (35%)]	Loss: -1.122860
Train Epoch: 0 [400000/852174 (47%)]	Loss: -2.892694
Train Epoch: 0 [500000/852174 (59%)]	Loss: -2.268665
Train Epoch: 0 [600000/852174 (70%)]	Loss: -3.088916
Train Epoch: 0 [700000/852174 (82%)]	Loss: -3.798649
Train Epoch: 0 [800000/852174 (94%)]	Loss: -2.554942

Validation set: Average loss: nan

Best validation at epoch 0: Average loss: inf

I'm trying to replicate the MNIST and CIFAR-10 experiments used in the paper. If I want to modify the code to add Conditional Masked Autoregressive Flow, which part of the neural network model should I modify?

Thanks for your help!!

I found that constructing masks using the original MADE paper https://arxiv.org/abs/1502.03509 (nodes are randomly assigned ids) works better than assigning an id deterministically as in

(The reason for this is likely simple, that if the output size is not an exact multiple of the number of hidden units, some input features will be assigned more hidden nodes than others)

This currently matches

The forward computation is on

u = (inputs - m) * torch.exp(-a)

The inverse of this is

inputs = u * torch.exp(a) + m

So the log determinant jacobian of the inverse is a.sum(-1, keepdim=True) not -a.sum(-1, keepdim=True).

Might be missing something here!