chunyuanli / psgld

AAAI & CVPR 2016: Preconditioned Stochastic Gradient Langevin Dynamics (pSGLD)

pSGLD

Preconditioned Stochastic Gradient Langevin Dynamics (pSGLD)

Links: Implementation on TensorFlow Website

Simulation (2D Gaussian Example in Figure 1 of the paper)

• Simulation 1 provides Average Absolute Error of Sample Covariance vs AutoCorrelation Time (ACT)
• Simulation 2 provides first 600 samples from SGLD and pSGLD

Experiments on Deep Neural Networks (Keep updating)

• Start to run 'test_FNN_mnist.m' to test a 2-layer FNN with 400 hidden units each .
• You may also modify line 'linSizes = [400 400 data.outSize]' to other configurations.

Citation

Please cite our AAAI paper if it helps your research:

@inproceedings{pSGLD_AAAI2016,
  title={Preconditioned stochastic gradient Langevin dynamics for deep neural networks},
  author={Li, Chunyuan and Chen, Changyou and Carlson, David and Carin, Lawrence},
  booktitle={AAAI},
  Year  = {2016}
}

psgld's Issues

Why is noise scaled by Ntrain in RMSProp

In SGLD_RMSprop.m the noise is scaled by opts.N which is set to Ntrain in DNN experiments:
https://github.com/ChunyuanLI/pSGLD/blob/master/pSGLD_DNN/algorithms/SGLD_RMSprop.m#L51

Why is this the case? In the paper (https://arxiv.org/pdf/1512.07666v1.pdf) there is no such scaling.

I also checked SGLD_Adagrad.m and there is no scaling by Ntrain for the noise.

Undefined function or variable 'environmentVariables'.

Hi,

I was trying to run test_FNN_mnist.m but I got the following error:
" Undefined function or variable 'environmentVariables'. "
Can you tell me to fix it?

Thanks

No learning rate annealing during training.

Hi Chunyuan,

Thanks for sharing. I found that in the 2-D simulation experiment the learning rate(injected Gaussian noise level) is kept constant, which doesn't satisfy the assumption 1 in your AAAI '16 paper. While in previous works e.g. Welling 2011, I found a polynomial decay scheme is applied. Will this be a problem?

Where the prior on the parameters is reflected in the code?

Thanks for sharing your code!

Your paper mentions about the prior on the parameters being , and that the variance () is set to 1 by default for DNN experiments and in some scenario it is set to 100. But I couldn't find it reflected anywhere in the code. Am I missing something obvious?