duvenaud / deep-limits

Repo for a paper about constructing priors on very deep models.

Avoiding Pathologies in Very Deep Networks

Experiment source code and latex source for http://arxiv.org/pdf/1402.5836.pdf

Abstract:

Choosing appropriate architectures and regularization strategies for deep networks is crucial to good predictive performance. To shed light on this problem, we analyze the analogous problem of constructing useful priors on compositions of functions. Specifically, we study the deep Gaussian process, a type of infinitely-wide, deep neural network. We show that in standard architectures, the representational capacity of the network tends to capture fewer degrees of freedom as the number of layers increases, retaining only a single degree of freedom in the limit. We propose an alternate network architecture which does not suffer from this pathology. We also examine deep covariance functions, obtained by composing infinitely many feature transforms. Lastly, we characterize the class of models obtained by performing dropout on Gaussian processes.

This paper appeared in the 2014 Artificial Intelligence and Statistics conference, held in Reykjavik, Iceland.

Authors: David Duvenaud, Oren Rippel, Ryan P. Adams, and Zoubin Ghahramani

Feel free to email me with any questions at ([email protected]).

deep-limits's Issues

A mistake in your paper

A little mistake in your thesis. In Equation(5.25), the sign before (x-x') term should be plus. Also the Equation(5.11), I get the mean value 1/2*(2*log(sigma/omega)-log(2)-gamma), It is the same as yours except for the 1/2 term in the front. If it is true, the variance should be pi^2/8. Would you like to check your answer? I can send my code to you and see if it goes wrong somewhere. I would be happy to discuss with you in any details.

What's the relationship between your definition of deep gaussian process and original one?

Hi, I read the paper <> written by Andreas C. Damianou and Neil D. Lawrence. And I am a little confused about the relationship and difference between you two. Personally, I think your definition is more apt to understand. Can you say something about your understanding ? Does Damianou's deep GP also has pathology you mentioned in your paper?

Question for your paper

Hi, I am reading your Phd thesis recently. I am a little confused about the Equation 5.11 in the chapter 5. Can you give me any clue or references to deduce the equation ? Thank you very much.

Question for the distribution of singular values

Hi, I run your code in the file deep_singular_value_dist.m. I have a question about the variable scale in the file. It turn out this variable is so important here. If it is set to be 1, the result of the figure 1 and figure 2 do not show any obvious difference. Why does it happen? What's the function of variable scale ? Can you clarify it to me? I do not find a clue in your PhD thesis. Thank you very much!