How to reproduce result of Fig.1 in the paper? about swa

[izmailovpavel]

Hi @roderickObrist, good to hear SWA is working well for you :) To make the figures we treat all the parameters of the network as one large (say 10 million-dimensional) vector. It includes both the biases and the weights. Say we have a total of D parameters. We treat the whole parameter space as just R^D. For each visualization we then pick three vectors v1, v2 and v3 in this R^D parameter space. These typically correspond to the weights of some networks, like SGD iterates from different iterations. Then, we construct the unique 2-d plane (affine subspace) that passes through these three vectors. We then plot the loss restricted to this 2-d subspace.

To answer your questions:

1. Weights include both weights and biases, and they are not from a single layer. This is the full vector of all the network's parameters.
2. We have a public implementation of a very similar visualization for our other paper here: https://github.com/timgaripov/dnn-mode-connectivity/blob/master/plane.py. I believe you would need to change this part here https://github.com/timgaripov/dnn-mode-connectivity/blob/master/plane.py#L96-L101, and load the weights of three networks v1, v2, v3 in the w list.

[roderickObrist]

Thank you kindly, I will implement this in my own project over the next few days.

[izmailovpavel]

Please see footnote 1 on page 2 of the camera-ready version of the paper:
http://auai.org/uai2018/proceedings/papers/313.pdf
There we tried to clarify the exact procedure for making the loss and test error surface visualizations.

[izmailovpavel]

I will close the issue for now, but I will be happy to answer if you will have further questions about those figures.

[roderickObrist]

@izmailovpavel Hi and thank you for the great work, I've been implementing SWA in my research project and the results are great. I just have a few questions regarding the illustrations.

1. Are the weight vectors literally the weights (not biases) from a single linear layer of a network or are they the concatenation of the entire model?
2. Would you be comfortable providing the snippet of code you used to make the figures? (Does not need to be functional/polished or commented). Just so I can double check my own implementation.

Thank you for what you have done for the community.