Initialization of variables about tensorflow-wavenet

[ibab]

There's no reason behind picking stddev=0.3 at the moment, and using Xavier initialization would be nice. Note that the learnable filter isn't just an (in_channels, out_channels) matrix, as needed for the Xavier initializer, but a (filter_width, in_channels, out_channels) tensor.
Maybe stacking together filter_width Xavier initializers would work well.

[jyegerlehner]

Right,well there's also tf.contrib.layers.xavier_initializer_conv2d. But no tf.contrib.layers.xavier_initializer_conv1d. But the only difference is amonst these is how they compute fan-in and fan-out; after that they compute the normal-std-dev or uniform-width from that. Should be straightforward to roll our own.

[ibab]

@jyegerlehner: I didn't actually know about xavier_initializer_conv2d. Looking at the code, I can see that it only accesses the last 2 dimensions of the tensor, so we can just use it for our rank 3 tensors.
This example just works:

import tensorflow as tf

initializer = tf.contrib.layers.xavier_initializer_conv2d()
variable = tf.get_variable('x', shape=[2, 32, 32], initializer=initializer)

sess = tf.Session()
sess.run(tf.initialize_all_variables())
print(sess.run(variable))

[jyegerlehner]

OK I'll try to get to this in the not-too-distant future. If anyone else wants to pick it up please speak up so we don't end up duplicating effort.

[ibab]

@jyegerlehner: It would make sense to define a _create_variable(name, shape) method that defines a new variable and initializes it using the Xavier initializer, so that it becomes easier to make the initializer configurable.

[jyegerlehner]

@ibab Sounds like a good idea.