Hyperparameter optimization about kaggle_diabetic

[sveitser]

I would probably try the following things, but I don't think there is much more to be gained this way either.

• Make the networks deeper where possible.
• Add another convnet to the ensemble.
• Try using different rectifier units. Notably the elu unit that is now available in lasagne sounds promising.
• You might be able to boost the score slightly if you try to compute optimal thresholds at the end rather than using 0.5, 1.5, 2.5, 3.5.
• I think you can at least save some time during training if you remove the fully connected layers and/or add 'nonlinearity': None to the parameters for the output layer. The default linearity is rectify so the minimum the convnets can output at the moment is 0. You might have to tune the learning rate schedule again for this to work well though.