Batch optimization about pix2latent

[minyoungg]

If I understood your question correctly ..

The goal is to not have diverse initialization but to show that an algorithm that is robust to the initialization and local-minima can find better solutions.

Adam has a nice property of letting you optimize a single sample. However, Adam is greatly affected by the initialization of latent variable z and therefore it's beneficial to do batched-optimization regardless and take the argmax_zs L(G(zs), y) where zs is the batch of latent variables z.

A more initialization-robust alternative is to use BasinCma/CMA. CMA-based algorithms are natively a batched es-optimizers. The optimizer will update the sampling distribution with the evaluation from the batched-candidates. Resample candidates from the updated distribution and repeat.

In the end, it boils down to what you care about. Speed vs quality of fit.

Adam: very fast + mediocre fit (allows you to optimize with any number of batch-size)
CMA: fast + decent fit (no backward pass required / fixed batch optimization)
BasinCMA: slow + great fit (backward pass required / fixed batch optimization)

[kkang831]

Thank you for the kind explanation.
That's a perfect explanation.