06_Neural_../04_single_..., batch size maters about tensorflow_cookbook

[nfmcclure]

Hi @leigaoyi , Thanks for your questions.

I think you have two main questions. The first is 'What is the effect of batch size on convergence?'. One way to think of this is to first think of a smooth convex solution space. In this situation, we have exactly one global minimum and ideally we just use the whole dataset in one single batch and converge cleanly towards the minimum solution. Of course, most sample spaces have multiple local minimum and using the whole dataset risks us getting stuck in a local minimum. To combat this, we can change the solution space by sampling the data, creating smaller batches of data. The smaller the sample the more 'jitter' we introduce in the convergence. We have to strike a balance between jittering the convergence too little (getting stuck in a local minimum) and jittering the convergence too much (not finding the global minimum).

Your second question is about how TensorFlow can apply operations across tensors of different sizes. In this case, TensorFlow operates the same way as Numpy, using broadcasting. Broadcasting is a way to apply an operation across multiple dimensions of a tensor (or array). I recommend reading more here:
https://www.tensorflow.org/api_guides/python/math_ops

I'm going to close this issue. If you have problems with the code, feel free to reopen this issue. Thanks!