Training with GPUs about deepmath

[kbansal]

I assume you are referring to the TF_NEED_CUDA in the Dockerfile. Yes, you may. We have only tested the evaluation externally on CPU, but it should work with GPU as well -- it possible some other Dockerfile changes need to be made for Docker image to see the GPU. To keep it broadly usable, we don't enable it by default. If you try it, we would happy to hear about your attempt.

[einsiedler0408]

@kbansal , I kind of figured out how to use multiple GPUs for training. The most important changes are (1) add a .bazelrc to deepmath root directory with content of "import %workspace%/tensorflow/.bazelrc", (2) change the version of tensorflow to r1.12 (because there is no compatible tensorflow-estimator version for your current TF version, when using tf.contrib.distribute.MirroredStrategy).
Currently, I am not using Docker. When I have time, I may give you a corresponding DockerFile.

I have some new questions: (1) I am running deepmath/deephol/train/ experiments.py. Am i right? (2) What is the batch size you actually use in the HOList paper?

[kbansal]

1 - yes, that is the training code. We had not ourselves tried the training code outside of Google and provided it as reference only, so glad to hear you were able to use it.

2- I asked my colleagues, and their recollection is 16 or 32. But try as large a batch_size as you are able to according to memory available.