core dump in running tensorflow benchmark about byteps

[ymjiang]

Looks like your gcc version is higher than our tested gcc-4.9.

Can you try to pin down gcc before you build BytePS? Here is an example: https://github.com/bytedance/byteps/blob/master/docker/Dockerfile.worker.tensorflow#L115-L123

[bobzhuyb]

I think it's a very similar issue as this tensorflow/tensorflow#13308 (comment)
It happens when BytePS is compiled with gcc 5 while TF is compiled with gcc 4.

It's still an open issue..

If your TF is compiled with gcc 4 (and it seems so), you have to use gcc-4.9 to build BytePS. You can try the suggestion given by @ymjiang , or use our pre-built docker image.

[un-knight]

I run the pytorch benchmark example, and meet the same core dump error.

/usr/local/byteps/example/pytorch/start_pytorch_byteps.sh: line 20:    30 Aborted                 (core dumped) python $path/benchmark_byteps.py $@
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/opt/anaconda/lib/python3.7/threading.py", line 917, in _bootstrap_inner
    self.run()
  File "/opt/anaconda/lib/python3.7/threading.py", line 865, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/byteps/launcher/launch.py", line 19, in worker
    subprocess.check_call(command, env=my_env, stdout=sys.stdout, stderr=sys.stderr, shell=True)
  File "/opt/anaconda/lib/python3.7/subprocess.py", line 347, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '/usr/local/byteps/example/pytorch/start_pytorch_byteps.sh --model resnet50 --num-iters 1000' returned non-zero exit status 134.

[ymjiang]

@un-knight Can you share more information about your OS and env?

[bobzhuyb]

@un-knight Would you reply to the issue thread you already opened #20? From your log, I don't see how your question is related to this issue.

[byronyi]

Just provide the binary release. No need to build one on users' environment, we do not need mpicc or mpicxx.

[bobzhuyb]

Our plan is to release binary installation package built for different frameworks/CUDA versions, in order to ease user installation process.

[ymjiang]

@myotheone We just uploaded some pypi lists for easier installation. See https://github.com/bytedance/byteps/blob/master/docs/pip-list.md

[bobzhuyb]

Closing this since we started providing pypi packages. Feel free to reopen.