mlcommons / training

Reference implementations of MLPerf™ training benchmarks

MLPerf™ Training Reference Implementations

This is a repository of reference implementations for the MLPerf training benchmarks. These implementations are valid as starting points for benchmark implementations but are not fully optimized and are not intended to be used for "real" performance measurements of software frameworks or hardware.

Please see the MLPerf Training Benchmark paper for a detailed description of the motivation and guiding principles behind the benchmark suite. If you use any part of this benchmark (e.g., reference implementations, submissions, etc.) in academic work, please cite the following:

@misc{mattson2019mlperf,
    title={MLPerf Training Benchmark},
    author={Peter Mattson and Christine Cheng and Cody Coleman and Greg Diamos and Paulius Micikevicius and David Patterson and Hanlin Tang and Gu-Yeon Wei and Peter Bailis and Victor Bittorf and David Brooks and Dehao Chen and Debojyoti Dutta and Udit Gupta and Kim Hazelwood and Andrew Hock and Xinyuan Huang and Atsushi Ike and Bill Jia and Daniel Kang and David Kanter and Naveen Kumar and Jeffery Liao and Guokai Ma and Deepak Narayanan and Tayo Oguntebi and Gennady Pekhimenko and Lillian Pentecost and Vijay Janapa Reddi and Taylor Robie and Tom St. John and Tsuguchika Tabaru and Carole-Jean Wu and Lingjie Xu and Masafumi Yamazaki and Cliff Young and Matei Zaharia},
    year={2019},
    eprint={1910.01500},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

These reference implementations are still very much "alpha" or "beta" quality. They could be improved in many ways. Please file issues or pull requests to help us improve quality.

Contents

We provide reference implementations for benchmarks in the MLPerf suite, as well as several benchmarks under development.

Each reference implementation provides the following:

• Code that implements the model in at least one framework.
• A Dockerfile which can be used to run the benchmark in a container.
• A script which downloads the appropriate dataset.
• A script which runs and times training the model.
• Documentation on the dataset, model, and machine setup.

Running Benchmarks

Follow instructions on the Readme of each benchmark. Generally, a benchmark can be run with the following steps:

1. Setup docker & dependencies. There is a shared script (install_cuda_docker.sh) to do this. Some benchmarks will have additional setup, mentioned in their READMEs.
2. Download the dataset using ./download_dataset.sh. This should be run outside of docker, on your host machine. This should be run from the directory it is in (it may make assumptions about CWD).
3. Optionally, run verify_dataset.sh to ensure the was successfully downloaded.
4. Build and run the docker image, the command to do this is included with each Benchmark.

Each benchmark will run until the target quality is reached and then stop, printing timing results.

Some these benchmarks are rather slow or take a long time to run on the reference hardware. We expect to see significant performance improvements with more hardware and optimized implementations.

MLPerf Training v4.0 (Submission Deadline May 10, 2024)

*Framework here is given for the reference implementation. Submitters are free to use their own frameworks to run the benchmark.

model	reference implementation	framework	dataset
resnet50v1.5	vision/classification_and_detection	tensorflow2	Imagenet
RetinaNet	vision/object detection	pytorch	OpenImages
3DUnet	vision/image segmentation	pytorch	KiTS19
Stable Diffusionv2	image generation	pytorch	LAION-400M-filtered
BERT-large	language/nlp	tensorflow	Wikipedia 2020/01/01
GPT3	language/llm	paxml,megatron-lm	C4
LLama2 70B-LoRA	language/LLM fine-tuning	pytorch	SCROLLS govtReport
DLRMv2	recommendation	torchrec	Criteo 4TB multi-hot
RGAT	GNN	pytorch	IGBFull

training's Issues

[transformer] Specify Versions of Python Packages in Dockerfile

Most of the versions were fixed in a recent commit, but numpy, scipy, and sklearn are still installing the latest rather than a fixed version.

typo in object detection loss description

The loss function descriptions mix up classification and detection loss functions: https://github.com/mlperf/reference/blob/master/object_detection/caffe2/README.md#loss-function. Smooth L1 is for box coordinates, log-loss is for object class.

[DeepSpeech2] RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation

DS2 code is throwing following error when run_and_time.sh is executed.

Traceback (most recent call last): File "train.py", line 289, in <module> main() File "train.py", line 214, in main loss.backward() File "/h/anandj/reference/speech_recognition/env/local/lib/python2.7/site-packages/torch/tensor.py", line 93, in backward torch.autograd.backward(self, gradient, retain_graph, create_graph) File "/h/anandj/reference/speech_recognition/env/local/lib/python2.7/site-packages/torch/autograd/__init__.py", line 89, in backward allow_unreachable=True) # allow_unreachable flag RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation

The only thing I changed from existing code is the pytorch version. I found some problems while installing pytorch 0.2.0. So I switched it to 0.4.0.

[transformer] Reduce Checkpointing Frequency

There are currently roughly 230 total checkpoints @ 2.5GB each. While only the latest 5 are retained, it is a lot of I/O that accounts for a few percent of the overall runtime. This is a result of using the default 10 minute interval in TF. I am in favor of reducing this to one checkpoint per bleu score evaluation (once per epoch) for benchmark purposes.

[reinforcement] Async MCTS implementation details

Hey,
Can you elaborate or point me to a reference regarding the threaded scheme used during self play MCTS.
What structure do the threads share, specifically how are the leaves collected and batched?

Thank you for your contribution!

[speech_recognition] run_and_time error: can't open input file `/tmp/tmp_GBQNr.wav': WAVE: RIFF header not found

speech_recognition workload

step1.

I have built up the docker image and launch the docker instance:

cd pytorch
cd docker
sh build-docker.sh
sh run-dev.sh

step2.

And modifying the run-dev.sh:

#!/bin/bash
nvidia-docker run \
-v /home/linlf/project/linlf/mlperf/reference/speech_recognition:/speech_recognition:rw \
-v /etc/passwd:/etc/passwd:ro \
-it --rm --user $(id -u) ds2-cuda9cudnn7:gpu

step3.

Launch the docker instance:

sh run-dev.sh

.......
Removing intermediate container 450a04f6173f
 ---> 86101cb30a55
Step 16/18 : RUN ls ctcdecode/third_party
 ---> Running in 1512a8bc9a65
ThreadPool
boost_1_63_0.tar.gz
kenlm
openfst-1.6.7.tar.gz
utf8
Removing intermediate container 1512a8bc9a65
 ---> c482d12d74b2
Step 17/18 : RUN cd ctcdecode; pip install .
 ---> Running in c9349e7fe2e2
Processing /tmp/ctcdecode
Requirement already satisfied: cffi>=1.0.0 in /usr/local/lib/python2.7/dist-packages (from ctcdecode==0.3) (1.11.5)
Requirement already satisfied: pycparser in /usr/local/lib/python2.7/dist-packages (from cffi>=1.0.0->ctcdecode==0.3) (2.18)
Building wheels for collected packages: ctcdecode
  Running setup.py bdist_wheel for ctcdecode: started
  Running setup.py bdist_wheel for ctcdecode: still running...
  Running setup.py bdist_wheel for ctcdecode: finished with status 'done'
  Stored in directory: /tmp/pip-ephem-wheel-cache-dMi3ud/wheels/30/4f/54/e41e2d0b75f971a3f8daad6bb79bf61fef10d0e95c5b28a93d
Successfully built ctcdecode
Installing collected packages: ctcdecode
Successfully installed ctcdecode-0.3
Removing intermediate container c9349e7fe2e2
 ---> e3048146fa23
Step 18/18 : ENV SHELL /bin/bash
 ---> Running in dae62270553e
Removing intermediate container dae62270553e
 ---> 3d2838335713
Successfully built 3d2838335713
Successfully tagged ds2-cuda9cudnn7:gpu

step4.

Next I ready to execute the run_and_time.sh, however it throws an error:

root@5316ed702b45:/speech_recognition/pytorch# time sh run_and_time.sh | tee speech_ds2.out
=======================================================
***acc                       = 23.0 
***continue_from             =  
***save_folder               = models/ 
***start_epoch               = -1 
***checkpoint                = False 
***seed                      = 1 
***model_path                = models/deepspeech_t1.pth.tar 
=======================================================
Directory already exists.
DataParallel(
  (module): DeepSpeech(
    (conv): Sequential(
      (0): Conv2d(1, 32, kernel_size=(41, 11), stride=(2, 2))
      (1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (2): Hardtanh(min_val=0, max_val=20, inplace)
      (3): Conv2d(32, 32, kernel_size=(21, 11), stride=(2, 1))
      (4): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (5): Hardtanh(min_val=0, max_val=20, inplace)
    )
    (rnns): Sequential(
      (0): BatchRNN(
        (rnn): GRU(672, 800, bidirectional=True)
      )
      (1): BatchRNN(
        (batch_norm): SequenceWise (
        BatchNorm1d(800, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True))
        (rnn): GRU(800, 800, bidirectional=True)
      )
      (2): BatchRNN(
        (batch_norm): SequenceWise (
        BatchNorm1d(800, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True))
        (rnn): GRU(800, 800, bidirectional=True)
      )
      (3): BatchRNN(
        (batch_norm): SequenceWise (
        BatchNorm1d(800, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True))
        (rnn): GRU(800, 800, bidirectional=True)
      )
      (4): BatchRNN(
        (batch_norm): SequenceWise (
        BatchNorm1d(800, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True))
        (rnn): GRU(800, 800, bidirectional=True)
      )
    )
    (fc): Sequential(
      (0): SequenceWise (
      Sequential(
        (0): BatchNorm1d(800, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (1): Linear(in_features=800, out_features=29, bias=False)
      ))
    )
    (inference_log_softmax): InferenceBatchLogSoftmax()
  )
)
Number of parameters: 38115968
formats: can't open input file `/tmp/tmp_GBQNr.wav': WAVE: RIFF header not found
ERROR: Unexpected segmentation fault encountered in worker.
Traceback (most recent call last):
  File "train.py", line 289, in <module>
    main()
  File "train.py", line 176, in main
    for i, (data) in enumerate(train_loader, start=start_iter):
  File "/usr/local/lib/python2.7/dist-packages/torch/utils/data/dataloader.py", line 280, in __next__
    idx, batch = self._get_batch()
  File "/usr/local/lib/python2.7/dist-packages/torch/utils/data/dataloader.py", line 259, in _get_batch
    return self.data_queue.get()
  File "/usr/lib/python2.7/multiprocessing/queues.py", line 378, in get
    return recv()
  File "/usr/local/lib/python2.7/dist-packages/torch/multiprocessing/queue.py", line 21, in recv
    buf = self.recv_bytes()
  File "/usr/local/lib/python2.7/dist-packages/torch/utils/data/dataloader.py", line 178, in handler
    _error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 81) is killed by signal: Segmentation fault.

real	0m20.221s
user	0m5.148s
sys	0m7.076s

What's error mean?

Where is all of the training data stored for the translation benchmark?

[translation] tf-nightly-gpu==1.9.0.dev20180419 is no longer available

When building translation docker image, pip cannot find tf-nightly-gpu==1.9.0.dev20180419. The oldest dev version for tf-nightly-gpu is dev20180601. Why not pass the version number into docker by ARG command?

[image classification] ResNet version for closed submission

Should we specify which version of Resnet (v1 or v2) that people should use when submitting closed performance numbers for image classification?

ONNX Support

Any plans to put ONNX models, so the benchmark can be more framework independent?

Random seeds should be fixed in benchmark code

Hi,
Thanks for doing such a great job on behalf of the ML/AI community to advance the state of the field. I did have one key piece of feedback. I'm concerned that allowing end users to decide on random seeds is likely not a good practice. Towards ensuring the maximum comparability, I would like to suggest that the random seed(s) be fixed in the benchmark script as opposed to being a user input (command line or otherwise).
Thanks very much and I hope this helps.

[image_classification] Raise ResNet accuracy to 75.9% [Edit from 76.0%]

ResNet can reach 76%. Similar to the proposal to raise the translation score, we should choose a high accuracy target.

If we want to allow some wiggle room, maybe 75.9%.

[transformer] Specify NVIDIA Driver Version

The instructions do not currently specify the driver version.

[transformer] Mounting Data Volumes in Docker Container

The instructions should specify that the processed_data and model folders are mounted rather than storing the data inside of the container. This bypasses Docker's storage driver and overheads. It follows Docker's suggested best practices as well.

For consistency, it may be good to have all of the benchmarks load/store from the same mount points.

For example, the command line could look like:
docker run -v ${INPUT_DATA_MOUNT}:/raw_data -v ${PROCESSED_DATA_MOUNT}:/research/transformer/processed_data -v ${OUTPUT_DATA_MOUNT}:/research/transformer/model --runtime=nvidia -e CUDA_VISIBLE_DEVICES="${GPUS}" -i ${IMAGE} "./run_and_time.sh" ${SEED} &> ${OUTPUT_DATA_MOUNT}/benchmark-${NOW}.log"

Reference:
https://docs.docker.com/develop/dev-best-practices/#where-and-how-to-persist-application-data
https://docs.docker.com/storage/storagedriver/

translation test files should be added to raw_data

the data_download script puts the newstest2014.en/de files under reference/translation/tensorflow
they should be put in raw_data
so they can be shared with translators for other frameworks

[transformer] Update TensorFlow Version

A recent commit locked down the version of TF to tf-nightly-gpu==1.9.0.dev20180419 that was used in the original reference runs, but this should be changed to a stable release version rather than a nightly build.

[image_classification] Log keeps saying 'train_accuracy = 0'

I've been training image_classification over hours but the log keeps saying that train_accuracy is zero. Loss changes at every logging time, however. Is that normal?

[object detection] git operation error in ubuntu 16.04

When I execute docker build under caffe2 directory, I run into this error when git clone caffe2. I start the container before this operation done, and git clone manually but also the same error. There may be a bug in libcurl-gnutls library relied on by git based on here https://curl.haxx.se/mail/lib-2012-10/0132.html
And centos doesn't have this problem when I try this on my host.

Error output when docker build

Step 19/42 : RUN git clone [email protected]:caffe2/caffe2.git
---> Running in 94f5e1a08d0b
Cloning into 'caffe2'...
error: cannot run ssh: No such file or directory
fatal: unable to fork
The command '/bin/sh -c git clone [email protected]:caffe2/caffe2.git' returned a non-zero code: 128

If I git clone manually in the container then I get this

root@ffa1a77b9f8b:/packages# git clone https://github.com/caffe2/caffe2.git
Cloning into 'caffe2'...
remote: Counting objects: 230439, done.
remote: Total 230439 (delta 0), reused 0 (delta 0), pack-reused 230438
Receiving objects: 100% (230439/230439), 393.44 MiB | 3.59 MiB/s, done.
error: RPC failed; curl 56 GnuTLS recv error (-110): The TLS connection was non-properly terminated.
Resolving deltas: 100% (211232/211232), done.

If I bypass git clone caffe2 by docker cp, then I run into another same error when do the git submodule update... It seems can't skip this error.

[speech_recognition]'IOError: Cannot assign requested address'. Changing download-site in speech_recognition workload

The download site of openfst-1.6.7.tar.gz and boost_1_63_0.tar.gz may not be reach in ctcdecode for Chinese. It would throw error just like:

Step 12/14 : RUN git clone --recursive https://github.com/parlance/ctcdecode.git
 ---> Using cache
 ---> 716635b0cded
Step 13/14 : RUN cd ctcdecode; pip install .
 ---> Running in eb02ef7ba1f6
Processing /tmp/ctcdecode
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-req-build-SmXxlu/setup.py", line 55, in <module>
        os.path.join(this_file, "build.py:ffi")
      File "/usr/lib/python2.7/distutils/core.py", line 111, in setup
        _setup_distribution = dist = klass(attrs)
      File "/usr/lib/python2.7/dist-packages/setuptools/dist.py", line 272, in __init__
        _Distribution.__init__(self,attrs)
      File "/usr/lib/python2.7/distutils/dist.py", line 287, in __init__
        self.finalize_options()
      File "/usr/lib/python2.7/dist-packages/setuptools/dist.py", line 327, in finalize_options
        ep.load()(self, ep.name, value)
      File "/usr/local/lib/python2.7/dist-packages/cffi/setuptools_ext.py", line 204, in cffi_modules
        add_cffi_module(dist, cffi_module)
      File "/usr/local/lib/python2.7/dist-packages/cffi/setuptools_ext.py", line 49, in add_cffi_module
        execfile(build_file_name, mod_vars)
      File "/usr/local/lib/python2.7/dist-packages/cffi/setuptools_ext.py", line 25, in execfile
        exec(code, glob, glob)
      File "/tmp/pip-req-build-SmXxlu/build.py", line 22, in <module>
        'third_party/openfst-1.6.7.tar.gz')
      File "/tmp/pip-req-build-SmXxlu/build.py", line 14, in download_extract
        out=dl_path)
      File "/usr/local/lib/python2.7/dist-packages/wget.py", line 526, in download
        (tmpfile, headers) = ulib.urlretrieve(binurl, tmpfile, callback)
      File "/usr/lib/python2.7/urllib.py", line 98, in urlretrieve
        return opener.retrieve(url, filename, reporthook, data)
      File "/usr/lib/python2.7/urllib.py", line 245, in retrieve
        fp = self.open(url, data)
      File "/usr/lib/python2.7/urllib.py", line 213, in open
        return getattr(self, name)(url)
      File "/usr/lib/python2.7/urllib.py", line 443, in open_https
        h.endheaders(data)
      File "/usr/lib/python2.7/httplib.py", line 1053, in endheaders
        self._send_output(message_body)
      File "/usr/lib/python2.7/httplib.py", line 897, in _send_output
        self.send(msg)
      File "/usr/lib/python2.7/httplib.py", line 859, in send
        self.connect()
      File "/usr/lib/python2.7/httplib.py", line 1270, in connect
        HTTPConnection.connect(self)
      File "/usr/lib/python2.7/httplib.py", line 836, in connect
        self.timeout, self.source_address)
      File "/usr/lib/python2.7/socket.py", line 575, in create_connection
        raise err
    IOError: [Errno socket error] [Errno 99] Cannot assign requested address

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-req-build-SmXxlu/

For Chinese users I suggest that you can change the file ~/speech_recognition/pytorch/docker/Dockerfile.gpu line59:

RUN git clone --recursive https://github.com/parlance/ctcdecode.git

to:

RUN git clone --recursive -b linrio-patch-1 https://github.com/linrio/ctcdecode.git

Thus you can download the openfst-1.6.7 and boost_1_63_0 reachablely and then complete installing ctcdecode, build-docker.

More info: branch linrio-patch-1in linrio/ctcdecode

[reference_results.md] Need units for the numbers

Hi,

I think it is better to add a unit column to all the results numbers in reference_results.md.

Thanks,
Christine

[reinforcement learning] NoneType error in some games

https://github.com/mlperf/reference/blob/cf9c7a4b52a714975852392c44f5b43fd02527f2/reinforcement/tensorflow/minigo/evaluation.py#L80
https://github.com/mlperf/reference/blob/cf9c7a4b52a714975852392c44f5b43fd02527f2/reinforcement/tensorflow/minigo/main.py#L177-L189

Sometimes, when the evaluation of a game terminates for some condition, it returns an empty list which is supposed to keep track of winners. In the reference implementation we perform computation assuming that list is not empty, which gives this error. I don't know what is the condition, since it isn't clear how the game decides the winner.

[translation] download-site unavailable.

translation workload

the download url in reference/translation/download_data.sh line 3 ,4 :

wget https://raw.githubusercontent.com/tensorflow/mlbenchmark/master/transformer/test_data/newstest2014.de?token=ABCvMNi6FA6_WuLcXj_UJLv_GUc8I9NWks5a9yfEwA -O tensorflow/newstest2014.en
wget https://raw.githubusercontent.com/tensorflow/mlbenchmark/master/transformer/test_data/newstest2014.en?token=ABCvMOlxzi69iyZbqRXa0tVuRzSz83v9ks5a9yfIwA -O tensorflow/newstest2014.de

Unavailable. Suggesting that replace them by:

wget https://nlp.stanford.edu/projects/nmt/data/wmt14.en-de/newstest2014.en -O tensorflow/newstest2014.en
wget https://nlp.stanford.edu/projects/nmt/data/wmt14.en-de/newstest2014.de -O tensorflow/newstest2014.de

[classification] Model is named RN50 v1 but is in fact not v1.

ResNet v1 has strides on 1x1 convolutions in the bottleneck block. This model has strides on 3x3 convolutions. This is not wrong, since this version is actually better, but it shouldn't be named v1.

Since v2 is already taken, and the version with strides on 3x3 conv is very similar to v1, I would propose naming it ResNet v1.5, to avoid misleading people who may not be familiar with this difference.

https://github.com/mlperf/reference/blob/5f41c7559cb5c478fa6320dc427f41aeaeb766d9/image_classification/tensorflow/official/resnet/resnet_model.py#L194-L246

[sentiment_analysis] How to install workload by my self?

@deepakn94
I want to install some other tools for docker container. And how to build the [sentiment_analysis] workload by my self like other workloads? The general method in Steps to set up the Paddle environment is that :

A docker image with PaddlePaddle pre-installed can be pulled using:

docker pull sidgoyal78/paddle:benchmark12042018

To run the docker container, use:

nvidia-docker run -it -v `pwd`:/paddle sidgoyal78/paddle:benchmark12042018 /bin/bash
cd paddle/

Can this workload support the self-build method ? Liking Dockerfile for build docker container.

error in runing translation

translation workload

Question 1.

I follow the Steps to run and time to run the workload, but it does not work, I suggest that we can step-by-step start the workload:

step1 $ cd ~/reference/translation/tensorflow
step2 $ sudo docker build .
(then we can get the docker ID, such as : c36bb132a842. Modify the $HOME, for example my $HOME is /home/linlf/project/linlf/mlperf)
step3 $sudo docker run -v  $HOME/reference/translation/raw_data:/raw_data --runtime=nvidia -t -i c36bb132a842
(if you do need to sign which GPU to use:)
($sudo NV_GPU=0 docker run -v $HOME/reference/translation/raw_data:/raw_data --runtime=nvidia -t -i c36bb132a842)
step4 root@c36bb132a842:~/transformer# ./run_and_time.sh 1 | tee benchmark-date "+%F-%T".log

That I can start the translation workload.

Question 2.

After start the workload, I find that the workload use much of GPU memory, which results an error:

root@975f79997b6d:~/transformer# ./run_and_time.sh 1 | tee benchmark-date "+%F-%T".log
STARTING TIMING RUN AT 2018-05-16 02:38:30 PM
running benchmark with seed 1
INFO:tensorflow:Step 1/4: Downloading data from source
INFO:tensorflow:Already downloaded and extracted http://data.statmt.org/wmt17/translation-task/training-parallel-nc-v12.tgz.
INFO:tensorflow:Already downloaded and extracted http://www.statmt.org/wmt13/training-parallel-commoncrawl.tgz.
INFO:tensorflow:Already downloaded and extracted http://www.statmt.org/wmt13/training-parallel-europarl-v7.tgz.
INFO:tensorflow:Already downloaded and extracted http://data.statmt.org/wmt17/translation-task/dev.tgz.
INFO:tensorflow:Step 2/4: Creating subtokenizer and building vocabulary
INFO:tensorflow:Vocab file already exists (processed_data/vocab.ende.32768)
INFO:tensorflow:Initializing Subtokenizer from file processed_data/vocab.ende.32768.
INFO:tensorflow:Step 3/4: Compiling training and evaluation data
INFO:tensorflow:Compiling files with tag train.
INFO:tensorflow:Reading files /raw_data/training/news-commentary-v12.de-en.en and /raw_data/training/news-commentary-v12.de-en.de.
INFO:tensorflow:Reading files /raw_data/commoncrawl.de-en.en and /raw_data/commoncrawl.de-en.de.
INFO:tensorflow:Reading files /raw_data/training/europarl-v7.de-en.en and /raw_data/training/europarl-v7.de-en.de.
INFO:tensorflow:Compiling files with tag dev.
INFO:tensorflow:Reading files /raw_data/dev/newstest2013.en and /raw_data/dev/newstest2013.de.
INFO:tensorflow:Step 4/4: Preprocessing and saving data
INFO:tensorflow:Files with tag train already exist.
INFO:tensorflow:Files with tag dev already exist.
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00001-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00002-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00003-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00004-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00005-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00006-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00007-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00008-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00009-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00010-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00011-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00012-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00013-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00014-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00015-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00016-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00017-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00018-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00019-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00020-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00021-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00022-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00023-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00024-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00025-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00026-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00027-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00028-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00029-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00030-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00031-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00032-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00033-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00034-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00035-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00036-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00037-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00038-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00039-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00040-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00041-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00042-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00043-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00044-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00045-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00046-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00047-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00048-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00049-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00050-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00051-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00052-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00053-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00054-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00055-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00056-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00057-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00058-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00059-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00060-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00061-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00062-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00063-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00064-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00065-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00066-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00067-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00068-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00069-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00070-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00071-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00072-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00073-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00074-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00075-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00076-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00077-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00078-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00079-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00080-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00081-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00082-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00083-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00084-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00085-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00086-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00087-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00088-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00089-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00090-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00091-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00092-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00093-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00094-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00095-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00096-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00097-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00098-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00099-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00100-of-00100
INFO:tensorflow:Using default config.
INFO:tensorflow:Using config: {'_master': '', '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7fe98e188c88>, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_model_dir': 'model', '_device_fn': None, '_is_chief': True, '_global_id_in_cluster': 0, '_num_worker_replicas': 1, '_log_step_count_steps': 100, '_evaluation_master': '', '_service': None, '_tf_random_seed': None, '_keep_checkpoint_max': 5, '_num_ps_replicas': 0, '_task_id': 0, '_train_distribute': None, '_save_summary_steps': 100, '_keep_checkpoint_every_n_hours': 10000, '_session_config': None, '_task_type': 'worker'}
INFO:tensorflow:Calling model_fn.
/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gradients_impl.py:100: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.
  "Converting sparse IndexedSlices to a dense Tensor of unknown shape. "
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Graph was finalized.
2018-05-16 14:41:45.417676: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2018-05-16 14:41:57.526465: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1349] Found device 0 with properties:
name: Tesla K80 major: 3 minor: 7 memoryClockRate(GHz): 0.8235
pciBusID: 0000:85:00.0
totalMemory: 11.17GiB freeMemory: 11.10GiB
2018-05-16 14:41:57.928098: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1349] Found device 1 with properties:
name: Tesla K80 major: 3 minor: 7 memoryClockRate(GHz): 0.8235
pciBusID: 0000:86:00.0
totalMemory: 11.17GiB freeMemory: 11.10GiB
2018-05-16 14:41:58.337921: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1349] Found device 2 with properties:
name: Tesla K80 major: 3 minor: 7 memoryClockRate(GHz): 0.8235
pciBusID: 0000:8c:00.0
totalMemory: 11.17GiB freeMemory: 11.10GiB
2018-05-16 14:41:58.767041: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1349] Found device 3 with properties:
name: Tesla K80 major: 3 minor: 7 memoryClockRate(GHz): 0.8235
pciBusID: 0000:8d:00.0
totalMemory: 11.17GiB freeMemory: 11.10GiB
2018-05-16 14:41:58.769543: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1428] Adding visible gpu devices: 0, 1, 2, 3
2018-05-16 14:41:59.984340: I tensorflow/core/common_runtime/gpu/gpu_device.cc:916] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-05-16 14:41:59.984407: I tensorflow/core/common_runtime/gpu/gpu_device.cc:922]      0 1 2 3
2018-05-16 14:41:59.984425: I tensorflow/core/common_runtime/gpu/gpu_device.cc:935] 0:   N Y Y Y
2018-05-16 14:41:59.984442: I tensorflow/core/common_runtime/gpu/gpu_device.cc:935] 1:   Y N Y Y
2018-05-16 14:41:59.984480: I tensorflow/core/common_runtime/gpu/gpu_device.cc:935] 2:   Y Y N Y
2018-05-16 14:41:59.984497: I tensorflow/core/common_runtime/gpu/gpu_device.cc:935] 3:   Y Y Y N
2018-05-16 14:41:59.985810: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1046] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10762 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:85:00.0, compute capability: 3.7)
2018-05-16 14:42:00.120836: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1046] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 10762 MB memory) -> physical GPU (device: 1, name: Tesla K80, pci bus id: 0000:86:00.0, compute capability: 3.7)
2018-05-16 14:42:00.235081: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1046] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 10762 MB memory) -> physical GPU (device: 2, name: Tesla K80, pci bus id: 0000:8c:00.0, compute capability: 3.7)
2018-05-16 14:42:00.350005: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1046] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 10762 MB memory) -> physical GPU (device: 3, name: Tesla K80, pci bus id: 0000:8d:00.0, compute capability: 3.7)
INFO:tensorflow:Restoring parameters from model/model.ckpt-0
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Saving checkpoints for 0 into model/model.ckpt.
INFO:tensorflow:step = 0, loss = 9.890151
2018-05-16 14:46:51.976901: W tensorflow/core/common_runtime/bfc_allocator.cc:275] Allocator (GPU_0_bfc) ran out of memory trying to allocate 511.51MiB.  Current allocation summary follows.
2018-05-16 14:46:51.977166: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (256): 	Total Chunks: 183, Chunks in use: 181. 45.8KiB allocated for chunks. 45.2KiB in use in bin. 764B client-requested in use in bin.
2018-05-16 14:46:51.977216: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (512): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-05-16 14:46:51.977254: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (1024): 	Total Chunks: 1, Chunks in use: 1. 1.2KiB allocated for chunks. 1.2KiB in use in bin. 1.0KiB client-requested in use in bin.
2018-05-16 14:46:51.977291: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (2048): 	Total Chunks: 2, Chunks in use: 2. 5.2KiB allocated for chunks. 5.2KiB in use in bin. 4.0KiB client-requested in use in bin.
2018-05-16 14:46:51.977330: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (4096): 	Total Chunks: 228, Chunks in use: 228. 916.2KiB allocated for chunks. 916.2KiB in use in bin. 912.0KiB client-requested in use in bin.
2018-05-16 14:46:51.977369: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (8192): 	Total Chunks: 43, Chunks in use: 42. 661.5KiB allocated for chunks. 646.5KiB in use in bin. 638.8KiB client-requested in use in bin.
2018-05-16 14:46:51.977406: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (16384): 	Total Chunks: 38, Chunks in use: 38. 653.8KiB allocated for chunks. 653.8KiB in use in bin. 604.8KiB client-requested in use in bin.
2018-05-16 14:46:51.977444: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (32768): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-05-16 14:46:51.977486: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (65536): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-05-16 14:46:51.977523: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (131072): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-05-16 14:46:51.977560: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (262144): 	Total Chunks: 4, Chunks in use: 4. 1.17MiB allocated for chunks. 1.17MiB in use in bin. 952.8KiB client-requested in use in bin.
2018-05-16 14:46:51.977596: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (524288): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-05-16 14:46:51.977631: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (1048576): 	Total Chunks: 1, Chunks in use: 0. 1.95MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-05-16 14:46:51.977665: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (2097152): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-05-16 14:46:51.977704: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (4194304): 	Total Chunks: 219, Chunks in use: 216. 883.23MiB allocated for chunks. 871.23MiB in use in bin. 864.00MiB client-requested in use in bin.
2018-05-16 14:46:51.977740: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (8388608): 	Total Chunks: 199, Chunks in use: 195. 2.99GiB allocated for chunks. 2.95GiB in use in bin. 2.95GiB client-requested in use in bin.
2018-05-16 14:46:51.977777: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (16777216): 	Total Chunks: 131, Chunks in use: 131. 2.62GiB allocated for chunks. 2.62GiB in use in bin. 2.51GiB client-requested in use in bin.
2018-05-16 14:46:51.977812: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (33554432): 	Total Chunks: 36, Chunks in use: 36. 2.02GiB allocated for chunks. 2.02GiB in use in bin. 2.02GiB client-requested in use in bin.
2018-05-16 14:46:51.977847: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (67108864): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-05-16 14:46:51.977885: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (134217728): 	Total Chunks: 5, Chunks in use: 5. 658.42MiB allocated for chunks. 658.42MiB in use in bin. 658.41MiB client-requested in use in bin.
2018-05-16 14:46:51.977922: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (268435456): 	Total Chunks: 3, Chunks in use: 2. 1.36GiB allocated for chunks. 1023.03MiB in use in bin. 1023.03MiB client-requested in use in bin.
2018-05-16 14:46:51.977960: I tensorflow/core/common_runtime/bfc_allocator.cc:646] Bin for 511.51MiB was 256.00MiB, Chunk State:
2018-05-16 14:46:51.978005: I tensorflow/core/common_runtime/bfc_allocator.cc:652]   Size: 373.75MiB | Requested Size: 12B | in_use: 0, prev:   Size: 511.51MiB | Requested Size: 511.51MiB | in_use: 1
2018-05-16 14:46:51.978044: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2314700000 of size 1280
2018-05-16 14:46:51.978078: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2314700500 of size 4096
2018-05-16 14:46:51.978109: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2314701500 of size 4096
2018-05-16 14:46:51.978140: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2314702500 of size 4096
2018-05-16 14:46:51.978169: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2314703500 of size 4096
2018-05-16 14:46:51.978202: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2314704500 of size 16781312
2018-05-16 14:46:51.978232: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2315705500 of size 256
2018-05-16 14:46:51.978265: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2315705600 of size 4352
2018-05-16 14:46:51.978295: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2315706700 of size 4096
2018-05-16 14:46:51.978326: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2315707700 of size 4194304
2018-05-16 14:46:51.978358: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2315b07700 of size 4194304
2018-05-16 14:46:51.978388: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2315f07700 of size 8379904
2018-05-16 14:46:51.978418: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2316705500 of size 4096
2018-05-16 14:46:51.978449: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2316706500 of size 4096
2018-05-16 14:46:51.978479: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2316707500 of size 4096
2018-05-16 14:46:51.978509: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2316708500 of size 4194304
2018-05-16 14:46:51.978539: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2316b08500 of size 4096
2018-05-16 14:46:51.978570: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2316b09500 of size 4096
2018-05-16 14:46:51.978628: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2316b0a500 of size 4194304
2018-05-16 14:46:51.978660: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2316f0a500 of size 4096
2018-05-16 14:46:51.978690: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2316f0b500 of size 16777216
2018-05-16 14:46:51.978721: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2317f0b500 of size 4096
2018-05-16 14:46:51.978752: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2317f0c500 of size 4096
2018-05-16 14:46:51.978782: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2317f0d500 of size 4096
2018-05-16 14:46:51.978813: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2317f0e500 of size 4096
2018-05-16 14:46:51.978844: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2317f0f500 of size 4096
2018-05-16 14:46:51.978873: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2317f10500 of size 4096
2018-05-16 14:46:51.978904: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2317f11500 of size 4096
2018-05-16 14:46:51.978933: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2317f12500 of size 4096
2018-05-16 14:46:51.978964: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2317f13500 of size 4198400
2018-05-16 14:46:51.979000: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318314500 of size 4194304
2018-05-16 14:46:51.979031: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318714500 of size 4194304
2018-05-16 14:46:51.979061: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318b14500 of size 4194304
2018-05-16 14:46:51.979092: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f14500 of size 4096
2018-05-16 14:46:51.979122: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f15500 of size 4096
2018-05-16 14:46:51.979152: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f16500 of size 4096
2018-05-16 14:46:51.979182: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f17500 of size 4096
2018-05-16 14:46:51.979214: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f18500 of size 24576
2018-05-16 14:46:51.979245: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f1e500 of size 4096
2018-05-16 14:46:51.979276: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f1f500 of size 4096
2018-05-16 14:46:51.979305: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f20500 of size 4096
2018-05-16 14:46:51.979336: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f21500 of size 4096
2018-05-16 14:46:51.979367: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f22500 of size 4096
2018-05-16 14:46:51.979397: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f23500 of size 4096
2018-05-16 14:46:51.979427: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f24500 of size 4096
2018-05-16 14:46:51.979458: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f25500 of size 4096
2018-05-16 14:46:51.979488: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f26500 of size 4096
2018-05-16 14:46:51.979517: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f27500 of size 4096
2018-05-16 14:46:51.979548: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f28500 of size 4096
2018-05-16 14:46:51.979578: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f29500 of size 4096
2018-05-16 14:46:51.979608: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f2a500 of size 4096
2018-05-16 14:46:51.979638: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f2b500 of size 4096
2018-05-16 14:46:51.979669: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f2c500 of size 16384
2018-05-16 14:46:51.979699: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f30500 of size 4096
2018-05-16 14:46:51.979730: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f31500 of size 4096
2018-05-16 14:46:51.979761: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f32500 of size 4096
2018-05-16 14:46:51.979792: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f33500 of size 4096
2018-05-16 14:46:51.979825: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f34500 of size 4096
2018-05-16 14:46:51.979855: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f35500 of size 4096
2018-05-16 14:46:51.979885: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f36500 of size 4096
2018-05-16 14:46:51.979916: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f37500 of size 4096
2018-05-16 14:46:51.979947: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f38500 of size 4096
2018-05-16 14:46:51.979978: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f39500 of size 4096
2018-05-16 14:46:51.980007: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f3a500 of size 4096
2018-05-16 14:46:51.980038: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f3b500 of size 4096
2018-05-16 14:46:51.980068: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f3c500 of size 4096
2018-05-16 14:46:51.980098: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f3d500 of size 4096
2018-05-16 14:46:51.980128: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f3e500 of size 4096
2018-05-16 14:46:51.980157: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f3f500 of size 4096
2018-05-16 14:46:51.980189: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f40500 of size 4096
2018-05-16 14:46:51.980220: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f41500 of size 4096
2018-05-16 14:46:51.980250: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f42500 of size 4096
2018-05-16 14:46:51.980281: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f43500 of size 16384
2018-05-16 14:46:51.980312: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f47500 of size 20480
2018-05-16 14:46:51.980343: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f4c500 of size 4096
2018-05-16 14:46:51.980374: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f4d500 of size 4096
2018-05-16 14:46:51.980405: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f4e500 of size 4096
2018-05-16 14:46:51.980436: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f4f500 of size 4096
2018-05-16 14:46:51.980467: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f50500 of size 4096
2018-05-16 14:46:51.980497: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f51500 of size 4096
2018-05-16 14:46:51.980527: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f52500 of size 4096
2018-05-16 14:46:51.980557: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f53500 of size 4096
2018-05-16 14:46:51.980587: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f54500 of size 4096
2018-05-16 14:46:51.980617: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f55500 of size 4096
2018-05-16 14:46:51.980649: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f56500 of size 4096
2018-05-16 14:46:51.980679: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f57500 of size 4096
2018-05-16 14:46:51.980711: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f58500 of size 20480
2018-05-16 14:46:51.980744: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f5d500 of size 16384
2018-05-16 14:46:51.980772: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f61500 of size 4096
2018-05-16 14:46:51.980803: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f62500 of size 4096
2018-05-16 14:46:51.980834: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f63500 of size 16384
2018-05-16 14:46:51.980865: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f67500 of size 4096
2018-05-16 14:46:51.980895: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f68500 of size 4096
2018-05-16 14:46:51.980935: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f69500 of size 4096
2018-05-16 14:46:51.980986: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f6a500 of size 4096
2018-05-16 14:46:51.981020: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f6b500 of size 4352
2018-05-16 14:46:51.981050: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f6c600 of size 4096
2018-05-16 14:46:51.981080: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f6d600 of size 4096
2018-05-16 14:46:51.981112: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f6e600 of size 4096
2018-05-16 14:46:51.981143: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f6f600 of size 16384
2018-05-16 14:46:51.981173: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f73600 of size 4096
2018-05-16 14:46:51.981203: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f74600 of size 4096
2018-05-16 14:46:51.981233: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f75600 of size 4096
2018-05-16 14:46:51.981263: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f76600 of size 4096
2018-05-16 14:46:51.981292: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f77600 of size 4096
2018-05-16 14:46:51.981324: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f78600 of size 20480
2018-05-16 14:46:51.981355: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f7d600 of size 4096
2018-05-16 14:46:51.981385: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f7e600 of size 4096
2018-05-16 14:46:51.981415: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f7f600 of size 4096
2018-05-16 14:46:51.981446: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f80600 of size 4096
2018-05-16 14:46:51.981476: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f81600 of size 4096
2018-05-16 14:46:51.981506: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f82600 of size 4096
2018-05-16 14:46:51.981536: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f83600 of size 4096
2018-05-16 14:46:51.981566: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f84600 of size 4096
2018-05-16 14:46:51.981597: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f85600 of size 4096
2018-05-16 14:46:51.981629: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f86600 of size 4096
2018-05-16 14:46:51.981659: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f87600 of size 4096
2018-05-16 14:46:51.981689: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f88600 of size 4096
2018-05-16 14:46:51.981719: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f89600 of size 4096
2018-05-16 14:46:51.981752: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f8a600 of size 24576
2018-05-16 14:46:51.981783: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f90600 of size 4096
2018-05-16 14:46:51.981814: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f91600 of size 4096
2018-05-16 14:46:51.981843: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f92600 of size 4096
2018-05-16 14:46:51.981873: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f93600 of size 4096
2018-05-16 14:46:51.981904: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f94600 of size 4096
2018-05-16 14:46:51.981934: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f95600 of size 4096
2018-05-16 14:46:51.981964: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f96600 of size 4096
2018-05-16 14:46:51.981994: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f97600 of size 4096
2018-05-16 14:46:51.982022: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f98600 of size 4096
2018-05-16 14:46:51.982053: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f99600 of size 4096
2018-05-16 14:46:51.982083: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f9a600 of size 4096
2018-05-16 14:46:51.982112: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f9b600 of size 4096
2018-05-16 14:46:51.982140: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f9c600 of size 4096
2018-05-16 14:46:51.982170: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f9d600 of size 4096
2018-05-16 14:46:51.982201: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f9e600 of size 4096
2018-05-16 14:46:51.982230: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f9f600 of size 4096
2018-05-16 14:46:51.982260: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fa0600 of size 4096
2018-05-16 14:46:51.982291: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fa1600 of size 4096
2018-05-16 14:46:51.982321: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fa2600 of size 4096
2018-05-16 14:46:51.982352: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fa3600 of size 4096
2018-05-16 14:46:51.982382: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fa4600 of size 256
2018-05-16 14:46:51.982414: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fa4700 of size 7936
2018-05-16 14:46:51.982445: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fa6600 of size 4096
2018-05-16 14:46:51.982474: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fa7600 of size 4096
2018-05-16 14:46:51.982504: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fa8600 of size 4096
2018-05-16 14:46:51.982534: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fa9600 of size 4096
2018-05-16 14:46:51.982565: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318faa600 of size 16384
2018-05-16 14:46:51.982607: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fae600 of size 256
2018-05-16 14:46:51.982642: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fae700 of size 256
2018-05-16 14:46:51.982673: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fae800 of size 256
2018-05-16 14:46:51.982702: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fae900 of size 256
2018-05-16 14:46:51.982733: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318faea00 of size 256
2018-05-16 14:46:51.982764: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318faeb00 of size 256
2018-05-16 14:46:51.982794: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318faec00 of size 256
2018-05-16 14:46:51.982825: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318faed00 of size 256
2018-05-16 14:46:51.982854: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318faee00 of size 256
2018-05-16 14:46:51.982884: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318faef00 of size 256
2018-05-16 14:46:51.982915: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318faf000 of size 256
2018-05-16 14:46:51.982945: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318faf100 of size 256
2018-05-16 14:46:51.982976: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318faf200 of size 256
2018-05-16 14:46:51.983007: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318faf300 of size 256
2018-05-16 14:46:51.983037: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318faf400 of size 256
2018-05-16 14:46:51.983068: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318faf500 of size 256
2018-05-16 14:46:51.983098: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318faf600 of size 4096
2018-05-16 14:46:51.983129: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fb0600 of size 4096
2018-05-16 14:46:51.983159: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fb1600 of size 4096
2018-05-16 14:46:51.983189: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fb2600 of size 4096
2018-05-16 14:46:51.983218: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fb3600 of size 256
2018-05-16 14:46:51.983248: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fb3700 of size 256
2018-05-16 14:46:51.983278: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fb3800 of size 256
2018-05-16 14:46:51.983310: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fb3900 of size 256
2018-05-16 14:46:51.983341: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fb3a00 of size 256
2018-05-16 14:46:51.983372: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fb3b00 of size 2816
2018-05-16 14:46:51.983403: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fb4600 of size 4096
2018-05-16 14:46:51.983433: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fb5600 of size 4096
2018-05-16 14:46:51.983463: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fb6600 of size 4096
2018-05-16 14:46:51.983492: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fb7600 of size 4096
2018-05-16 14:46:51.983521: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fb8600 of size 4096
2018-05-16 14:46:51.983551: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fb9600 of size 4096
2018-05-16 14:46:51.983580: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fba600 of size 256
2018-05-16 14:46:51.983610: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fba700 of size 256
2018-05-16 14:46:51.983641: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fba800 of size 256
2018-05-16 14:46:51.983671: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fba900 of size 256
2018-05-16 14:46:51.983701: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fbaa00 of size 256
2018-05-16 14:46:51.983730: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fbab00 of size 256
2018-05-16 14:46:51.983760: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fbac00 of size 256
2018-05-16 14:46:51.983790: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fbad00 of size 256
2018-05-16 14:46:51.983819: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fbae00 of size 256
2018-05-16 14:46:51.983849: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fbaf00 of size 256
2018-05-16 14:46:51.983885: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fbb000 of size 256
2018-05-16 14:46:51.983915: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fbb100 of size 256
2018-05-16 14:46:51.983945: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fbb200 of size 256
2018-05-16 14:46:51.983989: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fbb300 of size 256
2018-05-16 14:46:51.984021: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fbb400 of size 256
2018-05-16 14:46:51.984051: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fbb500 of size 256
2018-05-16 14:46:51.984082: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fbb600 of size 4096
2018-05-16 14:46:51.984112: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fbc600 of size 4096
2018-05-16 14:46:51.984142: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fbd600 of size 256
2018-05-16 14:46:51.984173: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fbd700 of size 256
2018-05-16 14:46:51.984203: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fbd800 of size 256
2018-05-16 14:46:51.984233: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fbd900 of size 256
2018-05-16 14:46:51.984263: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fbda00 of size 256
2018-05-16 14:46:51.984295: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fbdb00 of size 256
2018-05-16 14:46:51.984324: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fbdc00 of size 256
2018-05-16 14:46:51.984355: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fbdd00 of size 256
2018-05-16 14:46:51.984386: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fbde00 of size 256
2018-05-16 14:46:51.984416: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fbdf00 of size 256
2018-05-16 14:46:51.984445: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fbe000 of size 256
2018-05-16 14:46:51.984477: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fbe100 of size 256
2018-05-16 14:46:51.984507: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fbe200 of size 256
2018-05-16 14:46:51.984537: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fbe300 of size 256
2018-05-16 14:46:51.984567: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fbe400 of size 256
2018-05-16 14:46:51.984597: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fbe500 of size 256
2018-05-16 14:46:51.984628: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fbe600 of size 4096
2018-05-16 14:46:51.984659: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fbf600 of size 16384
2018-05-16 14:46:51.984689: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc3600 of size 256
2018-05-16 14:46:51.984719: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc3700 of size 256
2018-05-16 14:46:51.984749: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc3800 of size 256
2018-05-16 14:46:51.984780: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc3900 of size 256
2018-05-16 14:46:51.984810: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc3a00 of size 256
2018-05-16 14:46:51.984839: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc3b00 of size 256
2018-05-16 14:46:51.984869: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc3c00 of size 256
2018-05-16 14:46:51.984900: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc3d00 of size 256
2018-05-16 14:46:51.984929: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc3e00 of size 256
2018-05-16 14:46:51.984960: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc3f00 of size 256
2018-05-16 14:46:51.984990: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc4000 of size 256
2018-05-16 14:46:51.985020: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc4100 of size 256
2018-05-16 14:46:51.985050: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc4200 of size 256
2018-05-16 14:46:51.985080: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc4300 of size 256
2018-05-16 14:46:51.985111: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc4400 of size 256
2018-05-16 14:46:51.985140: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc4500 of size 256
2018-05-16 14:46:51.985170: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc4600 of size 4096
2018-05-16 14:46:51.985201: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc5600 of size 256
2018-05-16 14:46:51.985230: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc5700 of size 256
2018-05-16 14:46:51.985260: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc5800 of size 256
2018-05-16 14:46:51.985291: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc5900 of size 256
2018-05-16 14:46:51.985320: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc5a00 of size 256
2018-05-16 14:46:51.985351: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc5b00 of size 256
2018-05-16 14:46:51.985381: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc5c00 of size 2560
2018-05-16 14:46:51.985413: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc6600 of size 4096
2018-05-16 14:46:51.985443: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc7600 of size 256
2018-05-16 14:46:51.985473: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc7700 of size 256
2018-05-16 14:46:51.985503: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc7800 of size 256
2018-05-16 14:46:51.985534: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc7900 of size 256
2018-05-16 14:46:51.985564: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc7a00 of size 256
2018-05-16 14:46:51.985595: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc7b00 of size 256
2018-05-16 14:46:51.985625: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc7c00 of size 256
2018-05-16 14:46:51.985654: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc7d00 of size 256
2018-05-16 14:46:51.985685: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc7e00 of size 256
2018-05-16 14:46:51.985716: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc7f00 of size 256
2018-05-16 14:46:51.985746: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc8000 of size 256
2018-05-16 14:46:51.985777: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc8100 of size 256
2018-05-16 14:46:51.985808: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc8200 of size 256
2018-05-16 14:46:51.985839: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc8300 of size 256
2018-05-16 14:46:51.985871: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc8400 of size 256
2018-05-16 14:46:51.985901: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc8500 of size 256
2018-05-16 14:46:51.985931: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc8600 of size 4096
2018-05-16 14:46:51.985962: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc9600 of size 4096
2018-05-16 14:46:51.985992: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fca600 of size 20480
2018-05-16 14:46:51.986021: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fcf600 of size 4096
2018-05-16 14:46:51.986052: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd0600 of size 256
2018-05-16 14:46:51.986083: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd0700 of size 256
2018-05-16 14:46:51.986113: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd0800 of size 256
2018-05-16 14:46:51.986143: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free  at 0x2318fd0900 of size 256
2018-05-16 14:46:51.986173: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd0a00 of size 256
2018-05-16 14:46:51.986203: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd0b00 of size 256
2018-05-16 14:46:51.986233: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd0c00 of size 256
2018-05-16 14:46:51.986263: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd0d00 of size 256
2018-05-16 14:46:51.986294: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd0e00 of size 256
2018-05-16 14:46:51.986324: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd0f00 of size 256
2018-05-16 14:46:51.986354: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd1000 of size 256
2018-05-16 14:46:51.986384: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd1100 of size 256
2018-05-16 14:46:51.986413: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd1200 of size 256
2018-05-16 14:46:51.986443: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd1300 of size 256
2018-05-16 14:46:51.986473: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd1400 of size 256
2018-05-16 14:46:51.986502: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd1500 of size 256
2018-05-16 14:46:51.986531: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd1600 of size 256
2018-05-16 14:46:51.986561: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd1700 of size 256
2018-05-16 14:46:51.986590: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd1800 of size 256
2018-05-16 14:46:51.986630: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd1900 of size 256
2018-05-16 14:46:51.986661: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd1a00 of size 256
2018-05-16 14:46:51.986691: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd1b00 of size 256
2018-05-16 14:46:51.986721: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd1c00 of size 256
2018-05-16 14:46:51.986752: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd1d00 of size 256
2018-05-16 14:46:51.986782: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd1e00 of size 256
2018-05-16 14:46:51.986812: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd1f00 of size 256
2018-05-16 14:46:51.986842: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd2000 of size 256
2018-05-16 14:46:51.986871: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd2100 of size 256
2018-05-16 14:46:51.986901: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd2200 of size 256
2018-05-16 14:46:51.986931: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd2300 of size 256
2018-05-16 14:46:51.986962: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd2400 of size 256
2018-05-16 14:46:51.986993: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd2500 of size 256
2018-05-16 14:46:51.987022: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd2600 of size 4096
2018-05-16 14:46:51.987056: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd3600 of size 4096
2018-05-16 14:46:51.987086: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd4600 of size 256
2018-05-16 14:46:51.987115: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd4700 of size 256
2018-05-16 14:46:51.987151: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd4800 of size 256
2018-05-16 14:46:51.987182: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd4900 of size 256
2018-05-16 14:46:51.987214: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd4a00 of size 256
2018-05-16 14:46:51.987244: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd4b00 of size 256
2018-05-16 14:46:51.987274: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd4c00 of size 256
2018-05-16 14:46:51.987305: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd4d00 of size 256
2018-05-16 14:46:51.987335: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd4e00 of size 256
2018-05-16 14:46:51.987365: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd4f00 of size 256
2018-05-16 14:46:51.987395: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd5000 of size 256
2018-05-16 14:46:51.987425: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd5100 of size 256
2018-05-16 14:46:51.987455: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd5200 of size 256
2018-05-16 14:46:51.987484: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd5300 of size 256
2018-05-16 14:46:51.987515: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd5400 of size 256
2018-05-16 14:46:51.987545: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd5500 of size 256
2018-05-16 14:46:51.987578: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd5600 of size 16384
2018-05-16 14:46:51.987608: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd9600 of size 4096
2018-05-16 14:46:51.987638: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fda600 of size 256
2018-05-16 14:46:51.987669: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fda700 of size 256
2018-05-16 14:46:51.987699: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fda800 of size 256
2018-05-16 14:46:51.987729: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fda900 of size 256
2018-05-16 14:46:51.987759: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fdaa00 of size 256
2018-05-16 14:46:51.987793: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fdab00 of size 256
2018-05-16 14:46:51.987824: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fdac00 of size 256
2018-05-16 14:46:51.987854: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fdad00 of size 256
2018-05-16 14:46:51.987883: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fdae00 of size 256
2018-05-16 14:46:51.987913: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fdaf00 of size 256
2018-05-16 14:46:51.987944: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fdb000 of size 256
2018-05-16 14:46:51.987974: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fdb100 of size 256
2018-05-16 14:46:51.988004: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fdb200 of size 256
2018-05-16 14:46:51.988033: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fdb300 of size 256
2018-05-16 14:46:51.988063: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fdb400 of size 256
2018-05-16 14:46:51.988093: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fdb500 of size 256
2018-05-16 14:46:51.988123: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fdb600 of size 256
2018-05-16 14:46:51.988154: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fdb700 of size 256
2018-05-16 14:46:51.988183: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fdb800 of size 256
2018-05-16 14:46:51.988213: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fdb900 of size 256
2018-05-16 14:46:51.988243: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fdba00 of size 256
2018-05-16 14:46:51.988274: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fdbb00 of size 256
2018-05-16 14:46:51.988303: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fdbc00 of size 256
2018-05-16 14:46:51.988335: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fdbd00 of size 256
2018-05-16 14:46:51.988365: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fdbe00 of size 256
2018-05-16 14:46:51.988395: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fdbf00 of size 256
2018-05-16 14:46:51.988426: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fdc000 of size 256
2018-05-16 14:46:51.988456: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fdc100 of size 256
2018-05-16 14:46:51.988485: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fdc200 of size 256
2018-05-16 14:46:51.988514: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fdc300 of size 256
2018-05-16 14:46:51.988544: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fdc400 of size 256
2018-05-16 14:46:51.988575: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fdc500 of size 256
2018-05-16 14:46:51.988605: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fdc600 of size 4096
2018-05-16 14:46:51.988637: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fdd600 of size 4096
2018-05-16 14:46:51.988667: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fde600 of size 16384
2018-05-16 14:46:51.988698: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fe2600 of size 7544576
2018-05-16 14:46:51.988728: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2319714500 of size 4194304
2018-05-16 14:46:51.988757: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2319b14500 of size 4096
2018-05-16 14:46:51.988788: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2319b15500 of size 4096
2018-05-16 14:46:51.988818: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2319b16500 of size 4194304
2018-05-16 14:46:51.988849: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2319f16500 of size 4096
2018-05-16 14:46:51.988879: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2319f17500 of size 4096
2018-05-16 14:46:51.988908: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2319f18500 of size 4096
2018-05-16 14:46:51.988938: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2319f19500 of size 4096
2018-05-16 14:46:51.988970: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2319f1a500 of size 4198400
2018-05-16 14:46:51.989000: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231a31b500 of size 4194304
2018-05-16 14:46:51.989030: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231a71b500 of size 16777216
2018-05-16 14:46:51.989060: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231b71b500 of size 4096
2018-05-16 14:46:51.989090: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231b71c500 of size 4194304
2018-05-16 14:46:51.989119: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231bb1c500 of size 4096
2018-05-16 14:46:51.989149: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231bb1d500 of size 4194304
2018-05-16 14:46:51.989179: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231bf1d500 of size 4194304
2018-05-16 14:46:51.989208: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231c31d500 of size 4194304
2018-05-16 14:46:51.989239: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231c71d500 of size 16384
2018-05-16 14:46:51.989268: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231c721500 of size 4194304
2018-05-16 14:46:51.989299: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231cb21500 of size 4194304
2018-05-16 14:46:51.989328: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231cf21500 of size 4194304
2018-05-16 14:46:51.989359: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231d321500 of size 4096
2018-05-16 14:46:51.989389: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231d322500 of size 4096
2018-05-16 14:46:51.989418: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231d323500 of size 4194304
2018-05-16 14:46:51.989448: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231d723500 of size 4096
2018-05-16 14:46:51.989479: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231d724500 of size 4194304
2018-05-16 14:46:51.989508: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231db24500 of size 4194304
2018-05-16 14:46:51.989537: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231df24500 of size 4194304
2018-05-16 14:46:51.989566: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231e324500 of size 4194304
2018-05-16 14:46:51.989597: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231e724500 of size 4194304
2018-05-16 14:46:51.989627: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231eb24500 of size 4096
2018-05-16 14:46:51.989657: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231eb25500 of size 16384
2018-05-16 14:46:51.989687: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231eb29500 of size 16777216
2018-05-16 14:46:51.989717: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231fb29500 of size 16777216
2018-05-16 14:46:51.989746: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2320b29500 of size 4096
2018-05-16 14:46:51.989777: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2320b2a500 of size 4096
2018-05-16 14:46:51.989806: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2320b2b500 of size 256
2018-05-16 14:46:51.989837: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2320b2b600 of size 16781056
2018-05-16 14:46:51.989867: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2321b2c500 of size 4194304
2018-05-16 14:46:51.989896: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2321f2c500 of size 4194304
2018-05-16 14:46:51.989927: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232232c500 of size 4194304
2018-05-16 14:46:51.989957: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232272c500 of size 16384
2018-05-16 14:46:51.989986: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2322730500 of size 4194304
2018-05-16 14:46:51.990017: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2322b30500 of size 4194304
2018-05-16 14:46:51.990047: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2322f30500 of size 4194304
2018-05-16 14:46:51.990076: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2323330500 of size 4194304
2018-05-16 14:46:51.990106: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2323730500 of size 4194304
2018-05-16 14:46:51.990137: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2323b30500 of size 4194304
2018-05-16 14:46:51.990166: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2323f30500 of size 4194304
2018-05-16 14:46:51.990196: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324330500 of size 4194304
2018-05-16 14:46:51.990226: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324730500 of size 4194304
2018-05-16 14:46:51.990255: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324b30500 of size 4194304
2018-05-16 14:46:51.990284: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324f30500 of size 16777216
2018-05-16 14:46:51.990314: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2325f30500 of size 16777216
2018-05-16 14:46:51.990344: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2326f30500 of size 4096
2018-05-16 14:46:51.990374: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2326f31500 of size 4096
2018-05-16 14:46:51.990404: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2326f32500 of size 4096
2018-05-16 14:46:51.990434: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2326f33500 of size 4096
2018-05-16 14:46:51.990464: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2326f34500 of size 16384
2018-05-16 14:46:51.990493: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2326f38500 of size 16777216
2018-05-16 14:46:51.990523: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2327f38500 of size 16777216
2018-05-16 14:46:51.990554: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2328f38500 of size 4096
2018-05-16 14:46:51.990583: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2328f39500 of size 4096
2018-05-16 14:46:51.990636: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2328f3a500 of size 4096
2018-05-16 14:46:51.990668: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2328f3b500 of size 4096
2018-05-16 14:46:51.990698: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2328f3c500 of size 4096
2018-05-16 14:46:51.990727: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2328f3d500 of size 4096
2018-05-16 14:46:51.990759: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2328f3e500 of size 4194304
2018-05-16 14:46:51.990789: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232933e500 of size 4194304
2018-05-16 14:46:51.990819: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232973e500 of size 4194304
2018-05-16 14:46:51.990850: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2329b3e500 of size 4194304
2018-05-16 14:46:51.990880: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2329f3e500 of size 4194304
2018-05-16 14:46:51.990910: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232a33e500 of size 4194304
2018-05-16 14:46:51.990940: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232a73e500 of size 4194304
2018-05-16 14:46:51.990991: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232ab3e500 of size 4206592
2018-05-16 14:46:51.991021: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232af41500 of size 4096
2018-05-16 14:46:51.991050: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232af42500 of size 4194304
2018-05-16 14:46:51.991081: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232b342500 of size 4194304
2018-05-16 14:46:51.991110: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232b742500 of size 4194304
2018-05-16 14:46:51.991140: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232bb42500 of size 4194304
2018-05-16 14:46:51.991169: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232bf42500 of size 16384
2018-05-16 14:46:51.991198: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232bf46500 of size 4096
2018-05-16 14:46:51.991228: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232bf47500 of size 4096
2018-05-16 14:46:51.991257: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232bf48500 of size 16384
2018-05-16 14:46:51.991287: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232bf4c500 of size 16384
2018-05-16 14:46:51.991317: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232bf50500 of size 4096
2018-05-16 14:46:51.991347: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232bf51500 of size 4096
2018-05-16 14:46:51.991375: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232bf52500 of size 4096
2018-05-16 14:46:51.991404: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232bf53500 of size 4096
2018-05-16 14:46:51.991433: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232bf54500 of size 25096192
2018-05-16 14:46:51.991463: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232d743500 of size 4096
2018-05-16 14:46:51.991492: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232d744500 of size 16785408
2018-05-16 14:46:51.991523: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232e746500 of size 16793600
2018-05-16 14:46:51.991553: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232f74a500 of size 16793600
2018-05-16 14:46:51.991582: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x233074e500 of size 16781312
2018-05-16 14:46:51.991612: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x233174f500 of size 4096
2018-05-16 14:46:51.991641: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2331750500 of size 4096
2018-05-16 14:46:51.991675: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2331751500 of size 4096
2018-05-16 14:46:51.991705: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2331752500 of size 4096
2018-05-16 14:46:51.991734: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2331753500 of size 4096
2018-05-16 14:46:51.991762: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2331754500 of size 4198400
2018-05-16 14:46:51.991793: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2331b55500 of size 4096
2018-05-16 14:46:51.991823: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2331b56500 of size 16777216
2018-05-16 14:46:51.991853: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2332b56500 of size 16793600
2018-05-16 14:46:51.991881: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2333b5a500 of size 4096
2018-05-16 14:46:51.991910: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2333b5b500 of size 16777216
2018-05-16 14:46:51.991939: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2334b5b500 of size 138076160
2018-05-16 14:46:51.991969: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x233cf09500 of size 16777216
2018-05-16 14:46:51.992000: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x233df09500 of size 16777216
2018-05-16 14:46:51.992031: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x233ef09500 of size 4194304
2018-05-16 14:46:51.992060: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x233f309500 of size 16777216
2018-05-16 14:46:51.992087: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2340309500 of size 16777216
2018-05-16 14:46:51.992118: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2341309500 of size 16777216
2018-05-16 14:46:51.992147: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2342309500 of size 16777216
2018-05-16 14:46:51.992177: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2343309500 of size 33210368
2018-05-16 14:46:51.992207: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23452b5500 of size 4194304
2018-05-16 14:46:51.992237: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23456b5500 of size 4194304
2018-05-16 14:46:51.992266: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2345ab5500 of size 4194304
2018-05-16 14:46:51.992296: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2345eb5500 of size 4194304
2018-05-16 14:46:51.992325: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23462b5500 of size 16777216
2018-05-16 14:46:51.992357: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23472b5500 of size 20987904
2018-05-16 14:46:51.992387: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23486b9500 of size 16384
2018-05-16 14:46:51.992415: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23486bd500 of size 4096
2018-05-16 14:46:51.992444: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23486be500 of size 4096
2018-05-16 14:46:51.992473: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23486bf500 of size 4096
2018-05-16 14:46:51.992503: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23486c0500 of size 4096
2018-05-16 14:46:51.992533: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23486c1500 of size 16777216
2018-05-16 14:46:51.992562: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23496c1500 of size 16777216
2018-05-16 14:46:51.992591: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x234a6c1500 of size 4096
2018-05-16 14:46:51.992621: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x234a6c2500 of size 4096
2018-05-16 14:46:51.992650: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x234a6c3500 of size 4096
2018-05-16 14:46:51.992679: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x234a6c4500 of size 4096
2018-05-16 14:46:51.992707: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x234a6c5500 of size 4096
2018-05-16 14:46:51.992737: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x234a6c6500 of size 4096
2018-05-16 14:46:51.992766: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x234a6c7500 of size 4194304
2018-05-16 14:46:51.992795: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x234aac7500 of size 4194304
2018-05-16 14:46:51.992824: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x234aec7500 of size 4194304
2018-05-16 14:46:51.992854: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x234b2c7500 of size 4194304
2018-05-16 14:46:51.992883: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x234b6c7500 of size 4194304
2018-05-16 14:46:51.992912: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x234bac7500 of size 4194304
2018-05-16 14:46:51.992941: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x234bec7500 of size 4194304
2018-05-16 14:46:51.992970: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x234c2c7500 of size 4194304
2018-05-16 14:46:51.992999: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x234c6c7500 of size 4096
2018-05-16 14:46:51.993028: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x234c6c8500 of size 4096
2018-05-16 14:46:51.993058: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x234c6c9500 of size 4096
2018-05-16 14:46:51.993087: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x234c6ca500 of size 4096
2018-05-16 14:46:51.993117: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x234c6cb500 of size 16777216
2018-05-16 14:46:51.993147: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x234d6cb500 of size 16384
2018-05-16 14:46:51.993176: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x234d6cf500 of size 4194304
2018-05-16 14:46:51.993205: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x234dacf500 of size 4194304
2018-05-16 14:46:51.993235: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x234decf500 of size 4194304
2018-05-16 14:46:51.993265: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x234e2cf500 of size 4194304
2018-05-16 14:46:51.993294: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x234e6cf500 of size 16793600
2018-05-16 14:46:51.993324: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x234f6d3500 of size 16793600
2018-05-16 14:46:51.993354: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23506d7500 of size 4096
2018-05-16 14:46:51.993383: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23506d8500 of size 4096
2018-05-16 14:46:51.993412: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23506d9500 of size 4096
2018-05-16 14:46:51.993440: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23506da500 of size 4194304
2018-05-16 14:46:51.993470: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2350ada500 of size 4194304
2018-05-16 14:46:51.993499: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2350eda500 of size 4194304
2018-05-16 14:46:51.993529: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23512da500 of size 4194304
2018-05-16 14:46:51.993558: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23516da500 of size 4194304
2018-05-16 14:46:51.993586: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2351ada500 of size 4194304
2018-05-16 14:46:51.993615: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2351eda500 of size 4194304
2018-05-16 14:46:51.993644: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23522da500 of size 4096
2018-05-16 14:46:51.993675: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23522db500 of size 16384
2018-05-16 14:46:51.993703: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23522df500 of size 16384
2018-05-16 14:46:51.993733: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23522e3500 of size 4096
2018-05-16 14:46:51.993762: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23522e4500 of size 4096
2018-05-16 14:46:51.993792: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23522e5500 of size 4096
2018-05-16 14:46:51.993821: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23522e6500 of size 4096
2018-05-16 14:46:51.993852: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23522e7500 of size 16384
2018-05-16 14:46:51.993881: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23522eb500 of size 16384
2018-05-16 14:46:51.993911: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23522ef500 of size 16384
2018-05-16 14:46:51.993939: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23522f3500 of size 16384
2018-05-16 14:46:51.993968: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23522f7500 of size 16384
2018-05-16 14:46:51.993997: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23522fb500 of size 4096
2018-05-16 14:46:51.994027: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23522fc500 of size 16384
2018-05-16 14:46:51.994056: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2352300500 of size 16384
2018-05-16 14:46:51.994085: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2352304500 of size 16384
2018-05-16 14:46:51.994116: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2352308500 of size 4096
2018-05-16 14:46:51.994145: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2352309500 of size 4096
2018-05-16 14:46:51.994174: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x235230a500 of size 4096
2018-05-16 14:46:51.994204: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x235230b500 of size 16384
2018-05-16 14:46:51.994234: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x235230f500 of size 4096
2018-05-16 14:46:51.994264: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2352310500 of size 269824
2018-05-16 14:46:51.994292: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2352352300 of size 269824
2018-05-16 14:46:51.994322: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free  at 0x2352394100 of size 256
2018-05-16 14:46:51.994352: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2352394200 of size 256
2018-05-16 14:46:51.994381: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2352394300 of size 256
2018-05-16 14:46:51.994410: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2352394400 of size 256
2018-05-16 14:46:51.994440: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2352394500 of size 256
2018-05-16 14:46:51.994468: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2352394600 of size 256
2018-05-16 14:46:51.994498: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2352394700 of size 256
2018-05-16 14:46:51.994527: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2352394800 of size 256
2018-05-16 14:46:51.994557: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free  at 0x2352394900 of size 15360
2018-05-16 14:46:51.994586: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2352398500 of size 16128
2018-05-16 14:46:51.994627: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x235239c400 of size 30464
2018-05-16 14:46:51.994657: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23523a3b00 of size 16128
2018-05-16 14:46:51.994686: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23523a7a00 of size 16128
2018-05-16 14:46:51.994717: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23523ab900 of size 16128
2018-05-16 14:46:51.994746: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23523af800 of size 16128
2018-05-16 14:46:51.994776: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23523b3700 of size 268800
2018-05-16 14:46:51.994806: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23523f5100 of size 16128
2018-05-16 14:46:51.994837: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23523f9000 of size 13568
2018-05-16 14:46:51.994866: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23523fc500 of size 13568
2018-05-16 14:46:51.994896: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23523ffa00 of size 417792
2018-05-16 14:46:51.994925: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2352465a00 of size 13568
2018-05-16 14:46:51.994954: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2352468f00 of size 13568
2018-05-16 14:46:51.994984: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x235246c400 of size 13568
2018-05-16 14:46:51.995013: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x235246f900 of size 13568
2018-05-16 14:46:51.995043: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2352472e00 of size 16128
2018-05-16 14:46:51.995072: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2352476d00 of size 16128
2018-05-16 14:46:51.995103: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x235247ac00 of size 16128
2018-05-16 14:46:51.995132: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x235247eb00 of size 16128
2018-05-16 14:46:51.995161: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2352482a00 of size 16128
2018-05-16 14:46:51.995191: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2352486900 of size 16128
2018-05-16 14:46:51.995221: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x235248a800 of size 16128
2018-05-16 14:46:51.995250: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x235248e700 of size 16128
2018-05-16 14:46:51.995281: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2352492600 of size 16128
2018-05-16 14:46:51.995311: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2352496500 of size 16128
2018-05-16 14:46:51.995339: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x235249a400 of size 16128
2018-05-16 14:46:51.995369: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x235249e300 of size 16128
2018-05-16 14:46:51.995398: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23524a2200 of size 16128
2018-05-16 14:46:51.995427: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23524a6100 of size 16128
2018-05-16 14:46:51.995460: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23524aa000 of size 16128
2018-05-16 14:46:51.995489: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23524adf00 of size 16128
2018-05-16 14:46:51.995519: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23524b1e00 of size 16128
2018-05-16 14:46:51.995547: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23524b5d00 of size 16128
2018-05-16 14:46:51.995577: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23524b9c00 of size 16128
2018-05-16 14:46:51.995608: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23524bdb00 of size 16128
2018-05-16 14:46:51.995635: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23524c1a00 of size 16128
2018-05-16 14:46:51.995664: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23524c5900 of size 16128
2018-05-16 14:46:51.995694: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23524c9800 of size 16128
2018-05-16 14:46:51.995724: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23524cd700 of size 16128
2018-05-16 14:46:51.995754: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23524d1600 of size 16128
2018-05-16 14:46:51.995783: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23524d5500 of size 16128
2018-05-16 14:46:51.995812: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23524d9400 of size 16128
2018-05-16 14:46:51.995841: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23524dd300 of size 256
2018-05-16 14:46:51.995870: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23524dd400 of size 16128
2018-05-16 14:46:51.995902: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23524e1300 of size 16128
2018-05-16 14:46:51.995933: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23524e5200 of size 16128
2018-05-16 14:46:51.995962: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free  at 0x23524e9100 of size 2040832
2018-05-16 14:46:51.995990: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23526db500 of size 4096
2018-05-16 14:46:51.996020: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23526dc500 of size 4096
2018-05-16 14:46:51.996051: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23526dd500 of size 4096
2018-05-16 14:46:51.996078: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23526de500 of size 16777216
2018-05-16 14:46:51.996108: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23536de500 of size 16777216
2018-05-16 14:46:51.996139: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23546de500 of size 16384
2018-05-16 14:46:51.996169: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23546e2500 of size 4096
2018-05-16 14:46:51.996199: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23546e3500 of size 4096
2018-05-16 14:46:51.996228: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23546e4500 of size 4096
2018-05-16 14:46:51.996257: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23546e5500 of size 4096
2018-05-16 14:46:51.996286: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23546e6500 of size 16777216
2018-05-16 14:46:51.996315: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23556e6500 of size 16777216
2018-05-16 14:46:51.996344: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23566e6500 of size 4096
2018-05-16 14:46:51.996373: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23566e7500 of size 4096
2018-05-16 14:46:51.996403: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23566e8500 of size 4096
2018-05-16 14:46:51.996431: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23566e9500 of size 4096
2018-05-16 14:46:51.996460: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23566ea500 of size 4096
2018-05-16 14:46:51.996490: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23566eb500 of size 4096
2018-05-16 14:46:51.996520: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23566ec500 of size 4194304
2018-05-16 14:46:51.996549: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2356aec500 of size 4194304
2018-05-16 14:46:51.996578: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2356eec500 of size 4194304
2018-05-16 14:46:51.996607: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23572ec500 of size 4194304
2018-05-16 14:46:51.996637: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23576ec500 of size 4194304
2018-05-16 14:46:51.996666: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2357aec500 of size 4194304
2018-05-16 14:46:51.996697: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2357eec500 of size 4194304
2018-05-16 14:46:51.996727: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23582ec500 of size 4206592
2018-05-16 14:46:51.996758: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23586ef500 of size 4096
2018-05-16 14:46:51.996788: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23586f0500 of size 16777216
2018-05-16 14:46:51.996817: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23596f0500 of size 16793600
2018-05-16 14:46:51.996847: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x235a6f4500 of size 16793600
2018-05-16 14:46:51.996877: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x235b6f8500 of size 16777216
2018-05-16 14:46:51.996907: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x235c6f8500 of size 4096
2018-05-16 14:46:51.996936: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x235c6f9500 of size 4096
2018-05-16 14:46:51.996965: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x235c6fa500 of size 4096
2018-05-16 14:46:51.996994: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x235c6fb500 of size 16384
2018-05-16 14:46:51.997023: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x235c6ff500 of size 4198400
2018-05-16 14:46:51.997052: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x235cb00500 of size 4194304
2018-05-16 14:46:51.997081: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x235cf00500 of size 4194304
2018-05-16 14:46:51.997111: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x235d300500 of size 4194304
2018-05-16 14:46:51.997140: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x235d700500 of size 4194304
2018-05-16 14:46:51.997169: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x235db00500 of size 4194304
2018-05-16 14:46:51.997198: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x235df00500 of size 4194304
2018-05-16 14:46:51.997229: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x235e300500 of size 4194304
2018-05-16 14:46:51.997258: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x235e700500 of size 4194304
2018-05-16 14:46:51.997288: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x235eb00500 of size 16777216
2018-05-16 14:46:51.997317: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x235fb00500 of size 16777216
2018-05-16 14:46:51.997347: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2360b00500 of size 16777216
2018-05-16 14:46:51.997376: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2361b00500 of size 4194304
2018-05-16 14:46:51.997407: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2361f00500 of size 20971520
2018-05-16 14:46:51.997438: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2363300500 of size 4194304
2018-05-16 14:46:51.997470: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2363700500 of size 4194304
2018-05-16 14:46:51.997499: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2363b00500 of size 4194304
2018-05-16 14:46:51.997528: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2363f00500 of size 4194304
2018-05-16 14:46:51.997557: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2364300500 of size 4194304
2018-05-16 14:46:51.997586: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2364700500 of size 20971520
2018-05-16 14:46:51.997614: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2365b00500 of size 16777216
2018-05-16 14:46:51.997644: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2366b00500 of size 4194304
2018-05-16 14:46:51.997672: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2366f00500 of size 4194304
2018-05-16 14:46:51.997702: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2367300500 of size 4194304
2018-05-16 14:46:51.997731: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2367700500 of size 4194304
2018-05-16 14:46:51.997762: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2367b00500 of size 16777216
2018-05-16 14:46:51.997791: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2368b00500 of size 4194304
2018-05-16 14:46:51.997821: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2368f00500 of size 4194304
2018-05-16 14:46:51.997850: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2369300500 of size 4194304
2018-05-16 14:46:51.997879: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2369700500 of size 4194304
2018-05-16 14:46:51.997909: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2369b00500 of size 4194304
2018-05-16 14:46:51.997938: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2369f00500 of size 4194304
2018-05-16 14:46:51.997968: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236a300500 of size 4194304
2018-05-16 14:46:51.997997: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236a700500 of size 4194304
2018-05-16 14:46:51.998026: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236ab00500 of size 4194304
2018-05-16 14:46:51.998056: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236af00500 of size 4194304
2018-05-16 14:46:51.998085: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236b300500 of size 4194304
2018-05-16 14:46:51.998114: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236b700500 of size 4194304
2018-05-16 14:46:51.998144: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236bb00500 of size 4194304
2018-05-16 14:46:51.998174: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236bf00500 of size 29360128
2018-05-16 14:46:51.998203: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236db00500 of size 16777216
2018-05-16 14:46:51.998232: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236eb00500 of size 16777216
2018-05-16 14:46:51.998261: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236fb00500 of size 4194304
2018-05-16 14:46:51.998290: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236ff00500 of size 4194304
2018-05-16 14:46:51.998319: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2370300500 of size 4194304
2018-05-16 14:46:51.998349: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2370700500 of size 4194304
2018-05-16 14:46:51.998378: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2370b00500 of size 4194304
2018-05-16 14:46:51.998408: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2370f00500 of size 4194304
2018-05-16 14:46:51.998438: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2371300500 of size 4194304
2018-05-16 14:46:51.998468: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2371700500 of size 4194304
2018-05-16 14:46:51.998497: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2371b00500 of size 4194304
2018-05-16 14:46:51.998526: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2371f00500 of size 4194304
2018-05-16 14:46:51.998555: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2372300500 of size 4194304
2018-05-16 14:46:51.998585: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2372700500 of size 4194304
2018-05-16 14:46:51.998640: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2372b00500 of size 4194304
2018-05-16 14:46:51.998673: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2372f00500 of size 4194304
2018-05-16 14:46:51.998704: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2373300500 of size 4194304
2018-05-16 14:46:51.998733: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2373700500 of size 4194304
2018-05-16 14:46:51.998763: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2373b00500 of size 4194304
2018-05-16 14:46:51.998792: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2373f00500 of size 4194304
2018-05-16 14:46:51.998822: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2374300500 of size 4194304
2018-05-16 14:46:51.998852: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2374700500 of size 4194304
2018-05-16 14:46:51.998881: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2374b00500 of size 16777216
2018-05-16 14:46:51.998911: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2375b00500 of size 16777216
2018-05-16 14:46:51.998940: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2376b00500 of size 16777216
2018-05-16 14:46:51.998969: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2377b00500 of size 4194304
2018-05-16 14:46:51.998999: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2377f00500 of size 4194304
2018-05-16 14:46:51.999028: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2378300500 of size 4194304
2018-05-16 14:46:51.999057: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2378700500 of size 4194304
2018-05-16 14:46:51.999086: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2378b00500 of size 4194304
2018-05-16 14:46:51.999130: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2378f00500 of size 4194304
2018-05-16 14:46:51.999153: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2379300500 of size 4194304
2018-05-16 14:46:51.999174: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2379700500 of size 4194304
2018-05-16 14:46:51.999195: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2379b00500 of size 4194304
2018-05-16 14:46:51.999217: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2379f00500 of size 4194304
2018-05-16 14:46:51.999241: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x237a300500 of size 4194304
2018-05-16 14:46:51.999262: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x237a700500 of size 4194304
2018-05-16 14:46:51.999282: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x237ab00500 of size 4194304
2018-05-16 14:46:51.999303: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x237af00500 of size 4194304
2018-05-16 14:46:51.999324: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x237b300500 of size 4194304
2018-05-16 14:46:51.999345: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x237b700500 of size 4194304
2018-05-16 14:46:51.999365: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x237bb00500 of size 4194304
2018-05-16 14:46:51.999385: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x237bf00500 of size 4194304
2018-05-16 14:46:51.999407: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x237c300500 of size 4194304
2018-05-16 14:46:51.999428: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x237c700500 of size 4194304
2018-05-16 14:46:51.999449: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x237cb00500 of size 4194304
2018-05-16 14:46:51.999470: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x237cf00500 of size 4194304
2018-05-16 14:46:51.999492: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x237d300500 of size 4194304
2018-05-16 14:46:51.999514: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x237d700500 of size 4194304
2018-05-16 14:46:51.999536: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x237db00500 of size 4194304
2018-05-16 14:46:51.999556: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x237df00500 of size 16777216
2018-05-16 14:46:51.999578: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x237ef00500 of size 16777216
2018-05-16 14:46:51.999599: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x237ff00500 of size 4194304
2018-05-16 14:46:51.999620: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2380300500 of size 4194304
2018-05-16 14:46:51.999641: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2380700500 of size 4194304
2018-05-16 14:46:51.999663: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2380b00500 of size 4194304
2018-05-16 14:46:51.999684: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2380f00500 of size 16777216
2018-05-16 14:46:51.999703: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2381f00500 of size 4194304
2018-05-16 14:46:51.999725: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2382300500 of size 4194304
2018-05-16 14:46:51.999746: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2382700500 of size 4194304
2018-05-16 14:46:51.999768: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2382b00500 of size 4194304
2018-05-16 14:46:51.999789: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2382f00500 of size 4194304
2018-05-16 14:46:51.999808: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2383300500 of size 16777216
2018-05-16 14:46:51.999830: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2384300500 of size 16777216
2018-05-16 14:46:51.999852: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2385300500 of size 16293888
2018-05-16 14:46:51.999873: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x238628a500 of size 29499392
2018-05-16 14:46:51.999895: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2387eac500 of size 4194304
2018-05-16 14:46:51.999916: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23882ac500 of size 4194304
2018-05-16 14:46:51.999937: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23886ac500 of size 4194304
2018-05-16 14:46:51.999958: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2388aac500 of size 4194304
2018-05-16 14:46:51.999979: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2388eac500 of size 16777216
2018-05-16 14:46:52.000001: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2389eac500 of size 4194304
2018-05-16 14:46:52.000024: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x238a2ac500 of size 4194304
2018-05-16 14:46:52.000045: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x238a6ac500 of size 4194304
2018-05-16 14:46:52.000066: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x238aaac500 of size 4194304
2018-05-16 14:46:52.000088: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x238aeac500 of size 4194304
2018-05-16 14:46:52.000109: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x238b2ac500 of size 4194304
2018-05-16 14:46:52.000130: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x238b6ac500 of size 25165824
2018-05-16 14:46:52.000152: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x238ceac500 of size 20971520
2018-05-16 14:46:52.000173: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x238e2ac500 of size 4194304
2018-05-16 14:46:52.000195: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x238e6ac500 of size 4194304
2018-05-16 14:46:52.000216: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x238eaac500 of size 20971520
2018-05-16 14:46:52.000236: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x238feac500 of size 20971520
2018-05-16 14:46:52.000257: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23912ac500 of size 4194304
2018-05-16 14:46:52.000278: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23916ac500 of size 4194304
2018-05-16 14:46:52.000299: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2391aac500 of size 4194304
2018-05-16 14:46:52.000321: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2391eac500 of size 16777216
2018-05-16 14:46:52.000343: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2392eac500 of size 16777216
2018-05-16 14:46:52.000365: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2393eac500 of size 4194304
2018-05-16 14:46:52.000386: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23942ac500 of size 4194304
2018-05-16 14:46:52.000407: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23946ac500 of size 4194304
2018-05-16 14:46:52.000428: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2394aac500 of size 4194304
2018-05-16 14:46:52.000450: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2394eac500 of size 16777216
2018-05-16 14:46:52.000470: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2395eac500 of size 25165824
2018-05-16 14:46:52.000492: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23976ac500 of size 4194304
2018-05-16 14:46:52.000514: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2397aac500 of size 4194304
2018-05-16 14:46:52.000536: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2397eac500 of size 4194304
2018-05-16 14:46:52.000558: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23982ac500 of size 4194304
2018-05-16 14:46:52.000579: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23986ac500 of size 4194304
2018-05-16 14:46:52.000600: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2398aac500 of size 4194304
2018-05-16 14:46:52.000620: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2398eac500 of size 20971520
2018-05-16 14:46:52.000642: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x239a2ac500 of size 16777216
2018-05-16 14:46:52.000663: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x239b2ac500 of size 4194304
2018-05-16 14:46:52.000685: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free  at 0x239b6ac500 of size 4194304
2018-05-16 14:46:52.000706: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x239baac500 of size 4194304
2018-05-16 14:46:52.000727: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x239beac500 of size 4194304
2018-05-16 14:46:52.000747: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x239c2ac500 of size 4194304
2018-05-16 14:46:52.000769: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x239c6ac500 of size 4194304
2018-05-16 14:46:52.000790: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x239caac500 of size 16293888
2018-05-16 14:46:52.000812: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x239da36500 of size 25649152
2018-05-16 14:46:52.000833: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x239f2ac500 of size 4194304
2018-05-16 14:46:52.000855: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free  at 0x239f6ac500 of size 4194304
2018-05-16 14:46:52.000877: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x239faac500 of size 4194304
2018-05-16 14:46:52.000898: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x239feac500 of size 4194304
2018-05-16 14:46:52.000919: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23a02ac500 of size 4194304
2018-05-16 14:46:52.000939: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23a06ac500 of size 16293888
2018-05-16 14:46:52.000961: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23a1636500 of size 29843456
2018-05-16 14:46:52.000983: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23a32ac500 of size 4194304
2018-05-16 14:46:52.001006: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23a36ac500 of size 4194304
2018-05-16 14:46:52.001027: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free  at 0x23a3aac500 of size 4194304
2018-05-16 14:46:52.001048: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23a3eac500 of size 4194304
2018-05-16 14:46:52.001070: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23a42ac500 of size 4194304
2018-05-16 14:46:52.001091: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free  at 0x23a46ac500 of size 8388608
2018-05-16 14:46:52.001113: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23a4eac500 of size 4194304
2018-05-16 14:46:52.001134: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23a52ac500 of size 16777216
2018-05-16 14:46:52.001155: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23a62ac500 of size 16777216
2018-05-16 14:46:52.001176: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23a72ac500 of size 4194304
2018-05-16 14:46:52.001198: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free  at 0x23a76ac500 of size 12582912
2018-05-16 14:46:52.001219: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23a82ac500 of size 4194304
2018-05-16 14:46:52.001241: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free  at 0x23a86ac500 of size 12582912
2018-05-16 14:46:52.001261: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23a92ac500 of size 4194304
2018-05-16 14:46:52.001282: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23a96ac500 of size 16293888
2018-05-16 14:46:52.001302: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23aa636500 of size 17260544
2018-05-16 14:46:52.001325: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23ab6ac500 of size 4194304
2018-05-16 14:46:52.001347: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23abaac500 of size 4194304
2018-05-16 14:46:52.001368: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23abeac500 of size 16777216
2018-05-16 14:46:52.001388: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23aceac500 of size 4194304
2018-05-16 14:46:52.001410: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23ad2ac500 of size 16777216
2018-05-16 14:46:52.001431: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23ae2ac500 of size 138067968
2018-05-16 14:46:52.001453: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23b6658500 of size 138067968
2018-05-16 14:46:52.001475: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23bea04500 of size 16293888
2018-05-16 14:46:52.001496: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23bf98e500 of size 138096640
2018-05-16 14:46:52.001516: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23c7d41500 of size 138096640
2018-05-16 14:46:52.001538: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23d00f4500 of size 16293888
2018-05-16 14:46:52.001560: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23d107e500 of size 16293888
2018-05-16 14:46:52.001581: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23d2008500 of size 16293888
2018-05-16 14:46:52.001603: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23d2f92500 of size 16293888
2018-05-16 14:46:52.001623: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23d3f1c500 of size 25968384
2018-05-16 14:46:52.001644: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23d57e0400 of size 25968384
2018-05-16 14:46:52.001666: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23d70a4300 of size 25968384
2018-05-16 14:46:52.001687: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23d8968200 of size 16293888
2018-05-16 14:46:52.001708: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23d98f2200 of size 16293888
2018-05-16 14:46:52.001729: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23da87c200 of size 16293888
2018-05-16 14:46:52.001750: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23db806200 of size 16293888
2018-05-16 14:46:52.001771: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23dc790200 of size 25968384
2018-05-16 14:46:52.001794: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23de054100 of size 25968384
2018-05-16 14:46:52.001815: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23df918000 of size 25968384
2018-05-16 14:46:52.001835: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23e11dbf00 of size 16293888
2018-05-16 14:46:52.001857: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23e2165f00 of size 16293888
2018-05-16 14:46:52.001879: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23e30eff00 of size 16293888
2018-05-16 14:46:52.001901: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23e4079f00 of size 16293888
2018-05-16 14:46:52.001922: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23e5003f00 of size 16293888
2018-05-16 14:46:52.001943: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23e5f8df00 of size 16293888
2018-05-16 14:46:52.001967: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23e6f17f00 of size 16293888
2018-05-16 14:46:52.001989: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23e7ea1f00 of size 16293888
2018-05-16 14:46:52.002011: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23e8e2bf00 of size 13881344
2018-05-16 14:46:52.002033: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23e9b68f00 of size 16293888
2018-05-16 14:46:52.002055: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23eaaf2f00 of size 55525376
2018-05-16 14:46:52.002078: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23edfe6f00 of size 55525376
2018-05-16 14:46:52.002099: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23f14daf00 of size 55525376
2018-05-16 14:46:52.002120: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23f49cef00 of size 16293888
2018-05-16 14:46:52.002142: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23f5958f00 of size 16293888
2018-05-16 14:46:52.002163: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23f68e2f00 of size 16293888
2018-05-16 14:46:52.002184: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23f786cf00 of size 16293888
2018-05-16 14:46:52.002205: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23f87f6f00 of size 16293888
2018-05-16 14:46:52.002227: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23f9780f00 of size 16293888
2018-05-16 14:46:52.002248: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23fa70af00 of size 16293888
2018-05-16 14:46:52.002269: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23fb694f00 of size 25968384
2018-05-16 14:46:52.002291: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23fcf58e00 of size 25968384
2018-05-16 14:46:52.002312: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23fe81cd00 of size 25968384
2018-05-16 14:46:52.002333: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24000e0c00 of size 16293888
2018-05-16 14:46:52.002355: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x240106ac00 of size 16293888
2018-05-16 14:46:52.002376: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2401ff4c00 of size 16293888
2018-05-16 14:46:52.002397: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2402f7ec00 of size 16293888
2018-05-16 14:46:52.002418: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2403f08c00 of size 13881344
2018-05-16 14:46:52.002440: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2404c45c00 of size 55525376
2018-05-16 14:46:52.002461: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2408139c00 of size 55525376
2018-05-16 14:46:52.002483: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x240b62dc00 of size 55525376
2018-05-16 14:46:52.002504: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x240eb21c00 of size 16293888
2018-05-16 14:46:52.002525: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x240faabc00 of size 16293888
2018-05-16 14:46:52.002546: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2410a35c00 of size 16293888
2018-05-16 14:46:52.002567: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24119bfc00 of size 16293888
2018-05-16 14:46:52.002589: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2412949c00 of size 16293888
2018-05-16 14:46:52.002618: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24138d3c00 of size 16293888
2018-05-16 14:46:52.002639: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x241485dc00 of size 16293888
2018-05-16 14:46:52.002660: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24157e7c00 of size 25968384
2018-05-16 14:46:52.002682: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24170abb00 of size 25968384
2018-05-16 14:46:52.002703: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x241896fa00 of size 25968384
2018-05-16 14:46:52.002724: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x241a233900 of size 16293888
2018-05-16 14:46:52.002745: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x241b1bd900 of size 16293888
2018-05-16 14:46:52.002767: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x241c147900 of size 16293888
2018-05-16 14:46:52.002789: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x241d0d1900 of size 16293888
2018-05-16 14:46:52.002808: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x241e05b900 of size 13881344
2018-05-16 14:46:52.002830: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x241ed98900 of size 55525376
2018-05-16 14:46:52.002851: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x242228c900 of size 55525376
2018-05-16 14:46:52.002873: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2425780900 of size 55525376
2018-05-16 14:46:52.002893: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2428c74900 of size 16293888
2018-05-16 14:46:52.002915: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2429bfe900 of size 16293888
2018-05-16 14:46:52.002937: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x242ab88900 of size 16293888
2018-05-16 14:46:52.002956: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x242bb12900 of size 16293888
2018-05-16 14:46:52.002977: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x242ca9c900 of size 16293888
2018-05-16 14:46:52.002998: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x242da26900 of size 16293888
2018-05-16 14:46:52.003020: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x242e9b0900 of size 16293888
2018-05-16 14:46:52.003041: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x242f93a900 of size 25968384
2018-05-16 14:46:52.003063: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24311fe800 of size 25968384
2018-05-16 14:46:52.003083: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2432ac2700 of size 25968384
2018-05-16 14:46:52.003105: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2434386600 of size 16293888
2018-05-16 14:46:52.003127: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2435310600 of size 16293888
2018-05-16 14:46:52.003147: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x243629a600 of size 16293888
2018-05-16 14:46:52.003169: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2437224600 of size 16293888
2018-05-16 14:46:52.003191: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24381ae600 of size 13881344
2018-05-16 14:46:52.003213: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2438eeb600 of size 55525376
2018-05-16 14:46:52.003234: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x243c3df600 of size 55525376
2018-05-16 14:46:52.003256: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x243f8d3600 of size 55525376
2018-05-16 14:46:52.003277: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2442dc7600 of size 16293888
2018-05-16 14:46:52.003298: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2443d51600 of size 16293888
2018-05-16 14:46:52.003321: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2444cdb600 of size 16293888
2018-05-16 14:46:52.003343: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2445c65600 of size 16293888
2018-05-16 14:46:52.003364: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2446bef600 of size 16293888
2018-05-16 14:46:52.003385: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2447b79600 of size 16293888
2018-05-16 14:46:52.003407: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2448b03600 of size 16293888
2018-05-16 14:46:52.003430: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2449a8d600 of size 25968384
2018-05-16 14:46:52.003451: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x244b351500 of size 25968384
2018-05-16 14:46:52.003472: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x244cc15400 of size 25968384
2018-05-16 14:46:52.003492: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x244e4d9300 of size 16293888
2018-05-16 14:46:52.003514: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x244f463300 of size 16293888
2018-05-16 14:46:52.003535: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24503ed300 of size 16293888
2018-05-16 14:46:52.003557: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2451377300 of size 16293888
2018-05-16 14:46:52.003578: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2452301300 of size 13881344
2018-05-16 14:46:52.003600: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x245303e300 of size 55525376
2018-05-16 14:46:52.003621: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2456532300 of size 55525376
2018-05-16 14:46:52.003642: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2459a26300 of size 55525376
2018-05-16 14:46:52.003663: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x245cf1a300 of size 16293888
2018-05-16 14:46:52.003685: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x245dea4300 of size 16293888
2018-05-16 14:46:52.003707: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x245ee2e300 of size 16293888
2018-05-16 14:46:52.003728: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x245fdb8300 of size 16293888
2018-05-16 14:46:52.003749: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2460d42300 of size 16293888
2018-05-16 14:46:52.003771: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2461ccc300 of size 16293888
2018-05-16 14:46:52.003793: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2462c56300 of size 16293888
2018-05-16 14:46:52.003815: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2463be0300 of size 25968384
2018-05-16 14:46:52.003836: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24654a4200 of size 25968384
2018-05-16 14:46:52.003857: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2466d68100 of size 25968384
2018-05-16 14:46:52.003879: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x246862c000 of size 16293888
2018-05-16 14:46:52.003899: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24695b6000 of size 16293888
2018-05-16 14:46:52.003920: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x246a540000 of size 16293888
2018-05-16 14:46:52.003942: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x246b4ca000 of size 16293888
2018-05-16 14:46:52.003963: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x246c454000 of size 13881344
2018-05-16 14:46:52.003985: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x246d191000 of size 55525376
2018-05-16 14:46:52.004006: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2470685000 of size 55525376
2018-05-16 14:46:52.004027: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2473b79000 of size 55525376
2018-05-16 14:46:52.004048: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x247706d000 of size 16293888
2018-05-16 14:46:52.004070: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2477ff7000 of size 16293888
2018-05-16 14:46:52.004091: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2478f81000 of size 16293888
2018-05-16 14:46:52.004112: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2479f0b000 of size 16293888
2018-05-16 14:46:52.004133: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x247ae95000 of size 16293888
2018-05-16 14:46:52.004155: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x247be1f000 of size 16293888
2018-05-16 14:46:52.004176: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x247cda9000 of size 16293888
2018-05-16 14:46:52.004198: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x247dd33000 of size 16293888
2018-05-16 14:46:52.004219: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x247ecbd000 of size 16293888
2018-05-16 14:46:52.004240: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x247fc47000 of size 16293888
2018-05-16 14:46:52.004263: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2480bd1000 of size 16293888
2018-05-16 14:46:52.004285: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2481b5b000 of size 16293888
2018-05-16 14:46:52.004306: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2482ae5000 of size 16293888
2018-05-16 14:46:52.004327: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2483a6f000 of size 16293888
2018-05-16 14:46:52.004348: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24849f9000 of size 16293888
2018-05-16 14:46:52.004369: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2485983000 of size 25968384
2018-05-16 14:46:52.004393: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2487246f00 of size 25968384
2018-05-16 14:46:52.004414: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2488b0ae00 of size 25968384
2018-05-16 14:46:52.004436: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x248a3ced00 of size 16293888
2018-05-16 14:46:52.004457: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x248b358d00 of size 16293888
2018-05-16 14:46:52.004479: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x248c2e2d00 of size 16293888
2018-05-16 14:46:52.004500: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x248d26cd00 of size 16293888
2018-05-16 14:46:52.004522: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x248e1f6d00 of size 65175552
2018-05-16 14:46:52.004541: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x249201ed00 of size 65175552
2018-05-16 14:46:52.004563: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2495e46d00 of size 65175552
2018-05-16 14:46:52.004584: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2499c6ed00 of size 16293888
2018-05-16 14:46:52.004605: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x249abf8d00 of size 16293888
2018-05-16 14:46:52.004626: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x249bb82d00 of size 16293888
2018-05-16 14:46:52.004648: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x249cb0cd00 of size 16293888
2018-05-16 14:46:52.004669: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x249da96d00 of size 16293888
2018-05-16 14:46:52.004692: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x249ea20d00 of size 16293888
2018-05-16 14:46:52.004716: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x249f9aad00 of size 16293888
2018-05-16 14:46:52.004738: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24a0934d00 of size 16293888
2018-05-16 14:46:52.004759: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24a18bed00 of size 25968384
2018-05-16 14:46:52.004780: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24a3182c00 of size 25968384
2018-05-16 14:46:52.004803: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24a4a46b00 of size 25968384
2018-05-16 14:46:52.004824: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24a630aa00 of size 16293888
2018-05-16 14:46:52.004845: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24a7294a00 of size 16293888
2018-05-16 14:46:52.004866: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24a821ea00 of size 16293888
2018-05-16 14:46:52.004885: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24a91a8a00 of size 16293888
2018-05-16 14:46:52.004906: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24aa132a00 of size 16293888
2018-05-16 14:46:52.004928: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24ab0bca00 of size 16293888
2018-05-16 14:46:52.004950: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24ac046a00 of size 25968384
2018-05-16 14:46:52.004973: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24ad90a900 of size 25968384
2018-05-16 14:46:52.004994: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24af1ce800 of size 25968384
2018-05-16 14:46:52.005016: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24b0a92700 of size 16293888
2018-05-16 14:46:52.005038: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24b1a1c700 of size 16293888
2018-05-16 14:46:52.005060: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24b29a6700 of size 16293888
2018-05-16 14:46:52.005081: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24b3930700 of size 16293888
2018-05-16 14:46:52.005102: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24b48ba700 of size 65175552
2018-05-16 14:46:52.005124: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24b86e2700 of size 65175552
2018-05-16 14:46:52.005146: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24bc50a700 of size 65175552
2018-05-16 14:46:52.005169: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24c0332700 of size 16293888
2018-05-16 14:46:52.005190: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24c12bc700 of size 16293888
2018-05-16 14:46:52.005212: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24c2246700 of size 16293888
2018-05-16 14:46:52.005235: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24c31d0700 of size 16293888
2018-05-16 14:46:52.005256: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24c415a700 of size 16293888
2018-05-16 14:46:52.005277: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24c50e4700 of size 16293888
2018-05-16 14:46:52.005299: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24c606e700 of size 16293888
2018-05-16 14:46:52.005319: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24c6ff8700 of size 16293888
2018-05-16 14:46:52.005341: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24c7f82700 of size 25968384
2018-05-16 14:46:52.005362: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24c9846600 of size 25968384
2018-05-16 14:46:52.005384: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24cb10a500 of size 25968384
2018-05-16 14:46:52.005405: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24cc9ce400 of size 16293888
2018-05-16 14:46:52.005426: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24cd958400 of size 16293888
2018-05-16 14:46:52.005448: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24ce8e2400 of size 16293888
2018-05-16 14:46:52.005469: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24cf86c400 of size 16293888
2018-05-16 14:46:52.005492: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24d07f6400 of size 16293888
2018-05-16 14:46:52.005514: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24d1780400 of size 16293888
2018-05-16 14:46:52.005535: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24d270a400 of size 25968384
2018-05-16 14:46:52.005556: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24d3fce300 of size 25968384
2018-05-16 14:46:52.005578: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24d5892200 of size 25968384
2018-05-16 14:46:52.005600: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24d7156100 of size 16293888
2018-05-16 14:46:52.005620: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24d80e0100 of size 16293888
2018-05-16 14:46:52.005643: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24d906a100 of size 16293888
2018-05-16 14:46:52.005665: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24d9ff4100 of size 16293888
2018-05-16 14:46:52.005687: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24daf7e100 of size 65175552
2018-05-16 14:46:52.005709: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24deda6100 of size 65175552
2018-05-16 14:46:52.005730: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24e2bce100 of size 65175552
2018-05-16 14:46:52.005753: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24e69f6100 of size 16293888
2018-05-16 14:46:52.005775: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24e7980100 of size 16293888
2018-05-16 14:46:52.005796: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24e890a100 of size 16293888
2018-05-16 14:46:52.005818: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24e9894100 of size 16293888
2018-05-16 14:46:52.005839: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24ea81e100 of size 16293888
2018-05-16 14:46:52.005862: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24eb7a8100 of size 16293888
2018-05-16 14:46:52.005885: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24ec732100 of size 16293888
2018-05-16 14:46:52.005906: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24ed6bc100 of size 16293888
2018-05-16 14:46:52.005927: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24ee646100 of size 25968384
2018-05-16 14:46:52.005949: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24eff0a000 of size 25968384
2018-05-16 14:46:52.005971: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24f17cdf00 of size 25968384
2018-05-16 14:46:52.005993: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24f3091e00 of size 16293888
2018-05-16 14:46:52.006016: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24f401be00 of size 16293888
2018-05-16 14:46:52.006037: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24f4fa5e00 of size 16293888
2018-05-16 14:46:52.006059: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24f5f2fe00 of size 16293888
2018-05-16 14:46:52.006081: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24f6eb9e00 of size 16293888
2018-05-16 14:46:52.006102: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24f7e43e00 of size 25968384
2018-05-16 14:46:52.006125: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24f9707d00 of size 25968384
2018-05-16 14:46:52.006147: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24fafcbc00 of size 25968384
2018-05-16 14:46:52.006171: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24fc88fb00 of size 16293888
2018-05-16 14:46:52.006193: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24fd819b00 of size 16293888
2018-05-16 14:46:52.006214: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24fe7a3b00 of size 16293888
2018-05-16 14:46:52.006237: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24ff72db00 of size 16293888
2018-05-16 14:46:52.006259: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x25006b7b00 of size 16293888
2018-05-16 14:46:52.006282: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2501641b00 of size 16293888
2018-05-16 14:46:52.006303: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x25025cbb00 of size 65175552
2018-05-16 14:46:52.006325: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x25063f3b00 of size 65175552
2018-05-16 14:46:52.006346: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x250a21bb00 of size 65175552
2018-05-16 14:46:52.006368: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x250e043b00 of size 16293888
2018-05-16 14:46:52.006388: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x250efcdb00 of size 16293888
2018-05-16 14:46:52.006410: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x250ff57b00 of size 16293888
2018-05-16 14:46:52.006432: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2510ee1b00 of size 16293888
2018-05-16 14:46:52.006453: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2511e6bb00 of size 16293888
2018-05-16 14:46:52.006474: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2512df5b00 of size 16293888
2018-05-16 14:46:52.006494: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2513d7fb00 of size 16293888
2018-05-16 14:46:52.006516: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2514d09b00 of size 25968384
2018-05-16 14:46:52.006537: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x25165cda00 of size 25968384
2018-05-16 14:46:52.006559: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2517e91900 of size 25968384
2018-05-16 14:46:52.006581: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2519755800 of size 16293888
2018-05-16 14:46:52.008916: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x251a6df800 of size 16293888
2018-05-16 14:46:52.008939: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x251b669800 of size 16293888
2018-05-16 14:46:52.008956: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x251c5f3800 of size 16293888
2018-05-16 14:46:52.008974: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x251d57d800 of size 16293888
2018-05-16 14:46:52.008991: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x251e507800 of size 25968384
2018-05-16 14:46:52.009010: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x251fdcb700 of size 25968384
2018-05-16 14:46:52.009028: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x252168f600 of size 25968384
2018-05-16 14:46:52.009071: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2522f53500 of size 16293888
2018-05-16 14:46:52.009092: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2523edd500 of size 16293888
2018-05-16 14:46:52.009115: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2524e67500 of size 16293888
2018-05-16 14:46:52.009137: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2525df1500 of size 16293888
2018-05-16 14:46:52.009159: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2526d7b500 of size 16293888
2018-05-16 14:46:52.009180: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2527d05500 of size 16293888
2018-05-16 14:46:52.009202: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2528c8f500 of size 65175552
2018-05-16 14:46:52.009223: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x252cab7500 of size 65175552
2018-05-16 14:46:52.009245: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x25308df500 of size 65175552
2018-05-16 14:46:52.009265: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2534707500 of size 16293888
2018-05-16 14:46:52.009287: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2535691500 of size 16293888
2018-05-16 14:46:52.009309: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x253661b500 of size 16293888
2018-05-16 14:46:52.009330: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x25375a5500 of size 16293888
2018-05-16 14:46:52.009350: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x253852f500 of size 16293888
2018-05-16 14:46:52.009371: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x25394b9500 of size 16293888
2018-05-16 14:46:52.009392: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x253a443500 of size 16293888
2018-05-16 14:46:52.009412: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x253b3cd500 of size 25968384
2018-05-16 14:46:52.009432: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x253cc91400 of size 25968384
2018-05-16 14:46:52.009453: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x253e555300 of size 25968384
2018-05-16 14:46:52.009474: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x253fe19200 of size 16293888
2018-05-16 14:46:52.009495: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2540da3200 of size 16293888
2018-05-16 14:46:52.009516: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2541d2d200 of size 16293888
2018-05-16 14:46:52.009537: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2542cb7200 of size 16293888
2018-05-16 14:46:52.009558: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2543c41200 of size 16293888
2018-05-16 14:46:52.009579: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2544bcb200 of size 16293888
2018-05-16 14:46:52.009604: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2545b55200 of size 25968384
2018-05-16 14:46:52.009625: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2547419100 of size 25968384
2018-05-16 14:46:52.009646: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2548cdd000 of size 25968384
2018-05-16 14:46:52.009667: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x254a5a0f00 of size 16293888
2018-05-16 14:46:52.009687: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x254b52af00 of size 16293888
2018-05-16 14:46:52.009709: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x254c4b4f00 of size 16293888
2018-05-16 14:46:52.009730: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x254d43ef00 of size 16293888
2018-05-16 14:46:52.009752: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x254e3c8f00 of size 65175552
2018-05-16 14:46:52.009773: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x25521f0f00 of size 65175552
2018-05-16 14:46:52.009794: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2556018f00 of size 65175552
2018-05-16 14:46:52.009816: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2559e40f00 of size 16293888
2018-05-16 14:46:52.009835: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x255adcaf00 of size 16293888
2018-05-16 14:46:52.009856: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free  at 0x255bd54f00 of size 16293888
2018-05-16 14:46:52.009878: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x255ccdef00 of size 16293888
2018-05-16 14:46:52.009901: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x255dc68f00 of size 536361728
2018-05-16 14:46:52.009922: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x257dbeca00 of size 536361728
2018-05-16 14:46:52.009943: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free  at 0x259db70500 of size 391910400
2018-05-16 14:46:52.009964: I tensorflow/core/common_runtime/bfc_allocator.cc:671]      Summary of in-use Chunks by size:
2018-05-16 14:46:52.009992: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 181 Chunks of size 256 totalling 45.2KiB
2018-05-16 14:46:52.010015: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 1280 totalling 1.2KiB
2018-05-16 14:46:52.010038: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 2560 totalling 2.5KiB
2018-05-16 14:46:52.010059: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 2816 totalling 2.8KiB
2018-05-16 14:46:52.010082: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 225 Chunks of size 4096 totalling 900.0KiB
2018-05-16 14:46:52.010104: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 2 Chunks of size 4352 totalling 8.5KiB
2018-05-16 14:46:52.010126: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 7936 totalling 7.8KiB
2018-05-16 14:46:52.010148: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 6 Chunks of size 13568 totalling 79.5KiB
2018-05-16 14:46:52.010171: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 36 Chunks of size 16128 totalling 567.0KiB
2018-05-16 14:46:52.010194: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 31 Chunks of size 16384 totalling 496.0KiB
2018-05-16 14:46:52.010217: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 4 Chunks of size 20480 totalling 80.0KiB
2018-05-16 14:46:52.010240: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 2 Chunks of size 24576 totalling 48.0KiB
2018-05-16 14:46:52.010263: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 30464 totalling 29.8KiB
2018-05-16 14:46:52.010286: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 268800 totalling 262.5KiB
2018-05-16 14:46:52.010307: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 2 Chunks of size 269824 totalling 527.0KiB
2018-05-16 14:46:52.010330: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 417792 totalling 408.0KiB
2018-05-16 14:46:52.010352: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 208 Chunks of size 4194304 totalling 832.00MiB
2018-05-16 14:46:52.010375: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 4 Chunks of size 4198400 totalling 16.02MiB
2018-05-16 14:46:52.010399: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 2 Chunks of size 4206592 totalling 8.02MiB
2018-05-16 14:46:52.010420: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 7544576 totalling 7.19MiB
2018-05-16 14:46:52.010442: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 8379904 totalling 7.99MiB
2018-05-16 14:46:52.010465: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 6 Chunks of size 13881344 totalling 79.43MiB
2018-05-16 14:46:52.010487: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 189 Chunks of size 16293888 totalling 2.87GiB
2018-05-16 14:46:52.010512: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 50 Chunks of size 16777216 totalling 800.00MiB
2018-05-16 14:46:52.010534: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 16781056 totalling 16.00MiB
2018-05-16 14:46:52.010557: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 2 Chunks of size 16781312 totalling 32.01MiB
2018-05-16 14:46:52.010580: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 16785408 totalling 16.01MiB
2018-05-16 14:46:52.010621: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 7 Chunks of size 16793600 totalling 112.11MiB
2018-05-16 14:46:52.010646: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 17260544 totalling 16.46MiB
2018-05-16 14:46:52.010675: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 6 Chunks of size 20971520 totalling 120.00MiB
2018-05-16 14:46:52.010697: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 20987904 totalling 20.02MiB
2018-05-16 14:46:52.010721: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 25096192 totalling 23.93MiB
2018-05-16 14:46:52.010746: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 2 Chunks of size 25165824 totalling 48.00MiB
2018-05-16 14:46:52.010769: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 25649152 totalling 24.46MiB
2018-05-16 14:46:52.010792: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 54 Chunks of size 25968384 totalling 1.31GiB
2018-05-16 14:46:52.010814: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 29360128 totalling 28.00MiB
2018-05-16 14:46:52.010837: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 29499392 totalling 28.13MiB
2018-05-16 14:46:52.010860: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 29843456 totalling 28.46MiB
2018-05-16 14:46:52.010884: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 33210368 totalling 31.67MiB
2018-05-16 14:46:52.010906: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 18 Chunks of size 55525376 totalling 953.16MiB
2018-05-16 14:46:52.010928: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 18 Chunks of size 65175552 totalling 1.09GiB
2018-05-16 14:46:52.010952: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 2 Chunks of size 138067968 totalling 263.34MiB
2018-05-16 14:46:52.010975: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 138076160 totalling 131.68MiB
2018-05-16 14:46:52.010997: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 2 Chunks of size 138096640 totalling 263.40MiB
2018-05-16 14:46:52.011021: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 2 Chunks of size 536361728 totalling 1023.03MiB
2018-05-16 14:46:52.011044: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Sum Total of in-use chunks: 10.08GiB
2018-05-16 14:46:52.011071: I tensorflow/core/common_runtime/bfc_allocator.cc:680] Stats:
Limit:                 11284978074
InUse:                 10828579584
MaxInUse:              11220441600
NumAllocs:                  197840
MaxAllocSize:            618382336

2018-05-16 14:46:52.011159: W tensorflow/core/common_runtime/bfc_allocator.cc:279] *************************************************************************************************___
Setting random seed =  1
Training schedule:
	1. Train for 1 epochs.
	2. Evaluate model.
	3. Compute BLEU score.
Repeat above steps until the BLEU score reaches 25.0
Starting iteration 1
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1322, in _do_call
    return fn(*args)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1307, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1409, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: Ran out of GPU memory when allocating 536361696 bytes for
	 [[Node: model/loss/smoothing_cross_entropy/softmax_cross_entropy_with_logits = SoftmaxCrossEntropyWithLogits[T=DT_FLOAT, _class=["loc:@model...s_grad/mul"], _device="/job:localhost/replica:0/task:0/device:GPU:0"](model/loss/smoothing_cross_entropy/softmax_cross_entropy_with_logits/Reshape, model/loss/smoothing_cross_entropy/softmax_cross_entropy_with_logits/Reshape_1)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

	 [[Node: model/truediv/_7571 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_16885_model/truediv", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "transformer/transformer_main.py", line 404, in <module>
    tf.app.run(main=main, argv=[sys.argv[0]] + unparsed)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/platform/app.py", line 125, in run
    _sys.exit(main(argv))
  File "transformer/transformer_main.py", line 313, in main
    FLAGS.bleu_threshold)
  File "transformer/transformer_main.py", line 253, in train_schedule
    estimator.train(dataset.train_input_fn, steps=single_iteration_train_steps)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/estimator.py", line 363, in train
    loss = self._train_model(input_fn, hooks, saving_listeners)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/estimator.py", line 1055, in _train_model
    return self._train_model_default(input_fn, hooks, saving_listeners)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/estimator.py", line 1071, in _train_model_default
    saving_listeners)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/estimator.py", line 1272, in _train_with_estimator_spec
    _, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/monitored_session.py", line 577, in run
    run_metadata=run_metadata)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/monitored_session.py", line 1053, in run
    run_metadata=run_metadata)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/monitored_session.py", line 1144, in run
    raise six.reraise(*original_exc_info)
  File "/usr/local/lib/python3.5/dist-packages/six.py", line 693, in reraise
    raise value
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/monitored_session.py", line 1129, in run
    return self._sess.run(*args, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/monitored_session.py", line 1201, in run
    run_metadata=run_metadata)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/monitored_session.py", line 981, in run
    return self._sess.run(*args, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 900, in run
    run_metadata_ptr)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1135, in _run
    feed_dict_tensor, options, run_metadata)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1316, in _do_run
    run_metadata)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1335, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: Ran out of GPU memory when allocating 536361696 bytes for
	 [[Node: model/loss/smoothing_cross_entropy/softmax_cross_entropy_with_logits = SoftmaxCrossEntropyWithLogits[T=DT_FLOAT, _class=["loc:@model...s_grad/mul"], _device="/job:localhost/replica:0/task:0/device:GPU:0"](model/loss/smoothing_cross_entropy/softmax_cross_entropy_with_logits/Reshape, model/loss/smoothing_cross_entropy/softmax_cross_entropy_with_logits/Reshape_1)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

	 [[Node: model/truediv/_7571 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_16885_model/truediv", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.


Caused by op 'model/loss/smoothing_cross_entropy/softmax_cross_entropy_with_logits', defined at:
  File "transformer/transformer_main.py", line 404, in <module>
    tf.app.run(main=main, argv=[sys.argv[0]] + unparsed)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/platform/app.py", line 125, in run
    _sys.exit(main(argv))
  File "transformer/transformer_main.py", line 313, in main
    FLAGS.bleu_threshold)
  File "transformer/transformer_main.py", line 253, in train_schedule
    estimator.train(dataset.train_input_fn, steps=single_iteration_train_steps)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/estimator.py", line 363, in train
    loss = self._train_model(input_fn, hooks, saving_listeners)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/estimator.py", line 1055, in _train_model
    return self._train_model_default(input_fn, hooks, saving_listeners)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/estimator.py", line 1068, in _train_model_default
    features, labels, model_fn_lib.ModeKeys.TRAIN, self.config)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/estimator.py", line 1043, in _call_model_fn
    model_fn_results = self._model_fn(features=features, **kwargs)
  File "transformer/transformer_main.py", line 67, in model_fn
    logits, targets, params.label_smoothing, params.vocab_size)
  File "/research/transformer/transformer/utils/metrics.py", line 77, in padded_cross_entropy_loss
    logits=logits, labels=soft_targets)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/nn_ops.py", line 1879, in softmax_cross_entropy_with_logits_v2
    precise_logits, labels, name=name)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gen_nn_ops.py", line 7205, in softmax_cross_entropy_with_logits
    name=name)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 3417, in create_op
    op_def=op_def)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 1743, in __init__
    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access

ResourceExhaustedError (see above for traceback): Ran out of GPU memory when allocating 536361696 bytes for
	 [[Node: model/loss/smoothing_cross_entropy/softmax_cross_entropy_with_logits = SoftmaxCrossEntropyWithLogits[T=DT_FLOAT, _class=["loc:@model...s_grad/mul"], _device="/job:localhost/replica:0/task:0/device:GPU:0"](model/loss/smoothing_cross_entropy/softmax_cross_entropy_with_logits/Reshape, model/loss/smoothing_cross_entropy/softmax_cross_entropy_with_logits/Reshape_1)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

	 [[Node: model/truediv/_7571 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_16885_model/truediv", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.


2018-05-16 14:46:52.079801: E tensorflow/core/common_runtime/bfc_allocator.cc:381] tried to deallocate nullptr
2018-05-16 14:46:53.238813: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.239041: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.239165: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.239203: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.239268: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.239482: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.239586: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.239716: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.239755: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.239844: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.239915: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.239971: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.240035: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.240069: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.240192: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.240235: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.240298: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.240420: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.240470: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.240741: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.240990: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.241139: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.241385: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.241576: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.241713: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.241866: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.242081: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.242140: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.242270: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.242342: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.242442: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.242821: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.243032: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.243114: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.243226: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.243260: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.243327: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.243553: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.243617: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.243716: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.243806: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.243919: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.244004: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.244085: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.244172: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.244287: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.244483: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.244558: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.244652: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.244714: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.244851: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.244985: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.245080: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.245145: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.245236: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.245487: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.245594: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.245775: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.245845: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.245904: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.246145: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.246221: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.246434: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.246486: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.246564: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.246645: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.246934: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.247044: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.247218: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.247273: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.247360: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.247385: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.247436: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.247492: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.247611: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.247680: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.247709: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.247790: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.247835: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.247922: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.248004: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.248101: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.248170: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.248243: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.248280: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.248366: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.248446: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.248488: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.248555: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.248595: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.248640: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.248690: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.248735: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.248767: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.248813: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.248867: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.248981: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.249041: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.249092: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.249115: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.249138: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.249162: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.249184: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.249250: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.249291: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.249383: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.249429: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.249475: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.249507: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.249583: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.249633: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.249660: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.249694: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.249750: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.249813: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.249929: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.250022: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.250131: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.250217: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.250270: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.250477: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.250544: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.250680: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.250901: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.251009: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.251050: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.251220: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.251303: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.251494: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.251526: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.251546: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.251577: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.251598: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.251619: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.251638: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.251658: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.251678: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.251699: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.251719: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.251738: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.251761: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.251781: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.251804: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.251824: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.251843: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.251862: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.251883: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.251903: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.251923: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.251943: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.251966: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.251986: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252007: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252027: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252049: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252069: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252089: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252109: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252132: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252152: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252180: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252200: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252220: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252241: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252262: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252282: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252302: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252330: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252353: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252373: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252393: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252413: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252432: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252455: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252475: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252494: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252525: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252544: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252564: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252588: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252608: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252629: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252648: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252668: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252689: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252710: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252730: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252749: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252778: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252798: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252817: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252840: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252859: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252879: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252901: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252921: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252940: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252960: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252982: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253001: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253020: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253040: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253071: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253092: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253114: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253134: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253154: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253174: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253195: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253215: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253235: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253255: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253282: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253304: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253324: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253343: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253365: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253385: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253407: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253428: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253448: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253471: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253491: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253511: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253531: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253560: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253581: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253603: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253623: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253643: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253663: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253685: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253706: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253726: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253745: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253767: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253788: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253807: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253827: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253866: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253888: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253909: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253929: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253948: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253968: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253987: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.254009: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.254030: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.254057: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.254079: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.254102: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.254123: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.254142: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.254165: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.254185: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.254205: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.254224: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.254245: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.254265: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.254284: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.254303: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.254334: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.254355: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.254375: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.254396: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.254416: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.254435: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.254458: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.254477: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.254499: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.254519: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.254539: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.254566: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.254587: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.254761: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.254779: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.254797: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.254820: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.254851: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.254873: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.254892: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.254915: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.254934: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.254954: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.254976: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.254995: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255015: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255035: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255056: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255076: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255102: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255124: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255143: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255163: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255185: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255205: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255224: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255246: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255265: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255285: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255304: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255323: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255353: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255373: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255392: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255415: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255434: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255454: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255477: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255496: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255516: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255537: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255556: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255577: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255596: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255622: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255644: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255664: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255686: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255706: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255725: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255748: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255770: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255790: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255812: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255832: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255851: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255871: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255900: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255921: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255943: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255963: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255983: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256003: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256025: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256045: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256064: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256084: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256105: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256133: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256153: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256173: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256200: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256220: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256243: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256264: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256283: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256305: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256325: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256344: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256364: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256392: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256412: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256433: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256454: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256474: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256496: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256517: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256536: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256556: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256577: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256597: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256617: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256638: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256668: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256689: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256711: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256730: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256749: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256768: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256790: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256810: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256830: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256849: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256875: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256895: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256915: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256937: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256957: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256977: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256999: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257019: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257040: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257060: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257082: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257102: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257123: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257142: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257169: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257188: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257209: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257230: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257251: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257271: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257292: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257312: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257331: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257350: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257374: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257394: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257413: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257433: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257460: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257480: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257502: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257522: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257543: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257563: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257585: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257605: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257624: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257651: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257672: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257691: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257723: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257744: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257766: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257786: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257806: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257825: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257844: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257867: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257886: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257906: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257936: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257957: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257977: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257998: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258018: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258037: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258059: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258078: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258098: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258117: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258139: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258159: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258179: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258198: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258228: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258248: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258270: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258290: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258310: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258329: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258351: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258371: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258391: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258410: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258431: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258452: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258474: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258494: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258522: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258542: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258564: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258585: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258619: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258639: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258659: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258682: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258701: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258729: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258751: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258770: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258793: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258813: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258832: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258854: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258874: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258894: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258914: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258936: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258956: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258975: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258995: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259023: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259044: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259066: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259087: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259107: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259134: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259156: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259176: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259197: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259222: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259243: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259263: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259282: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259312: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259333: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259354: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259373: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259393: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259414: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259434: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259454: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259474: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259503: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259522: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259544: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259563: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259582: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259601: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259623: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259643: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259662: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259683: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259703: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259722: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259743: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259773: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259793: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259814: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259834: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259853: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259872: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259894: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259913: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259933: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259953: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259974: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259994: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.260014: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.260033: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.260060: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.260081: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.260102: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.260123: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.260142: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.260162: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.260184: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.260203: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.260225: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.260244: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.260264: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.260282: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.260310: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.260330: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.260350: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.260369: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.260391: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.260411: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.260431: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.260450: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS

Key point is that :

tensorflow.python.framework.errors_impl.ResourceExhaustedError: Ran out of GPU memory when allocating 536361696 bytes for
	 [[Node: model/loss/smoothing_cross_entropy/softmax_cross_entropy_with_logits = SoftmaxCrossEntropyWithLogits[T=DT_FLOAT, _class=["loc:@model...s_grad/mul"], _device="/job:localhost/replica:0/task:0/device:GPU:0"](model/loss/smoothing_cross_entropy/softmax_cross_entropy_with_logits/Reshape, model/loss/smoothing_cross_entropy/softmax_cross_entropy_with_logits/Reshape_1)]]

I manage to fix this problem by adding and modifying codes in transformer_main.py line 308:

  session_config = tf.ConfigProto()
  session_config.gpu_options.per_process_gpu_memory_fraction = 0.8
  run_config = tf.estimator.RunConfig().replace(session_config=session_config)
  estimator = tf.estimator.Estimator(
      model_fn=model_fn, model_dir=FLAGS.model_dir, config=run_config, params=params)

It does work, however it throws another problem:

...
2018-05-17 03:16:30.080699: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 171646976 totalling 163.70MiB
2018-05-17 03:16:30.080726: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Sum Total of in-use chunks: 5.53GiB
2018-05-17 03:16:30.080757: I tensorflow/core/common_runtime/bfc_allocator.cc:680] Stats: 
Limit:                  5997789184
InUse:                  5934368256
MaxInUse:               5934370816
NumAllocs:                    1918
MaxAllocSize:            171646976

2018-05-17 03:16:30.080839: W tensorflow/core/common_runtime/bfc_allocator.cc:279] ****************************************************************************************************
2018-05-17 03:16:30.080918: W tensorflow/core/framework/op_kernel.cc:1290] CtxFailure at matmul_op.cc:474: Resource exhausted: OOM when allocating tensor with shape[3955,4096] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
Setting random seed =  1
Training schedule:
	1. Train for 1 epochs.
	2. Evaluate model.
	3. Compute BLEU score.
Repeat above steps until the BLEU score reaches 25.0
Starting iteration 1
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1322, in _do_call
    return fn(*args)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1307, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1409, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[3955,4096] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
	 [[Node: model/Transformer/decode/decoder_stack/layer_0/ffn/feed_foward_network/filter_layer/Tensordot/MatMul = MatMul[T=DT_FLOAT, _class=["loc:@model...d/MatMul_1"], transpose_a=false, transpose_b=false, _device="/job:localhost/replica:0/task:0/device:GPU:0"](model/Transformer/decode/decoder_stack/layer_0/ffn/feed_foward_network/filter_layer/Tensordot/Reshape, model/Transformer/decode/decoder_stack/layer_0/ffn/feed_foward_network/filter_layer/Tensordot/ArithmeticOptimizer/ArithmeticOptimizer/RemoveIdempotent_RemoveIdempotent_Reshape_1)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

	 [[Node: model/Transformer/decode/decoder_stack/layer_4/self_attention/self_attention/q/Tensordot/Shape/_5073 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_4944_...rdot/Shape", tensor_type=DT_INT32, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
...

Data Error in Translation

When I tried running the training for my dataset using the following command ****python transformer_main.py --data_dir=/tmp/translate_ende_raw --model_dir=/mlperf/tensorflow/transformer/model --params=base****

It throws a data error like this
*W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at iterator_ops.cc:891 : Data loss: corrupted record at 0
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1327, in _do_call
return fn(args)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1312, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1420, in _call_tf_sessionrun
status, run_metadata)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/errors_impl.py", line 516, in exit
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.DataLossError: corrupted record at 0
[[Node: IteratorGetNext = IteratorGetNextoutput_shapes=[[?,?], [?,?]], output_types=[DT_INT64, DT_INT64], _device="/job:localhost/replica:0/task:0/device:CPU:0"]]
[[Node: model/Transformer/decode/decoder_stack/layer_3/ffn/feed_foward_network/output_layer/Tensordot/Gather/_2941 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_4584_...dot/Gather", tensor_type=DT_INT32, _device="/job:localhost/replica:0/task:0/device:GPU:0"]]

When I tried to convert the params from base to big , it started running but after 11500 steps in the first epoch, it again showed the same error

*W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at iterator_ops.cc:891 : Data loss: corrupted record at 0
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1327, in _do_call
return fn(args)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1312, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1420, in _call_tf_sessionrun
status, run_metadata)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/errors_impl.py", line 516, in exit
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.DataLossError: corrupted record at 0
[[Node: IteratorGetNext = IteratorGetNextoutput_shapes=[[?,?], [?,?]], output_types=[DT_INT64, DT_INT64], _device="/job:localhost/replica:0/task:0/device:CPU:0"]]
[[Node: model/Transformer/encode/encoder_stack/layer_2/self_attention/self_attention/q/Tensordot/Gather/_5161 = _HostRecvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_1750_...dot/Gather", tensor_type=DT_INT32, _device="/job:localhost/replica:0/task:0/device:GPU:0"]]

Is there a better dataset that we can refer for this case?

[speech_recognition] RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation

I have built the docker container and downloaded the dataset inside of the docker container, however when I try to run the training, it results in the following output:

root@0a6d2a309b97:/speech_recognition/pytorch# time sh run_and_time.sh | tee speech_ds2.out
=======================================================
***acc                       = 23.0 
***continue_from             =  
***save_folder               = models/ 
***start_epoch               = -1 
***checkpoint                = False 
***seed                      = 1 
***model_path                = models/deepspeech_t1.pth.tar 
=======================================================
Directory already exists.
DataParallel(
  (module): DeepSpeech(
    (conv): Sequential(
      (0): Conv2d(1, 32, kernel_size=(41, 11), stride=(2, 2))
      (1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (2): Hardtanh(min_val=0, max_val=20, inplace)
      (3): Conv2d(32, 32, kernel_size=(21, 11), stride=(2, 1))
      (4): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (5): Hardtanh(min_val=0, max_val=20, inplace)
    )
    (rnns): Sequential(
      (0): BatchRNN(
        (rnn): GRU(672, 800, bidirectional=True)
      )
      (1): BatchRNN(
        (batch_norm): SequenceWise (
        BatchNorm1d(800, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True))
        (rnn): GRU(800, 800, bidirectional=True)
      )
      (2): BatchRNN(
        (batch_norm): SequenceWise (
        BatchNorm1d(800, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True))
        (rnn): GRU(800, 800, bidirectional=True)
      )
      (3): BatchRNN(
        (batch_norm): SequenceWise (
        BatchNorm1d(800, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True))
        (rnn): GRU(800, 800, bidirectional=True)
      )
      (4): BatchRNN(
        (batch_norm): SequenceWise (
        BatchNorm1d(800, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True))
        (rnn): GRU(800, 800, bidirectional=True)
      )
    )
    (fc): Sequential(
      (0): SequenceWise (
      Sequential(
        (0): BatchNorm1d(800, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (1): Linear(in_features=800, out_features=29, bias=False)
      ))
    )
    (inference_log_softmax): InferenceBatchLogSoftmax()
  )
)
Number of parameters: 38115968
Traceback (most recent call last):
  File "train.py", line 289, in <module>
    main()
  File "train.py", line 214, in main
    loss.backward()
  File "/usr/local/lib/python2.7/dist-packages/torch/tensor.py", line 93, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/usr/local/lib/python2.7/dist-packages/torch/autograd/__init__.py", line 89, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation

real	0m7.935s
user	0m4.496s
sys	0m4.384s

I am running on:
Ubuntu 16.04 x86_64
Nvidia Tesla V100
130 GB Memory

Does anyone know where this error is coming from?

[classification and reinforcement learning] TF nightly is installed without specifing version.

https://github.com/mlperf/reference/blob/master/reinforcement/tensorflow/Dockerfile
https://github.com/mlperf/reference/blob/master/image_classification/tensorflow/Dockerfile

This can lead to very different SW being used each docker build.

There should be a version specified, or even better a stable, versioned package should be used.

Training Error on ncf model of recommendation

During training processing, there exists an RuntimeError on DataLoader worker that it will automatically kill itself by connection reset

Traceback (most recent call last):
File "./ncf.py", line 230, in
main()
File "./ncf.py", line 190, in main
for batch_index, (user, item, label) in enumerate(loader):
File "/data/anaconda3/envs/idp/lib/python3.6/site-packages/tqdm/_tqdm.py", line 897, in iter
for obj in iterable:
File "/data/anaconda3/envs/idp/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 272, in next
return self._process_next_batch(batch)
File "/data/anaconda3/envs/idp/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 305, in _process_next_batch
self._put_indices()
File "/data/anaconda3/envs/idp/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 298, in _put_indices
self.index_queues[self.worker_queue_idx].put((self.send_idx, indices))
File "/data/anaconda3/envs/idp/lib/python3.6/multiprocessing/queues.py", line 347, in put
self._writer.send_bytes(obj)
File "/data/anaconda3/envs/idp/lib/python3.6/multiprocessing/connection.py", line 200, in send_bytes
self._send_bytes(m[offset:offset + size])
File "/data/anaconda3/envs/idp/lib/python3.6/multiprocessing/connection.py", line 398, in _send_bytes
self._send(buf)
File "/data/anaconda3/envs/idp/lib/python3.6/multiprocessing/connection.py", line 368, in _send
n = write(self._handle, buf)
File "/data/anaconda3/envs/idp/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 178, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 14955) is killed by signal: Killed.
Exception in thread Thread-2:
Traceback (most recent call last):
File "/data/anaconda3/envs/idp/lib/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/data/anaconda3/envs/idp/lib/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/data/anaconda3/envs/idp/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 71, in _worker_manager_loop
r = in_queue.get()
File "/data/anaconda3/envs/idp/lib/python3.6/multiprocessing/queues.py", line 337, in get
return _ForkingPickler.loads(res)
File "/data/anaconda3/envs/idp/lib/python3.6/site-packages/torch/multiprocessing/reductions.py", line 70, in rebuild_storage_fd
fd = df.detach()
File "/data/anaconda3/envs/idp/lib/python3.6/multiprocessing/resource_sharer.py", line 57, in detach
with _resource_sharer.get_connection(self._id) as conn:
File "/data/anaconda3/envs/idp/lib/python3.6/multiprocessing/resource_sharer.py", line 87, in get_connection
c = Client(address, authkey=process.current_process().authkey)
File "/data/anaconda3/envs/idp/lib/python3.6/multiprocessing/connection.py", line 493, in Client
answer_challenge(c, authkey)
File "/data/anaconda3/envs/idp/lib/python3.6/multiprocessing/connection.py", line 737, in answer_challenge
response = connection.recv_bytes(256) # reject large message
File "/data/anaconda3/envs/idp/lib/python3.6/multiprocessing/connection.py", line 216, in recv_bytes
buf = self._recv_bytes(maxlength)
File "/data/anaconda3/envs/idp/lib/python3.6/multiprocessing/connection.py", line 407, in _recv_bytes
buf = self._recv(4)
File "/data/anaconda3/envs/idp/lib/python3.6/multiprocessing/connection.py", line 379, in _recv
chunk = read(handle, remaining)
ConnectionResetError: [Errno 104] Connection reset by peer

here is the error log

translation: cannot find newstest2014.en?

the newstest2014 files are in the $HOME/reference/translation/tensorflow directory per the data_download.sh
I even copied them also into the transformer directory but the invocation
sudo docker run -v $HOME/reference/translation/raw_data:/raw_data --runtime=nvidia -t -i $IMAGE "./run_and_time.sh" $SEED | tee benchmark-$NOW.log
results in
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00099-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00100-of-00100
Setting random seed = 1
Traceback (most recent call last):
File "transformer/transformer_main.py", line 404, in
tf.app.run(main=main, argv=[sys.argv[0]] + unparsed)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/platform/app.py", line 125, in run
_sys.exit(main(argv))
File "transformer/transformer_main.py", line 298, in main
raise ValueError("BLEU source file %s does not exist" % FLAGS.bleu_source)
ValueError: BLEU source file newstest2014.en does not exist
~/reference/translation/tensorflow$ ls -l newstest2014.*
-rw-rw-r-- 1 mlperf mlperf 386840 Aug 15 2015 newstest2014.de
-rw-rw-r-- 1 mlperf mlperf 354089 Aug 15 2015 newstest2014.en

Quality targets in Recommendation mismatch between github and mlperf.org

In MLPerf user guide
Quality target is 0.9652 HR@10.

But on this page is HR@10: 0.9562

Can hyper-parameters of MCTS be altered in Go benchmark under CLOSE division?

Hyper-parameters like below:

1. Number of search threads
2. Search depth
3. Maximum number of moves for each game
4. Algorithm used to determine end-of-game winner
5. Exploration constant
6. Alpha value of Dirichlet noise
7. Value of virtual loss

[translation] Incorrect Download Links

We have identified a problem with the download links for translation. The current download script will download the incorrect test files (the files are derived from the correct files and have additional data in them which our preprocessor does not expect). This will result in the model training correctly, but the computed BLEU scores will be incorrect.

I am currently working on a new download links and a new verify_dataset.sh.

For the time being, people can download the correct files from:

https://raw.githubusercontent.com/tensorflow/mlbenchmark/master/transformer/test_data/newstest2014.en?token=ABCvMNAuy84bgb5dC-YxSxNyE6qvuit6ks5bGaqtwA%3D%3D
https://raw.githubusercontent.com/tensorflow/mlbenchmark/master/transformer/test_data/newstest2014.de?token=ABCvMIGPgU8sEiCTPR4D-yovJ-TooXv2ks5bGau_wA%3D%3D

md5sum:
06e8840abe90cbfbd45cf2729807605d newstest2014.de BAD
f6c3818b477e4a25cad68b61cc883c17 newstest2014.de GOOD
4e4663b8de25d19c5fc1c4dab8d61703 newstest2014.en BAD
dabf51a9c02b2235632f3cee75c72d49 newstest2014.en GOOD

[sentiment analysis] Bug in calculating accuracy and loss.

There’s a bug in reference implementation that incorrectly calculates accuracy, resulting in a slightly different reported accuracy. Sometimes true accuracy does not reach the target.

The bug is in the way test accuracy is computed:
test_acc = [batch_i_mean_acc].mean()

Test accuracy is computed as mean of means per batch. However the last batch has less elements than the rest. We are talking here about 195 batches with 128 ‘elements’ each, and then a final batch with 40 elements which gets ~triple weight.
Might look minor, however since the model is very often on the edge of target accuracy I do observe cases when correctly computed accuracy doesn’t hit target, but the one used does. Note that it can work both ways. But since this is in testing where input order is not shuffled, the impact would depend on “how easy to learn are these last 40 elements” (and they are all positive).

loss has similar issue. Same holds for training. But test accuracy is the most important one as it is actually serving as termination condition. All others are only used for printout.

pytorch 0.2.0 do not support the torch.utils.cpp_extension module

speech_recognition workload

Question1:

During I run sudo sh build-docker.sh, it throws the following error:

linlf@ccrfox246:~/project/linlf/mlperf/reference/speech_recognition/pytorch/docker$ sudo sh build-docker.sh
Sending build context to Docker daemon  6.656kB
Step 1/14 : FROM nvidia/cuda:9.0-cudnn7-devel-ubuntu16.04
 ---> 803d7d264fd1
Step 2/14 : WORKDIR /tmp
 ---> Using cache
 ---> e603dcfc0c3d
Step 3/14 : RUN apt-get update && apt-get install -y   python   python-pip   sox   libsox-dev   libsox-fmt-all   git   cmake   tree   htop   bmon   iotop   tmux   vim   apt-utils
 ---> Using cache
 ---> 953f6f9ac680
Step 4/14 : RUN pip install --upgrade pip
 ---> Using cache
 ---> e2aa5d01505a
Step 5/14 : RUN pip install h5py                 hickle                 matplotlib                 tqdm                 http://download.pytorch.org/whl/cu80/torch-0.2.0.post3-cp27-cp27mu-manylinux1_x86_64.whl                 torchvision                 cffi                 python-Levenshtein                 librosa                 wget                 tensorboardX
 ---> Using cache
 ---> 5c22988db0ab
Step 6/14 : RUN apt-get update && apt-get install --yes --no-install-recommends cmake                                                                     sudo
 ---> Using cache
 ---> 2bb0e8a80162
Step 7/14 : ENV CUDA_HOME "/usr/local/cuda"
 ---> Using cache
 ---> 0dc8ba208277
Step 8/14 : RUN git clone https://github.com/SeanNaren/warp-ctc.git &&     cd warp-ctc &&     mkdir -p build && cd build && cmake .. && make &&     cd ../pytorch_binding && python setup.py install
 ---> Using cache
 ---> c9ae175af961
Step 9/14 : RUN apt-get install -y sox libsox-dev libsox-fmt-all
 ---> Using cache
 ---> c4a00a82418d
Step 10/14 : RUN git clone https://github.com/pytorch/audio.git
 ---> Using cache
 ---> f8d759701a93
Step 11/14 : RUN cd audio; python setup.py install
 ---> Running in b10a799e29f5
Traceback (most recent call last):
  File "setup.py", line 4, in <module>
    from torch.utils.cpp_extension import BuildExtension, CppExtension
ImportError: No module named cpp_extension
The command '/bin/sh -c cd audio; python setup.py install' returned a non-zero code: 1

I notice that the audio lib utilizes the torch.utils.cpp_extension module. However, it installs the pytorch 0.2.0 in Dockerfile.gpu file:

# Unlike apt-get, upgrading pip does not change which package gets installed,
# (since it checks pypi everytime regardless) so it's okay to cache pip.
# Install pytorch
# http://pytorch.org/
RUN pip install h5py \
                hickle \
                matplotlib \
                tqdm \
                http://download.pytorch.org/whl/cu80/torch-0.2.0.post3-cp27-cp27mu-manylinux1_x86_64.whl \
                torchvision \
                cffi \
                python-Levenshtein \
                librosa \
                wget \
                tensorboardX

It's worth notice that in the version 0.2.0 of pytorch, this version actually does not support the torch.utils.cpp_extension module pytorch version 0.2.0 utils module. It supports cpp_extension module until version 0.4.0 pytorch version 0.4.0 utils.cpp_extension module

How to fix this problem?

Formalize input preprocessing

In our experience, input preprocessing may significantly affect accuracy scores.

For example, when evaluating the Intel Caffe artifact submitted to ReQuEST@ASPLOS'18, we measured the following Top-1/Top-5 ImageNet scores for ResNet50 in single-precision floating-point:

Resolution	Top-1	Top-5
224	0.707186	0.898327
256	0.729173	0.911852
320	0.731754	0.912832
320'	0.736136	0.914713

This table shows that while ResNet50 operates on 224x224 input images, rescaling to a higher resolution and then cropping to 224x224 can boost the Top-1 accuracy score by up to 3%!

Furthermore, the accuracy may significantly depend on the mean values used. In the above table, the row indexed 320 was obtained with the mean calculated over the validation set, while the row indexed 320' with the mean calculated over the training set.

It is clear that this or similar preprocessing techniques can in principle be used to hide significant accuracy degradation from aggressive quantization or pruning techniques. Therefore, their use should be formalized.

[classification] LR warmup for large batch training

There should be a LR warmup method implemented which would make stability and convergence better for larger batch sizes.

For the first couple (5) of epochs learning rate for epoch e shoud equal lr = base_lr * e / 5.
This is a very small change and doesn't break convergence with small batch sizes, but stabilizes training for batch sizes > 256.

[Translation] Update Readme Documentation for Data Sources

The README.md references the Stanford files which are not part of the standard WMT data indicated in the download scripts. The README should reflect what the reference code is doing, as it was updated in this commit.

For clarity, the incorrect links are:
https://nlp.stanford.edu/projects/nmt/data/wmt14.en-de/newstest2014.en
https://nlp.stanford.edu/projects/nmt/data/wmt14.en-de/newstest2014.de

and the correct links are:
https://raw.githubusercontent.com/tensorflow/models/master/official/transformer/test_data/newstest2014.en
https://raw.githubusercontent.com/tensorflow/models/master/official/transformer/test_data/newstest2014.de

[recommendation] Python and apt packages not versioned

https://github.com/mlperf/reference/blob/876bf1a1d41db8ea105857bdae763686fd07a2eb/recommendation/pytorch/requirements.txt#L2-L3
https://github.com/mlperf/reference/blob/876bf1a1d41db8ea105857bdae763686fd07a2eb/recommendation/pytorch/Dockerfile#L7-L9

This packages are installed without specifying version. This can lead to different performance (or not work at all) in the future.

[object_detection] the eigen module in caffe2 is unreachtable and the old caffe2 is no longer maintained

Q1: the eigen module in caffe2 is unreachtable caffe2/.gitmodules line 10:

[submodule "third_party/eigen"]
	path = third_party/eigen
	url = https://github.com/RLovelett/eigen.git

It should be:

[submodule "third_party/eigen"]
	path = third_party/eigen
	url = https://github.com/eigenteam/eigen-git-mirror.git

During building the docker sudo docker build -t detectron . , it throws the error:

Cloning into 'third_party/eigen'...
fatal: could not read Username for 'https://github.com': No such device or address
fatal: clone of 'https://github.com/RLovelett/eigen.git' into submodule path 'third_party/eigen' failed

The PR and issue are there update, update.

However, the old caffe2 is no longer maintained. The code has been transfer under Pytorch. It has been merged into the Pytorch. Whether we select the old caffe2 or new caffe2 module in Pytorch? The maintainer @pjh5 said that the Caffe2 under the Pytorch repo should always work for people who just want the Caffe2 part. How to fix this problem?

reinforcement learning README typo

Typo (2 times) in README with the name of the file (params/final.josn -> params/final.json) in section 2. Directions.

can't build docker image in speech recognition example

Problem:

As in my terminal, the "build-docker" process was terminated at this step RUN cd audio; python setup.py install due to the Import Error.

Diagnosis:

The file in https://github.com/pytorch/audio/blob/master/setup.py requires torch.utils.cpp_extension module which was not supported by Torch0.2.0. It will cause the Import Error when building the docker image.

[reinforcement learning] Python requirements without version specification

https://github.com/mlperf/reference/blob/876bf1a1d41db8ea105857bdae763686fd07a2eb/reinforcement/tensorflow/minigo/requirements.txt#L1-L16

A lot of these packages doesn't have version specified, which can create differences in what is being build with docker build.

[image_classification] Quality should be an argument expected by run_and_time.sh

Currently quality is hardcoded into run_and_time.sh, which is not handy if one runs multiple time with different quality. It will be good if quality is an argument expected by run_and_time.sh.

[translation] ResourceExhaustedError: Ran out of GPU memory

I never modify the parameters of translation workload, it throws following error:

2018-05-16 14:46:52.011044: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Sum Total of in-use chunks: 10.08GiB
2018-05-16 14:46:52.011071: I tensorflow/core/common_runtime/bfc_allocator.cc:680] Stats:
Limit:                 11284978074
InUse:                 10828579584
MaxInUse:              11220441600
NumAllocs:                  197840
MaxAllocSize:            618382336

2018-05-16 14:46:52.011159: W tensorflow/core/common_runtime/bfc_allocator.cc:279] *************************************************************************************************___
Setting random seed =  1
Training schedule:
	1. Train for 1 epochs.
	2. Evaluate model.
	3. Compute BLEU score.
Repeat above steps until the BLEU score reaches 25.0
Starting iteration 1
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1322, in _do_call
    return fn(*args)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1307, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1409, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: Ran out of GPU memory when allocating 536361696 bytes for
	 [[Node: model/loss/smoothing_cross_entropy/softmax_cross_entropy_with_logits = SoftmaxCrossEntropyWithLogits[T=DT_FLOAT, _class=["loc:@model...s_grad/mul"], _device="/job:localhost/replica:0/task:0/device:GPU:0"](model/loss/smoothing_cross_entropy/softmax_cross_entropy_with_logits/Reshape, model/loss/smoothing_cross_entropy/softmax_cross_entropy_with_logits/Reshape_1)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

	 [[Node: model/truediv/_7571 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_16885_model/truediv", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

Actually because the too large batch size memory issues in ~/reference/translation/tensorflow/run.sh line 13:

python3 transformer/transformer_main.py --random_seed=$SEED --data_dir=processed_data/ --model_dir=model --params=big --bleu_threshold $QUALITY --bleu_source=newstest2014.en --bleu_ref=newstest2014.de

Main() function is in ~/reference/translation/tensorflow/transformer/transformer_main.py line267 :

def main(_):
  # Set logging level to INFO to display training progress (logged by the
  # estimator)
  tf.logging.set_verbosity(tf.logging.INFO)

  if FLAGS.params == "base":
    params = model_params.TransformerBaseParams
  elif FLAGS.params == "big":
    params = model_params.TransformerBigParams

Defining at ~/reference/translation/tensorflow/transformer/model/model_params.py :

class TransformerBigParams(TransformerBaseParams):
  """Parameters for the big Transformer model."""
  batch_size = 4096
  hidden_size = 1024
  filter_size = 4096
  num_heads = 16

How to set up the parameters adapting your server?

You can modify the parameters --params=big in ~/reference/translation/tensorflow/run.sh by:
--params=base if you GPU do not have large memory. Or you can self-adaption set up the parameters in ~/reference/translation/tensorflow/transformer/model/model_params.py

On the other hand, you can specify which GPU you want to use in ~/reference/translation/tensorflow/transformer/transformer_main.py . For example, I specify that the workload only uses the 3rd GPU:

import os
os.environ['CUDA_VISIBLE_DEVICES'] = '2'

And specify the use-rate of GPU in ~/reference/translation/tensorflow/transformer/transformer_main.py line 308 :

  session_config = tf.ConfigProto()
  session_config.gpu_options.per_process_gpu_memory_fraction = 0.8
  session_config.gpu_options.allow_growth = True
  session_config.gpu_options.allocator_type = 'BFC'

  run_config = tf.estimator.RunConfig().replace(session_config=session_config)

  estimator = tf.estimator.Estimator(
          model_fn=model_fn, model_dir=FLAGS.model_dir, config=run_config, params=params)
  train_schedule(
      estimator, train_eval_iterations, single_iteration_train_steps,
      single_iteration_train_epochs, FLAGS.bleu_source, FLAGS.bleu_ref,
      FLAGS.bleu_threshold)

paddle.fluid.core.EnforceNotMet: enforce allocating <= available failed. how to specify GPU?

In sentiment_analysis workload.

Question1:

I have 4 GPUs on server. I run the reinforcement workload on GPU-1,then I want to run the sentiment_analysis workload on the GPU-2, with the following setp:

$ export CUDA_VISIBLE_DEVICES=2
$ export NV_GPU=2
$ export FLAGS_fraction_of_gpu_memory_to_use=0.16
$ sudo nvidia-docker run -it -v `pwd`:/paddle sidgoyal78/paddle:benchmark12042018 /bin/bash

I execute the :

./run_and_time.sh 1

then it throws the following error:

root@402f3708b9fe:/paddle/sentiment_analysis/paddle# ./run_and_time.sh 1
STARTING TIMING RUN AT 2018-05-14 01:16:21 AM
Running sentiment benchmark with seed 1
Cache file /root/.cache/paddle/dataset/imdb/aclImdb_v1.tar.gz not found, downloading http://ai.stanford.edu/%7Eamaas/data/sentiment/aclImdb_v1.tar.gz
[==================================================]Traceback (most recent call last):
  File "train.py", line 217, in <module>
    save_dirname="understand_sentiment_conv.inference.model")
  File "train.py", line 191, in main
    save_dirname=save_dirname)
  File "train.py", line 176, in train
    train_loop(fluid.default_main_program())
  File "train.py", line 140, in train_loop
    exe.run(fluid.default_startup_program())
  File "/usr/local/lib/python2.7/dist-packages/paddle/fluid/executor.py", line 336, in run
    self.executor.run(program.desc, scope, 0, True, True)
paddle.fluid.core.EnforceNotMet: enforce allocating <= available failed, 10484135494 > 8700690176
 at [/paddle/paddle/fluid/platform/gpu_info.cc:119]
PaddlePaddle Call Stacks: 
0       0x7f82064e0628p paddle::platform::GpuMaxChunkSize() + 5080
1       0x7f820421496dp paddle::memory::GetGPUBuddyAllocator(int) + 253
2       0x7f8204214b3bp void* paddle::memory::Alloc<paddle::platform::CUDAPlace>(paddle::platform::CUDAPlace, unsigned long) + 43
3       0x7f820416fdc8p paddle::framework::Tensor::PlaceholderImpl<paddle::platform::CUDAPlace>::PlaceholderImpl(paddle::platform::CUDAPlace, unsigned long, std::type_index) + 72
4       0x7f82041766a0p paddle::framework::Tensor::mutable_data(boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_>, std::type_index) + 336
5       0x7f82044af2b9p paddle::operators::FillConstantOp::RunImpl(paddle::framework::Scope const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&) const + 1177
6       0x7f82042186bep paddle::framework::Executor::RunPreparedContext(paddle::framework::ExecutorPrepareContext*, paddle::framework::Scope*, bool, bool) + 414
7       0x7f8204219a57p paddle::framework::Executor::Run(paddle::framework::ProgramDesc const&, paddle::framework::Scope*, int, bool, bool) + 103
8       0x7f820418a253p void pybind11::cpp_function::initialize<pybind11::cpp_function::initialize<void, paddle::framework::Executor, paddle::framework::ProgramDesc const&, paddle::framework::Scope*, int, bool, bool, pybind11::name, pybind11::is_method, pybind11::sibling>(void (paddle::framework::Executor::*)(paddle::framework::ProgramDesc const&, paddle::framework::Scope*, int, bool, bool), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&)::{lambda(paddle::framework::Executor*, paddle::framework::ProgramDesc const&, paddle::framework::Scope*, int, bool, bool)#1}, void, paddle::framework::Executor*, paddle::framework::ProgramDesc const&, paddle::framework::Scope*, int, bool, bool, pybind11::name, pybind11::is_method, pybind11::sibling>(pybind11::cpp_function::initialize<void, paddle::framework::Executor, paddle::framework::ProgramDesc const&, paddle::framework::Scope*, int, bool, bool, pybind11::name, pybind11::is_method, pybind11::sibling>(void (paddle::framework::Executor::*)(paddle::framework::ProgramDesc const&, paddle::framework::Scope*, int, bool, bool), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&)::{lambda(paddle::framework::Executor*, paddle::framework::ProgramDesc const&, paddle::framework::Scope*, int, bool, bool)#1}&&, void (*)(paddle::framework::Executor*, paddle::framework::ProgramDesc const&, paddle::framework::Scope*, int, bool, bool), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&)::{lambda(pybind11::detail::function_call&)#3}::_FUN(pybind11::detail::function_call) + 579
9       0x7f8204186e14p pybind11::cpp_function::dispatcher(_object*, _object*, _object*) + 1236
10            0x4c37edp PyEval_EvalFrameEx + 31165
11            0x4b9ab6p PyEval_EvalCodeEx + 774
12            0x4c16e7p PyEval_EvalFrameEx + 22711
13            0x4b9ab6p PyEval_EvalCodeEx + 774
14            0x4c1e6fp PyEval_EvalFrameEx + 24639
15            0x4b9ab6p PyEval_EvalCodeEx + 774
16            0x4c16e7p PyEval_EvalFrameEx + 22711
17            0x4b9ab6p PyEval_EvalCodeEx + 774
18            0x4c16e7p PyEval_EvalFrameEx + 22711
19            0x4b9ab6p PyEval_EvalCodeEx + 774
20            0x4eb30fp
21            0x4e5422p PyRun_FileExFlags + 130
22            0x4e3cd6p PyRun_SimpleFileExFlags + 390
23            0x493ae2p Py_Main + 1554
24      0x7f823b87c830p __libc_start_main + 240
25            0x4933e9p _start + 41

ENDING TIMING RUN AT 2018-05-14 01:35:19 AM
RESULT,sentiment,1,1138,,2018-05-14 01:16:21 AM

The key error is that:
paddle.fluid.core.EnforceNotMet: enforce allocating <= available failed, 10484135494 > 8700690176

I tried to figure out the answer for this problem, with @luotao1, @rulai-huiyingl said in paddlepaddle issue 9230, paddlepaddle issue 6725, however they are no effect on me.

Question2:

Every time I restart the docker iamge of sidgoyal78/paddle:benchmark12042018 It always re-download the dataset:

Cache file /root/.cache/paddle/dataset/imdb/aclImdb_v1.tar.gz not found, downloading http://ai.stanford.edu/%7Eamaas/data/sentiment/aclImdb_v1.tar.gz
[==================================================]

How Can I save the dataset on my host machine?

Some questions about the user guide and rules

Hi,
I have some questions after going through the user guide and website descriptions.
• Can we submit for only few benchmarks instead of suit?
• Can we have multiple submissions of a set of benchmarks using different frameworks?
• What does it mean by "benchmark detection is not allowed"?
• What is a division result?
• What about converting data set in desired format of framework, e.g., lmdb for caffe2, TFRecord for tensorflow, etc? Is it allowed in Closed Benchmark? If yes, how the wall clock will be computed for this? Do we have to include this for each training run?

It may be a good idea to make a FAQ of questions like this if you think one is useful.

Benchmarks are run on Python 2 or 3, CPython or Pypy?

https://github.com/mlperf/reference#running-benchmarks should explain which implementation and version of Python these benchmarks will be run.

Does MLPerf target both training and inference performance or training only?

"This benchmark uses a RNN to classify images", it should be CNN instead.

Thank you for putting this up.
The very first sentence surprised me when I saw "RNN". It may confuse others as well.