That I can start the translation workload.
After start the workload, I find that the workload use much of GPU memory, which results an error:
root@975f79997b6d:~/transformer# ./run_and_time.sh 1 | tee benchmark-date "+%F-%T".log
STARTING TIMING RUN AT 2018-05-16 02:38:30 PM
running benchmark with seed 1
INFO:tensorflow:Step 1/4: Downloading data from source
INFO:tensorflow:Already downloaded and extracted http://data.statmt.org/wmt17/translation-task/training-parallel-nc-v12.tgz.
INFO:tensorflow:Already downloaded and extracted http://www.statmt.org/wmt13/training-parallel-commoncrawl.tgz.
INFO:tensorflow:Already downloaded and extracted http://www.statmt.org/wmt13/training-parallel-europarl-v7.tgz.
INFO:tensorflow:Already downloaded and extracted http://data.statmt.org/wmt17/translation-task/dev.tgz.
INFO:tensorflow:Step 2/4: Creating subtokenizer and building vocabulary
INFO:tensorflow:Vocab file already exists (processed_data/vocab.ende.32768)
INFO:tensorflow:Initializing Subtokenizer from file processed_data/vocab.ende.32768.
INFO:tensorflow:Step 3/4: Compiling training and evaluation data
INFO:tensorflow:Compiling files with tag train.
INFO:tensorflow:Reading files /raw_data/training/news-commentary-v12.de-en.en and /raw_data/training/news-commentary-v12.de-en.de.
INFO:tensorflow:Reading files /raw_data/commoncrawl.de-en.en and /raw_data/commoncrawl.de-en.de.
INFO:tensorflow:Reading files /raw_data/training/europarl-v7.de-en.en and /raw_data/training/europarl-v7.de-en.de.
INFO:tensorflow:Compiling files with tag dev.
INFO:tensorflow:Reading files /raw_data/dev/newstest2013.en and /raw_data/dev/newstest2013.de.
INFO:tensorflow:Step 4/4: Preprocessing and saving data
INFO:tensorflow:Files with tag train already exist.
INFO:tensorflow:Files with tag dev already exist.
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00001-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00002-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00003-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00004-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00005-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00006-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00007-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00008-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00009-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00010-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00011-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00012-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00013-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00014-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00015-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00016-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00017-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00018-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00019-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00020-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00021-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00022-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00023-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00024-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00025-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00026-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00027-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00028-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00029-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00030-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00031-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00032-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00033-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00034-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00035-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00036-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00037-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00038-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00039-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00040-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00041-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00042-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00043-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00044-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00045-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00046-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00047-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00048-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00049-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00050-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00051-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00052-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00053-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00054-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00055-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00056-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00057-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00058-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00059-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00060-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00061-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00062-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00063-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00064-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00065-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00066-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00067-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00068-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00069-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00070-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00071-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00072-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00073-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00074-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00075-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00076-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00077-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00078-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00079-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00080-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00081-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00082-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00083-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00084-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00085-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00086-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00087-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00088-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00089-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00090-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00091-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00092-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00093-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00094-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00095-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00096-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00097-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00098-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00099-of-00100
INFO:tensorflow:Shuffling records in file processed_data/wmt32k-train-00100-of-00100
INFO:tensorflow:Using default config.
INFO:tensorflow:Using config: {'_master': '', '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7fe98e188c88>, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_model_dir': 'model', '_device_fn': None, '_is_chief': True, '_global_id_in_cluster': 0, '_num_worker_replicas': 1, '_log_step_count_steps': 100, '_evaluation_master': '', '_service': None, '_tf_random_seed': None, '_keep_checkpoint_max': 5, '_num_ps_replicas': 0, '_task_id': 0, '_train_distribute': None, '_save_summary_steps': 100, '_keep_checkpoint_every_n_hours': 10000, '_session_config': None, '_task_type': 'worker'}
INFO:tensorflow:Calling model_fn.
/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gradients_impl.py:100: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.
"Converting sparse IndexedSlices to a dense Tensor of unknown shape. "
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Graph was finalized.
2018-05-16 14:41:45.417676: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2018-05-16 14:41:57.526465: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1349] Found device 0 with properties:
name: Tesla K80 major: 3 minor: 7 memoryClockRate(GHz): 0.8235
pciBusID: 0000:85:00.0
totalMemory: 11.17GiB freeMemory: 11.10GiB
2018-05-16 14:41:57.928098: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1349] Found device 1 with properties:
name: Tesla K80 major: 3 minor: 7 memoryClockRate(GHz): 0.8235
pciBusID: 0000:86:00.0
totalMemory: 11.17GiB freeMemory: 11.10GiB
2018-05-16 14:41:58.337921: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1349] Found device 2 with properties:
name: Tesla K80 major: 3 minor: 7 memoryClockRate(GHz): 0.8235
pciBusID: 0000:8c:00.0
totalMemory: 11.17GiB freeMemory: 11.10GiB
2018-05-16 14:41:58.767041: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1349] Found device 3 with properties:
name: Tesla K80 major: 3 minor: 7 memoryClockRate(GHz): 0.8235
pciBusID: 0000:8d:00.0
totalMemory: 11.17GiB freeMemory: 11.10GiB
2018-05-16 14:41:58.769543: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1428] Adding visible gpu devices: 0, 1, 2, 3
2018-05-16 14:41:59.984340: I tensorflow/core/common_runtime/gpu/gpu_device.cc:916] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-05-16 14:41:59.984407: I tensorflow/core/common_runtime/gpu/gpu_device.cc:922] 0 1 2 3
2018-05-16 14:41:59.984425: I tensorflow/core/common_runtime/gpu/gpu_device.cc:935] 0: N Y Y Y
2018-05-16 14:41:59.984442: I tensorflow/core/common_runtime/gpu/gpu_device.cc:935] 1: Y N Y Y
2018-05-16 14:41:59.984480: I tensorflow/core/common_runtime/gpu/gpu_device.cc:935] 2: Y Y N Y
2018-05-16 14:41:59.984497: I tensorflow/core/common_runtime/gpu/gpu_device.cc:935] 3: Y Y Y N
2018-05-16 14:41:59.985810: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1046] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10762 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:85:00.0, compute capability: 3.7)
2018-05-16 14:42:00.120836: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1046] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 10762 MB memory) -> physical GPU (device: 1, name: Tesla K80, pci bus id: 0000:86:00.0, compute capability: 3.7)
2018-05-16 14:42:00.235081: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1046] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 10762 MB memory) -> physical GPU (device: 2, name: Tesla K80, pci bus id: 0000:8c:00.0, compute capability: 3.7)
2018-05-16 14:42:00.350005: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1046] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 10762 MB memory) -> physical GPU (device: 3, name: Tesla K80, pci bus id: 0000:8d:00.0, compute capability: 3.7)
INFO:tensorflow:Restoring parameters from model/model.ckpt-0
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Saving checkpoints for 0 into model/model.ckpt.
INFO:tensorflow:step = 0, loss = 9.890151
2018-05-16 14:46:51.976901: W tensorflow/core/common_runtime/bfc_allocator.cc:275] Allocator (GPU_0_bfc) ran out of memory trying to allocate 511.51MiB. Current allocation summary follows.
2018-05-16 14:46:51.977166: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (256): Total Chunks: 183, Chunks in use: 181. 45.8KiB allocated for chunks. 45.2KiB in use in bin. 764B client-requested in use in bin.
2018-05-16 14:46:51.977216: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (512): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-05-16 14:46:51.977254: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (1024): Total Chunks: 1, Chunks in use: 1. 1.2KiB allocated for chunks. 1.2KiB in use in bin. 1.0KiB client-requested in use in bin.
2018-05-16 14:46:51.977291: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (2048): Total Chunks: 2, Chunks in use: 2. 5.2KiB allocated for chunks. 5.2KiB in use in bin. 4.0KiB client-requested in use in bin.
2018-05-16 14:46:51.977330: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (4096): Total Chunks: 228, Chunks in use: 228. 916.2KiB allocated for chunks. 916.2KiB in use in bin. 912.0KiB client-requested in use in bin.
2018-05-16 14:46:51.977369: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (8192): Total Chunks: 43, Chunks in use: 42. 661.5KiB allocated for chunks. 646.5KiB in use in bin. 638.8KiB client-requested in use in bin.
2018-05-16 14:46:51.977406: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (16384): Total Chunks: 38, Chunks in use: 38. 653.8KiB allocated for chunks. 653.8KiB in use in bin. 604.8KiB client-requested in use in bin.
2018-05-16 14:46:51.977444: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (32768): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-05-16 14:46:51.977486: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (65536): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-05-16 14:46:51.977523: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (131072): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-05-16 14:46:51.977560: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (262144): Total Chunks: 4, Chunks in use: 4. 1.17MiB allocated for chunks. 1.17MiB in use in bin. 952.8KiB client-requested in use in bin.
2018-05-16 14:46:51.977596: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (524288): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-05-16 14:46:51.977631: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (1048576): Total Chunks: 1, Chunks in use: 0. 1.95MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-05-16 14:46:51.977665: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (2097152): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-05-16 14:46:51.977704: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (4194304): Total Chunks: 219, Chunks in use: 216. 883.23MiB allocated for chunks. 871.23MiB in use in bin. 864.00MiB client-requested in use in bin.
2018-05-16 14:46:51.977740: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (8388608): Total Chunks: 199, Chunks in use: 195. 2.99GiB allocated for chunks. 2.95GiB in use in bin. 2.95GiB client-requested in use in bin.
2018-05-16 14:46:51.977777: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (16777216): Total Chunks: 131, Chunks in use: 131. 2.62GiB allocated for chunks. 2.62GiB in use in bin. 2.51GiB client-requested in use in bin.
2018-05-16 14:46:51.977812: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (33554432): Total Chunks: 36, Chunks in use: 36. 2.02GiB allocated for chunks. 2.02GiB in use in bin. 2.02GiB client-requested in use in bin.
2018-05-16 14:46:51.977847: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (67108864): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-05-16 14:46:51.977885: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (134217728): Total Chunks: 5, Chunks in use: 5. 658.42MiB allocated for chunks. 658.42MiB in use in bin. 658.41MiB client-requested in use in bin.
2018-05-16 14:46:51.977922: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (268435456): Total Chunks: 3, Chunks in use: 2. 1.36GiB allocated for chunks. 1023.03MiB in use in bin. 1023.03MiB client-requested in use in bin.
2018-05-16 14:46:51.977960: I tensorflow/core/common_runtime/bfc_allocator.cc:646] Bin for 511.51MiB was 256.00MiB, Chunk State:
2018-05-16 14:46:51.978005: I tensorflow/core/common_runtime/bfc_allocator.cc:652] Size: 373.75MiB | Requested Size: 12B | in_use: 0, prev: Size: 511.51MiB | Requested Size: 511.51MiB | in_use: 1
2018-05-16 14:46:51.978044: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2314700000 of size 1280
2018-05-16 14:46:51.978078: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2314700500 of size 4096
2018-05-16 14:46:51.978109: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2314701500 of size 4096
2018-05-16 14:46:51.978140: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2314702500 of size 4096
2018-05-16 14:46:51.978169: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2314703500 of size 4096
2018-05-16 14:46:51.978202: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2314704500 of size 16781312
2018-05-16 14:46:51.978232: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2315705500 of size 256
2018-05-16 14:46:51.978265: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2315705600 of size 4352
2018-05-16 14:46:51.978295: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2315706700 of size 4096
2018-05-16 14:46:51.978326: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2315707700 of size 4194304
2018-05-16 14:46:51.978358: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2315b07700 of size 4194304
2018-05-16 14:46:51.978388: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2315f07700 of size 8379904
2018-05-16 14:46:51.978418: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2316705500 of size 4096
2018-05-16 14:46:51.978449: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2316706500 of size 4096
2018-05-16 14:46:51.978479: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2316707500 of size 4096
2018-05-16 14:46:51.978509: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2316708500 of size 4194304
2018-05-16 14:46:51.978539: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2316b08500 of size 4096
2018-05-16 14:46:51.978570: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2316b09500 of size 4096
2018-05-16 14:46:51.978628: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2316b0a500 of size 4194304
2018-05-16 14:46:51.978660: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2316f0a500 of size 4096
2018-05-16 14:46:51.978690: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2316f0b500 of size 16777216
2018-05-16 14:46:51.978721: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2317f0b500 of size 4096
2018-05-16 14:46:51.978752: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2317f0c500 of size 4096
2018-05-16 14:46:51.978782: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2317f0d500 of size 4096
2018-05-16 14:46:51.978813: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2317f0e500 of size 4096
2018-05-16 14:46:51.978844: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2317f0f500 of size 4096
2018-05-16 14:46:51.978873: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2317f10500 of size 4096
2018-05-16 14:46:51.978904: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2317f11500 of size 4096
2018-05-16 14:46:51.978933: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2317f12500 of size 4096
2018-05-16 14:46:51.978964: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2317f13500 of size 4198400
2018-05-16 14:46:51.979000: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318314500 of size 4194304
2018-05-16 14:46:51.979031: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318714500 of size 4194304
2018-05-16 14:46:51.979061: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318b14500 of size 4194304
2018-05-16 14:46:51.979092: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f14500 of size 4096
2018-05-16 14:46:51.979122: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f15500 of size 4096
2018-05-16 14:46:51.979152: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f16500 of size 4096
2018-05-16 14:46:51.979182: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f17500 of size 4096
2018-05-16 14:46:51.979214: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f18500 of size 24576
2018-05-16 14:46:51.979245: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f1e500 of size 4096
2018-05-16 14:46:51.979276: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f1f500 of size 4096
2018-05-16 14:46:51.979305: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f20500 of size 4096
2018-05-16 14:46:51.979336: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f21500 of size 4096
2018-05-16 14:46:51.979367: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f22500 of size 4096
2018-05-16 14:46:51.979397: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f23500 of size 4096
2018-05-16 14:46:51.979427: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f24500 of size 4096
2018-05-16 14:46:51.979458: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f25500 of size 4096
2018-05-16 14:46:51.979488: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f26500 of size 4096
2018-05-16 14:46:51.979517: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f27500 of size 4096
2018-05-16 14:46:51.979548: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f28500 of size 4096
2018-05-16 14:46:51.979578: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f29500 of size 4096
2018-05-16 14:46:51.979608: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f2a500 of size 4096
2018-05-16 14:46:51.979638: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f2b500 of size 4096
2018-05-16 14:46:51.979669: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f2c500 of size 16384
2018-05-16 14:46:51.979699: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f30500 of size 4096
2018-05-16 14:46:51.979730: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f31500 of size 4096
2018-05-16 14:46:51.979761: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f32500 of size 4096
2018-05-16 14:46:51.979792: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f33500 of size 4096
2018-05-16 14:46:51.979825: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f34500 of size 4096
2018-05-16 14:46:51.979855: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f35500 of size 4096
2018-05-16 14:46:51.979885: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f36500 of size 4096
2018-05-16 14:46:51.979916: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f37500 of size 4096
2018-05-16 14:46:51.979947: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f38500 of size 4096
2018-05-16 14:46:51.979978: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f39500 of size 4096
2018-05-16 14:46:51.980007: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f3a500 of size 4096
2018-05-16 14:46:51.980038: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f3b500 of size 4096
2018-05-16 14:46:51.980068: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f3c500 of size 4096
2018-05-16 14:46:51.980098: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f3d500 of size 4096
2018-05-16 14:46:51.980128: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f3e500 of size 4096
2018-05-16 14:46:51.980157: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f3f500 of size 4096
2018-05-16 14:46:51.980189: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f40500 of size 4096
2018-05-16 14:46:51.980220: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f41500 of size 4096
2018-05-16 14:46:51.980250: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f42500 of size 4096
2018-05-16 14:46:51.980281: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f43500 of size 16384
2018-05-16 14:46:51.980312: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f47500 of size 20480
2018-05-16 14:46:51.980343: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f4c500 of size 4096
2018-05-16 14:46:51.980374: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f4d500 of size 4096
2018-05-16 14:46:51.980405: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f4e500 of size 4096
2018-05-16 14:46:51.980436: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f4f500 of size 4096
2018-05-16 14:46:51.980467: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f50500 of size 4096
2018-05-16 14:46:51.980497: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f51500 of size 4096
2018-05-16 14:46:51.980527: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f52500 of size 4096
2018-05-16 14:46:51.980557: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f53500 of size 4096
2018-05-16 14:46:51.980587: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f54500 of size 4096
2018-05-16 14:46:51.980617: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f55500 of size 4096
2018-05-16 14:46:51.980649: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f56500 of size 4096
2018-05-16 14:46:51.980679: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f57500 of size 4096
2018-05-16 14:46:51.980711: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f58500 of size 20480
2018-05-16 14:46:51.980744: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f5d500 of size 16384
2018-05-16 14:46:51.980772: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f61500 of size 4096
2018-05-16 14:46:51.980803: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f62500 of size 4096
2018-05-16 14:46:51.980834: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f63500 of size 16384
2018-05-16 14:46:51.980865: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f67500 of size 4096
2018-05-16 14:46:51.980895: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f68500 of size 4096
2018-05-16 14:46:51.980935: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f69500 of size 4096
2018-05-16 14:46:51.980986: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f6a500 of size 4096
2018-05-16 14:46:51.981020: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f6b500 of size 4352
2018-05-16 14:46:51.981050: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f6c600 of size 4096
2018-05-16 14:46:51.981080: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f6d600 of size 4096
2018-05-16 14:46:51.981112: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f6e600 of size 4096
2018-05-16 14:46:51.981143: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f6f600 of size 16384
2018-05-16 14:46:51.981173: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f73600 of size 4096
2018-05-16 14:46:51.981203: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f74600 of size 4096
2018-05-16 14:46:51.981233: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f75600 of size 4096
2018-05-16 14:46:51.981263: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f76600 of size 4096
2018-05-16 14:46:51.981292: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f77600 of size 4096
2018-05-16 14:46:51.981324: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f78600 of size 20480
2018-05-16 14:46:51.981355: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f7d600 of size 4096
2018-05-16 14:46:51.981385: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f7e600 of size 4096
2018-05-16 14:46:51.981415: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f7f600 of size 4096
2018-05-16 14:46:51.981446: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f80600 of size 4096
2018-05-16 14:46:51.981476: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f81600 of size 4096
2018-05-16 14:46:51.981506: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f82600 of size 4096
2018-05-16 14:46:51.981536: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f83600 of size 4096
2018-05-16 14:46:51.981566: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f84600 of size 4096
2018-05-16 14:46:51.981597: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f85600 of size 4096
2018-05-16 14:46:51.981629: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f86600 of size 4096
2018-05-16 14:46:51.981659: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f87600 of size 4096
2018-05-16 14:46:51.981689: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f88600 of size 4096
2018-05-16 14:46:51.981719: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f89600 of size 4096
2018-05-16 14:46:51.981752: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f8a600 of size 24576
2018-05-16 14:46:51.981783: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f90600 of size 4096
2018-05-16 14:46:51.981814: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f91600 of size 4096
2018-05-16 14:46:51.981843: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f92600 of size 4096
2018-05-16 14:46:51.981873: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f93600 of size 4096
2018-05-16 14:46:51.981904: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f94600 of size 4096
2018-05-16 14:46:51.981934: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f95600 of size 4096
2018-05-16 14:46:51.981964: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f96600 of size 4096
2018-05-16 14:46:51.981994: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f97600 of size 4096
2018-05-16 14:46:51.982022: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f98600 of size 4096
2018-05-16 14:46:51.982053: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f99600 of size 4096
2018-05-16 14:46:51.982083: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f9a600 of size 4096
2018-05-16 14:46:51.982112: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f9b600 of size 4096
2018-05-16 14:46:51.982140: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f9c600 of size 4096
2018-05-16 14:46:51.982170: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f9d600 of size 4096
2018-05-16 14:46:51.982201: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f9e600 of size 4096
2018-05-16 14:46:51.982230: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318f9f600 of size 4096
2018-05-16 14:46:51.982260: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fa0600 of size 4096
2018-05-16 14:46:51.982291: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fa1600 of size 4096
2018-05-16 14:46:51.982321: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fa2600 of size 4096
2018-05-16 14:46:51.982352: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fa3600 of size 4096
2018-05-16 14:46:51.982382: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fa4600 of size 256
2018-05-16 14:46:51.982414: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fa4700 of size 7936
2018-05-16 14:46:51.982445: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fa6600 of size 4096
2018-05-16 14:46:51.982474: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fa7600 of size 4096
2018-05-16 14:46:51.982504: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fa8600 of size 4096
2018-05-16 14:46:51.982534: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fa9600 of size 4096
2018-05-16 14:46:51.982565: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318faa600 of size 16384
2018-05-16 14:46:51.982607: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fae600 of size 256
2018-05-16 14:46:51.982642: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fae700 of size 256
2018-05-16 14:46:51.982673: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fae800 of size 256
2018-05-16 14:46:51.982702: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fae900 of size 256
2018-05-16 14:46:51.982733: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318faea00 of size 256
2018-05-16 14:46:51.982764: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318faeb00 of size 256
2018-05-16 14:46:51.982794: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318faec00 of size 256
2018-05-16 14:46:51.982825: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318faed00 of size 256
2018-05-16 14:46:51.982854: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318faee00 of size 256
2018-05-16 14:46:51.982884: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318faef00 of size 256
2018-05-16 14:46:51.982915: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318faf000 of size 256
2018-05-16 14:46:51.982945: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318faf100 of size 256
2018-05-16 14:46:51.982976: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318faf200 of size 256
2018-05-16 14:46:51.983007: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318faf300 of size 256
2018-05-16 14:46:51.983037: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318faf400 of size 256
2018-05-16 14:46:51.983068: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318faf500 of size 256
2018-05-16 14:46:51.983098: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318faf600 of size 4096
2018-05-16 14:46:51.983129: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fb0600 of size 4096
2018-05-16 14:46:51.983159: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fb1600 of size 4096
2018-05-16 14:46:51.983189: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fb2600 of size 4096
2018-05-16 14:46:51.983218: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fb3600 of size 256
2018-05-16 14:46:51.983248: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fb3700 of size 256
2018-05-16 14:46:51.983278: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fb3800 of size 256
2018-05-16 14:46:51.983310: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fb3900 of size 256
2018-05-16 14:46:51.983341: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fb3a00 of size 256
2018-05-16 14:46:51.983372: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fb3b00 of size 2816
2018-05-16 14:46:51.983403: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fb4600 of size 4096
2018-05-16 14:46:51.983433: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fb5600 of size 4096
2018-05-16 14:46:51.983463: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fb6600 of size 4096
2018-05-16 14:46:51.983492: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fb7600 of size 4096
2018-05-16 14:46:51.983521: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fb8600 of size 4096
2018-05-16 14:46:51.983551: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fb9600 of size 4096
2018-05-16 14:46:51.983580: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fba600 of size 256
2018-05-16 14:46:51.983610: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fba700 of size 256
2018-05-16 14:46:51.983641: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fba800 of size 256
2018-05-16 14:46:51.983671: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fba900 of size 256
2018-05-16 14:46:51.983701: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fbaa00 of size 256
2018-05-16 14:46:51.983730: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fbab00 of size 256
2018-05-16 14:46:51.983760: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fbac00 of size 256
2018-05-16 14:46:51.983790: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fbad00 of size 256
2018-05-16 14:46:51.983819: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fbae00 of size 256
2018-05-16 14:46:51.983849: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fbaf00 of size 256
2018-05-16 14:46:51.983885: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fbb000 of size 256
2018-05-16 14:46:51.983915: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fbb100 of size 256
2018-05-16 14:46:51.983945: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fbb200 of size 256
2018-05-16 14:46:51.983989: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fbb300 of size 256
2018-05-16 14:46:51.984021: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fbb400 of size 256
2018-05-16 14:46:51.984051: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fbb500 of size 256
2018-05-16 14:46:51.984082: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fbb600 of size 4096
2018-05-16 14:46:51.984112: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fbc600 of size 4096
2018-05-16 14:46:51.984142: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fbd600 of size 256
2018-05-16 14:46:51.984173: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fbd700 of size 256
2018-05-16 14:46:51.984203: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fbd800 of size 256
2018-05-16 14:46:51.984233: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fbd900 of size 256
2018-05-16 14:46:51.984263: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fbda00 of size 256
2018-05-16 14:46:51.984295: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fbdb00 of size 256
2018-05-16 14:46:51.984324: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fbdc00 of size 256
2018-05-16 14:46:51.984355: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fbdd00 of size 256
2018-05-16 14:46:51.984386: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fbde00 of size 256
2018-05-16 14:46:51.984416: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fbdf00 of size 256
2018-05-16 14:46:51.984445: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fbe000 of size 256
2018-05-16 14:46:51.984477: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fbe100 of size 256
2018-05-16 14:46:51.984507: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fbe200 of size 256
2018-05-16 14:46:51.984537: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fbe300 of size 256
2018-05-16 14:46:51.984567: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fbe400 of size 256
2018-05-16 14:46:51.984597: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fbe500 of size 256
2018-05-16 14:46:51.984628: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fbe600 of size 4096
2018-05-16 14:46:51.984659: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fbf600 of size 16384
2018-05-16 14:46:51.984689: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc3600 of size 256
2018-05-16 14:46:51.984719: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc3700 of size 256
2018-05-16 14:46:51.984749: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc3800 of size 256
2018-05-16 14:46:51.984780: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc3900 of size 256
2018-05-16 14:46:51.984810: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc3a00 of size 256
2018-05-16 14:46:51.984839: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc3b00 of size 256
2018-05-16 14:46:51.984869: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc3c00 of size 256
2018-05-16 14:46:51.984900: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc3d00 of size 256
2018-05-16 14:46:51.984929: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc3e00 of size 256
2018-05-16 14:46:51.984960: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc3f00 of size 256
2018-05-16 14:46:51.984990: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc4000 of size 256
2018-05-16 14:46:51.985020: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc4100 of size 256
2018-05-16 14:46:51.985050: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc4200 of size 256
2018-05-16 14:46:51.985080: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc4300 of size 256
2018-05-16 14:46:51.985111: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc4400 of size 256
2018-05-16 14:46:51.985140: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc4500 of size 256
2018-05-16 14:46:51.985170: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc4600 of size 4096
2018-05-16 14:46:51.985201: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc5600 of size 256
2018-05-16 14:46:51.985230: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc5700 of size 256
2018-05-16 14:46:51.985260: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc5800 of size 256
2018-05-16 14:46:51.985291: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc5900 of size 256
2018-05-16 14:46:51.985320: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc5a00 of size 256
2018-05-16 14:46:51.985351: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc5b00 of size 256
2018-05-16 14:46:51.985381: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc5c00 of size 2560
2018-05-16 14:46:51.985413: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc6600 of size 4096
2018-05-16 14:46:51.985443: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc7600 of size 256
2018-05-16 14:46:51.985473: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc7700 of size 256
2018-05-16 14:46:51.985503: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc7800 of size 256
2018-05-16 14:46:51.985534: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc7900 of size 256
2018-05-16 14:46:51.985564: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc7a00 of size 256
2018-05-16 14:46:51.985595: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc7b00 of size 256
2018-05-16 14:46:51.985625: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc7c00 of size 256
2018-05-16 14:46:51.985654: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc7d00 of size 256
2018-05-16 14:46:51.985685: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc7e00 of size 256
2018-05-16 14:46:51.985716: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc7f00 of size 256
2018-05-16 14:46:51.985746: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc8000 of size 256
2018-05-16 14:46:51.985777: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc8100 of size 256
2018-05-16 14:46:51.985808: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc8200 of size 256
2018-05-16 14:46:51.985839: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc8300 of size 256
2018-05-16 14:46:51.985871: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc8400 of size 256
2018-05-16 14:46:51.985901: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc8500 of size 256
2018-05-16 14:46:51.985931: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc8600 of size 4096
2018-05-16 14:46:51.985962: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fc9600 of size 4096
2018-05-16 14:46:51.985992: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fca600 of size 20480
2018-05-16 14:46:51.986021: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fcf600 of size 4096
2018-05-16 14:46:51.986052: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd0600 of size 256
2018-05-16 14:46:51.986083: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd0700 of size 256
2018-05-16 14:46:51.986113: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd0800 of size 256
2018-05-16 14:46:51.986143: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x2318fd0900 of size 256
2018-05-16 14:46:51.986173: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd0a00 of size 256
2018-05-16 14:46:51.986203: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd0b00 of size 256
2018-05-16 14:46:51.986233: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd0c00 of size 256
2018-05-16 14:46:51.986263: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd0d00 of size 256
2018-05-16 14:46:51.986294: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd0e00 of size 256
2018-05-16 14:46:51.986324: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd0f00 of size 256
2018-05-16 14:46:51.986354: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd1000 of size 256
2018-05-16 14:46:51.986384: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd1100 of size 256
2018-05-16 14:46:51.986413: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd1200 of size 256
2018-05-16 14:46:51.986443: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd1300 of size 256
2018-05-16 14:46:51.986473: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd1400 of size 256
2018-05-16 14:46:51.986502: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd1500 of size 256
2018-05-16 14:46:51.986531: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd1600 of size 256
2018-05-16 14:46:51.986561: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd1700 of size 256
2018-05-16 14:46:51.986590: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd1800 of size 256
2018-05-16 14:46:51.986630: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd1900 of size 256
2018-05-16 14:46:51.986661: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd1a00 of size 256
2018-05-16 14:46:51.986691: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd1b00 of size 256
2018-05-16 14:46:51.986721: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd1c00 of size 256
2018-05-16 14:46:51.986752: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd1d00 of size 256
2018-05-16 14:46:51.986782: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd1e00 of size 256
2018-05-16 14:46:51.986812: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd1f00 of size 256
2018-05-16 14:46:51.986842: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd2000 of size 256
2018-05-16 14:46:51.986871: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd2100 of size 256
2018-05-16 14:46:51.986901: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd2200 of size 256
2018-05-16 14:46:51.986931: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd2300 of size 256
2018-05-16 14:46:51.986962: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd2400 of size 256
2018-05-16 14:46:51.986993: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd2500 of size 256
2018-05-16 14:46:51.987022: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd2600 of size 4096
2018-05-16 14:46:51.987056: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd3600 of size 4096
2018-05-16 14:46:51.987086: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd4600 of size 256
2018-05-16 14:46:51.987115: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd4700 of size 256
2018-05-16 14:46:51.987151: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd4800 of size 256
2018-05-16 14:46:51.987182: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd4900 of size 256
2018-05-16 14:46:51.987214: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd4a00 of size 256
2018-05-16 14:46:51.987244: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd4b00 of size 256
2018-05-16 14:46:51.987274: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd4c00 of size 256
2018-05-16 14:46:51.987305: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd4d00 of size 256
2018-05-16 14:46:51.987335: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd4e00 of size 256
2018-05-16 14:46:51.987365: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd4f00 of size 256
2018-05-16 14:46:51.987395: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd5000 of size 256
2018-05-16 14:46:51.987425: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd5100 of size 256
2018-05-16 14:46:51.987455: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd5200 of size 256
2018-05-16 14:46:51.987484: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd5300 of size 256
2018-05-16 14:46:51.987515: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd5400 of size 256
2018-05-16 14:46:51.987545: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd5500 of size 256
2018-05-16 14:46:51.987578: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd5600 of size 16384
2018-05-16 14:46:51.987608: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fd9600 of size 4096
2018-05-16 14:46:51.987638: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fda600 of size 256
2018-05-16 14:46:51.987669: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fda700 of size 256
2018-05-16 14:46:51.987699: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fda800 of size 256
2018-05-16 14:46:51.987729: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fda900 of size 256
2018-05-16 14:46:51.987759: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fdaa00 of size 256
2018-05-16 14:46:51.987793: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fdab00 of size 256
2018-05-16 14:46:51.987824: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fdac00 of size 256
2018-05-16 14:46:51.987854: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fdad00 of size 256
2018-05-16 14:46:51.987883: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fdae00 of size 256
2018-05-16 14:46:51.987913: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fdaf00 of size 256
2018-05-16 14:46:51.987944: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fdb000 of size 256
2018-05-16 14:46:51.987974: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fdb100 of size 256
2018-05-16 14:46:51.988004: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fdb200 of size 256
2018-05-16 14:46:51.988033: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fdb300 of size 256
2018-05-16 14:46:51.988063: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fdb400 of size 256
2018-05-16 14:46:51.988093: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fdb500 of size 256
2018-05-16 14:46:51.988123: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fdb600 of size 256
2018-05-16 14:46:51.988154: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fdb700 of size 256
2018-05-16 14:46:51.988183: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fdb800 of size 256
2018-05-16 14:46:51.988213: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fdb900 of size 256
2018-05-16 14:46:51.988243: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fdba00 of size 256
2018-05-16 14:46:51.988274: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fdbb00 of size 256
2018-05-16 14:46:51.988303: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fdbc00 of size 256
2018-05-16 14:46:51.988335: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fdbd00 of size 256
2018-05-16 14:46:51.988365: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fdbe00 of size 256
2018-05-16 14:46:51.988395: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fdbf00 of size 256
2018-05-16 14:46:51.988426: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fdc000 of size 256
2018-05-16 14:46:51.988456: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fdc100 of size 256
2018-05-16 14:46:51.988485: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fdc200 of size 256
2018-05-16 14:46:51.988514: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fdc300 of size 256
2018-05-16 14:46:51.988544: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fdc400 of size 256
2018-05-16 14:46:51.988575: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fdc500 of size 256
2018-05-16 14:46:51.988605: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fdc600 of size 4096
2018-05-16 14:46:51.988637: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fdd600 of size 4096
2018-05-16 14:46:51.988667: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fde600 of size 16384
2018-05-16 14:46:51.988698: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2318fe2600 of size 7544576
2018-05-16 14:46:51.988728: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2319714500 of size 4194304
2018-05-16 14:46:51.988757: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2319b14500 of size 4096
2018-05-16 14:46:51.988788: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2319b15500 of size 4096
2018-05-16 14:46:51.988818: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2319b16500 of size 4194304
2018-05-16 14:46:51.988849: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2319f16500 of size 4096
2018-05-16 14:46:51.988879: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2319f17500 of size 4096
2018-05-16 14:46:51.988908: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2319f18500 of size 4096
2018-05-16 14:46:51.988938: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2319f19500 of size 4096
2018-05-16 14:46:51.988970: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2319f1a500 of size 4198400
2018-05-16 14:46:51.989000: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231a31b500 of size 4194304
2018-05-16 14:46:51.989030: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231a71b500 of size 16777216
2018-05-16 14:46:51.989060: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231b71b500 of size 4096
2018-05-16 14:46:51.989090: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231b71c500 of size 4194304
2018-05-16 14:46:51.989119: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231bb1c500 of size 4096
2018-05-16 14:46:51.989149: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231bb1d500 of size 4194304
2018-05-16 14:46:51.989179: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231bf1d500 of size 4194304
2018-05-16 14:46:51.989208: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231c31d500 of size 4194304
2018-05-16 14:46:51.989239: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231c71d500 of size 16384
2018-05-16 14:46:51.989268: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231c721500 of size 4194304
2018-05-16 14:46:51.989299: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231cb21500 of size 4194304
2018-05-16 14:46:51.989328: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231cf21500 of size 4194304
2018-05-16 14:46:51.989359: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231d321500 of size 4096
2018-05-16 14:46:51.989389: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231d322500 of size 4096
2018-05-16 14:46:51.989418: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231d323500 of size 4194304
2018-05-16 14:46:51.989448: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231d723500 of size 4096
2018-05-16 14:46:51.989479: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231d724500 of size 4194304
2018-05-16 14:46:51.989508: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231db24500 of size 4194304
2018-05-16 14:46:51.989537: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231df24500 of size 4194304
2018-05-16 14:46:51.989566: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231e324500 of size 4194304
2018-05-16 14:46:51.989597: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231e724500 of size 4194304
2018-05-16 14:46:51.989627: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231eb24500 of size 4096
2018-05-16 14:46:51.989657: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231eb25500 of size 16384
2018-05-16 14:46:51.989687: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231eb29500 of size 16777216
2018-05-16 14:46:51.989717: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x231fb29500 of size 16777216
2018-05-16 14:46:51.989746: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2320b29500 of size 4096
2018-05-16 14:46:51.989777: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2320b2a500 of size 4096
2018-05-16 14:46:51.989806: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2320b2b500 of size 256
2018-05-16 14:46:51.989837: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2320b2b600 of size 16781056
2018-05-16 14:46:51.989867: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2321b2c500 of size 4194304
2018-05-16 14:46:51.989896: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2321f2c500 of size 4194304
2018-05-16 14:46:51.989927: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232232c500 of size 4194304
2018-05-16 14:46:51.989957: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232272c500 of size 16384
2018-05-16 14:46:51.989986: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2322730500 of size 4194304
2018-05-16 14:46:51.990017: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2322b30500 of size 4194304
2018-05-16 14:46:51.990047: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2322f30500 of size 4194304
2018-05-16 14:46:51.990076: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2323330500 of size 4194304
2018-05-16 14:46:51.990106: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2323730500 of size 4194304
2018-05-16 14:46:51.990137: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2323b30500 of size 4194304
2018-05-16 14:46:51.990166: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2323f30500 of size 4194304
2018-05-16 14:46:51.990196: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324330500 of size 4194304
2018-05-16 14:46:51.990226: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324730500 of size 4194304
2018-05-16 14:46:51.990255: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324b30500 of size 4194304
2018-05-16 14:46:51.990284: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2324f30500 of size 16777216
2018-05-16 14:46:51.990314: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2325f30500 of size 16777216
2018-05-16 14:46:51.990344: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2326f30500 of size 4096
2018-05-16 14:46:51.990374: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2326f31500 of size 4096
2018-05-16 14:46:51.990404: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2326f32500 of size 4096
2018-05-16 14:46:51.990434: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2326f33500 of size 4096
2018-05-16 14:46:51.990464: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2326f34500 of size 16384
2018-05-16 14:46:51.990493: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2326f38500 of size 16777216
2018-05-16 14:46:51.990523: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2327f38500 of size 16777216
2018-05-16 14:46:51.990554: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2328f38500 of size 4096
2018-05-16 14:46:51.990583: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2328f39500 of size 4096
2018-05-16 14:46:51.990636: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2328f3a500 of size 4096
2018-05-16 14:46:51.990668: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2328f3b500 of size 4096
2018-05-16 14:46:51.990698: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2328f3c500 of size 4096
2018-05-16 14:46:51.990727: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2328f3d500 of size 4096
2018-05-16 14:46:51.990759: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2328f3e500 of size 4194304
2018-05-16 14:46:51.990789: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232933e500 of size 4194304
2018-05-16 14:46:51.990819: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232973e500 of size 4194304
2018-05-16 14:46:51.990850: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2329b3e500 of size 4194304
2018-05-16 14:46:51.990880: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2329f3e500 of size 4194304
2018-05-16 14:46:51.990910: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232a33e500 of size 4194304
2018-05-16 14:46:51.990940: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232a73e500 of size 4194304
2018-05-16 14:46:51.990991: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232ab3e500 of size 4206592
2018-05-16 14:46:51.991021: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232af41500 of size 4096
2018-05-16 14:46:51.991050: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232af42500 of size 4194304
2018-05-16 14:46:51.991081: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232b342500 of size 4194304
2018-05-16 14:46:51.991110: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232b742500 of size 4194304
2018-05-16 14:46:51.991140: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232bb42500 of size 4194304
2018-05-16 14:46:51.991169: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232bf42500 of size 16384
2018-05-16 14:46:51.991198: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232bf46500 of size 4096
2018-05-16 14:46:51.991228: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232bf47500 of size 4096
2018-05-16 14:46:51.991257: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232bf48500 of size 16384
2018-05-16 14:46:51.991287: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232bf4c500 of size 16384
2018-05-16 14:46:51.991317: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232bf50500 of size 4096
2018-05-16 14:46:51.991347: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232bf51500 of size 4096
2018-05-16 14:46:51.991375: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232bf52500 of size 4096
2018-05-16 14:46:51.991404: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232bf53500 of size 4096
2018-05-16 14:46:51.991433: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232bf54500 of size 25096192
2018-05-16 14:46:51.991463: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232d743500 of size 4096
2018-05-16 14:46:51.991492: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232d744500 of size 16785408
2018-05-16 14:46:51.991523: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232e746500 of size 16793600
2018-05-16 14:46:51.991553: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x232f74a500 of size 16793600
2018-05-16 14:46:51.991582: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x233074e500 of size 16781312
2018-05-16 14:46:51.991612: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x233174f500 of size 4096
2018-05-16 14:46:51.991641: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2331750500 of size 4096
2018-05-16 14:46:51.991675: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2331751500 of size 4096
2018-05-16 14:46:51.991705: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2331752500 of size 4096
2018-05-16 14:46:51.991734: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2331753500 of size 4096
2018-05-16 14:46:51.991762: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2331754500 of size 4198400
2018-05-16 14:46:51.991793: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2331b55500 of size 4096
2018-05-16 14:46:51.991823: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2331b56500 of size 16777216
2018-05-16 14:46:51.991853: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2332b56500 of size 16793600
2018-05-16 14:46:51.991881: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2333b5a500 of size 4096
2018-05-16 14:46:51.991910: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2333b5b500 of size 16777216
2018-05-16 14:46:51.991939: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2334b5b500 of size 138076160
2018-05-16 14:46:51.991969: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x233cf09500 of size 16777216
2018-05-16 14:46:51.992000: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x233df09500 of size 16777216
2018-05-16 14:46:51.992031: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x233ef09500 of size 4194304
2018-05-16 14:46:51.992060: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x233f309500 of size 16777216
2018-05-16 14:46:51.992087: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2340309500 of size 16777216
2018-05-16 14:46:51.992118: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2341309500 of size 16777216
2018-05-16 14:46:51.992147: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2342309500 of size 16777216
2018-05-16 14:46:51.992177: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2343309500 of size 33210368
2018-05-16 14:46:51.992207: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23452b5500 of size 4194304
2018-05-16 14:46:51.992237: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23456b5500 of size 4194304
2018-05-16 14:46:51.992266: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2345ab5500 of size 4194304
2018-05-16 14:46:51.992296: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2345eb5500 of size 4194304
2018-05-16 14:46:51.992325: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23462b5500 of size 16777216
2018-05-16 14:46:51.992357: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23472b5500 of size 20987904
2018-05-16 14:46:51.992387: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23486b9500 of size 16384
2018-05-16 14:46:51.992415: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23486bd500 of size 4096
2018-05-16 14:46:51.992444: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23486be500 of size 4096
2018-05-16 14:46:51.992473: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23486bf500 of size 4096
2018-05-16 14:46:51.992503: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23486c0500 of size 4096
2018-05-16 14:46:51.992533: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23486c1500 of size 16777216
2018-05-16 14:46:51.992562: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23496c1500 of size 16777216
2018-05-16 14:46:51.992591: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x234a6c1500 of size 4096
2018-05-16 14:46:51.992621: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x234a6c2500 of size 4096
2018-05-16 14:46:51.992650: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x234a6c3500 of size 4096
2018-05-16 14:46:51.992679: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x234a6c4500 of size 4096
2018-05-16 14:46:51.992707: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x234a6c5500 of size 4096
2018-05-16 14:46:51.992737: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x234a6c6500 of size 4096
2018-05-16 14:46:51.992766: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x234a6c7500 of size 4194304
2018-05-16 14:46:51.992795: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x234aac7500 of size 4194304
2018-05-16 14:46:51.992824: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x234aec7500 of size 4194304
2018-05-16 14:46:51.992854: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x234b2c7500 of size 4194304
2018-05-16 14:46:51.992883: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x234b6c7500 of size 4194304
2018-05-16 14:46:51.992912: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x234bac7500 of size 4194304
2018-05-16 14:46:51.992941: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x234bec7500 of size 4194304
2018-05-16 14:46:51.992970: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x234c2c7500 of size 4194304
2018-05-16 14:46:51.992999: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x234c6c7500 of size 4096
2018-05-16 14:46:51.993028: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x234c6c8500 of size 4096
2018-05-16 14:46:51.993058: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x234c6c9500 of size 4096
2018-05-16 14:46:51.993087: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x234c6ca500 of size 4096
2018-05-16 14:46:51.993117: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x234c6cb500 of size 16777216
2018-05-16 14:46:51.993147: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x234d6cb500 of size 16384
2018-05-16 14:46:51.993176: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x234d6cf500 of size 4194304
2018-05-16 14:46:51.993205: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x234dacf500 of size 4194304
2018-05-16 14:46:51.993235: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x234decf500 of size 4194304
2018-05-16 14:46:51.993265: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x234e2cf500 of size 4194304
2018-05-16 14:46:51.993294: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x234e6cf500 of size 16793600
2018-05-16 14:46:51.993324: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x234f6d3500 of size 16793600
2018-05-16 14:46:51.993354: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23506d7500 of size 4096
2018-05-16 14:46:51.993383: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23506d8500 of size 4096
2018-05-16 14:46:51.993412: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23506d9500 of size 4096
2018-05-16 14:46:51.993440: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23506da500 of size 4194304
2018-05-16 14:46:51.993470: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2350ada500 of size 4194304
2018-05-16 14:46:51.993499: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2350eda500 of size 4194304
2018-05-16 14:46:51.993529: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23512da500 of size 4194304
2018-05-16 14:46:51.993558: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23516da500 of size 4194304
2018-05-16 14:46:51.993586: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2351ada500 of size 4194304
2018-05-16 14:46:51.993615: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2351eda500 of size 4194304
2018-05-16 14:46:51.993644: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23522da500 of size 4096
2018-05-16 14:46:51.993675: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23522db500 of size 16384
2018-05-16 14:46:51.993703: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23522df500 of size 16384
2018-05-16 14:46:51.993733: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23522e3500 of size 4096
2018-05-16 14:46:51.993762: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23522e4500 of size 4096
2018-05-16 14:46:51.993792: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23522e5500 of size 4096
2018-05-16 14:46:51.993821: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23522e6500 of size 4096
2018-05-16 14:46:51.993852: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23522e7500 of size 16384
2018-05-16 14:46:51.993881: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23522eb500 of size 16384
2018-05-16 14:46:51.993911: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23522ef500 of size 16384
2018-05-16 14:46:51.993939: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23522f3500 of size 16384
2018-05-16 14:46:51.993968: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23522f7500 of size 16384
2018-05-16 14:46:51.993997: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23522fb500 of size 4096
2018-05-16 14:46:51.994027: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23522fc500 of size 16384
2018-05-16 14:46:51.994056: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2352300500 of size 16384
2018-05-16 14:46:51.994085: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2352304500 of size 16384
2018-05-16 14:46:51.994116: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2352308500 of size 4096
2018-05-16 14:46:51.994145: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2352309500 of size 4096
2018-05-16 14:46:51.994174: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x235230a500 of size 4096
2018-05-16 14:46:51.994204: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x235230b500 of size 16384
2018-05-16 14:46:51.994234: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x235230f500 of size 4096
2018-05-16 14:46:51.994264: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2352310500 of size 269824
2018-05-16 14:46:51.994292: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2352352300 of size 269824
2018-05-16 14:46:51.994322: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x2352394100 of size 256
2018-05-16 14:46:51.994352: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2352394200 of size 256
2018-05-16 14:46:51.994381: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2352394300 of size 256
2018-05-16 14:46:51.994410: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2352394400 of size 256
2018-05-16 14:46:51.994440: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2352394500 of size 256
2018-05-16 14:46:51.994468: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2352394600 of size 256
2018-05-16 14:46:51.994498: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2352394700 of size 256
2018-05-16 14:46:51.994527: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2352394800 of size 256
2018-05-16 14:46:51.994557: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x2352394900 of size 15360
2018-05-16 14:46:51.994586: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2352398500 of size 16128
2018-05-16 14:46:51.994627: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x235239c400 of size 30464
2018-05-16 14:46:51.994657: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23523a3b00 of size 16128
2018-05-16 14:46:51.994686: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23523a7a00 of size 16128
2018-05-16 14:46:51.994717: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23523ab900 of size 16128
2018-05-16 14:46:51.994746: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23523af800 of size 16128
2018-05-16 14:46:51.994776: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23523b3700 of size 268800
2018-05-16 14:46:51.994806: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23523f5100 of size 16128
2018-05-16 14:46:51.994837: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23523f9000 of size 13568
2018-05-16 14:46:51.994866: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23523fc500 of size 13568
2018-05-16 14:46:51.994896: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23523ffa00 of size 417792
2018-05-16 14:46:51.994925: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2352465a00 of size 13568
2018-05-16 14:46:51.994954: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2352468f00 of size 13568
2018-05-16 14:46:51.994984: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x235246c400 of size 13568
2018-05-16 14:46:51.995013: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x235246f900 of size 13568
2018-05-16 14:46:51.995043: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2352472e00 of size 16128
2018-05-16 14:46:51.995072: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2352476d00 of size 16128
2018-05-16 14:46:51.995103: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x235247ac00 of size 16128
2018-05-16 14:46:51.995132: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x235247eb00 of size 16128
2018-05-16 14:46:51.995161: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2352482a00 of size 16128
2018-05-16 14:46:51.995191: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2352486900 of size 16128
2018-05-16 14:46:51.995221: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x235248a800 of size 16128
2018-05-16 14:46:51.995250: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x235248e700 of size 16128
2018-05-16 14:46:51.995281: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2352492600 of size 16128
2018-05-16 14:46:51.995311: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2352496500 of size 16128
2018-05-16 14:46:51.995339: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x235249a400 of size 16128
2018-05-16 14:46:51.995369: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x235249e300 of size 16128
2018-05-16 14:46:51.995398: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23524a2200 of size 16128
2018-05-16 14:46:51.995427: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23524a6100 of size 16128
2018-05-16 14:46:51.995460: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23524aa000 of size 16128
2018-05-16 14:46:51.995489: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23524adf00 of size 16128
2018-05-16 14:46:51.995519: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23524b1e00 of size 16128
2018-05-16 14:46:51.995547: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23524b5d00 of size 16128
2018-05-16 14:46:51.995577: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23524b9c00 of size 16128
2018-05-16 14:46:51.995608: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23524bdb00 of size 16128
2018-05-16 14:46:51.995635: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23524c1a00 of size 16128
2018-05-16 14:46:51.995664: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23524c5900 of size 16128
2018-05-16 14:46:51.995694: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23524c9800 of size 16128
2018-05-16 14:46:51.995724: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23524cd700 of size 16128
2018-05-16 14:46:51.995754: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23524d1600 of size 16128
2018-05-16 14:46:51.995783: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23524d5500 of size 16128
2018-05-16 14:46:51.995812: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23524d9400 of size 16128
2018-05-16 14:46:51.995841: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23524dd300 of size 256
2018-05-16 14:46:51.995870: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23524dd400 of size 16128
2018-05-16 14:46:51.995902: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23524e1300 of size 16128
2018-05-16 14:46:51.995933: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23524e5200 of size 16128
2018-05-16 14:46:51.995962: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x23524e9100 of size 2040832
2018-05-16 14:46:51.995990: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23526db500 of size 4096
2018-05-16 14:46:51.996020: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23526dc500 of size 4096
2018-05-16 14:46:51.996051: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23526dd500 of size 4096
2018-05-16 14:46:51.996078: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23526de500 of size 16777216
2018-05-16 14:46:51.996108: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23536de500 of size 16777216
2018-05-16 14:46:51.996139: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23546de500 of size 16384
2018-05-16 14:46:51.996169: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23546e2500 of size 4096
2018-05-16 14:46:51.996199: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23546e3500 of size 4096
2018-05-16 14:46:51.996228: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23546e4500 of size 4096
2018-05-16 14:46:51.996257: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23546e5500 of size 4096
2018-05-16 14:46:51.996286: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23546e6500 of size 16777216
2018-05-16 14:46:51.996315: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23556e6500 of size 16777216
2018-05-16 14:46:51.996344: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23566e6500 of size 4096
2018-05-16 14:46:51.996373: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23566e7500 of size 4096
2018-05-16 14:46:51.996403: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23566e8500 of size 4096
2018-05-16 14:46:51.996431: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23566e9500 of size 4096
2018-05-16 14:46:51.996460: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23566ea500 of size 4096
2018-05-16 14:46:51.996490: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23566eb500 of size 4096
2018-05-16 14:46:51.996520: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23566ec500 of size 4194304
2018-05-16 14:46:51.996549: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2356aec500 of size 4194304
2018-05-16 14:46:51.996578: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2356eec500 of size 4194304
2018-05-16 14:46:51.996607: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23572ec500 of size 4194304
2018-05-16 14:46:51.996637: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23576ec500 of size 4194304
2018-05-16 14:46:51.996666: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2357aec500 of size 4194304
2018-05-16 14:46:51.996697: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2357eec500 of size 4194304
2018-05-16 14:46:51.996727: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23582ec500 of size 4206592
2018-05-16 14:46:51.996758: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23586ef500 of size 4096
2018-05-16 14:46:51.996788: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23586f0500 of size 16777216
2018-05-16 14:46:51.996817: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23596f0500 of size 16793600
2018-05-16 14:46:51.996847: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x235a6f4500 of size 16793600
2018-05-16 14:46:51.996877: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x235b6f8500 of size 16777216
2018-05-16 14:46:51.996907: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x235c6f8500 of size 4096
2018-05-16 14:46:51.996936: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x235c6f9500 of size 4096
2018-05-16 14:46:51.996965: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x235c6fa500 of size 4096
2018-05-16 14:46:51.996994: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x235c6fb500 of size 16384
2018-05-16 14:46:51.997023: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x235c6ff500 of size 4198400
2018-05-16 14:46:51.997052: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x235cb00500 of size 4194304
2018-05-16 14:46:51.997081: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x235cf00500 of size 4194304
2018-05-16 14:46:51.997111: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x235d300500 of size 4194304
2018-05-16 14:46:51.997140: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x235d700500 of size 4194304
2018-05-16 14:46:51.997169: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x235db00500 of size 4194304
2018-05-16 14:46:51.997198: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x235df00500 of size 4194304
2018-05-16 14:46:51.997229: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x235e300500 of size 4194304
2018-05-16 14:46:51.997258: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x235e700500 of size 4194304
2018-05-16 14:46:51.997288: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x235eb00500 of size 16777216
2018-05-16 14:46:51.997317: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x235fb00500 of size 16777216
2018-05-16 14:46:51.997347: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2360b00500 of size 16777216
2018-05-16 14:46:51.997376: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2361b00500 of size 4194304
2018-05-16 14:46:51.997407: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2361f00500 of size 20971520
2018-05-16 14:46:51.997438: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2363300500 of size 4194304
2018-05-16 14:46:51.997470: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2363700500 of size 4194304
2018-05-16 14:46:51.997499: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2363b00500 of size 4194304
2018-05-16 14:46:51.997528: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2363f00500 of size 4194304
2018-05-16 14:46:51.997557: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2364300500 of size 4194304
2018-05-16 14:46:51.997586: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2364700500 of size 20971520
2018-05-16 14:46:51.997614: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2365b00500 of size 16777216
2018-05-16 14:46:51.997644: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2366b00500 of size 4194304
2018-05-16 14:46:51.997672: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2366f00500 of size 4194304
2018-05-16 14:46:51.997702: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2367300500 of size 4194304
2018-05-16 14:46:51.997731: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2367700500 of size 4194304
2018-05-16 14:46:51.997762: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2367b00500 of size 16777216
2018-05-16 14:46:51.997791: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2368b00500 of size 4194304
2018-05-16 14:46:51.997821: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2368f00500 of size 4194304
2018-05-16 14:46:51.997850: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2369300500 of size 4194304
2018-05-16 14:46:51.997879: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2369700500 of size 4194304
2018-05-16 14:46:51.997909: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2369b00500 of size 4194304
2018-05-16 14:46:51.997938: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2369f00500 of size 4194304
2018-05-16 14:46:51.997968: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236a300500 of size 4194304
2018-05-16 14:46:51.997997: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236a700500 of size 4194304
2018-05-16 14:46:51.998026: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236ab00500 of size 4194304
2018-05-16 14:46:51.998056: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236af00500 of size 4194304
2018-05-16 14:46:51.998085: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236b300500 of size 4194304
2018-05-16 14:46:51.998114: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236b700500 of size 4194304
2018-05-16 14:46:51.998144: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236bb00500 of size 4194304
2018-05-16 14:46:51.998174: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236bf00500 of size 29360128
2018-05-16 14:46:51.998203: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236db00500 of size 16777216
2018-05-16 14:46:51.998232: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236eb00500 of size 16777216
2018-05-16 14:46:51.998261: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236fb00500 of size 4194304
2018-05-16 14:46:51.998290: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x236ff00500 of size 4194304
2018-05-16 14:46:51.998319: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2370300500 of size 4194304
2018-05-16 14:46:51.998349: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2370700500 of size 4194304
2018-05-16 14:46:51.998378: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2370b00500 of size 4194304
2018-05-16 14:46:51.998408: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2370f00500 of size 4194304
2018-05-16 14:46:51.998438: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2371300500 of size 4194304
2018-05-16 14:46:51.998468: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2371700500 of size 4194304
2018-05-16 14:46:51.998497: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2371b00500 of size 4194304
2018-05-16 14:46:51.998526: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2371f00500 of size 4194304
2018-05-16 14:46:51.998555: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2372300500 of size 4194304
2018-05-16 14:46:51.998585: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2372700500 of size 4194304
2018-05-16 14:46:51.998640: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2372b00500 of size 4194304
2018-05-16 14:46:51.998673: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2372f00500 of size 4194304
2018-05-16 14:46:51.998704: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2373300500 of size 4194304
2018-05-16 14:46:51.998733: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2373700500 of size 4194304
2018-05-16 14:46:51.998763: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2373b00500 of size 4194304
2018-05-16 14:46:51.998792: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2373f00500 of size 4194304
2018-05-16 14:46:51.998822: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2374300500 of size 4194304
2018-05-16 14:46:51.998852: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2374700500 of size 4194304
2018-05-16 14:46:51.998881: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2374b00500 of size 16777216
2018-05-16 14:46:51.998911: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2375b00500 of size 16777216
2018-05-16 14:46:51.998940: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2376b00500 of size 16777216
2018-05-16 14:46:51.998969: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2377b00500 of size 4194304
2018-05-16 14:46:51.998999: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2377f00500 of size 4194304
2018-05-16 14:46:51.999028: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2378300500 of size 4194304
2018-05-16 14:46:51.999057: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2378700500 of size 4194304
2018-05-16 14:46:51.999086: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2378b00500 of size 4194304
2018-05-16 14:46:51.999130: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2378f00500 of size 4194304
2018-05-16 14:46:51.999153: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2379300500 of size 4194304
2018-05-16 14:46:51.999174: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2379700500 of size 4194304
2018-05-16 14:46:51.999195: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2379b00500 of size 4194304
2018-05-16 14:46:51.999217: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2379f00500 of size 4194304
2018-05-16 14:46:51.999241: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x237a300500 of size 4194304
2018-05-16 14:46:51.999262: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x237a700500 of size 4194304
2018-05-16 14:46:51.999282: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x237ab00500 of size 4194304
2018-05-16 14:46:51.999303: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x237af00500 of size 4194304
2018-05-16 14:46:51.999324: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x237b300500 of size 4194304
2018-05-16 14:46:51.999345: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x237b700500 of size 4194304
2018-05-16 14:46:51.999365: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x237bb00500 of size 4194304
2018-05-16 14:46:51.999385: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x237bf00500 of size 4194304
2018-05-16 14:46:51.999407: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x237c300500 of size 4194304
2018-05-16 14:46:51.999428: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x237c700500 of size 4194304
2018-05-16 14:46:51.999449: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x237cb00500 of size 4194304
2018-05-16 14:46:51.999470: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x237cf00500 of size 4194304
2018-05-16 14:46:51.999492: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x237d300500 of size 4194304
2018-05-16 14:46:51.999514: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x237d700500 of size 4194304
2018-05-16 14:46:51.999536: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x237db00500 of size 4194304
2018-05-16 14:46:51.999556: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x237df00500 of size 16777216
2018-05-16 14:46:51.999578: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x237ef00500 of size 16777216
2018-05-16 14:46:51.999599: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x237ff00500 of size 4194304
2018-05-16 14:46:51.999620: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2380300500 of size 4194304
2018-05-16 14:46:51.999641: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2380700500 of size 4194304
2018-05-16 14:46:51.999663: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2380b00500 of size 4194304
2018-05-16 14:46:51.999684: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2380f00500 of size 16777216
2018-05-16 14:46:51.999703: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2381f00500 of size 4194304
2018-05-16 14:46:51.999725: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2382300500 of size 4194304
2018-05-16 14:46:51.999746: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2382700500 of size 4194304
2018-05-16 14:46:51.999768: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2382b00500 of size 4194304
2018-05-16 14:46:51.999789: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2382f00500 of size 4194304
2018-05-16 14:46:51.999808: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2383300500 of size 16777216
2018-05-16 14:46:51.999830: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2384300500 of size 16777216
2018-05-16 14:46:51.999852: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2385300500 of size 16293888
2018-05-16 14:46:51.999873: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x238628a500 of size 29499392
2018-05-16 14:46:51.999895: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2387eac500 of size 4194304
2018-05-16 14:46:51.999916: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23882ac500 of size 4194304
2018-05-16 14:46:51.999937: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23886ac500 of size 4194304
2018-05-16 14:46:51.999958: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2388aac500 of size 4194304
2018-05-16 14:46:51.999979: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2388eac500 of size 16777216
2018-05-16 14:46:52.000001: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2389eac500 of size 4194304
2018-05-16 14:46:52.000024: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x238a2ac500 of size 4194304
2018-05-16 14:46:52.000045: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x238a6ac500 of size 4194304
2018-05-16 14:46:52.000066: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x238aaac500 of size 4194304
2018-05-16 14:46:52.000088: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x238aeac500 of size 4194304
2018-05-16 14:46:52.000109: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x238b2ac500 of size 4194304
2018-05-16 14:46:52.000130: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x238b6ac500 of size 25165824
2018-05-16 14:46:52.000152: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x238ceac500 of size 20971520
2018-05-16 14:46:52.000173: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x238e2ac500 of size 4194304
2018-05-16 14:46:52.000195: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x238e6ac500 of size 4194304
2018-05-16 14:46:52.000216: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x238eaac500 of size 20971520
2018-05-16 14:46:52.000236: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x238feac500 of size 20971520
2018-05-16 14:46:52.000257: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23912ac500 of size 4194304
2018-05-16 14:46:52.000278: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23916ac500 of size 4194304
2018-05-16 14:46:52.000299: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2391aac500 of size 4194304
2018-05-16 14:46:52.000321: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2391eac500 of size 16777216
2018-05-16 14:46:52.000343: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2392eac500 of size 16777216
2018-05-16 14:46:52.000365: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2393eac500 of size 4194304
2018-05-16 14:46:52.000386: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23942ac500 of size 4194304
2018-05-16 14:46:52.000407: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23946ac500 of size 4194304
2018-05-16 14:46:52.000428: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2394aac500 of size 4194304
2018-05-16 14:46:52.000450: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2394eac500 of size 16777216
2018-05-16 14:46:52.000470: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2395eac500 of size 25165824
2018-05-16 14:46:52.000492: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23976ac500 of size 4194304
2018-05-16 14:46:52.000514: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2397aac500 of size 4194304
2018-05-16 14:46:52.000536: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2397eac500 of size 4194304
2018-05-16 14:46:52.000558: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23982ac500 of size 4194304
2018-05-16 14:46:52.000579: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23986ac500 of size 4194304
2018-05-16 14:46:52.000600: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2398aac500 of size 4194304
2018-05-16 14:46:52.000620: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2398eac500 of size 20971520
2018-05-16 14:46:52.000642: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x239a2ac500 of size 16777216
2018-05-16 14:46:52.000663: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x239b2ac500 of size 4194304
2018-05-16 14:46:52.000685: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x239b6ac500 of size 4194304
2018-05-16 14:46:52.000706: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x239baac500 of size 4194304
2018-05-16 14:46:52.000727: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x239beac500 of size 4194304
2018-05-16 14:46:52.000747: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x239c2ac500 of size 4194304
2018-05-16 14:46:52.000769: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x239c6ac500 of size 4194304
2018-05-16 14:46:52.000790: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x239caac500 of size 16293888
2018-05-16 14:46:52.000812: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x239da36500 of size 25649152
2018-05-16 14:46:52.000833: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x239f2ac500 of size 4194304
2018-05-16 14:46:52.000855: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x239f6ac500 of size 4194304
2018-05-16 14:46:52.000877: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x239faac500 of size 4194304
2018-05-16 14:46:52.000898: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x239feac500 of size 4194304
2018-05-16 14:46:52.000919: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23a02ac500 of size 4194304
2018-05-16 14:46:52.000939: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23a06ac500 of size 16293888
2018-05-16 14:46:52.000961: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23a1636500 of size 29843456
2018-05-16 14:46:52.000983: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23a32ac500 of size 4194304
2018-05-16 14:46:52.001006: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23a36ac500 of size 4194304
2018-05-16 14:46:52.001027: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x23a3aac500 of size 4194304
2018-05-16 14:46:52.001048: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23a3eac500 of size 4194304
2018-05-16 14:46:52.001070: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23a42ac500 of size 4194304
2018-05-16 14:46:52.001091: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x23a46ac500 of size 8388608
2018-05-16 14:46:52.001113: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23a4eac500 of size 4194304
2018-05-16 14:46:52.001134: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23a52ac500 of size 16777216
2018-05-16 14:46:52.001155: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23a62ac500 of size 16777216
2018-05-16 14:46:52.001176: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23a72ac500 of size 4194304
2018-05-16 14:46:52.001198: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x23a76ac500 of size 12582912
2018-05-16 14:46:52.001219: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23a82ac500 of size 4194304
2018-05-16 14:46:52.001241: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x23a86ac500 of size 12582912
2018-05-16 14:46:52.001261: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23a92ac500 of size 4194304
2018-05-16 14:46:52.001282: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23a96ac500 of size 16293888
2018-05-16 14:46:52.001302: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23aa636500 of size 17260544
2018-05-16 14:46:52.001325: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23ab6ac500 of size 4194304
2018-05-16 14:46:52.001347: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23abaac500 of size 4194304
2018-05-16 14:46:52.001368: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23abeac500 of size 16777216
2018-05-16 14:46:52.001388: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23aceac500 of size 4194304
2018-05-16 14:46:52.001410: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23ad2ac500 of size 16777216
2018-05-16 14:46:52.001431: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23ae2ac500 of size 138067968
2018-05-16 14:46:52.001453: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23b6658500 of size 138067968
2018-05-16 14:46:52.001475: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23bea04500 of size 16293888
2018-05-16 14:46:52.001496: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23bf98e500 of size 138096640
2018-05-16 14:46:52.001516: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23c7d41500 of size 138096640
2018-05-16 14:46:52.001538: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23d00f4500 of size 16293888
2018-05-16 14:46:52.001560: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23d107e500 of size 16293888
2018-05-16 14:46:52.001581: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23d2008500 of size 16293888
2018-05-16 14:46:52.001603: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23d2f92500 of size 16293888
2018-05-16 14:46:52.001623: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23d3f1c500 of size 25968384
2018-05-16 14:46:52.001644: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23d57e0400 of size 25968384
2018-05-16 14:46:52.001666: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23d70a4300 of size 25968384
2018-05-16 14:46:52.001687: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23d8968200 of size 16293888
2018-05-16 14:46:52.001708: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23d98f2200 of size 16293888
2018-05-16 14:46:52.001729: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23da87c200 of size 16293888
2018-05-16 14:46:52.001750: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23db806200 of size 16293888
2018-05-16 14:46:52.001771: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23dc790200 of size 25968384
2018-05-16 14:46:52.001794: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23de054100 of size 25968384
2018-05-16 14:46:52.001815: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23df918000 of size 25968384
2018-05-16 14:46:52.001835: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23e11dbf00 of size 16293888
2018-05-16 14:46:52.001857: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23e2165f00 of size 16293888
2018-05-16 14:46:52.001879: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23e30eff00 of size 16293888
2018-05-16 14:46:52.001901: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23e4079f00 of size 16293888
2018-05-16 14:46:52.001922: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23e5003f00 of size 16293888
2018-05-16 14:46:52.001943: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23e5f8df00 of size 16293888
2018-05-16 14:46:52.001967: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23e6f17f00 of size 16293888
2018-05-16 14:46:52.001989: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23e7ea1f00 of size 16293888
2018-05-16 14:46:52.002011: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23e8e2bf00 of size 13881344
2018-05-16 14:46:52.002033: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23e9b68f00 of size 16293888
2018-05-16 14:46:52.002055: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23eaaf2f00 of size 55525376
2018-05-16 14:46:52.002078: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23edfe6f00 of size 55525376
2018-05-16 14:46:52.002099: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23f14daf00 of size 55525376
2018-05-16 14:46:52.002120: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23f49cef00 of size 16293888
2018-05-16 14:46:52.002142: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23f5958f00 of size 16293888
2018-05-16 14:46:52.002163: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23f68e2f00 of size 16293888
2018-05-16 14:46:52.002184: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23f786cf00 of size 16293888
2018-05-16 14:46:52.002205: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23f87f6f00 of size 16293888
2018-05-16 14:46:52.002227: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23f9780f00 of size 16293888
2018-05-16 14:46:52.002248: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23fa70af00 of size 16293888
2018-05-16 14:46:52.002269: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23fb694f00 of size 25968384
2018-05-16 14:46:52.002291: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23fcf58e00 of size 25968384
2018-05-16 14:46:52.002312: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x23fe81cd00 of size 25968384
2018-05-16 14:46:52.002333: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24000e0c00 of size 16293888
2018-05-16 14:46:52.002355: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x240106ac00 of size 16293888
2018-05-16 14:46:52.002376: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2401ff4c00 of size 16293888
2018-05-16 14:46:52.002397: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2402f7ec00 of size 16293888
2018-05-16 14:46:52.002418: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2403f08c00 of size 13881344
2018-05-16 14:46:52.002440: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2404c45c00 of size 55525376
2018-05-16 14:46:52.002461: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2408139c00 of size 55525376
2018-05-16 14:46:52.002483: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x240b62dc00 of size 55525376
2018-05-16 14:46:52.002504: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x240eb21c00 of size 16293888
2018-05-16 14:46:52.002525: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x240faabc00 of size 16293888
2018-05-16 14:46:52.002546: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2410a35c00 of size 16293888
2018-05-16 14:46:52.002567: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24119bfc00 of size 16293888
2018-05-16 14:46:52.002589: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2412949c00 of size 16293888
2018-05-16 14:46:52.002618: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24138d3c00 of size 16293888
2018-05-16 14:46:52.002639: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x241485dc00 of size 16293888
2018-05-16 14:46:52.002660: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24157e7c00 of size 25968384
2018-05-16 14:46:52.002682: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24170abb00 of size 25968384
2018-05-16 14:46:52.002703: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x241896fa00 of size 25968384
2018-05-16 14:46:52.002724: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x241a233900 of size 16293888
2018-05-16 14:46:52.002745: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x241b1bd900 of size 16293888
2018-05-16 14:46:52.002767: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x241c147900 of size 16293888
2018-05-16 14:46:52.002789: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x241d0d1900 of size 16293888
2018-05-16 14:46:52.002808: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x241e05b900 of size 13881344
2018-05-16 14:46:52.002830: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x241ed98900 of size 55525376
2018-05-16 14:46:52.002851: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x242228c900 of size 55525376
2018-05-16 14:46:52.002873: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2425780900 of size 55525376
2018-05-16 14:46:52.002893: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2428c74900 of size 16293888
2018-05-16 14:46:52.002915: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2429bfe900 of size 16293888
2018-05-16 14:46:52.002937: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x242ab88900 of size 16293888
2018-05-16 14:46:52.002956: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x242bb12900 of size 16293888
2018-05-16 14:46:52.002977: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x242ca9c900 of size 16293888
2018-05-16 14:46:52.002998: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x242da26900 of size 16293888
2018-05-16 14:46:52.003020: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x242e9b0900 of size 16293888
2018-05-16 14:46:52.003041: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x242f93a900 of size 25968384
2018-05-16 14:46:52.003063: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24311fe800 of size 25968384
2018-05-16 14:46:52.003083: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2432ac2700 of size 25968384
2018-05-16 14:46:52.003105: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2434386600 of size 16293888
2018-05-16 14:46:52.003127: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2435310600 of size 16293888
2018-05-16 14:46:52.003147: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x243629a600 of size 16293888
2018-05-16 14:46:52.003169: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2437224600 of size 16293888
2018-05-16 14:46:52.003191: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24381ae600 of size 13881344
2018-05-16 14:46:52.003213: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2438eeb600 of size 55525376
2018-05-16 14:46:52.003234: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x243c3df600 of size 55525376
2018-05-16 14:46:52.003256: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x243f8d3600 of size 55525376
2018-05-16 14:46:52.003277: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2442dc7600 of size 16293888
2018-05-16 14:46:52.003298: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2443d51600 of size 16293888
2018-05-16 14:46:52.003321: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2444cdb600 of size 16293888
2018-05-16 14:46:52.003343: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2445c65600 of size 16293888
2018-05-16 14:46:52.003364: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2446bef600 of size 16293888
2018-05-16 14:46:52.003385: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2447b79600 of size 16293888
2018-05-16 14:46:52.003407: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2448b03600 of size 16293888
2018-05-16 14:46:52.003430: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2449a8d600 of size 25968384
2018-05-16 14:46:52.003451: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x244b351500 of size 25968384
2018-05-16 14:46:52.003472: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x244cc15400 of size 25968384
2018-05-16 14:46:52.003492: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x244e4d9300 of size 16293888
2018-05-16 14:46:52.003514: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x244f463300 of size 16293888
2018-05-16 14:46:52.003535: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24503ed300 of size 16293888
2018-05-16 14:46:52.003557: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2451377300 of size 16293888
2018-05-16 14:46:52.003578: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2452301300 of size 13881344
2018-05-16 14:46:52.003600: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x245303e300 of size 55525376
2018-05-16 14:46:52.003621: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2456532300 of size 55525376
2018-05-16 14:46:52.003642: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2459a26300 of size 55525376
2018-05-16 14:46:52.003663: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x245cf1a300 of size 16293888
2018-05-16 14:46:52.003685: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x245dea4300 of size 16293888
2018-05-16 14:46:52.003707: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x245ee2e300 of size 16293888
2018-05-16 14:46:52.003728: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x245fdb8300 of size 16293888
2018-05-16 14:46:52.003749: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2460d42300 of size 16293888
2018-05-16 14:46:52.003771: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2461ccc300 of size 16293888
2018-05-16 14:46:52.003793: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2462c56300 of size 16293888
2018-05-16 14:46:52.003815: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2463be0300 of size 25968384
2018-05-16 14:46:52.003836: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24654a4200 of size 25968384
2018-05-16 14:46:52.003857: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2466d68100 of size 25968384
2018-05-16 14:46:52.003879: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x246862c000 of size 16293888
2018-05-16 14:46:52.003899: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24695b6000 of size 16293888
2018-05-16 14:46:52.003920: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x246a540000 of size 16293888
2018-05-16 14:46:52.003942: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x246b4ca000 of size 16293888
2018-05-16 14:46:52.003963: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x246c454000 of size 13881344
2018-05-16 14:46:52.003985: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x246d191000 of size 55525376
2018-05-16 14:46:52.004006: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2470685000 of size 55525376
2018-05-16 14:46:52.004027: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2473b79000 of size 55525376
2018-05-16 14:46:52.004048: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x247706d000 of size 16293888
2018-05-16 14:46:52.004070: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2477ff7000 of size 16293888
2018-05-16 14:46:52.004091: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2478f81000 of size 16293888
2018-05-16 14:46:52.004112: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2479f0b000 of size 16293888
2018-05-16 14:46:52.004133: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x247ae95000 of size 16293888
2018-05-16 14:46:52.004155: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x247be1f000 of size 16293888
2018-05-16 14:46:52.004176: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x247cda9000 of size 16293888
2018-05-16 14:46:52.004198: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x247dd33000 of size 16293888
2018-05-16 14:46:52.004219: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x247ecbd000 of size 16293888
2018-05-16 14:46:52.004240: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x247fc47000 of size 16293888
2018-05-16 14:46:52.004263: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2480bd1000 of size 16293888
2018-05-16 14:46:52.004285: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2481b5b000 of size 16293888
2018-05-16 14:46:52.004306: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2482ae5000 of size 16293888
2018-05-16 14:46:52.004327: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2483a6f000 of size 16293888
2018-05-16 14:46:52.004348: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24849f9000 of size 16293888
2018-05-16 14:46:52.004369: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2485983000 of size 25968384
2018-05-16 14:46:52.004393: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2487246f00 of size 25968384
2018-05-16 14:46:52.004414: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2488b0ae00 of size 25968384
2018-05-16 14:46:52.004436: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x248a3ced00 of size 16293888
2018-05-16 14:46:52.004457: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x248b358d00 of size 16293888
2018-05-16 14:46:52.004479: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x248c2e2d00 of size 16293888
2018-05-16 14:46:52.004500: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x248d26cd00 of size 16293888
2018-05-16 14:46:52.004522: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x248e1f6d00 of size 65175552
2018-05-16 14:46:52.004541: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x249201ed00 of size 65175552
2018-05-16 14:46:52.004563: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2495e46d00 of size 65175552
2018-05-16 14:46:52.004584: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2499c6ed00 of size 16293888
2018-05-16 14:46:52.004605: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x249abf8d00 of size 16293888
2018-05-16 14:46:52.004626: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x249bb82d00 of size 16293888
2018-05-16 14:46:52.004648: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x249cb0cd00 of size 16293888
2018-05-16 14:46:52.004669: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x249da96d00 of size 16293888
2018-05-16 14:46:52.004692: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x249ea20d00 of size 16293888
2018-05-16 14:46:52.004716: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x249f9aad00 of size 16293888
2018-05-16 14:46:52.004738: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24a0934d00 of size 16293888
2018-05-16 14:46:52.004759: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24a18bed00 of size 25968384
2018-05-16 14:46:52.004780: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24a3182c00 of size 25968384
2018-05-16 14:46:52.004803: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24a4a46b00 of size 25968384
2018-05-16 14:46:52.004824: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24a630aa00 of size 16293888
2018-05-16 14:46:52.004845: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24a7294a00 of size 16293888
2018-05-16 14:46:52.004866: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24a821ea00 of size 16293888
2018-05-16 14:46:52.004885: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24a91a8a00 of size 16293888
2018-05-16 14:46:52.004906: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24aa132a00 of size 16293888
2018-05-16 14:46:52.004928: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24ab0bca00 of size 16293888
2018-05-16 14:46:52.004950: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24ac046a00 of size 25968384
2018-05-16 14:46:52.004973: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24ad90a900 of size 25968384
2018-05-16 14:46:52.004994: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24af1ce800 of size 25968384
2018-05-16 14:46:52.005016: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24b0a92700 of size 16293888
2018-05-16 14:46:52.005038: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24b1a1c700 of size 16293888
2018-05-16 14:46:52.005060: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24b29a6700 of size 16293888
2018-05-16 14:46:52.005081: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24b3930700 of size 16293888
2018-05-16 14:46:52.005102: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24b48ba700 of size 65175552
2018-05-16 14:46:52.005124: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24b86e2700 of size 65175552
2018-05-16 14:46:52.005146: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24bc50a700 of size 65175552
2018-05-16 14:46:52.005169: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24c0332700 of size 16293888
2018-05-16 14:46:52.005190: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24c12bc700 of size 16293888
2018-05-16 14:46:52.005212: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24c2246700 of size 16293888
2018-05-16 14:46:52.005235: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24c31d0700 of size 16293888
2018-05-16 14:46:52.005256: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24c415a700 of size 16293888
2018-05-16 14:46:52.005277: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24c50e4700 of size 16293888
2018-05-16 14:46:52.005299: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24c606e700 of size 16293888
2018-05-16 14:46:52.005319: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24c6ff8700 of size 16293888
2018-05-16 14:46:52.005341: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24c7f82700 of size 25968384
2018-05-16 14:46:52.005362: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24c9846600 of size 25968384
2018-05-16 14:46:52.005384: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24cb10a500 of size 25968384
2018-05-16 14:46:52.005405: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24cc9ce400 of size 16293888
2018-05-16 14:46:52.005426: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24cd958400 of size 16293888
2018-05-16 14:46:52.005448: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24ce8e2400 of size 16293888
2018-05-16 14:46:52.005469: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24cf86c400 of size 16293888
2018-05-16 14:46:52.005492: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24d07f6400 of size 16293888
2018-05-16 14:46:52.005514: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24d1780400 of size 16293888
2018-05-16 14:46:52.005535: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24d270a400 of size 25968384
2018-05-16 14:46:52.005556: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24d3fce300 of size 25968384
2018-05-16 14:46:52.005578: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24d5892200 of size 25968384
2018-05-16 14:46:52.005600: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24d7156100 of size 16293888
2018-05-16 14:46:52.005620: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24d80e0100 of size 16293888
2018-05-16 14:46:52.005643: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24d906a100 of size 16293888
2018-05-16 14:46:52.005665: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24d9ff4100 of size 16293888
2018-05-16 14:46:52.005687: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24daf7e100 of size 65175552
2018-05-16 14:46:52.005709: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24deda6100 of size 65175552
2018-05-16 14:46:52.005730: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24e2bce100 of size 65175552
2018-05-16 14:46:52.005753: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24e69f6100 of size 16293888
2018-05-16 14:46:52.005775: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24e7980100 of size 16293888
2018-05-16 14:46:52.005796: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24e890a100 of size 16293888
2018-05-16 14:46:52.005818: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24e9894100 of size 16293888
2018-05-16 14:46:52.005839: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24ea81e100 of size 16293888
2018-05-16 14:46:52.005862: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24eb7a8100 of size 16293888
2018-05-16 14:46:52.005885: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24ec732100 of size 16293888
2018-05-16 14:46:52.005906: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24ed6bc100 of size 16293888
2018-05-16 14:46:52.005927: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24ee646100 of size 25968384
2018-05-16 14:46:52.005949: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24eff0a000 of size 25968384
2018-05-16 14:46:52.005971: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24f17cdf00 of size 25968384
2018-05-16 14:46:52.005993: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24f3091e00 of size 16293888
2018-05-16 14:46:52.006016: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24f401be00 of size 16293888
2018-05-16 14:46:52.006037: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24f4fa5e00 of size 16293888
2018-05-16 14:46:52.006059: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24f5f2fe00 of size 16293888
2018-05-16 14:46:52.006081: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24f6eb9e00 of size 16293888
2018-05-16 14:46:52.006102: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24f7e43e00 of size 25968384
2018-05-16 14:46:52.006125: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24f9707d00 of size 25968384
2018-05-16 14:46:52.006147: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24fafcbc00 of size 25968384
2018-05-16 14:46:52.006171: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24fc88fb00 of size 16293888
2018-05-16 14:46:52.006193: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24fd819b00 of size 16293888
2018-05-16 14:46:52.006214: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24fe7a3b00 of size 16293888
2018-05-16 14:46:52.006237: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x24ff72db00 of size 16293888
2018-05-16 14:46:52.006259: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x25006b7b00 of size 16293888
2018-05-16 14:46:52.006282: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2501641b00 of size 16293888
2018-05-16 14:46:52.006303: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x25025cbb00 of size 65175552
2018-05-16 14:46:52.006325: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x25063f3b00 of size 65175552
2018-05-16 14:46:52.006346: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x250a21bb00 of size 65175552
2018-05-16 14:46:52.006368: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x250e043b00 of size 16293888
2018-05-16 14:46:52.006388: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x250efcdb00 of size 16293888
2018-05-16 14:46:52.006410: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x250ff57b00 of size 16293888
2018-05-16 14:46:52.006432: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2510ee1b00 of size 16293888
2018-05-16 14:46:52.006453: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2511e6bb00 of size 16293888
2018-05-16 14:46:52.006474: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2512df5b00 of size 16293888
2018-05-16 14:46:52.006494: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2513d7fb00 of size 16293888
2018-05-16 14:46:52.006516: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2514d09b00 of size 25968384
2018-05-16 14:46:52.006537: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x25165cda00 of size 25968384
2018-05-16 14:46:52.006559: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2517e91900 of size 25968384
2018-05-16 14:46:52.006581: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2519755800 of size 16293888
2018-05-16 14:46:52.008916: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x251a6df800 of size 16293888
2018-05-16 14:46:52.008939: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x251b669800 of size 16293888
2018-05-16 14:46:52.008956: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x251c5f3800 of size 16293888
2018-05-16 14:46:52.008974: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x251d57d800 of size 16293888
2018-05-16 14:46:52.008991: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x251e507800 of size 25968384
2018-05-16 14:46:52.009010: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x251fdcb700 of size 25968384
2018-05-16 14:46:52.009028: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x252168f600 of size 25968384
2018-05-16 14:46:52.009071: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2522f53500 of size 16293888
2018-05-16 14:46:52.009092: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2523edd500 of size 16293888
2018-05-16 14:46:52.009115: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2524e67500 of size 16293888
2018-05-16 14:46:52.009137: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2525df1500 of size 16293888
2018-05-16 14:46:52.009159: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2526d7b500 of size 16293888
2018-05-16 14:46:52.009180: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2527d05500 of size 16293888
2018-05-16 14:46:52.009202: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2528c8f500 of size 65175552
2018-05-16 14:46:52.009223: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x252cab7500 of size 65175552
2018-05-16 14:46:52.009245: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x25308df500 of size 65175552
2018-05-16 14:46:52.009265: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2534707500 of size 16293888
2018-05-16 14:46:52.009287: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2535691500 of size 16293888
2018-05-16 14:46:52.009309: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x253661b500 of size 16293888
2018-05-16 14:46:52.009330: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x25375a5500 of size 16293888
2018-05-16 14:46:52.009350: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x253852f500 of size 16293888
2018-05-16 14:46:52.009371: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x25394b9500 of size 16293888
2018-05-16 14:46:52.009392: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x253a443500 of size 16293888
2018-05-16 14:46:52.009412: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x253b3cd500 of size 25968384
2018-05-16 14:46:52.009432: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x253cc91400 of size 25968384
2018-05-16 14:46:52.009453: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x253e555300 of size 25968384
2018-05-16 14:46:52.009474: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x253fe19200 of size 16293888
2018-05-16 14:46:52.009495: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2540da3200 of size 16293888
2018-05-16 14:46:52.009516: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2541d2d200 of size 16293888
2018-05-16 14:46:52.009537: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2542cb7200 of size 16293888
2018-05-16 14:46:52.009558: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2543c41200 of size 16293888
2018-05-16 14:46:52.009579: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2544bcb200 of size 16293888
2018-05-16 14:46:52.009604: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2545b55200 of size 25968384
2018-05-16 14:46:52.009625: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2547419100 of size 25968384
2018-05-16 14:46:52.009646: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2548cdd000 of size 25968384
2018-05-16 14:46:52.009667: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x254a5a0f00 of size 16293888
2018-05-16 14:46:52.009687: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x254b52af00 of size 16293888
2018-05-16 14:46:52.009709: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x254c4b4f00 of size 16293888
2018-05-16 14:46:52.009730: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x254d43ef00 of size 16293888
2018-05-16 14:46:52.009752: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x254e3c8f00 of size 65175552
2018-05-16 14:46:52.009773: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x25521f0f00 of size 65175552
2018-05-16 14:46:52.009794: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2556018f00 of size 65175552
2018-05-16 14:46:52.009816: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x2559e40f00 of size 16293888
2018-05-16 14:46:52.009835: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x255adcaf00 of size 16293888
2018-05-16 14:46:52.009856: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x255bd54f00 of size 16293888
2018-05-16 14:46:52.009878: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x255ccdef00 of size 16293888
2018-05-16 14:46:52.009901: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x255dc68f00 of size 536361728
2018-05-16 14:46:52.009922: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x257dbeca00 of size 536361728
2018-05-16 14:46:52.009943: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x259db70500 of size 391910400
2018-05-16 14:46:52.009964: I tensorflow/core/common_runtime/bfc_allocator.cc:671] Summary of in-use Chunks by size:
2018-05-16 14:46:52.009992: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 181 Chunks of size 256 totalling 45.2KiB
2018-05-16 14:46:52.010015: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 1280 totalling 1.2KiB
2018-05-16 14:46:52.010038: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 2560 totalling 2.5KiB
2018-05-16 14:46:52.010059: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 2816 totalling 2.8KiB
2018-05-16 14:46:52.010082: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 225 Chunks of size 4096 totalling 900.0KiB
2018-05-16 14:46:52.010104: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 2 Chunks of size 4352 totalling 8.5KiB
2018-05-16 14:46:52.010126: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 7936 totalling 7.8KiB
2018-05-16 14:46:52.010148: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 6 Chunks of size 13568 totalling 79.5KiB
2018-05-16 14:46:52.010171: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 36 Chunks of size 16128 totalling 567.0KiB
2018-05-16 14:46:52.010194: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 31 Chunks of size 16384 totalling 496.0KiB
2018-05-16 14:46:52.010217: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 4 Chunks of size 20480 totalling 80.0KiB
2018-05-16 14:46:52.010240: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 2 Chunks of size 24576 totalling 48.0KiB
2018-05-16 14:46:52.010263: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 30464 totalling 29.8KiB
2018-05-16 14:46:52.010286: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 268800 totalling 262.5KiB
2018-05-16 14:46:52.010307: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 2 Chunks of size 269824 totalling 527.0KiB
2018-05-16 14:46:52.010330: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 417792 totalling 408.0KiB
2018-05-16 14:46:52.010352: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 208 Chunks of size 4194304 totalling 832.00MiB
2018-05-16 14:46:52.010375: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 4 Chunks of size 4198400 totalling 16.02MiB
2018-05-16 14:46:52.010399: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 2 Chunks of size 4206592 totalling 8.02MiB
2018-05-16 14:46:52.010420: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 7544576 totalling 7.19MiB
2018-05-16 14:46:52.010442: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 8379904 totalling 7.99MiB
2018-05-16 14:46:52.010465: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 6 Chunks of size 13881344 totalling 79.43MiB
2018-05-16 14:46:52.010487: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 189 Chunks of size 16293888 totalling 2.87GiB
2018-05-16 14:46:52.010512: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 50 Chunks of size 16777216 totalling 800.00MiB
2018-05-16 14:46:52.010534: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 16781056 totalling 16.00MiB
2018-05-16 14:46:52.010557: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 2 Chunks of size 16781312 totalling 32.01MiB
2018-05-16 14:46:52.010580: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 16785408 totalling 16.01MiB
2018-05-16 14:46:52.010621: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 7 Chunks of size 16793600 totalling 112.11MiB
2018-05-16 14:46:52.010646: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 17260544 totalling 16.46MiB
2018-05-16 14:46:52.010675: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 6 Chunks of size 20971520 totalling 120.00MiB
2018-05-16 14:46:52.010697: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 20987904 totalling 20.02MiB
2018-05-16 14:46:52.010721: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 25096192 totalling 23.93MiB
2018-05-16 14:46:52.010746: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 2 Chunks of size 25165824 totalling 48.00MiB
2018-05-16 14:46:52.010769: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 25649152 totalling 24.46MiB
2018-05-16 14:46:52.010792: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 54 Chunks of size 25968384 totalling 1.31GiB
2018-05-16 14:46:52.010814: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 29360128 totalling 28.00MiB
2018-05-16 14:46:52.010837: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 29499392 totalling 28.13MiB
2018-05-16 14:46:52.010860: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 29843456 totalling 28.46MiB
2018-05-16 14:46:52.010884: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 33210368 totalling 31.67MiB
2018-05-16 14:46:52.010906: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 18 Chunks of size 55525376 totalling 953.16MiB
2018-05-16 14:46:52.010928: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 18 Chunks of size 65175552 totalling 1.09GiB
2018-05-16 14:46:52.010952: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 2 Chunks of size 138067968 totalling 263.34MiB
2018-05-16 14:46:52.010975: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 138076160 totalling 131.68MiB
2018-05-16 14:46:52.010997: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 2 Chunks of size 138096640 totalling 263.40MiB
2018-05-16 14:46:52.011021: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 2 Chunks of size 536361728 totalling 1023.03MiB
2018-05-16 14:46:52.011044: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Sum Total of in-use chunks: 10.08GiB
2018-05-16 14:46:52.011071: I tensorflow/core/common_runtime/bfc_allocator.cc:680] Stats:
Limit: 11284978074
InUse: 10828579584
MaxInUse: 11220441600
NumAllocs: 197840
MaxAllocSize: 618382336
2018-05-16 14:46:52.011159: W tensorflow/core/common_runtime/bfc_allocator.cc:279] *************************************************************************************************___
Setting random seed = 1
Training schedule:
1. Train for 1 epochs.
2. Evaluate model.
3. Compute BLEU score.
Repeat above steps until the BLEU score reaches 25.0
Starting iteration 1
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1322, in _do_call
return fn(*args)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1307, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1409, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: Ran out of GPU memory when allocating 536361696 bytes for
[[Node: model/loss/smoothing_cross_entropy/softmax_cross_entropy_with_logits = SoftmaxCrossEntropyWithLogits[T=DT_FLOAT, _class=["loc:@model...s_grad/mul"], _device="/job:localhost/replica:0/task:0/device:GPU:0"](model/loss/smoothing_cross_entropy/softmax_cross_entropy_with_logits/Reshape, model/loss/smoothing_cross_entropy/softmax_cross_entropy_with_logits/Reshape_1)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
[[Node: model/truediv/_7571 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_16885_model/truediv", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "transformer/transformer_main.py", line 404, in <module>
tf.app.run(main=main, argv=[sys.argv[0]] + unparsed)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/platform/app.py", line 125, in run
_sys.exit(main(argv))
File "transformer/transformer_main.py", line 313, in main
FLAGS.bleu_threshold)
File "transformer/transformer_main.py", line 253, in train_schedule
estimator.train(dataset.train_input_fn, steps=single_iteration_train_steps)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/estimator.py", line 363, in train
loss = self._train_model(input_fn, hooks, saving_listeners)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/estimator.py", line 1055, in _train_model
return self._train_model_default(input_fn, hooks, saving_listeners)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/estimator.py", line 1071, in _train_model_default
saving_listeners)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/estimator.py", line 1272, in _train_with_estimator_spec
_, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/monitored_session.py", line 577, in run
run_metadata=run_metadata)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/monitored_session.py", line 1053, in run
run_metadata=run_metadata)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/monitored_session.py", line 1144, in run
raise six.reraise(*original_exc_info)
File "/usr/local/lib/python3.5/dist-packages/six.py", line 693, in reraise
raise value
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/monitored_session.py", line 1129, in run
return self._sess.run(*args, **kwargs)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/monitored_session.py", line 1201, in run
run_metadata=run_metadata)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/monitored_session.py", line 981, in run
return self._sess.run(*args, **kwargs)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 900, in run
run_metadata_ptr)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1135, in _run
feed_dict_tensor, options, run_metadata)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1316, in _do_run
run_metadata)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1335, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: Ran out of GPU memory when allocating 536361696 bytes for
[[Node: model/loss/smoothing_cross_entropy/softmax_cross_entropy_with_logits = SoftmaxCrossEntropyWithLogits[T=DT_FLOAT, _class=["loc:@model...s_grad/mul"], _device="/job:localhost/replica:0/task:0/device:GPU:0"](model/loss/smoothing_cross_entropy/softmax_cross_entropy_with_logits/Reshape, model/loss/smoothing_cross_entropy/softmax_cross_entropy_with_logits/Reshape_1)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
[[Node: model/truediv/_7571 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_16885_model/truediv", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
Caused by op 'model/loss/smoothing_cross_entropy/softmax_cross_entropy_with_logits', defined at:
File "transformer/transformer_main.py", line 404, in <module>
tf.app.run(main=main, argv=[sys.argv[0]] + unparsed)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/platform/app.py", line 125, in run
_sys.exit(main(argv))
File "transformer/transformer_main.py", line 313, in main
FLAGS.bleu_threshold)
File "transformer/transformer_main.py", line 253, in train_schedule
estimator.train(dataset.train_input_fn, steps=single_iteration_train_steps)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/estimator.py", line 363, in train
loss = self._train_model(input_fn, hooks, saving_listeners)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/estimator.py", line 1055, in _train_model
return self._train_model_default(input_fn, hooks, saving_listeners)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/estimator.py", line 1068, in _train_model_default
features, labels, model_fn_lib.ModeKeys.TRAIN, self.config)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/estimator.py", line 1043, in _call_model_fn
model_fn_results = self._model_fn(features=features, **kwargs)
File "transformer/transformer_main.py", line 67, in model_fn
logits, targets, params.label_smoothing, params.vocab_size)
File "/research/transformer/transformer/utils/metrics.py", line 77, in padded_cross_entropy_loss
logits=logits, labels=soft_targets)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/nn_ops.py", line 1879, in softmax_cross_entropy_with_logits_v2
precise_logits, labels, name=name)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gen_nn_ops.py", line 7205, in softmax_cross_entropy_with_logits
name=name)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 3417, in create_op
op_def=op_def)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 1743, in __init__
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access
ResourceExhaustedError (see above for traceback): Ran out of GPU memory when allocating 536361696 bytes for
[[Node: model/loss/smoothing_cross_entropy/softmax_cross_entropy_with_logits = SoftmaxCrossEntropyWithLogits[T=DT_FLOAT, _class=["loc:@model...s_grad/mul"], _device="/job:localhost/replica:0/task:0/device:GPU:0"](model/loss/smoothing_cross_entropy/softmax_cross_entropy_with_logits/Reshape, model/loss/smoothing_cross_entropy/softmax_cross_entropy_with_logits/Reshape_1)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
[[Node: model/truediv/_7571 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_16885_model/truediv", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
2018-05-16 14:46:52.079801: E tensorflow/core/common_runtime/bfc_allocator.cc:381] tried to deallocate nullptr
2018-05-16 14:46:53.238813: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.239041: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.239165: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.239203: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.239268: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.239482: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.239586: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.239716: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.239755: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.239844: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.239915: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.239971: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.240035: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.240069: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.240192: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.240235: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.240298: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.240420: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.240470: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.240741: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.240990: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.241139: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.241385: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.241576: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.241713: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.241866: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.242081: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.242140: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.242270: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.242342: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.242442: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.242821: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.243032: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.243114: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.243226: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.243260: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.243327: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.243553: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.243617: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.243716: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.243806: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.243919: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.244004: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.244085: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.244172: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.244287: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.244483: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.244558: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.244652: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.244714: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.244851: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.244985: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.245080: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.245145: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.245236: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.245487: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.245594: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.245775: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.245845: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.245904: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.246145: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.246221: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.246434: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.246486: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.246564: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.246645: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.246934: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.247044: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.247218: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.247273: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.247360: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.247385: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.247436: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.247492: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.247611: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.247680: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.247709: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.247790: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.247835: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.247922: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.248004: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.248101: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.248170: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.248243: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.248280: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.248366: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.248446: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.248488: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.248555: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.248595: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.248640: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.248690: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.248735: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.248767: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.248813: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.248867: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.248981: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.249041: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.249092: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.249115: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.249138: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.249162: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.249184: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.249250: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.249291: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.249383: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.249429: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.249475: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.249507: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.249583: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.249633: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.249660: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.249694: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.249750: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.249813: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.249929: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.250022: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.250131: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.250217: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.250270: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.250477: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.250544: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.250680: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.250901: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.251009: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.251050: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.251220: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.251303: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.251494: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.251526: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.251546: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.251577: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.251598: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.251619: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.251638: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.251658: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.251678: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.251699: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.251719: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.251738: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.251761: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.251781: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.251804: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.251824: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.251843: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.251862: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.251883: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.251903: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.251923: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.251943: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.251966: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.251986: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252007: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252027: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252049: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252069: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252089: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252109: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252132: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252152: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252180: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252200: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252220: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252241: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252262: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252282: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252302: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252330: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252353: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252373: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252393: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252413: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252432: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252455: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252475: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252494: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252525: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252544: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252564: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252588: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252608: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252629: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252648: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252668: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252689: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252710: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252730: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252749: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252778: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252798: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252817: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252840: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252859: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252879: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252901: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252921: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252940: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252960: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.252982: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253001: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253020: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253040: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253071: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253092: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253114: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253134: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253154: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253174: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253195: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253215: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253235: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253255: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253282: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253304: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253324: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253343: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253365: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253385: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253407: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253428: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253448: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253471: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253491: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253511: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253531: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253560: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253581: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253603: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253623: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253643: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253663: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253685: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253706: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253726: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253745: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253767: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253788: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253807: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253827: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253866: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253888: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253909: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253929: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253948: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253968: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.253987: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.254009: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.254030: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.254057: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.254079: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.254102: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.254123: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.254142: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.254165: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.254185: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.254205: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.254224: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.254245: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.254265: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.254284: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.254303: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.254334: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.254355: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.254375: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.254396: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.254416: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.254435: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.254458: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.254477: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.254499: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.254519: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.254539: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.254566: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.254587: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.254761: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.254779: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.254797: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.254820: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.254851: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.254873: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.254892: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.254915: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.254934: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.254954: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.254976: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.254995: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255015: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255035: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255056: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255076: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255102: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255124: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255143: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255163: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255185: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255205: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255224: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255246: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255265: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255285: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255304: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255323: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255353: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255373: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255392: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255415: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255434: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255454: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255477: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255496: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255516: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255537: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255556: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255577: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255596: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255622: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255644: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255664: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255686: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255706: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255725: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255748: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255770: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255790: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255812: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255832: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255851: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255871: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255900: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255921: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255943: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255963: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.255983: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256003: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256025: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256045: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256064: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256084: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256105: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256133: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256153: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256173: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256200: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256220: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256243: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256264: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256283: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256305: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256325: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256344: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256364: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256392: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256412: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256433: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256454: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256474: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256496: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256517: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256536: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256556: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256577: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256597: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256617: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256638: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256668: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256689: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256711: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256730: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256749: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256768: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256790: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256810: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256830: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256849: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256875: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256895: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256915: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256937: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256957: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256977: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.256999: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257019: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257040: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257060: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257082: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257102: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257123: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257142: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257169: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257188: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257209: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257230: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257251: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257271: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257292: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257312: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257331: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257350: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257374: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257394: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257413: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257433: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257460: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257480: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257502: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257522: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257543: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257563: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257585: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257605: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257624: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257651: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257672: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257691: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257723: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257744: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257766: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257786: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257806: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257825: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257844: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257867: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257886: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257906: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257936: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257957: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257977: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.257998: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258018: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258037: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258059: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258078: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258098: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258117: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258139: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258159: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258179: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258198: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258228: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258248: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258270: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258290: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258310: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258329: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258351: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258371: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258391: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258410: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258431: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258452: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258474: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258494: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258522: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258542: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258564: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258585: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258619: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258639: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258659: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258682: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258701: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258729: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258751: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258770: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258793: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258813: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258832: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258854: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258874: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258894: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258914: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258936: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258956: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258975: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.258995: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259023: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259044: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259066: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259087: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259107: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259134: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259156: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259176: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259197: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259222: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259243: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259263: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259282: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259312: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259333: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259354: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259373: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259393: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259414: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259434: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259454: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259474: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259503: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259522: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259544: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259563: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259582: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259601: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259623: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259643: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259662: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259683: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259703: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259722: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259743: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259773: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259793: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259814: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259834: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259853: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259872: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259894: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259913: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259933: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259953: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259974: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.259994: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.260014: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.260033: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.260060: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.260081: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.260102: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.260123: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.260142: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.260162: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.260184: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.260203: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.260225: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.260244: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.260264: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.260282: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.260310: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.260330: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.260350: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.260369: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.260391: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.260411: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.260431: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
2018-05-16 14:46:53.260450: E tensorflow/stream_executor/event.cc:32] error destroying CUDA event in context 0xddfc780: CUDA_ERROR_ILLEGAL_ADDRESS
I manage to fix this problem by adding and modifying codes in transformer_main.py line 308:
...
2018-05-17 03:16:30.080699: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 171646976 totalling 163.70MiB
2018-05-17 03:16:30.080726: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Sum Total of in-use chunks: 5.53GiB
2018-05-17 03:16:30.080757: I tensorflow/core/common_runtime/bfc_allocator.cc:680] Stats:
Limit: 5997789184
InUse: 5934368256
MaxInUse: 5934370816
NumAllocs: 1918
MaxAllocSize: 171646976
2018-05-17 03:16:30.080839: W tensorflow/core/common_runtime/bfc_allocator.cc:279] ****************************************************************************************************
2018-05-17 03:16:30.080918: W tensorflow/core/framework/op_kernel.cc:1290] CtxFailure at matmul_op.cc:474: Resource exhausted: OOM when allocating tensor with shape[3955,4096] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
Setting random seed = 1
Training schedule:
1. Train for 1 epochs.
2. Evaluate model.
3. Compute BLEU score.
Repeat above steps until the BLEU score reaches 25.0
Starting iteration 1
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1322, in _do_call
return fn(*args)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1307, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1409, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[3955,4096] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[Node: model/Transformer/decode/decoder_stack/layer_0/ffn/feed_foward_network/filter_layer/Tensordot/MatMul = MatMul[T=DT_FLOAT, _class=["loc:@model...d/MatMul_1"], transpose_a=false, transpose_b=false, _device="/job:localhost/replica:0/task:0/device:GPU:0"](model/Transformer/decode/decoder_stack/layer_0/ffn/feed_foward_network/filter_layer/Tensordot/Reshape, model/Transformer/decode/decoder_stack/layer_0/ffn/feed_foward_network/filter_layer/Tensordot/ArithmeticOptimizer/ArithmeticOptimizer/RemoveIdempotent_RemoveIdempotent_Reshape_1)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
[[Node: model/Transformer/decode/decoder_stack/layer_4/self_attention/self_attention/q/Tensordot/Shape/_5073 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_4944_...rdot/Shape", tensor_type=DT_INT32, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
...