faustomilletari / cfcm-2d

This repository contains a publicly available version of CFCM: segmentation via Coarse to Fine Context Memory, a paper accepted at MICCAI 2018 for presentation.

CFCM

This repository contains a publicly available version of CFCM: segmentation via Coarse to Fine Context Memory, a paper accepted at MICCAI 2018 for presentation. An arXiv version of the paper is available here, and the final version is available through Springer.

In this repository we provide only training/validation code. In the future it will be possibile to use CFCM to obtain predictions on the Montgomery XRay lung dataset over the cloud using TOMAAT and a compatible client (for example through 3D Slicer and its TOMAAT extension).

The code is provided without guarantees of correctness and functionality. This implementation is derived directly from the original implementation used for the CFCM paper, but it has been modified in a way that can result in unexpected behaviors. Limited support will be given through the means offered by GitHub issues reporting system.

The experiments of the CFCM paper were run on NVIDIA DGX systems with either 16Gb or 32Gb GPUs. To the best of our knowledge most experiments can run on workstations with smaller GPUs such as 1080ti, and all the experiments can surely run on GPUs equipped with 24Gb of memory (Quadro P6000 and above).

Cite

If you use CFCM for your own research please cite our paper using this bibtext:

@inbook{inbook,
    author = {Milletari, Fausto and Rieke, Nicola and Baust, Maximilian and Esposito, Marco and Navab, Nassir},
    year = {2018},
    month = {09},
    pages = {667-674},
    title = {CFCM: Segmentation via Coarse to Fine Context Memory: 21st International Conference, Granada, Spain, September 16-20, 2018, Proceedings, Part IV},
    isbn = {978-3-030-00936-6},
    doi = {10.1007/978-3-030-00937-3_76}
}

Local usage

Only recommended for development

Setup

• install C++ dependencies

  • Ubuntu: sudo apt install python-opencv ffmpeg

• (Recommended) create a python 2.7 virtual environment and activate it

virtualenv <path-to-virtualenv>
source <path-to-virtualenv>/bin/activate

• install the python dependencies

pip install -r requirements.txt

Usage

• activate the virtualenv

source <path-to-virtualenv>/bin/activate

• run training or evaluation

In order to run the training routine, a file describing the desired configuration must be specified. A set of predefined experiment setups is contained in the cfcm/experiments directory.

python cfcm/cli.py train cfcm/experiments/<experiment_name>.json

Docker usage

This section describes the usage of docker-compose, which allows to avoid verbose shell commands. For a lower-level nvidia-docker experience, please refer to the later section "Usage without docker-compose".

Prerequisites

• nvidia-container-runtime
• docker-compose >= 1.20

Setup

Register the NVidia docker runtime

By default, docker-compose is not aware of the NVidia docker runtime, so running cfcm could lead to the following error:

ERROR: Cannot create container for service cfcm: Unknown runtime specified nvidia   

In order to fix it, create a file named /etc/docker/daemon.json with the following content:

{
    "runtimes": {
        "nvidia": {
            "path": "/usr/bin/nvidia-container-runtime",
            "runtimeArgs": []
        }
    }
}

In order to apply the changes, reload the service configuration and restart it:

sudo systemctl deamon-reload
sudo systemctl restart docker

Login into the NVidia container registry

This image builds on the official NVidia Tensorflow Docker image, which is hosted on the NVidia container registry. The registry requires authentication in order to download the base image.

• create an account for Nvidia Cloud http://ngc.nvidia.com
• create an API key
• login

docker login nvcr.io
  username: $oauthtoken 
  password: <YOUR API KEY>

In order to avoid having to login manually, you can use pass.

Usage with docker-compose

• update env/local.env with the paths to this folder and to the data folder
• do the same with the paths for your remote setup in env/remote.env

Local

docker-compose run cfcm

Remote

docker-compose -f docker-compose.yml -f docker-compose-remote.yml run cfcm

Usage without Docker compose

Prerequisites

• nvidia-docker

Building

The following command builds a Docker image with all required dependencies, with the name "cfcm":

cd cfcm
nvidia-docker build -t cfcm .

Running an interactive shell

This command will start a bash shell in a container:

nvidia-docker run -it --rm --shm-size=1g --ulimit memlock=-1 --ulimit stack=67108864 cfcm bash

Acknowledgments

The code we provide in this repository is a refactoring of the original version. Most of the work for this refactoring has been done by Marco Esposito.

cfcm-2d's Issues

Result of the paper

Hi, the test code calculate the dice of one test-set.However, there are six sets in the endovis2015. How did you get the final dice in the paper form?

where is the train.py

i dont find the train.py and how to use it without docker? can you offer more details how to run this code?

what is the problem of this issue

when i run the command
python cfcm/cli.py train cfcm/experiments/ENDOVIS/endovis_[1-2-4]_[3]_LSTMResNet_18.json
the error occurs:
2018-06-19 10:57:46.264595: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX
2018-06-19 10:57:46.467164: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:892] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2018-06-19 10:57:46.467528: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Found device 0 with properties:
name: GeForce GTX 1060 6GB major: 6 minor: 1 memoryClockRate(GHz): 1.7845
pciBusID: 0000:01:00.0
totalMemory: 5.93GiB freeMemory: 5.66GiB
2018-06-19 10:57:46.467550: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: GeForce GTX 1060 6GB, pci bus id: 0000:01:00.0, compute capability: 6.1)

Load Data

/usr/local/lib/python2.7/dist-packages/skimage/transform/_warps.py:84: UserWarning: The default mode, 'constant', will be changed to 'reflect' in skimage 0.15.
warn("The default mode, 'constant', will be changed to 'reflect' in "

Load Data

2018-06-19 11:06:43.811790: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: GeForce GTX 1060 6GB, pci bus id: 0000:01:00.0, compute capability: 6.1)
epoch 0
2018-06-19 11:07:21.630296: W tensorflow/core/common_runtime/bfc_allocator.cc:217] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.12GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory is available.
2018-06-19 11:07:21.652514: W tensorflow/core/common_runtime/bfc_allocator.cc:217] Allocator (GPU_0_bfc) ran out of memory trying to allocate 868.50MiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory is available.
2018-06-19 11:07:31.729612: W tensorflow/core/common_runtime/bfc_allocator.cc:273] Allocator (GPU_0_bfc) ran out of memory trying to allocate 256.00MiB. Current allocation summary follows.
2018-06-19 11:07:31.729690: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (256): Total Chunks: 160, Chunks in use: 160. 40.0KiB allocated for chunks. 40.0KiB in use in bin. 22.3KiB client-requested in use in bin.
2018-06-19 11:07:31.729718: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (512): Total Chunks: 69, Chunks in use: 69. 34.8KiB allocated for chunks. 34.8KiB in use in bin. 34.5KiB client-requested in use in bin.
2018-06-19 11:07:31.729740: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (1024): Total Chunks: 75, Chunks in use: 75. 75.5KiB allocated for chunks. 75.5KiB in use in bin. 75.0KiB client-requested in use in bin.
2018-06-19 11:07:31.729770: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (2048): Total Chunks: 62, Chunks in use: 62. 125.8KiB allocated for chunks. 125.8KiB in use in bin. 124.0KiB client-requested in use in bin.
2018-06-19 11:07:31.729791: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (4096): Total Chunks: 8, Chunks in use: 8. 43.0KiB allocated for chunks. 43.0KiB in use in bin. 43.0KiB client-requested in use in bin.
2018-06-19 11:07:31.729811: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (8192): Total Chunks: 4, Chunks in use: 4. 32.0KiB allocated for chunks. 32.0KiB in use in bin. 32.0KiB client-requested in use in bin.
2018-06-19 11:07:31.729832: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (16384): Total Chunks: 5, Chunks in use: 4. 83.8KiB allocated for chunks. 64.0KiB in use in bin. 64.0KiB client-requested in use in bin.
2018-06-19 11:07:31.729852: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (32768): Total Chunks: 8, Chunks in use: 8. 275.0KiB allocated for chunks. 275.0KiB in use in bin. 275.0KiB client-requested in use in bin.
2018-06-19 11:07:31.729872: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (65536): Total Chunks: 5, Chunks in use: 5. 368.0KiB allocated for chunks. 368.0KiB in use in bin. 320.0KiB client-requested in use in bin.
2018-06-19 11:07:31.729905: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (131072): Total Chunks: 26, Chunks in use: 24. 3.59MiB allocated for chunks. 3.33MiB in use in bin. 3.31MiB client-requested in use in bin.
2018-06-19 11:07:31.729926: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (262144): Total Chunks: 4, Chunks in use: 4. 1.12MiB allocated for chunks. 1.12MiB in use in bin. 1.12MiB client-requested in use in bin.
2018-06-19 11:07:31.729956: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (524288): Total Chunks: 21, Chunks in use: 20. 11.31MiB allocated for chunks. 10.81MiB in use in bin. 10.75MiB client-requested in use in bin.
2018-06-19 11:07:31.729977: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (1048576): Total Chunks: 9, Chunks in use: 8. 10.12MiB allocated for chunks. 9.00MiB in use in bin. 9.00MiB client-requested in use in bin.
2018-06-19 11:07:31.729997: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (2097152): Total Chunks: 17, Chunks in use: 16. 37.25MiB allocated for chunks. 35.25MiB in use in bin. 35.00MiB client-requested in use in bin.
2018-06-19 11:07:31.730029: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (4194304): Total Chunks: 12, Chunks in use: 11. 52.50MiB allocated for chunks. 48.50MiB in use in bin. 48.00MiB client-requested in use in bin.
2018-06-19 11:07:31.730060: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (8388608): Total Chunks: 45, Chunks in use: 43. 385.00MiB allocated for chunks. 369.00MiB in use in bin. 364.00MiB client-requested in use in bin.
2018-06-19 11:07:31.730081: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (16777216): Total Chunks: 36, Chunks in use: 34. 588.00MiB allocated for chunks. 556.00MiB in use in bin. 548.00MiB client-requested in use in bin.
2018-06-19 11:07:31.730101: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (33554432): Total Chunks: 29, Chunks in use: 27. 928.00MiB allocated for chunks. 864.00MiB in use in bin. 864.00MiB client-requested in use in bin.
2018-06-19 11:07:31.730130: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (67108864): Total Chunks: 36, Chunks in use: 36. 2.28GiB allocated for chunks. 2.28GiB in use in bin. 2.28GiB client-requested in use in bin.
2018-06-19 11:07:31.730161: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (134217728): Total Chunks: 4, Chunks in use: 3. 627.04MiB allocated for chunks. 384.00MiB in use in bin. 384.00MiB client-requested in use in bin.
2018-06-19 11:07:31.730185: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (268435456): Total Chunks: 2, Chunks in use: 2. 512.00MiB allocated for chunks. 512.00MiB in use in bin. 512.00MiB client-requested in use in bin.
2018-06-19 11:07:31.730205: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin for 256.00MiB was 256.00MiB, Chunk State:
2018-06-19 11:07:31.730223: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffca00000 of size 1280
2018-06-19 11:07:31.730237: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffca00500 of size 1280
2018-06-19 11:07:31.730253: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffca00a00 of size 256
2018-06-19 11:07:31.730269: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffca00b00 of size 256
2018-06-19 11:07:31.730284: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffca00c00 of size 256
2018-06-19 11:07:31.730299: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffca00d00 of size 256
2018-06-19 11:07:31.730314: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffca00e00 of size 256
2018-06-19 11:07:31.730329: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffca00f00 of size 256
2018-06-19 11:07:31.730344: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffca01000 of size 256
2018-06-19 11:07:31.730359: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffca01100 of size 256
2018-06-19 11:07:31.730374: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffca01200 of size 256
2018-06-19 11:07:31.730389: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffca01300 of size 256
2018-06-19 11:07:31.730404: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffca01400 of size 256
2018-06-19 11:07:31.730419: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffca01500 of size 256
2018-06-19 11:07:31.730434: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffca01600 of size 256
2018-06-19 11:07:31.730449: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffca01700 of size 256
2018-06-19 11:07:31.730464: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffca01800 of size 256
2018-06-19 11:07:31.730478: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffca01900 of size 256
2018-06-19 11:07:31.730493: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffca01a00 of size 256
2018-06-19 11:07:31.730508: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffca01b00 of size 256
2018-06-19 11:07:31.730523: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffca01c00 of size 256
2018-06-19 11:07:31.730539: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffca01d00 of size 256
2018-06-19 11:07:31.730554: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffca01e00 of size 256
2018-06-19 11:07:31.730569: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffca01f00 of size 256
2018-06-19 11:07:31.730584: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffca02000 of size 256
2018-06-19 11:07:31.730599: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffca02100 of size 256
2018-06-19 11:07:31.730614: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffca02200 of size 256
2018-06-19 11:07:31.730628: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffca02300 of size 256
2018-06-19 11:07:31.730644: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffca02400 of size 256
2018-06-19 11:07:31.730658: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffca02500 of size 256
2018-06-19 11:07:31.730673: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffca02600 of size 256
2018-06-19 11:07:31.730688: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffca02700 of size 256
2018-06-19 11:07:31.730703: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffca02800 of size 256
2018-06-19 11:07:31.730718: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffca02900 of size 256
2018-06-19 11:07:31.730735: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffca02a00 of size 512
2018-06-19 11:07:31.730751: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffca02c00 of size 256
2018-06-19 11:07:31.730766: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffca02d00 of size 256
2018-06-19 11:07:31.730781: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffca02e00 of size 512
2018-06-19 11:07:31.730796: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffca03000 of size 512
2018-06-19 11:07:31.730812: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffca03200 of size 512
2018-06-19 11:07:31.730827: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffca03400 of size 512
2018-06-19 11:07:31.730842: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffca03600 of size 256
2018-06-19 11:07:31.730857: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffca03700 of size 256
2018-06-19 11:07:31.730872: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffca03800 of size 512
2018-06-19 11:07:31.730888: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffca03a00 of size 512
2018-06-19 11:07:31.730903: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffca03c00 of size 512
2018-06-19 11:07:31.730918: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffca03e00 of size 512
2018-06-19 11:07:31.730933: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffca04000 of size 256
2018-06-19 11:07:31.730948: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffca04100 of size 512
2018-06-19 11:07:31.730964: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffca04300 of size 512
2018-06-19 11:07:31.730979: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffca04500 of size 512
2018-06-19 11:07:31.730994: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffca04700 of size 512
2018-06-19 11:07:31.731009: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffca04900 of size 256
2018-06-19 11:07:31.731024: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffca04a00 of size 512
2018-06-19 11:07:31.731039: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffca04c00 of size 512
2018-06-19 11:07:31.731055: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffca04e00 of size 512
2018-06-19 11:07:31.731070: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffca05000 of size 512
2018-06-19 11:07:31.731085: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffca05200 of size 256
2018-06-19 11:07:31.731100: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffca05300 of size 256
2018-06-19 11:07:31.731116: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffca05400 of size 1024
2018-06-19 11:07:31.731131: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffca05800 of size 256
2018-06-19 11:07:31.731146: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffca05900 of size 256
2018-06-19 11:07:31.731162: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffca05a00 of size 1024
2018-06-19 11:07:31.731177: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffca05e00 of size 1024
2018-06-19 11:07:31.731192: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffca06200 of size 1024
2018-06-19 11:07:31.731206: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffca06600 of size 1024
2018-06-19 11:07:31.731221: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffca06a00 of size 256
2018-06-19 11:07:31.731237: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffca06b00 of size 256
2018-06-19 11:07:31.731252: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffca06c00 of size 1024
2018-06-19 11:07:31.731267: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffca07000 of size 1024
2018-06-19 11:07:31.731282: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffca07400 of size 1024
2018-06-19 11:07:31.731297: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffca07800 of size 1024
2018-06-19 11:07:31.731313: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffca07c00 of size 9437184
2018-06-19 11:07:31.731328: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffd307c00 of size 256
2018-06-19 11:07:31.731343: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffd307d00 of size 1024
2018-06-19 11:07:31.731358: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffd308100 of size 1024
2018-06-19 11:07:31.731374: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffd308500 of size 1024
2018-06-19 11:07:31.731389: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffd308900 of size 1024
2018-06-19 11:07:31.731404: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffd308d00 of size 256
2018-06-19 11:07:31.731419: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffd308e00 of size 1024
2018-06-19 11:07:31.731434: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffd309200 of size 1024
2018-06-19 11:07:31.731449: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffd309600 of size 1024
2018-06-19 11:07:31.731464: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffd309a00 of size 1024
2018-06-19 11:07:31.731479: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffd309e00 of size 256
2018-06-19 11:07:31.731494: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffd309f00 of size 256
2018-06-19 11:07:31.731509: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffd30a000 of size 2048
2018-06-19 11:07:31.731525: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffd30a800 of size 256
2018-06-19 11:07:31.731540: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffd30a900 of size 256
2018-06-19 11:07:31.731555: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffd30aa00 of size 2048
2018-06-19 11:07:31.731570: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffd30b200 of size 2048
2018-06-19 11:07:31.731585: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffd30ba00 of size 2048
2018-06-19 11:07:31.731599: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffd30c200 of size 2048
2018-06-19 11:07:31.731615: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffd30ca00 of size 256
2018-06-19 11:07:31.731630: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffd30cb00 of size 256
2018-06-19 11:07:31.731645: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffd30cc00 of size 2048
2018-06-19 11:07:31.731660: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffd30d400 of size 2048
2018-06-19 11:07:31.731675: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffd30dc00 of size 2048
2018-06-19 11:07:31.731690: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffd30e400 of size 2048
2018-06-19 11:07:31.731705: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffd30ec00 of size 256
2018-06-19 11:07:31.731720: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffd30ed00 of size 2048
2018-06-19 11:07:31.731735: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffd30f500 of size 2048
2018-06-19 11:07:31.731750: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffd30fd00 of size 2048
2018-06-19 11:07:31.731765: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffd310500 of size 2048
2018-06-19 11:07:31.731780: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffd310d00 of size 256
2018-06-19 11:07:31.731795: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffd310e00 of size 2048
2018-06-19 11:07:31.731810: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffd311600 of size 256
2018-06-19 11:07:31.731825: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffd311700 of size 256
2018-06-19 11:07:31.731841: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffd311800 of size 8192
2018-06-19 11:07:31.731856: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffd313800 of size 256
2018-06-19 11:07:31.731871: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffd313900 of size 256
2018-06-19 11:07:31.731886: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffd313a00 of size 2048
2018-06-19 11:07:31.731902: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffd314200 of size 256
2018-06-19 11:07:31.731917: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffd314300 of size 256
2018-06-19 11:07:31.731933: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffd314400 of size 4096
2018-06-19 11:07:31.731948: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffd315400 of size 256
2018-06-19 11:07:31.731963: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffd315500 of size 256
2018-06-19 11:07:31.731978: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffd315600 of size 1024
2018-06-19 11:07:31.731993: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffd315a00 of size 256
2018-06-19 11:07:31.732008: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffd315b00 of size 2048
2018-06-19 11:07:31.732023: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffd316300 of size 256
2018-06-19 11:07:31.732039: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffd316400 of size 256
2018-06-19 11:07:31.732054: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffd316500 of size 512
2018-06-19 11:07:31.732069: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffd316700 of size 256
2018-06-19 11:07:31.732084: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffd316800 of size 1024
2018-06-19 11:07:31.732100: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffd316c00 of size 256
2018-06-19 11:07:31.732115: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffd316d00 of size 256
2018-06-19 11:07:31.732130: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffd316e00 of size 256
2018-06-19 11:07:31.732145: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffd316f00 of size 256
2018-06-19 11:07:31.732161: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffd317000 of size 256
2018-06-19 11:07:31.732176: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffd317100 of size 256
2018-06-19 11:07:31.732191: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffd317200 of size 256
2018-06-19 11:07:31.732206: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffd317300 of size 256
2018-06-19 11:07:31.732221: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffd317400 of size 256
2018-06-19 11:07:31.732236: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffd317500 of size 256
2018-06-19 11:07:31.732252: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffd317600 of size 37632
2018-06-19 11:07:31.732267: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffd320900 of size 16384
2018-06-19 11:07:31.732283: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f4ffd324900 of size 75497472
2018-06-19 11:07:31.732299: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5001b24900 of size 147456
2018-06-19 11:07:31.732315: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5001b48900 of size 147456
2018-06-19 11:07:31.732330: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5001b6c900 of size 8388608
2018-06-19 11:07:31.732346: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500236c900 of size 147456
2018-06-19 11:07:31.732361: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5002390900 of size 147456
2018-06-19 11:07:31.732376: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50023b4900 of size 32768
2018-06-19 11:07:31.732391: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50023bc900 of size 18874368
2018-06-19 11:07:31.732407: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50035bc900 of size 294912
2018-06-19 11:07:31.732422: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5003604900 of size 147456
2018-06-19 11:07:31.732437: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5003628900 of size 589824
2018-06-19 11:07:31.732452: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50036b8900 of size 589824
2018-06-19 11:07:31.732468: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5003748900 of size 2097152
2018-06-19 11:07:31.732502: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5003948900 of size 589824
2018-06-19 11:07:31.732519: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50039d8900 of size 131072
2018-06-19 11:07:31.732536: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50039f8900 of size 4718592
2018-06-19 11:07:31.732552: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5003e78900 of size 1179648
2018-06-19 11:07:31.732569: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5003f98900 of size 2359296
2018-06-19 11:07:31.732584: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50041d8900 of size 2359296
2018-06-19 11:07:31.732600: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5004418900 of size 524288
2018-06-19 11:07:31.758168: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5004498900 of size 2359296
2018-06-19 11:07:31.758216: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50046d8900 of size 524288
2018-06-19 11:07:31.758235: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5004758900 of size 4718592
2018-06-19 11:07:31.758252: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5004bd8900 of size 1179648
2018-06-19 11:07:31.758276: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5004cf8900 of size 6912
2018-06-19 11:07:31.758289: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5004cfa400 of size 9437184
2018-06-19 11:07:31.758302: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50055fa400 of size 9437184
2018-06-19 11:07:31.758317: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005efa400 of size 65536
2018-06-19 11:07:31.758331: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005f0a400 of size 16384
2018-06-19 11:07:31.758352: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005f0e400 of size 1024
2018-06-19 11:07:31.758367: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005f0e800 of size 1024
2018-06-19 11:07:31.758381: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005f0ec00 of size 1024
2018-06-19 11:07:31.758394: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005f0f000 of size 1024
2018-06-19 11:07:31.758415: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005f0f400 of size 1024
2018-06-19 11:07:31.758439: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005f0f800 of size 1024
2018-06-19 11:07:31.758453: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005f0fc00 of size 1024
2018-06-19 11:07:31.758466: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005f10000 of size 1024
2018-06-19 11:07:31.758478: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005f10400 of size 1024
2018-06-19 11:07:31.758493: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005f10800 of size 2048
2018-06-19 11:07:31.758507: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005f11000 of size 2048
2018-06-19 11:07:31.758520: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005f11800 of size 2048
2018-06-19 11:07:31.758532: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005f12000 of size 2048
2018-06-19 11:07:31.758547: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005f12800 of size 3840
2018-06-19 11:07:31.758561: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005f13700 of size 256
2018-06-19 11:07:31.758576: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005f13800 of size 256
2018-06-19 11:07:31.758590: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005f13900 of size 256
2018-06-19 11:07:31.758604: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005f13a00 of size 256
2018-06-19 11:07:31.758619: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005f13b00 of size 6912
2018-06-19 11:07:31.758633: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005f15600 of size 256
2018-06-19 11:07:31.758647: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005f15700 of size 256
2018-06-19 11:07:31.758661: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005f15800 of size 512
2018-06-19 11:07:31.758676: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005f15a00 of size 256
2018-06-19 11:07:31.758690: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005f15b00 of size 256
2018-06-19 11:07:31.758704: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005f15c00 of size 256
2018-06-19 11:07:31.758719: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005f15d00 of size 256
2018-06-19 11:07:31.758733: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005f15e00 of size 256
2018-06-19 11:07:31.758747: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005f15f00 of size 256
2018-06-19 11:07:31.758761: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005f16000 of size 256
2018-06-19 11:07:31.758776: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005f16100 of size 256
2018-06-19 11:07:31.758790: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005f16200 of size 256
2018-06-19 11:07:31.758805: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005f16300 of size 256
2018-06-19 11:07:31.758819: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005f16400 of size 1024
2018-06-19 11:07:31.758834: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005f16800 of size 512
2018-06-19 11:07:31.758848: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005f16a00 of size 512
2018-06-19 11:07:31.758862: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005f16c00 of size 512
2018-06-19 11:07:31.758877: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005f16e00 of size 512
2018-06-19 11:07:31.758891: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005f17000 of size 512
2018-06-19 11:07:31.758906: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005f17200 of size 512
2018-06-19 11:07:31.758920: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005f17400 of size 512
2018-06-19 11:07:31.758934: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005f17600 of size 512
2018-06-19 11:07:31.758949: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005f17800 of size 768
2018-06-19 11:07:31.758964: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005f17b00 of size 256
2018-06-19 11:07:31.758979: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005f17c00 of size 147456
2018-06-19 11:07:31.758993: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005f3bc00 of size 256
2018-06-19 11:07:31.759007: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005f3bd00 of size 256
2018-06-19 11:07:31.759022: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005f3be00 of size 256
2018-06-19 11:07:31.759036: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005f3bf00 of size 256
2018-06-19 11:07:31.759051: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005f3c000 of size 147456
2018-06-19 11:07:31.759065: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005f60000 of size 256
2018-06-19 11:07:31.759079: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005f60100 of size 256
2018-06-19 11:07:31.759094: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005f60200 of size 256
2018-06-19 11:07:31.759108: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005f60300 of size 256
2018-06-19 11:07:31.759122: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005f60400 of size 147456
2018-06-19 11:07:31.759137: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005f84400 of size 256
2018-06-19 11:07:31.759151: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005f84500 of size 256
2018-06-19 11:07:31.759165: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005f84600 of size 256
2018-06-19 11:07:31.759180: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005f84700 of size 256
2018-06-19 11:07:31.759194: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005f84800 of size 32768
2018-06-19 11:07:31.765983: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005f8c800 of size 114688
2018-06-19 11:07:31.766020: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005fa8800 of size 256
2018-06-19 11:07:31.766041: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005fa8900 of size 256
2018-06-19 11:07:31.766056: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005fa8a00 of size 256
2018-06-19 11:07:31.766071: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005fa8b00 of size 256
2018-06-19 11:07:31.766086: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005fa8c00 of size 2048
2018-06-19 11:07:31.766100: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005fa9400 of size 2048
2018-06-19 11:07:31.766114: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005fa9c00 of size 2048
2018-06-19 11:07:31.766129: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005faa400 of size 1024
2018-06-19 11:07:31.766144: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005faa800 of size 256
2018-06-19 11:07:31.766158: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005faa900 of size 256
2018-06-19 11:07:31.766173: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005faaa00 of size 512
2018-06-19 11:07:31.766187: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005faac00 of size 2048
2018-06-19 11:07:31.766202: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005fab400 of size 256
2018-06-19 11:07:31.766216: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005fab500 of size 256
2018-06-19 11:07:31.766230: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005fab600 of size 512
2018-06-19 11:07:31.766245: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005fab800 of size 256
2018-06-19 11:07:31.766259: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005fab900 of size 1024
2018-06-19 11:07:31.766273: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005fb0c00 of size 512
2018-06-19 11:07:31.766288: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005fb0e00 of size 131072
2018-06-19 11:07:31.766304: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005fd0e00 of size 163840
2018-06-19 11:07:31.766318: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005ff8e00 of size 512
2018-06-19 11:07:31.766333: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005ff9000 of size 512
2018-06-19 11:07:31.766347: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005ff9200 of size 512
2018-06-19 11:07:31.766361: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005ff9400 of size 512
2018-06-19 11:07:31.766376: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5005ff9600 of size 589824
2018-06-19 11:07:31.766390: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5006089600 of size 512
2018-06-19 11:07:31.766404: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5006089800 of size 512
2018-06-19 11:07:31.766419: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5006089a00 of size 512
2018-06-19 11:07:31.766433: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5006089c00 of size 512
2018-06-19 11:07:31.766448: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5006089e00 of size 589824
2018-06-19 11:07:31.766462: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5006119e00 of size 589824
2018-06-19 11:07:31.766476: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50061a9e00 of size 512
2018-06-19 11:07:31.766491: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50061aa000 of size 512
2018-06-19 11:07:31.766505: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50061aa200 of size 512
2018-06-19 11:07:31.766519: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50061aa400 of size 512
2018-06-19 11:07:31.766534: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50061aa600 of size 589824
2018-06-19 11:07:31.766548: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500623a600 of size 589824
2018-06-19 11:07:31.766562: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50062ca600 of size 2097152
2018-06-19 11:07:31.766577: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50064ca600 of size 2097152
2018-06-19 11:07:31.766591: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50066ca600 of size 589824
2018-06-19 11:07:31.766605: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500675a600 of size 589824
2018-06-19 11:07:31.766619: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50067ea600 of size 131072
2018-06-19 11:07:31.766634: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500680a600 of size 131072
2018-06-19 11:07:31.766648: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500682a600 of size 4718592
2018-06-19 11:07:31.766663: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5006caa600 of size 1179648
2018-06-19 11:07:31.766677: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5006dca600 of size 1179648
2018-06-19 11:07:31.766691: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5006eea600 of size 4718592
2018-06-19 11:07:31.766705: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500736a600 of size 2359296
2018-06-19 11:07:31.766720: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50075aa600 of size 2359296
2018-06-19 11:07:31.766734: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50077ea600 of size 2359296
2018-06-19 11:07:31.766748: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5007a2a600 of size 2359296
2018-06-19 11:07:31.766762: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5007c6a600 of size 524288
2018-06-19 11:07:31.766776: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5007cea600 of size 524288
2018-06-19 11:07:31.766791: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5007d6a600 of size 2359296
2018-06-19 11:07:31.766805: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5007faa600 of size 2359296
2018-06-19 11:07:31.766819: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50081ea600 of size 524288
2018-06-19 11:07:31.766833: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500826a600 of size 524288
2018-06-19 11:07:31.766848: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50082ea600 of size 4718592
2018-06-19 11:07:31.766862: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500876a600 of size 4718592
2018-06-19 11:07:31.766877: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5008bea600 of size 1179648
2018-06-19 11:07:31.766891: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5008d0a600 of size 1179648
2018-06-19 11:07:31.766905: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5008e2a600 of size 6912
2018-06-19 11:07:31.766920: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5008e2c100 of size 9437184
2018-06-19 11:07:31.766934: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500972c100 of size 9437184
2018-06-19 11:07:31.766948: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500a02c100 of size 6912
2018-06-19 11:07:31.766963: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500a02dc00 of size 9437184
2018-06-19 11:07:31.766977: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500a92dc00 of size 9437184
2018-06-19 11:07:31.766992: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500b22dc00 of size 65536
2018-06-19 11:07:31.767006: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500b23dc00 of size 65536
2018-06-19 11:07:31.767020: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500b24dc00 of size 512
2018-06-19 11:07:31.767034: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500b24de00 of size 589824
2018-06-19 11:07:31.767049: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500b2dde00 of size 512
2018-06-19 11:07:31.767063: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500b2de000 of size 512
2018-06-19 11:07:31.767078: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500b2de200 of size 512
2018-06-19 11:07:31.767093: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500b2de400 of size 512
2018-06-19 11:07:31.767107: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500b2de600 of size 512
2018-06-19 11:07:31.767122: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500b2de800 of size 512
2018-06-19 11:07:31.767136: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500b2dea00 of size 512
2018-06-19 11:07:31.767151: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500b2dec00 of size 512
2018-06-19 11:07:31.767165: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500b2dee00 of size 589824
2018-06-19 11:07:31.767180: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500b36ee00 of size 512
2018-06-19 11:07:31.767194: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500b36f000 of size 512
2018-06-19 11:07:31.767209: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500b36f200 of size 512
2018-06-19 11:07:31.767223: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500b36f400 of size 512
2018-06-19 11:07:31.767238: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500b36f600 of size 512
2018-06-19 11:07:31.767252: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500b36f800 of size 512
2018-06-19 11:07:31.767267: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500b36fa00 of size 512
2018-06-19 11:07:31.767281: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500b36fc00 of size 512
2018-06-19 11:07:31.767296: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500b38fe00 of size 1024
2018-06-19 11:07:31.767310: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500b390200 of size 1024
2018-06-19 11:07:31.767325: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500b390600 of size 1024
2018-06-19 11:07:31.825644: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500b390a00 of size 1179648
2018-06-19 11:07:31.825688: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500b4b0a00 of size 1024
2018-06-19 11:07:31.825708: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500b4b0e00 of size 1024
2018-06-19 11:07:31.825721: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500b4b1200 of size 1024
2018-06-19 11:07:31.825735: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500b4b1600 of size 1024
2018-06-19 11:07:31.825746: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500b4b1a00 of size 1024
2018-06-19 11:07:31.825758: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500b4b1e00 of size 1024
2018-06-19 11:07:31.825770: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500b4b2200 of size 1024
2018-06-19 11:07:31.825782: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500b4b2600 of size 1024
2018-06-19 11:07:31.825793: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500b4b2a00 of size 2359296
2018-06-19 11:07:31.825805: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500b6f2a00 of size 1024
2018-06-19 11:07:31.825825: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500b6f2e00 of size 1024
2018-06-19 11:07:31.825838: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500b6f3200 of size 1024
2018-06-19 11:07:31.825851: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500b6f3600 of size 1024
2018-06-19 11:07:31.825864: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500b6f3a00 of size 1024
2018-06-19 11:07:31.825877: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500b6f3e00 of size 1024
2018-06-19 11:07:31.825890: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500b6f4200 of size 1024
2018-06-19 11:07:31.825908: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500b6f4600 of size 1024
2018-06-19 11:07:31.825922: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500b6f4a00 of size 2359296
2018-06-19 11:07:31.825935: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500b934a00 of size 9437184
2018-06-19 11:07:31.825948: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500c234a00 of size 1024
2018-06-19 11:07:31.825960: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500c234e00 of size 1024
2018-06-19 11:07:31.825973: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500c235200 of size 1024
2018-06-19 11:07:31.825986: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500c235600 of size 1024
2018-06-19 11:07:31.825999: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500c235a00 of size 1024
2018-06-19 11:07:31.826012: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500c235e00 of size 1024
2018-06-19 11:07:31.826025: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500c236200 of size 1024
2018-06-19 11:07:31.826038: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500c236600 of size 1024
2018-06-19 11:07:31.826051: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500c236a00 of size 2359296
2018-06-19 11:07:31.826064: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500c476a00 of size 1024
2018-06-19 11:07:31.826076: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500c476e00 of size 1024
2018-06-19 11:07:31.826090: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500c477200 of size 1024
2018-06-19 11:07:31.826103: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500c477600 of size 1024
2018-06-19 11:07:31.826115: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500c477a00 of size 1024
2018-06-19 11:07:31.826128: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500c477e00 of size 1024
2018-06-19 11:07:31.826140: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500c478200 of size 1024
2018-06-19 11:07:31.826153: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500c478600 of size 1024
2018-06-19 11:07:31.826166: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500c478a00 of size 524288
2018-06-19 11:07:31.826178: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500c4f8a00 of size 2048
2018-06-19 11:07:31.826191: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500c4f9200 of size 2048
2018-06-19 11:07:31.826204: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500c4f9a00 of size 2048
2018-06-19 11:07:31.826217: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500c4fa200 of size 4718592
2018-06-19 11:07:31.826229: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500c97a200 of size 2048
2018-06-19 11:07:31.826242: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500c97aa00 of size 2048
2018-06-19 11:07:31.826255: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500c97b200 of size 2048
2018-06-19 11:07:31.826267: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500c97ba00 of size 2048
2018-06-19 11:07:31.826280: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500c97c200 of size 2048
2018-06-19 11:07:31.826293: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500c97ca00 of size 2048
2018-06-19 11:07:31.826305: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500c97d200 of size 2048
2018-06-19 11:07:31.826318: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500c97da00 of size 2048
2018-06-19 11:07:31.826331: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500c97e200 of size 9437184
2018-06-19 11:07:31.826343: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500d27e200 of size 2048
2018-06-19 11:07:31.826357: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500d27ea00 of size 2048
2018-06-19 11:07:31.826369: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500d27f200 of size 2048
2018-06-19 11:07:31.826382: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500d27fa00 of size 2048
2018-06-19 11:07:31.826394: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500d280200 of size 2048
2018-06-19 11:07:31.826407: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500d280a00 of size 2048
2018-06-19 11:07:31.826420: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500d281200 of size 2048
2018-06-19 11:07:31.826432: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500d281a00 of size 2048
2018-06-19 11:07:31.826445: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500d282200 of size 9437184
2018-06-19 11:07:31.826457: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500db82200 of size 9437184
2018-06-19 11:07:31.826470: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500e482200 of size 2048
2018-06-19 11:07:31.826483: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500e482a00 of size 2048
2018-06-19 11:07:31.826496: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500e483200 of size 2048
2018-06-19 11:07:31.826508: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500e483a00 of size 2048
2018-06-19 11:07:31.826521: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500e484200 of size 2048
2018-06-19 11:07:31.826534: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500e484a00 of size 2048
2018-06-19 11:07:31.826546: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500e485200 of size 2048
2018-06-19 11:07:31.826559: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500e485a00 of size 2048
2018-06-19 11:07:31.826572: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500e486200 of size 9437184
2018-06-19 11:07:31.826584: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500ed86200 of size 2048
2018-06-19 11:07:31.826597: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500ed86a00 of size 18874368
2018-06-19 11:07:31.826611: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f500ff86a00 of size 12582912
2018-06-19 11:07:31.826624: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5010b86a00 of size 12582912
2018-06-19 11:07:31.826637: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5011786a00 of size 12582912
2018-06-19 11:07:31.826650: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5012386a00 of size 18874368
2018-06-19 11:07:31.826663: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5013586a00 of size 8192
2018-06-19 11:07:31.826675: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5013588a00 of size 8388608
2018-06-19 11:07:31.826688: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5013d88a00 of size 2048
2018-06-19 11:07:31.826700: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5013d89200 of size 18874368
2018-06-19 11:07:31.826713: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5014f89200 of size 4096
2018-06-19 11:07:31.826726: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5014f8a200 of size 4096
2018-06-19 11:07:31.826739: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5014f8b200 of size 4096
2018-06-19 11:07:31.826751: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f501518c200 of size 1024
2018-06-19 11:07:31.826764: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f501518c600 of size 1024
2018-06-19 11:07:31.826776: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f501518ca00 of size 1024
2018-06-19 11:07:31.826789: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f501518ce00 of size 4718592
2018-06-19 11:07:31.826801: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f501560ce00 of size 2048
2018-06-19 11:07:31.826814: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f501560d600 of size 2048
2018-06-19 11:07:31.826826: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f501560de00 of size 2048
2018-06-19 11:07:31.826839: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f501568e600 of size 512
2018-06-19 11:07:31.826851: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f501568e800 of size 512
2018-06-19 11:07:31.826864: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f501568ea00 of size 512
2018-06-19 11:07:31.826877: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50157aec00 of size 2048
2018-06-19 11:07:31.826890: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50157af400 of size 1024
2018-06-19 11:07:31.826902: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50157af800 of size 1024
2018-06-19 11:07:31.826915: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50157afc00 of size 1024
2018-06-19 11:07:31.826927: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50157b0000 of size 65536
2018-06-19 11:07:31.826940: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50157c0000 of size 256
2018-06-19 11:07:31.826952: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50157e4100 of size 256
2018-06-19 11:07:31.826965: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50157e4200 of size 256
2018-06-19 11:07:31.826977: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50157e4300 of size 256
2018-06-19 11:07:31.826990: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50157e4400 of size 512
2018-06-19 11:07:31.827003: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50157e4600 of size 256
2018-06-19 11:07:31.827015: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50157e4700 of size 256
2018-06-19 11:07:31.827028: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50157e4800 of size 256
2018-06-19 11:07:31.827041: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50157e4900 of size 256
2018-06-19 11:07:31.827053: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50157e4a00 of size 256
2018-06-19 11:07:31.827066: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50157e4b00 of size 256
2018-06-19 11:07:31.827078: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50157e4c00 of size 256
2018-06-19 11:07:31.827090: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50157e4d00 of size 2048
2018-06-19 11:07:31.827109: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50157e5500 of size 1024
2018-06-19 11:07:31.827123: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50157e5900 of size 256
2018-06-19 11:07:31.827136: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50157e5a00 of size 256
2018-06-19 11:07:31.827149: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50157e5b00 of size 256
2018-06-19 11:07:31.827161: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50157e5c00 of size 256
2018-06-19 11:07:31.827174: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50157e5d00 of size 256
2018-06-19 11:07:31.827187: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50157e5e00 of size 256
2018-06-19 11:07:31.827200: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50157e5f00 of size 256
2018-06-19 11:07:31.827213: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50157e6000 of size 256
2018-06-19 11:07:31.827226: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50157e6100 of size 256
2018-06-19 11:07:31.827238: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50157e6200 of size 256
2018-06-19 11:07:31.827251: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50157e6300 of size 37632
2018-06-19 11:07:31.827264: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50157ef600 of size 37632
2018-06-19 11:07:31.827277: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50157f8900 of size 256
2018-06-19 11:07:31.827290: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50157f8a00 of size 2048
2018-06-19 11:07:31.827303: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50157f9200 of size 256
2018-06-19 11:07:31.827316: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50157f9300 of size 256
2018-06-19 11:07:31.827328: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50157f9400 of size 256
2018-06-19 11:07:31.827341: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50157f9500 of size 256
2018-06-19 11:07:31.827354: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50157f9600 of size 16384
2018-06-19 11:07:31.827367: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50157fd600 of size 16384
2018-06-19 11:07:31.827379: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5015801600 of size 75497472
2018-06-19 11:07:31.827392: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f501a001600 of size 256
2018-06-19 11:07:31.827405: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f501a001700 of size 256
2018-06-19 11:07:31.827418: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f501a001800 of size 75497472
2018-06-19 11:07:31.827431: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f501e801800 of size 147456
2018-06-19 11:07:31.827444: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f501e825800 of size 147456
2018-06-19 11:07:31.827456: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f501e849800 of size 256
2018-06-19 11:07:31.827469: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f501e849900 of size 256
2018-06-19 11:07:31.854081: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f501e849a00 of size 8192
2018-06-19 11:07:31.854128: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f501e84ba00 of size 256
2018-06-19 11:07:31.854144: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f501e84bb00 of size 256
2018-06-19 11:07:31.854158: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f501e84bc00 of size 147456
2018-06-19 11:07:31.854173: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f501e86fc00 of size 147456
2018-06-19 11:07:31.854188: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f501e893c00 of size 8192
2018-06-19 11:07:31.854202: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f501e895c00 of size 256
2018-06-19 11:07:31.854214: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f501e895d00 of size 256
2018-06-19 11:07:31.854227: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f501e895e00 of size 256
2018-06-19 11:07:31.854249: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f501e895f00 of size 256
2018-06-19 11:07:31.854263: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f501e896000 of size 256
2018-06-19 11:07:31.854276: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f501e896100 of size 8388608
2018-06-19 11:07:31.854299: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f501f096100 of size 147456
2018-06-19 11:07:31.854313: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f501f0ba100 of size 147456
2018-06-19 11:07:31.854327: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f501f0de100 of size 256
2018-06-19 11:07:31.854341: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f501f0de200 of size 8388608
2018-06-19 11:07:31.854355: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f501f8de200 of size 256
2018-06-19 11:07:31.854368: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f501f8de300 of size 256
2018-06-19 11:07:31.854382: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f501f8de400 of size 256
2018-06-19 11:07:31.854396: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f501f8de500 of size 256
2018-06-19 11:07:31.854410: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f501f8de600 of size 2048
2018-06-19 11:07:31.854424: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f501f8dee00 of size 147456
2018-06-19 11:07:31.854437: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f501f902e00 of size 147456
2018-06-19 11:07:31.854451: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f501f926e00 of size 256
2018-06-19 11:07:31.854465: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f501f926f00 of size 256
2018-06-19 11:07:31.854479: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f501f927000 of size 2048
2018-06-19 11:07:31.854493: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f501f927800 of size 256
2018-06-19 11:07:31.854507: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f501f927900 of size 256
2018-06-19 11:07:31.854521: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f501f927a00 of size 32768
2018-06-19 11:07:31.854535: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f501f92fa00 of size 32768
2018-06-19 11:07:31.854548: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f501f937a00 of size 512
2018-06-19 11:07:31.854562: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f501f937c00 of size 18874368
2018-06-19 11:07:31.854576: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5020b37c00 of size 512
2018-06-19 11:07:31.854590: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5020b37e00 of size 294912
2018-06-19 11:07:31.854603: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5020b7fe00 of size 294912
2018-06-19 11:07:31.854617: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5020bc7e00 of size 18874368
2018-06-19 11:07:31.854631: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5021dc7e00 of size 512
2018-06-19 11:07:31.854645: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5021dc8000 of size 147456
2018-06-19 11:07:31.854659: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5021dec000 of size 147456
2018-06-19 11:07:31.854672: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5021e10000 of size 512
2018-06-19 11:07:31.854686: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5021e10200 of size 512
2018-06-19 11:07:31.854700: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5021e10400 of size 37632
2018-06-19 11:07:31.854714: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5021e19700 of size 147456
2018-06-19 11:07:31.854727: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5021e3d700 of size 294912
2018-06-19 11:07:31.854741: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5021e85700 of size 589824
2018-06-19 11:07:31.854755: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5021f15700 of size 1179648
2018-06-19 11:07:31.854769: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5022035700 of size 2359296
2018-06-19 11:07:31.854782: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5022275700 of size 4718592
2018-06-19 11:07:31.854796: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50226f5700 of size 9437184
2018-06-19 11:07:31.854810: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5022ff5700 of size 75497472
2018-06-19 11:07:31.854825: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50277f5700 of size 268435456
2018-06-19 11:07:31.854838: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50377f5700 of size 67108864
2018-06-19 11:07:31.854852: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f503b7f5700 of size 67108864
2018-06-19 11:07:31.854864: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f503f7f5700 of size 67108864
2018-06-19 11:07:31.854878: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50437f5700 of size 67108864
2018-06-19 11:07:31.854892: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50477f5700 of size 67108864
2018-06-19 11:07:31.854905: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f504b7f5700 of size 67108864
2018-06-19 11:07:31.854919: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f504f7f5700 of size 67108864
2018-06-19 11:07:31.854933: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50537f5700 of size 67108864
2018-06-19 11:07:31.854946: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50577f5700 of size 67108864
2018-06-19 11:07:31.854960: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f505b7f5700 of size 67108864
2018-06-19 11:07:31.854974: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f505f7f5700 of size 33554432
2018-06-19 11:07:31.854989: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50617f5700 of size 16777216
2018-06-19 11:07:31.855003: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50627f5700 of size 8388608
2018-06-19 11:07:31.855022: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5062ff5700 of size 8388608
2018-06-19 11:07:31.855037: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50637f5700 of size 67108864
2018-06-19 11:07:31.855051: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50677f5700 of size 67108864
2018-06-19 11:07:31.855064: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f506b7f5700 of size 16777216
2018-06-19 11:07:31.855079: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f506c7f5700 of size 33554432
2018-06-19 11:07:31.855093: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f506e7f5700 of size 33554432
2018-06-19 11:07:31.855107: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50707f5700 of size 33554432
2018-06-19 11:07:31.855138: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50727f5700 of size 33554432
2018-06-19 11:07:31.855155: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50747f5700 of size 33554432
2018-06-19 11:07:31.855169: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50767f5700 of size 33554432
2018-06-19 11:07:31.855183: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50787f5700 of size 33554432
2018-06-19 11:07:31.855197: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f507a7f5700 of size 33554432
2018-06-19 11:07:31.855211: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f507c7f5700 of size 16777216
2018-06-19 11:07:31.855224: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f507d7f5700 of size 16777216
2018-06-19 11:07:31.855239: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f507e7f5700 of size 33554432
2018-06-19 11:07:31.855252: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50807f5700 of size 33554432
2018-06-19 11:07:31.855266: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50827f5700 of size 16777216
2018-06-19 11:07:31.855280: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50837f5700 of size 16777216
2018-06-19 11:07:31.855294: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50847f5700 of size 16777216
2018-06-19 11:07:31.855308: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50857f5700 of size 8388608
2018-06-19 11:07:31.855321: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5085ff5700 of size 8388608
2018-06-19 11:07:31.855335: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50867f5700 of size 16777216
2018-06-19 11:07:31.855349: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50877f5700 of size 16777216
2018-06-19 11:07:31.855362: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50887f5700 of size 16777216
2018-06-19 11:07:31.855376: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50897f5700 of size 16777216
2018-06-19 11:07:31.855389: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f508a7f5700 of size 16777216
2018-06-19 11:07:31.855403: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f508b7f5700 of size 16777216
2018-06-19 11:07:31.855426: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f508c7f5700 of size 16777216
2018-06-19 11:07:31.855442: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f508d7f5700 of size 4194304
2018-06-19 11:07:31.855456: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f508dbf5700 of size 8388608
2018-06-19 11:07:31.855470: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f508e3f5700 of size 8388608
2018-06-19 11:07:31.855484: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f508ebf5700 of size 8388608
2018-06-19 11:07:31.855498: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f508f3f5700 of size 4194304
2018-06-19 11:07:31.855512: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f508fbf5700 of size 8388608
2018-06-19 11:07:31.855533: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50903f5700 of size 8388608
2018-06-19 11:07:31.855548: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5090bf5700 of size 8388608
2018-06-19 11:07:31.855562: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50913f5700 of size 8388608
2018-06-19 11:07:31.855576: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5091bf5700 of size 8388608
2018-06-19 11:07:31.855589: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50923f5700 of size 8388608
2018-06-19 11:07:31.855603: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5092bf5700 of size 67108864
2018-06-19 11:07:31.855617: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5096bf5700 of size 67108864
2018-06-19 11:07:31.855630: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f509abf5700 of size 8388608
2018-06-19 11:07:31.855644: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f509b3f5700 of size 8388608
2018-06-19 11:07:31.855657: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f509bbf5700 of size 8388608
2018-06-19 11:07:31.855671: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f509c3f5700 of size 8388608
2018-06-19 11:07:31.855685: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f509cbf5700 of size 8388608
2018-06-19 11:07:31.855698: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f509d3f5700 of size 8388608
2018-06-19 11:07:31.855711: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f509dbf5700 of size 16777216
2018-06-19 11:07:31.855725: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f509ebf5700 of size 8388608
2018-06-19 11:07:31.855738: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f509fbf5700 of size 8388608
2018-06-19 11:07:31.855752: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50a03f5700 of size 8388608
2018-06-19 11:07:31.855765: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50a0bf5700 of size 8388608
2018-06-19 11:07:31.855779: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50a1bf5700 of size 16777216
2018-06-19 11:07:31.855792: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50a2bf5700 of size 33554432
2018-06-19 11:07:31.855806: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50a4bf5700 of size 16777216
2018-06-19 11:07:31.855819: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50a5bf5700 of size 16777216
2018-06-19 11:07:31.855833: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50a6bf5700 of size 33554432
2018-06-19 11:07:31.855846: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50a8bf5700 of size 16777216
2018-06-19 11:07:31.855860: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50a9bf5700 of size 16777216
2018-06-19 11:07:31.855873: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50aabf5700 of size 16777216
2018-06-19 11:07:31.855887: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50abbf5700 of size 16777216
2018-06-19 11:07:31.855900: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50acbf5700 of size 16777216
2018-06-19 11:07:31.855914: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50adbf5700 of size 16777216
2018-06-19 11:07:31.855927: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50aebf5700 of size 33554432
2018-06-19 11:07:31.855941: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50b0bf5700 of size 16777216
2018-06-19 11:07:31.855954: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50b2bf5700 of size 16777216
2018-06-19 11:07:31.855968: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50b3bf5700 of size 16777216
2018-06-19 11:07:31.855981: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50b4bf5700 of size 16777216
2018-06-19 11:07:31.855995: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50b6bf5700 of size 33554432
2018-06-19 11:07:31.856008: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50b8bf5700 of size 67108864
2018-06-19 11:07:31.856030: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50bcbf5700 of size 33554432
2018-06-19 11:07:31.856053: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50bebf5700 of size 33554432
2018-06-19 11:07:31.856066: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50c0bf5700 of size 67108864
2018-06-19 11:07:31.856083: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50c4bf5700 of size 33554432
2018-06-19 11:07:31.856097: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50c6bf5700 of size 33554432
2018-06-19 11:07:31.856110: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50c8bf5700 of size 33554432
2018-06-19 11:07:31.856124: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50cabf5700 of size 33554432
2018-06-19 11:07:31.856136: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50ccbf5700 of size 33554432
2018-06-19 11:07:31.856147: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50cebf5700 of size 33554432
2018-06-19 11:07:31.856159: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50d0bf5700 of size 67108864
2018-06-19 11:07:31.856172: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50d4bf5700 of size 33554432
2018-06-19 11:07:31.856185: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50d8bf5700 of size 33554432
2018-06-19 11:07:31.856198: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50dabf5700 of size 33554432
2018-06-19 11:07:31.856211: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50dcbf5700 of size 33554432
2018-06-19 11:07:31.856224: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50e0bf5700 of size 67108864
2018-06-19 11:07:31.856238: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50e4bf5700 of size 134217728
2018-06-19 11:07:31.856251: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50ecbf5700 of size 67108864
2018-06-19 11:07:31.856264: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50f0bf5700 of size 67108864
2018-06-19 11:07:31.856277: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50f4bf5700 of size 134217728
2018-06-19 11:07:31.856290: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f50fcbf5700 of size 67108864
2018-06-19 11:07:31.856303: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5100bf5700 of size 67108864
2018-06-19 11:07:31.856316: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5104bf5700 of size 67108864
2018-06-19 11:07:31.856330: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5108bf5700 of size 67108864
2018-06-19 11:07:31.856343: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f510cbf5700 of size 67108864
2018-06-19 11:07:31.856356: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5110bf5700 of size 67108864
2018-06-19 11:07:31.856369: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5114bf5700 of size 134217728
2018-06-19 11:07:31.856382: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f511cbf5700 of size 67108864
2018-06-19 11:07:31.856395: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5120bf5700 of size 67108864
2018-06-19 11:07:31.856409: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5124bf5700 of size 67108864
2018-06-19 11:07:31.856421: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5128bf5700 of size 67108864
2018-06-19 11:07:31.856434: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f512cbf5700 of size 67108864
2018-06-19 11:07:31.856447: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5130bf5700 of size 268435456
2018-06-19 11:07:31.856460: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x7f5140bf5700 of size 67108864
2018-06-19 11:07:31.856490: I tensorflow/core/common_runtime/bfc_allocator.cc:670] Free at 0x7f5005fabd00 of size 20224
2018-06-19 11:07:31.856507: I tensorflow/core/common_runtime/bfc_allocator.cc:670] Free at 0x7f500b36fe00 of size 131072
2018-06-19 11:07:31.856522: I tensorflow/core/common_runtime/bfc_allocator.cc:670] Free at 0x7f5014f8c200 of size 2097152
2018-06-19 11:07:31.856536: I tensorflow/core/common_runtime/bfc_allocator.cc:670] Free at 0x7f501560e600 of size 524288
2018-06-19 11:07:31.856549: I tensorflow/core/common_runtime/bfc_allocator.cc:670] Free at 0x7f501568ec00 of size 1179648
2018-06-19 11:07:31.856562: I tensorflow/core/common_runtime/bfc_allocator.cc:670] Free at 0x7f50157c0100 of size 147456
2018-06-19 11:07:31.856576: I tensorflow/core/common_runtime/bfc_allocator.cc:670] Free at 0x7f508f7f5700 of size 4194304
2018-06-19 11:07:31.856590: I tensorflow/core/common_runtime/bfc_allocator.cc:670] Free at 0x7f509f3f5700 of size 8388608
2018-06-19 11:07:31.856603: I tensorflow/core/common_runtime/bfc_allocator.cc:670] Free at 0x7f50a13f5700 of size 8388608
2018-06-19 11:07:31.856614: I tensorflow/core/common_runtime/bfc_allocator.cc:670] Free at 0x7f50b1bf5700 of size 16777216
2018-06-19 11:07:31.856626: I tensorflow/core/common_runtime/bfc_allocator.cc:670] Free at 0x7f50b5bf5700 of size 16777216
2018-06-19 11:07:31.856645: I tensorflow/core/common_runtime/bfc_allocator.cc:670] Free at 0x7f50d6bf5700 of size 33554432
2018-06-19 11:07:31.856665: I tensorflow/core/common_runtime/bfc_allocator.cc:670] Free at 0x7f50debf5700 of size 33554432
2018-06-19 11:07:31.856680: I tensorflow/core/common_runtime/bfc_allocator.cc:670] Free at 0x7f5144bf5700 of size 254847232
2018-06-19 11:07:31.856693: I tensorflow/core/common_runtime/bfc_allocator.cc:676] Summary of in-use Chunks by size:
2018-06-19 11:07:31.856713: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 160 Chunks of size 256 totalling 40.0KiB
2018-06-19 11:07:31.856729: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 68 Chunks of size 512 totalling 34.0KiB
2018-06-19 11:07:31.856744: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 1 Chunks of size 768 totalling 768B
2018-06-19 11:07:31.856759: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 73 Chunks of size 1024 totalling 73.0KiB
2018-06-19 11:07:31.856774: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 2 Chunks of size 1280 totalling 2.5KiB
2018-06-19 11:07:31.856789: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 61 Chunks of size 2048 totalling 122.0KiB
2018-06-19 11:07:31.856804: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 1 Chunks of size 3840 totalling 3.8KiB
2018-06-19 11:07:31.856819: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 4 Chunks of size 4096 totalling 16.0KiB
2018-06-19 11:07:31.856833: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 4 Chunks of size 6912 totalling 27.0KiB
2018-06-19 11:07:31.856848: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 4 Chunks of size 8192 totalling 32.0KiB
2018-06-19 11:07:31.856862: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 4 Chunks of size 16384 totalling 64.0KiB
2018-06-19 11:07:31.856877: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 4 Chunks of size 32768 totalling 128.0KiB
2018-06-19 11:07:31.856892: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 4 Chunks of size 37632 totalling 147.0KiB
2018-06-19 11:07:31.856907: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 4 Chunks of size 65536 totalling 256.0KiB
2018-06-19 11:07:31.856922: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 1 Chunks of size 114688 totalling 112.0KiB
2018-06-19 11:07:31.856937: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 4 Chunks of size 131072 totalling 512.0KiB
2018-06-19 11:07:31.856952: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 19 Chunks of size 147456 totalling 2.67MiB
2018-06-19 11:07:31.856967: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 1 Chunks of size 163840 totalling 160.0KiB
2018-06-19 11:07:31.856981: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 4 Chunks of size 294912 totalling 1.12MiB
2018-06-19 11:07:31.856996: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 7 Chunks of size 524288 totalling 3.50MiB
2018-06-19 11:07:31.857010: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 13 Chunks of size 589824 totalling 7.31MiB
2018-06-19 11:07:31.857026: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 8 Chunks of size 1179648 totalling 9.00MiB
2018-06-19 11:07:31.857040: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 3 Chunks of size 2097152 totalling 6.00MiB
2018-06-19 11:07:31.857056: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 13 Chunks of size 2359296 totalling 29.25MiB
2018-06-19 11:07:31.857071: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 2 Chunks of size 4194304 totalling 8.00MiB
2018-06-19 11:07:31.857086: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 9 Chunks of size 4718592 totalling 40.50MiB
2018-06-19 11:07:31.857101: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 27 Chunks of size 8388608 totalling 216.00MiB
2018-06-19 11:07:31.857116: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 13 Chunks of size 9437184 totalling 117.00MiB
2018-06-19 11:07:31.857131: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 3 Chunks of size 12582912 totalling 36.00MiB
2018-06-19 11:07:31.857147: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 28 Chunks of size 16777216 totalling 448.00MiB
2018-06-19 11:07:31.857171: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 6 Chunks of size 18874368 totalling 108.00MiB
2018-06-19 11:07:31.857197: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 27 Chunks of size 33554432 totalling 864.00MiB
2018-06-19 11:07:31.857221: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 32 Chunks of size 67108864 totalling 2.00GiB
2018-06-19 11:07:31.857245: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 4 Chunks of size 75497472 totalling 288.00MiB
2018-06-19 11:07:31.857268: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 3 Chunks of size 134217728 totalling 384.00MiB
2018-06-19 11:07:31.857292: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 2 Chunks of size 268435456 totalling 512.00MiB
2018-06-19 11:07:31.857308: I tensorflow/core/common_runtime/bfc_allocator.cc:683] Sum Total of in-use chunks: 5.01GiB
2018-06-19 11:07:31.857325: I tensorflow/core/common_runtime/bfc_allocator.cc:685] Stats:
Limit: 5759827968
InUse: 5379246080
MaxInUse: 5755642880
NumAllocs: 952
MaxAllocSize: 4369459456

2018-06-19 11:07:31.857383: W tensorflow/core/common_runtime/bfc_allocator.cc:277] ************************************************************************************************____
2018-06-19 11:07:31.857425: W tensorflow/core/framework/op_kernel.cc:1192] Resource exhausted: OOM when allocating tensor with shape[16,64,256,256]
Traceback (most recent call last):
File "cfcm/cli.py", line 55, in
cli()
File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 722, in call
return self.main(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "cfcm/cli.py", line 48, in train
fit_runner.run_epoch_train()
File "/home/lvhao/CFCM-2D-master/cfcm/utilities/timing.py", line 9, in newfunc
retargs = func(*args, **kwargs)
File "/home/lvhao/CFCM-2D-master/cfcm/zoo/fitrunners.py", line 541, in run_epoch_train
summaries=[],
File "/home/lvhao/CFCM-2D-master/cfcm/utilities/fitting/fitrunners.py", line 42, in run_iteration
output_args = self.sess.run(op_list, feed_dict=feed_dict)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 889, in run
run_metadata_ptr)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1120, in _run
feed_dict_tensor, options, run_metadata)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1317, in _do_run
options, run_metadata)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1336, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[16,64,256,256]
[[Node: Conv2d_transpose_3/conv2d_transpose = Conv2DBackpropInput[T=DT_FLOAT, data_format="NHWC", padding="SAME", strides=[1, 2, 2, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](Conv2d_transpose_3/stack, Conv2d_transpose_3/weights/read, rnn_3/BasicConvLSTMCell_1/mul_2)]]
[[Node: Mean/_27 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_18892_Mean", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

Caused by op u'Conv2d_transpose_3/conv2d_transpose', defined at:
File "cfcm/cli.py", line 55, in
cli()
File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 722, in call
return self.main(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "cfcm/cli.py", line 36, in train
fit_runner = LSTMResNetEndovis(jdata, data_folder=data_folder, model_folder=model_folder)
File "/home/lvhao/CFCM-2D-master/cfcm/zoo/fitrunners.py", line 628, in init
LSTMResNet.init(self, jdata=jdata, model_folder=model_folder)
File "/home/lvhao/CFCM-2D-master/cfcm/zoo/fitrunners.py", line 473, in init
super(LSTMResNet, self).init()
File "/home/lvhao/CFCM-2D-master/cfcm/utilities/fitting/fitrunners.py", line 15, in init
self.create_outputs()
File "/home/lvhao/CFCM-2D-master/cfcm/zoo/fitrunners.py", line 501, in create_outputs
self.placeholders['images'], self.placeholders['is_training'], self.resnet_type, self.num_output_channels
File "/home/lvhao/CFCM-2D-master/cfcm/zoo/models.py", line 34, in lstm_resnet
output = model(inputs=input, is_training=is_training)
File "/home/lvhao/CFCM-2D-master/cfcm/zoo/tf_resnet.py", line 210, in model
hidden = conv2d_transpose(output[-1], kernel_size=2, output_channels=base_num_kernels * factor)
File "/home/lvhao/CFCM-2D-master/cfcm/zoo/tf_resnet.py", line 25, in conv2d_transpose
stride=2,
File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/framework/python/ops/arg_scope.py", line 181, in func_with_args
return func(*args, **current_args)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/layers/python/layers/layers.py", line 1250, in convolution2d_transpose
outputs = layer.apply(inputs)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/layers/base.py", line 671, in apply
return self.call(inputs, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/layers/base.py", line 575, in call
outputs = self.call(inputs, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/layers/convolutional.py", line 1289, in call
data_format=utils.convert_data_format(self.data_format, ndim=4))
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/nn_ops.py", line 1211, in conv2d_transpose
name=name)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_nn_ops.py", line 779, in conv2d_backprop_input
use_cudnn_on_gpu=use_cudnn_on_gpu, data_format=data_format, name=name)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2956, in create_op
op_def=op_def)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1470, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[16,64,256,256]
[[Node: Conv2d_transpose_3/conv2d_transpose = Conv2DBackpropInput[T=DT_FLOAT, data_format="NHWC", padding="SAME", strides=[1, 2, 2, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](Conv2d_transpose_3/stack, Conv2d_transpose_3/weights/read, rnn_3/BasicConvLSTMCell_1/mul_2)]]
[[Node: Mean/_27 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_18892_Mean", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

Why ConvLSTM is used in only decoder?

Thank you very much for sharing this wonderful project, and I have a question about the network.

I find that the ConvLSTM layer is used in only decoder, and the encoder used the general convolution. Why the encoder don't use the ConvLSTM? I am looking forward to you suggestion.

can you offer the test demon script?

i have trained the network and get the model.
but i dont know how to visualize the result?
i obtain the csv_results.csv and folder(train and valid).

about the xray dataset

Thank you very much for your code
I have a question, is the xray data set how to implement sequence processing
Is it a case study?
Or does a patient only have one xray image, corresponding to two labels?

can you offer the /Segmentation/Training/Dataset1/Video.avi

how to download the Video.avi？i want to know the details of the video.avi. tks!