nicolov / segmentation_keras

DilatedNet in Keras for image segmentation

Keras implementation of DilatedNet for semantic segmentation

A native Keras implementation of semantic segmentation according to Multi-Scale Context Aggregation by Dilated Convolutions (2016). Optionally uses the pretrained weights by the authors'.

The code has been tested on Tensorflow 1.3, Keras 1.2, and Python 3.6.

Using the pretrained model

Download and extract the pretrained model:

curl -L https://github.com/nicolov/segmentation_keras/releases/download/model/nicolov_segmentation_model.tar.gz | tar xvf -

Install dependencies and run:

pip install -r requirements.txt
# For GPU support
pip install tensorflow-gpu==1.3.0

python predict.py --weights_path conversion/converted/dilation8_pascal_voc.npy

The output image will be under images/cat_seg.png.

Converting the original Caffe model

Follow the instructions in the conversion folder to convert the weights to the TensorFlow format that can be used by Keras.

Training

Download the Augmented Pascal VOC dataset here:

curl -L http://www.eecs.berkeley.edu/Research/Projects/CS/vision/grouping/semantic_contours/benchmark.tgz | tar -xvf -

This will create a benchmark_RELEASE directory in the root of the repo. Use the convert_masks.py script to convert the provided masks in .mat format to RGB pngs:

python convert_masks.py \
    --in-dir benchmark_RELEASE/dataset/cls \
    --out-dir benchmark_RELEASE/dataset/pngs

Start training:

python train.py --batch-size 2

Model checkpoints are saved under trained/, and can be used with the predict.py script for testing.

The training code is currently limited to the frontend module, and thus only outputs 16x16 segmentation maps. The augmentation pipeline does mirroring but not cropping or rotation.

---

Fisher Yu and Vladlen Koltun, Multi-Scale Context Aggregation by Dilated Convolutions, 2016

segmentation_keras's Issues

display segmented mask

How do I display the segmented mask instead of the segmented image?

AtrousConvolution2D vs.Conv2DTranspose

Hi @nicolov
I was wondering whether in your model, you wouldn't need to have a Conv2DTranspose or Upsample layer to compensate for the maxpool and obtain predictions with the same size as your input image?

Modifying number of class

Hi Nicolov,

Thanks for the great work! I tried to train new dataset by generating my own set of jpg and png masks. However I realized it only works for pre-defined 20 classes. For example I wanted to re-train this network to segment screws from background, I wasn't able to find way to add new classes but to use a existed color 0x181818 which was originally trained for cats. After training it did segmented the screw. However I'm still wondering is there any way to change the number of classes and specify which color value are associated with certain class?

No module named '_pywrap_tensorflow_internal'

File "C:\Users\DGRE.conda\envs\env\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 18, in swig_import_helper
return importlib.import_module(mname)
File "C:\Users\DGRE.conda\envs\env\lib\importlib_init_.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 985, in _gcd_import
File "", line 968, in _find_and_load
File "", line 957, in _find_and_load_unlocked
File "", line 666, in _load_unlocked
File "", line 577, in module_from_spec
File "", line 938, in create_module
File "", line 222, in _call_with_frames_removed
ImportError: DLL load failed: The specified module could not be found.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\Users\DGRE.conda\envs\env\lib\site-packages\tensorflow\python\pywrap_tensorflow.py", line 41, in
from tensorflow.python.pywrap_tensorflow_internal import *
File "C:\Users\DGRE.conda\envs\env\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 21, in
_pywrap_tensorflow_internal = swig_import_helper()
File "C:\Users\DGRE.conda\envs\env\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 20, in swig_import_helper
return importlib.import_module('pywrap_tensorflow_internal')
File "C:\Users\DGRE.conda\envs\env\lib\importlib_init.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
ImportError: No module named '_pywrap_tensorflow_internal'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "predict.py", line 19, in
from model import get_frontend, add_softmax, add_context
File "C:\Users\DGRE\segmentation_keras\model.py", line 1, in
from keras.layers import Activation, Reshape, Dropout
File "C:\Users\DGRE.conda\envs\env\lib\site-packages\keras_init_.py", line 2, in
from . import backend
File "C:\Users\DGRE.conda\envs\env\lib\site-packages\keras\backend_init_.py", line 67, in
from .tensorflow_backend import *
File "C:\Users\DGRE.conda\envs\env\lib\site-packages\keras\backend\tensorflow_backend.py", line 1, in
import tensorflow as tf
File "C:\Users\DGRE.conda\envs\env\lib\site-packages\tensorflow_init_.py", line 24, in
from tensorflow.python import *
File "C:\Users\DGRE.conda\envs\env\lib\site-packages\tensorflow\python_init_.py", line 49, in
from tensorflow.python import pywrap_tensorflow
File "C:\Users\DGRE.conda\envs\env\lib\site-packages\tensorflow\python\pywrap_tensorflow.py", line 52, in
raise ImportError(msg)
ImportError: Traceback (most recent call last):
File "C:\Users\DGRE.conda\envs\env\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 18, in swig_import_helper
return importlib.import_module(mname)
File "C:\Users\DGRE.conda\envs\env\lib\importlib_init_.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 985, in _gcd_import
File "", line 968, in _find_and_load
File "", line 957, in _find_and_load_unlocked
File "", line 666, in _load_unlocked
File "", line 577, in module_from_spec
File "", line 938, in create_module
File "", line 222, in _call_with_frames_removed
ImportError: DLL load failed: The specified module could not be found.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\Users\DGRE.conda\envs\env\lib\site-packages\tensorflow\python\pywrap_tensorflow.py", line 41, in
from tensorflow.python.pywrap_tensorflow_internal import *
File "C:\Users\DGRE.conda\envs\env\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 21, in
_pywrap_tensorflow_internal = swig_import_helper()
File "C:\Users\DGRE.conda\envs\env\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 20, in swig_import_helper
return importlib.import_module('pywrap_tensorflow_internal')
File "C:\Users\DGRE.conda\envs\env\lib\importlib_init.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
ImportError: No module named '_pywrap_tensorflow_internal'

Failed to load the native TensorFlow runtime.

Docker command produces error

Traceback (most recent call last):
File "./convert.py", line 60, in
main()
File "./convert.py", line 56, in main
args.phase)
File "./convert.py", line 32, in convert
with open(data_output_path, 'wb') as data_out:
IOError: [Errno 2] No such file or directory: '../converted/dilation8_pascal_voc.npy'

How do I fix this? Thanks so much and thanks for the awesome code!

The pascal_mean in image_reader.py

I am wondering why the mean is got in the pascal dataset?
should it in the vgg is

Cannot locate Dockerfile: Dockerfile

Probably a rookie error but when I am trying to run the conversion step in conversion by running the docker I get the following error:

$sudo docker run -v $(pwd):/workspace -ti `docker build -q .`
time="2017-02-09T09:15:11-08:00" level=fatal msg="Cannot locate Dockerfile: Dockerfile" 
docker: "run" requires a minimum of 1 argument. See 'docker run --help'.

weight conversion error

After downloading the caffe model and prototxt I tried running the weight conversion code as follows and got an error:

± sudo docker run -v $(pwd):/workspace -ti `docker build -q .`
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.27/build?buildargs=%7B%7D&cachefrom=%5B%5D&cgroupparent=&cpuperiod=0&cpuquota=0&cpusetcpus=&cpusetmems=&cpushares=0&dockerfile=Dockerfile&labels=%7B%7D&memory=0&memswap=0&networkmode=default&q=1&rm=1&shmsize=0&ulimits=null: dial unix /var/run/docker.sock: connect: permission denied
"docker run" requires at least 1 argument(s).
See 'docker run --help'.

Usage:  docker run [OPTIONS] IMAGE [COMMAND] [ARG...]

Run a command in a new container

I'm on ubuntu 16.04 and sudo docker run hello-world works.

wrong flip

Here, x is a single image array, not a batch data. During flip, the row_index should be 0, and the col_index should be 1.

def transform(x: np.array, params: TransformParams) -> np.array:
    if params.do_hor_flip:
        x = flip_axis(x, 2)
    if params.do_vert_flip:
        x = flip_axis(x, 1)

https://github.com/fchollet/keras/blob/master/keras/preprocessing/image.py#L538
> # x is a single image, so it doesn't have image number at index 0

MIT License?

Could you consider applying the MIT license to this repository, the same as keras, so people may make use of it?

about the MIT license and here is the text:

The MIT License (MIT)

Copyright (c) <year> <copyright holders>

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

i have downloaded the dataset,but i don't know how to use it

Thanks for sharing your codes.
i downloaded the dataset and the file has no suffi. I don't find any README in the package.
could you help me to solve this proplem? Thanks a lot.

convert to mask looks bad

when i run the convert_masks.py i get the img like this

is it right?

palette conversion not needed

https://github.com/nicolov/segmentation_keras/blob/master/convert_masks.py isn't necessary.

Just use Pillow and you can load the classes separately from the color palette, which means it will already be in the format you want!

from https://github.com/aurora95/Keras-FCN/blob/master/utils/SegDataGenerator.py#L203

                from PIL import Image
                label = Image.open(label_filepath)
                if self.save_to_dir and self.palette is None:
                    self.palette = label.palette

cool right?

When the mask image is converted to array, is there a value of <21 in it?

My mask image is converted to array, which values are 255, then when I was training, I program a long time without any output, there is no error, the only difference between me with you with the sample pictures of the mask is converted to array mask, my future value are 255

What is the reason behind the removal of padding?

In this model the original VGG16 padding operations are removed. The paper also calls for this removal but I don't understand why. The authors note, "We also remove the padding of the intermediate feature maps. Intermediate padding was used in the original classification network, but is neither necessary nor justified in dense prediction."

Can you help me understand why the padding is not necessary? Given that the receptive fields of atrous convolutions are so large, wouldn't this be counterproductive?

Why sparse_categorical_crossentropy is used?

Why sparse_categorical_crossentropy is used?

Looks like here is softmax is used:
https://github.com/nicolov/segmentation_keras/blob/master/model.py#L60

Also I don't understand why our mask batch is:
mask_batch = np.zeros((batch_size, mask_target_size[0] * mask_target_size[1], 1)
and not
mask_batch = np.zeros((batch_size, mask_target_size[0] * mask_target_size[1], 21)
?
https://github.com/nicolov/segmentation_keras/blob/master/utils/image_reader.py#L194

How do you determine the width and height of the input, and the size of the label_margin?

@nicolov
Hello, I'd like to know if input_width, input_height, label_margin values are related to width, height of the picture? Do you have formulas or something like that?
Because I want to predict a new picture now, each picture of different sizes, some are large, some relatively small lot, which requires a different input_width, input_height, label_margin in size, that will not throw the error in "AssertionError". I try to use Image.resize method to predict the images, making them the size difference is not big, but I found that after the resize images and the size of the original picture of the forecast results are not the same, you can help me to answer this question? Thank you

how to add one more category in dataset

How to handle high resolution images

Hello @nicolov ,

let me first express my appreciation to your work in image segmentation its great (Y)

small suggestion , i just want to notify you that there is a missing -- in input parsing . very minor change

parser.add_argument('--input_path', nargs='?', default='images/cat.jpg',
                        help='Required path to input image') 

I'm hoping you can help me in understanding how to handle high res images as 1028 and 4k ,

also in the code i found you set input_width, input_height = 900, 900 and label_margin = 186
can you please illustrate what is the reason for this static number and how they effect on the output high and width

output_height = input_height - 2 * label_margin
output_width = input_width - 2 * label_margin

Submit to keras-contrib + see Keras-FCN

Might you be interested in a pull request of this code to the official keras-contrib repository?

These keras-contrib issues are also relevant to this repository:

• keras-team/keras-contrib#47 incorporating coco + voc 2012
• keras-team/keras-contrib#59 dealing with gaps in FCN training performance
• keras-team/keras-contrib#63 training on other FCN networks

Keras-FCN, which I was planning to adapt for a merge into keras-contrib also has a SegDataGenerator implementation with some of the features you are looking for in your comments, plus additional models and experimental support for coco. My Keras-FCN branch with a README.md may also be informative, there is a pull request in progress to merge upstream.

Link for "instructions to convert the weights" is broken

Can you please provide the instructions for converting pascal16 weights to TF?

Training freezes

On executing command: python train.py --batch-size 2 ,training freezes at last step of first epoch.

All the libraries are according to the requirement.txt file

Fine tuning ...

Hello,

You have provided the pre-trained model of VOC. I have a small dataset with 2 classes, which I annotated based on VOC and I want to fine-tune it. Would you please guide me through the process?

Black image after segmentation

Hi! I have val accuracy = 1, but when i am trying to predict mask on the image from train set it displays me black image. Does anybody know what is the reason of this behaviour?

About the pretrain model

Can you tell me the pretrained model if use the function
def add_context(model: Sequential)

Keras JS

Great work. Thank you very much!
Can I use this with Keras JS? Is there any way to easily rewrite from python to JS?

masaki

regarding the function definition used in model.py

Hi nicolov,

Thanks for sharing the code. By the way, I see you define the model function as follows:

I am not very clear about the usage of -> Sequential . What does it used for, when do we need to add this when defining a function. Also it seems to me you pass the input parameter as model: Sequential. Is that because we only need to ensure model is of sequential. I think I am not very clear about the parameter passing mechanism here.

def add_softmax(model: Sequential) -> Sequential:
    """ Append the softmax layers to the frontend or frontend + context net. """
    # The softmax layer doesn't work on the (width, height, channel)
    # shape, so we reshape to (width*height, channel) first.
    # https://github.com/fchollet/keras/issues/1169
    _, curr_width, curr_height, curr_channels = model.layers[-1].output_shape

    model.add(Reshape((curr_width * curr_height, curr_channels)))
    model.add(Activation('softmax'))
    # Technically, we need another Reshape here to reshape to 2d, but TF
    # the complains when batch_size > 1. We're just going to reshape in numpy.
    # model.add(Reshape((curr_width, curr_height, curr_channels)))

    return model

Please confirm download link

Could you please point to the tar file for the dataset?
Is it http://www.eecs.berkeley.edu/Research/Projects/CS/vision/grouping/semantic_contours/benchmark.tgz

Thank you.

Weights Conversion Error: wget: unable to resolve host address 'vobj.cs.princeton.edu'

I ran into a download error while converting the model weights. Please confirm the download link.

Thanks.

(dl4cvpy3) ubuntu@ip-172-31-0-76:~/cviz/segmentation_keras/conversion$ wget http://vobj.cs.princeton.edu/models/dilation8_pascal_voc.caffemodel
--2018-12-30 18:45:11-- http://vobj.cs.princeton.edu/models/dilation8_pascal_voc.caffemodel
Resolving vobj.cs.princeton.edu (vobj.cs.princeton.edu)... failed: Name or service not known.
wget: unable to resolve host address 'vobj.cs.princeton.edu'

How to deal with unbalanced classes?

Looks like your model is working with Pascal Voc dataset (20 classes + 1 background class), this dataset is unbalanced, if I remember correct, 'person' class is have dominant pixel area.

So my question is model use some additional tricks to deal with class unbalance?

training and validation loss nan

First of all I just want to thank you for the great work.
I am having an issue during training, my loss and val_loss is nan, however I am still getting values for accuracy and val_acc. I am training on the PASCAL_VOC 2012 dataset with the segmentation class pngs.

keras 1.2.1 & 2.0.6
tensorflow-gpu 1.2.1
python 3.6.1

about the required pre-trained vgg model

Hi, @nicolov ,

According to this line, vgg_conv.npy is needed as pre-trained vgg model in training. Could you list the download location for corresponding caffemodel and prototxt file? And, is the conversion step the same as here?

Thanks!

The role of label_margin

Hi,

I am trying to fine-tune the segmentation model.
I'd like to modify utils/image_reader.py to make it fit to the network with input 900x900 since it's 500x500 now in image_reader.py.
I am confused with the role of label_margin. How do I determine the value of label_margin?
Thanks.

Hao-Wei

AttributeError: module 'keras.utils.generic_utils' has no attribute 'populate_dict_with_module_objects'

Failed to parse NetParameter file: ../dilation8_pascal_voc.caffemodel

Hi, @nicolov ,

I got the following conversion (Failed to parse NetParameter file: ../dilation8_pascal_voc.caffemodel) error, any suggestion to fix it? (Thanks)

I0613 03:05:11.228221     7 layer_factory.hpp:77] Creating layer input
I0613 03:05:11.228240     7 net.cpp:84] Creating Layer input
I0613 03:05:11.228255     7 net.cpp:380] input -> data
I0613 03:05:11.239449     7 net.cpp:122] Setting up input
I0613 03:05:11.239488     7 net.cpp:129] Top shape: 1 3 900 900 (2430000)
I0613 03:05:11.239500     7 net.cpp:137] Memory required for data: 9720000
I0613 03:05:11.239513     7 layer_factory.hpp:77] Creating layer conv1_1
I0613 03:05:11.239536     7 net.cpp:84] Creating Layer conv1_1
I0613 03:05:11.239549     7 net.cpp:406] conv1_1 <- data
I0613 03:05:11.239574     7 net.cpp:380] conv1_1 -> conv1_1
I0613 03:05:11.240721     7 net.cpp:122] Setting up conv1_1
I0613 03:05:11.240762     7 net.cpp:129] Top shape: 1 64 898 898 (51609856)
I0613 03:05:11.240774     7 net.cpp:137] Memory required for data: 216159424
I0613 03:05:11.240805     7 layer_factory.hpp:77] Creating layer relu1_1
I0613 03:05:11.240833     7 net.cpp:84] Creating Layer relu1_1
I0613 03:05:11.240844     7 net.cpp:406] relu1_1 <- conv1_1
I0613 03:05:11.240854     7 net.cpp:367] relu1_1 -> conv1_1 (in-place)
I0613 03:05:11.240883     7 net.cpp:122] Setting up relu1_1
I0613 03:05:11.240895     7 net.cpp:129] Top shape: 1 64 898 898 (51609856)
I0613 03:05:11.240909     7 net.cpp:137] Memory required for data: 422598848
I0613 03:05:11.240923     7 layer_factory.hpp:77] Creating layer conv1_2
I0613 03:05:11.240943     7 net.cpp:84] Creating Layer conv1_2
I0613 03:05:11.240953     7 net.cpp:406] conv1_2 <- conv1_1
I0613 03:05:11.240964     7 net.cpp:380] conv1_2 -> conv1_2
I0613 03:05:11.241690     7 net.cpp:122] Setting up conv1_2
I0613 03:05:11.241717     7 net.cpp:129] Top shape: 1 64 896 896 (51380224)
I0613 03:05:11.241727     7 net.cpp:137] Memory required for data: 628119744
I0613 03:05:11.241744     7 layer_factory.hpp:77] Creating layer relu1_2
I0613 03:05:11.241770     7 net.cpp:84] Creating Layer relu1_2
I0613 03:05:11.241778     7 net.cpp:406] relu1_2 <- conv1_2
I0613 03:05:11.241798     7 net.cpp:367] relu1_2 -> conv1_2 (in-place)
I0613 03:05:11.241812     7 net.cpp:122] Setting up relu1_2
I0613 03:05:11.241822     7 net.cpp:129] Top shape: 1 64 896 896 (51380224)
I0613 03:05:11.241830     7 net.cpp:137] Memory required for data: 833640640
I0613 03:05:11.241839     7 layer_factory.hpp:77] Creating layer pool1
I0613 03:05:11.241850     7 net.cpp:84] Creating Layer pool1
I0613 03:05:11.241859     7 net.cpp:406] pool1 <- conv1_2
I0613 03:05:11.241871     7 net.cpp:380] pool1 -> pool1
I0613 03:05:11.241888     7 net.cpp:122] Setting up pool1
I0613 03:05:11.241899     7 net.cpp:129] Top shape: 1 64 448 448 (12845056)
I0613 03:05:11.241907     7 net.cpp:137] Memory required for data: 885020864
I0613 03:05:11.241915     7 layer_factory.hpp:77] Creating layer conv2_1
I0613 03:05:11.241928     7 net.cpp:84] Creating Layer conv2_1
I0613 03:05:11.241937     7 net.cpp:406] conv2_1 <- pool1
I0613 03:05:11.241947     7 net.cpp:380] conv2_1 -> conv2_1
I0613 03:05:11.242280     7 net.cpp:122] Setting up conv2_1
I0613 03:05:11.242295     7 net.cpp:129] Top shape: 1 128 446 446 (25461248)
I0613 03:05:11.242305     7 net.cpp:137] Memory required for data: 986865856
I0613 03:05:11.242318     7 layer_factory.hpp:77] Creating layer relu2_1
I0613 03:05:11.242331     7 net.cpp:84] Creating Layer relu2_1
I0613 03:05:11.242342     7 net.cpp:406] relu2_1 <- conv2_1
I0613 03:05:11.242352     7 net.cpp:367] relu2_1 -> conv2_1 (in-place)
I0613 03:05:11.242363     7 net.cpp:122] Setting up relu2_1
I0613 03:05:11.242373     7 net.cpp:129] Top shape: 1 128 446 446 (25461248)
I0613 03:05:11.242382     7 net.cpp:137] Memory required for data: 1088710848
I0613 03:05:11.242390     7 layer_factory.hpp:77] Creating layer conv2_2
I0613 03:05:11.242403     7 net.cpp:84] Creating Layer conv2_2
I0613 03:05:11.242413     7 net.cpp:406] conv2_2 <- conv2_1
I0613 03:05:11.242424     7 net.cpp:380] conv2_2 -> conv2_2
I0613 03:05:11.242842     7 net.cpp:122] Setting up conv2_2
I0613 03:05:11.242859     7 net.cpp:129] Top shape: 1 128 444 444 (25233408)
I0613 03:05:11.242868     7 net.cpp:137] Memory required for data: 1189644480
I0613 03:05:11.242880     7 layer_factory.hpp:77] Creating layer relu2_2
I0613 03:05:11.242893     7 net.cpp:84] Creating Layer relu2_2
I0613 03:05:11.242913     7 net.cpp:406] relu2_2 <- conv2_2
I0613 03:05:11.242923     7 net.cpp:367] relu2_2 -> conv2_2 (in-place)
I0613 03:05:11.242934     7 net.cpp:122] Setting up relu2_2
I0613 03:05:11.242944     7 net.cpp:129] Top shape: 1 128 444 444 (25233408)
I0613 03:05:11.242954     7 net.cpp:137] Memory required for data: 1290578112
I0613 03:05:11.242960     7 layer_factory.hpp:77] Creating layer pool2
I0613 03:05:11.242971     7 net.cpp:84] Creating Layer pool2
I0613 03:05:11.242980     7 net.cpp:406] pool2 <- conv2_2
I0613 03:05:11.242991     7 net.cpp:380] pool2 -> pool2
I0613 03:05:11.243005     7 net.cpp:122] Setting up pool2
I0613 03:05:11.243016     7 net.cpp:129] Top shape: 1 128 222 222 (6308352)
I0613 03:05:11.243024     7 net.cpp:137] Memory required for data: 1315811520
I0613 03:05:11.243033     7 layer_factory.hpp:77] Creating layer conv3_1
I0613 03:05:11.243058     7 net.cpp:84] Creating Layer conv3_1
I0613 03:05:11.243068     7 net.cpp:406] conv3_1 <- pool2
I0613 03:05:11.243088     7 net.cpp:380] conv3_1 -> conv3_1
I0613 03:05:11.243517     7 net.cpp:122] Setting up conv3_1
I0613 03:05:11.243533     7 net.cpp:129] Top shape: 1 256 220 220 (12390400)
I0613 03:05:11.243541     7 net.cpp:137] Memory required for data: 1365373120
I0613 03:05:11.243556     7 layer_factory.hpp:77] Creating layer relu3_1
I0613 03:05:11.243588     7 net.cpp:84] Creating Layer relu3_1
I0613 03:05:11.243599     7 net.cpp:406] relu3_1 <- conv3_1
I0613 03:05:11.243610     7 net.cpp:367] relu3_1 -> conv3_1 (in-place)
I0613 03:05:11.243623     7 net.cpp:122] Setting up relu3_1
I0613 03:05:11.243633     7 net.cpp:129] Top shape: 1 256 220 220 (12390400)
I0613 03:05:11.243641     7 net.cpp:137] Memory required for data: 1414934720
I0613 03:05:11.243649     7 layer_factory.hpp:77] Creating layer conv3_2
I0613 03:05:11.243662     7 net.cpp:84] Creating Layer conv3_2
I0613 03:05:11.243671     7 net.cpp:406] conv3_2 <- conv3_1
I0613 03:05:11.243683     7 net.cpp:380] conv3_2 -> conv3_2
I0613 03:05:11.244474     7 net.cpp:122] Setting up conv3_2
I0613 03:05:11.244493     7 net.cpp:129] Top shape: 1 256 218 218 (12166144)
I0613 03:05:11.244503     7 net.cpp:137] Memory required for data: 1463599296
I0613 03:05:11.244514     7 layer_factory.hpp:77] Creating layer relu3_2
I0613 03:05:11.244526     7 net.cpp:84] Creating Layer relu3_2
I0613 03:05:11.244535     7 net.cpp:406] relu3_2 <- conv3_2
I0613 03:05:11.244549     7 net.cpp:367] relu3_2 -> conv3_2 (in-place)
I0613 03:05:11.244561     7 net.cpp:122] Setting up relu3_2
I0613 03:05:11.244571     7 net.cpp:129] Top shape: 1 256 218 218 (12166144)
I0613 03:05:11.244580     7 net.cpp:137] Memory required for data: 1512263872
I0613 03:05:11.244599     7 layer_factory.hpp:77] Creating layer conv3_3
I0613 03:05:11.244612     7 net.cpp:84] Creating Layer conv3_3
I0613 03:05:11.244621     7 net.cpp:406] conv3_3 <- conv3_2
I0613 03:05:11.244632     7 net.cpp:380] conv3_3 -> conv3_3
I0613 03:05:11.245380     7 net.cpp:122] Setting up conv3_3
I0613 03:05:11.245398     7 net.cpp:129] Top shape: 1 256 216 216 (11943936)
I0613 03:05:11.245405     7 net.cpp:137] Memory required for data: 1560039616
I0613 03:05:11.245417     7 layer_factory.hpp:77] Creating layer relu3_3
I0613 03:05:11.245434     7 net.cpp:84] Creating Layer relu3_3
I0613 03:05:11.245442     7 net.cpp:406] relu3_3 <- conv3_3
I0613 03:05:11.245463     7 net.cpp:367] relu3_3 -> conv3_3 (in-place)
I0613 03:05:11.245474     7 net.cpp:122] Setting up relu3_3
I0613 03:05:11.245484     7 net.cpp:129] Top shape: 1 256 216 216 (11943936)
I0613 03:05:11.245492     7 net.cpp:137] Memory required for data: 1607815360
I0613 03:05:11.245501     7 layer_factory.hpp:77] Creating layer pool3
I0613 03:05:11.245512     7 net.cpp:84] Creating Layer pool3
I0613 03:05:11.245522     7 net.cpp:406] pool3 <- conv3_3
I0613 03:05:11.245532     7 net.cpp:380] pool3 -> pool3
I0613 03:05:11.245545     7 net.cpp:122] Setting up pool3
I0613 03:05:11.245558     7 net.cpp:129] Top shape: 1 256 108 108 (2985984)
I0613 03:05:11.245568     7 net.cpp:137] Memory required for data: 1619759296
I0613 03:05:11.245576     7 layer_factory.hpp:77] Creating layer conv4_1
I0613 03:05:11.245587     7 net.cpp:84] Creating Layer conv4_1
I0613 03:05:11.245596     7 net.cpp:406] conv4_1 <- pool3
I0613 03:05:11.245609     7 net.cpp:380] conv4_1 -> conv4_1
I0613 03:05:11.246632     7 net.cpp:122] Setting up conv4_1
I0613 03:05:11.246650     7 net.cpp:129] Top shape: 1 512 106 106 (5752832)
I0613 03:05:11.246659     7 net.cpp:137] Memory required for data: 1642770624
I0613 03:05:11.246670     7 layer_factory.hpp:77] Creating layer relu4_1
I0613 03:05:11.246685     7 net.cpp:84] Creating Layer relu4_1
I0613 03:05:11.246695     7 net.cpp:406] relu4_1 <- conv4_1
I0613 03:05:11.246704     7 net.cpp:367] relu4_1 -> conv4_1 (in-place)
I0613 03:05:11.246716     7 net.cpp:122] Setting up relu4_1
I0613 03:05:11.246726     7 net.cpp:129] Top shape: 1 512 106 106 (5752832)
I0613 03:05:11.246734     7 net.cpp:137] Memory required for data: 1665781952
I0613 03:05:11.246742     7 layer_factory.hpp:77] Creating layer conv4_2
I0613 03:05:11.246755     7 net.cpp:84] Creating Layer conv4_2
I0613 03:05:11.246764     7 net.cpp:406] conv4_2 <- conv4_1
I0613 03:05:11.246775     7 net.cpp:380] conv4_2 -> conv4_2
I0613 03:05:11.248570     7 net.cpp:122] Setting up conv4_2
I0613 03:05:11.248602     7 net.cpp:129] Top shape: 1 512 104 104 (5537792)
I0613 03:05:11.248610     7 net.cpp:137] Memory required for data: 1687933120
I0613 03:05:11.248633     7 layer_factory.hpp:77] Creating layer relu4_2
I0613 03:05:11.248649     7 net.cpp:84] Creating Layer relu4_2
I0613 03:05:11.248659     7 net.cpp:406] relu4_2 <- conv4_2
I0613 03:05:11.248672     7 net.cpp:367] relu4_2 -> conv4_2 (in-place)
I0613 03:05:11.248687     7 net.cpp:122] Setting up relu4_2
I0613 03:05:11.248697     7 net.cpp:129] Top shape: 1 512 104 104 (5537792)
I0613 03:05:11.248704     7 net.cpp:137] Memory required for data: 1710084288
I0613 03:05:11.248713     7 layer_factory.hpp:77] Creating layer conv4_3
I0613 03:05:11.248728     7 net.cpp:84] Creating Layer conv4_3
I0613 03:05:11.248739     7 net.cpp:406] conv4_3 <- conv4_2
I0613 03:05:11.248749     7 net.cpp:380] conv4_3 -> conv4_3
I0613 03:05:11.251756     7 net.cpp:122] Setting up conv4_3
I0613 03:05:11.251847     7 net.cpp:129] Top shape: 1 512 102 102 (5326848)
I0613 03:05:11.251868     7 net.cpp:137] Memory required for data: 1731391680
I0613 03:05:11.251891     7 layer_factory.hpp:77] Creating layer relu4_3
I0613 03:05:11.251914     7 net.cpp:84] Creating Layer relu4_3
I0613 03:05:11.251929     7 net.cpp:406] relu4_3 <- conv4_3
I0613 03:05:11.251948     7 net.cpp:367] relu4_3 -> conv4_3 (in-place)
I0613 03:05:11.251981     7 net.cpp:122] Setting up relu4_3
I0613 03:05:11.251991     7 net.cpp:129] Top shape: 1 512 102 102 (5326848)
I0613 03:05:11.251998     7 net.cpp:137] Memory required for data: 1752699072
I0613 03:05:11.252007     7 layer_factory.hpp:77] Creating layer conv5_1
I0613 03:05:11.252032     7 net.cpp:84] Creating Layer conv5_1
I0613 03:05:11.252041     7 net.cpp:406] conv5_1 <- conv4_3
I0613 03:05:11.252053     7 net.cpp:380] conv5_1 -> conv5_1
I0613 03:05:11.255393     7 net.cpp:122] Setting up conv5_1
I0613 03:05:11.255506     7 net.cpp:129] Top shape: 1 512 98 98 (4917248)
I0613 03:05:11.255515     7 net.cpp:137] Memory required for data: 1772368064
I0613 03:05:11.255539     7 layer_factory.hpp:77] Creating layer relu5_1
I0613 03:05:11.255578     7 net.cpp:84] Creating Layer relu5_1
I0613 03:05:11.255591     7 net.cpp:406] relu5_1 <- conv5_1
I0613 03:05:11.255615     7 net.cpp:367] relu5_1 -> conv5_1 (in-place)
I0613 03:05:11.255640     7 net.cpp:122] Setting up relu5_1
I0613 03:05:11.255651     7 net.cpp:129] Top shape: 1 512 98 98 (4917248)
I0613 03:05:11.255659     7 net.cpp:137] Memory required for data: 1792037056
I0613 03:05:11.255668     7 layer_factory.hpp:77] Creating layer conv5_2
I0613 03:05:11.255692     7 net.cpp:84] Creating Layer conv5_2
I0613 03:05:11.255702     7 net.cpp:406] conv5_2 <- conv5_1
I0613 03:05:11.255713     7 net.cpp:380] conv5_2 -> conv5_2
I0613 03:05:11.258558     7 net.cpp:122] Setting up conv5_2
I0613 03:05:11.258667     7 net.cpp:129] Top shape: 1 512 94 94 (4524032)
I0613 03:05:11.258677     7 net.cpp:137] Memory required for data: 1810133184
I0613 03:05:11.258700     7 layer_factory.hpp:77] Creating layer relu5_2
I0613 03:05:11.258738     7 net.cpp:84] Creating Layer relu5_2
I0613 03:05:11.258754     7 net.cpp:406] relu5_2 <- conv5_2
I0613 03:05:11.258772     7 net.cpp:367] relu5_2 -> conv5_2 (in-place)
I0613 03:05:11.258796     7 net.cpp:122] Setting up relu5_2
I0613 03:05:11.258807     7 net.cpp:129] Top shape: 1 512 94 94 (4524032)
I0613 03:05:11.258815     7 net.cpp:137] Memory required for data: 1828229312
I0613 03:05:11.258823     7 layer_factory.hpp:77] Creating layer conv5_3
I0613 03:05:11.258851     7 net.cpp:84] Creating Layer conv5_3
I0613 03:05:11.258862     7 net.cpp:406] conv5_3 <- conv5_2
I0613 03:05:11.258872     7 net.cpp:380] conv5_3 -> conv5_3
I0613 03:05:11.261682     7 net.cpp:122] Setting up conv5_3
I0613 03:05:11.261719     7 net.cpp:129] Top shape: 1 512 90 90 (4147200)
I0613 03:05:11.261728     7 net.cpp:137] Memory required for data: 1844818112
I0613 03:05:11.261740     7 layer_factory.hpp:77] Creating layer relu5_3
I0613 03:05:11.261754     7 net.cpp:84] Creating Layer relu5_3
I0613 03:05:11.261765     7 net.cpp:406] relu5_3 <- conv5_3
I0613 03:05:11.261778     7 net.cpp:367] relu5_3 -> conv5_3 (in-place)
I0613 03:05:11.261793     7 net.cpp:122] Setting up relu5_3
I0613 03:05:11.261801     7 net.cpp:129] Top shape: 1 512 90 90 (4147200)
I0613 03:05:11.261808     7 net.cpp:137] Memory required for data: 1861406912
I0613 03:05:11.261816     7 layer_factory.hpp:77] Creating layer fc6
I0613 03:05:11.261827     7 net.cpp:84] Creating Layer fc6
I0613 03:05:11.261837     7 net.cpp:406] fc6 <- conv5_3
I0613 03:05:11.261847     7 net.cpp:380] fc6 -> fc6
I0613 03:05:11.368047     7 net.cpp:122] Setting up fc6
I0613 03:05:11.368094     7 net.cpp:129] Top shape: 1 4096 66 66 (17842176)
I0613 03:05:11.368101     7 net.cpp:137] Memory required for data: 1932775616
I0613 03:05:11.368114     7 layer_factory.hpp:77] Creating layer relu6
I0613 03:05:11.368129     7 net.cpp:84] Creating Layer relu6
I0613 03:05:11.368139     7 net.cpp:406] relu6 <- fc6
I0613 03:05:11.368154     7 net.cpp:367] relu6 -> fc6 (in-place)
I0613 03:05:11.368166     7 net.cpp:122] Setting up relu6
I0613 03:05:11.368175     7 net.cpp:129] Top shape: 1 4096 66 66 (17842176)
I0613 03:05:11.368180     7 net.cpp:137] Memory required for data: 2004144320
I0613 03:05:11.368186     7 layer_factory.hpp:77] Creating layer drop6
I0613 03:05:11.368202     7 net.cpp:84] Creating Layer drop6
I0613 03:05:11.368227     7 net.cpp:406] drop6 <- fc6
I0613 03:05:11.368235     7 net.cpp:367] drop6 -> fc6 (in-place)
I0613 03:05:11.368244     7 net.cpp:122] Setting up drop6
I0613 03:05:11.368252     7 net.cpp:129] Top shape: 1 4096 66 66 (17842176)
I0613 03:05:11.368257     7 net.cpp:137] Memory required for data: 2075513024
I0613 03:05:11.368263     7 layer_factory.hpp:77] Creating layer fc7
I0613 03:05:11.368275     7 net.cpp:84] Creating Layer fc7
I0613 03:05:11.368283     7 net.cpp:406] fc7 <- fc6
I0613 03:05:11.368290     7 net.cpp:380] fc7 -> fc7
I0613 03:05:11.392370     7 net.cpp:122] Setting up fc7
I0613 03:05:11.392410     7 net.cpp:129] Top shape: 1 4096 66 66 (17842176)
I0613 03:05:11.392416     7 net.cpp:137] Memory required for data: 2146881728
I0613 03:05:11.392427     7 layer_factory.hpp:77] Creating layer relu7
I0613 03:05:11.392441     7 net.cpp:84] Creating Layer relu7
I0613 03:05:11.392457     7 net.cpp:406] relu7 <- fc7
I0613 03:05:11.392468     7 net.cpp:367] relu7 -> fc7 (in-place)
I0613 03:05:11.392482     7 net.cpp:122] Setting up relu7
I0613 03:05:11.392488     7 net.cpp:129] Top shape: 1 4096 66 66 (17842176)
I0613 03:05:11.392511     7 net.cpp:137] Memory required for data: 2218250432
I0613 03:05:11.392518     7 layer_factory.hpp:77] Creating layer drop7
I0613 03:05:11.392526     7 net.cpp:84] Creating Layer drop7
I0613 03:05:11.392534     7 net.cpp:406] drop7 <- fc7
I0613 03:05:11.392545     7 net.cpp:367] drop7 -> fc7 (in-place)
I0613 03:05:11.392557     7 net.cpp:122] Setting up drop7
I0613 03:05:11.392565     7 net.cpp:129] Top shape: 1 4096 66 66 (17842176)
I0613 03:05:11.392570     7 net.cpp:137] Memory required for data: 2289619136
I0613 03:05:11.392575     7 layer_factory.hpp:77] Creating layer fc-final
I0613 03:05:11.392587     7 net.cpp:84] Creating Layer fc-final
I0613 03:05:11.392593     7 net.cpp:406] fc-final <- fc7
I0613 03:05:11.392621     7 net.cpp:380] fc-final -> fc-final
I0613 03:05:11.392732     7 net.cpp:122] Setting up fc-final
I0613 03:05:11.392742     7 net.cpp:129] Top shape: 1 21 66 66 (91476)
I0613 03:05:11.392747     7 net.cpp:137] Memory required for data: 2289985040
I0613 03:05:11.392756     7 layer_factory.hpp:77] Creating layer ct_conv1_1
I0613 03:05:11.392766     7 net.cpp:84] Creating Layer ct_conv1_1
I0613 03:05:11.392772     7 net.cpp:406] ct_conv1_1 <- fc-final
I0613 03:05:11.392781     7 net.cpp:380] ct_conv1_1 -> ct_conv1_1
I0613 03:05:11.392822     7 net.cpp:122] Setting up ct_conv1_1
I0613 03:05:11.392832     7 net.cpp:129] Top shape: 1 42 130 130 (709800)
I0613 03:05:11.392838     7 net.cpp:137] Memory required for data: 2292824240
I0613 03:05:11.392853     7 layer_factory.hpp:77] Creating layer ct_relu1_1
I0613 03:05:11.392861     7 net.cpp:84] Creating Layer ct_relu1_1
I0613 03:05:11.392868     7 net.cpp:406] ct_relu1_1 <- ct_conv1_1
I0613 03:05:11.392874     7 net.cpp:367] ct_relu1_1 -> ct_conv1_1 (in-place)
I0613 03:05:11.392882     7 net.cpp:122] Setting up ct_relu1_1
I0613 03:05:11.392890     7 net.cpp:129] Top shape: 1 42 130 130 (709800)
I0613 03:05:11.392895     7 net.cpp:137] Memory required for data: 2295663440
I0613 03:05:11.392901     7 layer_factory.hpp:77] Creating layer ct_conv1_2
I0613 03:05:11.392911     7 net.cpp:84] Creating Layer ct_conv1_2
I0613 03:05:11.392918     7 net.cpp:406] ct_conv1_2 <- ct_conv1_1
I0613 03:05:11.392925     7 net.cpp:380] ct_conv1_2 -> ct_conv1_2
I0613 03:05:11.392971     7 net.cpp:122] Setting up ct_conv1_2
I0613 03:05:11.392982     7 net.cpp:129] Top shape: 1 42 128 128 (688128)
I0613 03:05:11.392987     7 net.cpp:137] Memory required for data: 2298415952
I0613 03:05:11.392995     7 layer_factory.hpp:77] Creating layer ct_relu1_2
I0613 03:05:11.393003     7 net.cpp:84] Creating Layer ct_relu1_2
I0613 03:05:11.393009     7 net.cpp:406] ct_relu1_2 <- ct_conv1_2
I0613 03:05:11.393018     7 net.cpp:367] ct_relu1_2 -> ct_conv1_2 (in-place)
I0613 03:05:11.393026     7 net.cpp:122] Setting up ct_relu1_2
I0613 03:05:11.393033     7 net.cpp:129] Top shape: 1 42 128 128 (688128)
I0613 03:05:11.393039     7 net.cpp:137] Memory required for data: 2301168464
I0613 03:05:11.393045     7 layer_factory.hpp:77] Creating layer ct_conv2_1
I0613 03:05:11.393054     7 net.cpp:84] Creating Layer ct_conv2_1
I0613 03:05:11.393060     7 net.cpp:406] ct_conv2_1 <- ct_conv1_2
I0613 03:05:11.393069     7 net.cpp:380] ct_conv2_1 -> ct_conv2_1
I0613 03:05:11.393127     7 net.cpp:122] Setting up ct_conv2_1
I0613 03:05:11.393138     7 net.cpp:129] Top shape: 1 84 124 124 (1291584)
I0613 03:05:11.393144     7 net.cpp:137] Memory required for data: 2306334800
I0613 03:05:11.393152     7 layer_factory.hpp:77] Creating layer ct_relu2_1
I0613 03:05:11.393160     7 net.cpp:84] Creating Layer ct_relu2_1
I0613 03:05:11.393167     7 net.cpp:406] ct_relu2_1 <- ct_conv2_1
I0613 03:05:11.393173     7 net.cpp:367] ct_relu2_1 -> ct_conv2_1 (in-place)
I0613 03:05:11.393182     7 net.cpp:122] Setting up ct_relu2_1
I0613 03:05:11.393188     7 net.cpp:129] Top shape: 1 84 124 124 (1291584)
I0613 03:05:11.393193     7 net.cpp:137] Memory required for data: 2311501136
I0613 03:05:11.393199     7 layer_factory.hpp:77] Creating layer ct_conv3_1
I0613 03:05:11.393209     7 net.cpp:84] Creating Layer ct_conv3_1
I0613 03:05:11.393216     7 net.cpp:406] ct_conv3_1 <- ct_conv2_1
I0613 03:05:11.393225     7 net.cpp:380] ct_conv3_1 -> ct_conv3_1
I0613 03:05:11.393357     7 net.cpp:122] Setting up ct_conv3_1
I0613 03:05:11.393368     7 net.cpp:129] Top shape: 1 168 116 116 (2260608)
I0613 03:05:11.393373     7 net.cpp:137] Memory required for data: 2320543568
I0613 03:05:11.393381     7 layer_factory.hpp:77] Creating layer ct_relu3_1
I0613 03:05:11.393388     7 net.cpp:84] Creating Layer ct_relu3_1
I0613 03:05:11.393395     7 net.cpp:406] ct_relu3_1 <- ct_conv3_1
I0613 03:05:11.393402     7 net.cpp:367] ct_relu3_1 -> ct_conv3_1 (in-place)
I0613 03:05:11.393410     7 net.cpp:122] Setting up ct_relu3_1
I0613 03:05:11.393417     7 net.cpp:129] Top shape: 1 168 116 116 (2260608)
I0613 03:05:11.393422     7 net.cpp:137] Memory required for data: 2329586000
I0613 03:05:11.393429     7 layer_factory.hpp:77] Creating layer ct_conv4_1
I0613 03:05:11.393437     7 net.cpp:84] Creating Layer ct_conv4_1
I0613 03:05:11.393445     7 net.cpp:406] ct_conv4_1 <- ct_conv3_1
I0613 03:05:11.393452     7 net.cpp:380] ct_conv4_1 -> ct_conv4_1
I0613 03:05:11.393869     7 net.cpp:122] Setting up ct_conv4_1
I0613 03:05:11.393882     7 net.cpp:129] Top shape: 1 336 100 100 (3360000)
I0613 03:05:11.393887     7 net.cpp:137] Memory required for data: 2343026000
I0613 03:05:11.393896     7 layer_factory.hpp:77] Creating layer ct_relu4_1
I0613 03:05:11.393903     7 net.cpp:84] Creating Layer ct_relu4_1
I0613 03:05:11.393909     7 net.cpp:406] ct_relu4_1 <- ct_conv4_1
I0613 03:05:11.393916     7 net.cpp:367] ct_relu4_1 -> ct_conv4_1 (in-place)
I0613 03:05:11.393923     7 net.cpp:122] Setting up ct_relu4_1
I0613 03:05:11.393930     7 net.cpp:129] Top shape: 1 336 100 100 (3360000)
I0613 03:05:11.393936     7 net.cpp:137] Memory required for data: 2356466000
I0613 03:05:11.393942     7 layer_factory.hpp:77] Creating layer ct_conv5_1
I0613 03:05:11.393951     7 net.cpp:84] Creating Layer ct_conv5_1
I0613 03:05:11.393959     7 net.cpp:406] ct_conv5_1 <- ct_conv4_1
I0613 03:05:11.393965     7 net.cpp:380] ct_conv5_1 -> ct_conv5_1
I0613 03:05:11.397289     7 net.cpp:122] Setting up ct_conv5_1
I0613 03:05:11.397331     7 net.cpp:129] Top shape: 1 672 68 68 (3107328)
I0613 03:05:11.397343     7 net.cpp:137] Memory required for data: 2368895312
I0613 03:05:11.397362     7 layer_factory.hpp:77] Creating layer ct_relu5_1
I0613 03:05:11.397378     7 net.cpp:84] Creating Layer ct_relu5_1
I0613 03:05:11.397398     7 net.cpp:406] ct_relu5_1 <- ct_conv5_1
I0613 03:05:11.397419     7 net.cpp:367] ct_relu5_1 -> ct_conv5_1 (in-place)
I0613 03:05:11.397436     7 net.cpp:122] Setting up ct_relu5_1
I0613 03:05:11.397449     7 net.cpp:129] Top shape: 1 672 68 68 (3107328)
I0613 03:05:11.397456     7 net.cpp:137] Memory required for data: 2381324624
I0613 03:05:11.397465     7 layer_factory.hpp:77] Creating layer ct_fc1
I0613 03:05:11.397483     7 net.cpp:84] Creating Layer ct_fc1
I0613 03:05:11.397493     7 net.cpp:406] ct_fc1 <- ct_conv5_1
I0613 03:05:11.397505     7 net.cpp:380] ct_fc1 -> ct_fc1
I0613 03:05:11.404548     7 net.cpp:122] Setting up ct_fc1
I0613 03:05:11.404587     7 net.cpp:129] Top shape: 1 672 66 66 (2927232)
I0613 03:05:11.404592     7 net.cpp:137] Memory required for data: 2393033552
I0613 03:05:11.404604     7 layer_factory.hpp:77] Creating layer ct_fc1_relu
I0613 03:05:11.404620     7 net.cpp:84] Creating Layer ct_fc1_relu
I0613 03:05:11.404628     7 net.cpp:406] ct_fc1_relu <- ct_fc1
I0613 03:05:11.404639     7 net.cpp:367] ct_fc1_relu -> ct_fc1 (in-place)
I0613 03:05:11.404652     7 net.cpp:122] Setting up ct_fc1_relu
I0613 03:05:11.404662     7 net.cpp:129] Top shape: 1 672 66 66 (2927232)
I0613 03:05:11.404670     7 net.cpp:137] Memory required for data: 2404742480
I0613 03:05:11.404676     7 layer_factory.hpp:77] Creating layer ct_final
I0613 03:05:11.404687     7 net.cpp:84] Creating Layer ct_final
I0613 03:05:11.404693     7 net.cpp:406] ct_final <- ct_fc1
I0613 03:05:11.404701     7 net.cpp:380] ct_final -> ct_final
I0613 03:05:11.404747     7 net.cpp:122] Setting up ct_final
I0613 03:05:11.404757     7 net.cpp:129] Top shape: 1 21 66 66 (91476)
I0613 03:05:11.404762     7 net.cpp:137] Memory required for data: 2405108384
I0613 03:05:11.404770     7 layer_factory.hpp:77] Creating layer prob
I0613 03:05:11.404780     7 net.cpp:84] Creating Layer prob
I0613 03:05:11.404786     7 net.cpp:406] prob <- ct_final
I0613 03:05:11.404794     7 net.cpp:380] prob -> prob
I0613 03:05:11.404810     7 net.cpp:122] Setting up prob
I0613 03:05:11.404819     7 net.cpp:129] Top shape: 1 21 66 66 (91476)
I0613 03:05:11.404825     7 net.cpp:137] Memory required for data: 2405474288
I0613 03:05:11.404831     7 net.cpp:200] prob does not need backward computation.
I0613 03:05:11.404837     7 net.cpp:200] ct_final does not need backward computation.
I0613 03:05:11.404844     7 net.cpp:200] ct_fc1_relu does not need backward computation.
I0613 03:05:11.404850     7 net.cpp:200] ct_fc1 does not need backward computation.
I0613 03:05:11.404855     7 net.cpp:200] ct_relu5_1 does not need backward computation.
I0613 03:05:11.404861     7 net.cpp:200] ct_conv5_1 does not need backward computation.
I0613 03:05:11.404868     7 net.cpp:200] ct_relu4_1 does not need backward computation.
I0613 03:05:11.404875     7 net.cpp:200] ct_conv4_1 does not need backward computation.
I0613 03:05:11.404881     7 net.cpp:200] ct_relu3_1 does not need backward computation.
I0613 03:05:11.404886     7 net.cpp:200] ct_conv3_1 does not need backward computation.
I0613 03:05:11.404892     7 net.cpp:200] ct_relu2_1 does not need backward computation.
I0613 03:05:11.404899     7 net.cpp:200] ct_conv2_1 does not need backward computation.
I0613 03:05:11.404906     7 net.cpp:200] ct_relu1_2 does not need backward computation.
I0613 03:05:11.404911     7 net.cpp:200] ct_conv1_2 does not need backward computation.
I0613 03:05:11.404917     7 net.cpp:200] ct_relu1_1 does not need backward computation.
I0613 03:05:11.404922     7 net.cpp:200] ct_conv1_1 does not need backward computation.
I0613 03:05:11.404929     7 net.cpp:200] fc-final does not need backward computation.
I0613 03:05:11.404935     7 net.cpp:200] drop7 does not need backward computation.
I0613 03:05:11.404942     7 net.cpp:200] relu7 does not need backward computation.
I0613 03:05:11.404947     7 net.cpp:200] fc7 does not need backward computation.
I0613 03:05:11.404958     7 net.cpp:200] drop6 does not need backward computation.
I0613 03:05:11.404968     7 net.cpp:200] relu6 does not need backward computation.
I0613 03:05:11.404978     7 net.cpp:200] fc6 does not need backward computation.
I0613 03:05:11.404986     7 net.cpp:200] relu5_3 does not need backward computation.
I0613 03:05:11.404995     7 net.cpp:200] conv5_3 does not need backward computation.
I0613 03:05:11.405005     7 net.cpp:200] relu5_2 does not need backward computation.
I0613 03:05:11.405015     7 net.cpp:200] conv5_2 does not need backward computation.
I0613 03:05:11.405025     7 net.cpp:200] relu5_1 does not need backward computation.
I0613 03:05:11.405035     7 net.cpp:200] conv5_1 does not need backward computation.
I0613 03:05:11.405043     7 net.cpp:200] relu4_3 does not need backward computation.
I0613 03:05:11.405053     7 net.cpp:200] conv4_3 does not need backward computation.
I0613 03:05:11.405063     7 net.cpp:200] relu4_2 does not need backward computation.
I0613 03:05:11.405074     7 net.cpp:200] conv4_2 does not need backward computation.
I0613 03:05:11.405086     7 net.cpp:200] relu4_1 does not need backward computation.
I0613 03:05:11.405097     7 net.cpp:200] conv4_1 does not need backward computation.
I0613 03:05:11.405109     7 net.cpp:200] pool3 does not need backward computation.
I0613 03:05:11.405122     7 net.cpp:200] relu3_3 does not need backward computation.
I0613 03:05:11.405135     7 net.cpp:200] conv3_3 does not need backward computation.
I0613 03:05:11.405146     7 net.cpp:200] relu3_2 does not need backward computation.
I0613 03:05:11.405158     7 net.cpp:200] conv3_2 does not need backward computation.
I0613 03:05:11.405169     7 net.cpp:200] relu3_1 does not need backward computation.
I0613 03:05:11.405182     7 net.cpp:200] conv3_1 does not need backward computation.
I0613 03:05:11.405194     7 net.cpp:200] pool2 does not need backward computation.
I0613 03:05:11.405203     7 net.cpp:200] relu2_2 does not need backward computation.
I0613 03:05:11.405212     7 net.cpp:200] conv2_2 does not need backward computation.
I0613 03:05:11.405222     7 net.cpp:200] relu2_1 does not need backward computation.
I0613 03:05:11.405232     7 net.cpp:200] conv2_1 does not need backward computation.
I0613 03:05:11.405244     7 net.cpp:200] pool1 does not need backward computation.
I0613 03:05:11.405253     7 net.cpp:200] relu1_2 does not need backward computation.
I0613 03:05:11.405263     7 net.cpp:200] conv1_2 does not need backward computation.
I0613 03:05:11.405273     7 net.cpp:200] relu1_1 does not need backward computation.
I0613 03:05:11.405283     7 net.cpp:200] conv1_1 does not need backward computation.
I0613 03:05:11.405293     7 net.cpp:200] input does not need backward computation.
I0613 03:05:11.405303     7 net.cpp:242] This network produces output prob
I0613 03:05:11.405333     7 net.cpp:255] Network initialization done.
F0613 03:05:11.606521     7 upgrade_proto.cpp:95] Check failed: ReadProtoFromBinaryFile(param_file, param) Failed to parse NetParameter file: ../dilation8_pascal_voc.caffemodel
*** Check failure stack trace: ***
Aborted (core dumped)
(segment) root@milton-OptiPlex-9010:/data/code/segmentation_keras/conversion# 

Ur download of dataset is useless,would u please give a new link?

Ur download of dataset is useless,would u please give a new link?

Padding of 255 for label during training

Hi,
For Pascal VOC, there are 21 classes including background. To my knowledge the padding of 255 is not belong to those 21 class. How does keras handling the padded pixel during training. Will those pixels be just ignored?

Context module training implementation plans

Thanks for creating this implementation. Do you have any plans to implement training of the context module (to allow producing full resolution segmentation maps)?

IoU results

Have you by any chance compared this to the original implementation with regards to the mean IoU?
If so, what implementation of IoU did you use and what were your results?

Slow prediction

The generation of mask is too slow. is there any ways to speed it up?

the way of loading the weight

Hi nicolov,

In the post, you explained how to do the weight conversion. Due to the development environment constraints, it is a little bit hard for me to follow exactly your step.

In keras blog, author also show a way to load VGG16 weight from Keras directly. Do you think this weight can be used for your implementation? Do we have to use the converted caffe model weight for pascal_voc. The data set I will be using is of different domain with the data set published in the paper. Thanks for your advice.

regarding loading_weights

Hi nicolov,

In the train.py, you have included the function of load_weights(model, weights_path):. My understanding is that you are trying to load a pre-training vcg model. If I do not want to use this pretrained model because the problem I am working one may belong to a totally different domain, should I just skip calling this load_weights function? Or using a pre-trained model is always preferable, I am kind of confusing about this.

In the notes, you mentioned that The training code is currently limited to the frontend module, and thus only outputs 16x16 segmentation maps. If I would like to leverage this code for my own data set, what are the modifications that I have to make? Do I still have to load the weights?

Thank you very much!

Colour meaning in result seg-files

Hi there,

Thanks for the model. Would you please tell me the which label corresponds to the color in the
result image?

Problems with CuDNN library

While running train.py, this is the error message:
Epoch 1/20 E tensorflow/stream_executor/cuda/cuda_dnn.cc:378] Loaded runtime CuDNN library: 6021 (compatibility version 6000) but source was compiled with 5105 (compatibility version 5100). If using a binary install, upgrade your CuDNN library to match. If building from sources, make sure the library loaded at runtime matches a compatible version specified during compile configuration.

Since I don't have the root account, I can't install CuDNN v5. Do you know how I can fix this?
Thanks!

Questions on training part

Hi Nicolov,

Thanks a lot for this nice work, it's been of great help to me.

I would like to get your opinion on various topics regarding training part:

• can you provide some insights on how to have a system returning image size results ? (instead of 16*16)
• to adapt to a problem with only two classes (say foreground/background), I only changed every 21 to 2 in the code, but the results are not as nice as I expected for now... Am I doing smth wrong ?
• do you suggest fixing the weights of the first layers or not ?
• I'm currently working on data augmentation to add elastic deformations, do you plan to add it as well ? I could share it once it's done.

Thanks a lot

vgg_conv.npy

Run train.py Error: Invalid value for "--weights-path": Path "conversion/converted/vgg_conv.npy" does not exist.

tensorflow 1.3 support

Hello,

May I ask you to update the repo to the latest Tensorflow version?

Nice work, are you planning to complete the training part ?

docker running error

Hi, @nicolov ,

For the caffe weight conversion, I got the following error:

(tf_1.0) root@milton-All-Series:/data/code/segmentation_keras/conversion# docker run -v $(pwd):/workspace -ti `docker build -q .`
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
"docker run" requires at least 1 argument(s).
See 'docker run --help'.

Usage:  docker run [OPTIONS] IMAGE [COMMAND] [ARG...]

Run a command in a new container
(tf_1.0) root@milton-All-Series:/data/code/segmentation_keras/conversion#

It shows that docker daemon is not running. Any other command should I input before it?

Thanks

Failure running conversion through provided Docker container

Hello,

While trying to experiment with the project, I ran into the issue with the conversion of the caffe model.

Sending build context to Docker daemon 3.072kB
Step 1/4 : FROM bvlc/caffe:cpu
---> 0b577b836386
Step 2/4 : ENV MODEL_NAME dilation8_pascal_voc
---> Using cache
---> 75c63a3c3735
Step 3/4 : RUN pip install tensorflow
---> Running in 7ce61c615947
Collecting tensorflow
Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f32ca357d10>, 'Connection to pypi.python.org timed out. (connect timeout=15)')': /simple/tensorflow/
Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f32ca3573d0>, 'Connection to pypi.python.org timed out. (connect timeout=15)')': /simple/tensorflow/
Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f32ca357310>, 'Connection to pypi.python.org timed out. (connect timeout=15)')': /simple/tensorflow/
Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f32ca357b90>, 'Connection to pypi.python.org timed out. (connect timeout=15)')': /simple/tensorflow/
Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f32ca357d50>, 'Connection to pypi.python.org timed out. (connect timeout=15)')': /simple/tensorflow/
Could not find a version that satisfies the requirement tensorflow (from versions: )
No matching distribution found for tensorflow
The command '/bin/sh -c pip install tensorflow' returned a non-zero code: 1

Any recommendations on how to fix the problem?