ascust / sec-mxnet Goto Github PK

I just have one object. My CLASS_NUM=2.When I run train_fg_cues.py, I got this:

Epoch[1] Batch[10] Speed: 29.01 samples/sec MultiLogisticLoss=0.100427 time=17/01/2019--21:28:53
Epoch[1] Batch[20] Speed: 29.69 samples/sec MultiLogisticLoss=0.000000 time=17/01/2019--21:28:58
Epoch[1] Batch[30] Speed: 29.92 samples/sec MultiLogisticLoss=0.000000 time=17/01/2019--21:29:03
Epoch[1] Batch[40] Speed: 29.68 samples/sec MultiLogisticLoss=0.000000 time=17/01/2019--21:29:08
Epoch[1] Batch[50] Speed: 29.80 samples/sec MultiLogisticLoss=0.000000 time=17/01/2019--21:29:13
Epoch[1] Batch[60] Speed: 28.67 samples/sec MultiLogisticLoss=0.000000 time=17/01/2019--21:29:18
Epoch[1] Batch[70] Speed: 29.99 samples/sec MultiLogisticLoss=0.000000 time=17/01/2019--21:29:23
Epoch[1] Batch[80] Speed: 28.36 samples/sec MultiLogisticLoss=0.000000 time=17/01/2019--21:29:29
Epoch[1] Batch[90] Speed: 28.10 samples/sec MultiLogisticLoss=0.000000 time=17/01/2019--21:29:34
Epoch[1] Batch[100] Speed: 28.93 samples/sec MultiLogisticLoss=0.000000 time=17/01/2019--21:29:39
Epoch[1] Batch[110] Speed: 28.58 samples/sec MultiLogisticLoss=0.000000 time=17/01/2019--21:29:44
Epoch[1] Batch[120] Speed: 29.89 samples/sec MultiLogisticLoss=0.000000 time=17/01/2019--21:29:49

U have used Segmentation classes for creating Augmented images.Will the Augmented images be used for training?If so aren't we using fully supervised data for training? Please let me know

generating cue file
cue file generated
{'VOC_TRAIN_LIST': 'train_list.txt', 'WD': 0.0005, 'EPOCH_SIZE': 1000, 'WORKSPACE': 512, 'CRF_BI_W': 10, 'BG_CUE_FILE': 'bg_cue_initsec.p', 'EPOCH_SIZE_BG': 8000, 'EPOCH_SIZE_FG': 8000, 'MAX_EPOCH': 8, 'LOG_FOLDER': 'log', 'CPU_WORKER_NUM': 8, 'CRF_BI_XY_STD': 80, 'BATCH_SIZE': 15, 'OUTPUT_FOLDER': 'outputs', 'LR_DECAY': 2000, 'LR': 0.001, 'CRF_POS_XY_STD': 3, 'CLASS_NUM': 21, 'VOCDEVKIT_PATH': '/home/hkt/Videos/VOCdevkit/VOC2012/', 'MOMENTUM': 0.9, 'CRF_BI_RGB_STD': 10, 'Q_FG': 0.996, 'FG_CUE_FILE': 'fg_cue_initsec.p', 'Q_BG': 0.999, 'CRF_POS_W': 3, 'INPUT_SIZE_SEC': 320, 'LR_FG': 0.001, 'BASE_NET': 'vgg16', 'SNAPSHOT_FOLDER': 'snapshots', 'SBD_PATH': '/home/hkt/Videos/benchmark_RELEASE/dataset/', 'BATCH_SIZE_FG': 15, 'VOC_VAL_IM_FOLDER': 'val_images', 'BATCH_SIZE_BG': 15, 'LR_BG': 0.001, 'DATASET_PATH': 'dataset', 'VOC_VAL_LIST': 'val_list.txt', 'LD_DECAY_BG': 2000, 'MEAN_RGB': [123, 117, 104], 'DOWN_SAMPLE_SEC': 8, 'VOC_TRAIN_MULTI_FILE': 'voc_multi_file.p', 'CACHE_PATH': 'cache', 'SALIENCY_TH_FG': 0.2, 'SALIENCY_TH_BG': 0.1, 'CUE_FILE_INITSEC': 'sec_cue.p', 'VOC_TRAIN_IM_FOLDER': 'train_images', 'VOC_VAL_MASK_FOLDER': 'val_masks', 'LR_DECAY_FG': 2000}
start training model SEC_vgg16
Traceback (most recent call last):
File "/home/hkt/PycharmProjects/SEC-MXNet-master/train_SEC.py", line 58, in
seg_net = net_symbol.create_training(class_num=conf.CLASS_NUM, outputsize=output_size, workspace=conf.WORKSPACE)
File "/home/hkt/PycharmProjects/SEC-MXNet-master/cores/symbols/SEC_vgg16.py", line 75, in create_training
bi_w=10, maxiter=10, scale_factor=12.0, min_prob=0.0001)
File "", line 52, in Custom
File "/home/hkt/anaconda3/envs/paddlepaddle/lib/python2.7/site-packages/mxnet/_ctypes/symbol.py", line 125, in _symbol_creator
ctypes.byref(sym_handle)))
File "/home/hkt/anaconda3/envs/paddlepaddle/lib/python2.7/site-packages/mxnet/base.py", line 251, in check_call
raise MXNetError(py_str(_LIB.MXGetLastError()))
mxnet.base.MXNetError: [13:15:07] src/operator/custom/custom.cc:104: Check failed: CustomOperator::Get()->Find(params.op_type) != nullptr Cannot find custom operator crf_layer

Stack trace returned 10 entries:
[bt] (0) /home/hkt/anaconda3/envs/paddlepaddle/lib/python2.7/site-packages/mxnet/libmxnet.so(+0x39008a) [0x7fcf0796608a]
[bt] (1) /home/hkt/anaconda3/envs/paddlepaddle/lib/python2.7/site-packages/mxnet/libmxnet.so(+0x3906c1) [0x7fcf079666c1]
[bt] (2) /home/hkt/anaconda3/envs/paddlepaddle/lib/python2.7/site-packages/mxnet/libmxnet.so(+0x56d564) [0x7fcf07b43564]
[bt] (3) /home/hkt/anaconda3/envs/paddlepaddle/lib/python2.7/site-packages/mxnet/libmxnet.so(+0x71811fa) [0x7fcf0e7571fa]
[bt] (4) /home/hkt/anaconda3/envs/paddlepaddle/lib/python2.7/site-packages/mxnet/libmxnet.so(MXSymbolCreateAtomicSymbol+0x6dc) [0x7fcf0a098e1c]
[bt] (5) /home/hkt/anaconda3/envs/paddlepaddle/lib/python2.7/lib-dynload/../../libffi.so.6(ffi_call_unix64+0x4c) [0x7fcf2fbcdec0]
[bt] (6) /home/hkt/anaconda3/envs/paddlepaddle/lib/python2.7/lib-dynload/../../libffi.so.6(ffi_call+0x22d) [0x7fcf2fbcd87d]
[bt] (7) /home/hkt/anaconda3/envs/paddlepaddle/lib/python2.7/lib-dynload/_ctypes.so(_ctypes_callproc+0x4de) [0x7fcf2fde48de]
[bt] (8) /home/hkt/anaconda3/envs/paddlepaddle/lib/python2.7/lib-dynload/_ctypes.so(+0x9b31) [0x7fcf2fddab31]
[bt] (9) /home/hkt/anaconda3/envs/paddlepaddle/bin/../lib/libpython2.7.so.1.0(PyObject_Call+0x43) [0x7fcf3bccd973]

Process finished with exit code 1

just as the title says, may i ask whether you get the identical seeds as provided by the original repo, as I find the localization cue important for training the segmentation network

Hi, May I ask you questions about your implementation of softmax function?
in ./cores/symbols/SEC_syms.py, you implement softmax as follows:

def create_softmax(preds, min_prob=0.0001):
    min_prob_const = mx.sym.ones((1, 1, 1, 1)) * min_prob
    preds_max = mx.sym.max(preds, keepdims=True, axis=1, name="softmax_max")
    preds_exp = mx.sym.exp(mx.sym.broadcast_minus(preds, preds_max, name="softmax_minus"))
    tmp1 = mx.sym.sum(preds_exp, keepdims=True, axis=1)
    tmp2 = mx.sym.broadcast_div(preds_exp, tmp1)
    tmp3 = mx.sym.broadcast_add(tmp2, min_prob_const)
    probs = mx.sym.broadcast_div(tmp3, mx.sym.sum(tmp3, axis=1, keepdims=True))
    return probs

In my mind, the variable 'tmp2' is exactly the softmax output, why do you add another small value and normalize again?

I trained the model for generating the seed cues for my own dataset.But the foreground seed cue file contains blank arrays.Can you please suggest me how to correct it?

first i follow the readme training voc for vgg16, val miou = 0.5005 , result is ok.
then i training voc for resnet50 ,i only change the core.config.py file params for these :
conf.BASE_NET = "resnet50",
conf.LOG_FOLDER = "log_resnet50"
conf.SNAPSHOT_FOLDER = "snapshots_resnet50"
conf.OUTPUT_FOLDER = "outputs_resnet50"
val miou = 0.4883, result even became lower than vgg16, i did not know why? hope you can help me!
there are some final loss for vgg16 resnet50 log, first is vgg16,second is resnet50
//train_bg_cue_net.log
Train-MultiLogisticLoss= 0.008143 0.027573
//train_fg_cue_net.log
Train-MultiLogisticLoss= 0.057217 0.065230
//train_sec_model.log final loss
Train-SEC_seed_loss=0.246602 0.336656
Train-SEC_constrain_loss=0.226370 0.324345
Train-SEC_expand_loss=0.861525 0.847647

Hi, Thank you very much for sharing the code. There was an error while running: no such file or directory: 'dataset/voc_multi_file.p'， bg_cue_net-1.params， fg_cue_net-1.params. Can you share this file?

do you test the single-gpu training performance?

I cann't find the net_symbol file.You used it in train_SEC.py

I have used your code to generate cues for a 2 category dataset.I got this runtime error while training the cues.Please do look into it
start training fg cues for SEC.
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/mxnet/symbol/symbol.py", line 1623, in simple_bind
ctypes.byref(exe_handle)))
File "/usr/local/lib/python3.5/dist-packages/mxnet/base.py", line 253, in check_call
raise MXNetError(py_str(_LIB.MXGetLastError()))
mxnet.base.MXNetError: [08:47:13] src/storage/storage.cc:119: Compile with USE_CUDA=1 to enable GPU usage

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "train_fg_cues.py", line 56, in
momentum=conf.MOMENTUM, lr_decay=conf.LR_DECAY_FG, workspace=conf.WORKSPACE)
File "/home/rashmi/SEC-MXNet-master/cores/train_multi_wrapper.py", line 40, in train_multi_wrapper
label_shapes=data_iter.provide_label)
File "/usr/local/lib/python3.5/dist-packages/mxnet/module/module.py", line 429, in bind
state_names=self._state_names)
File "/usr/local/lib/python3.5/dist-packages/mxnet/module/executor_group.py", line 279, in init
self.bind_exec(data_shapes, label_shapes, shared_group)
File "/usr/local/lib/python3.5/dist-packages/mxnet/module/executor_group.py", line 375, in bind_exec
shared_group))
File "/usr/local/lib/python3.5/dist-packages/mxnet/module/executor_group.py", line 662, in _bind_ith_exec
shared_buffer=shared_data_arrays, **input_shapes)
File "/usr/local/lib/python3.5/dist-packages/mxnet/symbol/symbol.py", line 1629, in simple_bind
raise RuntimeError(error_msg)
RuntimeError: simple_bind error. Arguments:
label: (15, 20)
data: (15, 3, 320, 320)
[08:33:20] src/storage/storage.cc:119: Compile with USE_CUDA=1 to enable GPU usage

The URL of VGG16 is broken. Could you update the URL?

Sorry I'm new to MXNet, but the usage of your crf layer in SEC_vgg16.py and SEC_resnet50.py seems incorrect. I have looked into the CRF layer, it requires the input unary to be 'log(prob)' form, as indicated below:

def inference(self, im, unary):
	#im: image HxWx3
	#unary: CxHxW should be "log(p)". take log of probability

	H, W = im.shape[:2]
	C = unary.shape[0]
	d = dcrf.DenseCRF2D(W, H, C)
	d.setUnaryEnergy(-unary.reshape(C, -1))

however, in SEC_vgg16.py and SEC_resnet50.py, when you use the CRF layer, your input unary term seems not satisfy the requirement. for example, in SEC_vgg16.py

preds = create_classifier(body, class_num=class_num, workspace=workspace, lr_type="alex10")
probs = SEC_syms.create_softmax(preds)
crf = mx.symbol.Custom(data=preds, small_ims=small_ims, name='crf', op_type='crf_layer',
                           pos_xy_std=3, pos_w=3, bi_xy_std=80, bi_rgb_std=13, bi_w=10,
                            maxiter=10, scale_factor=12.0, min_prob=0.0001)

variable 'preds' is the predicted score map and I don't see you perform any futher transformations in 'crf_layer' before feeding it to inference function,

I just have one object. So My CLASS_NUM=2. After I run train_fg_cues.py and train_bg_cues.py, I run train_SEC.py, and I got Epoch[1] Batch[10] Speed: 20.38 samples/sec SEC_seed_loss=nan SEC_constrain_loss=nan SEC_expand_loss=nan time=17/01/2019--22:11:03.
Why this happened?

Hi
I trained bg_cues and fg_cues successfully. After that, when I train SEC then loss for S,E and C are nan with default setting of conf.Q_FG = 0.996, conf.Q_BG = 0.999 and others. When I tweak different parameters by lowering learning rate etc., no changes in loss as nan before. So after tweaking conf.Q_FG and conf.Q_BG by lowering it, loss giving some values and reducing till 5th batch. And then again giving nan for each. Problem lies in this when this warning comes then loss become nan.
Projects/SEC-MXNet/cores/symbols/crf_layer.py:24: RuntimeWarning: invalid value encountered in less
self.result[self.result<self.min_prob] = self.min_prob

Please enlighten me about that and conf.Q_FG & conf.Q_BG

More details:
def inference(self, im, unary):
#im: image HxWx3
#unary: CxHxW should be "log(p)". take log of probability

This unary is nan actually

        unary = in_data[0].asnumpy()
        small_ims = in_data[1].asnumpy()
        N = unary.shape[0]
	print 'in_data: ', in_data[0], in_data[1]
        self.result = np.zeros(unary.shape, dtype=np.float32)
        for i in range(N):
            self.result[i] = self.CRF.inference(small_ims[i], unary[i])
            # print np.min(unary[i]), np.max(unary[i]), np.mean(np.abs(unary[i]))
            # print np.any(np.isnan(unary[i])), np.any(np.isnan(self.result[i]))

	#print 'self.min_prob: ', self.result, self.min_prob

        self.result[self.result<self.min_prob] = self.min_prob

@ascust Hello, your work is very amazing! I just want to segment humans and backgrounds, and I don't want to segment other objects on my own datasets. I have about 5000 images, 2500 of them contain humans, 2500 of them have no humans. However, in my dataset, the images contain humans are taken in the same indoor scene (scene A) and images without humans are taken in different indoor scenes (scene B, scene C, scene D ... ). None of the images without humans is taken in the scene A. Also, the images without humans may contains other objects, such as bottles, books, chairs, desks, etc. Can I generate accurate seeds to locate only human seeds as foreground in this case? Thank you very much!

http://www.eecs.berkeley.edu/Research/Projects/CS/vision/grouping/semantic_contours/benchmark.tgz will be ok