lim-anggun / fgsegnet

FgSegNet: Foreground Segmentation Network, Foreground Segmentation Using Convolutional Neural Networks for Multiscale Feature Encoding

FgSegNet : Foreground Segmentation Network

This repository contains source codes and training sets for the following paper:

"Foreground Segmentation Using Convolutional Neural Networks for Multiscale Feature Encoding" by Long Ang LIM and Hacer YALIM KELES

The preprint version of the above paper is available at: https://arxiv.org/abs/1801.02225

Citation

If you find FgSegNet useful in your research, please consider citing:

@article{LIM2018256,
        title = "Foreground segmentation using convolutional neural networks for multiscale feature encoding",
        journal = "Pattern Recognition Letters",
        volume = "112",
        pages = "256 - 262",
        year = "2018",
        issn = "0167-8655",
        doi = "https://doi.org/10.1016/j.patrec.2018.08.002",
        url = "http://www.sciencedirect.com/science/article/pii/S0167865518303702",
        author = "Long Ang Lim and Hacer Yalim Keles",
        keywords = "Foreground segmentation, Background subtraction, Deep learning, Convolutional neural networks, Video surveillance, Pixel classification"
}

Requirements

This work was implemented with the following frameworks:

• Spyder 3.2.x (recommended)
• Python 3.6.3
• Keras 2.0.6
• Tensorflow-gpu 1.1.0

Usage

Easy to train! Just a single click, go!

1. Clone this repo: git clone https://github.com/lim-anggun/FgSegNet.git

2. Modify the following file:

   • <Your PYTHON 3.6>\site-packages\skimage\transform\pyramids.py
     in pyramid_reduce function, replace the following two lines
     out_rows = math.ceil(rows / float(downscale))
out_cols = math.ceil(cols / float(downscale))
     with
     out_rows = math.floor(rows / float(downscale))
out_cols = math.floor(cols / float(downscale))

3. Download VGG16 weights from Here and put it in FgSegNet/FgSegNet/ dir, or it will be downloaded and stored in /.keras/models/ automatically.

4. Download CDnet2014 dataset, then put it in the following directory structure:
   

   Example:

     FgSegNet/
          FgSegNet/FgSegNet_M_S_CDnet.py
                  /FgSegNet_M_S_SBI.py
                  /FgSegNet_M_S_UCSD.py
                  /FgSegNet_M_S_module.py
          SBI2015_dataset/
          SBI2015_train/
          UCSD_dataset/
          UCSD_train20/
          UCSD_train50/
          FgSegNet_dataset2014/
                    baseline/
                         highway50
                         highway200
                         pedestrians50
                         pedestrians200
                         ...
                     badWeather/
                         skating50
                         skating200
                         ...
                    ...
           CDnet2014_dataset/
                     baseline/
                          highway
                          pedestrians
                          ...
                     badWeather/
                          skating
                          ...
                     ...
   

5. There are two methods; i.e. FgSegNet_M and FgSegNet_S. Choose a method that you want to train by setting method_name=='FgSegNet_M' or method_name=='FgSegNet_S'.

6. Run the codes with Spyder IDE. Note that all trained models will be automatically saved (in current working directory) for you.

Evaluation

on CDnet2014 dataset

We perform two separated evaluations and report our results on two test splits (test dev & test challenge):

1. We compute our results locally. (on test dev dataset)
2. We upload our results to Change Detection 2014 Challenge. (on test challenge dataset where ground truth values are not shared with the public dataset)
   (Both results are reported in our paper. Please refer to it for details)
   

Compute metrics locally using changedetection.net > UTILITIES tab.

Note:

• test dev: by considering only the range of the frames that contain the ground truth labels by excluding training frames (50 or 200 frames)
• test challenge: dataset on the server side (http://changedetection.net)

on SBI2015 dataset

We split 20% for training (denoted by n frames, where n ∈ [2−148]) and 80% for testing.

on UCSD Background Subtraction dataset

We perform two sets of experiment: first, we split the frames 20% for training (denoted by n frames, where n ∈ [3 − 23]) and 80% for testing, second we split 50% for training (where n ∈ [7 − 56]) and remaining 50% for testing.

Results

Results on CDnet2014 dataset

Table below shows overall results across 11 categories obtained from Change Detection 2014 Challenge.

Methods	PWC	F-Measure	Speed (320x240, batch-size=1) on NVIDIA GTX 970 GPU
FgSegNet_M	0.0559	0.9770	18fps
FgSegNet_S	0.0461	0.9804	21fps

Results on SBI2015 dataset

Table below shows overall test results across 14 video sequences.

Methods	PWC	F-Measure
FgSegNet_M	0.9431	0.9794
FgSegNet_S	0.8524	0.9831

Results on UCSD Background Subtraction dataset

Tables below show overall test results across 18 video sequences.

For 20% split

Methods	PWC(th=0.4)	F-Measure(th=0.4)	PWC(th=0.7)	F-Measure(th=0.7)
FgSegNet_M	0.6260	0.8948	0.6381	0.8912
FgSegNet_S	0.7052	0.8822	0.6273	0.8905

For 50% split

Methods	PWC(th=0.4)	F-Measure(th=0.4)	PWC(th=0.7)	F-Measure(th=0.7)
FgSegNet_M	0.4637	0.9203	0.4878	0.9151
FgSegNet_S	0.5024	0.9139	0.4676	0.9149

Updates

07/08/2018:

• combine FgSegNet_S with FgSegNet_M and more
• add SBI2015, UCSD dataset and more results
• add published paper and a new citation
• integrate custom loss and other codes in a single file, for ease of use
• other improvements

09/06/2018:

• add quantity results on changedetection.net (CDnet2014 dataset)
• add supporting codes

29/04/2018:

• add Jupyter notebook and a model for testing (FgSegNet_M).

27/01/2018:

• add FgSegNet_M (a triplet network) source code and training frames

Contact

lim.longang at gmail.com
Any issues/discussions are welcome.

fgsegnet's Issues

How should I extract the foreground mask

How should I extract the foreground mask？

test program

I want test those remaining pictures. But I did not find the test program. Can you share the test program for me? I 'm very confused.Thank you!

how to train?

where is the main function? how to train it?

适合背景变化的场景吗

我想提取画面中的人物轮廓，如果背景中有随风摆动的树叶，或者来往的车辆，还可以很好的提取人体轮廓吗？

train speed

I want to know the train speed for 3000 pictures with one object...and I have a 1080Ti...

Testing: Algorithm does not segment foreground

Dear Long Ang Lim,

Thank you for providing your source code with detail guide to run it.

I successfully trained your method on several CDNet videos. However, when I tried to test it on remaining frames say highway sequence, it does not segment foreground object. The binary mask obtained shows only background values.
I tried every possible solution. All in vain.
Do you have any idea what going wrong?

Thank You.

how many models after trainning cdw_dataset2014

"A common approach for moving objects segmentation in a scene is to perform a background subtraction.Several methods have been proposed in this domain."
above in your paper, but I readed your code,
for scene in scene_list:
print ('Training ->>> ' + category + ' / ' + scene)

    # training frame path and dataset2014 path
    # '..' up dir 1 level 
    train_dir = os.path.join('..', 'FgSegNet_dataset2014', category, scene + str(num_frames))
    dataset_dir = os.path.join('..', 'CDnet2014_dataset', category, scene)
        
    mdl_path = os.path.join(mdl_dir, 'mdl_' + scene + '.h5')

    results = generateData(train_dir, dataset_dir, scene)
    train(results, scene, mdl_path, log_dir, vgg_weights_path)

I have a question
how many models after trainning cdw_dataset2014? is it common approach ？

Regarding the Requirement.txt file and Spefic version of Numpy

Hello,
I want to use your model as the backbone of my research, I tried to run the model, but it gave errors. Also, I install TensorFlow-gpu == 1.1.0 and Keras == 2.0.6.
Error:
When I run the get_data function, there was an output of the data format:
My Output --->> (5, 240, 320, 3) (5, 120, 160, 2) (5, 60, 80, 1)
Your OutPut---->> (5, 240, 320, 3) (5, 120, 160, 3) (5, 60, 80, 3)

This is not the same as your output format. So, I kindly request to you please, upload the requirements.txt file.

how to load DataSet ? (train_images, train_labels), (test_images, test_labels)

hi, i want to organize the CDnet dataset into (train_images, train_labels), (test_images, test_labels) for training and testing my own systeme mainly based on CNN with keras.

How can I solve this error?

->>> baseline / highway
Traceback (most recent call last):

File "", line 1, in
runfile('D:/ShiYanchnegxu/FgSegNet_v2-master/testing_scripts/extract_mask.py', wdir='D:/ShiYanchnegxu/FgSegNet_v2-master/testing_scripts')

File "D:\ProgramData\Anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.py", line 705, in runfile
execfile(filename, namespace)

File "D:\ProgramData\Anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.py", line 102, in execfile
exec(compile(f.read(), filename, 'exec'), namespace)

File "D:/ShiYanchnegxu/FgSegNet_v2-master/testing_scripts/extract_mask.py", line 239, in
model = load_model(mdl_path)

File "D:\ProgramData\Anaconda3\envs\TF1.1.0\lib\site-packages\keras\models.py", line 227, in load_model
with h5py.File(filepath, mode='r') as f:

File "D:\ProgramData\Anaconda3\lib\site-packages\h5py_hl\files.py", line 269, in init
fid = make_fid(name, mode, userblock_size, fapl, swmr=swmr)

File "D:\ProgramData\Anaconda3\lib\site-packages\h5py_hl\files.py", line 99, in make_fid
fid = h5f.open(name, flags, fapl=fapl)

File "h5py_objects.pyx", line 54, in h5py._objects.with_phil.wrapper

File "h5py_objects.pyx", line 55, in h5py._objects.with_phil.wrapper

File "h5py\h5f.pyx", line 78, in h5py.h5f.open

OSError: Unable to open file (unable to open file: name = 'FgSegNet_v2\models25\baseline\mdl_highway.h5', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0)

the test file and the training parameter model

Hello, I did not find the test file in the file you posted, can you publish the test file and the training parameter model?

Tensorflow version upgrade

Anyone could upgrade it to at least tensorflow 1.14?

Use on new video sequence

Dear @lim-anggun,
How one can apply your implemented method on a new video sequence (i.e., a new video with, for example 50 first frames of it)? The network must be trained again on the new video sequence?

Save models

I have a question, after training the model ,how to save the model's weight? For every scene,I need to save corresponding weight,?if so ,it needs to save 52 weight files, could you help me?

Can i inference the keras model by OpenCV DNN module？

I want to apply FgSegNet on my project ，which programmed by C++

Evaluation Code

Hello how to evaluate the result of the method in CDnet 2014 dataset? I follow your links and tried to follow everything but I got this error.

The file C:\dataset\badWeather\blizzard\stats.txt doesn't exist.
It means there was an error calling the comparator.
Traceback (most recent call last):
  File "modified_process_folder.py", line 129, in <module>
    main()
  File "modified_process_folder.py", line 46, in main
    processFolder(datasetPath, binaryRootPath)
  File "modified_process_folder.py", line 62, in processFolder
    confusionMatrix = compareWithGroungtruth(videoPath, binaryPath)  #STATS
  File "modified_process_folder.py", line 84, in compareWithGroungtruth
    return readCMFile(statFilePath)
  File "modified_process_folder.py", line 90, in readCMFile
    raise Exception('error')
Exception: error

Should we recompile the comparator?, but in original file comparator.exe is exist.

Question of Modify the following file

Hi , i have find the file "pyramids.py" and find the function "pyramid_reduce",but i didn't find the line of
"out_rows = math.ceil(rows / float(downscale))
out_cols = math.ceil(cols / float(downscale)) ",any soluvation,thanks

Hi lawpdas, Have you solved the problem?

ValueError: Error when checking input: expected ip_scale2 to have shape (None, 157, 322, 3) but got array with shape (50, 158, 323, 3)

I have a problem, do you know how to solve it, thank you very much!
Training ->>> turbulence / turbulence2
(50, 315, 645, 3) (50, 158, 323, 3) (50, 79, 162, 3)

ValueError Traceback (most recent call last)
/apsarapangu/disk6/wuting/FgSegNet/FgSegNet-master/FgSegNet/FgSegNet.py in ()
219
220 results = generateData(train_dir, dataset_dir, scene)
--> 221 train(results, scene, mdl_path, log_dir, vgg_weights_path)
222 del results

/apsarapangu/disk6/wuting/FgSegNet/FgSegNet-master/FgSegNet/FgSegNet.py in train(results, scene, mdl_path, log_dir, vgg_weights_path)
147 redu = keras.callbacks.ReduceLROnPlateau(monitor='val_loss', factor=reduce_factor, patience=num_patience, verbose=1, mode='auto', epsilon=0.0001, cooldown=0, min_lr=0)
148 model.fit([results[0], results[1], results[2]], results[3], validation_split=0.2, epochs=epoch, batch_size=batch_size,
--> 149 callbacks=[redu, chk, tb], verbose=1, class_weight=results[4], shuffle = True)
150
151 del model, results, tb, chk, redu

/apsarapangu/disk6/wuting/Anaconda3/lib/python3.5/site-packages/keras/engine/training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, **kwargs)
1356 class_weight=class_weight,
1357 check_batch_axis=False,
-> 1358 batch_size=batch_size)
1359 # Prepare validation data.
1360 if validation_data:

/apsarapangu/disk6/wuting/Anaconda3/lib/python3.5/site-packages/keras/engine/training.py in _standardize_user_data(self, x, y, sample_weight, class_weight, check_batch_axis, batch_size)
1232 self._feed_input_shapes,
1233 check_batch_axis=False,
-> 1234 exception_prefix='input')
1235 y = _standardize_input_data(y, self._feed_output_names,
1236 output_shapes,

/apsarapangu/disk6/wuting/Anaconda3/lib/python3.5/site-packages/keras/engine/training.py in _standardize_input_data(data, names, shapes, check_batch_axis, exception_prefix)
138 ' to have shape ' + str(shapes[i]) +
139 ' but got array with shape ' +
--> 140 str(array.shape))
141 return arrays
142

ValueError: Error when checking input: expected ip_scale2 to have shape (None, 157, 322, 3) but got array with shape (50, 158, 323, 3)

Testing results

When I test those remaining pictures,I found the testing results are bad.The segmentation mask max is about 0.3,I don't know why ,But I read your paper,you said the testing results are good,so I want to ask you how to test ,Can you help me ? I 'm very confused.Thank you!

Unknown layer: MyUpSampling2D

hi :
I run your test_prediction.ipynb, and use my train weight xxx.h5,but I get a error "Unknown layer: MyUpSampling2D" when load_model(mdl_path), do you got this issue ?
thanks vary much

UnboundLocalError: local variable 'inlist' referenced before assignment

any idea?

Traceback (most recent call last):
File "/Users/alpullu/Downloads/FgSegNet-master/FgSegNet/FgSegNet.py", line 372, in
results = generateData(train_dir, dataset_dir, scene)
File "/Users/alpullu/Downloads/FgSegNet-master/FgSegNet/FgSegNet.py", line 242, in generateData
del X_list_temp, inlist, gtlist
UnboundLocalError: local variable 'inlist' referenced before assignment

about train and test

I have trained a model with your code, but it can not segment foreground object.
The acc become 0.8937 and 0.8999 at first two epoch, and it will remain unchanged.
Is this normal?

Epoch 55/60
40/40 [==============================] - 3s - loss: 1.7747 - acc: 0.8937 - val_loss: 1.6751 - val_acc: 0.8999
Epoch 56/60
40/40 [==============================] - 2s - loss: 1.7732 - acc: 0.8937 - val_loss: 1.6737 - val_acc: 0.8999
Epoch 57/60
40/40 [==============================] - 2s - loss: 1.7717 - acc: 0.8937 - val_loss: 1.6722 - val_acc: 0.8999
Epoch 58/60
40/40 [==============================] - 2s - loss: 1.7703 - acc: 0.8937 - val_loss: 1.6708 - val_acc: 0.8999
Epoch 59/60
40/40 [==============================] - 2s - loss: 1.7689 - acc: 0.8937 - val_loss: 1.6694 - val_acc: 0.8999
Epoch 60/60
40/40 [==============================] - 2s - loss: 1.7675 - acc: 0.8937 - val_loss: 1.6681 - val_acc: 0.8999

Can I use this trained netowork on my videos

@lim-anggun Hi
I wanna know that whether I can run this trained model directly on my videos as we can do with opencv foregroundextractor and other vision libraries foreground and background extractors
Thanks

Issue when training

I followed the steps mentioned in the README. After I downloaded the dataset and placed it in the right directory, I used FgSegNet.py to start the training. Pasting below the error log:

(50, 240, 320, 3) (50, 120, 160, 3) (50, 60, 80, 3)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
/FgSegNet/FgSegNet/FgSegNet.py in <module>()
    219 
    220         results = generateData(train_dir, dataset_dir, scene)
--> 221         train(results, scene, mdl_path, log_dir, vgg_weights_path)
    222         del results

/FgSegNet/FgSegNet/FgSegNet.py in train(results, scene, mdl_path, log_dir, vgg_weights_path)
    141     img_shape = results[0][0].shape
    142     model = FgSegNetModule(lr, reg, img_shape, scene, vgg_weights_path)
--> 143     model = model.initModel()
    144 
    145     tb = keras.callbacks.TensorBoard(log_dir=log_dir, histogram_freq=0, batch_size=batch_size, write_graph=False, write_grads=True, write_images=True, embeddings_freq=0, embeddings_layer_names=None, embeddings_metadata=None)

/content/FgSegNet/FgSegNet/FgSegNetModule.py in initModel(self)
     85 
     86         input_1 = Input(shape=(h, w, d), name='ip_scale1')
---> 87         vgg_layer_output = self.VGG16(input_1)
     88         shared_model = Model(inputs=input_1, outputs=vgg_layer_output, name='shared_model')
     89         shared_model.load_weights(self.vgg_weights_path, by_name=True)

/content/FgSegNet/FgSegNet/FgSegNetModule.py in VGG16(self, x)
     27 
     28         # Block 1
---> 29         x = Conv2D(64, (3, 3), activation='relu', padding='same', name='block1_conv1', data_format='channels_last')(x)
     30         x = Conv2D(64, (3, 3), activation='relu', padding='same', name='block1_conv2')(x)
     31         x = MaxPooling2D((2, 2), strides=(2, 2), name='block1_pool')(x)

/usr/local/lib/python3.6/dist-packages/keras/legacy/interfaces.py in wrapper(*args, **kwargs)
     89         return wrapper
     90     return legacy_support
---> 91 
     92 
     93 generate_legacy_method_interface = functools.partial(generate_legacy_interface,

/usr/local/lib/python3.6/dist-packages/keras/layers/convolutional.py in __init__(self, filters, kernel_size, strides, padding, data_format, dilation_rate, activation, use_bias, kernel_initializer, bias_initializer, kernel_regularizer, bias_regularizer, activity_regularizer, kernel_constraint, bias_constraint, **kwargs)
    439                  bias_constraint=None,
    440                  **kwargs):
--> 441         super(Conv2D, self).__init__(
    442             rank=2,
    443             filters=filters,

TypeError: super(type, obj): obj must be an instance or subtype of type 

Did I miss something?

Loading in custom personal dataset & question about methods

Hello @lim-anggun, I would like to use your model on a collection of image frames I have. However, the initModel methods always assert the dataset name:

assert dataset_name in ['CDnet', 'SBI', 'UCSD'], 'dataset_name must be either one in ["CDnet", "SBI", "UCSD"]]'

So, I'm sure I have to edit the initModel methods for my own use.

It seems like there are different calls to the MyUpSampling2D method for each dataset name I choose, and I'm confused with what this portion of code really does or why it's needed (especially why there are differences in the num_pixels argument):

# Scale 2
x2_ups = {'tunnelExit_0_35fps':(0,1),'tramCrossroad_1fps':(1,0),'bridgeEntry':(1,1),
                      'busyBoulvard':(1,0),'fluidHighway':(0,1),'streetCornerAtNight':(1,1), 
                      'tramStation':(2,0),'winterStreet':(1,0),'twoPositionPTZCam':(1,0),
                      'peopleInShade':(1,1),'turbulence2':(1,1),'turbulence3':(1,0),
                      'skating':(1,1), 'wetSnow':(0,0)}

            for key, val in x2_ups.items():
                if self.scene == key and self.scene in ['skating', 'wetSnow']:
                    x2 = Cropping2D(cropping=((1, 1), val))(x2)
                    break
                elif self.scene==key:
                    x2 = MyUpSampling2D(size=(1, 1), num_pixels=val)(x2)
                    break

(FgSegNet_M_S_module.py, line 153-165)

I would like to know how I would have to modify this method for my dataset. I would appreciate it if you can help me with this, and explain what initModel does and why it has to change slightly according to the dataset.

Thank you, and great work on this!

Evaluation Mode

Hi @lim-anggun thank you for your research in this current field. I have a question. In this FgSegNet, I tried to train it on CDNet dataset and I have this weight model.

Soon I realized that the model names is same with video names. Is it means that this method generate different model files (one model one video) for comparison in CDNet dataset? Is it possible to make generalized model where all of video only use 1 model file?, and how about the performance? Suppose I have different video should I must have different model file?

Thank you.

How to Modify Training and Testing Image Size

Is it possible to use arbitrary image size for training and testing? In your test_prediction_demo,the input image size must be(240, 320, 3), how can I modify it to test an arbitrary image? Thanks a lot!

Training time

I read your paper,In your paper. it said that it takes  around 23.7 minutes to train for 50 epochs. Considering a video sequence which has 1700 frames with spatial size of 320x240,with a 200-frame traing ,but when I run your code ,I found it takes more time than 23.7 minutes to train in the CDnet2014_dataset.I don't know why ,I use the more version GPU than yours. you mean you take around 23.7 minutes to train for 50 epochs for a category's  one scene or all categories' all scene  ?

I have a question ,you said the video sequence with size of 320x240,you mean you resize all video sequences to 320x240?but in the code ,I didn't find you process the picture size.
Anthor question , after training .I use the training model weights to test the picture , the result is bad ,the predict picture is black . what is the reason? Thank you!