tesparrow / crack_detection_cnn_masonry

This GitHub Repository was produced to share material relevant to the Journal paper "Automatic crack classification and segmentation on masonry surfaces using convolutional neural networks and transfer learning" by D. Dais, İ. E. Bal, E. Smyrou, and V. Sarhosis published in "Automation in Construction".

Crack detection for masonry surfaces

This GitHub Repository was produced to share material relevant to the Journal paper Automatic crack classification and segmentation on masonry surfaces using convolutional neural networks and transfer learning by D. Dais, İ. E. Bal, E. Smyrou, and V. Sarhosis published in Automation in Construction.

While in the paper models were trained both for patch classification and pixel segmentation, in the herein repository only the codes for crack segmentation are made available for the time being. Extra material regarding the patch classification will be added in the future.

A sub-set of the masonry dataset used in the paper is made available as well in the dataset folder.

Ιndicative examples of photos from our masonry dataset can be found in the folder dataset_sample.

The paper can be downloaded from the following links:

• https://doi.org/10.1016/j.autcon.2021.103606
• https://www.researchgate.net/publication/349645935_Automatic_crack_classification_and_segmentation_on_masonry_surfaces_using_convolutional_neural_networks_and_transfer_learning

In case you use or find interesting our work please cite the following Journal publication:

D. Dais, İ.E. Bal, E. Smyrou, V. Sarhosis, Automatic crack classification and segmentation on masonry surfaces using convolutional neural networks and transfer learning, Automation in Construction. 125 (2021), pp. 103606. https://doi.org/10.1016/j.autcon.2021.103606.

@article{Dais2021,  
  author = {Dais, Dimitris and Bal, İhsan Engin and Smyrou, Eleni and Sarhosis, Vasilis},  
  doi = {10.1016/j.autcon.2021.103606},  
  journal = {Automation in Construction},  
  pages = {103606},  
  title = {{Automatic crack classification and segmentation on masonry surfaces using convolutional neural networks and transfer learning}},  
  url = {https://linkinghub.elsevier.com/retrieve/pii/S0926580521000571},  
  volume = {125},  
  year = {2021}  
}  

Author and Moderator of the Repository: Dimitris Dais

For further information please follow me in the below links
LinkedIn: https://www.linkedin.com/in/dimitris-dais/
Email: [email protected]
ResearchGate: https://www.researchgate.net/profile/Dimitris_Dais2
Research Group Page: https://www.linkedin.com/company/earthquake-resistant-structures-promising-groningen
YouTube Channel: https://www.youtube.com/channel/UCuSdAarhISVQzV2GhxaErsg

Your feedback is welcome. Feel free to reach out to explore any options for collaboration.

Table of Contents

Crack detection for masonry surfaces
Table of Contents
Publication - Brief Preview
| --- Image patch classification for crack detection
| --- Crack segmentation on pixel level
How to download and run the code
Configuration file
Build data
Train
Evaluate
Acknowledgements
References

Publication - Brief Preview

The aim of this study is to examine deep learning techniques for crack detection on images from masonry walls. A dataset with photos from masonry structures is produced containing complex backgrounds and various crack types and sizes. Different deep learning networks are considered and by leveraging the effect of transfer learning crack detection on masonry surfaces is performed on patch level with 95.3% accuracy and on pixel level with 79.6% F1 score. This is the first implementation of deep learning for pixel-level crack segmentation on masonry surfaces.

Image patch classification for crack detection

Different state of the art CNNs pretrained on ImageNet are examined herein for their efficacy to classify images from masonry surfaces on patch level as crack or non-crack.

Crack segmentation on pixel level

U-net, a deep Fully Convolutional Network (FCN), and Feature Pyramid Networks (FPN), a generic pyramid representation, are considered herein and combined with different pretrained CNNs performing as the backbone of the encoder part of the network. Moreover, other networks found in the literature and performed well in crack segmentation are examined as well.

Schematic representation of the encoder-decoder architecture of Fully Convolutional Networks..

Schematic representation of Feature Pyramid Network.

Illustration of the architecture of U-net as implemented in the herein study.

The original image, the ground truth and the prediction with U-net-MobileNet.

How to download and run the code

Simply download the code and copy it your desired folder path.
In order to use the code you need to define the Configuration file (config_class.py) suitably. There are three modes:

• Build data
• Train
• Evaluate

To use the Build data mode you need to run the build_data.py file while the Train and Evaluate modes are implemented with the train_evaluate.py file. See the documentation below for extra info.

Configuration file

In the Configuration file (config_class.py) you can define the parameters that will be used when running the code.
See below some examples of different parameters that can be defined:

        # Define the mode that will be used when running the code
        self.mode = 'evaluate' # 'train', 'evaluate' or 'build_data'
        # Info that will be used as prefix of any output files
        self.info = 'crack_detection'
        # Dimensions of the images that will be fed to the network
        self.IMAGE_DIMS = (224 , 224 , 3)
        # Batch size
        self.BS = 4
        # Number of epochs to train the model
        self.epochs = 10
        # Initial learning rate
        self.INIT_LR = 0.0005

When the Train mode is used, define which network to be used. The options are:

• U-net
• Deeplabv3
• DeepCrack
• Segmentation Models: Different models architectures for segmentation combined with pretrained CNNs performing as the backbone of the encoder part of the network

        # 1) 'Unet'
        # 2) 'Deeplabv3': https://github.com/tensorflow/models/tree/master/research/deeplab
        # 3) 'DeepCrack': https://github.com/hanshenChen/crack-detection
        # 4) Different segmentation models as defined by https://github.com/qubvel/segmentation_models
        #    The name of the model should be structured as follows: sm_ModelName_Backbone
        #    e.g. 'sm_Unet_mobilenet': Unet will be used as model and mobilenetv2 as backbone
        #    The acceptable ModelNames are Unet and FPN
        #    For the acceptable Backbone networks refer to the documentation of the GitHub repository
        #    https://github.com/qubvel/segmentation_models#models-and-backbones
        self.args['model'] = 'Unet' # 'Deeplabv3', 'Unet', 'DeepCrack', 'sm_Unet_mobilenet'

In order to use the Deeplabv3 network copy model.py to the networks folder.

In order to use the DeepCrack network copy edeepcrack_cls.py and indices_pooling.py to the networks folder.

In order to use the Segmentation Models use the pip install method as shown in in the corresponding GitHub repository.

Networks folder.

Is it possible to select among the following Loss Functions:

        # Define Loss Function
        # 'Focal_Loss'
        # 'WCE': Weighted Cross-Entropy
        # 'BCE': Binary Cross-Entropy
        # 'F1_score_Loss'
        # 'F1_score_Loss_dil': Background pixels predicted as cracks (FP) are considered as
        #                      TP if they are a few pixels apart from the annotated cracks
        #                      Refer to the Journal paper for extra clarification
        self.args['loss'] = 'WCE'

Build data

Under the folder dataset there are two sub-folders with the images and the corresponding masks.

Execute the build_data.py to split the dataset into train and validation set and create HDF5 files that can be easily accessed when in Train and Evaluate modes. The output folder will be created and the output files will be stored in the hdf5 sub-folder.

It is noted that the output folder will be created automatically after executing the code.

Use the Build data mode by suitably setting the config_class.py.

        # Define the mode that will be used when running the code
        self.mode = 'build_data' # 'train', 'evaluate' or 'build_data'

In the build_data.py define appropriately the folder["main"] to correspond to the folder where you stored the repository on your local machine.

# The path where the repository is stored
folder['main'] = folder['initial'] + 'crack_detection_CNN_masonry/'

# if folder['main'] == '', then the current working directory will be used
if folder['main'] == '':
    folder['main'] = os.getcwd()

Train

Use the Train mode by suitably setting the config_class.py.

        # Define the mode that will be used when running the code
        self.mode = 'train' # 'train', 'evaluate' or 'build_data'

In the train_evaluate.py define appropriately the folder["main"] to correspond to the folder where you stored the repository on your local machine. Then execute the train_evaluate.py to train the selected network on the provided dataset.

# The path where the repository is stored
folder['main'] = folder['initial'] + 'crack_detection_CNN_masonry/'

# if folder['main'] == '', then the current working directory will be used
if folder['main'] == '':
    folder['main'] = os.getcwd()

In order to be able to run different trials without overwriting any existing files, any output files of each trial will have a different suffix. This suffix will be obtained from the file named counter.txt. Create a txt file and type any number that will be used as a counter; during each trial you will be asked whether this value needs to be incremented by 1. If the counter file has not been created, then the os.getpid() will be used as counter.

When training the model the following files will be created:

• The Loss/Metrics will be plotted during training to a png file in the folder output
• The model will be serialized and stored in JSON format in the sub-folder model_json
• The results (i.e. metrics, loss) will be serialized and stored in JSON format in the folder output
• The architecture of the network will be stored in the folder output
• The summary of the model will be stored to txt file in the folder output
• The results (i.e. metrics, loss) will be stored to CSV format in the folder output
• The weights of the trained model will be stored for different epochs in the sub-folder weights. See TrainingMonitor for extra details.

Output file created when Train mode is used.

The counter file.

The Loss/Metrics will be plotted during training.

The weights of the trained model will be stored for different epochs.

Evaluate

Use the Evaluate mode by suitably setting the config_class.py.

        # Define the mode that will be used when running the code
        self.mode = 'evaluate' # 'train', 'evaluate' or 'build_data'

In the train_evaluate.py define appropriately the folder["main"] to correspond to the folder where you stored the repository on your local machine. If folder["main"] was set during the Train mode, it does not need to be changed again.

# The path where the repository is stored
folder['main'] = folder['initial'] + 'crack_detection_CNN_masonry/'

# if folder['main'] == '', then the current working directory will be used
if folder['main'] == '':
    folder['main'] = os.getcwd()

Define the file with the pretrained weights or the model with weights that will be used to evaluate the model by setting correctly self.args['counter'] and self.args['pretrained_filename'].

            # Define the counter suitably in order to read the correct JSON file etc.
            self.args['counter'] = 1
            # Define the file with the pretrained weights or the model with weights that will be used to evaluate model
            # e.g. 'crack detection_1_epoch_7_F1_score_dil_0.762.h5'
            self.args['pretrained_filename'] = 'crack_detection_1_epoch_8_F1_score_dil_0.681.h5'

The filename of the pretrained weights provides information regarding several parameters. For exampe from 'crack_detection_1_epoch_8_F1_score_dil_0.681.h5' it can be inferred that:

self.info = 'crack_detection'
self.args['counter'] = 1
The epoch that the best value of the monitored metric was obtained: 8
Monitored metric: F1_score_dil
Best value of the monitored metric: 0.681

For extra clarification, refer to the documentation of ModelCheckpoint

Subsequently, execute the train_evaluate.py to use the pre-trained network to perform predictions and visualize them.

Predictions with the pretrained model.

It is noted that the metrics displayed in the figure above were obtained with training only for 10 epochs and a sub-set of the whole masonry dataset was used.

When using the default configuration, the validation set will be used to evaluate the trained model. In case a different test needs to be used, then define suitably the self.args['EVAL_HDF5'] in the config_class.py.

        # In case you need to test the model on a set other than the validation set,
        # define the EVAL_HDF5 suitably
        self.args['EVAL_HDF5'] = self.args['hdf5'] + temp + 'val.hdf5'

Acknowledgements

I would like to thank my friend and Deep Learning expert Nekartios Lianos for his insightful guidance along my first steps in the world of CV and DL.

I couldn not skip referring to the contribution of Adrian Rosebrock; the material he shares on his blog pyimagesearch and his books ensure that the transition to the magical world of CV and DL will be as smooth as it can be.

References

The following codes are based on material provided by Adrian Rosebrock shared on his blog (https://www.pyimagesearch.com/) and his books:

build_data.py
hdf5datasetgenerator_mask.py
hdf5datasetwriter_mask.py
epochcheckpoint.py
trainingmonitor.py

• Adrian Rosebrock, Deep Learning for Computer Vision with Python - Practitioner Bundle, PyImageSearch, https://www.pyimagesearch.com/deep-learning-computer-vision-python-book/, accessed on 24 February 2021
• Adrian Rosebrock, Keras: Starting, stopping, and resuming training, PyImageSearch, https://www.pyimagesearch.com/2019/09/23/keras-starting-stopping-and-resuming-training/, accessed on 24 February 2021
• Adrian Rosebrock, How to use Keras fit and fit_generator (a hands-on tutorial), PyImageSearch, https://www.pyimagesearch.com/2018/12/24/how-to-use-keras-fit-and-fit_generator-a-hands-on-tutorial/, accessed on 24 February 2021

The Segmentation Models with pre-trained CNNs are implemented based on the work of Pavel Yakubovskiy and his GitHub Repository https://github.com/qubvel/segmentation_models

DeepCrack is implemented as provided by the corresponding GitHub Repository

Deeplabv3 is implemented as provided by the corresponding GitHub Repository

Unet is based on the implementation found in the link below:
https://www.depends-on-the-definition.com/unet-keras-segmenting-images/

The Weighted Cross-Entropy (WCE) Loss is based on the implementation found in the link below:
https://jeune-research.tistory.com/entry/Loss-Functions-for-Image-Segmentation-Distribution-Based-Losses

The Focal Loss is based on the implementation found in the link below:
https://github.com/umbertogriffo/focal-loss-keras