hfawaz / miccai18

Evaluating surgical skills from kinematic data using convolutional neural networks

Evaluating surgical skills from kinematic data using convolutional neural networks

This is the companion repository for our paper also available on ArXiv titled "Evaluating surgical skills from kinematic data using convolutional neural networks". This paper has been accepted at the International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI) 2018.

Architecture

The source code

The software is developed using Python 3.6, it takes as input the kinematic data and predicts the subject's skill level. We trained the model on an NVIDIA GPU GTX 1080 (this is only necessary to speed up the calculations). You will need the JIGSAWS dataset to re-run the experiments of the paper. The source code is composed of one python file with the code needed to re-run the experiments. The hyper-parameters published in the paper are present in the source code. The empty folders (in the repository) are necessary so the code could run without the "folder not found error". The content of JIGSAWS.zip (once downloaded) should be placed in the empty folder "JIGSAWS". If you are looking for more information on how to classify time series with deep learning, also have a look to our paper "Deep learning for time series classification: a review". The code of the models compared in this review is available here.

Prerequisites

• Keras 2.12.2
• Tensorflow 1.4.1
• Numpy 1.13.3
• Scikit-learn 0.19.1
• Pandas 0.21.1
• Matplotlib 2.0.0
• Imageio 2.2.0

We used the versions listed above, but you should be able to run the code with much recent versions. Different versions of the above-listed packages can be found in the archives of the packages' corresponding websites.

Visualizing the movements' contribution to the classification

The proposed method uses the Class Activation Map to localize the regions of the surgical task and their corresponding contribution to a certain classification.

A video illustrating the method can be seen here.

The functions that are used to visualize the trajectories are present in the source code.

Figure 2 in the paper, illustrates the trajectory of subject (Novice) H's movements for the left master manipulator.

The colors are obtained using the Class Activation Map values of a given class and then projected on the (x,y,z) coordinates of the master left hand.

You can find here the visualizations for all the subjects and all their trials for the Suturing task for each one of the five cross-validation folds.

Frame	Trajectory explained

Reference

If you re-use this work, please cite:

@InProceedings{IsmailFawaz2018evaluating,
  Title                    = {Evaluating surgical skills from kinematic data using convolutional neural networks},
  Author                   = {Ismail Fawaz, Hassan and Forestier, Germain and Weber, Jonathan and Idoumghar, Lhassane and Muller, Pierre-Alain},
  booktitle                = {International Conference On Medical Image Computing and Computer Assisted Intervention (MICCAI)},
  Year                     = {2018},
  pages = {214--221}
}

Ipynb version of "Evaluating surgical skills from kinematic data using convolutional neural networks"

This section is added by @Hosseinhashemiir.

This modified project of @https://github.com/hfawaz/miccai18 cause I don't have a good hardware (GPU) for running this code by directly, I used "Google's Colaboratory".

you don't need any setup for the run, just refer this Repo to Colab website http://colab.research.google.com and find the miccai18.ipynb.

notice that processing need around 2 hours.

miccai18's Issues

keras version

Hi! Great work.
Did you use keras 2.2.2 or keras 2.1.2? I couldn't find the version 2.12.2 so I suppose its something between the first two versions.

Thanks in advance,
Dayvid Castro.

invalid syntax

Dear Weber and Forestier,

Your article is so interesting.
I'm trying to run the code's, but there are some problems with the syntax error.
like:

File "miccai2018.py", line 256
    class_activation_map(time_series_original,original_binary_class,model,path_to_images+surgery_name+'__'+str(f'{i:06}')+'.png',max_frame=i+1,slave_manipulator=slave_manipulator,angle=15*6,elev=75+180)
                                                                                                                       ^
SyntaxError: invalid syntax

File "miccai2018.py", line 261
    os.system('ffmpeg -f image2 -framerate 30 -i '+path_to_images+surgery_name+'__%06d.png -s 640x480 '+ path_to_video+surgery_name+slave_manipulator+'.mp4')
     ^
IndentationError: expected an indented block

Do you know what should be the problem?
I looking forward to hearing from you.
All the best

facing issues installing dependencies

in the miccai.ipynb notebook, facing problem installing them

!pip install Keras==2.1.2
!pip install Tensorflow==1.4.1
!pip install Numpy==1.13.3
!pip install Scikit-learn==0.19.1
!pip install Pandas==0.21.1
!pip install Matplotlib==2.0.0
!pip install Imageio==2.2.0
!pip install imageio
clear_output()