jrzech / reproduce-chexnet

Reproduce CheXNet

reproduce-chexnet

Provides Python code to reproduce model training, predictions, and heatmaps from the CheXNet paper that predicted 14 common diagnoses using convolutional neural networks in over 100,000 NIH chest x-rays.

Getting Started:

Click on the launch binder button at the top of this README to launch a remote instance in your browser using binder. This requires no local configuration, but it can take a couple minutes to launch. Open Explore_Predictions.ipynb, run all cells, and follow the instructions provided to review a selection of included chest x-rays from NIH.

To configure your own local instance (assumes Anaconda is installed; can be run on paperspace GPU instance if you do not have a GPU):

cd reproduce-chexnet
conda env create -f environment.yml
source activate reproduce-chexnet
python -m ipykernel install --user --name reproduce-chexnet --display-name "Python (reproduce-chexnet)"

Updates (2022)

Changes in libraries available through conda channels and non-compability of older versions of pytorch / torchvision with newer CUDA drivers caused the original build to break.

This library was updated in late 2022 so anyone interested could continue to use it. (1) environment.yml was updated and slight edits to code were made to ensure compatibility with newer version of pytorch/torchvision (2) NIH CXR labels were updated to latest version. Given changes in torchvision model naming conventions and updated labels, a new model was retrained; reported AUC numbers are based on this retrained model.

Given expected variability in predictions of retrained deep learning models, predictions vary from the model originally posted in 2018. The original 2018 model and predictions are shared in a /pretrained-old folder, but will require you to create a compatible environment with the older pytorch 0.4.0 and torchvision 0.2.0 to use them interactively.

Replicated results:

This reproduction achieved diagnosis-level AUC as given below compared to original paper:

	retrained auc	chexnet auc
label
Atelectasis	0.8180	0.8094
Cardiomegaly	0.9090	0.9248
Consolidation	0.8002	0.7901
Edema	0.8945	0.8878
Effusion	0.8827	0.8638
Emphysema	0.9316	0.9371
Fibrosis	0.8251	0.8047
Hernia	0.9175	0.9164
Infiltration	0.7156	0.7345
Mass	0.8377	0.8676
Nodule	0.7756	0.7802
Pleural_Thickening	0.7889	0.8062
Pneumonia	0.7617	0.7680
Pneumothorax	0.8776	0.8887

Results available in pretrained folder:

• aucs.csv: test AUCs of retrained model vs original ChexNet reported results
• checkpoint: saved model checkpoint
• log_train: log of train and val loss by epoch
• preds.csv: individual probabilities for each finding in each test set image predicted by retrained model

NIH Dataset

To explore the full dataset, download images from NIH (large, ~40gb compressed), extract all tar.gz files to a single folder, and provide path as needed in code. You can use batch download script provided by NIH researchers included in this repo:

python nih_batch_download_zips.py

Train your own model!

Please note: a GPU is required to train the model. You will encounter errors if you do not have a GPU available and compatible CUDA installed and you attempt to retrain. With a GPU, you can retrain the model with retrain.py. Make sure you download the full NIH dataset before trying this. If you run out of GPU memory, reduce BATCH_SIZE from its default setting of 16.

Please ensure your CUDA driver is compatible with the CUDA toolkit (v11.3) installed by default.

If you do not have a GPU, but wish to retrain the model yourself to verify performance, you can replicate the model with paperspace, Amazon EC2, Google Colaboratory, or other online cloud GPU services. If you're starting from scratch, paperspace is easy to get started with.

Note on training

I use SGD+momentum rather than the Adam optimizer as described in the original paper. I achieved better results with SGD+momentum, as has been reported in other work.

Note on data

A sample of 621 test NIH chest x-rays enriched for positive pathology is included with the repo to faciliate immediate use and exploration in the Explore Predictions.ipynb notebook. The full NIH dataset is required for model retraining.

Use and citation

My goal in releasing this code is to increase transparency and replicability of deep learning models in radiology. I encourage you to use this code to start your own projects. If you do, please cite the repo:

  author = {Zech, J.},
  title = {reproduce-chexnet},
  year = {2018},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/jrzech/reproduce-chexnet}}
}

Acknowledgements

With deep gratitude to researchers and developers at PyTorch, NIH, Stanford, and Project Jupyter, on whose generous work this project relies. With special thanks to Sasank Chilamkurthy, whose demonstration code was incorporated into this project. PyTorch is an incredible contribution to the research community.

reproduce-chexnet's Issues

What are the values in label_baseline_probs, how to come up with that

in calc_cam function in visualize_prediction.py
label_baseline_probs={ 'Atelectasis':0.103, 'Cardiomegaly':0.025, 'Effusion':0.119, 'Infiltration':0.177, 'Mass':0.051, 'Nodule':0.056, 'Pneumonia':0.012, 'Pneumothorax':0.047, 'Consolidation':0.042, 'Edema':0.021, 'Emphysema':0.022, 'Fibrosis':0.015, 'Pleural_Thickening':0.03, 'Hernia':0.002 }

Excuse me, could you tell me how to slove the question:can't calculate auc for Atelectasis:thinks

"I don't know how to run it specifically, but when I executed retrain.py, it outputted somecan't calculate auc for Atelectasis
can't calculate auc for Cardiomegaly
can't calculate auc for Effusion
can't calculate auc for Infiltration
can't calculate auc for Mass
can't calculate auc for Nodule
can't calculate auc for Pneumonia
can't calculate auc for Pneumothorax
can't calculate auc for Consolidation
can't calculate auc for Edema
can't calculate auc for Emphysema
can't calculate auc for Fibrosis
can't calculate auc for Pleural_Thickening
can't calculate auc for Hernia. Could you please tell me how to solve these problems? Also, the output result was created with a new optimizer with LR 1.0000000000000003e-09 at the 11th round."

Can be used on images outside dataset?

First of all great work.
Secondly:
How do i use it on images that are outside the dataset, i wanna use the pretrained model checkpoint and put 1 image at a time to print it's preds. From my testing if i change the folder and put a random image innit it tells me some other image is not found? is the preds just printing from whatever is stored?

some labels inconsistent with NIHCC/Data_Entry_2017.csv

Hi,

Thanks for your great work.

I am confused why there are some inconsistency between labels from NIHCC Data_Entry_2017.csv and from your repo (nih_labels.csv). For example:

00000150_002.png
label in Data_Entry_2017.csv: Atelectasis|Infiltration|Pneumonia
label in nih_labels.csv: Atelectasis|Infiltration

00000193_019.png
label in Data_Entry_2017: Pneumonia
label in nih_labels.csv: No Finding(normal)

('00000830_000.png', '00000902_001.png',... etc.)

Errors for '_DenseLayer' object has no attribute 'memory_efficient'

When running the code on google colab, this error always happen when executing:
raw_cam = calc_cam(inputs, LABEL, model)
I have followed the steps of the following to setup the environment, but the error is consistent. Thank you so much for any help!

!wget -c https://repo.continuum.io/archive/Anaconda3-5.1.0-Linux-x86_64.sh
!chmod +x Anaconda3-5.1.0-Linux-x86_64.sh
!bash ./Anaconda3-5.1.0-Linux-x86_64.sh -b -f -p /usr/local
!conda install -y --prefix /usr/local -c <<>>>

import sys
sys.path.append('/usr/local/lib/python3.6/site-packages/‘). — deleted, because there is error

Then, set environment

!cd reproduce-chexnet (already in the directory by mounting)
!conda env create -f environment.yml. (only needed the first time, for second time, only need !source postBuild, and !source activate reproduce-chexne )
!source postBuild
!source activate reproduce-chexnet

Horizontal Flipping in Training?

I briefly looked through the paper again and didn't see any explicit steps on their augmentation workflow, but I would be curious on how performance changes with the horizontal flipping removed.

reproduce-chexnet/model.py

Line 223 in 79ffc2b

transforms.RandomHorizontalFlip(),

In particular with cardiomegaly (big heart), the heart is always expected to be on the left-side and so flipping could cause a lower score there.

“no improvement in 3 epochs, break” happened at epoch 20

I used 8 gpus and default settings, but it was broke at epoch 20 with acu around 0.7, do you have any ideas?

Official dataset split

Hi,
I am curious about you dataset split. Is it same as the split provided by NIH? Thanks

Deployed Using Streamlit

Hi John!

I have created a Streamlit app using your model checkpoint.

It can be found using the link below:
https://report-cxr-with-ai.streamlit.app/

The repository for this application can be found using the link below:
https://github.com/ashishbamania/ChestXNet-Streamlit

Thanks for your work! Cheers!

Input/Output error 5 at colab when reading NIH x-ray data in google drive

Weighted Loss

The original paper implements a weighted binary cross entropy loss. I don't see how you could have reproduced the results from the paper without it, as some tasks have nearly a 2:112 class imbalance. Am I missing something?

Link in the description for Fast AI course is broken.

Please change it to https://course.fast.ai/start_aws.html

Excuse me, could you tell me how to slove the question "Reloaded modules: cxr_dataset, eval_model"? Thanks.

Reason for using imagenet mean and std for normalization

In model.py, it is mentioned that imagenet mean, std are used for normalization. However, the mean and std of NIH dataset is much larger than that of imagenet. Can you please share the reason for using imagenet mean and std for normalization? Maybe I am missing something.

imbalanced dataset

Hi John,

Great work. I am also trying to reproduce the results from the chexnet paper for learning purposes. I just have 2 quick questions:

1. Did you modify the loss function to account for the imbalanced dataset? The original authors used the unweighted version of binary cross entropy, which seems weird to me because negative cases outweigh the positive ones.

2. How long did you train the model for?

Thank you!