Expert-level detection of pathologies from unannotated chest X-ray images via self-supervised learning
Expert-level detection of pathologies from unannotated chest X-ray images via self-supervised learning, Nat. Biomed. Eng (2022).
[Paper]
Ekin Tiu, Ellie Talius, Pujan Patel, Curtis P. Langlotz, Andrew Y. Ng, Pranav Rajpurkar
Tiu, E., Talius, E., Patel, P. et al. Expert-level detection of pathologies from unannotated chest X-ray images via self-supervised learning. Nat. Biomed. Eng (2022). https://doi.org/10.1038/s41551-022-00936-9
This repository contains code to train a self-supervised learning model on chest X-ray images that lack explicit annotations and evalute this model's performance on pathology-classification tasks.
Main Findings
- Automatically detecting pathologies in chest x-rays without explicit annotations: Our method learns directly from the combination of images and unstructured radiology reports, thereby avoiding time-consuming labeling efforts. Our deep learning method is capable of predicting multiple pathologies and differential diagnoses that it had not explicitly seen during training.
- Matching radiologist performance on different tasks on an external test set: Our method performed on par with human performance when evaluated on an external validation set (CheXpert) of chest x-ray images labeled for the presence of 14 different conditions by multiple radiologists.
- Outperforming approaches that train on explicitly labeled data on an external test set: Using no labels, we outperformed a fully supervised approach (100% of labels) on 3 out of the 8 selected pathologies on a dataset (PadChest) collected in a different country. We further demonstrated high performance (AUC > 0.9) on 14 findings and at least 0.700 on 53 findings out of 107 radiographic findings that the method had not seen during training.
To clone all files:
git clone https://github.com/rajpurkarlab/CheXzero.git
To install Python dependencies:
pip install -r requirements.txt
- Download images come from [MIMIC-CXR JPG] https://physionet.org/content/mimic-cxr-jpg/2.0.0/ and reports from MIMIC-CXR Database Note: in order to gain access to the data, you must be a credentialed user as defined on PhysioNet.
- Copy the dataset into the
data/directory. - Run
python run_preprocess.py - This should preprocess the chest x-ray images into a Hierarchical Data Format (HDF) format used for training stored at
data/cxr.h5and extract the impressions section as text from the corresponding chest x-ray radiology report stored atdata/mimic_impressions.csv.
The CheXpert dataset consists of chest radiographic examinations from Stanford Hospital, performed between October 2002 and July 2017 in both inpatient and outpatient centers. Population-level characteristics are unavailable for the CheXpert test dataset, as they are used for official evaluation on the CheXpert leaderboard.
The main data (CheXpert data) supporting the results of this study are available at https://aimi.stanford.edu/chexpert-chest-x-rays.
The CheXpert test dataset has recently been made public, and can be found by following the steps in the cheXpert-test-set-labels repository.
The PadChest dataset contains chest X-rays that were interpreted by 18 radiologists at the Hospital Universitario de San Juan, Alicante, Spain, from January 2009 to December 2017. The dataset contains 109,931 image studies and 168,861 images. PadChest also contains 206,222 study reports.
The PadChest is publicly available at https://bimcv.cipf.es/bimcv-projects/padchest. Those who would like to use PadChest for experimentation should request access to PadChest at the link.
Model checkpoints of CheXzero pre-trained on MIMIC-CXR are publicly available at the following link. Download files and save them in the ./checkpoints/chexzero_weights directory.
Run the following command to perform CheXzero pretraining.
python run_train.py --cxr_filepath "./data/cxr.h5" --txt_filepath "data/mimic_impressions.csv"--cxr_filepathDirectory to load chest x-ray image data from.--txt_filepathDirectory to load radiology report impressions text from.
Use -h flag to see all optional arguments.
See the following notebook for an example of how to use CheXzero to perform zero-shot inference on a chest x-ray dataset. The example shows how to output predictions from the model ensemble and evaluate performance of the model if ground truth labels are available.
import zero_shot
# computes predictions for a set of images stored as a np array of probabilities for each pathology
predictions, y_pred_avg = zero_shot.ensemble_models(
model_paths=model_paths,
cxr_filepath=cxr_filepath,
cxr_labels=cxr_labels,
cxr_pair_template=cxr_pair_template,
cache_dir=cache_dir,
)model_paths: List[str]: List of paths to all checkpoints to be used in the ensemble. To run on a single model, input a list containing a single path.cxr_filepath: str: Path to images.h5filecxr_labels: List[str]: List of pathologies to query in each imagecxr_pair_templates: Tuple[str, str]: constrasting templates used to query model (see Figure 1 in article for visual explanation).cache_dir: str: Directory to cache predictions of each checkpoint, use to avoid recomputing predictions.
In order to use CheXzero for zero-shot inference, ensure the following requirements are met:
- All input
imagesmust be stored in a single.h5(Hierarchical Data Format). See theimg_to_h5function in preprocess_padchest.py for an example of how to convert a list of paths to.pngfiles into a valid.h5file. - The ground truth
labelsmust be in a.csvdataframe where rows represent each image sample, and each column represents the binary labels for a particular pathology on each sample. - Ensure all model checkpoints are stored in
checkpoints/chexzero_weights/, or themodel_dirthat is specified in the notebook.
Given a numpy array of predictions (obtained from zero-shot inference), and a numpy array of ground truth labels, one can evaluate the performance of the model using the following code:
import zero_shot
import eval
# loads in ground truth labels into memory
test_pred = y_pred_avg
test_true = zero_shot.make_true_labels(cxr_true_labels_path=cxr_true_labels_path, cxr_labels=cxr_labels)
# evaluate model, no bootstrap
cxr_results: pd.DataFrame = eval.evaluate(test_pred, test_true, cxr_labels) # eval on full test datset
# boostrap evaluations for 95% confidence intervals
bootstrap_results: Tuple[pd.DataFrame, pd.DataFrame] = eval.bootstrap(test_pred, test_true, cxr_labels) # (df of results for each bootstrap, df of CI)
# print results with confidence intervals
print(bootstrap_results[1])The results are represented as a pd.DataFrame which can be saved as a .csv.
In order to replicate the results in the paper, zero-shot inference and evaluation can be performed on the now publicly available CheXpert test dataset.
- Download labels at cheXpert-test-set-labels and image files from Stanford AIMI and save in the
./datadirectory inCheXzero/. The test dataset images should have the following directory structure:
data/
├─ CheXpert/
│ ├─ test/
│ │ ├─ patient64741/
│ │ │ ├─ study1/
│ │ │ │ ├─ view1_frontal.jpg
│ │ ├─ .../
- Run
run_preprocess.pyscript with the following arguments:
python run_preprocess.py --dataset_type "chexpert-test" --cxr_out_path "./data/chexpert_test.h5" --chest_x_ray_path "./data/CheXpert/test/"This should save a .h5 version of the test dataset images which can be used for evaluation.
- Open sample zero-shot notebook and run all cells. If the directory structure is set up correctly, then all cells should run without errors.
Please open new issue threads specifying the issue with the codebase or report issues directly to [email protected].
Tiu, E., Talius, E., Patel, P. et al. Expert-level detection of pathologies from unannotated chest X-ray images via self-supervised learning. Nat. Biomed. Eng (2022). https://doi.org/10.1038/s41551-022-00936-9The source code for the site is licensed under the MIT license, which you can find in the LICENSE file. Also see NOTICE.md for attributions to third-party sources.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent