sarah145 / scib

Benchmarking analysis of data integration tools

Benchmarking atlas-level data integration in single-cell genomics

This repository contains code and analysis for the benchmarking study for data integration tools. In this study, we benchmark 10 methods (see here) with 4 combinations of preprocessing steps leading to 40 methods combinations on 60 batches of gene expression and chromatin accessibility data.

We created the python package scIB to streamline the integration process and to integrate it into a scanpy workflow. Additionally, we created an environment to allow easy integration of R integration methods into the scanpy workflow.

Furthermore, the package allows for evaluation of integration quality of different datasets if cell type annotations are present using our novel metrics.

The scIB python package is in the folder scIB. It can be installed using pip install -e . run in the root directory. R helper functions for R integration methods can be found in the R directory. The scripts folder contains scripts for preparing the data, running the methods, postprocessing and calculation of the metrics. The notebooks folder contains jupyter notebooks for testing and demonstrating functions of the scIB package as well as notebooks for preprocessing of the data.

Installation

To reproduce the results from this study, three different conda environments are needed. There are different environments for the python integration methods, the R integration methods and the conversion of R data types to anndata objects.

For the installation of conda, follow these instructions or use your system's package manager. The environments have only been tested on linux operating systems although it should be possible to run the pipeline using Mac OS.

To create the conda environments use the .yml files in the envs directory. To install the envs, use

conda env create -f FILENAME.yml

To set the correct paths so that R the correct R libraries can be found, copy env_vars_activate.sh to etc/conda/activate.d/ and env_vars_deactivate.sh to etc/conda/deactivate.d/ to every environment. In the scIB-R-integration environment, R packages need to be installed manually. Activate the environment and install the packages scran, Seurat and Conos in R. Conos needs to be installed using R devtools. See here.

Running the integration methods

This package allows to run a multitude of single cell data integration methods in both R and python. We use Snakemake to run the pipeline. The parameters of the run are configured using the config.yaml file. See the DATA_SCENARIOS section to change the data used for integration. The script expects one .h5ad file containing all batches per data scenario.

To load the config file run snakemake --configfile config.yaml. Define the number of CPU threads you want to use with snakemake --cores N_CORES. To produce an overview of tasks that will be run, use snakemake -n. To run the pipeline, simply run snakemake.

Structure of the scIB package

The package contains several modules for the different steps of the integration and benchmarking pipeline. Functions for the integration methods are in scIB.integrate. The methods are called using scIB.integration.runMETHOD(adata, BATCH). scIB.preprocessing contains methods for preprocessing of the data such as normalisation, scaling or highly variable gene selection per batch. The metrics are located at scIB.metrics. To run multiple metrics in one run, use the scIB.metrics.metrics() function.

Metrics

For a detailed description of the metrics implemented in this package, please see the manuscript.

Batch removal metrics include:

• Principal component regression (pcr_comparison())
• Batch ASW (silhouette())
• K-nearest neighbour batch effect (kBET())
• Graph connectivity (graph_connectivity())
• Graph iLISI (lisi_graph())

Biological conservation metrics include:

• Normalised mutual information (nmi())
• Adjusted Rand Index (ari())
• Cell type ASW (silhouette_batch())
• Isolated label score F1 (isolated_labels())
• Isolated label score ASW (isolated_labels())
• Cell cycle conservation (cell_cycle())
• Highly variable gene conservation (hvg_overlap())
• Trajectory conservation (trajectory_conservation())
• Graph cLISI (lisi_graph())

Tools

Tools to be compared include:

• Seurat v3
• TrVae
• scVI
• CONOS tutorial
• MNN
• Scanorama
• LIGER
• BBKNN
• Harmony
• Combat paper