moka-guys / dnanexus_happy

DNAnexus app comparing a query VCF to a truth VCF to calculate performance metrics including sensitivity and precision using hap.py and vcfeval

vcfeval_hap.py

hap.py version

v0.3.9 (Docker: https://hub.docker.com/r/pkrusche/hap.py/)

What does this app do?

Compares a query VCF to a truth VCF to calculate performance metrics including sensitivity and precision using hap.py and vcfeval. It is equivalent to running the precisionFDA GA4GH benchmarking app in 'vcfeval-partialcredit' mode with other options left as default. More information available at the following links:

• https://precision.fda.gov/apps/app-F5YXbp80PBYFP059656gYxXQ
• https://github.com/ga4gh/benchmarking-tools/tree/master/doc/ref-impl

What are typical use cases for this app?

Validating an NGS workflow using the NA12878 (NIST Genome in a Bottle) benchmarking sample.

What data are required for this app to run?

Input files:

1. A query VCF (.vcf | .vcf.gz) - output from the workflow being validated
2. A truth VCF (.vcf | .vcf.gz)
3. A panel BED file (.bed) - region covered in query vcf
4. A high confidence region BED file (.bed) - high confidence region for truth set

Parameters:

1. Skip - default = false. If set to true will exit without performing any analysis
2. Output files prefix (required)
3. Output folder (optional)
4. Indication if additional stratification for NA12878 samples should be performed (default = False)
   • If truth set is NA12878, additional stratification of results can be performed and output in extended.csv file
   • HOWEVER the instance type will need to be upgraded to have at least 7GB of RAM, and the app will take significantly longer to run
5. Reference Genome build GRCh37 (default) or GRCh38

Note:

• The BED file names must not contain spaces or characters such as + and -

What does this app output?

This app outputs:

1. Summary csv file containing separate performance metrics for SNPs and Indels
2. Summary report HTML (generated using ga4gh rep.py https://github.com/ga4gh/benchmarking-tools/tree/master/reporting/basic)
3. Detailed results folder containing:
   • Extended csv file - Including results stratification and confidence intervals
   • VCF file - annotated vcf showing TP, FP and FN variants
   • runinfo JSON - detailed information about hap.py run
   • version log - version numbers of software used in app
   • metrics JSON - JSON file containing all computed metrics and tables

How does this app work?

• 'chr' is stripped from the chromosome field of the VCF and BED files (if hg19 format used)
• Indexed and zipped VCF files passed to hap.py:
  • Uses vcfeval comparison engine
  • If the sample is NA12878, additional stratification is performed using bed files found here: https://github.com/ga4gh/benchmarking-tools/tree/master/resources/stratification-bed-files
  • Summary HTML is generated

What are the limitations of this app

• Only works with inputs mapped to GRCh37 or GRCh38

This app was made by Viapath Genome Informatics

dnanexus_happy's Issues

Turn off the pass only option so that filtered variants are included in result set

--pass-only not required, because results are split into ALL and PASS anyway. With --pass-only on, filtered variants are completely ignored

No need to have separate truth_vcf/query_vcf variables, just reassign the ..._vcf_path environment variable

Output version numbers of apps and GIAB

Removed commented code

or mark it as for debugging purposes

test

test

Specify version numbers in exec depends

update instance type

dnanexus_happy/dxapp.json

Line 121 in 07f59c5

"instanceType": "mem1_ssd1_x2"

Output ga4gh html report

https://github.com/ga4gh/benchmarking-tools/tree/master/reporting/basic

Get rid of /work directory and symbolic home links

In original app provided to us all inputs are transfered to a /work directory under root, and then a symbolic link is set up to redirect home/in to work/in, and the HOME variable is set to the work folder:

#Move inputs into 'work' folder and create symbolic links mkdir /work cd /work export HOME=/work mv /home/dnanexus/in in ln -sf /work/in /home/dnanexus/in

Never quite understood why this was done like this, possibly to do with installation of hap.py. Now hap.py is run in docker remove these steps to make code less confusing.

precsionFDA version of hap.py ignores X chromosome

This causes problems particularly with our indel data which is largely on the X chromosome

From precisionFDA site:

It should be noted that chromosome X is not included in the confident regions of the GiaB truth data for HG002, so the comparisons have not evaluated that chromosome.

upgrade os

add:
"distribution" : "Ubuntu",
"release":"14.04",
to the runspec section of dxapp.json

Bug when assessing identical files

DNAnexus project 003_170713_GATKReproducibility

Query VCF: 3_WES3_Test2_48_136819_PN_WES_3_S2_R1_001.vcf.gz
Truth VCF: 1_WES3_Test2_48_136819_PN_WES_3_S2_R1_001.vcf.gz

High conf & panel BED files: Agilent WES

If this is run through PrecisionFDA vcfeval -> 100% sensitivity
If run through DNAnexus VCFeval_happy -> 2 FP
If run through DNAnexus VCFeval -> 22 FP

Running a diff on the vcfs -> identical except timestamp

Might be worth updating vcfeval_happy to catch up with PrecisionFDA version?

ROC plot scripts don't work if there are 0 indels. Replace the ga4gh html reports.

Maybe change this to output as .zip file so it's more user friendly for Windows users

dnanexus_happy/src/code.sh

Line 110 in a1d1989

tar zcvf /home/dnanexus/out/detailed_results/"$prefix".tar.gz "$prefix".*

Remove bedtools from exec depends (no longer required)

Remove java from exec depends (no longer required)

is the asset bundle version controlled anywhere?

Use -T option with bed in hap.py to remove 'UNK' records in output

clarify contents of output folder in readme

Output ROC curves

See: https://github.com/Illumina/hap.py/blob/d51d111e494b561b37c66299daf5a6c65a8d2ca9/doc/microbench.md
https://github.com/Illumina/hap.py/blob/d51d111e494b561b37c66299daf5a6c65a8d2ca9/src/R/rocplot.Rscript

Output confidence intervals

R code:

install.packages("epiR")
library("epiR")

#Provide numbers in order TP,FP,FN,TN   
dat <- as.table(matrix(c(11068,36,105,14141122), nrow = 2, byrow = TRUE))
rval <- epi.tests(dat, conf.level = 0.95)
print(rval); summary(rval)

Move app and assets to 001_ToolsReferenceData.

test2

2

Replace special characters in filenames

Limit number or characters in sample name for ROC curve

Long names result in a wide legend which squashes the plot.

Make build 38 compatible