ohdsi / empiricalcalibration

An R package for performing empirical calibration of observational study estimates

EmpiricalCalibration

EmpiricalCalibration is part of HADES.

Introduction

This R package contains routines for performing empirical calibration of observational study estimates. By using a set of negative control hypotheses we can estimate the empirical null distribution of a particular observational study setup. This empirical null distribution can be used to compute a calibrated p-value, which reflects the probability of observing an estimated effect size when the null hypothesis is true taking both random and systematic error into account, as described in the paper Interpreting observational studies: why empirical calibration is needed to correct p-values.

Also supported is empirical calibration of confidence intervals, based on the results for a set of negative and positive controls, as described in the paper Empirical confidence interval calibration for population-level effect estimation studies in observational healthcare data.

Features

• Estimate the empirical null distribution given the effect estimates of a set of negative controls.
• Estimate the calibrated p-value of a given hypothesis given the estimated empirical null distribution.
• Estimate a systematic error distribution given the effect estimates for a set of negative and positive controls.
• Estimate the calibrated confidence interval for a given estimate given the systematic error distribution.
• Estimate a calibrated log likelihood ratio, for use in maximum sequential probability ratio testing (MaxSPRT).
• Produce various plots for evaluating the empirical calibration.
• Contains the data sets from the papers for illustration.

Screenshots and examples

data(sccs) #Load one of the included data sets
negatives <- sccs[sccs$groundTruth == 0,] #Select the negative controls
null <- fitNull(logRr = negatives$logRr, seLogRr = negatives$seLogRr) #Fit the null distribution
positive <- sccs[sccs$groundTruth == 1,]  #Select the positive control

#Create the plot above:
plotCalibrationEffect(logRrNegatives = negatives$logRr,
                      seLogRrNegatives = negatives$seLogRr,
                      logRrPositives = positive$logRr,
                      seLogRrPositives = positive$seLogRr,
                      null = null)

#Compute the calibrated p-value:
calibrateP(null = null, logRr = positive$logRr, seLogRr = positive$seLogRr) #Compute calibrated p-value
[1] 0.8390598

Technology

This is a pure R package.

System requirements

Requires R (version 3.1.0 or newer).

Installation

In R, use the following commands to install the latest stable version from CRAN:

install.packages("EmpiricalCalibration")

To install the latest development version directly from GitHub, use:

install.packages("remotes")
library(remotes)
install_github("ohdsi/EmpiricalCalibration", ref = "develop")

User Documentation

Documentation can be found on the package website.

PDF versions of the documentation is also available:

• Vignette: Empirical calibration of p-values
• Vignette: Empirical calibration of confidence intervals
• Vignette: Empirical calibration and MaxSPRT
• Package manual: EmpiricalCalibration.pdf

Support

• Developer questions/comments/feedback: OHDSI Forum
• We use the GitHub issue tracker for all bugs/issues/enhancements

Contributing

Read here how you can contribute to this package.

License

EmpiricalCalibration is licensed under Apache License 2.0

Development

This package has been developed in RStudio.

Development status

This package is ready for use.

empiricalcalibration's Issues

GGPLOT2 V3.3.0 throwing Errors

I had upgraded all my packages to run PLE and was getting this issue in plotCalibrationEffect

ggplot2::ggsave(fileName, plot, width = 6, height = 4.5, dpi = 400)

Error in frame[[1]] == ".signalSimpleWarning" : comparison (1) is possible only for atomic and list types

Rolling back GGPLOT to V3.2.0 fixed it:
devtools::install_github("tidyverse/ggplot2",ref = "v3.2.0")

plotCalibrationEffect hides extreme values

PR incoming

Bug description

When using plotCalibrationEffect with values far to the left or right they are hidden from display.

Steps to reproduce

Produce a plot with crazy values, e.g.

inputDf <- data.frame(logRr = c( 1.1, 1.3, 1, 2, 1.1, 1.3, 1, 2, 1.1, 1.3, 1, 2), seLogRr = c(0.1, 0.5, 0.3))
EmpiricalCalibration::plotCalibrationEffect(inputDf$logRr, inputDf$seLogRr, logRrPositives = c(-3,-2), seLogRrPositives = c(0.1, 1.2))

Warning: Removed 2 rows containing missing values (geom_point).

Proposed solution

Allow x axis limits to be flexible, instead of being fixed at ~0.25 - 10.

In README.md, order of parameters is wrong

Where it reads:
calibrateP(positive$logRr,positive$seLogRr, null) #Compute calibrated p-value
null is shown as the last argument, but it's actually the first.

Add calibration to `computeCvPoissonRegression()` function

The computeCvPoissonRegression() currently does not allow for calibration like the other critical value computation functions. This should be added.