reifjulian / wyoung

Control the family-wise error rate when performing multiple hypothesis tests

WYOUNG: control the family-wise error rate when performing multiple hypothesis tests

• Current version: 1.3.3 18jan2024
• Jump to: overview installation examples update history citation

---

Overview:

wyoung is a Stata command that calculates adjusted p-values using the free step-down resampling methodology of Westfall and Young (1993). It also computes the Bonferroni-Holm and Sidak-Holm adjusted p-values. Algorithm details and simulation test results are documented here.

This command was developed as part of the Illinois Workplace Wellness Study.

Installation:

Type which wyoung at the Stata prompt to determine your current version number. To install the most recent version, copy and paste the following line of code:

net install wyoung, from("https://raw.githubusercontent.com/reifjulian/wyoung/master") replace

To install the version that was uploaded to SSC, copy/paste the following line of code:

ssc install wyoung, replace

After installing, type help wyoung to learn the syntax.

Examples

Example 1. Estimate a model separately for three outcomes (mpg, headroom, and turn) and calculated adjusted p-value for displacement (3 hypotheses).

sysuse auto.dta, clear
set seed 20
wyoung mpg headroom turn, cmd(regress OUTCOMEVAR displacement length) familyp(displacement) bootstraps(100)

For each regression, the output reports unadjusted and adjusted p-values for the null hypothesis that the coefficient on the variable displacement is equal to 0. For example, in the regression regress turn displacment length, the unadjusted p-value is 0.09 and the Westfall-Young adjusted p-value is 0.14.

Example 2. Estimate a model separately for three outcomes and for two subgroups defined by foreign (3 X 2 = 6 hypotheses).

sysuse auto.dta, clear
set seed 20
local yvars "mpg headroom turn"
wyoung `yvars', cmd(reg OUTCOMEVAR displacement length) familyp(displacement) subgroup(foreign) boot(100)

Example 3. Estimate a model for three outcomes, for two subgroups defined by foreign, and calculate adjusted p-values for both displacement and length (3 X 2 X 2 = 12 hypotheses).

sysuse auto.dta, clear
set seed 20
local yvars "mpg headroom turn"
wyoung `yvars', cmd(reg OUTCOMEVAR displacement length) familyp(displacement length) subgroup(foreign) boot(100)

Example 4. Estimate a model for three outcomes and test the linear restriction _b[length] + 50*_b[displacement] = 0 (3 hypotheses).

sysuse auto.dta, clear
set seed 20
local yvars "mpg headroom turn"
wyoung `yvars', cmd(reg OUTCOMEVAR displacement length) familyp(length+50*displacement) familypexp boot(100)

Update History:

• 1.3.3

  • fixed bug where unadjusted p-val was reported assuming normality (affected Stata versions 14 and lower only)

• 1.3.2

  • better error handling added for detail option

• 1.3.1

  • controls() option edited; previous functionality moved to controlsinteract()

• 1.3

  • controls() option added

• 1.2

  • familyp() option now supports multiple variables. subgroup() option added

• 1.1

  • familyp() option now supports the testing of linear and nonlinear combinations of parameters

• 1.0.5

  • familyp() option now supports factor variables and time-series operators

• 1.0.4

  • Added support for commands that don't store p-values in r(table) (eg ivreg2)

• 1.0.3

  • Better error handling for missing observations

• 1.0.2

  • Cluster bootstrap now required when clustered standard errors are present; force option added

• 1.0.1

  • Cluster bootstrap option added

Citation:

wyoung is not an official Stata command. It is a free contribution to the research community. You may cite it as:

Jones, D., D. Molitor, and J. Reif. "What Do Workplace Wellness Programs Do? Evidence from the Illinois Workplace Wellness Study." Quarterly Journal of Economics, November 2019, 134(4): 1747-1791.

wyoung's Issues

controls option

Add a controls() option, similar to subgroup(), that calculates adjusted p-vals for different permutations of the model that have varying sets of controls.

Feature enhancement

Hello,
first of all thanks a lot for developing such a useful command. It is clear efficient and simple to use!
I am writing to ask if you could add in the help file, an example of how to export the results from your command.
For someone like me that usually uses R it is not immediately clear, how I can export the results of the wyoung analysis into latex. I usually use esttab, but I am struggling to find ways to add the estimated, p values under the coefficients of my regressions.
Ideally, it would great if you could provide a short example showing how one can produce a table of all the regressions and below the coefficients a row for each of the p-values (p, pwyoung,pbonf,psidak).

I apologise in advance if this is trivial, but it can be really time intensive to find it out for someone not very skilled with Stata.

thanks a lot.

Extend familyp capability

Add the ability to test (1) a linear combination of coefficients (eg b1+b2) and (2) other coefficients (eg b1, b2 separately) all at the same time.

Confidence intervals

Prof. Reif,

I am very gratefull for this contribution of yours. However, I am missing the confidence intervarls for the coef calculation. Do you think is possible to add that option?

Thanks a lot,

Dr. José M. Valdés, MD

Lack of variation in one of the outcome variables

Hi Julian,

I have recently been using wyoung and it's great and handy, so thanks for sharing this public good! 😊

Just one minor issue I came across: When including one outcome variable which has no variation across treatment (which means Stata estimates a coefficient equal to 0 and omits the standard errors), p-values are not computed, and I get the following error

p-value not obtainable from matrix r(table)

It would be good if you could either

• better specify the error message, mentioning it arose from the regression of that outcome variable
• or still compute the adjusted p-values excluding that outcome for the rest of the variables,
  returning a missing p-value for that outcome variable coefficient and a warning to make the user aware of this

All the best,
Matteo

Bug when reporting unadjusted p-vals on Stata version 14 or older

Hi there,

I noticed what appears to be an error in the unadjusted p-values which occurs when including standard errors. Take the example included in your Stata help file:

sysuse auto.dta, clear
wyoung mpg headroom turn, cmd(regress OUTCOMEVAR displacement length, cluster(rep78)) cluster(rep78) familyp(displacement) bootstraps(100) seed(20)

Let's then take the example where headroom is the outcome variable. The wyoung table produces an unadjusted p-value of .25000982 for the coefficient on displacement. But if we run the underlying model outside of the wyoung package:

regress headroom displacement length, cluster(rep78)

then we get a p-value of 0.314 for the coefficient on displacement. It appears that the coefficient and standard errors are coming through correctly to the wyoung table, but unless I'm misunderstanding (apologies if I am missing something obvious!), something is going wrong with the p-value computation.

Wish list

• Add support for multiple treatments, ie allow familyp() option to take varlist as input
  • Reference: https://twitter.com/dmckenzie001/status/1154392345162522624

To-do list (minor fixes)

1. Problem: if user wrongly asks to calculate family-wise p-values for an instrument when running IV, wyoung thinks the request is valid because the variable name is in fact included in the command line.

   Solution: add a capture to the following of code, along with appropriate error handling:

    local beta_`k' = _b[`familyp']

Controls option with a blank

Allow the user to specify a blank for the number of variables. E.g., following code should run without an error:

sysuse auto, clear
wyoung mpg rep78, cmd(regress OUTCOMEVAR displacement CONTROLVARS) familyp(displacement) controls("headroom" "") bootstraps(100) seed(20)

Number of varlists in controls() option does not equal the number of outcomes

Potential solution: allow the user to specify fewer controls than outcomes, but not more.