acoppock / green-lab-sop Goto Github PK

We will routinely perform three types of checks for asymmetrical attrition: ... In checks #2 and #3, p-values below 0.05 will be considered evidence of asymmetrical attrition., If any of those checks raises a red flag, and if the PAP has not specified methods for addressing attrition bias, we will follow these procedures

This seems too lenient. The test for bias from attrition may not be powerful. You shouldn't give yourself the benefit of the doubt for something that may cause substantial bias. Why not make the Lee bounds or the Horowitz bounds the default, and only do the first proposed thing if you can somehow very convincingly demonstrate that "it is extremely unlikely that the attrition was asymmetric"?

Also

2. Consult a disinterested “jury” of colleagues to decide whether the monotonicity assumption for trimming bounds (Lee 2009; Gerber and Green 2012, 227) is plausible.

Where/how do you find this jury in practice? And what do you propose doing if they say it is not plausible?

Hello! I refer to this site frequently, thanks for making it!

Issue is: I get an error when trying to go to the HTML version of the SOP: http://htmlpreview.github.io/?https://github.com/acoppock/Green-Lab-SOP/blob/master/Green_Lab_SOP.html

Hi! I'm now preparing my experiment and referring to this nice SOP.
Could we use BMlmSE() function with fixed effects? It only accepts lm() objects, so with factor() we get very sparse X and BMlmSE() stops working.

There's a couple of things I'd like to have a default on:

1. Some say you have to report all measured outcome variables. This is hard in some cases
2. Some say you should make a scale. What's the best all-purpose way to make a scale. We've been doing principal components, but there must be tradeoffs to doing that.
3. What about multiple comparisons?

There's a couple of analysis choices to worry about:

1. Which comparisons to report (all pairwise? all versus some control condition?)
2. Which multiple comparisons correction to employ? Holm? RI? Some comparisons are in the same "family" some are not.
3. Do multi-arm trials change how we think about covariate adjustment?
4. When/how are we allowed to collapse over conditions?
5. When/how are we allowed to parameterize (e.g., in Guess and Coppock, we code conditions for their "information content" and estimate a linear model)

Jas's paper on this seems like a nice way to include arbitrarily many covariates without needing to worry about degrees of freedom? Plus then you don't need to worry as much about selecting the few you include based on "principle"?

http://sekhon.berkeley.edu/papers/lasso.pdf

In the section on permutation tests, we've written, "We recommend attempting the permutation test with mock outcome data and actual covariate data before analyzing the actual outcome data. The mock permutation test may reveal that on some randomizations, the t-statistic cannot be computed because the regressors are collinear or because the HC2 or BM SE is undefined (see the section above on 'Avoiding regression models that do not allow the BM adjustment'). In such cases, covariates should be dropped from the model until the mock permutation test runs without errors."

I'm thinking to change this so that if the t-statistic is uncomputable on only a small % of randomizations (e.g., less than 5%), we do a conditional permutation test (i.e., randomizations where the t-stat is undefined are excluded from both the numerator and the denominator of the p-value).

One situation where this might happen is if the PAP specifies poststratification and there are some randomizations where all units in some poststratum are assigned to one treatment condition.