Resampling techniques are modern statistical techniques that involve taking repeated subsamples from a sample. These procedures tend to be computationally intensive, since they involve computing statistics of a subsample, creating new subsamples and repeating the process thousands or perhaps millions of times. This can allow for additional analysis of the subsamples leading to increased confidence and knowledge of the larger population. The three main techniques we will discuss here are bootstraping, jacknife and permutation tests.
You will be able to:
- Understand permutation testing
- Understand what jacknife is
- Understand what bootstrapping is
Let's start by defining the sampling methodology for these techniques. The bootstrap method works by taking random samples with replacement. In contrast, the jacknife, the older of the two methods, works by taking samples by removing on, or more, observations at a time. Both aim estimate the variability of point estimators derived from the original samples.
The motivating principle behind both is that by analyzing the variance of parameter estimates from these synthetic samples, we can also gauge the variance of our point estimate for the population itself. For example, we might take an original sample from our population and then use the jacknife or bootstrapping method to generate additional synthetic samples. By calculating the point estimate of interest for these synthetic samples, we can better gauge the confidence interval and variability of our original point estimator.
Another related methodology is permutation tests. Permutation tests can be used in lieu of assumed parameter distributions for any statistical test. For example, we discussed the central limit theorem: that when taking the mean of repeated sample from a population, the means of these samples will form a normal distribution. From this, we were then able to extrapolate confidence intervals surrounding our estimate for the mean of the entire population by assuming that our sample mean was from a normal distribution. This allowed us to define our confidence bands associated with various levels of type I errors which we set with
For example, let's say we had two samples, one with 37 observations and the other with 45 observations. We calculate the mean of both samples and wish to perform a hypothesis test with a 5% confidence interval for whether the two samples belong to the same overall population. In our previous work, we would use a t-test to perform this comparison. The permutation test alternative would be to compare the difference in these sample means to the difference in sample means of all possible combinations of 37-45 splits between our 82 data points. In other words, we compare the difference between our actual sample means to the difference in sample means between all variations of all those 82 points in order to calculate our p-values and determine whether we accept or reject the null-hypothesis.
- http://hydrodictyon.eeb.uconn.edu/eebedia/images/9/9d/FelsensteinChap20.pdf
- https://www.scss.tcd.ie/Rozenn.Dahyot/453Bootstrap/05_Permutation.pdf
- http://iasri.res.in/seminar/AS-299/ebooks%5C2006-2007%5CMsc%5Ctrim1%5C4.%20Resampling%20Techniques%20Jackknife%20and%20Bootstrap-%20Vinayanand.pdf
In this lesson, we continued discussing non-parametric statistics and investigated resampling techniques. This included bootstrapping, jacknife and permutation tests. In the upcoming lab, you'll define functions that implement these techniques and then use them to conduct statistical simultions and tests.