Asymptotics
I’m thinking a lot about sampling these days. The idea of classical hypothesis testing is that we choose a null hypothesis and a testing scheme, and ask how often we would accidentally reject the null hypothesis if we took repeated samples and if the null were actually true. The null might be that the mean of the population is 0. The testing scheme might be that we take a sample of N=100 observations and we reject the null if the mean of that sample is less than -1 or greater than +1. The significance level of the test, the probability that we falsely reject the null, might be 14%. That means that if we took 10,000 100-observation samples, and the true mean is M=0, we would expect about 1,400 sample means to be outside of the [-1,+1] interval.
Any null hypothesis takes some set of background facts as given, as assumptions that aren’t tested. In the case above, these assumptions might include that (a) the population is distributed according to some shape f(x) around the mean M, and we know the entire shape, just not the value of M, (b) the observations are chosen independently (which also means they are chosen with replacement, I think), and (c) the shape f(x) does not change during the course of our sampling. If we wanted, we could instead take M=0 as an assumption and test statement (b) instead, or use some other combination of assumptions and null.
Well, I haven’t even gotten to asymptotics yet, but I’m out of time. I’ll have to continue this another day. Asymptotics concern the properties of a testing scheme when the sample size N is large enough that the Central Limit Theorem can be called into play and estimator errors are close to being normal even if the underlying distribution shapes are not.