Tuesday, 29 October 2013

The Problem with Small Sample Sizes

If you've ever read through the comment section of an online science article or entered into a discussion on recent research amongst science enthusiasts, then undoubtedly you've heard the complaint of a study being flawed as a result of having a small sample size. Recently, this has been particularly true for fields like neuroscience where costly and time-consuming techniques like fMRI limit your subject pool (for example: "Power failure: why small sample size undermines the reliability of neuroscience").

The knee-jerk response of rejecting studies with small sample sizes has become quite common, in a way that I'll argue is similar to the way the popular notion of "correlation does not equal causation" is used. To be clear, such responses aren't wrong because they are never accurate but rather they are wrong in their flippancy and blanket-use. What this means is that when writers like Steven Novella criticise the use of "correlation does not equal causation", they aren't saying that it's always wrong to point out the problems of trying to determine causation from a simple correlation but rather it is wrong to reject the importance that correlations play in the determination of causation.