The Problem with Small Sample Sizes

If you've ever read through the comment section of an online science article or entered into a discussion on recent research amongst science enthusiasts, then undoubtedly you've heard the complaint of a study being flawed as a result of having a small sample size. Recently, this has been particularly true for fields like neuroscience where costly and time-consuming techniques like fMRI limit your subject pool (for example: "Power failure: why small sample size undermines the reliability of neuroscience").

The knee-jerk response of rejecting studies with small sample sizes has become quite common, in a way that I'll argue is similar to the way the popular notion of "correlation does not equal causation" is used. To be clear, such responses aren't wrong because they are never accurate but rather they are wrong in their flippancy and blanket-use. What this means is that when writers like Steven Novella criticise the use of "correlation does not equal causation", they aren't saying that it's always wrong to point out the problems of trying to determine causation from a simple correlation but rather it is wrong to reject the importance that correlations play in the determination of causation.

BUT SMALL SAMPLE SIZES ARE A PROBLEM!

It's true that there are problems with small sample sizes, however, what many people fail to recognise is that practically all studies suffer from small sample sizes. In any given study there are a number of relevant samples that can be limited in size; 1) the subject pool, 2) the stimuli, and 3) the behaviors/responses. The first case is the most obvious and it is usually what people have in mind when they criticise small sample sizes, but the second presents the lesser known problem of biases in sample materials as described in this study: "Treating stimuli as a random factor in social psychology: a new and comprehensive solution to a pervasive but largely ignored problem".

As explained by Neuroskeptic here, this describes the problem where the advantages of having a large subject pool can be offset by limitations in the size of the sample materials:

Going with the blonde vs. dark example, suppose you take 1000 volunteers, show them some pictures of blonde guys and dark guys, and get them to rate them on trustworthiness. You find a significant difference between the two groups of stimuli. You conclude that your volunteers are hair-bigots and submit it as a paper. The reviewers think, 1000 volunteers? That’s a big sample size. They publish it.  

Now that study I just described might be perfectly valid. But it might be seriously flawed. The problem is that while your sample size may be large in terms of volunteers, it might be very small in another way. Suppose you have just 10 photos per group. Your ‘sample size’, as regards the sample of stimuli, is only 20. And that sample size is just as important as the other one.  

It might be that there’s no real hair difference in perceived trustworthiness, but there are individual differences – some men just look dodgy and it’s nothing to do with hair – and in your stimuli, you’ve happened to pick some dodgy looking blonde guys. Or whatever.  

Now you can run your statistical analyses taking these possible stimulus variation effects into account. But according to Judd, Westfall and Kenny, authors of this paper, this is rarely done. They show with both real and hypothetical data, that unless you take care of this, you can find “statistically significant” differences from pure random noise. This is not a new argument, but they say it’s been ignored for too long.

The third problem with sample size which is often discussed informally amongst some researchers but rarely communicated to the public, is the problem with the number of behaviors and responses collected. This distinction can be seen in the debate between using "large-N vs small-N" statistics or between using "between-subjects vs within-subject" designs, with the former categories referring to the research most people are familiar with where you gather together large numbers of people, place them into different conditions, and then compare the results from the two different groups. The latter categories refer to the less common research that involves using only a few people in the study who are placed in multiple conditions, and the results are analysed by comparing their own responses across conditions.

WHAT DOES THIS MEAN?

At this point, the problem might not to be apparent so I'll explain it in terms of the idea of a "learning curve". This is a concept that everybody is familiar with: when you start learning a new skill, you will experience a gradual increase in ability until it plateaus at some level. Makes sense, right? Well actually, it doesn't. Think about some time when you learnt a new skill, like playing a sport. Your improvement doesn't follow an increasing curve but rather it will often be composed of disjointed periods of steep increases and periods of little-to-not improvement, which makes the "learning curve" look more like a "learning staircase".

In other words, it will look a little like this:

(Yes, that graph was made in Paint. What of it?).

The red line represents what we think of as the "learning curve", whereas the blue line and dotted green line represent the "learning staircases" of two hypothetical subjects. So, if people learn in 'staircases', why does the myth of the curve persist? The answer can be found in the graph above: when we average out the results of two (or more) people, these observable differences in individuals are ironed out - and this is the learning curve.

Some people understand concepts best by looking at dry graphs (even beautifully coloured ones like above), and some learn best by relating them to an everyday situation so I'll illustrate the basic issue by retelling a joke about statistics:

A physicist, a biologist, and a statistician go hunting one day. They're creeping around the forest until they come across a deer in the distance, so they crouch down and all three take aim. The physicist is the first to fire but his shot veers off to the left by about 5 metres. The biologist takes his turn and his shot also misses by 5 metres, but this time it goes off to the right of the deer. At this point the statistician yells, "We hit it!".

As much as I am loath to explain a joke, the relevance to this discussion is in the fact that even though data such as averages can represent useful and important information to us, we also have to recognise that averages applied to individuals can lead to erroneous (and frankly ridiculous) conclusions. The statistician suggesting that they've hit (and thus killed) a deer between two missed shots average out to a direct hit is just as wrong as a statistician looking at the averages of thousands of people learning a new skill and suggesting that we progressively learn in the form of a curve.

WHAT SHOULD WE BE DOING THEN?

Ideally, we would have massive studies consisting of thousands of subjects, thousands of stimuli, and thousands of behaviors/responses taken from a number of conditions. Practically speaking, however, this is not really going to happen so our next best step is to view samples within context of the research they appear in.

This means that our first response to a study utilising a small subject sample shouldn't be something along the lines of, "...but we can't conclude much given the small sample" and instead we should be asking whether the small sample affects the results or whether the appropriate statistical designs and analyses were used.

The conclusion being that large and small studies both have their advantages and disadvantages. The large studies allow us to accurately identify trends and generalise results to populations, whereas the small studies allow us to examine the behavior of individuals to a much greater degree. So whilst large studies consisting of few responses from each subject can be useful, they don't always supplant or eliminate the need for small studies consisting of many responses for a few subjects.