The Problem with Small Sample Sizes

If you've ever read through the comment section of an online science article or entered into a discussion on recent research amongst science enthusiasts, then undoubtedly you've heard the complaint of a study being flawed as a result of having a small sample size. Recently, this has been particularly true for fields like neuroscience where costly and time-consuming techniques like fMRI limit your subject pool (for example: "Power failure: why small sample size undermines the reliability of neuroscience").

The knee-jerk response of rejecting studies with small sample sizes has become quite common, in a way that I'll argue is similar to the way the popular notion of "correlation does not equal causation" is used. To be clear, such responses aren't wrong because they are never accurate but rather they are wrong in their flippancy and blanket-use. What this means is that when writers like Steven Novella criticise the use of "correlation does not equal causation", they aren't saying that it's always wrong to point out the problems of trying to determine causation from a simple correlation but rather it is wrong to reject the importance that correlations play in the determination of causation.

Why World of Warcraft isn't a Skinner Box

The title of my post, for those with an understanding of the concept under discussion, is intentionally controversial. Alternatively I had considered using the title: "What are the mechanisms underpinning the partial reinforcement effect?", but I feared that such a heading would limit my audience to a very particular niche. To anyone whose understanding of "gaming" ended some time around the advent of Pac-Man, you might understand many of the words used in my title but not understand how they fit together. So, for starters, I'll briefly describe what the issue is and then I'll explain the problems with it.

WHAT IS A 'SKINNER BOX' GAME?

Games like "World of Warcraft" are often accused of being "Skinner boxes", for example Cracked and Penny Arcade have popular articles on the topic. This is a pejorative term and refers to the style of gameplay that they employ; specifically, they scatter various rewards throughout the game to get you to keep performing the same actions over and over again, in lieu of keeping you interested through the use of compelling plot lines or stories.

The 'Skinner box' label comes from research done by B.F.Skinner who looked at how different schedules of reinforcement affect behavior. In particular, he found that when pigeons couldn't reliably predict how many responses were required, or how much time needed to elapse, to receive a reward then a more persistent behavior was generated. The easiest way to think of this is to compare your behavior when operating a vending machine to your behavior when operating a slot machine. With the former, the requirements for receiving your 'reward' (your chocolate or drink) are clear - you put your money in and you get your selection after putting in the code. With the latter, the requirements for receiving a reward are much fuzzier - you can put money in a number of times before getting a 'win', and sometimes there are large gaps between wins.

What effect would you expect each contingency to have on behavior? Unless you are routinely faced with dodgy vending machines, you should expect your behavior to be more resistant to periods without rewards in the slot machine situation compared to the vending machine. That is, if you put your money into a vending machine and don't get your can of coke, you're unlikely to keep putting money into it in hopes of finally getting your can of coke. Instead you'll just conclude that the machine is broken and either call the company to complain, or just type out an angry status update on your Facebook. Conversely, if you put money into a slot machine and don't win on the first spin, you're likely to continue playing and maybe even put more money in when it runs out. This is because you understand that not every spin is a winner and sometimes you have to wait for a reward.

The argument, therefore, is that games like 'World of Warcraft' scatter rewards in this variable way so that players can't predict when the next reward is going to come, and this creates the same kind of addictive behavior we observe in people who play slot machines. If my explanation above isn't very clear, then this video from the Penny Arcade should make the position easier to understand: