Questionable Research Practices: Is data fishing a part of the scientific method?

There's been a lot of discussion lately regarding the "replication crisis" in science at the moment, where researchers have been testing Ioannidis' argument that most published research findings are false. The basic idea here being that there are a number of features inherent to scientific processes, as well as a collection of bad scientific practices, that culminate in a number of reported positive findings sneaking into the literature and not being corrected until it's already been cited as evidence of a false claim throughout entire fields of science.

For example, one major issue is the problem of publication bias. This describes the effect of journals favouring positive or "sexy" results over studies that get negative results or report finding no effect where previous results had reported effects. So what ends up happening is that ten researchers might choose to research whether X causes Y, and eight of them find no effect but two find an effect. This can happen due to bad methodology, fraud, experimenter bias, or just simply chance and statistical noise. But as a result of studies favouring positive findings, those two papers will get published and the eight negative findings will get rejected. Since they're never published, a review of the published literature will tell us that 100% of studies on the topic find a result, and without getting access to the unpublished studies we'd be unaware that only 20% found a result - which is much less impressive.

THE QRPs ARE COMING FROM INSIDE THE HOUSE

So far we can see that there is a serious issue here that needs addressing but it doesn't seem like all hope is lost. Things like publication bias, once identified, seem fairly easy to address - encourage journals to publish negative findings, start journals for negative findings, emphasise the importance of negative findings when teaching students to avoid the assumption that they're a waste of time, and so on. Most of these problems have relatively easy solutions because everyone agrees that it's bad science and weakens the confidence in our published results.

But what do we do when the people doing bad science think that they're doing good science?

Why World of Warcraft isn't a Skinner Box

The title of my post, for those with an understanding of the concept under discussion, is intentionally controversial. Alternatively I had considered using the title: "What are the mechanisms underpinning the partial reinforcement effect?", but I feared that such a heading would limit my audience to a very particular niche. To anyone whose understanding of "gaming" ended some time around the advent of Pac-Man, you might understand many of the words used in my title but not understand how they fit together. So, for starters, I'll briefly describe what the issue is and then I'll explain the problems with it.

WHAT IS A 'SKINNER BOX' GAME?

Games like "World of Warcraft" are often accused of being "Skinner boxes", for example Cracked and Penny Arcade have popular articles on the topic. This is a pejorative term and refers to the style of gameplay that they employ; specifically, they scatter various rewards throughout the game to get you to keep performing the same actions over and over again, in lieu of keeping you interested through the use of compelling plot lines or stories.

The 'Skinner box' label comes from research done by B.F.Skinner who looked at how different schedules of reinforcement affect behavior. In particular, he found that when pigeons couldn't reliably predict how many responses were required, or how much time needed to elapse, to receive a reward then a more persistent behavior was generated. The easiest way to think of this is to compare your behavior when operating a vending machine to your behavior when operating a slot machine. With the former, the requirements for receiving your 'reward' (your chocolate or drink) are clear - you put your money in and you get your selection after putting in the code. With the latter, the requirements for receiving a reward are much fuzzier - you can put money in a number of times before getting a 'win', and sometimes there are large gaps between wins.

What effect would you expect each contingency to have on behavior? Unless you are routinely faced with dodgy vending machines, you should expect your behavior to be more resistant to periods without rewards in the slot machine situation compared to the vending machine. That is, if you put your money into a vending machine and don't get your can of coke, you're unlikely to keep putting money into it in hopes of finally getting your can of coke. Instead you'll just conclude that the machine is broken and either call the company to complain, or just type out an angry status update on your Facebook. Conversely, if you put money into a slot machine and don't win on the first spin, you're likely to continue playing and maybe even put more money in when it runs out. This is because you understand that not every spin is a winner and sometimes you have to wait for a reward.

The argument, therefore, is that games like 'World of Warcraft' scatter rewards in this variable way so that players can't predict when the next reward is going to come, and this creates the same kind of addictive behavior we observe in people who play slot machines. If my explanation above isn't very clear, then this video from the Penny Arcade should make the position easier to understand:

Debunking Evolutionary Psychology

I wanted to discuss this topic because recently there have been a few disgruntled comments made about Rebecca Watson's talk at the "Skepticon" conference called: "How Girls Evolved to Shop" where she brings up a number of dodgy claims made by researchers in the field as well as how they're presented in the media. The main focus of her discussion, from my point of view at least, was on how we should be skeptical of the main assumptions of evolutionary psychology, how we should question how science is presented in the media, and also a good discussion on the effect that sexist and misogynistic attitudes have on the direction of some research in the field. Some of the criticisms against Watson are just plain silly, like the idea that since she is not an evolutionary psychologist then she should just remain quiet on the topic (or, "Shut Up and Sing", as P.Z. Myers puts it), but some arguably carry a little more substance.

An argument which is potentially more troubling is one presented by Ed Clint here which suggests that Watson's talk was an attack on the entire field of evolutionary psychology, and is thus an example of science denialism. This characterisation of her position seems unfair to me given that it seemed that she was attacking the bad science, not the entire field, but I thought it might be a good idea to discuss why the field of evolutionary psychology is often dismissed and what distinguishes the good science from the bad.

The latest deadweight dragging us (evolutionary biology) closer to phrenology is evolutionary psychology, or the science formerly known as sociobiology. If evolutionary biology is a soft science, then evolutionary psychology is its flabby underbelly. - Jerry Coyne¹.

Given the somewhat controversial title of this essay, it is perhaps necessary for me to preface it with a few disclaimers. Firstly, I am not a creationist and, for all intents and purposes, evolution is True^TM. Secondly, whenever somebody voices their skepticism over the veracity of evolutionary psychology, they are often met with the retort, “Do you not believe that the brain is a product of evolution?” with the implication that since behaviors are the product of the brain, and the brain is a product of evolution, then behaviors are the product of evolution. This logic, however, is flawed for reasons I will discuss later but I do accept that the brain is an evolved organ with implications for resulting behaviors. And thirdly, this is not a broad scale attack on evolutionary psychology – instead, my focus is on the particular approach to evolutionary psychology known as the “Santa Barbara church of psychology”².

To distinguish between the two approaches, I will follow the nomenclature used by Gray, Heaney and Fairhall³ where they refer to this approach as Evolutionary Psychology (EP). This approach (used by popular authors like Steven Pinker in his “How the Mind Works”) attempts to explain a wide range of human behaviors, like whether we have an evolutionary preference for green lawns, with an emphasis on the concept of a modular mind, and utilises a cartoonish view of the Pleistocene – with all considered, we have to wonder whether it should be rebranded as the “Hanna-Barbera church of psychology”.