Questionable Research Practices: Is data fishing a part of the scientific method?

There's been a lot of discussion lately regarding the "replication crisis" in science at the moment, where researchers have been testing Ioannidis' argument that most published research findings are false. The basic idea here being that there are a number of features inherent to scientific processes, as well as a collection of bad scientific practices, that culminate in a number of reported positive findings sneaking into the literature and not being corrected until it's already been cited as evidence of a false claim throughout entire fields of science.

For example, one major issue is the problem of publication bias. This describes the effect of journals favouring positive or "sexy" results over studies that get negative results or report finding no effect where previous results had reported effects. So what ends up happening is that ten researchers might choose to research whether X causes Y, and eight of them find no effect but two find an effect. This can happen due to bad methodology, fraud, experimenter bias, or just simply chance and statistical noise. But as a result of studies favouring positive findings, those two papers will get published and the eight negative findings will get rejected. Since they're never published, a review of the published literature will tell us that 100% of studies on the topic find a result, and without getting access to the unpublished studies we'd be unaware that only 20% found a result - which is much less impressive.

THE QRPs ARE COMING FROM INSIDE THE HOUSE

So far we can see that there is a serious issue here that needs addressing but it doesn't seem like all hope is lost. Things like publication bias, once identified, seem fairly easy to address - encourage journals to publish negative findings, start journals for negative findings, emphasise the importance of negative findings when teaching students to avoid the assumption that they're a waste of time, and so on. Most of these problems have relatively easy solutions because everyone agrees that it's bad science and weakens the confidence in our published results.

But what do we do when the people doing bad science think that they're doing good science?

Misandry in Mice: Is sexism among rodents destroying science?

"This is a nightmare. It isn't real. It isn't... DON'T TOUCH ME!"

In world-breaking cute news, it turns out that mice are scared of men. Robert Sorge and colleagues decided freak some mice out by seeing whether the gender of the researcher could affect the behavior of mice, and they discovered that mice experienced greater stress in the presence of men compared to women.

Besides being funny to imagine a mouse atop a chair shrieking as a man approaches him and trying to bat him away with a mini broom, it does have some serious implications for not only how we should conduct future research but also in how much confidence we should attribute to past research that hasn't controlled for this factor.

SHRIEKING MICE

Before discussing the complications this research raises, we'll first look at what the study did and exactly what they found. One of the main measures used in the study was the "mouse grimace scale", which adorably looks like this:

and presumably the judgement of where a mouse fell on the scale simply involved a group of researchers huddled around a mouse cooing: "Who's a grumpy mouse? You are! Yes, you're a grumpy mouse!". The pain response (or 'grimace') was used as a proxy for stress and this was associated with an observed "stress induced analgesia" (SIA). SIA simply means that when an individual is stressed, any pain it experiences is effectively numbed - like when an athlete injures themselves during a game but doesn't realise they are in pain until after the game has ended.

I'd love to be able to say that the researchers also used a handful of other semi-anthropomorphic measures, like a tiny mouse-sized Freudian couch where it talks about its feelings, but unfortunately they just looked at body temperature and corticosterone. However, these proxies for stress did support the conclusions reached based on the grimace metric.

The Parable of the Strawbehaviorist

My stimulus-response reflex chain is set to "Kill!".

In another time and in another place, giant men made of straw roamed the lands. They were known as 'strawbehaviorists'. Their heads were hollow, like a doughnut made more of hole than doughnut, and upon creation they knew nothing of the world. All of their temperaments and responses were simply the imprints written into them by their experiences in the world, like behavioral Golems whose behaviors are masses of unformed clay waiting to be shaped by their environment.

The people feared the absurdity of these creatures; didn't they know that they are logical impossibilities? A number of heroes came forth over the years to rip their straw bodies apart with ease but somehow the strawbehaviorists would rebuild themselves. Even the great Noah Chumsky, champion of champions, defeated these strawbehaviorists with the power of The Good Book (Review) but still they would not die.

Many theories were constructed to explain the regenerative capabilities of the strawbehaviorist. Some said that their own refusal to accept the incoherence of their existence was in itself enough to keep them alive, and this led to the Dissonance Revolution. This was an attempt to highlight their inconsistencies, like pointing out that they could not possibly learn language without a Learning Ordinary Language (LOL) device in the brain (since they had no brain and barely a head at all), with the hopes that these facts would cause them to crumple in a puff of logic.

However, despite all these fierce attacks and despite their apparently weak frames that should have been easy to dismantle, the strawbehaviorists grew in numbers. They were seemingly unperturbed by the onslaughts, as if they were too arrogant to just accept that they cannot exist. With time though, the people tried a new approach: they simply ignored the strawbehaviorists.

They wrote books about how they were defeated, how society had moved on, and whenever a strawbehaviorist would dare reveal itself to the people, it would be reminded that history tells us that they were beaten and that they no longer exist.

This put the minds of the people at ease and they lived their life as if the strawbehaviorists no longer existed. They told tales to their children of how they were so weak that they stood no chance against the people's greatest heroes. But every now and then, after these stories of horror and adventure have been shared with a child, a strawbehaviorist can be seen peering in through the darkened bedroom window.

The parents will comfort the children, carefully scanning the now empty window, and they'll reassure them with the words that their own parents used to reassure them in days gone by: "Don't worry, you're safe. They're not real".

And then they all lived happily ever after... Until "Strawbehaviorists 2: Rise of the Tin Machines" was released, and then no matter who wins, we lose.

---

This story was inspired by the recent high profile misrepresentations of behaviorism that described the philosophy as blank slatist or as a black box approach to psychology, and I hope that it will serve as a reminder that those people don't need to be afraid any more. Those behaviorists not only don't exist but they never have.

Some good reading on strawbehaviorism:

Why Pinker Needs Behaviorism: A Critique of The Blank Slate

What Happened to Behaviorism?

On behaviorism in the cognitive revolution: Myth and reactions.

Skinner was an evil sociopath

The great thing about eye-grabbing titles is that you can say whatever you want and justify it in terms of generating attention towards the article itself. However, problems arise when the ridiculousness of your title isn't explained away or justified in your article and instead you continue to engage in the hyperbolic misinformation. One such example of this is: Why B.F. Skinner May Have Been The Most Dangerous Psychologist Ever (thanks to @yurihbl for linking me to this absurdity of an article).

Is Skinner more dangerous than Dr.Satan himself? 

When I read the title I was obviously taken aback given that it's a fairly strong claim to make. Skinner was well-recognised as an incredibly gentle man who spent most of his life fighting against people attempting to use punishment as part of behavioral modification programs, and was awarded the Humanist of the Year in 1972. So what was happening here? Initially I thought that maybe it was going to be a thought-provoking discussion on how an effective theory of psychology could have negative implications on society, like an effective theory of nuclear physics did, but the big screenshot from 'A Clockwork Orange' didn't leave me hopeful...

There's an app for that

One of the awesome things about science and technology is that as it advances, it becomes more accessible to the everyday person. In behavioral psychology we now have a couple of possible options for cheaper, open-source contraptions to use in our experiments:

"Yeah, yeah, just a minute - I'm about to beat my top score in Flappy Bird!"

The first of these programs was described  in this article, "Need to train your rat? There is an App for that: A touchscreen behavioral evaluation system" and this is an app freely available in the Apple store which allows researchers (or the general public) to create flexible behavioral experiments on a very tight budget. One of it's advantages is also that it doesn't require a great amount of coding knowledge either so it shouldn't be that hard to set up.

The second article, "ArduiPod Box: A low-cost and open-source Skinner box using an iPod Touch and an Arduino microcontroller" describes the equipment needed to build your own Skinner box, complete with the open-source program to run it. All up, you can supposedly build a functioning operant chamber for less than $300 (US, I assume).

Here's a demonstration of it:



Even if you aren't interested in becoming a mad scientist yourself and running the neighbourhood animals through complicated behavioral experiments just to serve your own maniacal curiosity, you have to admit, seeing a rat using an ipad is pretty cool.

Sokalian Diatribes

(http://xkcd.com/451/)

For those who have been paying attention to the latest news in science, you might have heard of computer scientist Cyril Labbé. For the past two years, Labbé has been using a program he created to detect fake articles created by another program called "SCIgen" that strings together words to create fake science papers and submits these articles to a range of journals - as a result, major publishers like Springer and IEEE have been alerted to over 120 instances of these computer generated articles that have been accepted by their journals.

This is pretty shocking (even though maybe it shouldn't be as it's not the first time it's occurred) but what I find interesting is the reaction from the scientific and skeptical community. The overwhelming response I've seen at a grassroots level is one of excuses; for example, I've seen many arguments that this isn't really a problem for scientific publishing because the only place that has been identified as being scammed was a conference rather than a mainstream journal. This is similar to the criticism of the Science investigation I mention above, where the author was criticised on various grounds, including the idea that the methodology was flawed for not having a control group.

To be clear, my complaint here isn't the validity of the complaints. The ones I mention above I even agree with and they were my immediate thoughts when faced with the news. My complaint, on the other hand, is more about the difference in the reaction to this paper compare to the infamous "Sokal Affair".

The Law of Feynman

I've considered writing this entry a number of times from multiple different angles but in light of the recent Edge.org debacle, I've gone with this: The Law of Feynman. Stated simply, I view the relationship as such:

As a scientist's popularity increases, the probability of them making silly statements outside of their area of knowledge also increases. 

To help clarify this unique interaction of a scientist's popularity with silliness, I also propose that it be quantified and measured in terms of "Dawks". The unit of measurement here ranges from 0-10, with a Dawk score of "10" being comparable to the silliness of Richard Dawkins himself. If you don't think Dawkins makes silly statements outside of his area, I recommend you browse his Twitter feed. I particularly liked this one:

You showed them, Richard! Just er, let's not talk about Western Medicine or even (more broadly) Western Science.