Scientists replicated 100 recent psychology experiments. More than half of them failed.

HeinieKaboobler · 19時間前

Quite a conclusion. It's rare to find such good prose in scientific literature. "Any temptation to interpret these results as a defeat for psychology, or science more generally, must contend with the fact that this project demonstrates science behaving as it should. Hypotheses abound that the present culture in science may be negatively affecting the reproducibility of findings. An ideological response would discount the arguments, discredit the sources, and proceed merrily along. The scientific process is not ideological. Science does not always provide comfort for what we wish to be; it confronts us with what is"

knightsvalor · 20時間前

Full text of the actual journal article for the lazy: http://www.sciencemag.org/content/349/6251/aac4716.full

NeuroLawyer · 17時間前

1-2 controlled studies = no significance. 3-5 controlled studies = slightly significant. 6+ controlled studies and meta-analysis to determine if publication bias = moving more towards "fact".

Runoo · 16時間前

Co-author here! Great to see it gets so much love from Reddit. The real interesting part is seeing how other disciplines hold up in terms of reproducibility. A new project has been started: Reproducibility Project: Cancer Biology, they will try to replicate 50 studies. I am very curious how this will turn out, I highly encourage other disciplines to also start a reproducibility project to test how consistent their findings actually will be. I don't see these results as discouraging, instead, I see it as a big step in developing scientific methods. Now we know which methods and standards might be wrong, we can try to fix it (for example by developing guidlines).

angrygolfer1 · 17時間前

Saddest part is that this is a high water mark for scientific reproducibility. "Landmark" cancer studies were only 11% reproducible.

ProfessorSoAndSo · 20時間前

I'm social psychologist and one of the co-authors of this paper. This is sobering news for psychological science. I think everyone in the field hoped that more of the studies would have replicated. At the same time, it is a simple fact of science that findings will frequently fail to replicate. My wife is a neuroscientist, and many of the most basic and well-accepted findings in her field also fail to replicate. This does not mean that the findings are "wrong." It speaks instead to the complexity of science. Outcomes vary drastically based on countless factors that cannot always be anticipated or controlled for.

To those wanting to dismiss psychological science as "cult science" based on these findings, note how ironic your response is. You're discrediting the very people whose data you are using to back up your claim. This massive, groundbreaking project was conducted on psychological science by psychological scientists. In my view, psychological scientists are among the most dedicated and rigorous scientists there are. No other field has had the courage to instantiate a project like this. And I am sure that many of you would be shocked to find out how low the reproducibility rates are in other fields. Problems of non-reproducibility, publication bias, data faking, lack of transparency, and the like plague every scientific field. The people you are labeling as "cult" scientists are leading the movement to improve science of all types in a much needed way.

flounder19 · 20時間前

Original study effect size versus replication effect size (correlation coefficients).

Diagonal line represents replication effect size equal to original effect size. Dotted line represents replication effect size of 0. Points below the dotted line were effects in the opposite direction of the original. Density plots are separated by significant (blue) and nonsignificant (red) effects.

(source)

Dan_Keane · 18時間前

For me, the take away from this is distilled into the great quote that I heard on the SGU:

Science is the only thing that disproves science, and it does it all the time.

Matt Dillahunty

Cr3X1eUZ · 20時間前

What a surprise.

Cargo Cult Science, Richard Feynman:

"When I was at Cornell, I often talked to the people in the psychology department. One of the students told me she wanted to do an experiment that went something like this--it had been found by others that under certain circumstances, X, rats did something, A. She was curious as to whether, if she changed the circumstances to Y, they would still do A. So her proposal was to do the experiment under circumstances Y and see if they still did A.

I explained to her that it was necessary first to repeat in her laboratory the experiment of the other person--to do it under condition X to see if she could also get result A, and then change to Y and see if A changed. Then she would know the the real difference was the thing she thought she had under control.

She was very delighted with this new idea, and went to her professor. And his reply was, no, you cannot do that, because the experiment has already been done and you would be wasting time. This was in about 1947 or so, and it seems to have been the general policy then to not try to repeat psychological experiments, but only to change the conditions and see what happened.

...in 1937 a man named Young did a very interesting one. He had a long corridor with doors all along one side where the rats came in, and doors along the other side where the food was. He wanted to see if he could train the rats to go in at the third door down from wherever he started them off. No. The rats went immediately to the door where the food had been the time before.

The question was, how did the rats know, because the corridor was so beautifully built and so uniform, that this was the same door as before? Obviously there was something about the door that was different from the other doors. So he painted the doors very carefully, arranging the textures on the faces of the doors exactly the same. Still the rats could tell. Then he thought maybe the rats were smelling the food, so he used chemicals to change the smell after each run. Still the rats could tell. Then he realized the rats might be able to tell by seeing the lights and the arrangement in the laboratory like any commonsense person. So he covered the corridor, and still the rats could tell.

He finally found that they could tell by the way the floor sounded when they ran over it. And he could only fix that by putting his corridor in sand. So he covered one after another of all possible clues and finally was able to fool the rats so that they had to learn to go in the third door. If he relaxed any of his conditions, the rats could tell.

Now, from a scientific standpoint, that is an A-number-one experiment. That is the experiment that makes rat-running experiments sensible, because it uncovers that clues that the rat is really using-- not what you think it's using. And that is the experiment that tells exactly what conditions you have to use in order to be careful and control everything in an experiment with rat-running.

I looked up the subsequent history of this research. The next experiment, and the one after that, never referred to Mr. Young. They never used any of his criteria of putting the corridor on sand, or being very careful. They just went right on running the rats in the same old way, and paid no attention to the great discoveries of Mr. Young, and his papers are not referred to, because he didn't discover anything about the rats. In fact, he discovered all the things you have to do to discover something about rats. But not paying attention to experiments like that is a characteristic example of cargo cult science."

http://neurotheory.columbia.edu/~ken/cargo_cult.html

Vegerot · 17時間前

Why is this a bad thing? This is how science works, how it always works. The (truncated) steps of science are: People test something, come to a conclusion, and publish their findings. However, that actually misses one of the biggest parts of science: peer review. Publishing a paper is not the last step of discovery.

This happens all the time in science. A scientist comes to a conclusion, and someone else discovers that their conclusion was wrong. This is good. It's all part of building knowledge.

However, it's clearly a problem that over 50% of them turned out to be false. This is definitely bad.

18時間前

[deleted]

aggie_fan · 17時間前

Sometimes random assignment creates comparable treatment and control groups, sometimes it doesn't. This alone is justification for every randomized experiment to be replicated a dozen times.

jswan28 · 15時間前

I think the hate for psychology from a lot of scientists comes from the fact that it is so young that there are no laws of psychology. Psychology is a bit shaky because we haven't built a solid foundation yet, but that doesn't mean that we won't one day. Disparaging those that are trying to build that foundation will only delay it's completion.

Series_of_Accidents · 16時間前

I'm a quantitative psychologist, and while disappointing, this is not at all surprising to me. There are two fatal flaws of our field that lead to this, and they are highly interrelated: publish or perish, and a dearth of null-hypothesis journals. These two factors lead to the temptation to hunt for findings (often spurious) and search for explanations later. This is lying with statistics, plain and simple.

Sadly, statistics are not properly utilized by a large proportion of scientists (in all fields-- psychologists are far from the only, or even the worst offenders) because they fail to understand or test for the underlying assumptions for any given analysis. That said, I would like to reiterate that this problem is not unique to psychology. Far from. In fact, on NIH panels, it is often the psychologist that is asked if the statistical methods proposed are solid. As /u/ProfessorSoAndSo stated, "psychological scientists are among the most dedicated and rigorous scientists there are. No other field has had the courage to instantiate a project like this."

Let's fight for more access to raw data, null hypothesis journals, and an employment model that doesn't depend upon your ability to make the lucky hypotheses, but upon your ability to do good science.

ConsummateK · 17時間前

I know people love jumping on the "psychology isn't real science" train but this is widespread in academia. Only "novel" contributions are valued which leads to this.

Nightgloom · 17時間前

I recently read an article that touched on the same topic. It noted how many failed experiments are never published and how this prevents future researchers from being able to learn from past mistakes and unproven theories. It was shedding a lot of light on how much work is put into publishing results that when an experiment fails, the work required to publish the results is too significant for it to be worth it for the researcher to release it.

That information along with the information in this article, as well as the comments here, leads me to believe that there should be a prominent website dedicated to releasing failed experiments. The requirements for publishing a failed experiment should be lowered so that researchers are actually inclined to release the information, but the requirements should still be high enough that any data can be validated and detailed enough.

jam11249 · 16時間前

One of the big problems in science is a bias to only publishing successful results.

Let's say 20 research groups do the same experiment on demonstrating aspirin reduces the chance of stroke. It's a total guess that it might, but it's a relatively easy study to do so why not. 19 conclude that the drug has no effect. One outlier concludes with p=0.05 significance that aspirin does work.

To claim the outlier is the truth would be absurd. But why would you submit a paper saying that your wild guess punt of a study doesn't work.

Headlines the next day, "aspirin provides barrier against stroke!"

Doglatine · 16時間前

"~~Scientists~~ Psychologists replicated 100 recent psychology experiments. More than half of them failed."

There's something a little awkward with the title of this Vox piece. It falsely insinuates (perhaps accidentally) that when real scientists tried to replicate the results of psychologists' research, they failed. Instead, this is a very commendable 'inside job'. As psychology matures as a science, and the amount of experimental data it has access to increases, it's getting tighter and more rigorous, as demonstrated by this research. It's also worth noting that different subdisciplines have been maturing at different rates; perceptual psychology, in particular, allows for highly controlled experiments with far fewer confounding variables than social psychology.

_neutral_person · 15時間前

This is freaking awesome. One of the biggests issues with science today is everyone is trying so hard to be famous by doing explorative science; unfortunately the pressure to publish astonishing results has pressured researchers to either fudge data or pick lucrative areas of research. A certain percentage of NSF funding should go to reproducing experiments performed. It's the only way we can be sure besides the usually data combing during editorial phase.

BarrelRoll1996 · 17時間前

*but almost half of them succeeded !*

bananafreesince93 · 16時間前

Uh, yeah?

That sounds pretty good, actually.

CarbonEmitter · 13時間前

As a geologist, I do not see this issue in the academic sphere. We are well aware of our uncertainties and tend to keep multiple hypothesis even after completed work. There is usually value in proving or disproving something in our field.

The complexity of the human mind with regards to psychology is much more uncertain with our infantile understanding compared to something we can measure, visualize, and map. I can easily see how it is too difficult to control and understand all the variables in social science whereas in hard science there are more controls and objective analysis can be performed.

futuremachine · 18時間前

Isn't this what science is supposed to do? Replicate old experiments to see which ones remain true and which ones aren't supported by new research. Isn't it very hard to prove something but very easy to disprove something?

InVivoVeritas · 16時間前

That they could not replicate the findings does not negate the prior studies... It simply shows that those results did not hold true a second time around. This is the process by which all of our great discoveries were made. Who is to say that the 50% that did replicate would not replicate a third time? We are constantly reevaluating based on new evidence for this very reason. Confidence is a sliding scale.

FuckTylerH · 15時間前

I bet this is true for biology/medical experiments as well.

RagingNerdaholic · 14時間前

I guess this is ultimately a good thing? Science should continually aim to improve and refine towards factual solidity, and disproving results previously assumed to be factual (or as close as one can be to factual) is one of the ways it's done.

mewarmo990 · 14時間前

Yes, this is a normal part of the scientific process. Independent replication makes for better science.

jadelombard · 8時間前

One of the big problems in science is a bias to only publishing successful results. Let's say 20 research groups do the same experiment on demonstrating aspirin reduces the chance of stroke. It's a total guess that it might, but it's a relatively easy study to do so why not. 19 conclude that the drug has no effect. One outlier concludes with p=0.05 significance that aspirin does work. To claim the outlier is the truth would be absurd. But why would you submit a paper saying that your wild guess punt of a study doesn't work. Headlines the next day, "aspirin provides barrier against stroke!"

zebrahair743 · 17時間前

I wonder if this experiment would pass or fail if someone were to replicate it.

17時間前

[deleted]

Date	Time (Eastern Time- USA)	Person	Description
28 Aug	1pm	Noah Wardrip-Fruin (UC Santa Cruz)	Video Games Beyond Genre: Game Discovery with Natural Language Processing
28 Aug	1pm	Rush Holt and Frank von Hippel	The Iranian Nuclear Deal
31 Aug	1pm	UCSF Office of Career & Prof. Development	Career Advice for Young Scientists
1 Sep	11am	American Chemical Society AMA
2 Sep	1pm	Caitlin Dunn, Deborah McFarland, and Kelly Callahan	Global Health: Neglected Tropical Diseases
2 Sep	1pm	PLOS Science Wednesday
3 Sep	1pm	Matt Thomson, UCSF	Biophysics
4 Sep	8am	Oil-Climate Researchers	Creating a Global Oil-Climate Index
7 Sep	1pm	Stephanie Huette	Language processing
8 Sep	11am	American Chemical Society AMA
9 Sep	8am	Forensic Psychology Unit	How psychological science is improving criminal justice
9 Sep	1pm	PLOS Science Wednesday: Stuart Kim	Genome sequencing of supercentenarians
10 Sep	1pm	Alain Laederach UNC-Chapel Hill	RNA Biology
11 Sep	1pm	Ying-Hui Fu, UCSF	The Genetics Behind Sleep
14 Sep	1pm	Sri Kosuri and the UCLA iGEM Team	Genetic and Protein Engineering of Novel Programmble Synthetic Silks
15 Sep	11am	American Chemical Society AMA
16 Sep	1pm	PLOS Science Wednesday
17 Sep	1pm	Debbie Cory-Slechta	Effects of Toxins and Chemicals
18 Sep	11am	Brian Tomaszewski, Rochester Institute of Technology	Using mapping and Geographic Information Systems to provide disaster relief
21 Sep	1pm	Rob Guralnick and Michael Denslow	Notes From Nature
22 Sep	11am	American Chemical Society AMA
23 Sep	1pm	PLOS Science Wednesday
24 Sep	1pm	Jason Hoyt and Peter Binfield - Co-founders of open access journal PeerJ
25 Sep	1pm	Disk Detective Science Team	Circumstellar Disks and Citizen Science
26 Sep	1pm	SfN AMA Guest #1	Hold for AMA (SfN #1)

science

Make sure to check out our sister subreddit /r/EverythingScience

Filter by Field (Click to Filter)

Upcoming AMAs

調停者