![]() |
@BrianNosek | |||||
Using prediction markets we found that researchers were very accurate in predicting which studies would replicate and which would not. (blue=successful replications; yellow=failed replications; x-axis=market closing price) socarxiv.org/4hmb6/ nature.com/articles/s4156… #SSRP pic.twitter.com/4xfv77NFqT
|
||||||
|
![]() |
Brian Nosek
@BrianNosek
|
Aug 27 |
We replicated 21 social science experiments in Science or Nature. We succeeded with 13. Replication effect sizes were half of originals. All materials, data, code, & reports: osf.io/pfdyw/, preprint socarxiv.org/4hmb6/, Nature Human Behavior nature.com/articles/s4156… pic.twitter.com/dVrSK922Cb
|
||
![]() ![]() ![]() |
![]() |
Brian Nosek
@BrianNosek
|
Aug 27 |
Design ensured 90% power to detect an effect size half as large as original study. Replications averaged 5x the sample size of originals. We obtained original materials in all but one case, and original authors provided very helpful feedback on design. socarxiv.org/4hmb6/
|
||
![]() ![]() ![]() |
![]() |
Brian Nosek
@BrianNosek
|
Aug 27 |
Eight original authors & Malcolm Macleod wrote commentaries published in Nature Human Behavior offering a variety of insights. Particularly attend to the ideas about why our results might have differed from the originals. See commentaries linked to dx.doi.org/10.1038/s41562…
|
||
![]() ![]() ![]() |
![]() |
Brian Nosek
@BrianNosek
|
Aug 27 |
For successful ones, replication effect sizes were about 3/4ths the original effect size. For failures, replication effect sizes were near 0. Could suggest influence of publication bias and false positives but does not demand that conclusion. nature.com/articles/s4156…
|
||
![]() ![]() ![]() |
![]() |
Brian Nosek
@BrianNosek
|
Aug 27 |
An outstanding example of replication fostering deeper understanding is the Shah, @m_sendhil, & @eshafir. We failed to replicate their Study 1 of 5. Then they conducted replications of all 5. They also failed on Study 1, but other parts replicated: osf.io/7z38r/
|
||
![]() ![]() ![]() |
![]() |
Brian Nosek
@BrianNosek
|
Aug 27 |
The social-behavioral sciences are undergoing a reformation. First, replication is becoming a normal activity for reducing uncertainty and improving understanding of novel claims. See @RolfZwaan et al. cambridge.org/core/journals/…
|
||
![]() ![]() ![]() |
![]() |
Brian Nosek
@BrianNosek
|
Aug 27 |
Second, social scientists are adopting preregistration to improve rigor & demarcate confirmatory and exploratory results—the latter opening possibilities but w/greater uncertainty. pnas.org/content/early/… Preregistration has doubled each year since 2012: osf.io/registries/
|
||
![]() ![]() ![]() |
![]() |
Brian Nosek
@BrianNosek
|
Aug 27 |
Preregistration and Registered Report growth is non-linear, particularly in social-behavioral sciences, and most particularly (so far) in psychology. psychologicalscience.org/observer/prere… pic.twitter.com/TD64f1X265
|
||
![]() ![]() ![]() |
![]() |
Brian Nosek
@BrianNosek
|
Aug 27 |
Third, journals are adopting transparency policies to improve reproducibility. Ex: We reviewed 33 journals that publish social-personality psych. In 2013, 0 had transparency policies. Today, 73% have some policy & 58% are relatively assertive based on TOP: cos.io/top/
|
||
![]() ![]() ![]() |
![]() |
Brian Nosek
@BrianNosek
|
Aug 27 |
Social-behavioral sciences' self-scrutiny is changing the norms for rigor and reproducibility. Change is occurring by grassroots efforts by individuals and labs, and by policy interventions by disciplinary leaders and editors. e.g., @improvingpsych improvingpsych.org
|
||
![]() ![]() ![]() |
![]() |
Brian Nosek
@BrianNosek
|
Aug 27 |
Change is also spreading. Ex: More than 100K researchers are using osf.io to register their research and make data and materials more accessible. All fields will benefit from this reformation toward greater rigor and transparency.
|
||
![]() ![]() ![]() |
![]() |
Brian Nosek
@BrianNosek
|
Aug 27 |
Vox: "What scientists learn from failed replications: how to do better science." vox.com/science-and-he…
|
||
![]() ![]() ![]() |
![]() |
Brian Nosek
@BrianNosek
|
Aug 27 |
Science: "‘Generous’ approach to replication confirms many high-profile social science findings" sciencemag.org/news/2018/08/g…
|
||
![]() ![]() ![]() |
![]() |
Brian Nosek
@BrianNosek
|
Aug 27 |
Nature: "Researchers replicated 62% of social-behaviour findings published in Science and Nature — a result matched almost exactly by a prediction market." nature.com/articles/d4158…
|
||
![]() ![]() ![]() |
![]() |
Brian Nosek
@BrianNosek
|
Aug 27 |
Washington Post: "the experiments scrutinized in this latest effort were published prior to a decision several years ago by Science, Nature and other journals to adopt new guidelines designed to increase reproducibility, in part by greater sharing of data" washingtonpost.com/news/speaking-…
|
||
![]() ![]() ![]() |
![]() |
Brian Nosek
@BrianNosek
|
Aug 27 |
Wired: "Thousands of researchers now pre-register their methodology and hypothesis before publication, to head off concerns that they’ll massage data after the fact. " wired.com/story/social-s…
|
||
![]() ![]() ![]() |
![]() |
Brian Nosek
@BrianNosek
|
Aug 27 |
NPR "'The social-behavioral sciences are in the midst of a reformation'... Scientists are..announcing in advance the hypothesis they are testing; they are making their data and computer code available so their peers can evaluate and check their results." npr.org/sections/healt…
|
||
![]() ![]() ![]() |
![]() |
Brian Nosek
@BrianNosek
|
Aug 27 |
The Atlantic: "Fortunately, there are signs of progress. The number of pre-registered experiments—in which researchers lay out all their plans beforehand to obviate the possibility of p-hacking—has been doubling every year since 2012. " theatlantic.com/science/archiv…
|
||
![]() ![]() ![]() |
![]() |
Brian Nosek
@BrianNosek
|
Aug 27 |
Science News: "‘Replication crisis’ spurs reforms in how science studies are done" sciencenews.org/blog/science-p…
|
||
![]() ![]() ![]() |