A large-scale, cross-disciplinary project has concluded that roughly half of tested claims in the social and behavioural sciences hold up when independent researchers attempt to repeat the work. The findings, published in Nature, come from a Defence Advanced Research Projects Agency SCORE programme experiment led by Andrew H. Tyner of the Center for Open Science, with senior colleague Timothy M. Errington and corresponding author Brian A. Nosek. The study united hundreds of researchers from 31 countries to test 274 scientific claims drawn from 164 papers across 54 journals, sourced from a random sample of 3,900 quantitative papers published between 2009 and 2018 across six disciplines: business, economics, education, political science, psychology and sociology.
Replication attempts involve testing the exact same question as the original study, using independent data to see if the original conclusion holds true. When a genuine regularity in human behaviour exists, independent researchers following a similar procedure or analysing a fresh dataset should observe a similar result. Repeating past studies helps verify whether a finding is reliable enough to inform public policy, education or clinical practice.
Historically, verification efforts have raised questions about how often published findings endure scrutiny. A 2015 replication project tested 100 psychology studies and found that only about a third produced the original results, prompting widespread discussion about reliability. Subsequent investigations looked at other disciplines to see if the issue extended beyond psychology.
Across other fields, a pattern emerged. A 2016 study of laboratory experiments in economics found that 11 of 18 studies produced a statistically significant effect in the same direction as the original. A 2018 probe of social science experiments published in top-tier general science journals found about 62 per cent replicated, though the strength of findings was typically halved. Taken with further replication efforts, researchers suggested that, across hundreds of attempts, roughly half of well-powered replications yielded a statistically significant result in the same direction as the original.
Experts have proposed several systemic explanations for why many published findings fail to repeat. Journals have traditionally favoured novel, positive discoveries, creating a publication bias towards results that overestimate effects by chance. Small sample sizes, flawed measurement tools and flexible data analysis practices can also generate false positives. The culture of academia has often discouraged replication work; for instance, a 2017 study found that only about 3 per cent of psychology journals explicitly welcomed replication studies, with a third implicitly discouraging them.
Most prior verification projects focused on a single discipline or small paper groups. To assess whether the patterns hold across the broader landscape of human behaviour research, a vast, international collaboration led by Andrew H. Tyner, with Timothy M. Errington and Brian A. Nosek, launched a systematic cross-disciplinary investigation.
Nosek, who coordinated the 2015 psychology project and co-authored the 2018 analysis of studies in top general science journals, described the SCORE project as part of a broader research programme begun in 2019. “This project was part of a larger research program called SCORE that involved thousands of researchers around the world and began in 2019,” Nosek told PsyPost. “We were investigating whether algorithms could predict the replicability of findings to support decision-making about what research is needed next to advance knowledge.”
The team assembled a random sample of 3,900 quantitative papers published between 2009 and 2018 across 62 journals, representing six major disciplines: business, economics, education, political science, psychology and sociology. To be eligible, a paper had to use real human data and report a positive statistical result. From a pool of 1,500 eligible papers, the researchers matched specific claims with independent groups able to test them.
In total, 296 replication attempts were completed, covering 274 unique scientific claims drawn from 164 published papers across 54 journals. For 98 papers, researchers collected brand-new data from participants, while for 66 papers, teams analysed existing secondary datasets not used in the original research. Among studies where standard correlation metrics could be calculated, the original median sample size was 206, compared with 545 in the replication studies.
To ensure fairness and rigour, the replication studies were designed with high statistical power, giving a median 99.6 per cent chance of detecting an effect as large as the one originally reported. Whenever feasible, teams used the original study materials. Before data collection or analysis, each study’s design and analysis plan underwent internal peer review, inviting feedback from the original authors, and was publicly registered in advance on the Open Science Framework to prevent post-hoc changes.
The study’s primary yardstick was whether the replication produced a statistically significant result in the same direction as the original. Because some papers contained multiple claims, the authors calculated results both with claims weighted by paper and with claims counted individually.
When weighted by paper, 80.8 of the 164 papers, or 49.3 per cent, yielded results in the same pattern that met the statistical significance threshold. A further 40.4 per cent showed a null effect, while 9.7 per cent produced a statistically significant result in the opposite direction. Looking at the 274 unweighted claims individually, 151 claims, or 55.1 per cent, replicated successfully, while 35.8 per cent showed a null effect and 8.8 per cent went in the opposite direction.
Beyond significance testing, the researchers rescaled outcomes to a common metric, converting 157 papers (representing 249 claims) to a standard Pearson’s r. Across the original papers, the median effect size was 0.25; in the replication studies, it fell to 0.10. This represents a 58.1 per cent relative reduction in the correlation and an 82.4 per cent drop in shared variance. Overall, 80.3 per cent of the replicated papers (126 of 157) and 70.3 per cent of the individual claims (175 of 249) showed smaller effect sizes than the originals. Regression to the mean explains some of this shrinkage, since the team tested only positively reported results.
“The basic finding is that establishing the replicability of published results cannot be taken for granted,” Nosek said. “The main practical implication is that this research exposes greater uncertainty in the published literature than we might recognize. Establishing credibility and verifying research claims is an effortful process and our assumptions about the strength and nature of a finding are sometimes incorrect.”
Discipline-wise, there was only modest variation. Weighted by paper, success rates were 42.5 per cent in economics, 47.2 per cent in business, 49.0 per cent in psychology, 51.1 per cent in sociology, 52.0 per cent in political science and 63.1 per cent in education, though the education sample was small (13 papers) and thus uncertain. The year of publication between 2009 and 2018 did not significantly affect outcomes.
The team also compared studies using brand-new data with those relying on secondary data. Both approaches showed similar success rates on the basic significance test—47.8 per cent for new data and 51.4 per cent for secondary data. However, effect-size declines were steeper for new-data replications: from 0.28 originally to 0.11 in replication, whereas secondary-data studies began with smaller original effects (0.13) and fell only slightly to 0.10.
Across a spectrum of alternative replication criteria—such as whether results fall within the original margin of error or combining studies in a meta-analysis—the estimated success rate ranged from 28.6 per cent to 74.8 per cent, with a median of 49.3 per cent. The highest rate emerged from meta-analysis, which blends original and new data, while methods that penalise smaller or larger effects tended to yield the lowest rates.
“Many Labs projects aim to test the replicability of specific research findings using a multisite collaborative approach. This work extends beyond this approach by comprehensively testing a wider range of research findings across behavioral and social science disciplines. The authors used a systematic, representative, and random approach to identify claims for testing, though there was some non-random selection introduced by teams choosing to start a specific replication project,” said Stephanie Zellers, a postdoctoral researcher at the University of Helsinki who was not involved in the study. “I am quite confident in these results and that is due in large part to the transparent and nuanced reporting.”
There are important caveats. A single successful replication does not prove a finding beyond doubt, and no two social-science studies are identical. Shifting a study to a different year, participant group or cultural setting can legitimately alter outcomes, scientists warn.
“Failing to replicate a finding does not mean that the original finding is wrong,” Nosek emphasised. “One possibility is that the original finding is a false positive or exaggerated. Other possibilities are that we made an error in the replication study, or that unrecognised factors influenced the finding differently between the original and replication studies.”
The authors caution against interpreting the results as showing social and behavioural sciences are correct only about half the time. They stress that replication is not inherently good or bad; rather, it should be viewed as part of evaluating the total body of evidence for a claim. They also acknowledge that the studied corpus was not a perfectly random slice of social science, with some fields over- or under-represented due to practicality and data availability.
“I would be interested to see a project that tests replicability in a broader group of journals, broader research results like null and qualitative findings, and in different time periods of published results,” Zellers added. “These types of studies could help us understand drivers of replicability and unreplicability.”
The researchers propose that the core challenge is not a lack of replicability per se but overconfidence: treating a single published paper as established fact rather than provisional evidence. They advocate focusing on the broader evidence base, context and boundaries of the effect, rather than declaring an entire discipline in crisis.
Nosek reflected on the reception of the findings: “I was surprised that most of the research community readily accepted these findings … there was much less heated rhetoric. I think that is a sign of progress.”
The study, titled Investigating the replicability of the social and behavioural sciences, was led by Andrew H. Tyner, with Anna Lou Abatayo, Mason Daley, Samuel Field and many co-authors, and coordinated under the SCORE programme. It represents a milestone in systematic replication work, aiming to illuminate how best to balance discovery with replication as researchers navigate finite funding and the evolving landscape of open science.
