ReproducibiliTea: A grassroots movement promoting open science

During their PhD studies in biomedicine, Jane Fisher and Daniela Grassi developed an interest in understanding how scientists conduct their research. For a long time, society's focus on science has primarily been on the results—after all, it's the easiest way for most people to assess whether research is valuable. However, are all published research findings accurate?

Can we trust scientific publications?

In 2006, a paper published in Nature claimed that a specific protein known as amyloid-β oligomer 56 (Aβ*56) was related to Alzheimer’s disease (1). This “landmark paper” became the foundation for many subsequent studies on treatments for Alzheimer’s disease (2). But with more and more labs reporting that they were unable to replicate these results, researchers found that the Aβ*56 oligomer was an “elaborate mirage.’’ This was only after the National Institutes of Health and pharmaceutical companies had invested millions or even billions of dollars to develop drugs targeting soluble Aβ. It not only dashed the hopes of 55 million people with Alzheimer’s disease and their families, but also slowed the race to save lives.  “You can cheat to get a paper. You can cheat to get a degree. You can cheat to get a grant. You can’t cheat to cure a disease,” Holden Thorp, editor-in-chief of Science said. “Biology doesn’t care” (6).

According to Jane and Daniela, reproducibility and replicability are the benchmarks of good science. Although sometimes the terms “reproducibility” and “replicability” can be used interchangeably, there are some differences between them. Reproducibility refers to obtaining the same results by reanalyzing existing data with the same methods, ensuring transparency and accuracy (3). Replicability involves conducting the same study with new data and still achieving the same results, demonstrating the reliability of the original findings. Reproducibility and replicability are vital to ensuring that the conclusions drawn from research are trustworthy.

Unfortunately, the Alzheimer’s case mentioned above is not an isolated one. Across various disciplines in social and biomedical sciences, studies carried out later by other scientists have failed to replicate published results, making them question the conclusions of the original studies. In a 2016 survey published in Nature, 90% of respondents agreed that a "reproducibility crisis" exists. Although this description is debatable, reproducibility is truly facing some threats (1).

This situation refers to the high number of studies that cannot be reproduced. This phenomenon is causing researchers to lose confidence in the integrity of many published findings. According to the Nature survey, over 70% of researchers reported being unable to reproduce the findings of other scientists (1).

On the other hand, some scientists think that the current outpouring of concern about replicability is overblown. One of the arguments is that science can self-correct. This is the conclusion reached by Pashler and Harris from the University of California, San Diego, who wrote: “Although some erroneous results may get published, eventually these will be discarded. Current discussions of a replicability crisis reflect an unreasonable impatience” (5).

So, are these issues overblown?

A grassroots solution

Since it is a widespread problem in research, Jane and Daniela believe that instead of waiting for science to self-correct, people could take certain measures to prevent this situation, thus avoiding worse outcomes or further waste of resources. While they were exploring ways to address various issues related to metascience—the study of scientific research aimed at understanding and improving scientific practices—they discovered that numerous replication attempts have shown that a significant portion of findings are not reproducible. With many studies lacking adequate reporting, it is difficult for others to interpret the results or replicate the methods. In this case, this is a challenge many scientists encounter, whether in the lab or when reviewing publications.

Several solutions have been proposed to address the reproducibility crisis, including open science initiatives, pre-registration of studies, and an increased commitment to data sharing and transparency. Additionally, some researchers believe that grassroots initiatives promoting bottom-up reform could be the solution. One such initiative is the ReproducibiliTea journal club.

Jane and Daniela discovered ReproducibiliTea through the Open Science and metascience networks on LinkedIn. Originally founded in the UK, this international journal club initiative has expanded to universities worldwide, fostering discussions on open science and reproducibility. According to its official website, the goal of ReproducibiliTea is to help early career researchers build a local community for those interested in open and reproducible research (7). It also aims to reduce the feeling of isolation within researchers who are actively engaged or have interest in improving research practices.

In order to inspire researchers to explore reproducibility challenges and stay informed about current initiatives working to address them, Jane and Daniela started the Lund Biomedicine ReproducibiliTea journal club, which contributes by bringing these critical conversations to one of Sweden’s largest universities and its surrounding research communities. The club offers researchers a supportive environment that brings like-minded individuals together to discuss the latest research topics, such as the threats to reproducibility, practices to enhance research integrity, diversity, equity, inclusion, and open science (8).

ReproducibiliTea comes to Lund

The first session of the Lund Biomedicine journal club was in September. Jane and Daniela chose a paper titled “A manifesto for reproducible science” (9) for the first meeting because they felt it broadly addressed several aspects of the reproducibility crisis. The session focused on the challenges and opportunities surrounding reproducibility and research quality. Those present discussed the benefits of study pre-registration and the limitations of current open science initiatives. A notable part of the discussion centered on how initiatives and practices differ across various scientific disciplines. Furthermore, they plan to offer an online option for future meetings to make it more accessible and inclusive for a wider audience to attend.

The impact of ReproducibiliTea extends beyond mere discussions. As one of the ReproducibiliTea advisory board members Amy Orben remarked, “These conversations can become a crucial line of support for young scientists” (4). For many researchers, these meetings have offered new insights, improved methodologies, and a sense of community support in their reproducibility struggles. The idea of ReproducibiliTea is not only to provide researchers with resources, but also to connect with others who face similar challenges. Practical, real-world changes may not always succeed every time, but ReproducibiliTea discussions foster a sense of belonging in an often isolating field.

The continuing problem

Despite increasing awareness, the reproducibility crisis remains a pressing concern across various scientific fields. Recent studies indicate that while efforts are underway, the problem persists, underscoring the need for ongoing awareness initiatives like ReproducibiliTea. This kind of simple but effective event provides researchers with a low-barrier entry platform for people at different stages of their careers to participate in discussions, thereby facilitating the broader dissemination of ideas and support.

However, as a grassroots organization without funding support or widespread promotion, ReproducibiliTea has its limits. While existing ReproducibiliTea journal clubs serve as a valuable platform for discussion, most are based in Europe, with limited representation in Asia, the Americas, and Africa. The limited reach may make it difficult for researchers in places that lack such organizations to find platforms for discussing reproducibility, which doesn’t help address the reproducibility crisis. “When I found that there are only a few clubs in North America and Asia, I was very surprised," said medical writer Sean Kim, PhD. "You can imagine how many researchers there are there.” 

While the club primarily focuses on reading and discussing papers to understand the issues and explore potential solutions, it does not serve as a platform for direct action. It is up to individual researchers to apply the solutions they discuss. Many current challenges to achieving reproducible research, such as funding pressures and journal expectations, must be addressed at a systemic level.

Beyond the ReproducibiliTea initiative

To address issues in research practices, Jane and Daniela founded AdvanSci Research Solutions, a company dedicated to transforming innovative ideas into impactful, reproducible science. AdvanSci provides consulting services that encourage good research practices, medical writing services that adhere to published reporting guidelines, and has developed a web application designed to assist researchers with study design and reporting. AdvanSci aims to address the reproducibility problem not only through paid services, but also through educating researchers at large through their social media pages and, of course, by bringing a ReproducibiliTea journal club to Lund.

Reproducibility today

“In science, once you publish your data, if it’s not readily replicated, then there is real concern that it’s not correct or true,” says Dennis Selkoe, MD,  a physician at Harvard University who tried but could not find the Alzheimer’s disease protein Aβ*56 in human samples (6). In July 2024, Lesné et al.’s 2006 Nature article on Aβ*56 was retracted by the authors due to “concerns…regarding figures” which “cannot be verified from the records” (10).

Growing awareness of the reproducibility crisis in science is gradually driving change. At this moment, it’s impossible to predict whether current awareness efforts, such as the ReproducibiliTea journal club initiative, will succeed in resolving the reproducibility crisis. While the crisis remains a pressing concern, current discussions offer a roadmap to rebuilding trust in scientific research. Although there is still a long way to go, we are already on the path of progress.

References

1. Baker, M. (2016) 1,500 scientists lift the lid on reproducibility. Nature 533, 452–454. https://doi.org/10.1038/533452a.

2. DESAI, B. (2023) Alzheimer Disease research and Aβ*56: The star that never was. Indian Journal of Medical Ethics, IX (1 (NS)), 44. Retrieved from https://ijme.in/articles/alzheimer-disease-research-and-a%ce%b256-the-star-that-never-was/.

3. Nikolopoulou, K. (2022) Reproducibility vs Replicability | Difference & Examples. Available online: https://www.scribbr.com/methodology/reproducibility-repeatability-replicability/.

4. Orben, A. (2019) ‘A journal club to fix science’, Nature: International weekly journal of science, 573(7775), p. 465. doi:10.1038/d41586-019-02842-8.

5. Pashler, H., & Harris, C. R. (2012) Is the Replicability Crisis Overblown? Three Arguments Examined. Perspectives on Psychological Science, 7(6), 531-536. https://doi.org/10.1177/1745691612463401.

6. Piller, C. (2022) Blots on a field? A neuroscience image sleuth finds signs of fabrication in scores of Alzheimer’s articles, threatening a reigning theory of the disease. Available online: https://www.science.org/content/article/potential-fabrication-research-images-threatens-key-theory-alzheimers-disease.

7.    ReproducibiliTea journal club official website: https://reproducibilitea.org/about/.

8.    Fisher, J. & Grassi, D. (n.d.) AdvanSci: advancing science with scientific advice, medical   writing and editing, and study management, Interview with Kate Gardner. Available at: https://www.scientifyresearch.org/blog/advansci-advancing-science-scientific-advice-medical-writing-editing-study-management/.

9.   Munafò, M., Nosek, B., Bishop, D. et al. (2017) Manifesto for reproducible science. Nat Hum Behav 1, 0021. https://doi.org/10.1038/s41562-016-0021.

10.  Lesné, S., Koh, M. T., Kotilinek, L., Kayed, R., Glabe, C. G., Yang, A., Gallagher, M., & Ashe, K. H. (2006). A specific amyloid-beta protein assembly in the brain impairs memory. Nature, 440(7082), 352–357. https://doi.org/10.1038/nature04533 (Retraction published Nature. 2024 Jul;631(8019):240. doi: 10.1038/s41586-024-07691-8)

Author

Wanyi Zhu is a master's student in Media and Communication at Lund University. She is passionate about creative storytelling and fostering meaningful connections through communication.

Editor

Rebeca Cardoso, PhD, is a research consultant at AdvanSci Research Solutions. Committed to making science accessible, Rebeca strives to bring cutting-edge research to a broader audience through her writing and outreach initiatives.