The evidence underpinning modern clinical practice rests on a fragile foundation: systematic reviews and meta-analyses that combine data from hundreds of studies to guide treatment decisions, yet according to a policy analysis published in PLOS Medicine, these syntheses are increasingly compromised by researchers’ inability to access the raw datasets and analytical code used in original investigations. When underlying data remains locked behind institutional barriers or researcher discretion, evidence synthesizers are forced to rely on incomplete summary statistics and unverifiable assumptions—introducing distortions that can mislead clinicians and policymakers about what treatments actually work.
Key takeaways
- Systematic reviews and meta-analyses—the foundation of clinical guidelines—often lack access to underlying raw data and analytical code from original studies
- Incomplete or inconsistently reported summary statistics force researchers to use imputation and unverifiable assumptions, which can distort effect estimates
- Historical cases such as hormone replacement therapy demonstrate how flawed evidence synthesis can influence treatment recommendations, healthcare spending, and patient safety
- Despite widespread data-sharing policies, compliance is low, enforcement is weak, and monitoring is almost non-existent
- Stronger enforceable data-sharing mechanisms and pragmatic verification approaches within editorial workflows are urgently needed
The Data Transparency Gap in Evidence Synthesis
Barriers preventing access to underlying research datasets in systematic reviews, by stage
Source: PLOS Medicine Policy Forum, 2024 | Georgian Medical Journal News
Why Data Transparency Matters to Clinical Practice
Systematic reviews and meta-analyses hold an outsized influence over healthcare. Guidelines from major organizations including the World Health Organization, the US National Institutes of Health, and the UK National Institute for Health and Care Excellence (NICE) rely directly on these syntheses to formulate treatment recommendations that affect millions of patients globally. When the underlying datasets remain inaccessible, evidence synthesizers cannot verify the original analyses, recalculate effect sizes, or detect errors in statistical reporting.
According to the PLOS Medicine analysis by researchers including Saul Martin-Rodriguez and David Moher, this limitation forces meta-analysts to rely on imputation strategies—educated guesses about missing data—and unverifiable assumptions about how original studies were conducted. These workarounds introduce systematic bias into effect estimates, subtly distorting the very conclusions that clinicians use to decide which treatments to prescribe.
Reliance on incomplete or inconsistently reported summary statistics forces researchers to use imputation and unverifiable assumptions, which can distort effect estimates and mislead clinical decision-making.
— Saul Martin-Rodriguez, David Moher, and colleagues, PLOS Medicine (2024)
Historical Lessons: When Flawed Evidence Synthesis Harms Patients
The consequences of poor data transparency are not theoretical. The PLOS Medicine article cites the hormone replacement therapy (HRT) case as a cautionary example: for decades, meta-analyses of observational studies suggested that HRT reduced cardiovascular risk in postmenopausal women, leading to widespread prescribing. When large randomized controlled trials finally provided access to robust, verifiable data, the evidence reversed—HRT was found to increase cardiovascular and breast cancer risk in many populations. This reversal led to a dramatic decline in HRT use and prompted a painful reassessment of patient safety across millions of treated women.
The HRT case illustrates a broader vulnerability: when data synthesis relies on incomplete information and unverifiable assumptions, the resulting recommendations can mislead clinicians and harm patients. According to the PLOS Medicine policy forum, flawed evidence synthesis can influence not only treatment recommendations but also healthcare spending and resource allocation, with cascading economic and clinical consequences.
The Policy-Practice Gap: Data-Sharing Promises Unfulfilled
Major funders and publishers have issued data-sharing mandates for years. The US National Institutes of Health requires data-sharing plans for federally funded research, and leading journals including The Lancet, The BMJ, and others have adopted data-availability policies. Yet the PLOS Medicine analysis reveals that compliance remains low, enforcement is weak, and monitoring is almost non-existent.
Researchers cite practical obstacles: privacy concerns, lack of standardized data formats, inadequate institutional infrastructure, and legitimate intellectual property interests. Publishers struggle to verify compliance without dedicated resources. The result is a fragmented ecosystem in which some researchers openly share data while others claim confidentiality or simply do not respond to requests—with little consequence.
What this means
Toward Enforceable Transparency: Practical Reforms
The PLOS Medicine policy forum argues for strengthening enforceable data-sharing mechanisms, including clearer enforcement pathways and pragmatic verification approaches within editorial workflows. Rather than relying on voluntary compliance, the authors propose that journals, funders, and institutions implement concrete accountability measures: mandatory data-sharing as a condition of publication, regular audits of compliance, and clear consequences for non-compliance.
Pragmatic verification could occur at the point of peer review and editorial assessment, with meta-analysts and systematic reviewers empowered to request raw data during the manuscript evaluation process. Funding agencies could condition grant renewal on demonstrated data-sharing. Institutional review boards could integrate data-sharing requirements into research ethics approval. These mechanisms would shift data transparency from a rhetorical commitment to an enforceable standard, closing the gap between policy aspiration and practice reality.
The challenge is significant but surmountable. Standardized data-sharing platforms, clear confidentiality protections, and shared investment in infrastructure can reduce researcher burden. Professional societies and editorial boards can clarify expectations and provide guidance. Most importantly, the evidence base for clinical practice—and ultimately patient safety—depends on moving beyond the current regime of weak compliance and non-existent enforcement toward a healthcare research ecosystem in which transparency is the default and inaccessibility requires explicit justification.
Source: The data transparency crisis in research: Lessons from systematic reviews and meta-analyses, PLOS Medicine (2024)
Frequently asked questions
Why do systematic reviews need access to raw data?
Systematic reviewers combine findings from multiple studies to synthesize evidence for clinical practice. Access to raw datasets and analytical code allows reviewers to verify that the original analyses were conducted correctly, recalculate effect estimates if needed, and detect errors or inconsistencies in reporting. Without this access, reviewers must rely on imputed or assumed values, which can introduce bias and distort the conclusions used in clinical guidelines.
How does the HRT case demonstrate the risks of data opacity?
For decades, meta-analyses of observational studies suggested that hormone replacement therapy reduced cardiovascular risk in postmenopausal women, leading to widespread prescribing. When large randomized controlled trials with verifiable data became available, the evidence reversed—HRT was found to increase cardiovascular and breast cancer risk in many populations. This reversal would have been detected earlier if the underlying data from original studies had been transparent and accessible to evidence synthesizers.
What practical steps can enforce data-sharing?
Enforceable mechanisms include: mandatory data-sharing as a condition of journal publication; regular audits of compliance by funding agencies; integration of data-sharing requirements into research ethics approval; and pragmatic verification by peer reviewers and meta-analysts during manuscript evaluation. Professional societies and editorial boards can clarify expectations, and standardized platforms can reduce researcher burden. The goal is to shift data transparency from a voluntary commitment to an institutional standard with clear consequences for non-compliance.
Was this article helpful?
Disclaimer. This article is health journalism intended for general information and education. It is not medical advice and is not a substitute for professional diagnosis or treatment. Always consult a qualified healthcare provider about your individual circumstances. Full disclaimer →
Related Coverage
Medically reviewed by Prof. Giorgi Pkhakadze, MD, MPH, PhD. Spotted an error? Contact the editorial team.
