Medical research: why so many studies rely on data no one can actually check
- Restricted access to empirical data undermines the validity of certain meta-analyses and, consequently, the reliability of medical studies.
- The demand for transparency often comes up against the need to protect participants’ data and regulatory constraints.
- This lack of accurate data has led to tens of millions of women being prescribed treatments that cause harm.
- These limitations also affect government decisions, with considerable financial consequences.
- Improving data transparency remains the most appropriate response to these uncertainties.
Every year, thousands of clinical recommendations, drug reimbursement decisions, and public health policies rest on meta-analyses. These studies pool results from many individual reports to draw a conclusion more robust than any single one could offer on its own. Although pillars of evidence-based medicine, they may be contributing to a systemic and widely underestimated problem because most rely on data that no one outside the original authors has ever been able to see or verify.
The idea behind a meta-analysis is simple enough. Researchers gather results from several studies on the same question to identify a more reliable overall trend. In doing so, they almost never have access to the raw data behind the original studies such as individual results for each patient or participant. Instead, they must work from the summary statistics published in the articles such as means, medians, standard deviations, which are often reported incompletely or inconsistently from one study to the next.
This lack of access forces researchers to rely on estimation and imputation methods to fill in the gaps. For instance, guessing at missing values based on statistical assumptions. Even when carried out carefully, these approximations introduce additional uncertainty that can distort the final results. Moreover, when several studies within the same meta-analysis require this kind of estimation, small errors can accumulate and, in some cases, completely change the conclusions, sometimes even reversing a result altogether.
A fragile link at the heart of evidence-based medicine
The problem isn’t a lack of rules. For several years now, many scientific journals and funding bodies have required researchers to commit to sharing their data. Although, in practice, these commitments largely go unfulfilled. A study of 7,750 biomedical research papers published between 2015 and 2020 found that only 9% of papers pledged to make their data publicly available and that among those, just 3% actually followed through.
The picture becomes even more striking when you look closely at what happens once a paper officially promises to share its data. In another study analysing 3,556 articles, 1,792 manuscripts included an explicit pledge to share data. Among the authors who were then contacted for that data, 93% either did not respond or declined. In the end, only 6.8% of authors provided the requested datasets. In other words, the data-sharing statement now found in nearly every publication almost never translates into real access.
In some settings, legal and regulatory constraints imposed by ethics committees, governments, or data governance frameworks genuinely limit what can be shared, particularly when data span multiple countries or come from large cohort studies and biobanks.
The reasons behind this lack of transparency are varied, and they don’t always reflect bad faith. In some settings, legal and regulatory constraints imposed by ethics committees, governments, or data governance frameworks genuinely limit what can be shared, particularly when data span multiple countries or come from large cohort studies and biobanks. Protecting participants’ privacy and honouring the terms of their original consent can also restrict what can be made public. On top of that, there are more subjective factors at play: some researchers worry about losing ownership of their data, are uncertain how it might be reused, or fear intense scrutiny of their original analyses. Limited data availability, then, reflects a tangled mix of structural, ethical, and cultural constraints, not simply a refusal to cooperate.
Missing data can change the course of medicine
The publication in question1 draws on a particularly striking historical example to illustrate the real-world consequences of this problem: menopausal hormone replacement therapy. For nearly two decades, meta-analyses based on aggregate data from observational studies suggested that this therapy reduced the risk of coronary heart disease in menopausal women by almost 50%. That conclusion shaped clinical guidelines worldwide and influenced the care of tens of millions of women. In the most influential meta-analysis of the period, the pooled relative risk of coronary heart disease among hormone therapy users was estimated at 0.56 — an impressive figure that reinforced the narrative of cardio-protection.
That narrative collapsed, however once detailed, participant-level data finally became accessible through the Women’s Health Initiative randomised trial in the United States. With full access to adjudicated outcomes, confounding factors, and time-to-event data, the trial revealed that combined oestrogen-progestin therapy increased the risk of coronary events, stroke, pulmonary embolism, and breast cancer. The reversal was so dramatic that the trial had to be halted early on safety grounds.
What makes this episode particularly instructive is that it wasn’t a case of unpublished studies or selective reporting: every major observational study involved was, in fact, publicly available. What was missing were the underlying datasets needed to detect confounding, test alternative models, and examine effects across different subgroups of women. Working only from published summary statistics, researchers had produced estimates that looked confident and reproducible, but that turned out to be inconsistent with evidence from far more rigorous randomised trials after shaping treatment decisions for decades. Hormone therapy became one of the most widely prescribed treatments in high-income countries during the 1990s, representing billions of dollars in annual spending. When the Women’s Health Initiative results came out, prescribing rates collapsed within months.
Consequences that go well beyond academic debate
Far from being a purely methodological dispute among researchers, this lack of transparency has very real consequences for health policy. Government agencies and public health bodies frequently base reimbursement and coverage decisions on meta-analytic findings, which directly shapes how healthcare resources (and sometimes entire markets) are allocated. For instance, according to the Institute for Clinical and Economic Review, unjustified price increases on certain drugs in the United States, made without any new evidence of clinical benefit, added an estimated $815 million to healthcare spending in 2023 alone.
The long-running debate over statin safety offers a telling illustration. These widely prescribed cholesterol-lowering drugs have been at the centre of persistent disagreements over how often side effects occur and how clinically significant they really are. Such disagreements have fuelled conflicting interpretations of the same evidence and eroded public trust in treatment recommendations, alongside complicating clinical practice. Addressing this calls for a progressive roadmap toward genuine data transparency in research. The most immediate and low-cost step is simply to enforce rules that already exist: data and analytic code should be made available at the time of publication, and manuscripts that fail to meet this commitment should be rejected outright, barring a clearly justified legal or ethical constraint.
Beyond this first step lie changes to the broader research ecosystem. Data sharing needs to be embedded into how research institutions are evaluated, funder requirements need strengthening, and the possibility of future data sharing and reuse should be built into the ethics and consent process from the very start of a study, not bolted on afterward. On the infrastructure side, data and code should be systematically deposited in recognised, trustworthy repositories (Zenodo, the Open Science Framework, or Dryad among them) with documentation thorough enough to allow meaningful reuse rather than mere archiving. A more ambitious step still would introduce dedicated editorial processes to verify the availability and quality of deposited data before publication: essentially, a formal quality check woven into peer review itself.
None of these measures, on their own, would have prevented every questionable price hike or resolved every statin debate. But taken together, they would shift the burden of proof back to where it belongs: on the evidence.
