PUBLICATION BIAS UNDERMINES EVIDENCE BASED MEDICINE
- Posted by Alan Eppel
- Posted in Papers
Mark Bennett MD, FRCPC
The bias towards publishing positive results in research is well known, yet despite this nearly half of all clinical trial data goes unpublished. The consequences are severe; biasing clinicians, harming patients, and affecting health care costs.
Without access to all clinical trial data harmful effects of treatments can go undetected. Examples of this include the COX-2 inhibitors and the diabetes drug rosiglitazone. In the case of rofecoxib cases of myocardial infarction and causes of death were missing from clinical trial reports, and the association between the drug and heart attacks was undetected, until the drug was eventually pulled from market1.
Meta-analysis that included unpublished data examining the cardiac effects of celecoxib also found an association with myocardial infarction2. In 2004, a legal settlement required the makers of rosiglitazone to make available all clinical trial results. A meta-analysis with access to this data, showed a significant association with the drug to MI
and cardiovascular death3.
Publication bias misleads clinicians and ultimately their patients by inflating effect size estimates of the treatments they provide. This effects the risk-benefit discussion clinicians have with their patients.
Turner et al.4 demonstrated this in a meta-analysis that accessed all phase 2 and 3 trials for 12 antidepressant agents approved by the FDA between 1987-2004, accessed through Freedom of Information Act requests. Of these 74 trials, 38 were deemed by the FDA to support the efficacy of the drug. Of those 37 were published in academic journals. The remaining 36 trials were deemed negative or questionable by the FDA. Of those 22 were unpublished while the others were published as positive conflicting with the FDA’s assessment. When effect size estimates from these trials do not include unpublished trial data, they are overestimated4. One might think that trials were unpublished due to methodological flaws, however unpublished trials were not found to differ significantly in terms of sample size or methodology. Systematic review and meta-analysis were revolutionary in terms of their impact on modern medicine, and they are undermined by publication bias4.
Perhaps the example of publication bias that has gathered the most media attention is that of the anti-viral oseltamivir. Billions of dollars of the drug have been stockpiled by governments across the globe in preparation for influenza outbreaks, generating approximately 18 billions dollars in sales for its maker. These decisions were made without approximately 60% of the clinical trial data.
However following a five-year campaign by Cochrane reviewers and the British Medical Journal, the full data set was made available, and the revised meta-analysis suggested the small benefits provided by Tamiflu do not outweight the harms5.
Why does publication bias occur? It’s commonly believed that medical journals are more likely to publish clinical trials if they are positive. However the largest systematic review of studies investigating publication bias did not find this to be the case6. In a pooled analysis of four prospective studies following clinical trials submitted for publication, the direction of the results did not predict publication6.
Inception cohort studies however, where trials are followed to publication from their design, found that trials were approximately three times more likely to be published if positive. This suggests that it is the investigator’s decision to publish, more so than editorial decisions, that leads to publication bias6.
When authors are surveyed regarding their reasons for not submitting articles for publication, most common reasons include, lack of time or low priority, results not seeming important enough, and fear of journal rejection. Finally industry sponsorship has been repeatedly associated with publication bias, with positive industry trial found be three to four times more likely to be published if positive6.
Not only are the consequences of publication bias severe, methods in detection, such as funnel plots, are often ineffective7. Efforts in prevention have focused primarily on clinical trial registries. Clinicaltrials.gov and similar online clinical trial registries in Europe were created in the early 2000’s. In 2005 the International Committee of Medical Journal Editors (ICMJE) required, in a nonbinding agreement, that all clinical trials should be indexed in a clinical trial registry to qualify for publication in a journal.
Perhaps, the legislation with the most potential for increasing access to clinical trial data was the FDA Amendment Act (2008). This legislation required that all clinical trial results be disseminated on clinicaltrials.gov within one year of the studies completion. However Gopal et al8 evaluated the rate of compliance with the act and found only 19.1% of trials had available results in 2007-8 and 10.8% in 2008-9. A more recent systematic review of interventions against publication bias demonstrated that although the use of prospective trial registration has been increasing, the over representation of positive results in the published literature persists9.
Within psychiatry, Scott et al.10reviewed the registry information of all published clinical trials in five journals adhering to the ICMJE agreement, including The American Journal of Psychiatry, Archives of General Psychiatry/JAMA Psychiatry, Biological Psychiatry, The Journal of Child and Adolescent Psychiatry, and the Journal of Clinical Psychiatry; 181 trials in total. It was found that only 15% of trials were registered prospectively, without any changes to their primary outcome measure.
Advances in evidence-based medicine, such as systematic review and meta-analysis have shaped clinical practice in dramatic ways. With new technologies and decision making tools evidence is more accessible to clinicians to inform their patients better. These tools however rely on access to all the clinical trial data to truly be effective; otherwise they may be leading physicians and their patients astray.
Public awareness of the impact of publication bias is increasing, much in thanks to the work of the AllTrials campaign11, an international organization founded by psychiatrist and epidemiologist Dr. Ben Goldacre, which calls for all clinical trials in humans to be registered and results be reported. Systems are in place to achieve this but monitoring and enforcement are lacking. There is an urgency to implement these measures to improve the health and safety of our patients.
References
11. Topol EJ. Failing the Public Health — Rofecoxib, Merck, and the FDA. N Engl J Med 2004; 351:1707-170.
22. Caldwell B, Aldington, S, Weatherall M, Shirtcliff, P, Beassley R. J R Soc Med 2006, 99: 132-140
33. Nissen SE, Wolski K. N Engl J Med 2007, 356:2457-2471
44. Turner EH, Matthews AM, Linardatos E, Tell RA, Rosenthal. N Engl J Med 2008; 358(3):252-60.
55. Jefferson T, Jones M, Doshi P, Spencer EA, Onakpoya I, Heneghan CJ BMJ2014;348:g2545.
66. Song F, Parekh S, Hooper L, Loke YK, Ryder J, Sutton AJ, Hing C, Kwok CS, Pang C, Harvey I. Dissemination and publication or research finding: an updated related biases. Health Technology Assessment 10 (9); 2010.
77. Terrin N, Schmid CH, Lau J. In an empirical evaluation of the funnel plot, researchers could not visually identify publication bias. J Clin Epidemiol 2005.
88. Gopal RK, Yamashita TE, Prochazka AV. Research without results: Inadequate public reporting of clinical trial results. Contemp Clin Trials 2012; 33 (3):486-491.
99. Thaler K, Kien C, Nussbaumer B, Van Noord MG, Griebler U, Klerings I, Gartlehner G. Inadequate use and regulation of interventions against publication bias decreases their effectiveness: a systematic review. J Clin Epidemiol 2015; 68(7):792-802.
110. Scott A, Rucklidge J, Mulder R. Is Mandatory Prospective Trial Registration Working to Prevent Publication of Unregistered Trials and Selective Outcome Reporting? An Observational Study of Five Psychiatry Journals That Mandate Prospective Clinical Trial Registration. PloS One 2015:e0133718.
111. alltrials.net
