Network Meta-analysis of Antidepressants: A Bridge Too Far?

Volume 5 # 1 January 2019

 

Alan Eppel MB, FRCPC McMaster University

 

ABSTRACT

 

Network Meta-analysis (NMA) is a powerful  statistical tool that expands the methods of  pairwise Meta-analysis. In this overview NMA is explained for non-statisticians to allow physicians and other clinicians to critically appraise published NMAs. This requires understanding the underlying assumptions of NMA: indirect comparisons, transitivity, consistency, and effect modifiers. The recent NMA of 21 antidepressants (Cipriani et al, 2018) is discussed.

Disclosures of conflicts of interest:

None

Address correspondence or comments to [email protected]

and @PsychiatReform on Twitter.

Acknowledgements.

The author acknowledges with appreciation the assistance of Professor David Streiner McMaster University.

 

 

What is Network  Meta-analysis?

Network meta-analysis (NMA) is a powerful statistical tool that expands the methods of pairwise meta-analysis (MA). Conventional MA is based on the synthesis of data from randomized controlled trials consisting of direct comparisons of two drugs or between a drug and placebo. MA is limited to these pairwise comparisons that have been directly compared in head-to-head trials. NMA is a further development of the methodologies used in MA. NMA is designed to make comparisons of multiple treatments simultaneously.

The main theoretical addition that underlies NMA is the incorporation of indirect comparisons, based on common comparators, between medications. Mixed evidence refers to the combination of both direct and indirect evidence. In order to rely on indirect comparisons, NMA depends on several key assumptions.

Direct evidence refers to evidence from randomized controlled trials,  for example studies comparing  fluoxetine to venlafaxine. Indirect evidence is when there are no direct head to head comparisons  but  both drugs have been compared to a third drug that serves as a common comparator.

 

 

Drug A is directly compared to drug C. Drug B is directly compared to drug C. Drug C is a common comparator allowing an indirect comparison to be made between A and B (Fig1).

 

The goal of NMA is to determine the most effective intervention among all possible relevant interventions. This is important for clinicians, funding agencies and drug regulators. NMA attempts to rank interventions in the order of efficacy.

NMA has been touted as the pinnacle of methodologies in the hierarchy of  evidence for  the effectiveness of psychotropic medications (Leucht et al, 2016). Yet these claims must be qualified by the potential for introduction of error  at any of the multiple steps in the process ( Efthimiou et al, 2016; Egger et al, 2016; Jansen & Naci 2013; Jansen et al, 2014; Salanti et al, 2014; Tonin et al, 2017).

Clinicians may be overawed by the complexity of the analysis and find it difficult to appraise the validity of these methods. Physicians may not be aware that there is ongoing debate about different methodologies that are used in NMA because these are published in epidemiological journals such as the Journal of Clinical Epidemiology and the Journal of Research Synthesis which rely heavily on mathematical theory such as Bayesian Analysis and Metaregression. Many clinical journal article reviewers may not be equipped to assess these procedures. Nevertheless it is possible to appraise these studies by examining the individual RCTs regarding population characteristics, trial duration, and dosages of comparator medications.

 As in pairwise MA,  NMA involves critical decision choices based on underlying assumption  at multiple levels of the analytic process. NMA has additional levels of complexity which means researchers have to make more judgements about which techniques to utilize at each stage of the process. These assumptions are by no means trivial or undisputed and are not transparent to clinicians. This article  examines some of these assumptions in relation to a recent comprehensive  NMA  of 21 antidepressants (Cipriani et al, 2018).

 

The Transitivity Assumption

Transitivity refers to the condition in which the indirect comparisons validly estimates a true head-to-head comparison. Consistency is when both direct and indirect evidence agree. In other words it is possible to estimate the relative effects of two treatments A and B by comparison to the common comparator C.

 

For example (Fig 1) if drug A has been found to be directly superior to drug C , and if drug B has been found to be inferior to drug C,  then NMA leads to the conclusion that drug A is superior to drug B.  This is known as the Transitivity Assumption. Table 1 illustrates some of the key concepts of NMA.

 Table 1                                                       Network  Meta-analysis Key Concepts

Consistency	Agreement between the treatment effect estimates from head-to-head direct comparisons and treatment effect estimates obtained from indirect comparisons.
Effect modifiers Examples: Baseline illness severity Demographics of study populations Duration of trials Dosages of medications Measures of outcome	Study or patient characteristic that can influence the treatment effect. Effect modifiers can cause inconsistency in both MAs and NMAs  which may lead to lack of validity.
Transitivity	Transitivity is when there is no difference in the treatment effects  between analyses that provide  direct evidence of   treatment effect and those analyses that provide indirect evidence.
Treatment effect	Difference in outcome between an active intervention and a  comparator.
Loop of evidence	When 2 or more direct comparisons contribute to an indirect estimate. ABC in Fig 1 is a loop.

 

 

If there is little direct evidence for a particular drug then it is difficult to identify inconsistency. Indirect comparisons are not based on randomization and are influenced by differences in the trials. Consistency occurs when all the trials have been conducted in a similar fashion, where participants are similar on the relevant characters that might modify the treatment response such as disease severity, age, sex, concomitant treatments, duration of treatment, and dosages of medications. If these effect modifiers are different between the trials then the NMA  is not valid. If direct and indirect evidence are not consistent the transitivity assumption cannot be made.

 

The transitivity assumption can be evaluated by reviewing the distribution of effect modifiers across trials. The transitivity assumption implies that it is possible to combine direct evidence from AC and BC studies to learn about the indirect comparison AB. This will not be valid if there are important differences in the distribution of the effect modifiers which will impact the outcomes.

An alternative way of looking at the transitivity assumption is to state that it assumes that all included patients could in principal be randomized to any of the treatments within the network.

 

DOES TRANSITIVITY HOLD IN THE GAME OF TENNIS?

 The game of tennis can provide us with a partial analogy of transitivity. In the 2018 Wimbledon Tennis Tournament Novak Djokovic beat Kevin Anderson in the final. This was a direct head to head comparison between Djokevic and Anderson.

Anderson had beaten Roger Federer in the quarter final. Can we deduce that Djokovic would beat Roger Federer? That would be an indirect comparison. However there are other factors to consider.

Conditions may not be the same in each game: the  type of surface, clay or grass, temperature, wind conditions, umpire judgments etc. These factors are analogous to effect modifiers in NMA.

 

BIAS

Bias may occur in all study types including RCTs, MAs and NMAs. Three areas of bias are of particular importance:

 

• Publication bias
• Sponsorship bias
• Author allegiance bias and conflict of interest

 

Publication and sponsorship bias have been well established as problematic. Less often considered is author allegiance bias (Munder et al, 2012; Rasmussen et al, 2018, Shinohara et al, 2017). This was first studied in the psychotherapy literature but may well apply to pharmacological studies.

Allegiance bias refers to the preference of the therapist for one form of therapy versus another. For example in a comparison study between cognitive behavior therapy and psychodynamic psychotherapy the results may be influenced by the affinities of the study investigators. This may be a source of unconscious bias not only in the carrying out of the study but also when it comes to writing up the study. One particular example of allegiance bias is evident when the conclusions in the article abstract are not consistent with the data or discussion within the main body of the article. It is very important to know which authors participated in the final write-up  of  a study and whether there was any unconscious bias related to authors’ affiliations to particular modalities or commercial interests. This becomes critical when evaluating the effects of new medications.

 

John Ioannidis, at the Meta-Research Innovation Center at Stanford (METRICS), is one of the world’s leading experts on “research of research” He has written that many systematic reviews and MAs are misleading and can be used as marketing tools (Ioannidis, 2016). Ioannidis  identifies as areas of potential bias the choice of studies included in MAs, the nature of comparison populations, outcomes of interest, and the statistical methods used. This may lead to different results in MAs on the same  questions. The interpretation of the findings may also vary between authors.

 

In a review of 185 MAs ( Ebrahim et al, 2016)  found that 54 MAs (29%) involved authors who were employed by the manufacturer of the study drug. In 147 (79%) authors had links to industry or had conflicts of interest. Only 58 MAs (31%) had unfavourable conclusions of the study drug in the article abstract. Authors who were employees of the manufacturer of the assessed drug were 22-fold less likely to contain negative statements about the drug than other MAs.

 

NETWORK  META-ANALYSIS OF 21 ANTIDEPRESSANTS

 

A recent NMA  of 21 antidepressants published in the Lancet (Cipriani et al, 2018) has provoked lively debate.  The authors include some of the most eminent researchers in the field of NMA research: Andrea Cipriani, Toshi A Furukawa, Georgia Salanti, Anna Chaimani, Lauren Z Atkinson, Yusuke Ogawa, Stefan Leucht, Henricus G Ruhe, Erick H Turner, Julian P T Higgins, Matthias Egger, Nozomi Takeshima, Yu Hayasaka, Hissei Imai, Kiyomi Shinohara, Aran Tajika, John P A Ioannidis and John R Geddes.

The following disclosures were made in accordance with journal requirements:

“Toshi A Furukawa has received lecture fees from Eli Lilly, Janssen, Meiji, Mitsubishi-Tanabe, Merck Sharp & Dohme, and Pfizer; consultancy fees from Takeda Science Foundation; and research support from Mochida and Mitsubishi-Tanabe.

Stefan Leucht has received honoraria for consulting from  LB Pharma, Lundbeck, Otsuka, TEVA, Geodon Richter, Recordati,  LTS Lohmann, and Boehringer Ingelheim; and for lectures from Janssen,  Lilly, Lundbeck, Otsuka, SanoiAventis, and Servier.

 Nozomi Takeshima has received  lecture fees from Otsuka and Meiji.

 Yusuke Ogawa has received lecture fees from  Yoshitomi”.

 

This was an exhaustive NMA done over almost six years. It included 522 RCTs. The authors carefully gathered unpublished studies to reduce the effects of publication bias. Yet questions have been raised about the validity of the article conclusions. Of note the conclusions of the study ran counter to other findings and the published conclusions may have overreached the data. The ranking of efficacy of antidepressants were in fact not a basis to choose any particular drug as differences in the rankings did not exceed chance.

One major concern was the fact that a Cochrane review of vortioxetine had quite different conclusions than the NMA (De Giorgi, 2019; Fox, 2018; Koesters et al, 2017).  This raised questions about the validity of the methodology and whether or not there were unidentified biases in the way the conclusions were presented in the NMA paper.

By contrast the Cochrane review concluded in part:

“The place of vortioxetine in the treatment of acute depression is unclear. Our analyses showed vortioxetine may be more effective than placebo in terms of response, remission and depressive symptoms, but the clinical relevance of these effects is uncertain……. the quality of evidence to support these findings was generally low…”

In the Cipriani article NMA vortioxetine is ranked as having the highest odds ratio when placebo studies are excluded. Escitalopram and agomelatine  are also ranked high whereas fluoxetine ranked low.

Vortioxetine, escitalopram and agomelatine are manufactured by the same company, Lundbeck.

In the Cipriani study the assessment of effect modifiers appears limited in relation to variation in baseline severity measures. Also of 522 original  trials 380 (73%) were rated to have a moderate risk of bias, 46 (9%) as high risk of bias and only 96 (18%) were rated as low risk of bias.

Diagnostic consistency for depression is notoriously difficult in psychiatry. Some trials were limited to outpatients which had elements of variation. Others had up to 20% with other diagnoses  such as bipolar disorder, psychotic depression or comorbid medical illness.

Comparative drug dosages is another source of error. Average dose ranges were permitted. Comparisons with maximal dosages would be more relevant.

In the article and abstract, rank order of efficacy were featured prominently although such rankings are probabilistic and must be regarded with caution.

 

Some Factors to Consider in Appraising NMAs  

 

Physicians can begin to appraise published NMAs by examining effect modifiers. This may require examination of supplementary data available on the journal website (Al Khalifah et al, 2018).

 

Are there study population differences in any of the following?

• Baseline severity of illness
• Co-morbid psychiatric or medical illness
• Proportion of inpatients v outpatients
• Treatment setting
• Medication dosages
• Duration of treatment
• Concurrent medications
• Rescue medications
• Outcome measures
• Population demographics

 

Is there evidence for risk of bias?

• Conflict of interest
• Sponsorship
• Allegiance

 

CONCLUSIONS

 

Physicians need to be aware that existing NMAs in psychiatry cannot be relied upon for definitive answers to longstanding questions about the longterm effectiveness of psychotropic medications. There are multiple sources of error resulting from statistical assumptions and potential for bias.

 

References

 

Al Khalifah R,  Florez I, Guyatt G, Thabane L. Network meta-analysis: user’s guide for pediatricians. BMC Pediatrics. 2018, 18:180

Cipriani A, Furukawa TA, Salanti G, et al. Comparative efficacy and

acceptability of 21 antidepressant drugs for the acute treatment of adults with major

depressive disorder: a systematic review and network meta-analysis. Lancet 2018;

published online Feb 21. http://dx.doi.org/10.1016/S0140-6736(17)32802-7.

Cristea I , Ioannidis J. Improving Disclosure of Financial Conflicts of Interest for Research on Psychosocial Interventions. JAMA Psychiatry. 2018;75(6):541-542. doi:10.1001/jamapsychiatry.2018.0382

De Giorgi R. Vortioxetine for depression: the evidence for its current use in the UK. BJPsych Advances (2019), vol. 25, 3–6 doi: 10.1192/bja.2018.62

Ebrahim S, Bance S, Athale A, Malachowski C, Ioannidis JP. Meta-analyses with

industry involvement are massively published and report no caveats for

antidepressants. J Clin Epidemiol. 2016 Feb;70:155-63. doi:

10.1016/j.jclinepi.2015.08.021. Epub 2015 Sep 21.

Efthimiou O, Debray T, van Valkenhoef G et al. GetReal in network meta-analysis: a

review of the methodology. Res. Syn. Meth. 2016, 7 236–263

Egger M, Karel Moons G, Fletcher C, and GetReal Workpackage 4. GetReal: from efficacy in clinical trials to relative effectiveness in the real world. Res. Syn. Meth. 2016, 7 278–281

Fox J, Zhang P. Lancet. 2018 Sep 22;392(10152):1011. doi: 10.1016/S0140-6736(18)31787-2. Epub 2018 Sep 20.

González-Castro TB, Tovilla-Zárate C. Meta-analysis: A tool for clinical and experimental research in psychiatry. Nord J Psychiatry 2014;68:243–250.

Ioannidis J . The mass production of redundant, misleading and conflicted systematic reviews and meta-analyses: The Milbank Quarterly; vol 94 (3),  2016 page 485 – 514

Jansen J, Naci H. Is network meta-analysis as valid as standardpairwise meta-analysis? It all depends on the distribution of effect modifiers. BMC Medicine 2013, 11:159

http://www.biomedcentral.com/1741-7015/11/159

Jansen J, Trikalinos T, Cappelleri J et al. Indirect Treatment Comparison/Network Meta-Analysis Study Questionnaire to Assess Relevance and Credibility to Inform Health Care Decision Making: An ISPOR-AMCP-NPC Good Practice Task Force Report. Value in Health 17, 2014, 157-173 [International Society for Pharmacoeconomics and Outcomes Research and the the Academy of  Managed Care Pharmacy (AMCP)  and the National  Pharmaceutical  Council (NPC)

Koesters M, Ostuzzi G, Guaiana G, et al (2017) Vortioxetine for depression in adults. Cochrane Database of Systematic Reviews, 7: CD011520 (doi:10.1002/14651858.CD011520.pub2).

Leucht S, Chaimani A, Cipriani A,  et al. Network meta-analysis should be the highest level of evidence in treatment guidelines. Eur. Arch. Psychiatry Clin Neurosci (2016) 266; 477-480.

Munder, T., Flückiger C. Gerger H, Wampold B. E., & Barth, J. (2012). Is the allegiance effect an epiphenomenon of true efficacy differences between treatments? A meta-analysis. Journal of Counseling Psychology, 59(4), 631-637.

http://dx.doi.org/10.1037/a0029571

Rasmussen K, Bero l, Redberg R et al. Collaboration between academics and industry in clinical trials: BMJ 2018;363:k3654 | doi: 10.1136/bmj.k3654

Rouse B, Chaimani A, Li T. Network Meta-Analysis: An Introduction for Clinicians. Intern Emerg Med. 2017 February ; 12(1): 103–111. doi:10.1007/s11739-016-1583-7.

Salanti G, Giovane C, Chimani A et al. Evaluating the quality of evidence from network met-analysis. PLOS one 2014, vol 9, issue 7 e99682.

Shadish W ,Lecy J. The meta-analytic big bang. Res. Syn. Meth. 2015, 6 246–264

Shinohara K, Aoki T, So R, et al.  Influence of overstated abstract conclusions on clinicians: a web-based randomized controlled trial. BMJ Open 2017;7:e018355. doi:10.1136/bmjopen-2017-018355

Tonin F, Rotta I, Mendes A, Pontarolo R. Network meta-analysis: a technique to gather evidence from direct and indirect comparisons. Pharmacy  Practice 2017, 15 (1) 943.