INNOVATIONS in pharmacy

Imaginary Worlds: A Systematic Review of the Status of Modeled Cost-Effectiveness Claims Published in the Journal of Medical Economics from January 2015 to December 2015

Paul C Langley

University of Minnesota

Taeho Greg Rhee

University of Minnesota, Twin Cities

DOI: https://doi.org/10.24926/iip.v7i2.439

Keywords: cost-effectiveness modeling, credibility, imaginary worlds, scientific method

---

Abstract

The purpose of this review is to evaluate the credibility of modeled claims for cost-effectiveness studies published in the Journal of Medical Economics (JME) in the period January 2015 to December 2015. Credibility is assessed in terms of the standards of normal science. Are the claims made capable of falsification and replication? Following the PRISMA-P recommendations, abstracts of all papers published in the JME were evaluated by both authors independently against a MeSH terms checklist: cost, cost-effectiveness, cost-utility, outcomes, QALY and Markov. A total of 32 studies were identified. A systematic review of each study was then evaluated. The consensus was that all studies be included in the review. Each study was judged against four criteria: (i) Is the model capable of generating evaluable claims? (ii) did the author(s) attempt to generate evaluable claims? (iii) did the author(s) suggested how the claims might be evaluated? and (iv) did the author(s) caution readers as to the implications of generating non-evaluable clams for the credibility of the analysis? None of the studies presented their claims or projections in an evaluable form and none suggested how they might be evaluated. None met the standards of normal science. The claims made for cost-effectiveness were either impossible to verify, or if potentially verifiable, were not presented in an evaluable form. The studies lacked credibility. There was no basis for assessing whether the claims were right or even if they were wrong. The JME is not alone. As part of an ongoing systematic review program, covering the leading journals and focusing on chronic condition areas, it is clear that the majority of cost-effectiveness papers also fail the test of credibility. Of course, it might be that the consensus opinion among practitioners is that non-evaluable projections are acceptable. Evidence is not discovered, but is constructed through models and simulations. This lack of scientific credibility is a major concern. If medical economics is to advance through the formulation and testing of hypotheses, then editors of journals should consider whether or not to set standards for the acceptance of publications to include the requirement for evaluable claims and the results of claims assessment. If this is not acceptable, then it should be made clear that published papers are simply imaginary worlds or thought experiments.

 

Type: Commentary