Understanding the Value of Individualized Information: The Impact of Poor Calibration or Discrimination in Outcome Prediction Models.

Pubmed ID: 28399375

Pubmed Central ID: PMC5577366

Journal: Medical decision making : an international journal of the Society for Medical Decision Making

Publication Date: Oct. 1, 2017

MeSH Terms: Humans, Randomized Controlled Trials as Topic, Risk Assessment, Treatment Outcome, Computer Simulation, Cost of Illness, Myocardial Infarction, Cost-Benefit Analysis, Quality-Adjusted Life Years, Decision Support Techniques, Decision Making, Fibrinolytic Agents, Tissue Plasminogen Activator, Streptokinase

Grants: U01 NS086294

Authors: Wong JB, Kent DM, Olchanski N, Cohen JT, Neumann PJ

Cite As: Olchanski N, Cohen JT, Neumann PJ, Wong JB, Kent DM. Understanding the Value of Individualized Information: The Impact of Poor Calibration or Discrimination in Outcome Prediction Models. Med Decis Making 2017 Oct;37(7):790-801. Epub 2017 Apr 11.

Studies:

Abstract

BACKGROUND: Risk prediction models allow for the incorporation of individualized risk and clinical effectiveness information to identify patients for whom therapy is most appropriate and cost-effective. This approach has the potential to identify inefficient (or harmful) care in subgroups at different risks, even when the overall results appear favorable. Here, we explore the value of personalized risk information and the factors that influence it. METHODS: Using an expected value of individualized care (EVIC) framework, which monetizes the value of customizing care, we developed a general approach to calculate individualized incremental cost effectiveness ratios (ICERs) as a function of individual outcome risk. For a case study (tPA v. streptokinase to treat possible myocardial infarction), we used a simulation to explore how an EVIC is influenced by population outcome prevalence, model discrimination (c-statistic) and calibration, and willingness-to-pay (WTP) thresholds. RESULTS: In our simulations, for well-calibrated models, which do not over- or underestimate predicted v. observed event risk, the EVIC ranged from $0 to $700 per person, with better discrimination (higher c-statistic values) yielding progressively higher EVIC values. For miscalibrated models, the EVIC ranged from -$600 to $600 in different simulated scenarios. The EVIC values decreased as discrimination improved from a c-statistic of 0.5 to 0.6, before becoming positive as the c-statistic reached values of ~0.8. CONCLUSIONS: Individualizing treatment decisions using risk may produce substantial value but also has the potential for net harm. Good model calibration ensures a non-negative EVIC. Improvements in discrimination generally increase the EVIC; however, when models are miscalibrated, greater discriminating power can paradoxically reduce the EVIC under some circumstances.