An official website of the United States government
Understanding the Value of Individualized Information: The Impact of Poor Calibration or Discrimination in Outcome Prediction Models.
Pubmed ID: 28399375
Pubmed Central ID: PMC5577366
Journal: Medical decision making : an international journal of the Society for Medical Decision Making
Publication Date: Oct. 1, 2017
Affiliation: Institute for Clinical Research and Health Policy Studies, Tufts Medical Center, Boston, MA (NO, JTC, PJN, JBW, DMK).
MeSH Terms: Humans, Randomized Controlled Trials as Topic, Risk Assessment, Treatment Outcome, Computer Simulation, Cost of Illness, Myocardial Infarction, Cost-Benefit Analysis, Quality-Adjusted Life Years, Decision Support Techniques, Decision Making, Fibrinolytic Agents, Tissue Plasminogen Activator, Streptokinase
Grants: U01 NS086294
Authors: Wong JB, Kent DM, Olchanski N, Cohen JT, Neumann PJ
Cite As: Olchanski N, Cohen JT, Neumann PJ, Wong JB, Kent DM. Understanding the Value of Individualized Information: The Impact of Poor Calibration or Discrimination in Outcome Prediction Models. Med Decis Making 2017 Oct;37(7):790-801. Epub 2017 Apr 11.
Studies:
- Action to Control Cardiovascular Risk in Diabetes (ACCORD)
- Antihypertensive and Lipid-Lowering Treatment to Prevent Heart Attack Trial (ALLHAT)
- Aspirin-Myocardial Infarction Study (AMIS)
- Atrial Fibrillation Follow-Up Investigation of Rhythm Management (AFFIRM)
- Beta-Blocker Evaluation in Survival Trial (BEST)
- Beta-Blocker Heart Attack Trial (BHAT)
- Bypass Angioplasty Revascularization Investigation (BARI)
- Cardiac Arrhythmia Suppression Trial (CAST)
- Digitalis Investigation Group (DIG)
- Enhancing Recovery in Coronary Heart Disease Patients (ENRICHD)
- Hypertension Detection and Follow-Up Program (HDFP)
- Lipid Research Clinics (LRC) Coronary Primary Prevention Trial (CPPT)
- Multiple Risk Factor Intervention Trial for the Prevention of Coronary Heart Disease (MRFIT)
- Occluded Artery Trial (OAT)
- Prevention of Events With Angiotensin-Converting Enzyme Inhibitor Therapy (PEACE)
- Resuscitation Outcomes Consortium (ROC) Hypertonic Saline Trial Shock Study (HS) and Traumatic Brain Injury Study (TBI)
- Studies of Left Ventricular Dysfunction (SOLVD)
- Systolic Hypertension in the Elderly Program (SHEP)
- Thrombolysis in Myocardial Ischemia Trial II (TIMI II)
Abstract
BACKGROUND: Risk prediction models allow for the incorporation of individualized risk and clinical effectiveness information to identify patients for whom therapy is most appropriate and cost-effective. This approach has the potential to identify inefficient (or harmful) care in subgroups at different risks, even when the overall results appear favorable. Here, we explore the value of personalized risk information and the factors that influence it. METHODS: Using an expected value of individualized care (EVIC) framework, which monetizes the value of customizing care, we developed a general approach to calculate individualized incremental cost effectiveness ratios (ICERs) as a function of individual outcome risk. For a case study (tPA v. streptokinase to treat possible myocardial infarction), we used a simulation to explore how an EVIC is influenced by population outcome prevalence, model discrimination (c-statistic) and calibration, and willingness-to-pay (WTP) thresholds. RESULTS: In our simulations, for well-calibrated models, which do not over- or underestimate predicted v. observed event risk, the EVIC ranged from $0 to $700 per person, with better discrimination (higher c-statistic values) yielding progressively higher EVIC values. For miscalibrated models, the EVIC ranged from -$600 to $600 in different simulated scenarios. The EVIC values decreased as discrimination improved from a c-statistic of 0.5 to 0.6, before becoming positive as the c-statistic reached values of ~0.8. CONCLUSIONS: Individualizing treatment decisions using risk may produce substantial value but also has the potential for net harm. Good model calibration ensures a non-negative EVIC. Improvements in discrimination generally increase the EVIC; however, when models are miscalibrated, greater discriminating power can paradoxically reduce the EVIC under some circumstances.