An official website of the United States government
Generalizability of Cardiovascular Disease Clinical Prediction Models: 158 Independent External Validations of 104 Unique Models.
Pubmed ID: 35354282
Pubmed Central ID: PMC9015037
Journal: Circulation. Cardiovascular quality and outcomes
Publication Date: April 1, 2022
MeSH Terms: Humans, Cardiovascular Diseases, Risk Assessment, Heart Failure
Authors: Steyerberg EW, Kent DM, Nelson J, van Klaveren D, Gulati G, Upshaw J, Wessler BS, Brazil RJ, Lundquist CM, Park JG, McGinnes H, Van Calster B
Cite As: Gulati G, Upshaw J, Wessler BS, Brazil RJ, Nelson J, van Klaveren D, Lundquist CM, Park JG, McGinnes H, Steyerberg EW, Van Calster B, Kent DM. Generalizability of Cardiovascular Disease Clinical Prediction Models: 158 Independent External Validations of 104 Unique Models. Circ Cardiovasc Qual Outcomes 2022 Apr;15(4):e008487. Epub 2022 Mar 31.
Studies:
- Action to Control Cardiovascular Risk in Diabetes (ACCORD)
- Antihypertensive and Lipid-Lowering Treatment to Prevent Heart Attack Trial (ALLHAT)
- Aspirin-Myocardial Infarction Study (AMIS)
- Atherothrombosis Intervention in Metabolic Syndrome with Low HDL/High Triglyceride and Impact on Global Health Outcomes (AIM-HIGH)
- Atrial Fibrillation Follow-Up Investigation of Rhythm Management (AFFIRM)
- Beta-Blocker Evaluation in Survival Trial (BEST)
- Beta-Blocker Heart Attack Trial (BHAT)
- Bypass Angioplasty Revascularization Investigation (BARI)
- Bypass Angioplasty Revascularization Investigation in Type 2 Diabetes (BARI 2D)
- Cardiac Arrhythmia Suppression Trial (CAST)
- Digitalis Investigation Group (DIG)
- Enhancing Recovery in Coronary Heart Disease Patients (ENRICHD)
- Heart Failure: A Controlled Trial Investigating Outcomes of Exercise Training (HF-ACTION)
- Hypertension Detection and Follow-Up Program (HDFP)
- Lipid Research Clinics (LRC) Coronary Primary Prevention Trial (CPPT)
- Magnesium in Coronaries (MAGIC)
- Multiple Risk Factor Intervention Trial for the Prevention of Coronary Heart Disease (MRFIT)
- Post Coronary Artery Bypass Graft Study (CABG)
- Prevention of Events With Angiotensin-Converting Enzyme Inhibitor Therapy (PEACE)
- Resuscitation Outcomes Consortium (ROC) Hypertonic Saline Trial Shock Study (HS) and Traumatic Brain Injury Study (TBI)
- Studies of Left Ventricular Dysfunction (SOLVD)
- Sudden Cardiac Death in Heart Failure Trial (SCD-HeFT)
- Systolic Hypertension in the Elderly Program (SHEP)
- Thrombolysis in Myocardial Ischemia Trial II (TIMI II)
- Thrombolysis in Myocardial Ischemia Trial III (TIMI III)
- Treatment of Preserved Cardiac Function Heart Failure With an Aldosterone Antagonist (TOPCAT)
Abstract
BACKGROUND: While clinical prediction models (CPMs) are used increasingly commonly to guide patient care, the performance and clinical utility of these CPMs in new patient cohorts is poorly understood. METHODS: We performed 158 external validations of 104 unique CPMs across 3 domains of cardiovascular disease (primary prevention, acute coronary syndrome, and heart failure). Validations were performed in publicly available clinical trial cohorts and model performance was assessed using measures of discrimination, calibration, and net benefit. To explore potential reasons for poor model performance, CPM-clinical trial cohort pairs were stratified based on relatedness, a domain-specific set of characteristics to qualitatively grade the similarity of derivation and validation patient populations. We also examined the model-based C-statistic to assess whether changes in discrimination were because of differences in case-mix between the derivation and validation samples. The impact of model updating on model performance was also assessed. RESULTS: Discrimination decreased significantly between model derivation (0.76 [interquartile range 0.73-0.78]) and validation (0.64 [interquartile range 0.60-0.67], <i>P</i><0.001), but approximately half of this decrease was because of narrower case-mix in the validation samples. CPMs had better discrimination when tested in related compared with distantly related trial cohorts. Calibration slope was also significantly higher in related trial cohorts (0.77 [interquartile range, 0.59-0.90]) than distantly related cohorts (0.59 [interquartile range 0.43-0.73], <i>P</i>=0.001). When considering the full range of possible decision thresholds between half and twice the outcome incidence, 91% of models had a risk of harm (net benefit below default strategy) at some threshold; this risk could be reduced substantially via updating model intercept, calibration slope, or complete re-estimation. CONCLUSIONS: There are significant decreases in model performance when applying cardiovascular disease CPMs to new patient populations, resulting in substantial risk of harm. Model updating can mitigate these risks. Care should be taken when using CPMs to guide clinical decision-making.