The measure estimates the hospital-level, risk-standardized mortality rate (RSMR) for Medicare patients (Fee-for-Service [FFS] and Medicare Advantage[MA]) discharged from the hospital with a principal discharge diagnosis of acute ischemic stroke. The outcome is all-cause 30-day mortality, defined as death from any cause within 30 days of the index admission date, including in-hospital death, for stroke patients. The measure includes the National Institutes of Health (NIH) Stroke Scale as an assessment of stroke severity upon admission in the risk-adjustment model.
Measure Specs
- General Information(active tab)
- Numerator
- Denominator
- Exclusions
- Measure Calculation
- Supplemental Attachment
- Point of Contact
General Information
The goal of a stroke mortality measure* is to improve patient outcomes for patients hospitalized with acute ischemic stroke by providing patients, physicians, hospitals, and policymakers with hospital-level, risk-standardized mortality rates. Measurement of patient outcomes allows for a broad view of quality of care that encompasses more than what can be captured by individual process-of-care measures.
Stroke occurs in nearly a million people in the United States annually (CDC, 2020), is a common cause of death, and is a leading cause of disability (American Stroke Association, 2024). As the fifth-leading cause of death, stroke affects approximately 795,000 people in the United States annually (Tsao et al., 2023). Additionally, stroke accounted for 17.5% of all cardiovascular disease deaths in the U.S. in 2022 (National Center for Health Statistics, 2024).
Stroke mortality is an appropriate measure of the quality of care; stroke mortality rates vary across hospitals and can be influenced by the quality of care during the initial hospitalization; mortality following stroke is an important adverse outcome that can be measured reliably and objectively. Specifically, post-stroke mortality rates have been shown to be influenced by critical and modifiable aspects of care such as response to complications, timeliness of delivery of care, organization of care, and appropriate imaging (Bekelis et al., 2016; Bustamante et al., 2016; Xian et al., 2019; Jahan et al., 2019, Kuriakose & Xiao, 2020, Lip et al., 2020; Feigin & Owolabi, 2023).
*Note: There are two existing CMS stroke mortality measures that are referenced in this CBE submission: an existing measure that is publicly reported (and includes only Medicare Fee-For-Service [FFS] patients) and this (new) measure under review, which is a re-specification of the existing publicly reported measure. We define them below and provide nomenclature to differentiate them.
“FFS-only stroke mortality measure” is used to describe the measure that is currently publicly reported on Care Compare. This three-year measure includes only Medicare FFS admissions. This measure includes the NIH stroke scale (NIHSS) in the risk adjustment model. In the attachment, we provide the methodology report for the FFS-only measure, which is also available online on QualityNet:
https://qualitynet.cms.gov/files/663be566cc07c26dc8485e00?filename=2024_CSM_AUS_Report_v1.0.pdf
“Stroke Mortality measure” is used to indicate the new measure submitted in this CBE endorsement application, which is a re-specified version of the FFS-only measure. The Stroke Mortality measure is now a two-year measure, includes both Medicare Advantage (MA) and Medicare FFS admissions, includes the NIHSS in the risk adjustment model, and has an updated risk model (newly selected risk variables).
References
American Stroke Association. (2024). About stroke. Retrieved from https://www.stroke.org/en/about-stroke
Bekelis K, Marth NJ, Wong K, Zhou W, Birkmeyer JD, Skinner J. Primary stroke center hospitalization for elderly patients with stroke: implications for case fatality and travel times. JAMA Internal Medicine. 2016;176(9):1361-1368.
Bustamante, A., García-Berrocoso, T., Rodriguez, N., Llombart, V., Ribó, M., Molina, C., & Montaner, J. (2016). Ischemic stroke outcome: A review of the influence of post-stroke complications within the different scenarios of stroke care. European Journal of Internal Medicine, 29, 9-21. https://doi.org/10.1016/j.ejim.2015.11.030
Centers for Disease Control and Prevention: Stroke. Available at: https://www.cdc.gov/stroke/index.htm.
Feigin, V. L., & Owolabi, M. O., on behalf of the World Stroke Organization–Lancet Neurology Commission Stroke Collaboration Group. (2023). Pragmatic solutions to reduce the global burden of stroke: A World Stroke Organization–Lancet Neurology Commission. The Lancet Neurology, 22(12), 1160–1206. https://doi.org/10.1016/S1474-4422(23)00289-6
Jahan R, Saver JL, Schwamm LH, et al. Association between time to treatment with endovascular reperfusion therapy and outcomes in patients with acute ischemic stroke treated in clinical practice. JAMA. 2019;322(3):252-263.
Kuriakose, D., & Xiao, Z. (2020). Pathophysiology and treatment of stroke: Present status and future perspectives. International Journal of Molecular Sciences, 21(20), 7609. https://doi.org/10.3390/ijms21207609
Lip, G. Y. H., Lane, D. A., Lenarczyk, R., Boriani, G., Doehner, W., Benjamin, L. A., Fisher, M., Lowe, D., Sacco, R. L., Schnabel, R., Watkins, C., Ntaios, G., & Potpara, T. (2022). Integrated care for optimizing the management of stroke and associated heart disease: A position paper of the European Society of Cardiology Council on Stroke. European Heart Journal, 43(26), 2442–2460. https://doi.org/10.1093/eurheartj/ehac245
National Center for Health Statistics. Multiple Cause of Death 2018–2022 on CDC WONDER Database. https://wonder.cdc.gov/mcd.html
Tsao, C. W., Aday, A. W., Almarzooq, Z. I., et al. (2023). Heart disease and stroke statistics—2023 update: A report from the American Heart Association. Circulation, 147, e93–e621. https://doi.org/10.1161/CIR.0000000000001123
Xian Y, Xu H, O'Brien EC, et al. Clinical effectiveness of direct oral anticoagulants vs warfarin in older patients with atrial fibrillation and ischemic stroke: Findings from the Patient-Centered Research into Outcomes Stroke Patients Prefer and Effectiveness Research (PROSPER) Study. JAMA. 2019;322(3):252-263.
Medicare fee-for-service (FFS) claims and Medicare Advantage (MA) encounters, in addition to Medicare administrative data, are used to derive all components of the measure.
Numerator
The outcome for this measure is 30-day, all-cause mortality. We define mortality as death from any cause within 30 days of the start of the index admission.
The outcome for this measure is 30-day, all-cause mortality. We define mortality as death from any cause within 30 days of the start of the admission for patients discharged from the hospital with a principal discharge diagnosis of ischemic stroke.
Denominator
The cohort includes admissions for patients that meet all of the following inclusion criteria:
1. Discharged from the hospital with a principal discharge diagnosis of ischemic stroke;
2. Enrolled in Medicare (Fee-for-Service [FFS] or Medicare Advantage [MA]) for the 12 months prior to the date of admission and during the index admission;
3. Aged 65 or older;
4. Not transferred from another acute care facility.
The cohort includes admissions for patients who meet all the following inclusion criteria:
- Discharged from the hospital with a principal discharge diagnosis of ischemic stroke;
- Enrolled in Medicare Fee-for-Service (FFS) and/or Medicare Advantage (MA) for the 12 months prior to the date of admission and enrolled in FFS or MA during the index admission;
- Aged 65 and older;
- Not transferred from another acute care facility.
Codes that define the denominator are in the Data Dictionary in Section 1.13.
Exclusions
This measure excludes index admissions for patients who meet any of the following exclusion criteria:
- Inconsistent or unknown vital status or other unreliable demographic data (e.g., age and gender);
- Enrolled in the Medicare hospice program at any time in the 12 months prior to the index admission, including the first day of the index admission;
- Discharged against medical advice;
- With a secondary diagnosis code of COVID-19 coded as present on admission on the index admission claim.
For patients with more than one eligible stroke admission in the reporting period, only one index admission is randomly selected for inclusion in the cohort. Additional admissions within that time period are excluded.
The exclusion criteria for the Stroke Mortality measure are as follows:
- Inconsistent or unknown vital status or other unreliable demographic data;
- Rationale: We do not include stays for patients where the age is greater than 115, where the gender is neither male nor female, where the admission date is after the date of death in the Medicare Enrollment Database, or where the date of death occurs before the date of discharge, but the patient was discharged alive.
- Enrolled in the Medicare hospice program at any time in the 12 months prior to the index admission, including the first day of the index admission;
- Rationale: These patients are likely to continue to seek comfort measures only; thus, mortality is not necessarily an adverse outcome or signal of poor-quality care for these patients. Patients who transition to hospice during their hospital stay (after the first day of the index admission) are not excluded. Such transitions may be the result of quality failures that have led to poor clinical outcomes. Thus, excluding these patients could mask quality problems.
- Discharged against medical advice.
- Rationale: Providers did not have the opportunity to deliver full care and prepare the patient for discharge.
In addition, CMS has temporarily adjusted this (and other measures) in response to the COVID-10 public health emergency (Hospital Inpatient Value, Incentives, and Quality Reporting Outreach and Education Support Contractor, 2022). Temporarily, admissions with a principal diagnosis code of COVID-19 or with a secondary diagnosis code of COVID-19 coded as POA on the index admission claim are excluded from the measure. The 2022-2023 results presented in this submission have applied this exclusion.
Reference
Hospital Inpatient Value, Incentives, and Quality Reporting Outreach and Education Support Contractor (Health Services Advisory Group, Inc.). CMS Announces Updates on Hospital Quality Reporting and Value-based Payment Programs Due to the COVID-19 Public Health Emergency. Accessed March 3, 2022. https://qualitynet.cms.gov/files/5f0707a3b8112700239dca19?filename=2020-62-IP.pdf
Measure Calculation
The Stroke Mortality measure estimates hospital-level, 30-day, all-cause risk-standardized mortality rates (RSMRs) following hospitalization for stroke using a hierarchical logistic regression model. In brief, the approach simultaneously models data at the patient and hospital levels to account for variance in patient outcomes within and between hospitals (Normand and Shahian, 2007). At the patient level, it models the log odds of mortality within 30 days of index admission using age, selected clinical covariates, and a hospital-specific intercept. At the hospital level, it models the hospital-specific intercepts as arising from a normal distribution. The hospital intercept represents the underlying risk of mortality at the hospital after accounting for patient risk. The hospital-specific intercepts are given a distribution to account for the clustering (non-independence) of patients within the same hospital. If there were no differences among hospitals, then after adjusting for patient risk, the hospital intercepts should be identical across all hospitals.
The RSMR is calculated as the ratio of the number of “predicted” to the number of “expected” deaths at a given hospital, multiplied by the national observed mortality rate. For each hospital, the numerator of the ratio is the number of deaths within 30 days predicted based on the hospital’s performance with its observed case mix, and the denominator is the number of deaths expected based on the nation’s performance with that hospital’s case mix. This approach is analogous to a ratio of “observed” to “expected” used in other types of statistical analyses. It conceptually allows for a comparison of a particular hospital’s performance given its case mix to an average hospital’s performance with the same case mix. Thus, a lower ratio indicates lower-than-expected mortality rates or better quality, and a higher ratio indicates higher-than-expected mortality rates or worse quality.
The “predicted” number of deaths (the numerator) is calculated by using the coefficients estimated by regressing the risk factors and the hospital-specific intercept on the risk of mortality. The estimated hospital-specific intercept is added to the sum of the estimated regression coefficients multiplied by the patient characteristics. The results are transformed and summed over all patients attributed to a hospital to get a predicted value. The “expected” number of deaths (the denominator) is obtained in the same manner, but a common intercept using all hospitals in our sample is added in place of the hospital-specific intercept. The results are transformed and summed over all patients in the hospital to get an expected value. To assess hospital performance for each reporting period, we re-estimate the model coefficients using the years of data in that period.
This calculation transforms the ratio of predicted over expected into a rate that is compared to the national observed mortality rate. The hierarchical logistic regression models are described fully in the original methodology report posted on QualityNet (https://qualitynet.org/inpatient/measures/mortality/methodology).
References
Normand S-LT, Shahian DM. 2007. Statistical and Clinical Aspects of Hospital Outcomes Profiling. Stat Sci 22(2): 206-226.
The measure is not stratified.
The measure does not have a minimum sample size.
Supplemental Attachment
Point of Contact
Not applicable.
Raquel Myers
7500 Security Boulevard
Windsor Mill, MD 21244
United States
Amy Moyer
Yale-New Haven Health Services Corporation/Center for Outcomes Research and Evaluation
195 Church Street (Fifth Floor)
New Haven, CT 06510
United States
Importance
Evidence
Stroke continues to be a leading cause of mortality and morbidity in the United States (Tsao et al., 2022), with an estimated 795,000 people experiencing a stroke each year (Tsao et al., 2023). Among patients 65 and older, both stroke burden and subsequent mortality vary considerably by race and geographic region (Howard et al., 2016; Thompson et al., 2017). Most of these strokes are ischemic and increase in prevalence with advancing age (Benjamin et al., 2019; Benjamin et al., 2020). Some projections estimated that more than 3 million adults, representing almost 4% of the US adult population, will have had a stroke by 2030 (Ovbiagele et al., 2013). Recent data show that the economic burden of stroke (healthcare services, medications, and lost productivity) was nearly $56.2 billion between 2019 and 2020 (Strilciuc et al., 2021; Martin et al. 2024). As such, stroke mortality is a priority condition for outcomes measure development.
Many current hospital processes have been associated with lower stroke mortality rates within 30 days of hospital admission. In particular, mortality rates for patients experiencing a stroke have been shown to be influenced by critical aspects of care in the hospital, such as timeliness of care, organization of care, coordinated transitions to the outpatient environment, antihypertensive and anticoagulant therapies, and appropriate imaging (Fonarow et al., 2014; Bekelis et al., 2016; Bustamante et al., 2016; Xian et al., 2019; Jahan et al., 2019, Kuriakose & Xiao, 2020, Lip et al., 2022; Feigin & Owolabi, 2023).
Importantly, there is variation in stroke mortality rates at the hospital level that can be impacted by hospital-specific care delivery and organization. For example, hospitals participating in quality improvement registries like Get With The Guidelines (GWTG) have lower in-hospital mortality rates among stroke patients than hospitals not participating in similar programs (Fonarow et al., 2014). Research has also shown that patients treated at hospitals participating in the GWTG quality improvement registry for stroke were significantly more likely to receive multiple evidence-based care interventions, such as tissue plasminogen activator (tPA) administration and evaluation by a neurologist (Howard et al., 2018). Additionally, it has been shown that hospitals that use an integrated care model and employ multidisciplinary stroke care teams, (including neurologists, radiologists, nurses, and rehabilitation specialists) can deliver comprehensive and coordinated care that improves patient outcomes and post-stroke recovery (Clarke and Forster, 2015, Lip et al., 2022). This demonstrates that modifiable and controllable actions at the hospital level can influence care and subsequent outcomes for patients with stroke. Please see Section 6.2.1 for additional evidence for hospital-level interventions that impact stroke mortality rates.
References
Bekelis, K., Marth, N.J., Wong, K., Zhou, W., Birkmeyer, J.D., Skinner, J. (2016). Primary Stroke Center Hospitalization for Elderly Patients with Stroke: Implications for Case Fatality and Travel Times. Journal of the American Medical Association Internal Medicine, 176(9), 1361-1368. https://doi.org/10.1001/jamainternmed.2016.3919
Benjamin, E.J., Muntner, P., Alonso, A., Bittencourt, M.S., Callaway, C.W., Carson, A.P., et al. (2019). Heart Disease and Stroke Statistics-2019 Update: A Report from the American Heart Association. Circulation, 139(10), e56–e528. https://doi.org/10.1161/CIR.0000000000000659
Benjamin, E.J., Muntner, P., Alonso, A., Bittencourt, M.S., Callaway, C.W., Carson, A.P., et al. (2020). Correction: Heart Disease and Stroke Statistics-2019 Update: A Report from the American Heart Association. Circulation, 141(2), e33. https://doi.org/10.1161/cir.0000000000000746
Bustamante, A., García-Berrocoso, T., Rodriguez, N., Llombart, V., Ribó, M., Molina, C., & Montaner, J. (2016). Ischemic stroke outcome: A review of the influence of post-stroke complications within the different scenarios of stroke care. European Journal of Internal Medicine, 29, 9-21. https://doi.org/10.1016/j.ejim.2015.11.030
Centers for Disease Control and Prevention. (2020). Stroke. U.S. Department of Health and Human Services. Retrieved July 27, 2020 from https://www.cdc.gov/stroke/index.htm
Clarke, D. J., & Forster, A. (2015). Improving post-stroke recovery: The role of the multidisciplinary health care team. Journal of Multidisciplinary Healthcare, 8, 433-442. https://doi.org/10.2147/JMDH.S68764
Feigin, V. L., & Owolabi, M. O., on behalf of the World Stroke Organization–Lancet Neurology Commission Stroke Collaboration Group. (2023). Pragmatic solutions to reduce the global burden of stroke: A World Stroke Organization–Lancet Neurology Commission. The Lancet Neurology, 22(12), 1160–1206. https://doi.org/10.1016/S1474-4422(23)00289-6
Fonarow, G.C., Zhao, X., Smith, E.E., Saver, J.L., Reeves, M.J., Bhatt, D.L., Xian, Y., Hernandez, A.F., Peterson, E.D., Schwamm, L.H. (2014). Door-to-Needle Times for Tissue Plasminogen Activator Administration and Clinical Outcomes in Acute Ischemic Stroke Before and After a Quality Improvement Initiative. Journal of the American Medical Association, 311(16), 1632-1640. https://doi.org/10.1001/jama.2014.3203
Howard, G., Moy, C. S., Howard, V. J., McClure, L. A., Kleindorfer, D. O., Kissela, B. M., Judd, S. E., et al. (2016). Where to focus efforts to reduce the Black–White disparity in stroke mortality: Incidence versus case fatality? Stroke, 47(7), 1893–1898. https://doi.org/10.1161/STROKEAHA.115.012631
Howard, G., Schwamm, L.H., Donnelly, J.P., Howard, V.J., Jasne, A., Smith, E.E., Rhodes, J.D., Kissela, B.M., Fonarow, G.C., Kleindorfer, D.O., Albright, K.C. (2018). Participation in Get with the Guidelines-Stroke and its Association with Quality of Care for Stroke. Journal of the American Medical Association Neurology, 75(11), 1331-1337. https://doi.org/10.1001/jamaneurol.2018.2101
Jahan, R., Saver, J.L., Schwamm, L.H., Fonarow, G.C., Liang, L., Matsouaka, R.A., Xian, Y., Holmes, D.N., Peterson, E.D., Yavagal, D., Smith, E.E. (2019). Association Between Time to Treatment with Endovascular Reperfusion Therapy and Outcomes in Patients with Acute Ischemic Stroke Treated in Clinical Practice. Journal of the American Medical Association, 322(3), 252-263. https://doi.org/10.1001/jama.2019.8286
Kuriakose, D., & Xiao, Z. (2020). Pathophysiology and treatment of stroke: Present status and future perspectives. International Journal of Molecular Sciences, 21(20), 7609. https://doi.org/10.3390/ijms21207609
Lip, G. Y. H., Lane, D. A., Lenarczyk, R., Boriani, G., Doehner, W., Benjamin, L. A., Fisher, M., Lowe, D., Sacco, R. L., Schnabel, R., Watkins, C., Ntaios, G., & Potpara, T. (2022). Integrated care for optimizing the management of stroke and associated heart disease: A position paper of the European Society of Cardiology Council on Stroke. European Heart Journal, 43(26), 2442-2460. https://doi.org/10.1093/eurheartj/ehac245
Martin, S. S., Aday, A. W., Almarzooq, Z. I., et al.; American Heart Association Council on Epidemiology and Prevention Statistics Committee; Stroke Statistics Subcommittee. (2024). 2024 heart disease and stroke statistics: A report of US and global data from the American Heart Association. Circulation, 149(e347–913).
Ovbiagele, B., Goldstein, L.B., Higashida, R.T., Howard, V.J., Johnston, S.C., Khavjou, O.A., Lackland, D.T., Lichtman, J.H., Mohl, S., Sacco, R.L., Saver, J.L., Trogdon, J.G., American Heart Association Advocacy Coordinating Committee and Stroke Council. (2013). Forecasting the future of stroke in the United States: a policy statement from the American Heart Association and American Stroke Association. Stroke, 44(8), 2361-2375. https://doi.org/10.1161/str.0b013e31829734f2
Strilciuc, S., Grad, D. A., Radu, C., Chira, D., Stan, A., Ungureanu, M., Gheorghe, A., & Muresanu, F. D. (2021). The economic burden of stroke: A systematic review of cost of illness studies. Journal of Medicine and Life, 14(5), 606–619. https://doi.org/10.25122/jml-2021-0361
Thompson, M.P., Zhao, X., Bekelis, K., Gottlieb, D.J., Fonarow, G.C., Schulte, P.J., Xian, Y., Lytle, B.L., Schwamm, L.H., Smith, E.E., Reeves, M.J. (2017). Regional Variation in 30-Day Ischemic Stroke Outcomes for Medicare Beneficiaries Treated in Get With The Guidelines-Stroke Hospitals. Circulation, 10(8), e003604. https://doi.org/10.1161/circoutcomes.117.003604
Tsao CW, Aday AW, Almarzooq ZI, et al. Heart disease and stroke statistics—2022 update: a report from the American Heart Association. Circulation 2022;145:e153–639. https://doi.org/10.1161/CIR.0000000000001052PMID:35078371
Tsao, C. W., Aday, A. W., Almarzooq, Z. I., et al. (2023). Heart disease and stroke statistics—2023 update: a report from the American Heart Association. Circulation, 147, e93–e621. https://doi.org/10.1161/CIR.0000000000001052
Xian, Y., Xu, H., O'Brien, E.C., Shah, S., Thomas, L., Pencina, M.J., Fonarow, G.C., Olson, D.M., Schwamm, L.H., Bhatt, D.L., Smith, E.E., Hannah, D., Maisch, L., Lytle, B.L., Peterson, E.D., Hernandez, A.F. (2019). Clinical Effectiveness of Direct Oral Anticoagulants vs Warfarin in Older Patients with Atrial Fibrillation and Ischemic Stroke: Findings from the Patient-Centered Research into Outcomes Stroke Patients Prefer and Effectiveness Research (PROSPER) Study. Journal of the American Medical Association, 76(10), 1192-1202. https://doi.org/10.1001/jamaneurol.2019.2099
Measure Impact
With the increased focus on quality improvement for patients hospitalized for stroke and increasing scrutiny given to eliminating disparities, we expect that stroke mortality rates will continue to decline, as seen in the literature (described in Section 2.2) and described below.
Nationally, population-level rates of stroke mortality have declined over the past decades, although more recently, this decline has stalled (Mercy et al., 2023) and was exacerbated by a COVID-19-related increase in stroke mortality (Qureshi et al., 2021; Yang et al., 2023). At the hospital level, however, since the Fee-for-Service (FFS)-stroke mortality measure was implemented in 2014, there has been improvement across most of the distribution in 30-day risk-standardized outcomes; the range of performance has narrowed, and the mean (average) has improved. We compared performance between prior to measure implementation (dates of data, 7/2010-6/2013) and the most recent performance period prior to COVID and prior to the addition of the NIHSS in 2022 (7/1/2017-12/1/2019), and the current re-specification. These results show improvement across the distribution of the measure, with lower mean (FFS-only) RSMRs (13.5 vs. 15.4), and lower RSMRs across the rest of the distribution (25th percentile, 12.9 vs. 14.6; 50th percentile, 13.4 vs. 15.3; 75th percentile, 14.1 vs. 16.1).
Reference
Mercy, U. C., Farhadi, K., Ogunsola, A. S., Karaye, R. M., Baguda, U. S., Eniola, O. A., Yunusa, I., & Karaye, I. M. (2023). Revisiting recent trends in stroke death rates, United States, 1999-2020. Journal of the neurological sciences, 451, 120724. https://doi.org/10.1016/j.jns.2023.120724
Qureshi, A. I., Baskett, W. I., Huang, W., Shyu, D., Myers, D., Raju, M., Lobanova, I., Suri, M. F. K., Naqvi, S. H., French, B. R., Siddiq, F., Gomez, C. R., & Shyu, C. R. (2021). Acute Ischemic Stroke and COVID-19: An Analysis of 27 676 Patients. Stroke, 52(3), 905–912. https://doi.org/10.1161/STROKEAHA.120.031786
Yang, Q., Tong, X., Schieb, L., Coronado, F., & Merritt, R. (2023). Stroke Mortality Among Black and White Adults Aged ≥35 Years Before and During the COVID-19 Pandemic - United States, 2015-2021. MMWR. Morbidity and mortality weekly report, 72(16), 431–436. https://doi.org/10.15585/mmwr.mm7216a4
The Stroke Mortality measure (and the Fee-for-Service (FFS)-only version that is currently in use) fills an important gap in quality measurement. As described below, there are other stroke outcome (non-mortality) and process measures in use, but none of them capture the outcome of mortality and none of them are publicly reported.
There has been increased focus on the timeliness of care provided to stroke patients, and hospitals can opt for one of four different levels of stroke center certification (each with its own set of process and outcome measures) as assessed by The Joint Commission (TJC) (The Joint Commission, 2024). All but two measures in TJC’s certification program are process measures; there are two outcome measures based on the Modified Rankin Score (mRS) at 90 days, however, those outcome measures can, by definition, only apply to patients who survived the stroke. There are also process measures associated with quality improvement efforts around stroke (e.g., Get With the Guidelines). Importantly, none of these processes or outcome measures are reported to the public at the facility level. We note that it is possible for a hospital to perform well on mRS measures but have poor performance on the Stroke Mortality measure.
Other than the Stroke Mortality measure currently under review in this submission (and the prior FFS-only version, which is currently publicly reported), there are no other stroke mortality measures that are publicly reported. This Stroke Mortality measure will be publicly reported to patients and other stakeholders on Care Compare (and currently, the prior version of this measure, with only FFS patients, is publicly reported).
This illustrates the measurement gap that is filled by this publicly reported, risk-standardized mortality measure for patients with ischemic stroke as it complements the other stroke-related measures in the quality landscape and can be used by those organizations to track progress on quality improvement efforts. It also serves as a public reporting and transparency/accountability measure for public use.
Reference
The Joint Commission. (2024). Stroke certification. The Joint Commission. Retrieved October 1, 2024, from https://www.jointcommission.org/what-we-offer/certification/certifications-by-setting/hospital-certifications/stroke-certification/
By understanding hospital-level, risk-standardized mortality rates following hospitalization for acute ischemic stroke, providers can identify areas for improvement in stroke care, therefore improving patient outcomes and mortality rates following a stroke. In addition, the public reporting of hospital-level performance provides transparency/accountability to other stakeholders, including patients.
U.S. taxpayers who fund the care of Medicare-aged stroke patients deserve the transparency that publicly developed and reported measures provide. Because stroke is an emergent condition, patients and their families will not have the opportunity to evaluate and choose a hospital based on quality. However, it has been demonstrated that public reporting of hospital-based quality measures can result in performance improvement (Bozic et al., 2020; Cacace et al., 2019), which will benefit patients. The combination of public reporting (of a measure developed with taxpayer dollars that also funds the actual care of patients captured by the measure at these hospitals) and information for quality improvement (provided by CMS to hospitals) creates the accountability needed for improvements in care. In addition, this Stroke Mortality measure can be used to assess the nation’s overall progress in improving care for stroke patients. Indeed, since the original Fee-for-Service (FFS)-only stroke mortality measure has been implemented, stroke mortality rates have declined.
Overall, the stroke mortality measure is an important indicator of the quality of care provided by hospitals. High mortality rates may point to gaps in clinical practices, patient care protocols, and the overall delivery of healthcare services. By monitoring these rates, hospitals can identify areas needing improvement and implement specific interventions that improve outcomes for stroke patients. This measure offers insights into a hospital's performance and informs patients, healthcare providers, and policymakers about the standard of care that currently exists and is ongoing. We note that the Stroke Mortality measure described in this CBE submission includes adjustment for patient preferences using the Do Not Resuscitate (DNR) variable (see Table 9 in the attachment).
References
Bozic, K., Yu, H., Zywiel, M. G., Li, L., Lin, Z., Simoes, J. L., Dorsey Sheares, K., Grady, J., Bernheim, S. M., & Suter, L. G. (2020). Quality Measure Public Reporting Is Associated with Improved Outcomes Following Hip and Knee Replacement. The Journal of bone and joint surgery. American volume, 102(20), 1799–1806. https://doi.org/10.2106/JBJS.19.00964.
Cacace M, Geraedts M, Berger E. Public reporting as a quality strategy. In: Busse R, Klazinga N, Panteli D, et al., editors. Improving healthcare quality in Europe: Characteristics, effectiveness and implementation of different strategies [Internet]. Copenhagen (Denmark): European Observatory on Health Systems and Policies; 2019. (Health Policy Series, No. 53.) 13. Available from: https://www.ncbi.nlm.nih.gov/books/NBK549281/
Performance Gap
Table 1, Table 2, and Figure 2 (see "All Figures and Tables Stroke Mortality" attachment) show that there is wide variation in measure scores (risk-standardized mortality rates or RSMRs) among hospitals for the Stroke Mortality measure. As shown in Table 1, RSMRs range from 6.4%-40.1%. The median hospital RSMR was 12.4%; the 10th percentile was 10.4%, and the 90th percentile was 15.7%. This variation in RSMRs suggests that there are differences in the quality of care received across hospitals for stroke that support measurement to reduce this variation.
We further characterize variation in performance using the median odds ratio. The median odds ratio, in this context, calculates the odds of the outcome (mortality) if the same patient were treated at a higher-risk hospital compared with a lower-risk hospital. For this measure, the median odds ratio was 2.54, indicating that a patient’s risk of mortality is 2.5 times greater (or 155% greater) in a higher-risk hospital than in a lower-risk hospital.
| Overall | Minimum | Decile_1 | Decile_2 | Decile_3 | Decile_4 | Decile_5 | Decile_6 | Decile_7 | Decile_8 | Decile_9 | Decile_10 | Maximum | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Mean Performance Score | 12.92 | 6.43 | 9.66 | 10.88 | 11.52 | 11.95 | 12.26 | 12.56 | 13.18 | 13.92 | 14.90 | 18.37 | 40.28 |
| N of Entities | 4,028 | 1 | 402 | 403 | 403 | 403 | 403 | 403 | 403 | 403 | 403 | 402 | 1 |
| N of Persons / Encounters / Episodes | 573,699 | 107 | 116,218 | 96,188 | 70,250 | 46,694 | 28,125 | 51,057 | 54,761 | 43,443 | 34,753 | 32,210 | 29 |
Equity
Equity
Please see the "All Tables and Figures Stroke Mortality" attachment for all tables and figures that are referred to within this submission, unless otherwise specified.
Table 11 (in the attachment) shows the variation in the prevalence of social risk factors across hospitals with at least 25 admissions, using one year of data (CY2022). For the Stroke Mortality cohort, the median percentage of dually eligible (DE) patients was 12.6% (25th percentile-75th percentile: 8.7% - 19.1%), and the median percentage of high Area Deprivation Index (ADI) patients was 8.4% (25th percentile-75th percentile: 1.7% - 22.9%).
Unadjusted Stroke Mortality rates were higher for patients with social risk factors (see Table 12 in the attachment). Mean observed mortality rates were 12.6% for patients without DE, compared with 14.1% for patients with DE; similarly, unadjusted stroke mortality rates were 12.8% for patients with low ADI compared with 13.2% for patients with high ADI.
To determine the impact of each variable in a multivariable model, we calculated odds ratios for each variable (see Table 13 in the attachment). The results show that in a multivariable model, the odds ratios for the DE variable are below 1.0 (protective), whereas the odds ratio for the high ADI is above 1 (higher risk).
We then examined model calibration for each social risk factor to determine if the risk model without the social risk factor performs well for patients with each social risk factor. The results show that the model is adequately calibrated for patients with and without each social risk factor (see Figures 5A and 5B, and Figures 6A and 6B in the attachment).
To understand the impact of each variable on the Stroke Mortality measure score, we calculated measure scores with and without each social risk factor and then calculated the differences in measure scores and the correlation between measure scores (see Table 14 in attachment). Results show that measure scores calculated with and without social risk factors are highly correlated (correlation coefficient 0.999 and 0.985 for DE and high ADI, respectively), and differences between measure scores are very small.
Patients with social risk factors (DE, high ADI) have higher unadjusted rates of stroke mortality compared with patients without either social risk factor. After adding the clinical risk variables to the model, the high ADI variable is still significantly associated with a higher risk of the outcome, but the DE variable is protective. However, measure scores calculated with and without either social risk factors are highly correlated, and differences are small, demonstrating that the DE and high ADI variables have little impact on measure scores. In addition, the risk model is well calibrated for patients with either social risk factor. Furthermore, this measure is used in a pay-for-reporting program and not a pay-for-performance program. Therefore, at this time, CMS has chosen not to adjust the Stroke Mortality measure so as not to mask disparities in outcomes for patients with social risk factors.
Feasibility
Feasibility
The Stroke Mortality measure is a claims-based measure that uses data routinely generated during healthcare delivery. This measure does not require additional data collection by facilities, which eliminates the reporting burden for hospitals. Since data are collected through standard claims submissions and processed by CMS, this measure does not introduce implementation challenges.
Feedback from measured entities has not indicated any significant concerns regarding implementation burden or confidentiality. As this measure relies on claims data securely submitted by healthcare facilities to CMS, there are no risks to patient confidentiality. CMS manages the data used for both the payment process and measure calculations.
To ensure the completeness and accuracy of the data, CMS allows a minimum of three months between the end of the performance period and when data is accessed. This period ensures that all claims are finalized, enables accurate calculation of the measure score, and minimizes the potential bias from incomplete or missing data.
As with any mortality measure, there are some concerns regarding the potential for unintended consequences. For example, a stroke mortality measure could hypothetically incentivize hospitals to aggressively treat patients who are less likely to have good prospects for a meaningful recovery. Several factors are working against this incentive, however, including (1) measures in The Joint Commission’s Stroke Certification requirements (The Joint Commission, 2024) that focus on longer-term (90-day) functional outcomes and (2) adequate risk adjustment of this stroke mortality measure which includes adjustment for both stroke severity, and for the Do Not resuscitate (DNR) ICD-10 code (see Table 9 in the attachment).
References
American Heart Association (2021). Recommendations for regional stroke destination plans in rural, suburban, and urban communities from the Prehospital Stroke System of Care Consensus Conference. Stroke.org. Retrieved October 1, 2024, from https://stroke.org/stroketransportplans
The Joint Commission; Stroke https://www.jointcommission.org/measurement/measures/stroke/
Because this is a claims-based measure, there is no burden on measured entities (hospitals); measure scores are automatically calculated by CMS based on claims data submitted by hospitals for payment.
Proprietary Information
Scientific Acceptability
Testing Data
For most of the testing in this submission, we used two years of Medicare (Fee-for-Service (FFS) and Medicare Advantage (MA)) data (January 1, 2022-December 30, 2023). Differences in data used for testing are outlined in Table 3 ("All Figures and Tables Stroke Mortality" attachment) in Section 4.1.4.
Please see Section 4.1.4 for details.
Characteristics of measured entities differ depending on the dataset. Please see Section 4.1.4 for details.
The datasets, dates, number of measured hospitals, and number of admissions used in each type of testing are in Table 3 ("All Figures and Tables Stroke Mortality" attachment).
For most measure testing, we used Medicare data from January 1, 2022-December 30, 2023. These datasets also include data on each patient for the 12 months prior to the index admission and contain inpatient and facility outpatient and Medicare Enrollment Database (EDB) data.
Reliability
Measure Score Reliability
We tested facility-level measure score reliability using the signal-to-noise method, using the formula presented by Adams and colleagues (Adams et al., 2010; Yu et al., 2013). Specifically, for each facility we calculate the reliability as follows:
Reliability=(σ_(facility-to-facility)^2)/(σ_(facility-to-facility)^2+ (σ_(facility error variance)^2)/n)
Where facility-to-facility variance is estimated from the hierarchical logistic regression model, n is equal to each facility’s observed case size, and the facility error variance is estimated using the variance of the logistic distribution (pi^2/3).
Signal-to-noise reliability scores can range from 0 to 1. A reliability of zero implies that all the variability in a measure is attributable to measurement error. A reliability of one implies that all the variability is attributable to real differences in performance.
We calculated the measure score reliability for all facilities, and for facilities with a volume cutoff of 25 procedures, which is the public reporting threshold of the currently, publicly reported measure.
References
Adams J, Mehrota, A, Thoman J, McGlynn, E. (2010). Physician cost profiling – reliability and risk of misclassification. NEJM, 362(11): 1014-1021.
Yu, H, Mehrota, A, Adams J. (2013). Reliability of utilization measures for primary care physician profiling. Healthcare, 1, 22-29.
Please see the "All Tables and Figures Stroke Mortality" attachment for all tables and figures that are referred to within this submission, unless otherwise specified.
We calculated signal-to-noise reliability for all hospitals in the testing sample (n=4,028) and hospitals with at least 25 cases (n=2,473), the volume threshold for public reporting of measure scores on Care Compare (Table 4 in the attachment). We used two years of data for these analyses (CY2022/2023).
For hospitals with at least 25 cases, the median reliability score was 0.911, ranging from 0.623 to 0.994. The 25th and 75th percentiles were 0.818 and 0.952, respectively. The minimum signal-to-noise reliability meets Partnership for Quality Measurement’s (PQM) minimum threshold of 0.6.
Table 5 (in the attachment) provides the Battelle-required table of measure scores within deciles of reliability.
The minimum signal-to-noise reliability score for the Stroke Mortality measure for hospitals with at least 25 cases (the public reporting threshold) was 0.623, using two years of data (CY2022/2023). The minimum reliability for this measure exceeds PQM’s threshold of a minimum of 0.6 at the entity level.
| | Overall | Minimum | Decile_1 | Decile_2 | Decile_3 | Decile_4 | Decile_5 | Decile_6 | Decile_7 | Decile_8 | Decile_9 | Decile_10 | Maximum |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Reliability | 0.875 | 0.623 | 0.668 | 0.749 | 0.816 | 0.865 | 0.898 | 0.919 | 0.937 | 0.953 | 0.966 | 0.979 | 0.994 |
| Mean Performance Score | 1.27 | 1.33 | 1.38 | 1.32 | 1.35 | 1.32 | 1.28 | 1.24 | 1.24 | 1.21 | 1.19 | 1.16 | 0.99 |
| N of Entities | 2,473 | 19 | 256 | 235 | 247 | 254 | 243 | 245 | 253 | 246 | 247 | 247 | 1 |
| N of Persons / Encounters / Episodes | 561,932 | 475 | 7,883 | 10,735 | 16,746 | 24,879 | 32,579 | 42,720 | 57,039 | 75,340 | 107,929 | 186,082 | 2,325 |
Validity
For this submission, we present data element validity testing for the NIH stroke scale variable, and three types of measure validity testing (face validity and two empiric analyses).
Data Element Validity using GWTG-Stroke Registry
CMS has been publicly reporting a risk-adjusted stroke mortality measure since 2014. In response to stakeholder feedback (including feedback from a CBE Committee) in 2022, CMS implemented a version of this measure that included the NIH Stroke Scale (NIHSS) in the risk adjustment model. Because this was a new data element at that time, we conducted data element validity testing by comparing CMS Medicare claims with Get With the Guidelines (GWTG) Stroke Registry Data. We note that the NIHSS is also a required element for The Joint Commission Stroke Center Certification process.
To assess the data element validity of the NIH Stroke Scale scores coded in the claims data, we linked the Medicare claims (2016-2019 Fee-for-Service (FFS) Dataset) with the GWTG-Stroke Registry data derived from patients’ medical records and compared the scores. The GWTG-Stroke Registry draws data from medical records and has been shown to be reliable through studies comparing registry data to chart abstraction (Xian et al., 2012). Because only patients aged 65 years and older were included, and some data were excluded based on linkage and other factors, a total of 29,937 stroke hospitalizations were used in the analysis.
Of the linked stroke hospitalizations for which claims data had a non-missing NIH Stroke Scale, we compared the scores recorded in the claims data to the scores in the GWTG-Stroke Registry data. We also examined the distribution of the stroke severity scales within the claims data and the registry data.
Face Validity Using Technical Expert Panel
We systematically assessed the face validity of the measure score as an indicator of quality by soliciting Technical Expert Panel (TEP) members agreement with the following statement: “The risk-standardized mortality rate obtained from the measures as specified can be used to distinguish between better and worse quality hospitals.”
The TEP is comprised of clinicians, health services researchers, statisticians, patients, patient advocates/caregivers, health insurance representatives, and hospital administrators (see Table 6 in the attachment for additional details on panel members).
We provided TEP members with a link to a survey from an email. Response options were on a six-point Likert Scale:
- 1=Strongly disagree
- 2=Moderately disagree
- 3=Somewhat disagree
- 4=Somewhat agree
- 5=Moderately agree
- 6=Strongly agree
Empiric Validity
To demonstrate empirical validity for the Stroke Mortality measure, we assessed the measure score’s correlation (1) with volume and (2) with two components of the Overall Hospital Quality Star Ratings.
Volume
For certain procedures and conditions, there is an established volume-outcome relationship (Levaillant et al., 2021; Scharfe et al., 2023). For stroke, some, but not all, studies have found a volume-outcome relationship (Saposnik et al., 2007; Svendsen et al., 2012, Lee et al., 2020). Because we have a large national database, we decided to examine if higher volume was associated with better outcomes for the Stroke Mortality measure. For this analysis, we calculated Stroke Mortality Stroke Risk-Standardized Mortality Rates (RSMRs) within deciles of admission volume using two years of data (CY2022/2023). We hypothesized that higher hospital volume would be moderately/weakly associated with better (lower) measure scores (negative correlation).
Volume may be considered a quality construct in that it is a proxy for underlying quality-related factors. For example, as a hospital sees more patients with a particular condition (in particular, those for whom there are clear guidelines and processes associated with better outcomes, as in the case of stroke), staff gain greater experience and repeated performance of evidence-based processes allows hospitals to develop and implement standardized delivery of care, which can reduce errors and delays in care and improve outcomes. Higher volume hospitals may also invest in more specialized support systems and training (Jha, 2015) and may be more likely to have multidisciplinary teams, which have been shown in the literature to result in lower stroke mortality rates. Higher-volume hospitals may also have more resources to invest in quality improvement and specialty accreditation (Shen et al., 2019). These concepts are discussed in Section 2.2 and Section 6.2.1 and are shown in the logic model (Figure 1 in the attachment).
Overall Hospital Star Rating
CMS’s Overall Hospital Quality Star Ratings assesses hospitals’ overall performance based on a weighted average of group scores from five measure groups (Patient Experience, Timely & Effective Care, Readmission, Safety of Care, and Mortality). The Mortality Group is comprised of the mortality measures that are publicly reported on Care Compare, including the FFS-only Stroke Mortality measure. Group scores, including the Mortality Group Score, are derived from a simple average of standardized measures scores within the Group. For the validity testing in this submission, we used the January 2024 Star Rating results, and Stroke Mortality RSMRs using one year of data (CY2022). The full methodology for the Overall Hospital Star Rating can be found on QualityNet: https://qualitynet.cms.gov/files/603966dda413b400224ddf50?filename=Star_Rtngs_CompMthdlgy_v4.1.pdf
For the validity testing for this submission, we first compared Stroke Mortality measure scores with the Overall Standardized Summary Score (which includes all Groups, including the Mortality Group score), with and without the entire Mortality Group score. We separately assessed the association between Stroke Mortality measure scores and the Mortality Measure Group Score (which includes only mortality measures). Because the Mortality Group score contains the existing FFS-only stroke mortality measure, we analyzed the Mortality Group score with and without the FFS-only stroke measure. We calculated Pearson’s correlation coefficients for these analyses.
We hypothesized that there would be a weak-to-moderate, negative, significant correlation between the Stroke Mortality measure (as specified in this submission) for both comparator measures (Overall Standardized Summary Score and the Mortality Group Score) because better performance (lower measure scores on the Stroke Mortality measure) should be associated with higher Standardized Summary Scores and Mortality Group Scores (which are standardized in a higher-is-better direction). We further hypothesized that after removing the Mortality or FFS-only stroke mortality measure from both comparator measures, respectively, the correlation coefficients would be lower but still significant.
References
Jha AK. Back to the Future: Volume as a Quality Metric. JAMA Forum Archive. Published online June 10, 2015. doi:10.1001/jamahealthforum.2015.0024
Lee, K.-J., Kim, J. Y., Kang, J., Kim, B. J., Kim, S.-E., Oh, H., Park, H.-K., Cho, Y.-J., Park, J.-M., Park, K.-Y., Lee, K. B., Lee, S. J., Park, T. H., Lee, J. S., Lee, J., Yang, K. H., Choi, A. R., Kang, M. Y., Saposnik, G., & Bae, H.-J. (2020). Hospital volume and mortality in acute ischemic stroke patients: Effect of adjustment for stroke severity. Journal of Stroke and Cerebrovascular Diseases, 29(5), 104753. https://doi.org/10.1016/j.jstrokecerebrovasdis.2020.104753
Levaillant, M., Marcilly, R., Levaillant, L., Michel, P., Hamel-Broza, J.-F., Vallet, B., & Lamer, A. (2021). Assessing the hospital volume-outcome relationship in surgery: A scoping review. BMC Medical Research Methodology, 21, 204. https://doi.org/10.1186/s12874-021-01396-6
Saposnik, G., Baibergenova, A., O’Donnell, M., Hill, M. D., Kapral, M. K., & Hachinski, V.; Stroke Outcome Research Canada (SORCan) Working Group. (2007). Hospital volume and stroke outcome: Does it matter? Neurology, 69(11), 1142–1151. https://doi.org/10.1212/01.wnl.0000276995.71411.70
Scharfe, J., Pfisterer-Heise, S., Kugler, C. M., Shehu, E., Wolf, T., Mathes, T., & Pieper, D. (2023). The effect of minimum volume standards in hospitals (MIVOS)—Protocol of a systematic review. Systematic Reviews, 12(11). https://doi.org/10.1186/s13643-023-02066-y
Shen YC, Chen G, Hsia RY. Community and Hospital Factors Associated With Stroke Center Certification in the United States, 2009 to 2017. JAMA Netw Open. 2019 Jul 3;2(7):e197855. doi: 10.1001/jamanetworkopen.2019.7855. PMID: 31348507; PMCID: PMC6661722.
Svendsen, M. L., Ehlers, L. H., Ingeman, A., & Johnsen, S. P. (2012). Higher stroke unit volume associated with improved quality of early stroke care and reduced length of stay. Stroke, 43(11), 3041–3045. https://doi.org/10.1161/STROKEAHA.112.663468
Xian Y, Fonarow GC, Reeves MJ, et al. Data quality in the American Heart Association Get With The Guidelines-Stroke (GWTG-Stroke): Results from a National Data Validation Audit. American Heart Journal. 2012;163(3):392-398.e391. http://www.ahjonline.com/article/S0002-8703%2811%2900894-5/abstract
Please see the "All Tables and Figures Stroke Mortality" attachment for all tables and figures that are referred to within this submission, unless otherwise specified.
Data Element Validity
The NIH Stroke Scale (NIHSS) is used to assess the severity of stroke and can be documented using ICD-10-CM codes. The NIHSS scores range from 0 to 42, with higher scores indicating more severe stroke symptoms.
When comparing the NIH Stoke Scale scores within the GWTG-Stroke Registry and administrative claims data (2016-2019 Medicare Fee-for-Service (FFS) Dataset), 93% of the scores from the two data sources are within 5 points of each other, and 84% are within 2 points. The distributions of NIH Stroke Scale scores from the administrative and GWTG-Stroke Registry data are similar (Figure 3 in the attachment), with a Pearson Correlation Coefficient of 0.993 and a weighted kappa of 0.842.
Measure Validity
Face Validity
12 of 13 TEP members responded to the face validity survey 11 of 12 members (92%) agreed (strongly, moderately, or somewhat) with the face validity statement. Details for each response category are shown below.
- Strongly agree (4) (33%)
- Moderately agree (6) (50%)
- Somewhat agree (1) (8%)
- Somewhat disagree (0) (0%),
- Moderately disagree (1) (8%)
- Strongly disagree (0) (0%)
Empiric Validity
To demonstrate empiric validity, we assessed the Stroke Mortality measure score association with (1) volume and (2) with the Summary Score and Mortality Group Score components of the Overall Hospital Star Ratings (with and without the FFS-only stroke morality measure) as described in Section 4.3.3.
Volume
Table 7 (in the attachment) shows Stroke Mortality risk-standardized mortality rates (RSMRs) within deciles of hospital admission volume (cohort volume). Starting with the fourth decile, RSMRs decrease with each increasing decile of volume. The overall Pearson’s correlation between RSMR and hospital volume is -0.25 (p<.0001).
Overall Hospital Star Rating
As shown in Table 8, the Pearson correlation coefficient between Stroke Mortality measure scores and Star Rating Standardized Summary Scores (excluding the FFS-stroke mortality measure) was –0.21 (p<0.001). After removing the entire Mortality Measure Group, the correlation was -0.11 (p<0.001). There were 1,997 hospitals (limited to those with at least 25 admissions) in these analyses, which used CY2022 data with claims from January 1, 2022 to December 31, 2022.
The Pearson correlation coefficient for the association between Stroke Mortality measure scores and Mortality Group Scores was –0.27 (p<0.001) (see Table 8 in the attachment). After removing the FFS-stroke mortality measure from the Mortality Group score, the correlation coefficient was -0.24 (p<0.001). There were 1,966 hospitals in these analyses, (limited to those with at least 25 admissions) in these analyses, which used CY2022 data with claims from January 1, 2022 to December 31, 2022.
Overall, our data element validity, empiric measure validity, and face validity analysis results support the validity of the Stroke Mortality measure. Each is described in more detail below. Measure validity is further supported by evidence of improvement, described in section 6.2.4. in the setting of quality improvement efforts.
Data Element Validity using Get With The Guidelines (GWTG)-Stroke Registry
There was substantial concordance between the NIH Stroke Scale scores within the GWTG-Stroke Registry and administrative claims data, as demonstrated by the proximity of scores, as well as the Pearson correlation coefficient of 0.993 and weighted kappa of 0.842. When compared to the GWTG-Stroke Registry NIH Stroke Scale scores, which have been validated through comparison to chart abstraction, the NIH Stroke Scale scores coded on administrative claims can be considered valid and reliable data elements for the adjustment of stroke severity of patients upon admission within the measure.
Empiric Measure Validity Testing
Our empiric validity testing examined two quality constructs: volume, and quality measures in the same causal pathway. Our results show a trend in the expected direction for the comparison between hospital volume and Stroke Mortality risk-standardized mortality rates (RSMRs), with higher volume significantly associated with better RSMR scores. Our results also show that there is a significant relationship in the expected direction between the Star Rating Summary Score and the Stroke Mortality score, even after you remove the entire Mortality Group Score from the comparison, suggesting (as has been shown for other mortality outcomes (Peter et al., 2024) that there is an association between mortality for one condition, and other components of Overall Hospital Star Rating (such as the Safety, Timely & Effective Care, Readmission, and/or Patient Experience components). Likewise, the results showing an association between the Mortality Group Score (without the Fee-for-Service (FFS)-only stroke measure) and Stroke mortality measure are consistent with the literature showing that mortality rates for different conditions are correlated at the hospital level (Horwitz et al., 2012).
Face Validity
The validity of the measure is supported by strong face validity results, as measured by systematic feedback from the TEP. As described above, 11 of 12 (91.7%) TEP members strongly, moderately, or somewhat agreed that the measure as specified can be used to distinguish between better and worse quality hospitals.
References
Horwitz, L. I., Wang, Y., Desai, M. M., Curry, L. A., Bradley, E. H., Drye, E. E., & Krumholz, H. M. (2012). Correlations among risk-standardized mortality rates and among risk-standardized readmission rates within hospitals. Journal of hospital medicine, 7(9), 690–696. https://doi.org/10.1002/jhm.1965
Peter, D., Li, S. X., Wang, Y., Zhang, J., Grady, J., McDowell, K., Norton, E., Lin, Z., Bernheim, S., Venkatesh, A. K., Fleisher, L. A., Schreiber, M., Suter, L. G., & Triche, E. W. (2024). Pre-COVID-19 hospital quality and hospital response to COVID-19: examining associations between risk-adjusted mortality for patients hospitalised with COVID-19 and pre-COVID-19 hospital quality. BMJ open, 14(3), e077394. https://doi.org/10.1136/bmjopen-2023-077394
Risk Adjustment
The goal of risk adjustment is to adjust for case-mix differences between hospitals. Risk adjustment supports fair and accurate comparison of outcomes across measured entities by including an adjustment for patient-level factors such as age, comorbid diseases, and indicators of patient frailty, which are clinically relevant and have relationships with the outcome.
In pursuing an approach that best leverages the data and analytical advancements since initial measure development, we developed and evaluated a framework to use individual ICD-10 codes for risk adjustment. The main advantage of leveraging ICD-10 codes in place of the prior method (that used an ICD-10 grouper, CMS’s Condition Categories, or CCs) is the ability to address the clinical heterogeneity found in the broadly defined CCs. Our previous research indicates that the model performance of the mortality measures is significantly improved by using individual codes instead of CCs (Krumholz et al., 2019).
Selection of Clinical Risk Variables
For candidate risk variables, we included all secondary ICD-10 codes documented as present-on-admission (POA) during the index admission (except for the palliative care code of Z51.5, which, effective October 1, 2021, was considered POA-exempt), and both principal and secondary ICD-10 codes in the 12 months prior to admission from any inpatient, outpatient, and professional provider claims. We also considered age, frailty, sex, an indicator for whether the admission was Medicare Advantage (MA) vs. Fee-for-Service (FFS), and other non-individual-ICD variables in the existing publicly reported CMS mortality measures. The variable selection of individual ICD codes mainly relied on data-driven methodologies involving three key steps: 1) pre-processing, 2) evaluating association with outcome, and 3) consideration of associations between other non-individual-code variables, including frailty, with the outcome.
In pre-processing, we screened and included index and history (pre-index) codes if their prevalence exceeded 0.5% and 2.5%, respectively. Further, co-occurring index and pre-index codes with Pearson correlation coefficients greater than 0.8 were combined into one risk variable. Finally, pairs of identical index and pre-index ICD-10 codes with similar odds ratios that acted in the same direction (where the difference in association with the outcome, measured by odds ratio (OR), was less than 0.2) were merged. We additionally excluded ICD-10 codes that begin with R297 since they are components of the National Institutes of Health Stroke Scale (NIHSS) variable included in the risk adjustment model for the measure as a numeric variable in a later stage. We note that specific Z codes for social risk factors were removed from the candidate list to allow for the selection of clinical risk variables; we later tested the impact of adding social risk factors to the model (See Section 5, Equity).
In the second step, we included the remaining candidate variables including age in a multivariable logistic regression model that underwent variable selection through 1,000 iterations of bootstrapping. We selected variables that were statistically significantly associated with outcomes (p<0.05) in at least 80% of the bootstrapped samples. Additional variables were added if there was a resulting increase in c-statistic of at least 0.0005 for each additional variable or an increase of at least 0.005 for including additional variables within the next 5% of bootstrapped samples (e.g. moving from 80% to 75%). Lastly, we included other non-individual-ICD variables from the current CMS mortality measures if their regression coefficients were statistically significant when added to the models.
Lastly, based on evidence from the literature, expert input, guidance from the consensus-based entity for measure endorsement, the Assistant Secretary for Planning and Evaluation (ASPE, 2020), input from other stakeholders, as well as prior testing results, we included a claims-based indicator of frailty that was developed for CMS’s Multiple Chronic Conditions measure (Yale New Haven Health Services Corporation/Center for Outcomes Research & Evaluation (YNHHSC/CORE), 2019) in the final model for all measures. We did not include sex as a variable since sex can be considered a socio-demographic variable.
After variable selection, we also added into the model the history of coronavirus disease 2019 (COVID-19) variable to be consistent with current CMS policy. In addition, we added into the model the NIHSS score variable derived from ICD-10 codes R29701–R29742 corresponding to NIHSS score 1–42. For patients with no R297 codes, we imputed the NIHSS score as 0.
For the combined MA and FFS cohort, the risk adjustment model was updated to include an MA indicator (versus FFS) as a main effect. This was to adjust for the generally higher prevalence of comorbidities in the MA cohort, especially among the pre-index variables that were derived from services in the outpatient setting (e.g., physician visits).
Adjustment for Stroke Severity with the NIH Stroke Scale (NIHSS)
Stakeholders have previously stressed the importance of including stroke severity in mortality measures for risk adjustment, as several studies have demonstrated that initial stroke severity is the strongest predictor of mortality in acute ischemic stroke patients (Bhaskar et al., 2017; Soriano-Tárraga et al., 2018; Lichtman et al., 2019; Iluţ et al., 2023). In response to this feedback, we have included the National Institutes of Health Stroke Scale (NIHSS) as an assessment of stroke severity in the risk-adjustment model, thereby accounting for stroke severity at the time of admission to assess the condition of the patient before care has been administered. The 2019 Update to the American Heart Association (AHA)/American Stroke Association (ASA) Guidelines for the Early Management of Acute Ischemic Stroke emphasizes the role of the NIHSS in evaluating initial stroke severity and its association with patient outcomes, particularly mortality (Powers et al., 2019). The guidelines note that the NIHSS is important not only for assessing the patient's condition but also for guiding the application of intravenous and intra-arterial therapies. Furthermore, these guidelines reinforce that higher NIHSS scores are strongly associated with worse functional outcomes and greater mortality risk, which supports the inclusion of this scale in risk adjustment models for stroke care evaluation.
Moreover, the inclusion of the NIH Stroke Scale has been shown to improve model discrimination for the publicly reported stroke mortality measure (Schwart et al., 2017). Additionally, Farooque et al. (2020) found that accounting for stroke severity using the NIHSS significantly improved the predictive accuracy of mortality models in stroke care and could more accurately identify high-performing institutions, which leads to better-targeted quality improvement initiatives. The NIHSS could also act as an independent predictor of early mortality after stroke (Bhaskar et al., 2017; Soriano-Tárraga et al., 2018; Ramachandran et al., 2022; Iluţ et al., 2023). This predictive capability allows healthcare providers to stratify patients based on their initial stroke severity and enable more personalized and efficient interventions.
Social Risk Factors
To inform our conceptual framework regarding the impact of social risk factors on stroke outcomes, we reviewed the existing literature. Our search was centered on articles that included key terms related to stroke mortality, disparities in socioeconomic and sociodemographic factors, and access to healthcare.
Our findings highlighted several demographic and socioeconomic variables that significantly influence stroke outcomes. Age, for instance, was an example of a determinant of stroke outcomes, with older adults experiencing higher mortality rates and poorer recovery outcomes post-stroke. Smithard (2017) noted that individuals aged 80 and above are more prone to increased frailty and multiple health conditions, which increases their risk of mortality and complicated recovery. Kelly and Rothwell (2021) further supported this by showing that frail older adults have a six-fold increase in mortality compared to those who are not frail, largely due to the compounded effects of frailty and pre-existing health conditions such as cardiovascular disease and diabetes.
Socioeconomic status (SES) is another factor influencing stroke outcomes. A systematic review and meta-analysis by Wang et al. (2020) revealed that individuals with the lowest SES had a 39% higher risk of stroke-related mortality compared to those with the highest SES. The study highlighted that lower SES populations often have reduced access to high-quality acute care and rehabilitation services which results in delayed treatment and poorer recovery after a stroke.
Disparities in stroke outcomes and care by race are also well documented and have been identified at the geographic and hospital level. A recent analysis by the CDC confirmed the well-known race disparity: at the county level, median stroke mortality rates were higher for Black vs. white Medicare beneficiaries over age 65 (1,214 vs 1,115 deaths per 100,000, respectively, or an absolute disparity of 61.5 deaths per 100,000 individuals (Evans et al., 2024). While there are racial disparities in the underlying risk factors for stroke, and in the incidence of stroke, there is also evidence for disparities in treatment within the healthcare system. A 2022 systematic review analyzed 30 studies that examined racial differences in hospital-based care and found disparities in rates of evidence-based treatment of ischemic stroke, lower use of emergency services, longer waiting times (emergency department, and time-to-treat), and lower referral rates to higher-level facilities among Hispanic and Black patients compared with white patients (Ikeme et al., 2022).
Social Risk Factor Conceptual Model
Our social risk factor conceptual model described below builds on the literature cited above and envisions several different pathways, including:
- Patients with social risk factors may have worse health at the time of hospital admission. Patients who have lower income/education/literacy or unstable housing may have a worse general health status and may present for their hospitalization or procedure with a greater severity of underlying illness. These social risk factors, which are characterized by patient-level or neighborhood/community-level (as proxy for patient-level) variables, may contribute to worse health status at admission due to competing priorities (restrictions based on job, lack of childcare, etc.), lack of access to care (geographic, cultural, or financial), or lack of health insurance. Given that these risk factors all lead to worse general health status, this causal pathway should be largely accounted for by current clinical risk adjustment.
- Patients with social risk factors may receive care at lower-quality hospitals. Patients of lower income, lower education, or unstable housing may have inequitable access to high-quality facilities, in part, because such facilities may be less likely to be found in geographic areas with large populations of patients with social risk factors. Thus, patients with low income may be more likely to be seen in lower-quality hospitals, which can contribute to an increased risk of stroke mortality.
- Patients with social risk factors may receive differential care within a hospital. The third major pathway by which social risk factors may contribute to mortality risk is that patients may not receive equivalent care within a facility. Alternatively, patients with social risk factors such as lower education may require differentiated care – e.g., provision of lower literacy information – that they do not receive.
- Patients with social risk factors may experience worse health outcomes beyond the control of the healthcare system. Some social risk factors, such as income or wealth, may affect the likelihood of death without directly affecting health status at admission or the quality of care received during the hospital stay. For instance, while a hospital may make appropriate care decisions and provide tailored care and education, a lower-income patient may have a worse outcome post-discharge due to competing economic priorities or a lack of access to care outside of the hospital.
These proposed pathways are complex to distinguish analytically. They also have different implications on the decision to risk adjust or not. Based on the evidence we reviewed and the conceptual model, and given the limited availability of valid and reliable variables for social risk that can be tested in claims data, we selected the following variables for testing:
Dual-Eligible (DE) Status
Dual eligibility for Medicare and Medicaid is available at the patient level in the Medicare Master Beneficiary Summary File. The eligibility threshold for Medicare beneficiaries aged 65 or older considers both income and assets. For the dual-eligible (DE) indicator, there is a body of literature demonstrating differential health care and health outcomes among beneficiaries (ASPE, 2020).
Area Deprivation Index (ADI)
While we previously used the AHRQ SES variable in these types of analyses, we now use the validated ADI (Forefront Group, 2023). We made this change to align with other CMS work on social risk factors that now use the ADI. We describe the ADI variable below.
The ADI, initially developed by the Health Resources & Services Administration, is based on 17 measures across four domains: income, education, employment, and housing quality (Kind et al., 2018; Singh, 2003).
The 17 components are listed below:
- Population aged ≥ 25 y with < 9 y of education, %
- Population aged ≥ 25 y with at least a high school diploma, %
- Employed persons aged ≥ 16 y in white-collar occupations, %
- Median family income, $
- Income disparity
- Median home value, $
- Median gross rent, $
- Median monthly mortgage, $
- Owner-occupied housing units, % (homeownership rate)
- Civilian labor force population aged ≥16 y unemployed, % (unemployment rate)
- Families below poverty level, %
- Population below 150% of the poverty threshold, %
- Single-parent households with children aged < 18 y, %
- Households without a motor vehicle, %
- Households without a telephone, %
- Occupied housing units without complete plumbing, % (log)
- Households with more than one person per room, % (crowding)
ADI scores were derived using the beneficiary’s 9-digit ZIP Code of residence, which is obtained from the Master Beneficiary Summary File and is linked to 2017-2021 US Census/American Community Service (ACS) data. In accordance with the ADI developers’ methodology, an ADI score is calculated for the census block group corresponding to the beneficiary’s 9-digit ZIP Code using 17 weighted Census indicators. Raw ADI scores were then transformed into a national percentile ranking ranging from 1 to 100, with lower scores indicating lower levels of disadvantage and higher scores indicating higher levels of disadvantage. Percentile thresholds established by the ADI developers were then applied to the ADI percentile to dichotomize neighborhoods into more disadvantaged (high ADI areas=ranking equal to or greater than 85) or less disadvantaged areas (low ADI areas=ranking of less than 85).
References
Bhaskar, S., Stanwell, P., Bivard, A., Spratt, N., Walker, R., Kitsos, G. H., Parsons, M. W., Evans, M., Jordan, L., Nilsson, M., Attia, J., & Levi, C. (2017). The influence of initial stroke severity on mortality, overall functional outcome and in-hospital placement at 90 days following acute ischemic stroke: A tertiary hospital stroke register study. Neurology India, 65(6), 1252-1259. https://doi.org/10.4103/0028-3886.217947
Department of Health and Human Services, Office of the Assistant Secretary of Planning and Evaluation. Report to Congress: Social Risk Factors and Performance under Medicare’s Value-based Payment Programs. December 21, 2016. (https://aspe.hhs.gov/pdf-report/report-congress-social-risk-factors-and-performance-under-medicares-value-based-purchasing-programs.
Department of Health and Human Services, Office of the Assistant Secretary of Planning and Evaluation (ASPE). Second Report to Congress: Social Risk Factors and Performance in Medicare’s Value-based Purchasing Programs. 2020; https://aspe.hhs.gov/system/files/pdf/263676/Social-Risk-in-Medicare%E2%80%99s-VBP-2nd-Report.pdf. Accessed July 2, 2020.
Evans, K., Casper, M., Schieb, L., DeLara, D., & Vaughan, A. S. (2024). Stroke mortality and stroke hospitalizations: Racial differences and similarities in the geographic patterns of high burden communities among older adults. Preventing Chronic Disease, 21, 230339. https://doi.org/10.5888/pcd21.230339
Farooque, U., Lohano, A. K., Kumar, A., Karimi, S., Yasmin, F., Bollampally, V. C., & Ranpariya, M. R. (2020). Validity of National Institutes of Health Stroke Scale for Severity of Stroke to Predict Mortality Among Patients Presenting With Symptoms of Stroke. Cureus, 12(9), e10255. https://doi.org/10.7759/cureus.10255
Ikeme, S., Kottenmeier, E., Uzochukwu, G., & Brinjikji, W. (2022). Evidence-based disparities in stroke care metrics and outcomes in the United States: A systematic review. Stroke, 53(3), 670-679. https://doi.org/10.1161/STROKEAHA.121.036263
Iluţ, S., Vesa, Ş. C., Văcăraș, V., & Mureșanu, D. F. (2023). Predictors of short-term mortality in patients with ischemic stroke. Medicina (Kaunas), 59(6), 1142. https://doi.org/10.3390/medicina59061142
Kelly DM, Rothwell PM, on behalf of the Oxford Vascular Study. Impact of multimorbidity on risk and outcome of stroke: Lessons from chronic kidney disease. Int J Stroke. 2021;16(7):758-770. doi:10.1177/1747493020975250.
Kind AJH, Buckingham W. Making Neighborhood Disadvantage Metrics Accessible: The Neighborhood Atlas. New England Journal of Medicine, 2018. 378: 2456-2458. DOI: 10.1056/NEJMp1802313. PMCID: PMC6051533. AND University of Wisconsin School of Medicine Public Health. 2023 Area Deprivation Index v4.0. Downloaded from https://www.neighborhoodatlas.medicine.wisc.edu/.
Krumholz HM, Coppi AC, Warner F, et al. Comparative effectiveness of new approaches to improve mortality risk models from Medicare claims data. JAMA Network Open. 2019;2(7):e197314-e197314.
Lichtman, J.H., Leifheit, E.C., Wang, Y., Goldstein, L.B. (2019). Hospital Quality Metrics: "America's Best Hospitals" and Outcomes After Ischemic Stroke. Journal of Stroke and Cerebrovascular Diseases, 28(2), 430-434. https://doi.org/10.1016/j.jstrokecerebrovasdis.2018.10.022
Powers, W. J., Rabinstein, A. A., Ackerson, T., Adeoye, O. M., Bambakidis, N. C., Becker, K., Biller, J., Tirschwell, D. L., & the American Heart Association Stroke Council. (2019). Guidelines for the early management of patients with acute ischemic stroke: 2019 update to the 2018 guidelines for the early management of acute ischemic stroke: A guideline for healthcare professionals from the American Heart Association/American Stroke Association. Stroke, 50(12), e344–e418. https://doi.org/10.1161/STR.0000000000000211
Ramachandran, K., Radha, D., Gaur, A., Kaliappan, A., & Sakthivadivel, V. (2022). Is the National Institute of Health Stroke Scale a valid prognosticator of the aftermath in patients with ischemic stroke? Journal of Family Medicine and Primary Care, 11(11), 7185–7190. https://doi.org/10.4103/jfmpc.jfmpc_611_22
Schwart, J., Wang, Y., Qin, L., Schwamm, L.H., Fonarow, G.C., Cormier, N., Dorsey, K., McNamara, R.L., Suter, L.G., Krumholz, H.M., Bernheim, S.M. (2017). Incorporating Stroke Severity into Hospital Measures of 30-Day Mortality After Ischemic Stroke Hospitalization. Stroke, 48(11), 3101-3107. https://doi.org/10.1161/strokeaha.117.017960
Singh, G. K. (2003). Area deprivation and widening inequalities in US mortality, 1969–1998. American Journal of Public Health. 93(7), 1137–1143. https://doi.org/10.2105/ajph.93.7.1137
Smithard DG. Stroke in frail older people. Geriatrics (Basel). 2017;2(3):24. doi:10.3390/geriatrics2030024.
Soriano-Tárraga, C., Giralt-Steinhauer, E., Mola-Caminal, M., Ois, A., Rodríguez-Campello, A., Cuadrado-Godia, E., Fernández-Cadenas, I., Cullell, N., Roquer, J., & Jiménez-Conde, J. (2018). Biological age is a predictor of mortality in ischemic stroke. Scientific Reports, 8, Article 4148. https://doi.org/10.1038/s41598-018-22574-z
Wang S, Zhai H, Wei L, Shen B, Wang J. Socioeconomic status predicts the risk of stroke death: A systematic review and meta-analysis. Prev Med Rep. 2020;19:101124. doi:10.1016/j.pmedr.2020.101124. PMCID: PMC7264080.
Yale New Haven Health Services Corporation/Center for Outcomes Research & Evaluation (YNHHSC/CORE). (2019). Methodology report, measure testing report, and risk-adjustment report: Clinician and clinician group risk-standardized hospital admission rates for patients with multiple chronic conditions. Centers for Medicare & Medicaid Services: Measure Instrument Development and Support; Development, Reevaluation, and Implementation of Outpatient Outcome/Efficiency Measures (Contract Number HHSM-75FCMC18D0042, Task Order HHSM-75FCMC19F0002).
The process that was used to identify risk variables is described in Section 4.4.2. Frequencies of risk variables in the final risk model are shown in Table 9 ("All Figures and Tables Stroke Mortality" attachment) and Section 4.4.4.
Please see Section 5 for testing related to social risk factors.
Table 9 ("All Figures and Tables Stroke Mortality" attachment) shows the odds ratios for the final risk variables selected by the process described in Section 4.4.2.
To assess model performance, we assessed model discrimination, calibration, and overfitting. To assess discrimination, we computed two discrimination statistics, the c-statistic and predictive ability (see Table 10 in the attachment). These analyses used the CY2022 dataset (derivation and validation). For calibration, we provide calibration (risk-decile) plots (see Figures 4A and 4B in the attachment) for CY2022 and CY2023 data.
The c-statistic is the probability that predicting the outcome is better than chance, which is a measure of how accurately a statistical model can distinguish between a patient with and without an outcome.
Predictive ability measures the ability to distinguish high-risk subjects from low-risk subjects; therefore, for a model with good predictive ability, we would expect to see a wide range in observed outcomes between the lowest and highest deciles of predicted outcomes. To calculate the predictive ability, we calculated the range of mean observed stroke mortality between the lowest and highest predicted deciles of stroke mortality probabilities.
For model calibration, we assessed calibration plots, with mean predicted and mean observed stroke mortality plotted against deciles of predicted stroke mortality. The closer the predicted outcomes are to the observed outcomes, the better calibrated the model is. We provide results for CY2022 (derivation sample) and CY2023 (validation sample).
In addition, we provide an analysis of overfitting. Overfitting refers to the phenomenon in which a model accurately describes the relationship between predictive variables and outcomes in the development dataset but fails to provide valid predictions in new patients. Estimated calibration values of γ0 close to 0 and estimated values of γ1 close to 1 provide evidence of good calibration of the model.
Model Performance Testing Results
Please see Table 10 and Figures 4A and 4B in the attachment "All Figures and Tables Stroke Mortality " for the model testing results.
The c-statistic was 0.911 in the derivation sample, and 0.915 in the validation sample (Table 10). Predictive ability ranged from 0.6%-75.2% in the derivation sample, and 0.5%-74.1% in the validation sample. Risk decile plots show that higher deciles of the predicted outcomes are associated with higher observed outcomes in both CY2022 and CY2023 data (Figures 4A and 4B). Overfitting results are shown in Table 9.
Discrimination
The c-statistic of 0.991 in the development sample, and 0.915 in the validation sample indicate excellent model discrimination. The model indicated a wide range between the lowest decile and highest decile, indicating the ability to distinguish high-risk subjects from low-risk subjects.
Calibration
Higher deciles of the predicted outcomes are associated with higher observed outcomes, which show a good calibration of the model. The model has good predictive ability as shown by the wide range in observed outcomes between the lowest and highest deciles of predicted outcomes.
Over-fitting (γ0, γ1)
If γ0 is substantially far from zero and γ1 is substantially far from one, there is potential evidence of over-fitting. Our testing results show calibration values of almost 0 at one end and almost one on the other end indicating good calibration of the model.
Overall Interpretation
Interpreted together, our diagnostic results demonstrate the risk-adjustment model adequately controls for differences in patient characteristics (case mix).
Use & Usability
Use
Usability
Several hospital-level factors can significantly influence stroke mortality outcomes. For instance, lower case volume and the absence of specialized stroke care teams have been linked to increased mortality rates among stroke patients. Witrick et al. (2020) underscored the importance of the time of patient arrival and the hospital's involvement in a telestroke network in shaping stroke outcomes. Telestroke networks are a type of telemedicine specifically designed to improve the care of patients experiencing an acute ischemic stroke, particularly in regions where immediate access to specialized stroke care is limited. The study findings revealed that patients arriving at the hospital during nighttime hours had a 22% higher likelihood of in-hospital mortality compared to those who arrived during the day. However, hospitals that were part of a telestroke network experienced a significant decrease in mortality rates, with patients in these facilities having a 53% lower likelihood of dying compared to those in non-telestroke hospitals. This impact was even more noticeable in larger hospitals, which indicates that access to specialized stroke care and the adoption of telestroke networks are important in improving stroke survival outcomes.
Similarly, the involvement of less experienced providers or those outside of specialized stroke centers has been correlated with high stroke mortality. In a study by Zhao et al. (2024), nurses in neurology wards with less specialized stroke training showed lower proficiency in stroke assessment and care. This lack of expertise was associated with higher stroke mortality rates, especially in settings where comprehensive stroke care protocols were not strictly adhered to. The study highlighted the importance of targeted education and training programs for nurses in non-specialized or community settings to ensure that all stroke patients receive high-quality care, regardless of where they are treated. The findings suggest that less experienced providers or those working outside specialized stroke centers may contribute to poorer patient outcomes.
Additionally, it has been shown that the use of integrated, multidisciplinary stroke care teams (comprised of neurologists, rehabilitation specialists, and nurses with expertise in stroke care) results in significant reductions in mortality and improvement in functional outcomes for stroke patients. Rodgers and Price (2017) highlight the role of specialized stroke units, where a multidisciplinary team works together to deliver comprehensive care; patients who receive care in these specialized stroke units experience lower mortality rates and are less likely to need long-term institutional care compared to those treated in general medical wards. Other studies also support that the use of integrated, interdisciplinary care teams, including neurologists, radiologists, nurses, and rehabilitation specialists, can contribute to improved patient outcomes and improved post-stroke recovery (Clarke and Forster, 2015; Lip et al., 2022). These models address shared risk factors for stroke and cardiovascular disease and incorporate telemedicine and educational initiatives to engage and empower patients and caregivers. The European Society of Cardiology Council on Stroke has developed a consensus position on optimizing stroke and heart disease management that highlights the importance of coordinated and individualized care pathways (Lip, 2022). For example, the post-stroke ABC pathway, which encompasses antithrombotic therapy, functional and psychological status, and cardiovascular risk factors and comorbidities, is one example of a holistic framework to improve patient outcomes and quality of life (Lip, 2017).
Finally, for measures implemented in CMS hospital reporting programs, hospitals can use, for quality improvement, the hospital-specific reports (HSR) provided by CMS. This report provides hospitals with their detailed measure results, discharge-level data, and state and national results so that hospitals can target their quality improvement efforts.
References
Clarke, D. J., & Forster, A. (2015). Improving post-stroke recovery: The role of the multidisciplinary health care team. Journal of Multidisciplinary Healthcare, 8, 433-442. https://doi.org/10.2147/JMDH.S68764
Lip GYH. The ABC pathway: an integrated approach to improve AF management. Nat Rev Cardiol. 2017 Nov;14(11):627-628. doi: 10.1038/nrcardio.2017.153. Epub 2017 Sep 29. PMID: 28960189.
Lip, G. Y. H., Lane, D. A., Lenarczyk, R., Boriani, G., Doehner, W., Benjamin, L. A., Fisher, M., Lowe, D., Sacco, R. L., Schnabel, R., Watkins, C., Ntaios, G., & Potpara, T. (2022). Integrated care for optimizing the management of stroke and associated heart disease: A position paper of the European Society of Cardiology Council on Stroke. European Heart Journal, 43(26), 2442-2460. https://doi.org/10.1093/eurheartj/ehac245
Rodgers H, Price C. Stroke unit care, inpatient rehabilitation and early supported discharge. Clin Med (Lond). 2017;17(2):173-177. doi:10.7861/clinmedicine.17-2-173. PMCID: PMC6297619. PMID: 28365632.
Witrick B, Zhang D, Switzer JA, Hess DC, Shi L. The association between stroke mortality and time of admission and participation in a telestroke network. J Stroke Cerebrovasc Dis. 2020;29(2):104480. doi:10.1016/j.jstrokecerebrovasdis.2019.104480. PMCID: PMC6954319.
Zhao Y, Xu Y, Ma D, et al. The impact of education/training on nurses caring for patients with stroke: a scoping review. BMC Nurs. 2024;23:90. Published online 2024 Feb 2. doi:10.1186/s12912-024-01754-x. PMCID: PMC10835862. PMID: 38308293.
This Stroke Mortality measure is not yet implemented; it is a re-specified version of the currently reported Fee-for-Service (FFS)-only stroke measure that is implemented in the Inpatient Quality Reporting Program (IQR). The Stroke Mortality measure was re-specified twice, once in 2016, driven by stakeholder feedback that the measure should risk adjust for stroke severity upon admission. In 2016, a workgroup consisting of neurologists, cardiologists, and experts in biostatistics, measurement, and quality improvement was convened to provide clinical expertise on the measure. The workgroup met regularly throughout development to address key issues related to measure cohort, outcome, and usability. The measure also received feedback through rulemaking public comment.
In 2023, the measure was again respecified, again driven by stakeholder feedback to expand the cohort to include Medicare Advantage beneficiaries. We also convened a Technical Expert Panel to provide input into further improving the measure through the re-selection of clinical risk variables, described in Section 4.4.2.
On an ongoing basis, the measured entities (hospitals that provide acute inpatient and outpatient care) and other stakeholders or interested parties submit questions or comments about the measure through a Q&A portal Experts on measure specifications, calculation, or implementation prepare responses to those inquiries and reply directly to the sender. We consider issues raised through the Q&A process about measure specifications or measure calculation in measure reevaluation and in the re-specification of this measure. The majority of inquiries received from hospitals through the Q&A process relate to clarifying questions about data sources and aspects of the methodology, specific questions related to hospital performance outlined within Hospital-Specific Reports, and requests for code set files and Statistical Analysis System (SAS) code.
In addition, we routinely scan literature repositories for scholarly articles describing research related to this measure. We summarize new information obtained through these reviews every three years as a part of comprehensive reevaluation as mandated by the Measure Management System (MMS) Blueprint.
Of note, we received stakeholder feedback that the currently reported stroke mortality measure should adjust for stroke severity upon admission using the National Institute of Health Stroke Scale (NIHSS). As noted above, we also subsequently expanded the measure’s cohort to include Medicare Advantage beneficiaries in response to stakeholder feedback.
Each year, issues raised through the Q&A process or in the literature related to the currently reported stroke mortality measure are considered by measure and clinical experts. Any issues that warrant additional analytic work due to potential changes in the measure specifications are addressed as part of the annual measure reevaluation. If small changes are indicated after additional analytic work is complete, those changes are usually incorporated into the measure in the next measurement period. If the changes are substantial, CMS may propose the changes through rulemaking and adopt the changes only after CMS receives public comment on the changes and finalizes those changes in the Inpatient Prospective Payment System (IPPS) or another rule. A similar process will be maintained for this re-specified version of the measure.
As noted in Section 2.3, there has been improvement across most of the distribution of risk-standardized scores for this measure. We compared performance between prior to measure implementation (dates of data, 7/2010-6/2013) and the most recent performance period prior to COVID and prior to the addition of the NIHSS in 2022 (7/1/2017-12/1/2019), and the current re-specification. These results show improvement across the distribution of the measure, with lower mean (FFS-only) RSMRs (13.5 vs. 15.4), and lower RSMRs across the rest of the distribution (25th percentile, 12.9 vs. 14.6; 50th percentile, 13.4 vs. 15.3; 75th percentile, 14.1 vs. 16.1).
We did not identify any unintended consequences during measure development or model testing. However, we are committed to monitoring this measure’s use and assessing potential unintended consequences over time, such as the inappropriate shifting of care, increased patient morbidity and mortality, and other potential unintended consequences for patients.
Comments
Staff Preliminary Assessment
CBE #4595 Staff Assessment
Importance
Strengths:
- The developer cites evidence of the economic impact of stroke, with costs related to health care services, medications, and lost productivity nearing $56.2 billion between 2019 and 2020.
- Stroke mortality rates are expected to continue declining, supported by historical trends and recent hospital-level improvements since the implementation of the Fee-for-Service (FFS) stroke mortality measure in 2014, which has shown significant enhancements in 30-day risk-standardized outcomes.
- Hospital-level interventions, including participation in quality improvement programs like Get With The Guidelines (GWTG) and the use of integrated care models with multidisciplinary teams, have been shown to significantly reduce stroke mortality rates and improve patient outcomes through timely and coordinated care, highlighting the importance of targeted hospital practices in enhancing stroke care.
- The developer posits that the stroke mortality measure fills a crucial gap in publicly reported outcomes, complementing other stroke-related measures and enhancing transparency and accountability in stroke care quality. Mortality is an important outcome to patients, and public reporting of the measure provides transparency and accountability, crucial for improving stroke care and outcomes.
- The developer provided performance gap data from claims-only FFS and MA (2022 - 2023). The results show a mean of 6.43% with a decile range of 9.66% - 18.37% (a difference of 8.71%).
Limitations:
- As a new measure, performance gap analysis is not required. However, despite the range in performance scores by decile, a 8.71% difference across deciles suggests a narrowing performance gap between providers.
Rationale:
- This new measure, utilizing Medicare FFS and MA data, aims to replace the current claims-only measure in the Hospital Inpatient Quality Reporting program. It highlights that hospitals employing evidence-based actions and participating in programs like Get With The Guidelines often see improved outcomes through integrated care models and multidisciplinary teams. However, the measure's logic model diagram includes generalized hospital inputs that may not be fully supported by the evidence provided, and lacks some details such as high service volume and staff training. The measure is intended to enhance transparency and accountability in stroke care, informing both patients about hospital performance.
Feasibility Acceptance
Strengths:
- The measure utilizes routinely generated claims data, requiring no additional data collection by hospitals, thus eliminating reporting burdens and implementation challenges.
- Data are securely processed by CMS, ensuring no risks to patient confidentiality and allowing a three-month period post-performance for finalizing claims to ensure data accuracy and completeness.
- Feedback from health care facilities indicates no significant concerns regarding implementation burden or confidentiality issues, as the measure relies on securely submitted claims data managed by CMS for both payment processing and measure calculations. The measure is not a proprietary measure and no proprietary components.
Limitations:
- None.
Rationale:
- The measure leverages routinely generated claims data, requiring no additional data collection by hospitals, thus minimizing reporting burdens and implementation challenges. Managed and processed securely by CMS, this measure ensures patient confidentiality and data accuracy, with healthcare facilities reporting no significant concerns regarding its implementation.
Scientific Acceptability
Strengths:
- Data Sources and Dates: Data used for testing were sourced from CY2022-CY2023 Medicare Fee-for-Service and Medicare Advantage data and includes facilities with at least 25 cases. The reliability has been calculated using 2 years of data, and so the reliability results are only applicable if the measure is also calculated using two years of data.
- Accountable Entity-Level Reliability: The developer conducted signal-to-noise reliability testing at the accountable entity-level using the hospital intercepts from the hierarchical logistic regression model.
Limitations:
- Despite the signal-to-noise testing, the within hospital variance of the calculated RSRR may not scale with the variance of the logistic distribution that is used to calculate the reliability. So although more than 70% of accountable entities meet the expected threshold of 0.6, this estimate of reliability may not reflect the reliability of the RSRR.
Rationale:
- There are potential issues with the accuracy of the results using the current reliability metrics. However, the identified limitations are deemed addressable, as the developer may consider performing additional reliability testing such as split-half reliability. By addressing this issue, there is potential to enhance the reliability.
Recommended Actions to Address Reliability Concerns:- Refine Reliability Testing Methods: Adjust the reliability testing methods or analytic approaches to improve the assessment the reliability of the final measure.
Strengths:
- The developer provided person- or episode-level validity for the NIH Stroke Scale (NIHSS), demonstrating strong association between the GWTG registry and administrative claims. The developer also provided face validity evidence among the TEP (N=12) with the statement "The risk-standardized mortality rate obtained from the measures as specified can be used to distinguish between better and worse quality hospitals”; however the results did not represent consensus. The developer also provided an association study that demonstrated correlations with volume and with selected Star Ratings measures: Overall Summary Score, Overall Summary Score without Entire Mortality Group, Mortality Group Score, Mortality Group Score without the FFS only Stroke Mortality Measure.
- The developer conducted statistical risk adjustment, based on a conceptual model, selecting risk factors that have a significant correlation to the outcome. The developer also explored social risk factors, such as dual eligibility and the Area Deprivation Index. The developer [did not include these in the final models due to the minimal impact these social risk factors have on the measure scores and to the model overall. The developer reported a c-statistic of 0.911, indicating good model discrimination.
Limitations:
- The main limitation is the lack of material variation among accountable entities in the Importance Table which does not support a causal inference between the entity and the measure focus. For accountable entity validity, causal claims based on association (correlation) studies alone are prone to bias (i.e., confounding due to a common cause cannot be ruled-out). Additional support from mechanism studies that confirm the existence of a suitable (plausible) mechanism capable of accounting for the observed correlation would strengthen the causal claim. Otherwise, statements about the relative magnitude of the observed correlations and whether those magnitudes are greater or lesser than what one might anticipate are difficult to evaluate. Going forward, additional studies that either rule-out potential confounding or describe features of potential mechanisms will strengthen causal claims. A demonstrated association with volume would support ruling out potential confounders; however in this instance the association with volume, although conceptually supported, is not empirically compelling.
Rationale:
- As a new measure person- or episode-level validity is sufficient, although this measure is new only with respect to the risk adjustment model.
- The risk adjustment methods used are appropriate and demonstrate variation in the prevalence of risk factors across measured entities, contribute to unique variation in the outcome, and show the impact of risk adjustment for providers at high or low extremes of risk. The model performance is acceptable.
Equity
Strengths:
- The developer evaluated two social risk factors, dual eligibility status (DE) and high Area Deprivation Index (ADI).
- Analysis of one year of data (CY2022) for hospitals with at least 25 admissions shows that the median percentage of DE patients was 12.6%, and for high ADI patients, it was 8.4%.
- Unadjusted stroke mortality rates were higher for patients with these social risk factors, with DE patients showing a 14.1% mortality rate compared to 12.6% for non-DE patients, and high ADI patients showing a 13.2% rate compared to 12.8% for low ADI patients.
- The differences in measure scores with and without these social risk factors are minimal, indicating little impact on the overall measure scores.
- CMS has decided not to adjust the measure for social risk factors in order to avoid masking disparities in outcomes, as the measure is part of a pay-for-reporting program, not a pay-for-performance program.
Limitations:
- The developer provides rationale as to why DE and high ADI are not included in the risk adjustment model as the inclusion of these social risk factors did not significantly impact the measure scores. This is not the intent of this domain.
- Rather, the developer should consider stratification of the measure scores by identified social risk factors.
Rationale:
- The developer evaluated two social risk factors, DE and ADI. Despite higher unadjusted rates of adverse outcomes among patients with high social risk, the empirical results showed minimal impact on adjusted measure scores. Consequently, CMS decided not to adjust the measure for these factors. However, the limitation noted is the lack of stratification by social risk factors, which could provide deeper insights into disparities in outcomes.
Use and Usability
Strengths:
- The measure has a planned use in the Hospital Inpatient Quality Reporting (HIQR) program to replace the fee-for-service-only measure.
- The developer refers to evidence-based actions hospitals can take to reduce mortality rates, including being a part of telestroke networks, having specialized stroke care teams, and providing stroke care training to staff.
- The developer further notes that hospitals can also use detailed reports from CMS to support quality improvement. Hospitals and other stakeholders actively participate in shaping the measure through a Q&A portal, where they can submit inquiries about measure specifications and calculations. This ongoing feedback is considered in the measure's reevaluation and re-specification processes, supplemented by a routine review of related scholarly articles to ensure the measure remains current and effective.
- Potential changes to the measure, whether minor or substantial, are addressed through additional analytic work and, if necessary, proposed through rulemaking.
- The developer notes this measure is a re-specified version of the existing FFS stroke measure already in use. It has been modified based on stakeholder feedback to include risk adjustments for stroke severity and to expand the cohort to MA beneficiaries. This measure has undergone continuous refinement through input from a workgroup of neurologists, cardiologists, and biostatistics experts, and through public comments during rulemaking.
- Feedback has emphasized the need for the stroke mortality measure to adjust for stroke severity using the National Institute of Health Stroke Scale (NIHSS), leading to its inclusion in the measure's specifications. Since this respecified measure is not in use, the developer reported comparisons of data from before the FFS-only measure's implementation (2010-2013) to the period just before the NIHSS addition (2017-2019) and the re-specified measure. The results show a reduction in risk-standardized mortality rates (RSMRs), indicating improvement over time.
The developer reports no unintended consequences have been identified.
Limitations:
- None.
Rationale:
- This new measure, planned for use in the HIQR program, incorporates significant improvements based on extensive stakeholder feedback and evidence-based practices. Hospitals are encouraged to engage in telestroke networks, maintain specialized stroke care teams, and enhance staff training to reduce mortality rates, with additional support from detailed CMS reports for ongoing quality improvement. The measure, continuously refined through expert input and public comments, now includes risk adjustments for stroke severity and an expanded cohort to include MA beneficiaries.
Committee Independent Review
Share concern around 25…
Importance
Agree w staff.
Feasibility Acceptance
Agree with staff.
Scientific Acceptability
Agree with staff.
Agree with staff.
Equity
Agree with staff.
Use and Usability
Agree with staff.
Summary
Share concern around 25 cases being required to calculate but not report - why not make a requirement for calculation of performance.
support
Importance
agree with staff
Feasibility Acceptance
agree with staff
Scientific Acceptability
agree with staff regarding limitations
agree with staff
Equity
agree with staff regarding need for stratified results
Use and Usability
agree with staff
Summary
Has some relatively minor caveats that I believe the developer can address, as noted under the individual domains
Summary
Importance
Agree with staff recommendation.
Feasibility Acceptance
Agree with staff recommendation.
Scientific Acceptability
Agree with staff recommendation.
Agree with staff recommendation.
Equity
Agree with staff recommendation.
Use and Usability
Agree with staff recommendation.
Summary
I support the measure.
Support
Importance
Agree with staff
Feasibility Acceptance
Agree with staff
Scientific Acceptability
Agree with staff
Agree with staff
Equity
Agree with staff
Use and Usability
Agree with staff
Summary
Agree with staff
Approve
Importance
Agree with staff assessment
Feasibility Acceptance
Agree with staff assessment
Scientific Acceptability
Agree with staff assessment
Agree with staff assessment
Equity
Interesting prelim findings that need to e explored more.
Use and Usability
Agree with staff assessment
Summary
No substantial concerns but would like to see equity fleshed out
Support with caveat
Importance
agree with staff
Feasibility Acceptance
Agree with staff
Scientific Acceptability
Agree with staff. The AMA comments are relevant and although addressed by the authors I wonder about the benefit of generating a score for less than 25 admissions if it is unreliable.
Agree with staff
Equity
Agree with staff
Use and Usability
Agree with staff
Summary
AMA comment should be considered. All hospitals get scored even if they have less than 25 eligible patients. The authors indicate those with fewer than 25 don’t get reported on. Is there a risk that an unreliable score (less than 25 patients) would get reported; are there other risks to unreliability for smaller facilities? is there a risk to not generating a score for facilities in which the score could be unreliable due to too few cases?
Public Comments
4595 Hospital 30-Day, All-Cause, Risk-Standardized Mortality Rat
The American Medical Association (AMA) believes that a case minimum of at least 25 individuals must be required as a part of endorsement of this measure. This minimum would ensure that the measure’s minimum reliability is close to 0.7, which is what we believe should be the standard for endorsed measures.
Stroke Mortality Public Comment Response
We thank the commentor for their interest in the Stroke Mortality measure. The minimum case volume is not part of the measure specifications; all hospitals receive a measure score but the measure score is only reported for hospitals with at least 25 cases. We note that the minimum signal-to-noise reliability for this measure, for hospitals with at least 25 cases, is above the 0.6 threshold set by Battelle for endorsement maintenance.