The 30-Day Risk Standardized All-Cause Emergency Department Visit Following an Inpatient Psychiatric Facility (IPF) Discharge (IPF ED Visit) measure assesses the proportion of patients ages 18 and older with an emergency department (ED) visit, including observation stays, for any cause, within 30 days of discharge from an IPF, without subsequent admission. The IPF ED Visit measure is an outcome-based measure.
- Measure TypeComposite MeasureNoElectronic Clinical Quality Measure (eCQM)Level Of AnalysisCare SettingMeasure Rationale
The 30-Day Risk Standardized All-Cause Emergency Department Visit Following an Inpatient Psychiatric Facility Discharge (IPF ED Visit) is a new measure that has not been endorsed, and therefore does not have an associated web page at this time. It has been well established that the first three months following an inpatient psychiatric facility (IPF) discharge, particularly the first month, is a period of high risk for readmissions, rehospitalization, and suicidality (Mutschler et al. 2019). The rationale for the IPF ED Visit measure is to encourage IPFs to proactively focus on discharge planning and community reintegration during patients’ IPF stays. The measure is complementary to the 30-Day All-Cause Unplanned Readmission Following Psychiatric Hospitalization in an IPF (IPF Readmission) measure (NQF #2860) and Follow-Up After Psychiatric Hospitalization (FAPH) measure. The proposed measure complements the IPF Readmission measure by providing information on emergency department (ED) visits, including observation, without readmission, thereby more comprehensively assessing IPF discharge planning quality and promoting care coordination post-discharge.
Relevant literature:
- Mutschler, C., S. Lichtenstein, S.A. Kidd, and L. Davidson. “Transition Experiences Following Psychiatric Hospitalization: A Systematic Review of the Literature.” Community Mental Health Journal, vol. 55, no. 8, 2019, pp. 1255–1274. doi.org/10.1007/s10597-019-00413-9
Measure WebpageMAT output not attachedAttachedData dictionary not attachedYesNumeratorThe numerator is comprised of patients 18 and older with an emergency department (ED) visit, including observation stays, for any cause, within 30 days of discharge from an inpatient psychiatric facility (IPF), without subsequent admission. An ED visit is defined as any ED visit or observation stay that does not result in an admission or transfer and occurs within 30 days after the discharge date from an eligible index admission to an IPF during the measurement period.
Numerator DetailsThe numerator is comprised of patients 18 and older with an ED visit, including observation stays, for any cause, within 30 days of discharge from an IPF, without subsequent admission. An ED visit is defined as any ED visit or observation stay that does not result in an admission or transfer and occurs within 30 days after the discharge date from an eligible index admission to an IPF during the measurement period.
DenominatorThe measure population consists of patients with eligible index admissions to IPFs during the measurement period. Index admissions are defined as admissions to IPFs for patients with the following characteristics:
- Age 18 or older at admission
- Discharged alive
- Enrolled in Medicare FFS Parts A and B during the 12 months prior to, the month of, and at least three months after the month of discharge
- Discharged from IPF with a psychiatric or substance use disorder principal diagnosis
Denominator DetailsThe measure population consists of patients with eligible index admissions to IPFs during the measurement period. Index admissions are defined at admissions to IPFs for patients with the following characteristics:
- Age 18 or older at admission to an IPF;
- Discharged alive from an IPF;
- Enrolled in Medicare FFS Parts A and B during the 12 months prior to, the month of, and at least three months after the month of discharge;
- Discharged from IPF with a psychiatric or substance use disorder principal diagnosis.
Denominator ExclusionsDenominator exclusions include:
- Patients discharged against medical advice (AMA) from the IPF index admission;
- Patients transferred from the IPF to another care facility such as an acute care hospital, skilled nursing facility, long-term acute care facility, or residential program;
- Patients with unreliable demographic and vital status data defined as the following:
- Age greater than 115 years;
- Missing gender data;
- Discharge status of “dead” but with subsequent admissions;
- Death date prior to admission date;
- Death date within the admission and discharge dates but the discharge status was not “dead.”
Denominator Exclusions DetailsDenominator exclusions include the following:
- Discharged AMA. Index admissions where there is an indicator in the Medicare claims data (discharge status code ‘07’) that patients left AMA are excluded because the facility may have limited opportunity to complete treatment and prepare for discharge.
- Unreliable Data. Index admissions with unreliable demographic and death information are excluded from the denominator. Unreliable demographic information is defined as age greater than 115 years or missing gender. Unreliable death information is defined as:
- An admission with a discharge status of “dead” (discharge status code ‘20’) but the person has subsequent admissions;
- The death date is prior to the admission date; or
- The death date is within the admission and discharge dates for an admission, but the discharge status is not “dead” (discharge status code is not ‘20’).
- Transfers/Interrupted Stays. Index admissions that result in a transfer or interrupted stay are excluded because transfers and interrupted stays cannot always be distinguished from true readmissions in the claims data. This exclusion is defined as an index admission with a readmission on Days 0, 1, or 2 post-discharge.
Type of ScoreMeasure Score InterpretationBetter quality = Lower scoreCalculation of Measure ScoreDenominator. The denominator is the measure population which consists of patients with eligible index admissions to IPFs during the measurement period. Index admissions are defined at admissions to IPFs for patients with the following characteristics:
- Age 18 or older at admission,
- Discharged alive,
- Enrolled in Medicare FFS Parts A and B during the 12 months prior to, the month of, and at least three months after the month of discharge,
- Discharged from IPF with a psychiatric or substance use disorder principal diagnosis.
Denominator Exclusions. Denominator exclusions include:
- Patients discharged AMA from the IPF index admission,
- Patients transferred from the IPF to another care facility such as an acute care hospital, skilled nursing facility, long-term acute care facility, or residential program,
- Patients with unreliable demographic and vital status data defined as the following:
- Age greater than 115 years,
- Missing gender data,
- Discharge status of “dead” but with subsequent admissions,
- Death date prior to admission date,
- Death date within the admission and discharge dates but the discharge status was not “dead.”
Discharged AMA is calculated by index admissions where there is an indicator in the claims data (discharge status code ‘07’) that patients left AMA are excluded because the facility may have limited opportunity to complete treatment and prepare for discharge. Unreliable data is determined by index admissions with unreliable demographic and death information are excluded from the denominator. Unreliable demographic information is defined as age greater than 115 years or missing gender. Unreliable death information is defined as:
- An admission with a discharge status of “dead” (discharge status code ‘20’) but the person has subsequent admissions;
- The death date is prior to the admission date; or
- The death date is within the admission and discharge dates for an admission but the discharge status is not “dead” (discharge status code is not ‘20’).
Transfers/Interrupted Stays are determined as index admissions that result in a transfer or interrupted stay are excluded because transfers and interrupted stays cannot always be distinguished from true readmissions in the claims data. This exclusion is defined as an index admission with a readmission on Days 0, 1, or 2 post-discharge.
Numerator. The numerator is comprised of patients 18 and older with an emergency department (ED) visit, including observation stays, for any cause, within 30 days of discharge from an inpatient psychiatric facility (IPF), without subsequent admission. An ED visit is defined as any ED visit or observation stay that does not result in an admission or transfer and occurs within 30 days after the discharge date from an eligible index admission to an IPF during the measurement period.
Measure Period. The measure period includes adult IPF admission with admission and discharges between June 1, 2019, and July 31, 2021, were used to calculate the IPF ED Visit measure performance rate. We excluded cases during the first two quarters of 2020 due to the COVID-19 public health emergency, being discharged alive with a psychiatric principal discharge diagnosis, and being enrolled in FFS.
Measure Calculation. The key algorithm steps for measure calculation are as follows:
- Identify all IPF admissions in the performance period.
- Apply inclusion and exclusion criteria to identify index admissions.
- Identify ED visits within 30 days of discharge from each IPF index admission.
- Identify risk factors in the 12 months before index admission and during the index admission.
- Run a hierarchical logistic regression to compute the risk-standardized readmission rate for each IPF.
- Hierarchical logistic regression is used to model the log-odds of the ED visit. The two-level specification enables reliable estimates for small-volume hospitals, while accepting a certain amount of shrinkage toward the mean. The model includes risk factors as fixed effects and a hospital-specific intercept as a random effect. The estimate of hospital-specific intercept reflects the quality of care received at an IPF after adjusting for case mix.
- A standardized risk ratio (SRR), which is the predicted number of readmissions over the expected number of readmissions, is calculated for each IPF. The predicted number of ED visits is based on the IPF’s performance and observed case mix; it is calculated by taking the mean of the estimated probabilities of an ED visit for the index admissions at the IPF, based on the IPF-specific intercept and all other risk factors. The expected number of ED visits is based on the national observed ED visit rate in the measure cohort and the IPF’s observed case mix; it is calculated by taking the mean of the estimated probabilities of an ED visit for the index admissions contributing to the IPF, based on the average intercept across all IPFs nationwide and all other risk factors.
- The risk-standardized ED visit rate is calculated by multiplying the SRR by the overall national ED visit rate. An SRR greater than the national average indicates worse quality of care compared with the national average. An SRR less than the national average indicates better quality of care. The confidence interval of the SRR is calculated by bootstrapping to take into account the uncertainty of the estimate.
Measure Stratification DetailsNot applicable; this measure is not risk-stratified. Please refer to the Risk-factor coeff. all risk-fct tab and Candidate risk factor freq tab in the Risk-model specifications.xlsx file for the final risk-variables selected for the risk-adjustment model andthe candidate risk-variables that were considered for inclusion into the model. Please see the attached Excel file for the codebook as well.
All information required to stratify the measure resultsOffAll information required to stratify the measure resultsOffTesting Data SourcesData Sources- Medicare beneficiary and coverage files. Provides information on patient demographic, enrollment, and vital status to identify the measure population and certain risk factors.
- Medicare fee-for-service (FFS) Part A records. Contains final action claims submitted by acute care and critical access hospitals, IPFs, home health agencies, and skilled nursing facilities to identify the measure population, readmissions, and certain risk factors.
- Medicare FFS Part B records. Contains final action claims submitted by physicians, physician assistants, clinical social workers, nurse practitioners, and other outpatient providers to identify certain risk factors. For this measure, claims for services such as laboratory tests, medical supplies, or other ambulatory services were not used. This ensures that diagnoses result from an encounter with a provider trained to establish diagnoses and not from a claim for a diagnostic test.
Minimum Sample SizeThe minimum sample size for reporting this measure is IPFs with at least 25 discharges during the measurement period.
- Evidence of Measure Importance
Studies suggest readmissions and rehospitalizations, particularly within 30 days of discharge from an initial psychiatric hospitalization, are an indicator of poor service delivery and treatment across the psychiatric care continuum (Durbin et al. 2007; Feigenbaum et al. 2012). The first three months following IPF discharge, particularly the first month, are a period of high risk for the patient, particularly for suicidality, such as suicidal ideation and self-harm (Mutschler et al. 2019; Chung et al. 2017). In an assessment of 2008 Medicare claims data, Blair et al. (2019) found ED visits are common 30 days before and after an IPF stay; specifically, 29 percent of beneficiaries had an ED visit 30 days following an IPF discharge. Additionally, Venkatesh et al. (2020) found that Medicare beneficiaries disproportionately visit the ED compared to non-Medicare beneficiaries. One reason for an ED visit following an IPF discharge could be a lack of post-discharge follow-up. Blair et al. (2019) found that less than 40 percent of Medicare IPF patients access a mental health provider 30 days before and after an IPF stay.
Follow-up care post-IPF discharge occurs infrequently. Brown and Bell (2022), in their analysis of 2018 National Mental Health Services Survey data and IPF Quality Reporting program data, found only 28 percent of Medicare beneficiaries discharged from an IPF received follow-up care within seven days. Efforts to improve the timeliness of mental health care (Pearlmutter et al. 2017), focus on intervention strategies to “improve patient safety, quality of care, [and] well-being” (Lorine et al. 2015), and increasing access to affordable, preventative care can reduce reliance on ED visits for psychiatric care (Bruckner et al. 2019). Brown & Bell (2022) cite that improving follow-up care and discharge planning can increase continuity of care for Medicare beneficiaries and thus increase patient engagement following discharge from an IPF. Mutschler et al. (2019), in their systematic review, identified multiple studies that found that patients receive the necessary care while hospitalized, but are often left on their own to find post-discharge supports and services in their local community. They identified barriers to post-IPF discharge care, such as stigma and poverty, and actions that can be taken by IPFs to reduce the likelihood of ED visits and readmissions following IPF discharge such as aiding the transition to outpatient support, focusing on patient self-efficacy, and maximizing social and peer support.
There is limited literature on ED visits without resultant readmission following IPF discharge. However, researchers have examined hospital readmission rates following an IPF discharge. Because an ED visit is often the initial point of contact for a patient seeking care, and unplanned readmissions may result from an ED visit, the research on reducing hospital readmissions following IPF discharge can be informative. Callaly et al. (2010) found that IPF readmissions were 10 times less likely if post-discharge follow-up occurred within seven days of discharge as compared to eight days or more. If the initial ED visit can be avoided, IPF readmission rates may also go down.
Evidence suggests that discharge planning focused on continuity of care (e.g., care coordination) increases the likelihood of successful community reintegration for patients after an IPF discharge, resulting in reduced ED visits (Chung et al. 2017). Care coordination with outpatient services is an important predictor of acute hospital readmission, again often stemming from ED visits. IPFs can play a role in care coordination by arranging follow-up appointments for patients, ensuring medications are available at discharge, assisting patients with accessing medications from external providers, and engaging the patients’ social support system. Hamilton et al. (2015) found that patients who missed their first post-IPF discharge follow-up appointment had a 140 percent increased risk of readmission. Similarly, Abernathy et al. (2016) found that patients with poor post-discharge compliance were 1.7 times more likely to be readmitted following IPF discharge.
The rationale for this measure is to encourage IPFs to proactively focus on discharge planning and community reintegration, at the time of the patients IPF stay. The IPF ED Visit measure is intended to compliment the IPF Readmission measure and the FAPH measure. The IPF ED Visit measure gives IPF providers additional data points on how their patients fare after discharge. This measure will additionally complement existing measures by quantifying follow-up, ED visits, and IPF readmissions, which will assist IPFs in discharge planning, care coordination, and timely follow-up. The IPF ED Visit measure is intended to incentivize providers to conduct discharge planning in ways that ensure patients have an opportunity and ability to adhere to post-discharge instructions associated with medication use and follow-up with community-based providers.
Relevant literature:
- Abernathy, K., Zhang, J., Mauldin, P. et al. (2016). Acute care utilization in patients with concurrent mental health and complex chronic medical conditions. Journal of Primary Care & Community Health, 7(4), 226–233.
- Blair, R., Brown, J. D., Barry, X., & Schmitt, A. (2019). Transitions in Care and Service Use among Medicare Beneficiaries in Inpatient Psychiatric Facilities Issue Brief. Washington, DC, US Department of Health and Human Services, Office of the Assistant Secretary for Planning and Evaluation.
- Brown, J. D. & Bell, N. (2022). Factors Associated with the Receipt of Follow-Up Care Among Medicare Beneficiaries Discharged from Inpatient Psychiatric Facilities. The Journal of Behavioral Health Services & Research. https://doi.org/10.1007/s11414-022-09810-7
- Bruckner, T. A., Singh, P., Chakravarthy, B., Snowden, L., & Yoon, J. (2019). Psychiatric emergency department visits after regional expansion of community health centers. Psychiatric Services, 70(10), 901-906.
- Callaly, T., Hyland, M., Trauer, T., Dodd, S., & Berk, M. (2010). Readmission to an acute psychiatric unit within 28 days of discharge: Identifying those at risk. Australian Health Review, 34(3), 282–285
- Chung, D.T., Ryan, C.J., Hadzi-Pavlovic, D., Singh, S.P., Stanton, C., & Large, M.M. (2017). Suicide rates after discharge from psychiatric facilities: A systematic review and meta-analysis. JAMA Psychiatry, 74(7), 694–702. https://doi.org/10.1001/jamapsychiatry.2017.1044
- Durbin, J., Lin, E., Layne, C., et al. (2007). Is readmission a valid indicator of the quality of inpatient psychiatric care? Journal of Behavioral Health Services Research, 34, 137–150. doi:10.1007/s11414- 007-9055-5
- Feigenbaum, P., Neuwirth, E., Trowbridge, L., Teplitsky, S., Barnes, C. A., Fireman, E., ... & Bellows, J. (2012). Factors contributing to all-cause 30-day readmissions: a structured case series across 18 hospitals. Medical care, 599-605.
- Hamilton, J.E., Rhoades, H., Galvez, J. et al. (2015). Factors differentially associated with early readmission at a university teaching psychiatric hospital. Journal of Evaluation in Clinical Practice, 21(4), 572–578.
- Lorine, K., Goenjian, H., Kim, S., Steinberg, A. M., Schmidt, K., & Goenjian, A. K. (2015). Risk factors associated with psychiatric readmission. The Journal of Nervous and Mental disease, 203(6), 425- 430.
- Mutschler, C., S. Lichtenstein, S.A. Kidd, and L. Davidson. “Transition Experiences Following Psychiatric Hospitalization: A Systematic Review of the Literature.” Community Mental Health Journal, vol. 55, no. 8, 2019, pp. 1255–1274. doi.org/10.1007/s10597-019-00413-9
- Pearlmutter, M.D., Dwyer, K.H., Burke, L.G., Rathlev, N., & Maranda, L. Volturo, G. (2017). Analysis of emergency department length of stay for mental health patients at ten Massachusetts emergency departments. Annals of Emergency Medicine, 70(2),193-202.
- Venkatesh AK, Mei H, Shuling L, D'Onofrio G, Rothenberg C, Lin Z, Krumholz HM. Cross-sectional Analysis of Emergency Department and Acute Care Utilization Among Medicare Beneficiaries. Acad Emerg Med. 2020 Jul;27(7):570-579. doi: 10.1111/acem.13971. Epub 2020 May 20. PMID: 32302034.
Anticipated ImpactMeasure Impact on Care and Health Outcomes. Improvements to IPF care, follow-up, and reduced IPF readmission rates and ED utilization are imperative to improving patient health outcomes, including a reduction of suicidality (Mutschler at al. 2019). Moreover, the proposed measure could facilitate and improve continuity of care following an IPF discharge, including data to improve post-discharge follow-up to minimize ED uptake for psychiatric care. Additionally, the proposed measure may have positive implications on improving continuity to community resources following an IPF discharge. As seen across the literature, the 30 days following an IPF discharge are a key touchpoint ensuring successful facilitation to community reintegration.
Measure Impact on Healthcare Costs. Psychiatric-related care provided in the ED is costly. Using the Nationwide Emergency Department Sample (2017), Karaca & Moore (2020) found the average ED visit cost for psychiatric and substance use diagnoses (PSUD) was $520 per visit across 10.7 million PSUD ED visits. In 2017, PSUD visits cost more than $5.6 billion in ED service delivery, representing more than 7% of the annual ED service delivery costs (Karaca & Moore 2020). The costliest PSUDs are alcohol-related, anxiety-related, depressive diagnoses, suicidality, schizophrenia spectrum disorders and other psychotic disorders (Karaca & Moore 2020). Preventative strategies that focus on reducing risk factors associated with psychiatric care admissions (Lorine et al. 2015), and improving continuity of care upon discharge can reduce healthcare costs. Reducing ED visits for psychiatric care can also reduce hospital costs and improve healthcare outcomes (Oblath et al. 2022). Addressing gaps between an IPF discharge and an ED visit has the potential to reduce ED costs. Likewise, by adopting more preventative care strategies on the front end of an IPF stay, it could be anticipated there would be reduced costs associated with readmission to an IPF.
We developed a cell-based model to estimate the costs associated with ED visits within 30 days of IPF discharge (Exhibits 2 & 3 attached below). Estimates rely on data from two sources: empirical data from measure testing activities on the number of Medicare patients with ED visits within 30 days of IPF discharge (n = 37,054 of 178,143 IPF discharges; 20.8 percent) and empirical cost data from the Health Care Utilization Project. Exhibit 2 provides current cost estimates resulting from ED visits within 30 days of IPF discharge among Medicare patients. Current average hospital costs total $38 million and average hospitals charges total $278 million.
Relevant literature:
- Karaca, Z., & Moore, B. J. (2020). Costs of Emergency Department Visits for Mental and Substance Use Disorders in the United States, 2017. In Healthcare Cost and Utilization Project (HCUP) Statistical Briefs. Agency for Healthcare Research and Quality (US).
- Lorine, K., Goenjian, H., Kim, S., Steinberg, A. M., Schmidt, K., & Goenjian, A. K. (2015). Risk factors associated with psychiatric readmission. The Journal of Nervous and Mental disease, 203(6), 425- 430.
- Mutschler, C., S. Lichtenstein, S.A. Kidd, and L. Davidson. “Transition Experiences Following Psychiatric Hospitalization: A Systematic Review of the Literature.” Community Mental Health Journal, vol. 55, no. 8, 2019, pp. 1255–1274. doi.org/10.1007/s10597-019-00413-9
- Oblath, R., Herrera, C. N., Were, L. P. O., Syeda, H. S., Duncan, A., Ferguson, T., Kalesan, B., Perez, D. C., Taglieri, J., Borba, C. P. C., & Henderson, D. C. (2023). Long-Term Trends in Psychiatric Emergency Services Delivered by the Boston Emergency Services Team. Community mental health journal, 59(2), 370–380. https://doi.org/10.1007/s10597-022-01015-8
Exhibit 2. Average hospital cost and charges based on current IPF ED Visit measure rate among Medicare beneficiaries
Current population and estimates Total
Estimated number of Medicare beneficiaries discharged from IPFs in 2020a 178,143
Proportion with an ED visit within 30-days of discharge from IPFb 20.8%
Total patients with an ED visit within 30 days of IPF discharge 37,054
Average hospital cost per ED visit (Medicare), 2020c $1,031
Average hospital charges per ED visit (Medicare), 2020d $7,491
Total average hospital costs resulting from ED visits by patients discharged from IPFs (in $) $38,202,410
Total average hospital charges resulting from ED visits by patients discharged from IPFs (in $) $277,569,596
a Medicare beneficiaries discharged from IPFs between June 1, 2019 and July 31, 2021, excluding the first two quarters of 2020 due to COVID-19, that met denominator criteria for the measure totaled 282,060. We divided this number by 19 months to determine monthly discharges and then multiplied by 12 to generate an annual estimate of 178,143.
b 30-Day Risk Standardized All-Cause Emergency Department Visit Following an Inpatient Psychiatric Facility Discharge performance rate based on analysis of Medicare claims date for adult IPF discharged between June 1, 2019 and July 31, 20221 (n=282,060).
c HCUPNet data (2020), Treat-and-release ED visits, Average hospital cost per ED visit for cases with Medicare as expected payment source: $1,031. Costs reflect the actual expenses incurred in the production of hospital ser vices, such as wages, supplies, and utility costs. ‘Treat -and-release’ includes ED visits that did not result in admission to the same hospital and ED visits transferred to another hospital or ED.
d HCUPNet data (2020), Treat-and-release ED visits, Average hospital charges per Ed Visit for cases with Medicare as expected payment source: $7,491. Charges represent what the hospital billed for a case. ‘Treat-and-release’ includes ED visits that did not result in admission to the same hospital and ED visits transferred to another hospital or ED.
As seen in Exhibit 3, if the rate of ED visits following IPF discharge declined by 5 percentage points, the average hospital costs and average hospital charges would drop by about $9 million and $67 million, respectively.
Exhibit 3. Average hospital cost and charges with reduction in IPF ED Visit measure rate by 5 percentage points among Medicare beneficiaries
Current population and estimates Scenario: Reduce ED visits by 5 percentage points
Total Difference
Estimated number of Medicare beneficiaries discharged from IPFs in 2020a 178,143 -
Proportion with an ED visit within 30-days of discharge from IPFb 15.8% -5.0%
Total patients with an ED visit within 30 days of IPF discharge 28,147 -8,907
Average hospital cost per ED visit (Medicare), 2020c $1,031 -
Average hospital charges per ED visit (Medicare), 2020d $7,491 -
Total average hospital costs resulting from ED visits by patients
discharged from IPFs (in $) $29,019,138 -$9,183,272
Total average hospital charges resulting from ED visits by
patients discharged from IPFs (in $) $210,846,136 -$66,723,461
a Medicare beneficiaries discharged from IPFs between June 1, 2019 and July 31, 2021, excluding the first two quarters of 2020 due to COVID-19, that met denominator criteria for the measure totaled 282,060. We divided this number by 19 months to determine monthly discharges and then multiplied by 12 to generate an annual estimate of 178.143.
b 30-Day Risk Standardized All-Cause Emergency Department Visit Following an Inpatient Psychiatric Facility Discharge performance rate based on analysis of Medicare claims date for adult IPF discharged between June 1, 2019 and July 31, 20221 (n=282,060).
c HCUPNet data (2020), Treat-and-release ED visits, Average hospital cost per ED visit for cases with Medicare as expected payment source: $1,031. Costs reflect the actual expenses incurred in the production of hospital ser vices, such as wages, supplies, and utility costs. ‘Treat -and-release’ includes ED visits that did not result in admission to the same hospital and ED visits transferred to another hospital or ED.
d HCUPNet data (2020), Treat-and-release ED visits, Average hospital charges per Ed Visit for cases with Medicare as expected payment source: $7,491. Charges represent what the hospital billed for a case. ‘Treat-and-release’ includes ED visits that did not result in admission to the same hospital and ED visits transferred to another hospital or ED.
Health Care Quality LandscapeThe IPF Readmission and FAPH measures address IPF quality of care, yet they are not sufficient to address the need to reduce ED visits for acute behavioral health encounters following discharge from the IPF. The IPF Readmission is a facility-level measure that estimates unplanned, 30-day risk-standardized readmission rate for adult Medicare FFS patients discharged from an inpatient psychiatric facility with a principal discharge diagnosis of a psychiatric disorder, dementia, or Alzheimer's. FAPH assesses the percentage of IPF hospitalizations for mental health or substance use disorders that were followed by an outpatient mental health or substance user encounter within 7- and 30-days following discharge.
The IPF ED Visit measure assesses the proportion of patients ages 18 years and older with an ED visit, including observation stays, for any cause, within 30 days of discharge from an IPF, without subsequent admission. The IPF ED Visit measure is intended to compliment and harmonize with the measures mentioned above, to the extent possible (exclusions, age parameters, etc.). The IPF ED Visit measure complements the IPF Readmission by assessing the proportion of patients who may seek ED care without a readmission to an IPF.
The IPF ED Visit measure gives IPF providers additional data points on how their patients fare after discharge, as some patients may seek treatment in an ED instead of an IPF. This measure will additionally complement existing measures by quantifying follow-up, ED visits, and IPF readmissions, which will incentivize IPFs in high quality discharge planning, care coordination, and timely follow-up. The IPF ED Visit measure is intended to incentivize providers to conduct discharge planning in ways that ensure patients have an opportunity and ability to adhere to post-discharge instructions associated with medication use and follow-up with community-based providers.
Meaningfulness to Target PopulationIn the spring of 2023, we convened a behavioral health technical expert panel (TEP) (n = 7) that guided our work on this measure, which included three patient caregivers. TEP members were asked if they strongly agree, agree, disagree, or strongly disagree with the following statement, “The IPF ED Visit measure provides information that is easy to understand.” Two of the three patient caregivers that responded to the question agreed with the statement. One of these two patient caregivers agreed that providers can use facility-level scores on the IPF ED Visit measure to help make decisions about inpatient care and workflow. A 30-day public comment period was held for the IPF ED Visit measure. No other responses were received from patients or caregivers.
- Feasibility Assessment
Medicare claims data are used for this measure. Claims-based measures are feasible to implement comprehensively and efficiently for large populations of patients.
Data for claims-based measures are readily available, making the measure feasible to implement and report. Our testing work proved that it is feasible to calculate the measure, as specified.
Feasibility Informed Final MeasureNot applicable.
Proprietary InformationNot a proprietary measure and no proprietary componentsFees, Licensing, or Other RequirementsThere are no fees, licensing, or other requirements to use the measure, as specified.
- Data Used for Testing
The IPF ED Visit measure was tested using Medicare beneficiary and coverage files, Medicare FFS Part A and B records, data from the American Community Survey, Dartmouth Atlas of Health Care (which includes data elements from the Centers for Medicare & Medicaid Services (CMS), United States Census Bureau, the American Hospital Association, the American Medical Association, and the National Center for Health Statistics), and Health Resources and Services Administration HPSA files. Both administrative and claims data were used for testing.
For measure calculation, the following Medicare files are required:
- Medicare beneficiary and coverage files. Provides information on patient demographic, enrollment, and vital status to identify the measure population and certain risk factors.
- Medicare fee-for-service (FFS) Part A records. Contains final action claims submitted by acute care and critical access hospitals, IPFs, home health agencies, and skilled nursing facilities to identify the measure population, readmissions, and certain risk factors.
- Medicare FFS Part B records. Contains final action claims submitted by physicians, physician assistants, clinical social workers, nurse practitioners, and other outpatient providers to identify certain risk factors. For this measure, claims for services such as laboratory tests, medical supplies, or other ambulatory services were not used. This ensures that diagnoses result from an encounter with a provider trained to establish diagnoses and not from a claim for a diagnostic test.
Index admissions and ED visits are identified in the Medicare Part A data. Comorbid conditions for risk adjustment are identified in the Medicare Part A and Part B data in the 12 months prior to and including the index admission. Demographic and FFS enrollment information is identified in the Medicare beneficiary and coverage files. Adult IPF admission with admission and discharges between June 1, 2019, and July 31, 2021, were used to calculate the IPF ED Visit measure performance rate. We excluded cases during the first two quarters of 2020 due to the COVID-19 public health emergency.
Differences in DataWe calculated performance rates and conducted reliability and validity analyses for the 1,483 IPFs with 25 or more discharges during the measurement period (June 1, 2019, through July 31, 2021). We aligned with previous testing of the existing and conceptually related IPF Readmission measure to identify potential social determinants of health (SDOH) that may put patients at a higher risk of ED visits and tested their effect on the measured outcome in the risk-adjustment model. We included two patient-level variables (Dual Medicare-Medicaid status and race/ethnicity) and eighteen neighborhood-level SDOH variables in the analyses. Please refer to the Data sources for SDOH tab in Risk-model specifications.xlsx file, attached below.
Using the datasets listed above (i.e., Medicare beneficiary and coverage files, Medicare FFS Part A and B records) as well as the datasets provided in Data sources for comorbidities tab in Risk-model specifications.xlsx file, we constructed a set of variables capturing distinct aspects of patient- and area-level SDOH characteristics to use in the risk-adjustment model.
The only patient-level SDOH variable considered for inclusion in the final risk-adjustment model was Medicare-Medicaid dual enrollment, as an indicator of poverty. While we assessed the effect of race on the probability of the ED visit, we did not consider risk-adjusting for race or ethnicity, per the recommendation of the National Quality Forum (NQF) Risk-Adjustment Expert Panel and Disparities Standing Committee (National Quality Forum, 2014).
In the absence of patient-level data on beneficiary socioeconomic status (SES) characteristics, area-based variables offer the potential to capture characteristics of patient environments and exposure to social and economic conditions (Krieger, 2003). Studies that used both levels of factors showed similar results and found that area- and individual- factors independently and jointly affect some health outcomes (Roux, 2001). The area-level variables capture demographic and SES characteristics of patient neighborhoods, the local healthcare system, adjusted mortality rates of Medicare beneficiaries, and urbanization level. We were not able to create variables for patient housing stability, marital status, or availability of social support because these data are not collected for all Medicare beneficiaries.
We merged area-level SDOH data to patient-level data using the nine-digit zip-code variable. Recent studies show that block group-, census tract-, and zip-code level indicators detect expected gradients of the SES in health outcomes similarly (Berkowitz et al., 2014). In the patient-level data file use to test the IPF ED Visit measure, 133 IPF index admissions did not have any zip code; 98.4 percent (n = 277,534) had a nine-digit zip code, and 1.6 percent (n = 4,393) had a five-digit zip code only. For the beneficiaries with unavailable nine-digit zip codes, we computed the area-level characteristics at the five-digit zip code.
Relevant literature:
- Berkowitz, S. A., Traore, C. Y., Singer, D. E., & Atlas, S. J. (2014). Evaluating Area-Based Socioeconomic Status Indicators for Monitoring Disparities within Health Care Systems: Results from a Primary Care Network. Health Services Research, 2, 398–417. https://doi.org/10.1111/1475-6773.12229
- Krieger N, Chen JT, Waterman PD, Rehkopf DH, Subramanian SV. Painting a truer picture of US socioeconomic and racial/ethnic health inequalities: the Public Health Disparities Geocoding Project. Am J Public Health. 2005;95(2):312-323
- National Quality Forum (2014). Risk Adjustment for Socioeconomic Status or Other Sociodemographic Factors. Retrieved from http://www.qualityforum.org/Publications/2014/08/RA_SES_Technical_Report.aspx
- Roux, A. V. D. (2001). Investigating neighborhood and area effects on health. American Journal of Public Health, 91(11), 1783–1789. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1446876/pdf/0911783.pdf
Characteristics of Measured EntitiesThe measure was tested at the hospital level (IPF). Testing used claims data for patients discharged from any of the 1,617 IPFs participating in the Inpatient Psychiatric Facility Quality Reporting Program. Some analyses were limited to IPFs with 25 or more discharges during the measurement period (n = 1,483).
Characteristics of Units of the Eligible PopulationWe tested the measure using data from 194,531 patients across 282,060 index admissions across 1,617 IPFs. The patient characteristics are described in Exhibit 4.
Exhibit 4. Patient characteristics
Patient characteristics n %
Total patients: 194,531 100.0
Age (mean = 58.6, SD = 17.9)
18–64 110,617 56.9
65+ 83,914 43.1
18–34 23,258 12.0
35–64 87,359 44.9
65+ 83,914 43.1
Race/ethnicity
Black 29,024 14.9
White 149,989 77.1
Hispanic 5,933 3.1
Other 5,569 2.9
Unknown/missing 4,016 2.1
Sex
Female 99,951 51.4
Male 94,580 48.6
Dual Medicare-Medicaid status
Yes 104,188 53.6
No 90,343 46.4
Data source: Medicare claims date for adult IPF admissions with admission and discharge between June 1, 2019, and July 31, 2021, excluding the first two quarters of 2020 due to the COVID-19 public health emergency.
- Level(s) of Reliability Testing ConductedMethod(s) of Reliability Testing
The IPF ED Visit measure rates are computed using a hierarchical logistic regression. This approach adjusts the IPF ED Visit rate results for smaller facilities by smoothing their rates closer to the mean. Thus, signal-to-noise reliability analysis is less suitable for this measure because the hierarchical logistic regression adjustment removes IPF-level variation in the risk-standardized rates and could make the results more reliable. We assessed measure score reliability using a bootstrapped test-retest approach at the facility-level. A test-retest approach examines the agreement between repeated measure scores at the same IPF during the same period taking many different random sample from the test population. A total of 1,483 IPFs with at least 25 discharges during the measurement period were included in the reliability analysis. We assessed reliability using the intra-class correlation coefficient (ICC), a reliability coefficient that reflects both correlation and agreement between measurements (Ranganathan et al., 2017).
We used a bootstrap approach to draw 1,000 pairs of samples from the original measure cohort with replacement (stratified sampling by IPF), maintaining the sample size within each IPF. We estimated the ICC in the bootstrap sampling for each pair of the bootstrap samples. With the 1,000 ICC estimates, we determined the distribution of estimated ICCs and calculated the mean, median, and percentile distribution of the ICC. The randomly sampled sets of admissions from a given hospital are assumed to reflect an independent set of re-measurement of readmission rates for the hospital. Adequate reliability is assumed if the risk-standardized measure rates calculated from the random datasets for the same IPF are similar. Higher ICC values indicate stronger agreement between measure scores in the samples and better measure reliability. The bootstrapping approach has advantages over the split-half method because it avoids biased sampling, maintains the original sample size, and allows estimation of ICC confidence.
Relevant literature:
- Ranganathan P, Pramesh CS, Aggarwal R. Common pitfalls in statistical analysis: Measures of agreement. Perspect Clin Res. 2017 Oct-Dec;8(4):187-191.
Reliability Testing ResultsThe mean ICC obtained from the bootstrapping approach is 0.690 (95% confidence interval [CI] 0.550, 0.831) and the median ICC is 0.698 (ICC interquartile range: 0.690 – 0.705). Table 2 (below) provides detailed reliability testing results across the 1,483 IPFs with 25 or more discharges during the measurement period.
Accountable Entity-Level Reliability Testing ResultsAccountable Entity-Level Reliability Testing Results Overall Minimum Decile_1 Decile_2 Decile_3 Decile_4 Decile_5 Decile_6 Decile_7 Decile_8 Decile_9 Decile_10 Maximum Reliability 0.690 0.00 0.682550979 0.687673994 0.691466134 0.694924154 0.697561482 0.700271986 0.70340118 0.707332578 0.712465142 0.756010232 0.756 Mean Performance Score 1,483 N of Entities Interpretation of Reliability ResultsThe ICC obtained from the bootstrapping approach, comparing 1,000 pairs of samples of the original measurement cohort, which were sampled with replacement yielding an identical sample size as the original measurement cohort, is 0.690 (ICC range: 0.683 – 0.756). These results suggest the high reliability of the measure.
- Level(s) of Validity Testing ConductedType of accountable entity-level validity testing conductedMethod(s) of Validity Testing
Empirical validity testing was conducted for this measure. We assessed the relationship between the IPF ED Visit measure rate and the IPF Readmission measure rate. We used Spearman rank-order correlation to assess cross-measure validity to account for a possible non-linear relationship between the measure scores. We hypothesized IPFs with a high proportion of patients readmitted following an IPF hospitalization would also have a high proportion of patients with ED visits, without readmission, following an IPF hospitalization.
We also conducted hypothesis-driven validity testing to determine if performance rates varied among subgroups of patients in ways that were consistent with the empirical literature. Our literature-based hypotheses are as follows:
- Sex. We hypothesized that male patients will have higher ED visit rates than female patients.
- Race/ethnicity. We hypothesized that white, non-Hispanic patients will have lower ED visit rates than non-white, Hispanic patients.
- Dual eligibility status. We hypothesized that patients who are dually eligible for Medicare and Medicaid will have higher ED visit rates than patients who are not dually eligible.
- IPF length of stay (LOS). We hypothesized that patients with a longer LOS will have higher ED visit rates than patients with a shorter LOS.
Independent samples t-tests were used to compare mean group differences in ED visit scores based on sex, race, Medicare-Medicaid dual enrollment status, and IPF length of stay (LOS). With large sample sizes, small differences that are statistically significant may not always be practical or clinically meaningful. Therefore, we computed Cohen's d effect size (the difference in mean scores divided by the pooled standard deviation). Following Cohen’s (1988) definitions, we defined effect size values as small (0.2), medium (0.5), or large (0.8). In our hypothesis-driven validity testing, we used unadjusted performance rates, as risk-standardization reduces the variability in the distribution of measure scores and could obscure differences between subgroups of patients.
Relevant literature:
- Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Earlbaum Associates.
- Larkin, G. L., Claassen, C. A., Emond, J. A., Pelletier, A. J., & Camargo, C. A. (2005). Trends in US emergency department visits for mental health conditions, 1992 to 2001. Psychiatric services, 56(6), 671-677.
- Peters ZJ, Santo L, Davis D, DeFrances CJ. Emergency department visits related to mental health disorders among adults, by race and Hispanic ethnicity: United States, 2018–2020. National Health Statistics Reports; no 181. Hyattsville, MD: National Center for Health Statistics. 2023.
- Strashny A, Cairns C, Ashman JJ. Emergency department visits with suicidal ideation: United States, 2016–2020. NCHS Data Brief, no 463. Hyattsville, MD: National Center for Health Statistics. 2023. DOI: https://dx.doi.org/10.15620/cdc:125704.
- Stroever S, Brett C, Michael K, Petrini J. Emergency department utilization for mental health conditions before and after the COVID-19 outbreak. Am J Emerg Med. 2021 Sep;47:164-168.
Validity Testing ResultsFor the characteristics listed above in our validity testing, the statistical results are as follows:
- Sex. There is a 2.6 percent difference in the mean IPF ED Visit rate between male and female beneficiaries (Cohen’s D = 0.06).
- Race. The difference in the IPF ED Visit rates between Black and White beneficiaries is 3.7 percentage points (Cohen’s D = 0.09).
- Dual eligibility status. There is a difference of 5.5 percentage points between the mean IPF ED Visit rate among beneficiaries with dual eligibility for Medicare-Medicaid and Medicare-only (non-dual) status (Cohen’s D = 0.16).
- IPF LOS. There is an 8.9 percentage point difference in the mean IPF ED Visit rate for beneficiaries with long LOS (4th quartile) versus short LOS (1st quartile) (Cohen’s D = 0.22).
Interpretation of Validity ResultsThere is a moderate, yet meaningful, positive relationship between the facility rates on the IPF ED Visit measure and the IPF Readmission measure (Spearman ranked correlation ρ = 0.42). A positive relationship between the measure rates indicates that an increase in ED visit rates is associated with the increase in the unplanned 30-day IPF readmission. We observed small effect sizes for the differences in the observed ED visit measure rates by patient subgroups:
- Sex. There is a 2.6 percent difference in the mean IPF ED Visit rate between male and female beneficiaries (Cohen’s D = 0.06). There is a 51.7 percent chance that a male beneficiary will have higher probability of an ED visit compared to a female beneficiary.
- Race. The difference in the IPF ED Visit rates between Black and White beneficiaries is 3.7 percentage points (Cohen’s D = 0.09). There is a 52.5 percent chance that a Black patient will have higher probability of an ED visit relative to an average White patient.
- Dual eligibility status. There is a difference of 5.5 percentage points between the mean IPF ED Visit rate among beneficiaries with dual eligibility for Medicare-Medicaid and Medicare-only (non-dual) status (Cohen’s D = 0.16). There is a 54.5 percent chance that a patient with dual eligibility status will have higher probability of an ED visit as compared to a non-dual patient.
- IPF LOS. There is an 8.9 percentage point difference in the mean IPF ED Visit rate for beneficiaries with long LOS (4th quartile) versus short LOS (1st quartile) (Cohen’s D = 0.22). There is a 56.2 percent chance that a patient with longer LOS will have higher probability of an ED visit compared to a beneficiary with a short LOS.
Observed differences were small, as described above. However, they were in the expected directions and further support the ability of the measure to discriminate between the groups of beneficiaries known to have different ED visit measure rates.
- Methods used to address risk factorsConceptual Model Rationale
Patient health characteristics (i.e., principal diagnosis, comorbidities, age, gender, prior history of AMA, violence, and suicidality), patient- and area-level social determinants of health (SDOH) (i.e. disability status, dual eligibility status, race, and ethnicity, socioeconomic status, urbanization/rurality, housing stability, social support, adjusted mortality rate), and access to care/characteristics of local healthcare systems impact the structure (i.e. funding, staffing, clinical expertise, services, location) of an IPF, the quality of care received at an IPF, and therefore impact the risk of an ED visit 30 days following discharge from an IPF.
To identify candidate clinical and SDOH variables for risk adjustment we reviewed existing literature on risk factors for ED visit following psychiatric discharges, reviewed risk variables used in other ED visit measures, and performed bootstrap selection for candidate risk variables. For performance assessment, we only controlled for patient factors that were present prior to the start of care. The risk factors for health status at IPF admission included in the risk model are principal diagnosis of the IPF index admission, comorbidities, demographics of age and gender, and prior history of being discharged AMA, aggressive behavior, or suicidality. We used broad AHRQ CCS categories for the principal diagnoses in risk adjustment. However, while we collapsed unique principal discharge diagnosis ICD-10-CM codes into broader categories, we carefully reviewed crosswalks to ensure optimal capture of differences in readmission rates. This resulted in the development of subcategories for schizophrenia/psychosis and bipolar/depressive disorders and the further collapsing of developmental/childhood disorders and other psychiatric disorders.
For comorbidities, we used the CMS complication or comorbidity (CMS CC) categories to form clusters on comorbidities, but reviewed crosswalks to optimize the predictive performance of each cluster in capturing ICD-10-CM codes with similar associations with ED visits. This resulted in modification of the ICD-10-CM to CC crosswalk, mostly in following assignments in the comparable CCS category or collapsing certain CC categories based on similar ED visit rates. We obtained information on comorbidities from the secondary diagnosis of the index admission, after careful review and exclusion of conditions that may represent hospital-acquired complications rather than preexisting comorbidities, principal or secondary diagnoses of hospital admissions during the 12-month look-back period, or presence of at least two outpatient encounter claims with principal or secondary diagnoses of the same CC.
We also identified other variables in the literature that are relevant for the inpatient psychiatric population. These included history of discharge AMA, suicidality, electroconvulsive therapy/transcranial magnetic stimulation (ECT/TMS), or aggression; admission source (as proxy for involuntary admission); and count of psychiatric comorbidities. The key SDOH constructs that may affect the risk of ED visits for psychiatric patients include risk factors such as income/poverty, disability, race/ethnicity and language barriers, access to care, education, housing stability, and social support. As shown in the conceptual model, the impact of SDOH factors on ED visits can be direct or indirect through their effect on health status, the facility selected to obtain care, and the quality of the specific treatments and care received. Additionally, health status can influence SDOH factors. The mechanisms for the effect of sociodemographic factors on health are complex, interrelated, and may result from a lifelong, cumulative effect of social status on health. External factors related to local health-care markets and IPF structure can also affect patient’s access to services prior to admission. Quality of IPF care can directly affect ED visits after an IPF discharge. Risk models typically do not control for differences in such external factors.
Risk Factor Characteristics Across Measured EntitiesWe observed substantial variation in the distribution of most of the risk variables identified in the conceptual model. Please refer to the Descriptive Statistics tab in the Risk-model specifications.xlsx file, attached below.
Risk Adjustment Modeling and/or Stratification ResultsWe derived a parsimonious risk adjustment model by using logistic regression with a stepwise backward elimination process, which was repeated in 1,000 bootstrap samples from the entire population via random selection with replacement. This approach allows the use of the entire dataset for model development and a nearly unbiased estimate of predictive accuracy with relatively low variance compared with other validation approaches, such as data splitting and cross-validation. We retained candidate variables demonstrating a positive association with ED visits at p-value <0.15 in at least 70 percent of samples. The p-value cut-off of 0.15 was chosen to approximately mimic variable selection based on the Akaike Information Criterion (AIC). To select a candidate risk factor based on AIC, its chi-squared (χ2) value has to exceed twice its degrees of freedom (df). When considering a predictor with 1 df, such as gender or diagnosis code, this implies χ2 >2 with p < 0.157.
The final risk model includes 49 risk-factors (excluding individual categories for categorical variables) capturing patients demographics, principal discharge diagnosis for the index admission, psychiatric and non-psychiatric comorbidities, and risk-factors identified during literature review (that is, suicide attempt/self-harm, aggression, and discharge AMA). Four potential comorbidity risk factors were dropped from the final risk model. We assessed model discrimination using the C-statistic, which reflects how accurately the model distinguishes between an index admission that does or does not have an ED visit. The C-statistic for the final risk-adjustment model was 0.670. A C-statistic of 0.5 represents random prediction and a C-statistic of 1.0 represents perfect prediction.
The variables in the final risk-adjustment model are presented in Risk-factor coeff. all risk-fct tab in the Risk-model specifications.xlsx file. The Candidate risk factor freq tab in the Risk-model specifications.xlsx file lists the frequencies and ED visit rates of all candidate clinical risk variables and details the output of the selection process, including the percent of bootstrap samples in which a variable had a p-value of <0.15. As the tab shows, most of the clinical risk factors were retained during the bootstrap selection process. The variables that did not demonstrate a positive association with ED visits at p-value <0.15 in at least 70 percent of samples were diabetes acute complications, diabetes chronic complications, hematological disorder, and coagulation defects.
In our testing, most of the SDOH variables were not retained in the final model during bootstrap selection process. Therefore, we added all candidate SDOH variables to the risk-adjustment model after selecting clinical risk factors to compare model discrimination for the model with clinical factors only and the model with clinical and SDOH factors. We assessed the impact on the model performance compared to the clinical risk factor only model in terms of predictive ability, c-statistic, distribution of residuals, model chi square, and distributions of RSRRs. Considering the contribution of the SDOH variables on risk model performance, we evaluated the SDOH variables based on their feasibility for use in a national CMS measure.
Calibration and DiscriminationWe assessed model calibration (i.e., whether a model accurately predicts probability of an ED visit) via a Hosmer-Lemeshow test (Exhibit 13), and the risk-decile calibration plot (Exhibit 14). Hosmer-Lemeshow test divides the patients into deciles (i.e., ten groups with equal number of patients) based on the expected risk for an ED visit, from lowest to highest risk. The range of expected risks of having an ED visit within each decile is determined by the patients in that decile. The difference between the observed and expected ED visit for each decile is summarized by the Pearson chi-square statistic. The statistics are then summed over the ten deciles and are compared to the chi-square distribution. In decile assessment, we should see similar numbers of observations in each decile group and increasing observed rates when we move from low to high deciles.
The risk-decile calibration plot with observed outcomes versus expected probabilities of readmission was computed to localize possible deviations across risk strata. In the risk-decile calibration plot (Exhibit 14), the diagonal line is the line of perfect calibration. In a well-calibrated model, all markers representing deciles should be close to the diagonal line. In this graph, the markers appear close to the diagonal line, which indicates a close agreement between the observed and expected probabilities of the ED visit.
We assessed model discrimination using the C-statistic, which reflects how accurately the model distinguishes between an index admission that does or does not have an ED visit. The C-statistic for the final risk-adjustment model was 0.667. A C-statistic of 0.5 represents random prediction, and a C-statistic of 1.0 represents perfect prediction.
Risk adjustment model performance parameters showed excellent calibration with no indication of over-fitting. The mean observed ED visit rates range from 45.1 percent in the highest decile to 9.7 percent in the lowest decile, an absolute difference of 35.3 percentage points, suggesting good discrimination. The ratio of observed to predicted IPF readmission rates is close to 1.0 for each decile, suggesting adequate calibration of the model. The Hosmer-Lemeshow statistic was 131.29 (df = 8; p < 0.001). Given the sensitivity of the Hosmer-Lemeshow statistic to sample size, calibration was reassessed using 20 random samples of 5,000 patients taken from the sample. A total of 18 of the 20 randomly selected samples of 5,000 patients showed non-significant H-L statistics, supporting the evidence that the model is correctly specified and fits the data well.
The C-statistic of 0.667 suggests moderate predictive discrimination, expressed as the model’s ability to distinguish between index admissions that are and are not followed by a readmission. Statistical findings of excellent calibration are confirmed when comparing observed to predicted probabilities by risk deciles (Exhibit 14). The results are in-line with the NQF-endorsed measures capturing post-discharge care services (that is, ED use and hospital readmissions) developed for other settings, such as Short-Stay Residents who have had an Outpatient Emergency Department Visit (0.610); Short-Stay Residents who were Re-hospitalized after a Nursing Home Admission (0.614); Hospital 30-Day Heart Failure Readmission measure (0.601); Hospital 30-Day Pneumonia Readmission Measure (0.630); Hospital 30-Day Acute Myocardial Infarction Readmission Measure (0.630); and Hospital-Wide Readmission Measure (0.64 to 0.71) (Centers for Medicare and Medicaid Services. (2020). Nursing Home Compare Quality Measure Technical Specifications, Appendices. Abt Associates. https://www.cms.gov/Medicare/Provider-Enrollment-and-Certification/CertificationandComplianc/Downloads/APPENDIX-New-Claims-based-Measures-Technical-Specifications-January-2020.pdf).
Interpretation of Risk Factor FindingsSee above
Final Approach to Address Risk FactorsRisk adjustment approachOnRisk adjustment approachOffSpecify number of risk factorsWe selected a total of 53 candidate risk factors based on the conceptual risk model (excluding the SDOH). Of these 53 risk factors, 49 appeared in the final risk adjustment model. Please refer to the Risk-model specifications.xlsx file, attached below.
Conceptual model for risk adjustmentOffConceptual model for risk adjustmentOn
- Contributions Towards Advancing Health Equity
As previously indicated, we added each SDOH variable on its own to the risk model adjusted by the clinical risk factors, and nearly all SDOH variables had much weaker associations with the outcome (odds ratios closer to 1.0). These variables include dual eligibility status, percentage of patients who do not speak English or speak limited English, rural-urban commuting area (RUCA), which is used as a proxy for the geographical location, and Agency for Healthcare Research and Quality (AHRQ) SES Index.
In the univariate analyses, 14 out of 17 SDOH risk factors had a statistically significant association with the outcome. After controlling for the clinical risk factors, only 11 SDOH risk-factors remained statistically significant (p < = 0.05). At the patient-level, non-dually eligible patients have lower odds of having an ED visit as compared to the patients with dual eligibility status (0.901[95 percent CI: 0.882-0.921]); Black (non-Hispanic) patients had higher odds of having an ED visit (1.140[95 percent CI: 1.107-1.175]) compared to the rest of the patients. At the area-level, neighborhoods where more residents do not speak English or speak limited English, neighborhoods with higher percentage of Black residents, and more urban areas had higher ED visit rates. When controlling for the clinical factors, dual eligibility, race, neighborhoods with higher percentages of Black residents and residents who either do not speak English or speak limited English, more urban areas, and areas with higher patient-to-hospital bed ratios (at the area level) had a statistically significant association with higher ED visit rates. This measure may further advance health equity by increasing data points that further explain disparities in behavioral health care for individuals who are dually eligible, non-white, individuals with limited English language proficiency, and individuals who live in more urban communities.
- Current or planned use(s)Other planned or current usePay for reportingActions of Measured Entities to Improve Performance
The measure is currently not in use. Once implemented, IPFs can use measure results to better understand how patients fair in their community transition and, if warranted, develop strategies for improving post-discharge continuity of care with the goal of averting ED visits. Mathematica, on behalf of CMS, conducted a 30-day public comment period on the IPF ED Visit measure. One health insurer and one state health department provided comments. Both commenters cited the importance of reducing avoidable ED visits following IPF discharge. Both expressed concern regarding the IPF’s ability to impact what happens post-discharge. Recommendations included removal of the “all-cause” component of the measure specification and inclusion of patients discharged AMA. One commenter referenced existing post-discharge measures and questioned if the IPF ED Visit measure was necessary. The same commenter suggested resources be spent on topics such as collaboration and shared decision-making.
- Data DictionaryLogic ModelConceptual ModelRisk Adjustment Modeling and/or Stratification SpecificationsCalibration and Discrimination Testing Results
- Most Recent Endorsement ActivityCost and Efficiency Fall 2023Endorsement StatusE&M Committee Rationale/Justification
Due to No Consensus
- Do you have a secondary measure developer point of contact?OnMeasure Developer Secondary Point Of Contact
Laura McDermott
Mathematica
600 Alexander Park
Suit 100
Princeton, NJ 08540
United StatesMeasure Developer Secondary Point Of Contact EmailMeasure Developer Secondary Point Of Contact Phone NumberThe measure developer is NOT the same as measure stewardOnSteward Organization EmailSteward Phone NumberSteward AddressWilfred Agbenyikey
Centers for Medicare and Medicaid Services
7500 Security Boulevard
Woodlawn, MD 21244
United StatesSteward Organization URLSteward Organization CopyrightN/A
- Detailed Measure SpecificationsYesLogic ModelOnImpact and GapYesFeasibility assessment methodology and resultsYesMeasured/accountable entity (reliability and/or validity) methodology and results (if available)Address health equityNoMeasure’s use or intended useYesRisk-adjustment or stratificationYes, risk-adjusted only508 ComplianceOnIf no, attest that all information will be provided in other fields in the submission.Off
-
Measure 4190 Should Not Be Endorsed
Name or OrganizationPresident and CEO, Center for Healthcare Quality and Payment Reform
-
CBE# 4190 Staff Assessment
Importance
ImportanceStrengths:
- Submission includes references to published literature (N=6) that demonstrate the measure focus is a material outcome (high risk of occurance, high cost, association with significant behavior health conditions).
- Performance to the benchmark would reduce population rates almost in half (20.6% to 12.3%).
- The developer convened a 7-person technical expert panel, inclusive of 3 patient caregivers.
Limitations:
- Submission references the TEP were asked if the measure was easy to understand. Two of the three patient caregivers that responded to the question agreed the measure provides information that is easy to undersatnd. One of these two patient caregivers agreed that providers can use facility-level scores on the IPF ED Visit measure to help make decisions about inpatient care and workflow. The developer did not note why the third patient did not agree, nor was this question geared towards meaningfulness ot the target population.
- Submission does not identify any potential person or entity harms
Rationale:
- Strong conceptual and literature-supported rationale; submission would be strenthened with more qualitative input from entities and the target populaiton (patients, caregivers).
Feasibility Acceptance
Feasibility AcceptanceStrengths:
- For the numerator and denominator, all required data are from enrollment and administrative (claims) data.
- For the risk adjustment model, social risk factors require a linkage to national surveys that are widely available.
Limitations:
None
Rationale:
- The measure uses readily available adminsitrative and survey data.
Scientific Acceptability
Scientific Acceptability ReliabilityStrengths:
- Measure is well defined and specified.
- An average entity level signal-to-noise reliability was estimated with an ICC using a bootstrap method with a 2019-2021 dataset of 1,483 entities that met the criteria of at least 25 discharges. The median ICC from the bootstrap is 0.698, indicating that more than 50% of the enitites have a reliability >0.6.
Limitations:
The reported deciles seem to be deciles of the results of the bootstrap iterations, not entity-level reliabilities. Without estimates of reliability by entity it is only possible to conclude that less than 50% of the enitites have a reliability <0.6, although that number may be much lower.
Rationale:
Nearly 50% of the entities could have a reliability <0.6. Applying a method such as the Spearman-Brown formula (based on denominator size) would provide estimates of reliability by denominator (population) decile. In addition, if to support any low volume providers, the developer may consider some possible mitigation strategies to improve these estimates, such as:
- Empirical approaches outlined in the report, MAP 2019 Recommendations from the Rural Health Technical Expert Panel Final Report, https://www.qualityforum.org/WorkArea/linkit.aspx?LinkIdentifier=id&ItemID=89673
- Consider a higher minimum case volume.
- Extend the time frame.
- Focus on applying mitigation at the lower volume providers.
Scientific Acceptability ValidityStrengths:
- Submission includes references to published literature (N=12) demonstrating an association between the measure focus and during-discharge planning and post-discharge timely access to care
- For risk adjustment, the submission uses a robust approach to risk factor variable selection. Discrimination statistic (C-statistic) of 0.67 is in line with models for similar measures. Submission uses a robust method for assessing calibration, including random sampling to account for large sample sizes that generally limit the usefulness of the Hosmer-Lemeshow test.
Limitations:
- For face validity testing, which is acceptbale for new measures, the submission references a TEP (N=7) but does not report results.
- The developer aslo peformed empirical validity testing, in which performance rates are reported by sex, race, dual eligibility status, and IPF LOS. The effects were modest but in the anticipated direction, although no rationale was provided to support the hypothesis and rationale to explain why the modest results. The devleoper also reports results on an association between the measure of interest and IPF 30-day readmission rate. The effects (spearman rank correlation p=0.42) are in the antcipated direction, but again, no rationale provided.
- For risk adjustment, for a few persons (<2%) the data needed to link area-level risk factors (9 digit zip code) is missing.
- Risk adjustment: Some social risk factors are at the area-level rather than the person-level due to the lack of available data
Rationale:
- The committee may consider asking the developer for a rationale for the antcipated associations that explains the reasons the direction and magnitude of the association, and an explanation for how the known interventions could plausibly account for the between entity variation.
Equity
EquityStrengths:
- Developer evaluated 17 social determinants of health for disparities in the 30-day all-cause ED visit following IPF discharge in both unadjusted and adjusted models. After controlling for clinical risk factors, it found that patients with dual eligibility and Black non-Hispanic patients had significantly higher odds of an ED visit; at the area level, patients living in neighborhoods with higher % of residents who do not speak English, higher % Black residents, more urban areas, and with higher patient-to-hospital bed ratios also had higher odds of an ED visit
Limitations:
None
Rationale:
- The developer analyzed performance data to evaluate disparities at the individual and area levels. At the individual-level it found significantly higher odds of a 30-day all-cause ED visit after IPF discharge for NH Black patients relative to White, and dual-eligible (DE) patients relative to non-DE, controlling for clinical risk factors. At the area-level it found higher odds of an ED visit for patients living in neighborhoods with higher % of residents who do not speak English, higher % Black residents, more urban areas, and with higher patient-to-hospital bed ratios, controlling for clinical risk factors.
Use and Usability
Use and UsabilityStrengths:
- Developer indicates that the measure is planned for use in public reporting and internal/external QI.
- Developer conducted a 30-day public comment period on the measure and received responses from a health insurer and a state health department, and both stressed the importance of reducing avoidable readmissions.
- Developer suggests that IPFs focus on implementing strategies for improving post-discharge continuity of care.
Limitations:
- Developer indicates that the measure is planned for use in public reporting and internal/external QI but does not provide any other information such as program name, purpose, geographic coverage, level of analysis, etc.
- Comments received from the insurer and health department expressed reservations regarding the ability of IPFs to influence post-discharge events; recommended changes to the specifications included removing the all-cause reason and including patients discharged AMA - developers do not discuss any changes that might have been made to the specifications.
Rationale:
- The measure is planned for use in public reporting and internal and external QI initiatives, but no details are provided such as program name, purpose, geographic coverage, etc. The committee may consider asking the developer this information.
- A 30-day comment period yielded comments supporting the importance of reducing avoidable ED visits after discharge from IPF, but also expressed reservations regarding the ability of IPFs to influence the measure and suggested changes to the specifications, such as dropping the all-cause component. Developer suggests that IPFs can improve performance by supporting post-discharge continuity of care (no details provided).
Summary
N/A
-
Overall summary
Importance
ImportanceEvidence base is weak with the majority of studies focused on readmissions (not a criteria for this measure) and also not focused on psychiatric conditiolns. The systematic review which underpins the rationale focuses on three outcomes ie readmissions/rehospitalizations/suicide NONE of which are relevant to this measure. The exclusion of hospitalized patients are excluding the very patients the IPF serves well in nthat individuals would be responding to DC instructions. All cause ED admissions will be driven by the medical management of these individuals which is not a focus of the measure.
Feasibility Acceptance
Feasibility AcceptanceClaims based metric
Scientific Acceptability
Scientific Acceptability ReliabilityBoot strapping model: The randomly sampled sets of admissions are assumed to reflect an independent set of remeasurement of readmission rates for the hospital...fine but the measure isn't about readmissions. Any testing on the non admitted population would likely yield differing characteristics of the population
Scientific Acceptability ValidityNo results reported from TEP. Only 1 of 3 patient caregivers felt that metric is useful for IPF QI (unclear if pts know what that means). Modest at best results for empirical validity testing with no explanation. Again testing done on readmission rates which is not a focus of the measure. Risk adjustment model would be relevant if all cause component of measure is focused on psychiatric visits. Access to mental health services is not accounted for and represents a major determinant of health. Effect sizes are minimal
Equity
EquityIf measure is focused on Psychiatric admissions as well as including hospitalizations (which would represent the appropriate action post dc from the IPF as part of discharge instructions) in the denominator this measure may impact improvements in Equity
Use and Usability
Use and UsabilityThe measure is planned for use in public reporting and internal and external QI initiatives, but no details are provided such as program name, purpose, geographic coverage, etc. Information does not provide useful/usable information assessing the quality of psychiatric care in an IPF
Summary
I do not see how this measure as constructed can provide any opportunity back to the IPF for QI improvement.
Agree with Staff Comments
Importance
ImportanceWOuld be a useful measure, but limited to program IPF, which may not be able to influence outcomes
Feasibility Acceptance
Feasibility AcceptanceAgree with staff comments
Scientific Acceptability
Scientific Acceptability ReliabilityI question the data insofar as the readmission correlates to the quality measure.
Scientific Acceptability ValidityI would like more discussion of the scientific acceptability. I think the statistical sampling is OK, but the quality nexus is questionable, in my opinion
Equity
EquityI think you will get wide variation on readmits and outcomes on the equity axis depending on social factors
Use and Usability
Use and UsabilityAgree with staff comments
Summary
Agree generally with staff comments. Would like to hear some discussion of the objections raised in the public comment.
A
Importance
ImportanceMost of the supporting literature pertains to reducing readmissions not just reducing ED visits following discharge at IPF. Makes logical sense but all cause ED may not reflect the original IPF condition since not readmitted. May benefit from including BH-related conditions as well as readmissions to capture entire picture of post-discharge experience for a BH patient.
Although they included patient caregivers in the TEP, the questions did not specifically relate to meaningfulness.
Feasibility Acceptance
Feasibility AcceptanceClaims-based measure.
Scientific Acceptability
Scientific Acceptability ReliabilityAgree with staff on potential actions to verify reliability.
Scientific Acceptability ValidityAgree with staff on potential actions to support validity.
Equity
EquityAgree with staff.
Use and Usability
Use and UsabilityNot in use. Usability would improve with some adjustments to the measure, including limiting to BH diagnoses. Would discuss how to handle AMA-discharges.
Summary
N/A
NA
Importance
ImportanceExcited for the concept of reviewing quality of care and transition of care from inpatient psychiatric facilities.
Feasibility Acceptance
Feasibility AcceptanceNA
Scientific Acceptability
Scientific Acceptability ReliabilityHas research been done related to percentage of readmissions not related to inpatient stay?
Scientific Acceptability ValidityNA
Equity
EquityNA
Use and Usability
Use and UsabilityNA
Summary
NA
Measure Should Not Be Endorsed
Importance
ImportanceThere is little evidence to support the presumption that the rate of ED visits after discharge is a valid measure of the quality of care provided by an IPF. There is no information presented to justify the assumption in Exhibit 3 that ED visits can be reduced by 5%, so the calculation in that exhibit is not a valid assessment of the potential impact of the measure on healthcare costs.
There is no information on how many IPFs would be classified as having ED visit rates that are higher or lower than expected. Most measures that are similar to this have such high uncertainty in individual hospital scores that most hospitals can only be classified as “no different than expected.” If the classifications derived from this measure also change from year to year because of the low reliability of the measure, there would be little benefit to using it.
The weaknesses in the measure methodology could cause some hospitals to be inappropriately labeled as “worse than expected” because of the kinds of patients they treat. This would create an undesirable incentive for IPFs to avoid treating patients with characteristics that are likely to result in higher numbers of ED visits, such as patients with multiple comorbidities and patients who do not have good access to primary care and chronic disease management services. Public reporting of the results could mislead patients and families about where they should receive treatment for mental health problems, and use of the measure to modify hospital payments could worsen disparities in access and outcomes for patients.
Feasibility Acceptance
Feasibility AcceptanceIt is feasible to collect the data and calculate the measure.
Scientific Acceptability
Scientific Acceptability ReliabilityThis is not a reliable measure of the quality or efficiency of care provided by inpatient psychiatric hospitals (IPFs).
The Intraclass Correlation Coefficient (ICC) is used to assess measure reliability, and the mean ICC is reported as 0.690. Although this is described as “high” reliability, it is actually moderate at best. Moreover, a high ICC value can be misleading, because if the characteristics of patients differ from facility to facility and the risk adjustment model fails to adequately adjust for this, the ICC value will be higher. This is because more of the variation between patients will be inappropriately attributed to differences in the facilities. Because of the problems with the numerator and risk adjustment model in this measure, it is hard to know how to interpret the ICC.
Moreover, if a measure is going to be used to classify individual IPFs, what matters is the reliability of the measure for individual IPFs, not the mean reliability for all IPFs. The Measure Information Form contains no data on how reliability varies based on the number of patients treated by an IPF. Ordinarily, one would expect higher reliability for facilities with more patients, so the fact that the reliability rates in Table 2 of the Measure Information Form are almost identical across deciles suggests that the deciles are not defined based on IPF size. (It is not clear how the deciles are defined in the table; the Battelle Measure Submission Form is problematic because it requires submission of reliability measures by decile, but it does not specify what variable should be used to define the deciles.)
The Measure Information Form states that only IPFs with 25 or more discharges are included in the measure, but it provides no information as to how this threshold was chosen.
If the measure is going to be used to classify hospitals, an additional way to assess reliability is the extent to which the classifications of IPF ED visit rates differ from year to year. If random variation in patients and weaknesses in the risk adjustment model cause hospitals to change classifications from one year to the next (e.g., being classified as “worse than expected” in some years but not others), the measure will be of little value in guiding patients in their choice of facilities or determining whether there has actually been improvement (or deterioration) in performance. There is no indication in the Measure Information Form as to whether year-to-year reliability was assessed.
The measure developers do not report having made any effort to examine individual IPFs that would be classified as having high or low ED visit rates to ensure those classifications are not based on artifacts of the measure definition or the risk adjustment methodology.
Scientific Acceptability ValidityThis is not a valid measure of the quality or efficiency of care provided by inpatient psychiatric hospitals (IPFs) due to problems with the ways the numerator and denominator are defined and problems with the risk adjustment methodology.
Problems with the Numerator
The stated goal of the measure is to “encourage IPFs to proactively focus on discharge planning and community reintegration during patients’ IPF stays.” The definition of the measure is based on a presumption that an emergency department visit following discharge from the inpatient psychiatric hospital reflects a failure by the IPF to deliver high-quality care. However, the numerator includes emergency department visits that are unrelated to the care delivered by the IPF. At the same time, it excludes many ED visits that are related to the IPF’s care.
- Inclusion of Visits Unrelated to Mental Health Problems. The numerator includes any visit to an Emergency Department within 30 days following discharge from the IPF, regardless of whether the ED visit had anything to do with the problem treated during the IPF stay or other mental health problems. Individuals visit emergency departments for a wide variety of reasons, including accidental injuries, new acute illnesses, and exacerbations of chronic medical conditions. Including such visits overestimates the true rate of mental health problems following discharge.
- Exclusion of ED Visits That Result in Hospital Admissions. The numerator excludes emergency department visits that result in a hospital admission. As a result, if a patient comes to an ED for a mental health problem and is admitted to the hospital, that visit will not be counted in the measure. This will underestimate the true rate of mental health problems following discharge.
Inclusion of Unrelated Visits
It does not appear that the measure developers made any effort to assess what proportion of ED visits after an IPF discharge were actually related to mental health conditions or how that proportion varied by hospital. According to a 2019 ASPE study, the majority of Medicare IPF patients received outpatient health care services that were unrelated to mental health in the 30 days before their IPF admission, so it would not be surprising to find that a high proportion of ED visits after discharge are for things other than mental health problems, and it would also not be surprising to find that there are differences in the number and types of those problems among the patients treated by different IPFs. (Blair R, et al. Transitions in Care and Service Use Among Medicare Beneficiaries in Inpatient Psychiatric Facilities. HHS Office of the Assistant Secretary for Planning and Evaluation, April 2019.)
The inclusion of so many visits unrelated to mental health issues creates two different problems.
- First, it dilutes the effect of differences between IPFs that are related to mental health care. For example, if the rate of ED visits that are actually related to patients’ mental health problems is 50% higher at IPF #1 as at IPF #2, but only half of the visits are related to mental health issues, then the overall visit rate will only be 25% higher at IPF #1 than at IPF #2 (0.5 + 0.75)/(0.5 + 0.5) = 1.25.
- Second, differences between hospitals in the rate of visits that are unrelated to mental health issues may cause IPFs to appear to have higher or lower quality care than they actually do. For example, a patient with mental illness may come to the ED for an exacerbation of a chronic condition such as heart failure or COPD, not because of issues related to their mental illness. Rates of ED visits for chronic disease exacerbations will be higher in communities where chronic disease management services are less available, and patients who have lower incomes or lack of family support may have greater difficulty accessing the services that do exist. In these types of communities and for these types of patients, there will likely be more post-discharge ED visits for chronic disease exacerbations that are unrelated to outpatient mental health care. The risk adjustment methodology only controls for the presence of a chronic condition; it does not control for differences in the care patients receive for the chronic condition. Care for the chronic condition is not the responsibility of the IPF that delivered treatment for mental health problems, but under this measure, a higher rate of ED visits for problems related to chronic conditions will be interpreted as poor quality care for the patient’s mental health needs. Similarly, a patient may come to the ED for an acute condition that is unrelated to mental health (e.g., a viral infection). An IPF in a community experiencing high rates of such illnesses could have higher rates of post-discharge ED visits, and this measure would make it appear that the IPF delivers lower quality psychiatric care. The risk adjustment model used in the proposed measure has variables indicating the presence of chronic conditions, but there are no variables to indicate the presence of new acute conditions.
Rather than attempting to correct for these unrelated problems through a risk adjustment model, it would be better to simply limit the numerator to ED visits that are related to mental health problems. The Measure Information Form indicates that public comments on the draft measure recommended removing the “all-cause” definition used for the numerator. This is very feasible to do:
- For example, a study of the impact on psychiatric care resulting from improving access to community health centers measured changes in the number of psychiatric ED visits, which were defined as ED visits with a diagnosis code corresponding to mental health disorders. (Bruckner, TA, et al. “Psychiatric Emergency Department Visits After Regional Expansion of Community Health Centers,” Psychiatric Services 70(10): 901-906.)
- An analogous approach is used in the “Follow-Up After Psychiatric Hospitalization (FAPH)” measure that is already part of the measures CMS calculates for IPFs. That measure only includes outpatient mental health care encounters after discharge in its numerator, not all outpatient visits, and outpatient mental health care encounters are defined as “outpatient visits, intensive outpatient encounters, or partial hospitalizations provided by a mental health provider for which mental health or SUD diagnoses are mentioned anywhere on the follow-up visit claim.” A similar definition could be used to define ED visits related to mental health issues.
Exclusion of Related Visits
The fact that the measure excludes patients who are admitted to the hospital from the ED creates additional problems. If the patients who are discharged from some IPFs visit emergency departments that are less likely to admit patients to the hospital, those IPFs could have higher ED visit rates, even though the higher rates are due to differences in the way emergency departments operate, not due to differences in either the care delivered by the IPF or the mental health care the patients were receiving in the community after discharge.
It does not appear that the measure developers examined IPF discharges to assess the magnitude of the variation in the proportion of ED visits resulting in hospital admissions. A study of ED visits following hospital discharges for heart and lung conditions found that higher rates of total ED visits were associated with lower rates of hospital readmissions (Venkatesh AK, et al. “Association Between Postdischarge Emergency Department Visitation and Readmission Rates,” Journal of Hospital Medicine 13(9): 589-594). If there is a similar relationship for patients discharged from IPFs, then excluding the subset of ED visits that result in hospital admissions (as this measure proposes to do) would result in erroneous estimates of the differences in rates across hospitals.
This problem could easily be avoided by expanding the numerator to include all mental health-related ED visits, including those that result in a hospital admission.
Failure to Exclude Patients Who Die After Discharge from the Denominator
The denominator excludes patients who died during the IPF stay but it does not exclude patients who died during the 30 days after discharge. As a result, all else being equal, an IPF with a higher post-discharge mortality rate would have a lower ED visit rate and under this measure, the IPF would appear to be delivering higher-quality care. The Measure Information Form describes steps taken to identify patients who were recorded as being dead at discharge but who were actually alive, however, it does not address the issue of determining whether patients died after discharge but before the 30-day measurement window ended.
Problems with the Risk Adjustment Methodology
The measure does not report the actual rate of post-discharge ED visits; it reports a ratio of the “predicted” number of visits to the “expected” number of visits. Both of these parameters are calculated using a hierarchical logistic regression model that is supposed to adjust for differences in the characteristics of patients that affect the rate of ED visits. However, there are serious flaws with the model that is used:
- Failure to adjust for availability of community mental health care. The effectiveness of an IPF’s discharge planning and community reintegration activities is inherently constrained by the availability of mental health services in the communities where the patients discharged from the IPF will be living. As a result, an IPF will likely have a higher rate of ED visits if more of its patients live in communities that have limited community mental health services. There are no variables in the risk adjustment methodology to adjust for this.
- Failure to adjust for whether the patient was discharged to another facility. An ASPE study found that only 70% of IPF patients were discharged to their home or self-care; most of the others were discharged to Skilled Nursing Facilities, intermediate care facilities (ICFs), or inpatient settings (Blair R, et al. Transitions in Care and Service Use Among Medicare Beneficiaries in Inpatient Psychiatric Facilities. HHS Office of the Assistant Secretary for Planning and Evaluation, April 2019). It seems likely that rates of ED visits will differ for patients in these different post-discharge settings, but there is no adjustment for this in the risk adjustment methodology. There is no indication that the measure developers examined either the magnitude of these differences or the variation in the discharge locations across different IPFs in order to determine whether the measure should be stratified or adjusted based on discharge location so that it more accurately and fairly compares IPFs.
- Failure to adjust for propensity of patients to use the ED for chronic conditions. Patients with severe or poorly managed chronic conditions are more likely to have ED visits for exacerbations of those conditions than other patients with the same chronic conditions. Because the numerator of the measure includes ED visits made for any reason, not just visits that are related to mental health conditions, these chronic disease-related visits will increase the rate of ED visits. Although some types of chronic conditions are included as variables in the risk adjustment model, the presence of a condition does not indicate how severe it is or how well managed it is. No variables are included in the risk adjustment model to adjust for this (e.g., the rate at which a patient visited the ED for chronic disease exacerbations before the IPF admission), so an IPF that has more patients with poorly managed chronic conditions could inappropriately appear to be delivering lower-quality mental health care.
- Failure to include variables affecting patient access to other types of health services. Patients who have low incomes, patients who live in rural areas, and patients who live in areas with shortages of healthcare professionals will have greater difficulties accessing medical services and thereby be more likely to visit EDs for diagnosis and treatment of acute conditions. According to the Measure Information Form, variables for Medicaid enrollment (assessed through dual Medicare-Medicaid status), rurality, and health professional availability were tested in the modeling and found to have a large and statistically significant impact on the rate of ED visits. However, these variables were not included in the final model used for the measure, and there is no explanation as to why, other than a statement that “risk models typically do not control for differences in such external factors.” The Measure Information Form specifically notes that non-dually eligible patients had lower odds (0.901) of an ED visit than dually eligible patients, but it does not explain why dual eligibility was not included in the final model nor does it examine the implications of failing to do so. While this may have been done in an effort to prevent “hiding” income-related disparities in treatment by IPFs, it means that an IPF can be penalized for problems that low-income patients face in accessing medical care and other services for conditions that are unrelated to their mental health problems.
Inadequate Evaluation of Risk Adjustment Model Performance
Because of the many factors that could affect the rate of post-discharge ED visits, a thorough analysis is needed to determine how well the regression model in this measure adjusts for variations in outcomes and factors that are beyond the control of an IPF. The simplistic analysis provided in the Measure Information Form is not adequate to justify use of this measure.
The developers report that the c-statistic for the risk adjustment model was 0.667, which they describe as “moderate predictive discrimination.” What the statistic means is that there is only a 66.7% chance that a patient who has a post-discharge ED visit will be classified as higher risk than a patient who does not experience a visit, i.e., a patient will only be classified correctly 2/3 of the time. In comparison, a coin flip would have a 50% chance of predicting a visit accurately. Although one should not expect the model to predict 100% of visits if it is believed that hospitals differ in how well they are carrying out discharge planning and facilitating post-discharge transitions, it seems unlikely that a model with a c-statistic of .67 will accurately identify patients who are likely to make ED visits that are unrelated to the IPF’s care. The fact that this statistic is similar to the statistics reported for other measures that use similarly problematic numerators does not mean the model is good. There is nothing in the Measure Information Form indicating whether alternative risk adjustment models were tested, or whether the model discrimination was evaluated using only ED visits for non-mental health problems, which is where discrimination is most important.
The Measure Information Form states that the risk-decile calibration plot “indicates a close agreement,” even though there is 13% underprediction in the lowest decile, 2-3% underprediction in the highest two deciles, and 4% over-prediction in two of the middle-range deciles. These types of prediction errors could lead to erroneous classifications of hospitals with the types of patients with these risk characteristics. The Hosmer-Lemeshow statistic was reportedly statistically significant, which indicates poor calibration. There is no information indicating that any efforts were made to adjust the model or to test alternative models to achieve better calibration, nor were there any efforts to use alternatives to the Hosmer-Lemeshow statistic to assess calibration. Instead, the measure developers stated that they recalculated the Hosmer-Lemeshow statistic for 20 smaller samples of cases and found that the statistic was not significant in most of these cases. However, they do not show the actual results of these analyses. They state that the rationale for doing these additional estimates was that the Hosmer-Lemeshow statistic is more likely to be significant with larger numbers of cases, but the converse of this is also true – it is less likely to be significant with smaller samples. Failure to find statistically significant results in artificially-restricted samples does not mean that there is no calibration problem.
Equity
EquityThe weaknesses in the measure methodology could cause some hospitals to be inappropriately labeled as “worse than expected” because of the kinds of patients they treat. As a result, use of the measure would create an undesirable incentive for IPFs to avoid treating patients with characteristics that are likely to result in higher numbers of ED visits, such as patients with multiple comorbidities and patients who do not have good access to primary care and chronic disease management services. This would increase inequities in care.
Use and Usability
Use and UsabilityMost of the studies cited as demonstrating the importance of this measure do not actually discuss ED visits, so there is little evidence to support the presumption that the rate of ED visits after discharge is a valid measure of the quality of care provided by an IPF.
The Measure Information Form provides no information on how many IPFs would be classified as having ED visit rates that are higher or lower than expected. Most measures that are similar to this have such high uncertainty in individual hospital scores that most hospitals can only be classified as “no different than expected.” If the classifications derived from this measure also change from year to year because of the low reliability of the measure, there would be little benefit to using it.
Summary
This measure should not be endorsed. There is no business case for using it. It is not a valid measure of the quality or efficiency of care provided by inpatient psychiatric hospitals (IPFs) due to problems with the ways the numerator and denominator are defined and problems with the risk adjustment methodology. Public reporting of the results could mislead patients and families about where they should receive treatment for mental health problems, and use of the measure to modify hospital payments could worsen disparities in access and outcomes for patients.
This measure needs improved…
Importance
ImportanceThe literature and panels support the importance of this measure.
Feasibility Acceptance
Feasibility AcceptanceThe measure seems feasible and uses claims data that are available to anyone.
Scientific Acceptability
Scientific Acceptability ReliabilityAs indicated in the staff assessment there is a need to redo the ICC analysis.
Scientific Acceptability ValidityI don't believe that a c-statistic of 0.67 is sufficient to have a reliable or valid estimate of an expected outcome. Particularly if this is for use in public reporting or quality improvement. It could provide inaccurate estimates of performance.
Equity
EquityI thought the assessment for equity was sufficient.
Use and Usability
Use and UsabilityThere need to be adjustments to the inclusion exclusion criteria which were called out in public comments (i.e., failure to exclude patients who died after discharge). I also believe that given that this is a facility level hierarchical model, it would not be adequate for use in quality improvement. There is no capability to drill down to identify segments of populations where you need improvement with a measure like this.
Summary
This measure needs improved performance metrics to be acceptable for use in peer comparison reporting. It is an important measure, but I don't think it is quite sufficient at this point.
Viability
Importance
ImportanceAdding tracking of ER admission to the existing tracking is important to patients.
Feasibility Acceptance
Feasibility AcceptanceOne group with 4 professionals and 3 caregivers in one joint panel seems insufficient to obtain broad feedback.
Scientific Acceptability
Scientific Acceptability ReliabilityMeasuring all ER admissions, not just for mental health related issues reduces the reliability of the measure.
Issues of concern to patients that do not result in readmission are not tracked
Scientific Acceptability ValiditySignificant viability testing was completed
Equity
EquityMultiple social determinants of health (SDOH) are tracked enabling more detailed analysis. Economic date, which would ideally be included is not available in the electronic medical record.
Use and Usability
Use and UsabilityThe main barrier is counting all ER admissions, not just for mental health issues.
Summary
The intent of this measure is positive and there is a significant need for it. The patient review is insufficient and tracking all admissions from the ER makes the measure too broad to be viable.
too many unanswered questions to endorse
Importance
ImportanceLong term stability is an important goal for high risk patients with BH disorders.
The question as to who is the accountable party for this stability is relevant.
It may not be the hospital as many of these patients are in high risk carve out managed care entities which should be responsible for care coordination and services and oversight.
The acute psychiatric hospital should be accountable for stabilization and appropriate discharge, but has limited resources for assuring follow up of all clinical issues for complex patients that pose adherence challenges.
Is this measure of accountability for the acute care hospital important? Probably not depending on the systems of care designed for these patients in a particular community. The hospital is only one entity in this framework.
Feasibility Acceptance
Feasibility Acceptancedata are available
Scientific Acceptability
Scientific Acceptability Reliabilitydata collection available
Scientific Acceptability Validitywho/what is the accountable entity?
hard to address this item without understanding intended use
if the hospital is the accountable entity for the measure, validity is a major concern
Equity
Equity??
Use and Usability
Use and Usabilitywho/what is the accountable entity?
is it an insurance plan? a hospital? a Medicaid program? a state commission?
hard to endorse a measure when you do not know how it is to be used
if the measure is to be used to compare hospitals, it is quite flawed in concept
Summary
ER visits post inpatient BH events requires a system of care for these patients - that is often beyond the scope and expectations of the admission facility. This measure in the aggregate is useful for public administrators, insurance entities - but does not seem to be a fair expectation of a hospital in isolation from the regional programs to manage these patients.
This type of measure seems better suited for accountability of a managed care entity than a facility.
Review
Importance
ImportanceThe developer sufficiently described the importance of this measure
Feasibility Acceptance
Feasibility AcceptanceThe measure relies on readily available Medicare claims data, and no data availability issues were identified. This ensures the feasibility of implementing the measure comprehensively and efficiently for large patient populations.
Scientific Acceptability
Scientific Acceptability ReliabilityResults of the test as mentioned, suggests high reliability.
Scientific Acceptability ValidityThough the observed differences were small, I still believe that the results are valid.
Equity
EquityI think the developer did a good job in portraying the contribution to health equity.
Use and Usability
Use and UsabilityNo mention of whether this measure can be state-wide, regional, or national program or an internal quality improvement program, specific to an organization. Also, there is insufficient information on the impact on healthcare providers within the mentioned domain. Additionally, there were concerns regarding the IPF’s ability to impact what happens post-discharge.
Summary
N/A
Would not support endorsement
Importance
ImportanceThe first key question here is whether all-cause ED visits after IPF admission are a true marker of IPF care quality. The studies cited are observational and of weak quality, generally showing that having failed to follow up as an outpatient in a timely fashion is associated with higher rates of readmission (not ED visits). This association is unsurprising, but no supportive evidence is given that the relationship is causal—and more importantly, that improving rates of timely outpatient follow up improves readmissions, let alone all-cause ED visits. If the measure is intended to “encourage IPFs to proactively focus on discharge planning and community reintegration during patients’ IPF stays,” then causality will have to be proven.
The second key question, even if you escape the above shortcoming, is who is responsible for this transition. Certainly the IPF has an important role, but this may be dominated by patient-level factors, and more importantly by structural factors like community access to outpatient psychiatric care that the patient can access. It doesn’t seem appropriate to place the full weight of this metric (even if it were otherwise valid) on the IPF. By doing so, this risks worsening equity for IPFs that have fewer resources or practice in underserved settings where patients may have more difficulty accessing care (or may use the ED as a PCP).
For these reasons, this measure should not be endorsed.
Feasibility Acceptance
Feasibility AcceptanceAs designed, this is available from claims data.
Scientific Acceptability
Scientific Acceptability ReliabilityAgree with another commenter that capturing IPF ED visit rate differences over time would be important to gauge reliability. A C-statistic of 0.67, even though in line with other measures, is not compelling for something intended to drive individual patient care decisions – and is more a criticism of those other measures.
Scientific Acceptability ValidityThe use of all cause ED visits as the numerator rather than BH-related ED visits +/- hospitalizations is suboptimal. As the number of non-BH visits increases relative to BH visits this creates a bias toward the null while at the same time making the overall metric less modifiable by the IPF itself. Excluding patients who are admitted to the hospital from the ED can introduce bias based on the capacity and practice pattern of that ED. Would favor limiting the numerator to BH ED visits and BH admissions.
Lack of incorporation of SDOH in model may be because data quality for those elements is poor. Residual confounding by external factors such as SDOH not well captured in the model will likely manifest as apparent differences at the IPF level with this modeling. This also has equity implications.
Equity
EquityLabeling lower resource area centers as “poor quality” has the potential to worsen equity and may have an unintended distortionary effect on care. The current inability to account for local options for BH follow up or for how often patients may use the ED as their de facto primary care are limitations that could again adversely affect equity.
Use and Usability
Use and UsabilityThe lack of proven, modifiable, causality is a major flaw that can't be addressed until we have more data. With the potential adverse impacts on equity, the potential financial burden on IPF, and the attribution of "failure" to just the IPF, usability seems poor.
Summary
See detailed comments above. The causality and modifiability of this metric and it's potential to actually improve outcomes is unproven. Failing that, the potential adverse effects of creating this as a QI target outweigh the unproven benefit, however well intentioned.
Not met
Importance
ImportanceThere is at least one big flaw with this measure. For example, the numerator is comprised of patients 18 and older with an emergency department (ED) visit, including observation stays, for any cause.
Several occasions would negatively affect this measure. For example, the transfer of BH patients to acute care hospitals for non-BH medical or surgical interventions that are followed by an associated ED admission would be counted in this measure. Also if that is not the case any unrelated ED visit without BH issues is also not counted.
Why are ED admissions that are followed by an inpatient admission excluded?
Feasibility Acceptance
Feasibility AcceptanceData is readily available.
Scientific Acceptability
Scientific Acceptability ReliabilityAgree with staff review.
Scientific Acceptability ValidityAgree with staff review.
Equity
EquityAccess to IPF was not sufficiently studied. The availability of IPF could be directly correlated to BH ED admissions but was not addressed in this proposal.
Use and Usability
Use and UsabilityAgree with staff review.
Summary
N/A
Unfortunately, not met
Importance
Importance- I worked for many years in behavioral health. We convened a community collaboration to Increase 30-day follow-up after an inpatient MH stay. In two years, we increased follow-up from 18% to 75%.
- Agree with staff on the rest.
Feasibility Acceptance
Feasibility Acceptance- Agree with staff
Scientific Acceptability
Scientific Acceptability Reliability- Agree with staff
Scientific Acceptability Validity- Developers don't address the impact of medical-community collaboration on the measure.
- Developers don't consider the extraordinary boarding waits that people have until a bed or community services opens up. Discharge AMA can reflect this problem.
Equity
Equity- Agree with staff
- Response weak, but better than most
Use and Usability
Use and Usability- This very important measure is usable for quality management with community partners.
- The most readable report for laypeople I've seen from CMS measure developers
- It is not usable to compare or rate providers. Although important, feasible, and reliable, the measure lacks the sensitivity to incorporate all factors - social, political, business, and community culture, and access to services.
Summary
- This very important measure is usable for quality management with community partners.
- The most readable report for laypeople I've seen from CMS measure developers
- It is not usable to compare or rate providers. Although important, feasible, and reliable, the measure lacks the sensitivity to incorporate all factors - social, political, business, and community culture, and access to services. This very important measure is usable for quality management with community partners.
- The most readable report for laypeople I've seen from CMS measure developers
- It is not usable to compare or rate providers. Although important, feasible, and reliable, the measure lacks the sensitivity to incorporate all factors - social, political, business, and community culture, and access to services.
Do not recommend this measure at this time.
Importance
ImportanceThere is considerable evidence referenced regarding the importance of a follow-up visit post discharge, but little evidence provided on the importance of returns to the ED as a significant provider gap in care. Instead, a measure on the # of non-follow-up visits scheduled prior to discharge would have more clinical importance and drive more action as stated in the literature provided. Returns to the ED would be best suited as a descriptive or sub-measure of a broader measure of overall returns to the acute care facility, including both ED visits and readmissions rather than as a separate measure. As stated, there is a moderate, yet meaningful, positive relationship between the facility rates on the IPF ED Visit measure and the IPF Readmission measure (Spearman ranked correlation ρ = 0.42) indicating an opportunity to combine ED and readmission (return to the facility) rather than separate measures.
Additionally, the 30-day time window was not clearly stated. There was reference to follow-up visits within 3 months, but nothing mentioned regarding the importance of the 30-day window.
Feasibility Acceptance
Feasibility AcceptanceWithin the feasibility section, claims data is listed as only data source needed. While the claims data is readily available, the data needed to make this measure meaningful may not.
Scientific Acceptability
Scientific Acceptability ReliabilityConsiderable analysis was performed in assessing the measure. The reported deciles seem to be deciles of the results of the bootstrap iterations, not entity-level reliabilities. Without estimates of reliability by entity it is only possible to conclude that less than 50% of the entities have a reliability <0.6, although that number may be much lower.
Scientific Acceptability ValidityAs stated, the C-statistic for the final risk-adjustment model was 0.670. This is only 17% better than flipping a coin, indicating there is more variability not captured by the model which introduces validity challenges with this model's ability to drive clear action.
Equity
EquityWhile SDOH factors and race were evaluated and showed statistically meaningful impact on returns to the ED, the measure does not directly evaluate performance that would lead to a reduction in healthcare inequities. The final recommended measure is not focused on providing insights into healthcare disparities.
Use and Usability
Use and UsabilityGiven the correlation with the existing 30-day readmission measure and the lack of clear evidence that this measure offers clearer direction than the current follow-up visit measure, the utility of this measure is questionable, particularly for pay for performance reporting. Using this measure as a 'descriptive statistic' to understand how the patient is returning to the facility (a stratification between ED and readmissions), but the follow-up visit appears to be the key in supporting continuum of care. In turn, this measure has limited utility and would not recommend pay for performance reporting.
Summary
There is considerable evidence referenced regarding the importance of a follow-up visit post discharge, but little evidence provided on the importance of returns to the ED as a significant provider gap in care. Instead, a measure on the # of non-follow-up visits scheduled prior to discharge would have more clinical importance and drive more action as stated in the literature provided. Returns to the ED would be best suited as a descriptive or sub-measure of a broader measure of overall returns to the acute care facility, including both ED visits and readmissions rather than as a separate measure. As stated, there is a moderate, yet meaningful, positive relationship between the facility rates on the IPF ED Visit measure and the IPF Readmission measure (Spearman ranked correlation ρ = 0.42) indicating an opportunity to combine ED and readmission (return to the facility) rather than separate measures.
Additionally, the 30-day time window was not clearly stated. There was reference to follow-up visits within 3 months, but nothing mentioned regarding the importance of the 30-day window.
Given the correlation with the existing 30-day readmission measure and the lack of clear evidence that this measure offers clearer direction than the current follow-up visit measure, the utility of this measure is questionable, particularly for pay for performance reporting. Using this measure as a 'descriptive statistic' to understand how the patient is returning to the facility (a stratification between ED and readmissions), but the follow-up visit appears to be the key in supporting continuum of care. In turn, this measure has limited utility and would not recommend pay for performance reporting.
30-Day Risk Standardized All-Cause ED Visit following IPF
Importance
ImportanceThere is mixed thoughts on this measure. There is some evidence that the performance to the benchmark would decrease rates. The developed did have a TEP with technical experts and patient caregivers. Information from the TEP was not completely provided especially for the caregiver who did not agree with the measure and not clear that the questions addressed if the measure was meaningful.
Feasibility Acceptance
Feasibility AcceptanceThe measure uses available administrative and survey data.
Scientific Acceptability
Scientific Acceptability ReliabilityReliability is not met. Many of the entities have a reliability less than 0.6 (about 50%). Possibly could be do to case volume and need for higher volumes. Concern for lower volume providers.
Scientific Acceptability ValidityNo results from the TEP and only 1 of the 3 caregivers felt that the metric would be useful. Effect size is minimal
Equity
EquityThe develop did evaluate 17 social determinants of health for disparities in the measure.
Use and Usability
Use and UsabilityMeasure is planned for public reporting however it is not clear on the purpose, level of analysis, areas of the measure. There is importance to reduce avoidable emergency department visits after discharge from IPF but there were comments about the ability of IPFs to influence the measure and further specifications needed.
Summary
This measure should not be endorsed. There are reliability and validity issues. Public reporting of this measure may not provide information to assist patients and families about IPF care. Additionally, disparities and outcomes for patients with health disparities could worsen.
Overall Summary for Measure # 4190
Importance
ImportanceAgree with the staff assessment.
Feasibility Acceptance
Feasibility AcceptanceThe measure is a claims-based measure and is feasible to collect as the system to collect data and calculate the measure is an automated process using electronic standardized data already routinely generated for billing purposes
Scientific Acceptability
Scientific Acceptability ReliabilityBased on the results submitted, it is unclear whether the ICC reliability threshold is met at the accountable entity-level.
Scientific Acceptability ValidityAgree with the staff comments. The submission could be strengthened with rationale supporting the hypothesis and the expectations around the modest results and anticipated direction. Face validity and risk adjustment approaches seem reasonable.
Equity
EquityThe developer added social determinant of health variables to the risk model adjusted by clinical risk factors and nearly all had weak associations with the outcome. Social determinant of health risk factors were assessed but were not retained during the variable selection process.
Use and Usability
Use and UsabilityMy primary concern is whether performance scores yield actionable information that can be used to improve performance among measured entities. I agree with public comment regarding the "all-cause" component of the measure limiting the usefulness of the measure. IPFs are responsible for a specific scope of care and may not be able to affect other clinical areas. Additionally, I agree that excluding ED visits that result in inpatient admissions may limit the usefulness of the measure. It's not clear that the approach/strategies for improvement would differ for an ED visit that does or does not result in an inpatient admission. One strategy may be to include ED visits that result in inpatient admissions and present stratified results for assessment of both groups. This would also address the harmonization with the readmission measure.
Summary
The measure has some addressable issues regarding scientific validity. The measure provides important information for public reporting in terms of awareness of gaps and quality improvement information for IPFs to understand where patients go after discharge. Further discussion is needed as to the actionability of the measure as currently specified as highlighted by the public comments.
This measure needs more development and analysis to refine it
Importance
ImportanceNot clear that an ED only measure is important, particularly one that excludes ED visits leading to readmissions. Need for a patient to go to ED for follow on care will depend not only on adequacy of discharge planning but availability of follow on services in the community. This is not addressed in discussion. Need a better model of why ED visits post-discharge occur, how one should view ED visit without admission, and therefore what the measure is capturing. It is described as a measure to encourage better discharge planning, with assumption this will reduce ED visits, but assumption is not adequately tested or documented.
Cost analysis focuses only on payment for ED, does not consider what the cost of community-based care that might reduce ED visits or readmissions would be.
Feasibility Acceptance
Feasibility AcceptanceAs currently spec'd, uses only administrative data, so should be feasible.
Scientific Acceptability
Scientific Acceptability ReliabilityUse of bootstrapping to obtain an analog to split sample reliability measurement is fine.
Average correlation of 0.69 is on border of acceptable for correlation within facilities of different samples.
The method does not allow analysis of reliability of rankings of facilities or stability of estimates of where in the distribution facilties fall. This analysis needs to be augmented with analysis of stability of ranking or facility assignment to deciles or quartiles to adequately inform how reliable the measure is in differentiating facility performance.
Scientific Acceptability ValidityOne of the public comments asks why all cause ED use was used rather than mental health associated visits. This is a legitimate question.
One of the reasons why ED visits might be high are that there are fewer mental health treatment resources in some areas than others. Rural areas one example. Also, dual eligibles may have less access to private MH facilities or services. In these cases, ED might be serving as substitute for less access to private services. SDOH measures evaluated do not directly assess this, yet it is critical to understanding ED use.
Equity
EquitySee comment on SDOH and lack of direct measurement of access to non-hospital non ED MH services for those discharged. Failure to consider this makes measure potentially misleading on quality.
Developers also did not seem to reflect on fact that in this Medicare population, over half of those discharged from IPF were under 65. There is a need for more analysis of who the IPF population is, what resources they have available, and what expectations for ED use should be.
Use and Usability
Use and UsabilitySince measure is ambiguous in terms of reasonable expectations, use is suspect.
Summary
It is not clear what ED use without admission is measuring, under what circumstances it might be appropriate given potential unavailability of other resources for follow on care, the potential mismeasurement due to inclusion of all causes for ED visits. Reliability analysis needs to assess the stability of cross facility rankings if ICC's being used (a reasonable decision) rather than signal to noise measurement.
The measure needs refinement and not yet ready for endorsement.
Importance
ImportanceThe proposed measure is a reflection on community mental health resources which fall under state, county and city jurisdictions. It complements a different measure - 30-day all cause unplanned readmission (#2860) following psychiatric hospitalization. At best, it could be considered an intermediate measure but not as a clinically meaningful outcome measure. The measure does not meet criteria for importance as patient input does not support the conclusion that the measured outcome, process, or structure is meaningful or it does so with a low degree of certainty.
Psychiatric conditions are frequently associated with other comorbidities and socioeconomic vulnerability and substance use disorders. IPF ED visits not requiring a hospital admission could occur from a variety of clinical and socioeconomic reasons and the number of any cause ED visits following discharge from an IPF is not clinically meaningful and not always reflective of the care received during the index IPF stay. Community collaboratives, when they exist, do have the ability to reduce these. Loss of insurance or death after discharge are not factored in the measure.
Feasibility Acceptance
Feasibility AcceptanceThe measure does not meet feasibility criteria as long-term or no path is specified to support routine and electronic data capture with an implementable data collection strategy. This measure is not an eCQM and data collection can be onerous on already resource challenged facilities.
The patient population included in this metric, particularly those who live in inner city areas, frequently visit multiple EDs belonging to different health systems, which reduces the ability of any one facility to reduce these ED visits significantly. The numerator does not specify the location of the ED. IPFs rarely have their own ED, and even if they do, the patients move between EDs in the area.
Scientific Acceptability
Scientific Acceptability ReliabilityThe data measures all ED visits without specifying location of ED, whereas the proposed measure specifies ED visit to the IPF that discharged the patient.
Scientific Acceptability ValidityThe measure developer used Medicare claims data from July 2019 to July 2021 to develop the measure. 194,531 patients were included in the data analysis. The measure appears to have been developed with the best of intentions as they “identified barriers to post-IPF discharge care, such as stigma and poverty, and actions that can be taken by IPFs to reduce the likelihood of ED visits and readmissions following IPF discharge such as aiding the transition to outpatient support, focusing on patient self-efficacy, and maximizing social and peer support." The data seem to indicate that any ED visit was considered in the data analysis, but the measure is for ED visits back to the discharging IPF. Excluding patients who are admitted might incentivize them to admit patients from the ED. Post-discharge death needs to be excluded as well.
Equity
EquityThe measure developers gathered data on social determinants of health and risk stratified data by SDOH.
Use and Usability
Use and UsabilityThe criteria for use and usability is met. However, the planned use for pay for reporting is a cause for concern as the measure is not yet ready for use, and also incentivizes better resourced IPFs compared to those that are under-resourced and widens the support that the facilities receive.
Summary
For all the above reasons, this measure is not yet ready for endorsement. Development of structural measures and leading indicators for the patients having the necessary peer support and social support is a better approach to solving this important problem.
no additional comments
Importance
ImportanceDeveloper shared research on importance, including expert panel with patient caregivers.
Feasibility Acceptance
Feasibility AcceptanceMeasure uses widely available data
Scientific Acceptability
Scientific Acceptability ReliabilityDefer to staff response on limitations
Scientific Acceptability ValidityDefer to staff response on limitations
Equity
Equity17 separate SDOH factors analyzed
Use and Usability
Use and UsabilityDefer to staff response on limitations
Summary
n.a
Concern regarding validity of data
Importance
ImportancePopulation of interest with potential for material impact
Feasibility Acceptance
Feasibility AcceptanceUtilizes readily available data
Scientific Acceptability
Scientific Acceptability Reliabilitywork to further define denominator and minimum N
Scientific Acceptability ValidityFurther work to define risk adjustment methodology
Equity
EquitySDOH data utilized
Use and Usability
Use and Usabilityfurther definition required
Summary
While the measure developer does address some concerns regarding data collection and validity, there are numerous outstanding questions that remain. Further definition needs to be done along with consideration for SDOH and impact.
Agree with majority of staff comments
Importance
ImportanceThe rational is good but more input is needed
Feasibility Acceptance
Feasibility AcceptanceClaims data
Scientific Acceptability
Scientific Acceptability ReliabilityAgree with staff comments
Scientific Acceptability ValidityAgree with staff comments
Equity
EquityAgree with staff comments
Use and Usability
Use and UsabilityAgree with staff comments
Summary
N/A
Not Ready
Importance
ImportanceFollow-up encounters for psychiatric issues demonstrate effectiveness of psychiatric care. I don't think a visit to the ED for suspected appendicitis should count
Feasibility Acceptance
Feasibility AcceptanceEasily obtained through claims data
Scientific Acceptability
Scientific Acceptability ReliabilityAgree with staff assessment
Scientific Acceptability ValidityAgree with staff assessment
Equity
EquityAgree with staff assessment
Use and Usability
Use and UsabilityAgree with staff assessment
Summary
Ability to measure the effectiveness of inpatient psychiatric care is important, given the costs and chrosic nature of many illn esses. All-cause measures risk being too inclusive ands thereby clouding the role the psychiatric care played in sending the patient back to the hospital.
The "30-Day Risk Standardized All-Cause Emergency Department Visit Following Inpatient Psychiatric Facility Discharge" measure should not be endorsed. It is not a valid measure of the quality or efficiency of care provided by inpatient psychiatric hospitals (IPFs) due to problems with the ways the numerator and denominator are defined and problems with the risk adjustment methodology. Public reporting of the results could mislead patients and families about where they should receive treatment for mental health problems, and use of the measure to modify hospital payments could worsen disparities in access and outcomes for patients.
Problems with the Numerator
The stated goal of the measure is to “encourage IPFs to proactively focus on discharge planning and community reintegration during patients’ IPF stays.” The definition of the measure is based on a presumption that an emergency department visit following discharge from the inpatient psychiatric hospital reflects a failure by the IPF to deliver high-quality care. However, the numerator includes emergency department visits that are unrelated to the care delivered by the IPF. At the same time, it excludes many ED visits that are related to the IPF’s care.
Inclusion of Unrelated Visits
It does not appear that the measure developers made any effort to assess what proportion of ED visits after an IPF discharge were actually related to mental health conditions or how that proportion varied by hospital. According to a 2019 ASPE study, the majority of Medicare IPF patients received outpatient health care services that were unrelated to mental health in the 30 days before their IPF admission, so it would not be surprising to find that a high proportion of ED visits after discharge are for things other than mental health problems, and it would also not be surprising to find that there are differences in the number and types of those problems among the patients treated by different IPFs. (Blair R, et al. Transitions in Care and Service Use Among Medicare Beneficiaries in Inpatient Psychiatric Facilities. HHS Office of the Assistant Secretary for Planning and Evaluation, April 2019.)
The inclusion of so many visits unrelated to mental health issues creates two different problems.
Rather than attempting to correct for these unrelated problems through a risk adjustment model, it would be better to simply limit the numerator to ED visits that are related to mental health problems. The Measure Information Form indicates that public comments on the draft measure recommended removing the “all-cause” definition used for the numerator. This is very feasible to do:
Exclusion of Related Visits
The fact that the measure excludes patients who are admitted to the hospital from the ED creates additional problems. If the patients who are discharged from some IPFs visit emergency departments that are less likely to admit patients to the hospital, those IPFs could have higher ED visit rates, even though the higher rates are due to differences in the way emergency departments operate, not due to differences in either the care delivered by the IPF or the mental health care the patients were receiving in the community after discharge.
It does not appear that the measure developers examined IPF discharges to assess the magnitude of the variation in the proportion of ED visits resulting in hospital admissions. A study of ED visits following hospital discharges for heart and lung conditions found that higher rates of total ED visits were associated with lower rates of hospital readmissions (Venkatesh AK, et al. “Association Between Postdischarge Emergency Department Visitation and Readmission Rates,” Journal of Hospital Medicine 13(9): 589-594). If there is a similar relationship for patients discharged from IPFs, then excluding the subset of ED visits that result in hospital admissions (as this measure proposes to do) would result in erroneous estimates of the differences in rates across hospitals.
This problem could easily be avoided by expanding the numerator to include all mental health-related ED visits, including those that result in a hospital admission.
Failure to Exclude Patients Who Die After Discharge from the Denominator
The denominator excludes patients who died during the IPF stay but it does not exclude patients who died during the 30 days after discharge. As a result, all else being equal, an IPF with a higher post-discharge mortality rate would have a lower ED visit rate and under this measure, the IPF would appear to be delivering higher-quality care. The Measure Information Form describes steps taken to identify patients who were recorded as being dead at discharge but who were actually alive, however, it does not address the issue of determining whether patients died after discharge but before the 30-day measurement window ended.
Problems with the Risk Adjustment Methodology
The measure does not report the actual rate of post-discharge ED visits; it reports a ratio of the “predicted” number of visits to the “expected” number of visits. Both of these parameters are calculated using a hierarchical logistic regression model that is supposed to adjust for differences in the characteristics of patients that affect the rate of ED visits. However, there are serious flaws with the model that is used:
Inadequate Evaluation of Risk Adjustment Model Performance
Because of the many factors that could affect the rate of post-discharge ED visits, a thorough analysis is needed to determine how well the regression model in this measure adjusts for variations in outcomes and factors that are beyond the control of an IPF. The simplistic analysis provided in the Measure Information Form is not adequate to justify use of this measure.
The developers report that the c-statistic for the risk adjustment model was 0.667, which they describe as “moderate predictive discrimination.” What the statistic means is that there is only a 66.7% chance that a patient who has a post-discharge ED visit will be classified as higher risk than a patient who does not experience a visit, i.e., a patient will only be classified correctly 2/3 of the time. In comparison, a coin flip would have a 50% chance of predicting a visit accurately. Although one should not expect the model to predict 100% of visits if it is believed that hospitals differ in how well they are carrying out discharge planning and facilitating post-discharge transitions, it seems unlikely that a model with a c-statistic of .67 will accurately identify patients who are likely to make ED visits that are unrelated to the IPF’s care. The fact that this statistic is similar to the statistics reported for other measures that use similarly problematic numerators does not mean the model is good. There is nothing in the Measure Information Form indicating whether alternative risk adjustment models were tested, or whether the model discrimination was evaluated using only ED visits for non-mental health problems, which is where discrimination is most important.
The Measure Information Form states that the risk-decile calibration plot “indicates a close agreement,” even though there is 13% underprediction in the lowest decile, 2-3% underprediction in the highest two deciles, and 4% over-prediction in two of the middle-range deciles. These types of prediction errors could lead to erroneous classifications of hospitals with the types of patients with these risk characteristics. The Hosmer-Lemeshow statistic was reportedly statistically significant, which indicates poor calibration. There is no information indicating that any efforts were made to adjust the model or to test alternative models to achieve better calibration, nor were there any efforts to use alternatives to the Hosmer-Lemeshow statistic to assess calibration. Instead, the measure developers stated that they recalculated the Homer-Lemeshow statistic for 20 smaller samples of cases and found that the statistic was not significant in most of these cases. However, they do not show the actual results of these analyses. They state that the rationale for doing these additional estimates was that the Homer-Lemeshow statistic is more likely to be significant with larger numbers of cases, but the converse of this is also true – it is less likely to be significant with smaller samples. Failure to find statistically significant results in artificially-restricted samples does not mean that there is no calibration problem.
Inadequate Assessment of Measure Reliability or Case Threshold
The Intraclass Correlation Coefficient (ICC) is used to assess measure reliability, and the mean ICC is reported as 0.690. Although this is described as “high” reliability, it is actually moderate at best. Moreover, a high ICC value can be misleading, because if the characteristics of patients differ from facility to facility and the risk adjustment model fails to adequately adjust for this, the ICC value will be higher. This is because more of the variation between patients will be inappropriately attributed to differences in the facilities. Because of the problems with the numerator and risk adjustment model in this measure, it is hard to know how to interpret the ICC.
Moreover, if a measure is going to be used to classify individual IPFs, what matters is the reliability of the measure for individual IPFs, not the mean reliability for all IPFs. The Measure Information Form contains no data on how reliability varies based on the number of patients treated by an IPF. Ordinarily, one would expect higher reliability for facilities with more patients, so the fact that the reliability rates in Table 2 of the Measure Information Form are almost identical across deciles suggests that the deciles are not defined based on IPF size. (It is not clear how the deciles are defined in the table; the Battelle Measure Submission Form is problematic because it requires submission of reliability measures by decile, but it does not specify what variable should be used to define the deciles.)
The Measure Information Form states that only IPFs with 25 or more discharges are included in the measure, but it provides no information as to how this threshold was chosen.
If the measure is going to be used to classify hospitals, an additional way to assess reliability is the extent to which the classifications of IPF ED visit rates differ from year to year. If random variation in patients and weaknesses in the risk adjustment model cause hospitals to change classifications from one year to the next (e.g., being classified as “worse than expected” in some years but not others), the measure will be of little value in guiding patients in their choice of facilities or determining whether there has actually been improvement (or deterioration) in performance. There is no indication in the Measure Information Form as to whether year-to-year reliability was assessed.
The measure developers do not report having made any effort to examine individual IPFs that would be classified as having high or low ED visit rates to ensure those classifications are not based on artifacts of the measure definition or the risk adjustment methodology.
Lack of Business Case for Using the Measure and Undesirable Effects of Using It
Most of the studies cited as demonstrating the importance of this measure do not actually discuss ED visits, so there is little evidence to support the presumption that the rate of ED visits after discharge is a valid measure of the quality of care provided by an IPF. There is also no information presented to justify the assumption in Exhibit 3 that ED visits can be reduced by 5%, so the calculation in that exhibit is not a valid assessment of the potential impact of the measure on healthcare costs.
The Measure Information Form provides no information on how many IPFs would be classified as having ED visit rates that are higher or lower than expected. Most measures that are similar to this have such high uncertainty in individual hospital scores that most hospitals can only be classified as “no different than expected.” If the classifications derived from this measure also change from year to year because of the low reliability of the measure, there would be little benefit to using it.
In addition, since the weaknesses in the measure methodology could cause some hospitals to be inappropriately labeled as “worse than expected” because of the kinds of patients they treat, use of the measure would create an undesirable incentive for IPFs to avoid treating patients with characteristics that are likely to result in higher numbers of ED visits, such as patients with multiple comorbidities and patients who do not have good access to primary care and chronic disease management services.