We wish to provide the following information and context to clarify the committee’s understanding of the validity of our measure Engagement in Community-Based Mental Health Care After a Mental Health Hospitalization (EIC). In our measure application, we presented three different forms of validity – face validity, construct validity, and empirical validity. During the August 3, 2023 Prevention and Population Health Standing Committee Measure Evaluation meeting, the committee recognized that the face validity and construct validity of our measure were sufficient. For empirical validity, we showed three concordance (C) statistics. These statistics indicated the ability of our measure to predict outcomes after the measurement period. The C statistic for engagement in care six months after discharge was predictive, for medication adherence four to six months after discharge was borderline predictive, and for inpatient mental health readmissions in months four to nine after discharge was not predictive. We feel that the committee weighed the not predictive and borderline concordance findings too heavily and did not give enough consideration to the sufficient face validity, construct validity, and partial empirical validity demonstrated by the measure. Our measure is a process measure, and even widely used process measures in behavioral health are not always predictive of outcomes. The claims data we used to test the measure did not allow for adjustment of potential important confounding factors such as lack of housing. These three C statistics were included in our application to be fully transparent about the testing that we had completed during measure development.
National Quality Forum (NQF) endorsed measures of mental health follow-up widely used in HEDIS also did not show concordance with mental health readmissions in our analysis. In our 2018 NYS Medicaid cohort, when looking at the ability to predict an inpatient mental health readmission four to nine months after discharge, the C statistic for one mental health follow-up visit in the 30 days after a mental health discharge (HEDIS Follow-Up After Hospitalization for Mental Illness, FUH) is very similar to the EIC measure (C Statistic for EIC=0.5291, for FUH=0.5229). Additionally, when having at least one community-based mental health visit in month 6 is used as an outcome, FUH is less predictive than the EIC measure (C Statistic for EIC=0.7244, for FUH=0.6847).
It is important to recognize that previously endorsed measures of mental health follow-up also did not demonstrate the ability to predict outcomes in their NQF applications. In the validity section of the NQF application for FUH, NCQA described their face validity process and established construct validity through correlation of the two FUH measure components with each other and with similar measures. For EIC, we found a significant moderate positive correlation with FUH, which agreed with our hypothesis. NCQA did not show concordance statistics or correlations with outcomes for FUH. For validity of Follow-up after Discharge from the Emergency Department for Mental Health or Alcohol or Other Drug Dependence, NCQA described their face validity process and showed an analysis comparing state level performance on the measure with state level rates of MH and SUD inpatient use. They hypothesized that states with better performance on the measure would have lower rates of inpatient use. Their analysis did not show a significant difference in MH or SUD inpatient rates between states in the lower and upper quartiles of the follow-up measure. Despite these failures to establish empirical validity, these measures were endorsed by NQF.
As a final point we also note that the Standing Committee appears to have applied a higher standard for approval for the EIC measure than for the other two measures approved during the August 3, 2023 Measure Evaluation meeting. The application for the measure “Risk Adjusted Post-Ambulance Provider Triage Emergency Department (ED) Visit Rate” only presented face validity and did not demonstrate construct or empirical validity, but the committee passed the measure on the validity component. The application for the measure “Quality of Care Composite for Implantable Cardioverter-Defibrillator (ICD)/Cardiac Resynchronization Therapy Defibrillator (CRT-D)” demonstrated face validity and for empirical validity showed weak and not significant correlations with outcomes, but the committee passed the measure on the validity component.
We look forward to addressing any further comments or concerns during the Post Comment Meeting.
We wish to provide the following information and context to clarify the committee’s understanding of the validity of our measure Engagement in Community-Based Mental Health Care After a Mental Health Hospitalization (EIC). In our measure application, we presented three different forms of validity – face validity, construct validity, and empirical validity. During the August 3, 2023 Prevention and Population Health Standing Committee Measure Evaluation meeting, the committee recognized that the face validity and construct validity of our measure were sufficient. For empirical validity, we showed three concordance (C) statistics. These statistics indicated the ability of our measure to predict outcomes after the measurement period. The C statistic for engagement in care six months after discharge was predictive, for medication adherence four to six months after discharge was borderline predictive, and for inpatient mental health readmissions in months four to nine after discharge was not predictive. We feel that the committee weighed the not predictive and borderline concordance findings too heavily and did not give enough consideration to the sufficient face validity, construct validity, and partial empirical validity demonstrated by the measure. Our measure is a process measure, and even widely used process measures in behavioral health are not always predictive of outcomes. The claims data we used to test the measure did not allow for adjustment of potential important confounding factors such as lack of housing. These three C statistics were included in our application to be fully transparent about the testing that we had completed during measure development.
National Quality Forum (NQF) endorsed measures of mental health follow-up widely used in HEDIS also did not show concordance with mental health readmissions in our analysis. In our 2018 NYS Medicaid cohort, when looking at the ability to predict an inpatient mental health readmission four to nine months after discharge, the C statistic for one mental health follow-up visit in the 30 days after a mental health discharge (HEDIS Follow-Up After Hospitalization for Mental Illness, FUH) is very similar to the EIC measure (C Statistic for EIC=0.5291, for FUH=0.5229). Additionally, when having at least one community-based mental health visit in month 6 is used as an outcome, FUH is less predictive than the EIC measure (C Statistic for EIC=0.7244, for FUH=0.6847).
It is important to recognize that previously endorsed measures of mental health follow-up also did not demonstrate the ability to predict outcomes in their NQF applications. In the validity section of the NQF application for FUH, NCQA described their face validity process and established construct validity through correlation of the two FUH measure components with each other and with similar measures. For EIC, we found a significant moderate positive correlation with FUH, which agreed with our hypothesis. NCQA did not show concordance statistics or correlations with outcomes for FUH. For validity of Follow-up after Discharge from the Emergency Department for Mental Health or Alcohol or Other Drug Dependence, NCQA described their face validity process and showed an analysis comparing state level performance on the measure with state level rates of MH and SUD inpatient use. They hypothesized that states with better performance on the measure would have lower rates of inpatient use. Their analysis did not show a significant difference in MH or SUD inpatient rates between states in the lower and upper quartiles of the follow-up measure. Despite these failures to establish empirical validity, these measures were endorsed by NQF.
As a final point we also note that the Standing Committee appears to have applied a higher standard for approval for the EIC measure than for the other two measures approved during the August 3, 2023 Measure Evaluation meeting. The application for the measure “Risk Adjusted Post-Ambulance Provider Triage Emergency Department (ED) Visit Rate” only presented face validity and did not demonstrate construct or empirical validity, but the committee passed the measure on the validity component. The application for the measure “Quality of Care Composite for Implantable Cardioverter-Defibrillator (ICD)/Cardiac Resynchronization Therapy Defibrillator (CRT-D)” demonstrated face validity and for empirical validity showed weak and not significant correlations with outcomes, but the committee passed the measure on the validity component.
We look forward to addressing any further comments or concerns during the Post Comment Meeting.