Not endorsed due to no consensus.
This instrument assesses an individual’s experiences in paid employment as well as their experiences searching for paid employment. The target population for this instrument is adults with disabilities who receive HCBS or HCBS-like services. This is a self-contained instrument that can be administered independently of other RTCOM instruments.
The instrument is administered through an in-person or video-conferencing interview where an interviewer guides an individual through a series of questions (i.e., items). The responses for each item are scored on a Likert scale. The available research supports measure administration every four to six months.
The Job Experiences measure is one of two proposed measures in the RTC/OM suite of measures focused on employment. This measure assesses individuals’ perspectives and attitudes on their current place of employment and is comprised of 13 items:
G1. You are happy with your current job.
G2. Your staff helped you get paid work.
G3. Your staff help you keep working.
S1. How long have you worked at your current job?
S3. You work the number of hours you want to work.
S4. Your pay from work helps you buy the things you need.
S5. You get to learn new skills at your job that are valuable to you.
S6. You feel respected by the people you work for.
S7. You feel accepted by your coworkers.
S8. You have the accommodations you need to succeed at your job.
S9. Most of the people you work with have a disability
S10. You are happy with the benefits, such as paid time off, vacation time, and health insurance, you receive from your job.
S11. You have chances for advancement or promotion at your job.
The items “G1” through “G3”, and “S3” through “S8” are scored 0 to 3 on a frequency scale with response options:
“Never/Rarely”
“Sometimes”
“Often”
“Almost Always/Always”
The items “S9” through “S11” are scored 0 to 3 on an agreement scale with response options:
“Strongly Disagree”
“Disagree”
“Agree”
“Strongly Agree”
The item “S1” is scored 0 to 4 with the response options:
“Less than 3 months”
“3-6 months”
“Between 6-12 months”
“1-3 years”
“More than 3 years”
Measure Specs
General Information
Over the last decade, policy shifts have placed increased attention on integrated employment of people with disabilities. As a society, we have moved from sheltered workshops and day activity centers to supported employment, and most recently through the application of approaches to customized employment, to competitive integrated work opportunities for people with even the most intensive support needed, Yet despite a variety of policies, directives, and legislation, people with disabilities continue to experience limited participation in the labor market. (Almalky, 2020; Butterworth, et al, 2015a; Miethlich & Oldenburg, 2019). Recent estimate indicate that in the US, only one in three (34.9%) individuals with disabilities are employed compared to 76% of their counterparts without disabilities, and this disparity appears to be increasing over time (Bonaccio, et al 2020; Houtenville & Ruiz, 2012; Kraus,2017; Lauer & Houtenville, 2017).
Individuals with disabilities face challenges to employment beyond that experienced by their peers without disabilities. Not only are they less likely to be employed when they desire to be so (Andara et al, 2024) but the quality of their employment is significantly lower than their counterparts. Persons with disabilities consistently report lower levels employment quality than their counterparts without disabilities and are nearly twice as likely to report low-quality employment in the form of employment in which they feel trapped) or is insecure and unrewarding . This gap in employment quality is particularly pronounced for those who reported living with both a physical and mental/cognitive condition (Shahadi, et al, 2023). Given the relative stagnation that has occurred in disability employment over the past two decades, the emphasis in many states and among support providers is on raising overall employment rates. This, however, overlooks the problem that not all jobs are of equal quality, and some jobs offer few of the assumed benefits of paid employment Burgard & Lin, 2013; Burroway, 2017). Therefore, although participation in paid employment is an objective with the potential to support social and economic well-being, the long-term goal should be high-quality employment (Martin Ginis, et al, 2020; Tompa et al, 2022).
Specific external barriers to employment for individuals with disabilities that have been identified include a lack of job opportunities (Grant, 2008), insufficient supports to maintain employment (Shier, et al., 2009), and discrimination and stigmatization in the workplace (Winn & Hay, 2009). Individuals with mental health issues face similar difficulties finding and retaining employment, with an added burden of unique stigmatization due to perception of mental health and safety and when persons with disabilities do have jobs they tend to be underemployed (Stuart, 2006).
Beyond the financial benefits of the employment of people with disabilities, access to inclusive high quality employment that pays a living wage has been shown to be a key determinant of individual health and quality of life outcomes (Dean et al., 2018; Emerson et al., 2018; Randall et al., 2022; Robertson et al., 2019; Voermans et al., 2020), as well as being economically and socially beneficial for employers, communities, and broader society (Cimera, 2010; Taylor, et al., 2021).
Existing employment measures developed for use with people with disabilities typically assess a limited number of aspects of one’s work including rates of employment, type of employment, hours worked, and weekly or monthly earnings (Cimera & Burgess, 2011; Grigal, Hart, & Migliore, 2011). When considering the well-being of people with disabilities, earnings and hours worked are key outcome variables. However, this data only inform one about limited aspects of an individual’s employment experience (Nord, et al., 2013; Martin Ginis, et al, 2020; Migliorie et al., 2012; Tompa et al, 2022) and leave out a wide variety of outcomes strongly associated with both the quality of employment and the quality of life one leads. Second, only a few existing employment measures include items that solicit information about how effectively the provider system supports an individual’s employment. Just as important, few currently available instruments have demonstrated adequate reliability and validity and across different disability populations. A third limitation is that current measures do not account for the outcomes experienced by a large segment of the disability population, who although they desire to work, are not currently employed. Providers therefore have little information about the people they serve who are in the process of applying for work but have yet to be able to find employment or are in the process of applying for a new job. The submitted RTCOM employment measure concepts attempt to fill this gap in HCBS measurement.
The rationale for inclusion of a set of employment performance measures was based on:
(a) A systematic review of the disability employment research literature
(b) Use of a variety of approaches, including a participatory planning and decision-making process employed at a national level, to determine the extent to which stakeholders believed performance scores based on various components of the job experiences construct were important to include in the measure under development
(c) Input from several technical expert panels, and
(d) An examination of gaps or limitation inherent in current approaches to measuring outcomes and service quality in this area.
The Quality of Job Experiences and Experiences Seeking Employment measures developed as part of the Research and Training Center on HCBS Outcome Measurement (RTC/OM) were designed to target employment outcomes of people with disabilities at the agency level and be sufficiently sensitive to changes in policy and services that performance scores could be used to document improvement in quality of both services and HCBS recipient outcomes. This level of measurement needs to be more granular than that which is used to focus on compliance and requires measures able to demonstrate reliability, validity, accuracy, and sensitivity at the agency level with the specific groups of HCBS recipient organizations’ serve. The measure is constructed to be person-centered, taking into consideration personal preferences with respect to how one goes about a search for employment, the extent to which they desire support in applying for a job, and for those who work, the level and types of support they need on the job, the quality of interactions they have with co-workers and the general public, the number of hours worked, the extent to which these hours meet their needs, the degree to which person is looking for advancement in their employment, the types/forms of both supervision and job support they prefer as well as from whom, and the specific roles they serve for their employer.
These measure characteristics are necessary in order to ensure the provision of actionable data so that provider agencies can determine the effectiveness with which they are supporting individuals with disabilities to obtain employment, and not just survive in their jobs but thrive in them. Fortunately, the existing research base clearly indicates that people with intellectual and physical disabilities, autism, mental health challenges and TBI can all be matched to job positions in which they can be successful (Gustafsson, et al, 2013; Henry, et al 2019; Houtenville & Kalargyrou, 2012; Khayatzadeh-Mahani, et al 2020 Nevala et al, 2019; Suijkerbuijk, et al 2017) and a variety of approaches to support the employment of people with disabilities are now approaching or have met the criteria for consideration as evidenced-based or promising practices including supported (Wehman, 2023; Wehman et al, 2018) and customized (Inge, et al, 2018; Riesen et al, 2023) approaches to employment. As mentioned previously, however, it should be noted that employment quality is just as important an outcome to consider as the number of people with disabilities who are employed.
Information/data available based on performance scores on our two employment measures have the potential to provide support agencies with a variety of information that can be used to (a) document overall service quality and facilitate policy and/or programmatic changes needed as part of agency quality improvement efforts, (b) identify specific aspects of the Job Experiences subdomain where performance is less than desirable as well as those areas in which the agency is supporting exceptional outcomes, (c) longitudinally track changes that occur in service quality and job experiences, and provide families and persons with disabilities with information they can use to help make informed decisions as to which employment support agencies they desire to provide services to their family member with a disability.
(A complete reference list is provided as a supplemental attachment in section 7.1.)
Employment: Job Experiences
As a society, over the past decade, we have moved from sheltered workshops and day activity centers to supported employment, and most recently through the application of approaches to customized employment, to competitive integrated work opportunities for people with even the most intensive support needs. Yet despite a variety of policies, directives, and legislation, people with disabilities continue to experience limited participation in the labor market. (Almalky, 2020; Andara, et al, 2024; Bonaccio, et al 2020; Miethlich & Oldenburg, 2019). Not only are people with disabilities less likely to be employed when they desire, but the quality of their employment is significantly lower than their counterparts (Andara et al, 2024). Persons with disabilities report consistently lower levels of employment quality than their individuals without disabilities and are nearly twice as likely to report jobs in which they feel trapped, perceive as insecure, and/or unrewarding. This gap in employment quality is particularly pronounced for those who live with both physical and mental/cognitive challenges (Shahadi, et al, 2023).
Given the lack of improvement that has occurred in disability employment over the past two decades, the emphasis in many states and among support providers is on raising overall employment rates. This, however, overlooks the problem that not all jobs are of equal quality, and some jobs offer few of the assumed benefits of paid employment Burgard & Lin, 2013; Burroway, 2017). Therefore, although participation in paid employment should be an objective with the potential to support social and economic well-being, the long-term goal should be high-quality employment (Martin Ginis, et al, 2020; Tompa et al, 2022). Beyond the financial benefits of employment access to inclusive high quality occupations that pay a living wage has been shown to be a key determinant of individual health and quality of life outcomes (Dean et al., 2018; Emerson et al., 2018; Randall et al., 2022; Robertson et al., 2019; Voermans et al., 2020), as well as being economically and socially beneficial for employers, communities, and broader society (Cimera, 2010; Taylor, et al., 2021). Specific external barriers to high quality employment for individuals with disabilities that have been identified include a lack of quality job opportunities (Grant, 2008), insufficient supports to maintain employment (Shier, et al., 2009), employer fears and anxieties over accommodations Erickson, et al., 2014; Schur, et al., 2014), and discrimination and stigmatization in the workplace (Winn & Hay, 2009).
Existing employment measures developed for use with people with disabilities typically assess a limited number of aspects of one’s work including rates of employment, type of employment, hours worked, and weekly or monthly earnings (Cimera & Burgess, 2011; Grigal, Hart, & Migliore, 2011). When considering the well-being of people with disabilities, earnings and hours worked are key outcome variables. However, this data only inform one about limited aspects of an individual’s employment experience (Nord, et al., 2013; Martin Ginis, et al, 2020; Migliorie et al., 2012; Tompa et al, 2022) and leave out a wide variety of outcomes closely associated with both the quality of employment and the quality of life one leads.
Second, only a few existing employment measures include items that solicit information about how effectively the provider system supports the quality of an individual’s employment. Just as important, few currently available instruments have demonstrated adequate reliability and validity and across different disability populations.
The submitted RTCOM employment measure concepts attempt to fill this gap in HCBS measurement and are based on:
(a) A systematic review of the disability employment literature as it relates to a wide variety of aspects of “job quality.”
(b) Use of a variety of processes, employed at a national level, to determine the extent to which stakeholders believed performance scores based on various components of the job experiences construct were important to include in the measure under development that included utilization of a Participatory Planning and Decision-Making (PPDM) process.
(c) Input from several technical expert panels, and
(d) An examination of gaps or limitations inherent in current approaches to measuring outcomes and service quality in this area.
Additional information about each of these processes and how they contributed to the development of the submitted Job Experiences measure can be found in 5105 and in Sections 2.5 of this IDM.
The Quality of Job Experiences measure developed was designed to target employment outcomes of people with disabilities at the agency level and be sufficiently sensitive to changes in policy and services that performance scores could be used to document improvement in quality of both services and HCBS recipient outcomes. The measure is constructed to be person-centered, taking into consideration personal preferences with respect to the number of hours a person works and the extent to which these work hours meet their needs, extent to which a person is looking for advancement in their employment, the types/forms of both supervision and job support they prefer as well as from whom, and the specific roles they serve for their employer. These measure characteristics are necessary in order to ensure the provision of actionable data so that provider agencies can determine the effectiveness with which they are supporting individuals with disabilities to obtain employment, and not just survive in their jobs but thrive in them.
Information/data available based on performance scores on this measure have the potential to provide support agencies with a variety of information that can be used to (a) document overall service quality and facilitate policy and/or programmatic changes needed as part of agency quality improvement efforts, (b) identify specific aspects of the Job Experiences subdomain where performance is less than desirable as well as those areas in which the agency is supporting exceptional outcomes, (c) longitudinally track changes that occur in service quality and job experiences, and provide families and persons with disabilities with information they can use to help make informed decisions as to which employment support agencies they desire to provide services to their family member with a disability.
(A complete reference list is provided as a supplemental attachment in section 7.1.)
N/A
Numerator
The measure focus for the performance measure outcome is the number of individuals who have a composite score within the measure’s possible inter-quartile range (IQR). An individual's composite score is derived by summing the responses to 13 items on the Employment instrument. (See the response to section 1.6 for details on item scoring.) An individual's composite score can range from 0 to 40 where higher scores indicate a participant’s greater overall satisfaction with their current employment. The IQR is 10 to 30 and the numerator is the number of individuals whose composite scores fall in this range.
The numerator is calculated using a tabulation of composite scores from individual respondents. These individuals belong to the relevant population as defined in the denominator for this measure. Other details such as time period for data collection are equivalent to the denominator definitions and will be discussed in section 1.15a.
A composite score for an individual respondent is obtained by administering the Employment instrument to a respondent then calculating a sum score from the responses to 13 unique items on the instrument. This results in a composite score for one individual. See the attached data dictionary (1.13a) for a list of items, response options, and response scoring. An individual composite score (i.e., sum score) for the Job Experiences measure can range between 0 to 40.
Included in the numerator are all individual respondents included in the denominator who have composite scores within the measure-derived inter-quartile range (IQR) for the measure. In other words, composite scores that fall within the middle 50% of possible scores that can be obtained for the measure. For the Job Experiences measure this range is 10 to 30.
It is recommended that these calculations be performed prior to scaling of scores to be on a different, public-facing metric, e.g., T-scores.
Denominator
The target population for this measure is individuals receiving HCBS who are at least 18 years old and have a primary diagnosis of either: intellectual and/or developmental disability, physical disability, and psychiatric disability. Respondents must be able to complete an interview either independently or with assistance (e.g., support staff). Individuals must also currently be employed.
The target population for this measure is individuals receiving HCBS who are at least 18 years old and have a primary diagnosis of either: intellectual and/or developmental disability, physical disability, and psychiatric disability. Respondents must also currently be employed and able to complete an interview either independently or with assistance (e.g., support staff). The available research supports measure administration every four to six months.
Exclusions
None
None
Measure Calculation
The measure score for individual recipients is calculated by the sum of item responses on the instrument. See section 1.6 and the attached Data Codebook in 1.13 for the items and scoring codes on this instrument. Stratification does not modify the calculation of measure scores. Note that the measure score interpretation in section 1.17 is only applicable to this individual measure score.
Scores at the accountable-entity level are calculated using individual measure scores and incorporating the numerator and denominator criteria provided in sections 1.14/1.14a and 1.15/1.15a. For a calendar year, accountable entities will obtain individual measure scores for eligible HCBS recipients that they serve. The numerator is the number of individuals who obtained a score in the measure-derived interquartile range (IQR; see 1.14/1.14a). The denominator is the total number of eligible HCBS recipients that were assessed. The accountable entity score is this numerator/denominator ratio, which will be a proportion.
Each individual HCBS recipient should only be counted once in the numerator and denominator. In other words, an HCBS recipient with more than one assessment within a calendar year will still be only counted once for reporting purposes. For these individuals, use their most recent individual measure score that still falls within the calendar year.
A higher score indicates that there is a greater performance gap on measure outcomes for recipients at a given accountable entity. A higher proportion also suggests that the provider serves a wider population with greater variability in service needs. We have purposefully avoided an accountable entity score interpretation that is based on high (or the highest) individual measure scores as this would provide perverse incentives for those using our instruments. We also do not recommend higher or lower accountable entity scores be interpreted as “better” or “worse”. Rather, accountable entity scores should be accurate and informative.
See data dictionary attachment.
The data dictionary contains item and scoring schemes for questions that ask respondents about their Services Needs. These questions are found on the RTC/OM’s Demographic survey. This survey is appended to the instrument information in section 1.13a Attach Data Dictionary.
The Service Needs questions are used to develop a classification variable of a respondent’s functional disability (see section 5.4.2). The functional disability variable is defined as the overall level of services and supports a respondent is currently receiving. During field testing of the RTC/OM instruments, this collection of items was found to be superior in identifying level of disability compared to items that directly asked participants about their functional difficulties (see question 8 in the Demographic survey, attached in section 1.13a). In essence, most participants indicated low levels in functional difficulties despite varying levels of service needs. This indicates that services are effective: they reduce or eliminate the difficulties a person experiences in various areas of their life. As such, level of current service needs is a better indicator of functional disability.
Scores on the service needs items are summed to create a composite score of service needs for a respondent. Higher composite scores indicate greater service needs. Cut points are used to group individuals into functional disability categories:
- 0 - 10 = Few or no services & supports
- 11 - 20 = Moderate services & supports
- 21+ = Intense services & supports
Our research team is cognizant that polychotomizing a numerical scale into a few discrete categories is not best practice, as it reduces the amount of information (i.e., variance explained) in the outcome of interest. However, we chose this categorical approach to make allowances for those who have less technical expertise and want an easy-to-understand display of the relationship between level of functional disability and service needs.
HCBS Recipient
The Employment Measure is reported by HCBS recipients and administered either in person or through a HIPAA-compliant remote video conferencing platform. Currently, the instrument is available only in English; however, an interpreter can be used during the interview to improve accessibility. Response rates may be improved by offering accommodations that increase accessibility, such as environmental modifications for individuals with disabilities and visual displays of response scales.
There is no minimum sample size that is needed to calculate a performance score at the individual HCBS recipient level. Performance scores for all of the IDMs submitted are based on instrument scores and it is these scores that are reported.
Given that the instrument was developed were intended to be used at the provider level to document service quality and HCBS recipient outcomes, it is unlikely that the measure would be used to represent the quality of services received and outcome experienced by a single individual. As a result, while performance scores could be calculated on the basis of a single individual they would have little meaning in that isolated context. Focus group results and other discussions we have had with HCBS providers indicate that the most likely future use of the measures will be to (a) document the extent to which providers and sites within provider networks meet established benchmarks, (b) determine the extent to which quality improvement efforts have had a significant impact on service quality and HCBS beneficiary outcomes in targeted areas, and (c) potentially be used in the establishment of “report cards” for agencies within and between provider networks that can be used by external funders and families to determine programmatic strengths and areas in need of improvement and to facilitate informed decision-making when people with disabilities and/or their family members are looking to contract for services.
When used in this manner, sample sizes deemed adequate for analysis and interpretation of the submitted measures will depend on a variety of factors including, but not limited to the questions one desires to answer, comfort levels in committing Type I (false positive) and Type II (false negative) errors, the sampling methods used (i.e., probability vs. non-probability sampling) , the target population and its attributes. Attributes of HCBS samples include a wide variety of factors that impact sample size and include client demographic variables such as disability type, intensity of support needs and their variation within the population of interest, geographic location, the size of the population, the effect size one expects and a variety of other features of clients and the environment that could be relevant for answering a provider’s evaluation questions.
The University of Minnesota’s Institute on Community Integration, home of the RTC/OM has long-term history of working with HCBS service providers on a wide variety of evaluation projects that have included the use of measures similar to the ones we are submitting for review. These experiences suggest that if a provider desires to use performance scores aggregated across individuals to determine changes within that provider organization, a representative sample (adjusted for population size) that includes an absolute minimum of 35-50% of HCBS beneficiaries receiving services is needed to draw valid conclusions regarding quality improvement efforts.
Supplemental Attachment
Point of Contact
The measures themselves are placed in the public domain without copyright. To use the measures with institutional approval, organizations must demonstrate evidence of internal capacity to administer the measures, or complete training provided by the measure steward on the use of the measures. The training and technical assistance provided by the measure steward in regards to measures use, scoring, etc. will be copyrighted by the Regents of the University of Minnesota.
Brian Abery
Minneapolis, MN
United States
Brian Abery
Institute on Community Integration
Minneapolis, MN
United States
Importance
Evidence
Existing employment measures developed for use with people with disabilities typically assess a limited number of aspects of one’s work including rates of employment, type of employment, hours worked, and weekly or monthly earnings (Cimera & Burgess, 2011; Grigal, Hart, & Migliore, 2011). When considering the well-being of people with disabilities, earnings and hours worked are key outcome variables. However, this data only inform one about limited aspects of an individual’s employment experience (Nord, et al., 2013; Martin Ginis, et al, 2020; Migliorie et al., 2012; Tompa et al, 2022) and leave out a wide variety of outcomes closely associated with both the quality of employment and the quality of life one leads. Second, only a few existing employment measures include items that solicit information about how effectively the provider system supports an individual’s employment. Just as important, few currently available instruments have demonstrated adequate reliability and validity and across different disability populations. A third limitation is that current measures do not account for the outcomes experienced by a large segment of the disability population, who although they desire to work, are not currently employed. Providers therefore have little information about the people they serve who are in the process of applying for work but have yet to be able to find employment, or are in the process of applying for a new job. The submitted RTCOM employment measure concepts attempt to fill this gap in HCBS measurement.
The rationale for inclusion of a set of employment performance measures was based on:
· A systematic review of the disability employment research literature
· Use of a variety of processes, employed at a national level, to determine the extent to which stakeholders believed performance scores based on various components of the job experiences construct were important to include in the measure under development
· Input from several technical expert panels, and
· An examination of gaps or limitation inherent in current approaches to measuring outcomes and service quality in this area.
Of critical relevance related to the inclusion of employment experience measure is its Importance as defined by stakeholders. Importance was determined via three sources/processes. First, we began by considering all the domains and subdomains identified by the NQF Framework for Home & Community Based Services Outcome Measurement (2016). The framework for HCBS outcome measurement developed by the NQF mentioned employment but only included it as an element of the subdomain of Meaningful Activity which is itself a subdomain of the Community Inclusion domain.
University of Minnesota RTC/OM staff engaged stakeholders on whose lives HCBS has an impact in a series of participatory planning and decision-making groups. Stakeholders included: (a) people with disabilities (100 participants), including individuals with IDD, physical disabilities, traumatic brain injury, psychiatric disabilities, and age-related disabilities; (b) family members of people with the above noted disabilities (84 participants); (c) representatives of HCBS provider agencies (89 participants), and (d) state level HCBS program administrators/policy makers (47 participants) for a total of 320 participants in 58 small groups conducted nationally. Stakeholders took part in a participatory planning and decision-making (PPDM) process in which participants provided importance weights for each domain and subdomain of the NQF framework on a scale from 0-100 based on their perceived importance in determining the HCBS outcomes and quality. As part of the PPDM process, stakeholders are first asked to examine the basic domains and subdomains for the framework or theory under consideration and add or rearrange its components in order to refine the model. The large majority (83%) of PPDM groups across all stakeholder types reached consensus that employment was an outcome of sufficient importance as to assume the status of its own domain. Among stakeholder groups composed of with disabilities as well as groups representing families, over 90% of participants (people with disabilities = 94%; families = 91%) supporting this refinement of the NQF model with somewhat lower percentages of support among providers and program administrators. Because the existing NQF framework for HCBS outcome measurement did not include employment as a domain or subdomain but only included it as an element of the subdomain of Meaningful Activity no PPDM importance weightings of this domain can be provided.
As a third step in the process of developing a rationale for the inclusion of this measure, we solicited the input of measurement and content experts in disability-related fields through the use of technical expert panels that included people with disabilities. Two groups of experts were asked to rate the importance of all subdomains in the NQF framework in terms of their relevance, importance, and accuracy. Based on the 1-4 point Likert-type rating scales used for these measures, mean Job Experiences TEP scores for people with disabilities who were employed were as follows: Relevancy 3.50/4.00; Importance 3.53/4.00; and Accuracy 3.12/4.0. Mean scores for Experiences Seeking Employment (i.e., those unemployed but currently job seeking) were Relevancy 3.54/4.00; Importance 3.55/4.00; and Accuracy 3.00/4.0.
The Job Experiences and Experiences Seeking Employment measures developed as part of the Research and Training Center on HCBS Outcome Measurement (RTC/OM) were designed to target employment outcomes of people with disabilities at the agency level and be sufficiently sensitive to changes in policy and services that performance scores could be used to document improvement in quality of both services and HCBS recipient outcomes. This level of measurement needs to be more granular than that which is used to focus on compliance and requires measures able to demonstrate reliability, validity, accuracy and sensitivity at the agency level with the specific groups of HCBS recipient organizations’ serve. The measure is constructed to be person-centered, taking into consideration personal preferences with respect to personal preferences with respect to how one goes about a search for employment, the extent to which they desire support, the level and types of support they need, and for those currently employed he number of hours worked, the extent to which these hours meet their needs, the degree to which person is looking for advancement in their employment, the types/forms of both supervision and job support they prefer as well as from whom, and the specific roles they serve for their employer.
These measure characteristics are necessary in order to ensure the provision of actionable data so that provider agencies can determine the effectiveness with which they are supporting individuals with disabilities to obtain employment, and not just survive in their jobs but thrive in them. Fortunately, the existing research base clearly indicates that people with intellectual and physical disabilities, autism, mental health challenges and TBI can all be matched to job positions in which they can be successful (Gustafsson, et al, 2013; Henry, et al 2019; Houtenville & Kalargyrou, 2012; Khayatzadeh-Mahani, et al 2020 Nevala et al, 2019; Suijkerbuijk, et al 2017). A variety of approaches to support the employment of people with disabilities are now approaching or have met the criteria for consideration as evidenced-based or promising practices including supported (Wehman, 2023; Wehman et al, 2018) and customized (Inge, et al, 2018; Riesen et al, 2023) approaches to supporting employment success. As mentioned previously, however, it should be noted that employment quality is just as important an outcome to consider as the number of people with disabilities who are employed.
Information/data available based on performance scores on our two employment measures have the potential to provide support agencies with a variety of information that can be used to (a) document overall service quality and facilitate policy and/or programmatic changes needed as part of agency quality improvement efforts, (b) identify specific aspects of the Job Experiences subdomain where performance is less than desirable as well as those areas in which the agency is supporting exceptional outcomes, (c) longitudinally track changes that occur in service quality and job experiences, and provide families and persons with disabilities with information they can use to help make informed decisions as to which employment support agencies they desire to provide services to their family member with a disability.
(A complete reference list is provided as a supplemental attachment in section 7.1.)
General Importance
Importance is defined as the relevance of measures to the lives of people with disabilities who receive HCBS and the potential of these measures to reveal important aspects of quality of life experienced by individuals across disability groups.
Importance was determined via three sources/processes. First, we began by considering all the domains and subdomains identified by the National Quality Forum’s (NQF) Framework for Home & Community Based Services Outcome Measurement (2016). The framework for HCBS outcome measurement was developed by a panel of NQF experts with the goal of identifying key areas to measure in order to be able to track the effectiveness of HCBS services. The final recommendation of the panel included 11 domains with 2-7 subdomains within each domain.
Next, we engaged stakeholders on whose lives HCBS has an impact in a series of feedback and planning groups. Stakeholders included: (1) Persons with disabilities (100 participants), including individuals with intellectual disability and developmental disabilities, physical disabilities, traumatic brain injury, mental illness, and age-related disabilities; (2) family members (84 participants); (3) providers (89 participants), and (4) program administrators/policy makers (47 participants) for a total of 320 participants in 58 small groups conducted nationally. Stakeholders took part in a participatory planning and decision-making (PPDM) process in which participants weighted each domain and subdomain of the NQF framework on a scale from 0-100 based on their perceived importance in determining the HCBS outcomes and quality.
As a third step in the process, we solicited the input of measurement experts in disability-related fields. Two groups of experts rated all subdomains, including the new subdomains of employment and transportation, in terms of feasibility, usability, and importance. These expert ratings were used in conjunction with stakeholder weighting from PPDM groups to narrow our development process to nine NQF subdomains on which to initially focus our work.
Specific Importance
Research has clearly indicated that there are different aspects of having a job a person with a disability values and finds important in their lives, including positive experiences and relationships, support staff and person-centered supports (Hastings, 2010; van den Meer et al., 2018). Measuring these aspects of job experiences is critical for people with disabilities to be able to set goal to improve these experiences. It is also important for service providers and employers to understand how employment experiences are impacting quality of life of the people they support and what can be done to keep people with disabilities employed based on the measure.
During the iterative development process of the measures, our stakeholders clearly indicated and literature supports this distinction that the employment tool needed to separated into two distinct measures: Experiences Seeking Employment and Job Experiences. In this document we are focusing on the latter.
(A complete reference list is provided as a supplemental attachment in section 7.1.)
Measure Impact
Our intent for developing this measure of Job Experiences was to assess the outcomes of experiences and success and satisfaction with employment (Meltzer et al., 2020); of reduction in employment service needs and in the use of day services (Drake et al., 2009); and of greater efficacy, timeliness, and quality of services to support employment seeking service delivery (Cheng et al., 2018).
(A complete reference list is provided as a supplemental attachment in section 7.1.)
The capacity to monitor the extent to which the employment of people with disabilities meets their needs and reflects their personal goals and dreams is critical if we are to understand the extent to which community-based services are doing what they are intended to do. At one level, performance measures are needed to ensure compliance with federal and state regulations governing HCBS. Performance measures, however, are also needed that are person-centered, longitudinal, and capable of assessing outcomes associated with various aspects of employment as well as the quality of support service recipients receive. These measures need to be sufficiently sensitive to change so that the impact of policy, funding, and programmatic changes on the outcomes people experience can be determined over time. They would also preferably have the capacity to be used with different disability populations who receive community supports.
As a result of the variety of HCBS waiver programs and diversity of users, and numerous contextual factors, the measurement of the quality of job experiences supports that the recipients receive and the outcomes these individuals experience is far from a straightforward process. A nuanced approach needs to be taken that is responsive to a wide variety of potential influences on quality outcomes. This process needs to be decidedly different than that currently used in medical/healthcare contexts due to the dissimilarities in the constructs measured. Unlike many outcome measures related to health, outcomes associated with the quality of a person’s job experiences are both more complex and difficult to assess. A second set of critical contextual factors for which one needs to account are the policies and regulations under which HCBS is implemented which vary significantly between states. One must also consider additional external factors such as the recent pandemic, a time during which many people with disabilities lost their jobs.
In order to be confident that performance measures associated with HCBS adequately assess both the quality of services and the employment experiences of people with disabilities, data are needed with respect to the reliability, validity, and sensitivity to change of those measures intended for use. Indicators of quality of outcomes and the existence of unmet support needs as directly perceived by service recipients, must be considered paramount when developing, administering, and interpreting results based on these tools.
Over the past decade, CMS has championed the development and maintenance of the Home and Community-Based Services (HCBS) Quality Measure Set (QMS) - a set of quality measures for Medicaid-funded HCBS. The Human Services Research Institute’s National Core Indicators, the Center for Quality Leadership’s Personal Outcome Measures, and the CAHPS Home and Community-Based Services Survey have all been developed and are being used by both states and support providers as means through which to demonstrate compliance with CMS regulations, and in the case of the CQL-POM for quality assurance purposes. Each of these approaches has its strengths. However, all have significant limitations that lead to the need to develop new measures that address these shortcomings.
The QMS, developed through the support of CMS, focuses on the compliance level and is decidedly medical in its orientation. As such, “performance measures” are most often conceptualized as single items. While this approach might be appropriate to measure discrete healthcare outcomes, most psychometricians would argue that it is inadvisable to use when attempting to measure latent variables or constructs that cannot be directly observed, but instead, must be inferred from their relationship with other, directly measurable variables. The Quality of Job Experience construct clearly meets the criteria for being considered a latent variable and therefore should not be measured with single-item measures that all too often lead to inaccurate representation of a latent construct and limit the ability to assess measurement error. Using measures composed of multiple items provides more robust evidence of construct structure, allows for estimation of measurement error, and enhances the overall validity and reliability of
the measure.
The National Core Indicators (NCI/NCI-AD - https://www.nationalcoreindicators.org) was developed and validated as a state-level compliance measure and does an excellent job when used at that level. It is not, however, intended to be used at the provider or individual level for quality improvement, service plan development, and/or outcome assessment. In addition, although the NCI includes indicators in a variety of areas, it is intended to be administered (and was validated) at the instrument level as opposed to on an indicator-by indicator basis. Users are therefore required to administer items related to all indicators as opposed to only those in which there is a specific interest. It should also be noted that although NCI and NCI-AD have been used with populations beyond those for which they were intended (i.e., people with IDD, physical, and age-related disabilities) these tools have only been validated for use with the limited disability groups noted.
CQL’s Personal Outcome Measures (CQL, 2017) is one of the better developed and validated HCBS Outcome tools and part of a commercially available system of assessment and quality improvement. It has been validated with a much wider variety of people with disabilities than the NCI and possesses good psychometric properties. However, the instrument is time consuming with respect to administration (715 items; 12) limiting its feasibility for many providers. In addition, the CQL-POM, as part of a quality improvement package, is proprietary and not inexpensive.
A third approach to outcome assessment in the human services field that has been championed by CMS is the HCBS CAHPS Survey. The CAHPS, unfortunately, currently has limited data available with respect to its validity or reliability. Internal consistency reliabilities for seventeen of its nineteen measures fail to meet even the most basic criteria for psychometric acceptability, there are serious questions about the representativeness of the sample used for the field study as well as the evidence presented to support validity, and in a number of indicator areas, there appears to be a ceiling effect with the overwhelming majority of respondents indicating the highest possible level of service quality or personal outcomes (Nyce, et al, 2020).
In addition to the individual shortcomings of the most widely used HCBS outcome measures, there are additional limitations that cut across these instruments as well as other HCBS outcome assessment tools that contribute to the need for development of new measures of job experiences. The first of these entails the small percentage of items included in HCBS outcome measurement instruments that meet the criteria for person centeredness. Recent decades have seen a growing focus on providing HCBS in a person-centered manner thereby supporting outcomes that are both important for and to the person. One should not be assessing the quality of employment merely by counting the number of hours one works, the shift they work, or their wages. The personal preferences of employees as well as contextual factors must be taken into consideration.
A second shortcoming that cuts across existing HCBS employment outcome measures is the lack of evidence that they are sufficiently sensitive to change over time that they can be used in a longitudinal manner. Some developers, such as HSRI (NCI-ASC/NCI-AD) explicitly state that their measures are not intended to be used longitudinally. Others (e.g., CQL, CAHPS) have yet to provide sufficient evidence that, when used in a longitudinal manner, their measures are sufficiently sensitive to change that they can be used as evidence of the effectiveness/efficacy of quality improvement efforts, changes that take place in a HCBS recipient’s life, disability policy or funding or as part of value-based payment systems.
A third reason to consider the development of new quality of job experience outcome measures emanates from the resources needed to administer measures at a time when the human services field is experiencing serious workforce shortages. All of the tools mentioned above are intended to be administered in their entirety as full instruments. They are neither modular in format allowing for administration focused on only one or a few indicators, nor tiered and able to provide both a quick general overview of indicators as well as a more in-depth assessment needed for having utility at the provider level.
A final overarching rationale for considering the need and development of new Quality of Job Experience measures is that the best developed and most well-researched measures that are currently available in the field are proprietary and part of measurement systems. States, as well as large providers typically have the funding to pay for the use of these tools. Provider agencies, especially those focused on employment, which more often than not are of a small variety, however, often do not. As a result there is a need for measures at the HCBS employment provider level that are (a) able to be used a little to no cost, (b) person-centered, (c) of a composite nature with the ability to assess latent constructs, (d) based on recent theory and research pertaining to the outcome domains and subdomains assessed, (e) easily scored and interpreted, and (f) sufficiently sensitive to change over time so that they can be used longitudinally.
(A complete reference list is provided as a supplemental attachment in section 7.1.)
Stakeholders, particularly those who have disabilities and their supporters, should be at the heart of measurement development. In order to ensure that HCBS outcome measures are of high quality, the measure development process must include input from stakeholders and most importantly the intended population with which the measures will be used. Furthermore, we contend that the unique challenges associated with measuring employment outcomes among members of diverse populations of HCBS recipients requires strong stakeholder involvement throughout all stages of development. This process has been affirmed by NIDILRR, ACL, and the Centers for Medicare and Medicaid Services (CMS, 2019). Using a sound HCBS outcome measurement framework that has evidence of content validity provided by stakeholders including people with disabilities; putting all measures developed through multiple expert panel reviews; undertaking cognitive testing with people with a variety of disabilities are necessary strategies or processes in which one must engage to ensure for quality measure development.
The target populations for the submitted measures of employment (Quality of Job Experience and Experiences Seeking Employment) include people with intellectual and developmental disabilities, psychiatric disabilities, physical disabilities, and TBI/ABI as well as age-related disabilities. Although the large majority of people with such disabilities possess the capacity to articulate their thoughts and feelings about the outcomes in question and their importance, it must be recognized that some do not. The intensity of their support needs may be such that they experience difficulty understanding questions and articulating their thoughts and feelings. Not only are such individuals at elevated risk for experiencing poor services and outcomes but most HCBS performance measure programs are not set up to reflect their experiences. The RTC/OM development team therefore instituted a process in which not only HCBS beneficiaries, but other relevant stakeholders, provided input into the measure development process. These consisted of family members of people with the above noted disabilities (who are often direct caregivers on either a part or full-time basis), representatives of support provider agencies, and HCBS program administrators who are responsible for using performance measure data to improve both services and outcomes. At multiple steps along the measure development process, these individuals were consulted and their input incorporated into the measure development process. All of the measures submitted for review were based on:
- A systematic review of the disability employment literature
- The results of a national content validation study of the National Quality Forum’s HCBS Outcome Measurement Framework using a Participatory Planning and Decision-Making (PPDM) process.
- Utilization of multiple Technical Assistance Panels (TEPS) that included people with disabilities as well as members of other stakeholder groups
- Input from an RTC/OM Center Advisory Committee composed of people with a variety of disabilities in addition to other stakeholder groups, and
- Extensive cognitive testing of measure items and response options, and
- Vetting of measures by representatives of provider agencies
A Systematic Review of Employment of People with Disabilities Literature. First, we began by considering the current literature which clearly indicates that people with disabilities desire to have meaningful employment and that work enriches their lives in a number of ways (Dean et al., 2018; Emerson et al., 2018; Randall et al., 2022; Robertson et al., 2019; Voermans et al., 2020), as well as being economically and socially beneficial for employers, communities, and broader society (Cimera, 2010; Taylor, et al., 2021).
PPDM Process. Of critical relevance related to ensuring that HCBS recipients for whom the measures under development were intended viewed them as important, we then examined the domains and subdomains identified by the NQF Framework for Home & Community Based Services Outcome Measurement (2016). The initial framework for HCBS outcome measurement developed by the NQF covered 11 domains as well as 2-7 subdomains within each domain. However, employment was not included in the original framework at either a domain or subdomain level. As a second step in determining importance, relevance, and meaningfulness of including employment measures, RTC/OM staff engaged stakeholders from 29 states on whose lives HCBS has an impact in a series of participatory planning and decision-making (PPDM) groups. Stakeholders included: (a) people representing with disabilities (100 participants), including individuals with IDD, physical disabilities, traumatic brain injury, psychiatric disabilities, and age-related disabilities; (b) family members of people with the above noted disabilities (84 participants); (c) representatives of HCBS provider agencies (89 participants), and (d) state level HCBS program administrators/policy makers (47 participants) for a total of 320 participants who took part in 58 small (4-6 person) PPDM groups.
The PPDM process initially included meeting with homogeneous (with respect to disability and stakeholder type) stakeholder groups and providing them with an opportunity to evaluate the original NQF framework. They were provided with the opportunity to add to it, remove domains and/or subdomains they believed were not important, and then stipulate which personal outcomes and service characteristics were most important to measure. Following stakeholders reaching consensus with respect to the domains and subdomains of the original framework they wanted to add or remove, members of each group took part in a process in which they first independently assigned importance weights for each domain and subdomain of the original NQF framework on a scale from 0-100 (or 0-10 for persons with cognitive challenges) based on their perceived importance in determining the HCBS outcomes and service quality. As part of the PPDM process, stakeholders then discussed their weightings first at the subdomain level and later at the domain level examining why people in their group assigned the importance weightings they did. Stakeholders were then given the opportunity to assign a second set of importance weightings taking into consideration what they had heard during their discussion.
Overall, results from PPDM groups indicated a high degree of stakeholder support for inclusion of a separate Employment Domain in the NQF HCBS Outcome Measurement Framework as well as support for the development of measures for both persons with disabilities currently employed as well as those seeking work but currently unemployed. Across all four stakeholder groups, measurement of employment outcomes and both access to and the quality of vocational supports was viewed as of high importance (People with Disabilities = 94% agreement; Family members = 96% agreement; Providers = 83% agreement, Program administrators = 91% agreement).
Technical Expert Panels. As a second step in the process of developing measures, we solicited the input of people with lived experience with disability (N = 9) as well as measurement and content experts in disability-related fields (N = 12) for a series of technical expert panels (TEPs). Four TEP groups were formed and initially asked to rate the importance of all subdomains in the NQF framework that received the highest importance weightings across stakeholder groups on a 1-5 point Likert-type scale with respect to their feasibility, usability/utility, and importance, as well as provide an overall score for the subdomain. Across stakeholder groups, the Employment domain received the following weigthing with respect to feasibility (Mean = 4.6/5.0), usability/utility (Mean = 4.5/5.0), and importance (Mean = 3.96/5.0), as well as provide an overall score for the subdomain (Mean = 4.3/5.0). This information, in combination with PPDM results, the systematic review of the literature, and analysis of existing HCBS outcome measures indicated both the meaningfulness of the employment construct and perceptions among stakeholders that it is important to measure.
At follow-up TEP meetings following the initial development of items for those subdomains that were selected to be a focus of RTC/OM measures related to employment (Job Experiences & Experiences Seeking Work), TEPs engaged in a similar process using 1-4 point Likert-type rating scales to rate the relevance, importance, accessibility/understandability, and accuracy of each item developed for the employment measures under construction. Mean scores for Quality of Job Experiences for employed people with disabilities were as follows: Relevance (Mean = 3.49/4.00), Importance (Mean = 3.49/4.00); Accessibility/Understandability (Mean = 3.37/4.00); and Accuracy (Mean = 3.12/4.00). Mean TEP scores for Experiences Seeking Work were Relevance (Mean = 3.53/4.00), Importance (Mean = 3.55/4.00); Accessibility/Understandability (Mean = 3.20/4.00); and Accuracy (Mean = 3.12/4.00).
RTC/OM National Advisory Group. In conjunction with the results of PPDM groups and TEPs, ongoing input was also solicited from a national advisory group of HCBS stakeholders composed of 14 individuals including 5 with lived experience with disability. The RTC/OM Center Advisory Committee provided valuable feedback not only with respect to the item content of the measures under development but measure administrator training content and the medium (live versus virtual using HIPAA compliant version of Zoom Meeting) through which interviews would be conducted and the approach to cognitive testing that would be used after item refinement. This group not only provided critical qualitative information stressing the importance of employment measures but stressed the need to also included recommendations for the development of measures associated with employment including people with disabilities having access to affordable and safe transportation, and the availability of employment within one’s community of choice.
Cognitive Testing. The terminology used to describe services or supports can vary based upon disability group membership, the settings in which supports are received, and/or the services and supports themselves. The terms work, job, and employment, for example, have positive, neutral, and sometimes even different meaning, when used in different contexts. For example, is a person with a disability living on their family’s farm and working forty hours/week for no pay but for food, shelter, and clothing “employed,” “a volunteer,” or working but not employed.” Thoroughly testing with people with disabilities, the language and terminology used in measure concepts, and how the measure would typically be implemented (e.g., interview, survey, etc.) is therefore essential to ensure items are universally understood across HCBS recipients.
Cognitive testing (CT) focused on obtaining direct input from respondents to verify their interpretation of items and the words of which they are composed to ensure that these matched the developer’s intent (Ericsson & Simon, 1980; Willis, 2005; Willis, et al., 1991) and used a cognitive testing strategy referred to as the “Think Aloud Method” to address the core cognitive components of item responding: comprehension, retrieval of information needed to answer the item, making a judgment, and reporting a response (Tourangeu, 1984; 2018). This approach provided yet another way to involve people with disabilities in the measure development process.
Focus Groups with Potential Measure Users. It is essential that people with disabilities have multiple opportunities to provide input into the measure development process and resulting measures. The employment measures under development, however, also needed to undergo vetting by representatives of employment provider and vocational rehabilitation agencies. After final performance measure refinement based on the results of piloting and field-testing the final measures under submission were presented to two separate groups of potential users utilizing a focus group format. Participants were recruited from a large human service organization in Minnesota that provides residential, home health, and employment services, and a statewide network of human services providers located in Michigan that provide residential, and employment supports (Total N = 23). Measures were initially shared with participants several days prior to scheduled focus groups. Groups were initiated with RTC/OM providing background information and answering questions about the measures themselves, their administration, analysis, and use. After facilitators were assured that participants’ questions had been answered group discussion focused on, (a) the importance of the measures, (b) their overall quality and comprehensiveness, (c) the feasibility of provider agencies using the measures, and (d) the utility of the performance measures developed and how providers could foresee using them. {Overall, the employment measures developed were viewed as important, high in quality, feasible to administer and capable of providing actionable information. The sole negative response to the measures was concern over with whom this information would be shared if the measures were employed on a widescale basis.
(A complete reference list is provided as a supplemental attachment in section 7.1.)
Of critical relevance related to the inclusion of Quality of Job Experiences measure is its Meaningfulness to the Target Population defined by stakeholders including people with disabilities themselves. Importance was determined via multiple sources/processes.
A Systematic Review of Employment of People with Disabilities Literature. First, we began by considering the current literature which clearly indicates that people with disabilities desire to have meaningful employment and that work enriches their lives in a number of ways (Dean et al., 2018; Emerson et al., 2018; Randall et al., 2022; Robertson et al., 2019; Voermans et al., 2020), as well as being economically and socially beneficial for employers, communities, and broader society (Cimera, 2010; Taylor, et al., 2021).
Participatory Planning and Decision-Making Groups. A second approach followed to determine importance include a national study designed to validate the domains and subdomains of the NQF Framework for HCBS Outcome Measurement developed by the NQF (2016) which included employment but only as an element of the subdomain of Meaningful Activity which is itself a subdomain of the Community Inclusion domain. University of Minnesota RTC/OM staff engaged stakeholders on whose lives HCBS has an impact in a series of PPDM groups. The large majority (83%) of groups across all stakeholder types reached consensus that quality of employment experience was a critical but often overlooked component of work and was of sufficient importance as to assume the status of its own subdomain. Among stakeholder groups composed of people with disabilities as well as groups representing families, over 90% of participants (people with disabilities = 94%; families = 91%) supported this refinement of the NQF model with somewhat lower percentages of support among providers and program administrators.
Technical Expert Panels As a third step in the process of developing a rationale for the inclusion of this measure, we solicited the input of people with disability, family members, and measurement and content experts in disability-related fields through the use of technical expert panels. TEPs groups were asked to rate the importance of all subdomains in the NQF framework in terms of their relevance, importance, and accuracy. Based on the 1–4-point Likert-type rating scales used for these measures, mean Quality of Job Experiences TEP scores for employed people with disabilities were as follows: Relevancy 3.49/4.00; Importance 3.49/4.00; and Accuracy 3.12/4.0.
RTC/OM National Advisory Group. In conjunction with the results of PPDM groups and TEPs, ongoing input was also solicited from a national advisory group of HCBS stakeholders. The RTC/OM National Advisory Committee confirmed the importance of the need for a person-centered quality of employment performance measure and provided valuable feedback not only with respect to item content of the measure but suggested the addition of a separate measure focused on retirement planning.
Cognitive Testing. The Quality of Job Experiences measure was tested with 23 people with disabilities who did not have a current job but were interested in one using the “Think Aloud Method” to address the core cognitive components of item responding as included in the Cognitive Aspects of Survey Methodology (CASM) model the language and terminology used in measure concepts, their completeness, and the extent to which response options provided a person to accurately indicate their perspective on an item.
Focus Groups with Potential Measure Users. After final performance measure refinement based on the results of piloting and field-testing the final measures under submission were presented to two separate groups of potential users utilizing a focus group format. Participants were recruited from a large human service organization in Minnesota, and a statewide network of human services providers located in Michigan (Total N = 23). Group discussion focused on, (a) the importance of the measure, (b) its overall quality and comprehensiveness, (c) feasibility of measure use, and (d) measure utility.
(A complete reference list is provided as a supplemental attachment in section 7.1.)
Performance Gap
Measures are being submitted for initial endorsement.
Care Gaps
Closing Care Gaps
This domain is optional for Spring 2025
Feasibility
Feasibility
The composite scores (i.e., measure scores, see section 1.18) from both the Job Experiences and the Experiences Seeking Employment measure will be available through electronic platforms, such as Qualtrics or incorporated into existing electronic systems providers already use. It is important to understand that HCBS data are by definition not medical data and therefore require different electronic systems from the usual hospital managed system. The HCBS field is not as centralized at the provider level and their electronic systems are developing.
That said, in addition to working with the systems providers may already been using, we are in the process of developing an electronic system at the University of Minnesota’s Institute on Community Integration that houses this initiative. This system will offer providers an opportunity to utilize our electronic system and technical assistance to use the system to house their scores for a negotiated fee.
Administering the measure as an interview requires approximately 10 minutes per Home and Community-Based Services (HCBS) beneficiary. The primary human resource burden includes the time of a trained interviewer to administer the measure and a data entry specialist or the interviewer themselves to enter the responses into a secure data system. Minimal additional personnel time is needed for data validation, which may involve reviewing and cleaning inconsistent or incomplete data, and follow-up with interviewers to clarify discrepancies. While these tasks include some investment of staff time, they are essential to ensuring data quality and integrity. To support accurate and valid data collection, organizations may also use supplementary resources such as interviewer training sessions, standardized data entry protocols, and processes to conduct periodic checks of the data. Despite the staff requirements, the burden on respondents was minimal. During testing, the vast majority of HCBS recipients reported a positive experience with the interviews. Many beneficiaries expressed appreciation for the opportunity to share their perspectives and discuss their services and outcomes with interviewers.
As part of one of the nation’s prominent research universities with a large academic health sciences program, the Institute on Community Integration and RTC/OM have access to a wide variety of servers/data storage systems that meet or exceed all HIPAA security requirements and can be used to ensure the privacy/confidentiality of personally identified information. All clinical and human subjects’ data collected as part of RTC/OM performance measures will be secured with a University of Minnesota-approved resource at all times for the full extent of their life. This is true even when data has been de-identified. University approved methods of storing, clinical and human subjects’ data that will be used by the implementer (RTC/OM) include servers exclusively devoted to and supported by the University’s Health Sciences Technology (HST) group. HST supports University departments that need to store Private Highly Restricted data. The HST Operations and Infrastructure team supports and manages applications and infrastructure to meet University standards regarding HIPAA compliance. HST will work with RTC/OM staff to identify onboarding and maintenance requirements necessary to comply with UIS standards for PHR systems and data. Operating systems include professionally managed RHEL (Linux) and Windows Server environments. HST offers multiple storage options for data, documents, and folders to meet a variety of clinical, research, and storage needs. Network storage (SMB/NFS) is available on the Twin Cities campus for Healthcare Component (HCC) departments. The University’s Health Information Privacy & Compliance Office (HIPCO) and its Institutional Review Board RB strongly encourage the use of limited datasets to maintain confidentiality when some identifiers are needed and this recommendation will be followed. A Limited Data Set is a dataset that contains a limited set of indirect identifiers, and it's only used within the University's Health Care Components or under a Data Use Agreement (DUA). This will allow implementers (RTC/OM) to access data with limited privacy risks while still being able to provide critical service quality and outcome data to RTC/OM measure users in a way that has a high degree of utility. Key aspects of creating a Limited Data Set will entail:
(a) Removing Direct Identifiers: All direct identifiers including names, street addresses, and telephone numbers will be removed.
(b) Including Limited Indirect Identifiers: Certain indirect identifiers, such as dates (e.g., date of birth, death, admission), and geographic information (city, state, zip code) will be included
(c) Creating Data Use Agreements (DUAs): When PHI in a Limited Data Set is shared with a third party, a Data Use Agreement (DUA) will first be established and approved by the UMN’s IRB, Sponsored Projects Administration and Office of General Counsel. This agreement will outline the permissible uses of the data and ensures compliance with HIPAA.
(d) Review and Approval Processes: The process for creating and sharing the Limited Data Set will include review by HIPCO, the IRB, and the Sponsored Projects Administration (SPA).
The final measure is a result of multiple feasibility assessments. First, we conducted Participatory Planning and Decision Making (PPDM) groups with people with disabilities, family members, and staff to weigh importance of the NQF domains and subdomains. This process assisted our team with the selection of measure concepts to prioritize for developing within HCBS. Second, the items and their response options from which the measure was composed was reviewed by a technical expert panel to test whether the measure reflected the intent behind the measure concept. Third, the measure underwent cognitive testing during which the items and their response options from which the measure was composed were tested by people with intellectual and developmental disabilities, aging needs, TBI, mental health needs, and physical disabilities. We wanted to assure that the measure was reflective of its measure concept as understood by people with different challenges. Fourth, the measure underwent pilot testing in two states (MN, PA). Feasibility was one of the main objectives of the pilot study during which we were able to identify items and response options that were either not providing us with information that reflected the measure concept accurately or those that did not contribute meaningfully to the measure. Based on all these stages to the feasibility assessment, we used an iterative process to refine the measure. The final version was used for psychometric testing in the RTC/OM National Field Study.
Proprietary Information
The measure is not proprietary, but the training and technical assistance will have an associated cost.
If organizations and users have the requisite knowledge (e.g., a Quality Assurance staff person with knowledge of HCBS processes) to use and score the measure then proprietary training would not be necessary.
Training that is available (online or in-person), focuses on providing potential users with background in order to do high quality measurement. More specific training on the measures we have developed, which is focused on appropriate use, administration and interviewing techniques, strategies for data analysis, and interpretation.
Scientific Acceptability
Testing Data
The data presented in this submission were collected on a rolling basis during a multi-year field study of the RTC/OM measures between Spring 2021 and Spring 2024. This was a longitudinal data collection effort with three waves of data collection for each participant. However, only results from the first wave of data collection for each participant are presented unless otherwise noted. The first point of data collection for participants occurred between May 2021 and February 2024.
Field not required Spring 2025
To mitigate autocorrelation and other statistical artifacts during data analysis only the first wave of data collection is used for data analysis, with one exception. The one exception is the test-retest analysis which could have been collected during any wave. Test-retest responses were only collected once per participant.
Providers in the sample were recruited through a national directory of HCBS providers maintained by Medicaid.gov, as well as through networks of known HCBS providers and contacts recruited by University Centers for Excellence in Developmental Disabilities and other organizations contracted to support recruitment and data collection in several states (e.g., Utah, Pennsylvania, Florida, Georgia, Kansas). Additional HCBS providers were referred by participants who responded to national recruitment efforts (e.g., website, newsletters). Providers were not recruited for the study in every state, attempts to expand the representation of the sample were made in every state. This resulted in 67 organizations formally participating in the study across the states of Minnesota, Kansas, Florida, New Jersey, Pennsylvania, Massachusetts, Georgia, Arizona, Kentucky, Iowa, California, and New York. The size of participating organizations, in terms of the number of beneficiaries served, ranged from 10 or fewer to several hundred, with a variety of sizes in between. In Kansas, three large Managed Care Organizations also participated, as HCBS in that state are administered through them. The types of HCBS provided included residential services, in-home supports, home health or skilled nursing, employment services, community access, financial assistance, transportation, and more.
HCBS beneficiaries were recruited either directly by participating provider organizations through direct outreach or recruitment materials (e.g., flyers, videos), or through national-level postings and newsletters inviting participation. Each participant was screened to verify eligibility, including age and receipt of HCBS or HCBS-like services. All participants in the study who expressed interest in participating and met inclusionary criteria (age, currently receiving HCBS or HCBS-like services, able to provide consent/assent). Participant ability to understand the measure questions was first evaluated with the University of California, San Diego Brief Assessment of Capacity to Consent (UBACC). Capacity was also closely monitored by interviewers and if significant concerns about the validity of responses were raised, the participant’s data were excluded.
Participants reported their primary disability as Intellectual or Developmental Disability (181, 61.4%), Physical Disability (59, 20%), Traumatic Brain Injury (24, 8.1%), Psychiatric Disability (17, 5.8%), Age-related Disability (4, 1.4%), or Other (10, 3.4%). The age range of participants was between 19 and 76 years old. Participants between ages 18-34 made up 37.9% of the sample, 35-54 were 44.7% of the sample, and the remaining 17.4% of the sample were 55 or older. 160 participants (54.2%) identified as male, 133 participants (45.1%) identified as female, and 2 participants (<1%) identified as “other”. Participants identified their race as White (188, 63.7%), Black or African-American (66, 22.4%), Hispanic/Latino (12, 4.1%), Asian (2, <1%), or “Other race not listed” (5, 1.7%). No participants identified as solely American Indian or Alaska Native. Approximately 7.5% of participants identified with more than one race.
Reliability
Reliability testing was conducted for each individual IDM.
Internal consistency reliability and test-retest reliability methods were used to assess the reliability of person-level outcomes. Internal consistency reliability is a way to test the generalizability of a set of items to the broad domain of items that could have been used on the test. This type of reliability is used to gauge the level of error in content sampling of the items as well as errors of measurement arising from sampling, administration, or other secular effects. On the other hand, test-retest reliability is an estimate of errors around an examinee's “true” score over a short time frame (Crocker & Algina, 2008).
Internal consistency reliability of measure responses was assessed with Cronbach’s alpha, a well-researched and widely-used method that was listed by the NQF as a method to demonstrate scientific acceptability (NQF, 2021). Test-retest reliability was evaluated using PPM correlations between participant composite scores across time points. Test-retest data were collected 10-14 days apart.
All calculations were performed in R (2025; version 4.4.0 or later) with the psych package (Revelle, 2025). A full data matrix of all respondents was loaded into R containing participant responses to all items on the instrument. For Cronbach’s alpha calculations, a correlation matrix from these responses was computed using matrix smoothing and full information maximum likelihood (FIML), the latter an optimal technique for handling missing data (Enders, 2010). The sample size used was the average number of complete responses across all items. For test-retest reliability, missing data were handled by pairwise deletion.
(A complete reference list is provided as a supplemental attachment in section 7.1.)
The attachment in 5.2.3a provides reliability testing results at the measure level. The document contains results for both Experiences Seeking Employment and Job Experiences measures, respectively. Standardized Cronbach’s alpha and test-retest correlation statistics are highlighted in yellow. The detailed report on internal consistency also includes 95% confidence intervals as well as leave-one-out item analysis. Alongside the test-retest correlation coefficient is also reported the number of subjects and items in the sample.
The Employment Instrument is evaluated at the encounter-level separately for the Experiences Seeking Employment and Job Experiences IDM/sub-measures.
Experiences Seeking Employment
Internal consistency (.71) and test-retest (.80) were adequate and good. This demonstrates this measure produces an acceptable level of measurement error for scores obtained from this portion of the instrument.
Job Experiences
Internal consistency (.79) and test-retest (.92) were good and excellent. This demonstrates this measure produces an low level of measurement error for scores obtained from this portion of the instrument.
Reliability for the IDMs derived from the Employment instrument were restricted to encounter-level testing (see results from the primary Employment Instrument).
Validity
There were two approaches used to perform validity testing at the encounter-level. Content validity and construct validity were the two approaches used. An in-depth discussion of the content validation process undertaken in Study 1 of the RTC/OM project was provided in section 2.6 and will not be repeated here.
Construct validity of measure outcomes was evaluated with parallel analysis (PA) via scree plots as well as exploratory factor analysis (EFA). Both were used to evaluate the factor structure of each measure. Similar to reliability analyses, all calculations were performed in R. First, to determine the number of factors to retain during EFA, parallel analyses were performed then compared with the theoretical structure proposed during measure development. This helped guide the number of factors that were fit during the EFA procedure.
EFA models were fit with the psych package using ordinary least squares . Oblique solutions were produced with Oblimin rotation. Missing data were handled via full information maximum likelihood (FIML) when computing the correlation from the full data matrix (see section 5.2.2).
The attachment in 5.3.4a provides the results of parallel analyses and exploratory factor analyses for both the Experiences Seking Employment and Job Experiences IDMs, respectively. For both Experiences Seeking Employment and Job Experiences IDMs, three different EFA models were fit to the data and the model output is provided. Parallel analyses for both measures recommended retaining four factors in the model. The initial hypothesis for both measures was a one-factor model. To explore model adequacy, one-, two-, and three-factor models were fit to the data.
Similar to Reliability results, the IDMs that make up this Instrument will be evaluated separately as to their outcome validity.
Experiences Seeking Employment
Parallel analysis suggested retaining four factors, although the hypothesized factor structure was a one-factor model. The presence of several factor loadings below .30 and subpar fit statistics (TLI = -0.03, RMSEA = 0.34) show that a higher-order model may be a better fit tot the data. This result is echoed by reliability results that are sufficient for basic research but do not show excellent results compared to other measures. Follow up analysis found that a three-factor model was a much better fit to the data (TLI = .73, RMSEA = .17).
Job Experiences
Parallel analysis suggested retaining four factors, and the hypothesized strcuture was a one-factor model. The best fit to the data was a three-factor EFA (TLI = .90, RMSEA = .06).
At the accountable entity-level, the RTC/OM team led a focus group with six leaders of HCBS organizations across Minnesota to evaluate the face validity of our measures.
In 2024, a focus group was conducted with six leaders of Home and Community-Based Services (HCBS) organizations in Minnesota. During the session, participants received information about the measure under discussion, including the associated outcome area (Job Experiences) and the method used to calculate the performance score. Participants were then asked to provide input on several aspects of the performance score using a scale ranging from 1-4:
Relevance: How relevant is the performance score to the HCBS services they provide? (Mean = _3.67_ )
Representation: To what extent does the score represent the employment outcomes it is intended to measure? (Mean = _3.4_ )
Accuracy: Does the score accurately reflect the outcomes experienced by their beneficiaries? (Mean = _3.4_ )
Feasibility: Would their organization have the capacity to collect the data needed for this measure? (Mean = _3.4_ )
Usefulness: How useful would the performance score be in helping the organization support its beneficiaries? (Mean = _3.4_ )
These results indicate that, overall, potential users of the measure rated the performance score as having adequate face validity. Participants rated the IDM as having high relevance, representation, accuracy, feasibility, and usefulness (all greater than 3 on a scale of 1 to 4). This demonstrates that this measure would have public-facing appeal for those want to use the measure to address individual's experiences at work.
Risk Adjustment
The National Quality Forum (NQF) emphasizes the importance of risk adjustment in evaluating outcome measures to ensure that potential threats to validity are addressed. Sociodemographic factors such as income and race have been explored as possible elements for risk adjustment by the NQF, which aims to develop guidelines in this area. In 2017, the NQF reviewed 303 submitted measures to assess their applicability for adjusting social risk factors that could affect health outcomes. The NQF panel recommended that these social risk factors follow the same criteria as clinical and health-related risk factors, although it noted a lack of a conceptual framework for their inclusion (National Quality Forum, 2014).
In the context of risk adjustment for this study, the NQF panel advised that sociodemographic factors should: (1) have a conceptual link to the outcome,(2) show an empirical relationship to the outcome , (3) display variability, (4) exist prior to intervention or care, (5) remain unaffected by intervention or policy changes, (6) be resistant to change, (7) be based on data that can be easily collected, (8) uniquely explain variations in the outcome, (9) contribute to the overall model, and (10) be considered valid and acceptable (NQF, 2014). These guidelines help differentiate risk adjusters from other variables.
More recently, the NQF conducted a review and convened a technical expert panel to develop further guidance for developers of outcome measures (National Quality Forum, 2020). They found that social risk factors mostly emerged at the individual and community levels, derived from various socioeconomic and demographic indicators. Functional risk factors, however, were often specific to individuals and based on self-reported survey data, with fewer clear definitions available. Statistical methods such as regression analyses were frequently used, though other models like hierarchical linear modeling were also applied to accommodate a broader range of risk factors.
Other risk adjustment models reflect similar themes to the NQF’s recommendations. For example, the Centers for Medicare and Medicaid Services (CMS) sought expert input on risk adjustment, and the Department of Health and Human Services (HHS) reviewed and integrated those recommendations into 10 key principles for risk adjustment (Centers for Medicare and Medicaid Services, 2016). Five of these principles overlap with the NQF’s guidelines, including recommendations that risk adjusters should be clinically relevant, predictive of medical costs, based on adequate sample sizes, encourage specific coding, and maintain internal consistency.
Additional support for the NQF’s guidelines comes from the Research Agency for Healthcare Research and Quality (Velentgas, n.d.), which recommended that risk adjustment should not include variables affected by the outcomes, that variable selection should be based on prior knowledge of their relationship to outcomes, and that risk adjusters should have statistical ties to outcomes.
In our systematic review of studies involving risk adjustment for individuals with disabilities receiving home or community-based services, we categorized studies based on the type of risk adjusters used and their relation to specific outcomes. Panels helped prioritize the risk adjusters, and our findings suggest that four factors—chronic conditions, functional disability, mental health status, and cognitive functioning—may be recommended as candidate risk adjusters (Houseworth et al., 2022).
We have not currently collected data related to mental health status and cognitive functioning due to feasibility issues during the pilot study. We did collect data allowing us to stratify by functional disability/chronic conditions. Therefore, we are currently unable to determine if large difference between providers on those factors would impact performance scores, as the literature suggest. This could lead to some inappropriate conclusions.
The attachment in 5.4.4a contains descriptive statistics across levels of functional disability for the measured outcomes. There are three levels of functional disability ranging from low to high service needs. Descriptive statistics reported for each level of functional disability are means, medians, standard deviations, minimum scores, and maximum scores.
The attachment in 5.4.4a also contains an ANOVA analysis to determine if there were significant differences between the aforementioned functional disability categories on the measure outcome. No statistically-significant differences were found for the Job Experiences measure outcome.
ANOVA analysis found that there were no significant differences between different levels of functional disability on individual's experiences at work.
Use & Usability
Use
Usability
The measured entities are bound by CMS regulations to provide HCBS services to support QoL outcomes in alignment with HCBS Final Settings Rule and Access Rule. As a minimum, this measure will provide evidence for the level of participation in meaningful activities, including engagement in activities at home and in the community to the extent they desire, expansion of natural supports and community connections, and service quality in supporting engagement in meaningful activities, of their clients as indicators of Access and HCBS. There is currently a gap in measures available to provide reliable and valid data to service providers on HCBS outcomes, including the outcome of job seeking experiences (UMN RTC/OM, 2020). The RTC/OM measures of Job Experiences and Experiences with Seeking Employment have the potential to fill this gap for providers to be able to report to CMS on their progress toward HCBS and Access. Most importantly, providers will be able to make informed judgments about the way people with disabilities receiving HCBS experience their services and their effectiveness to successfully seek employment of their choice.
In our work with PAVE in CA, the measure score will be used to validate a PAVE measure score related to job seeking experineces to assess the quality of services as related to the Quality Incentive Program (QIP) for CA disability service providers. A pilot study is currently under way to collect data using RTC/OM and PAVE measures.
The measured entities are bound by CMS regulations to provide HCBS services to support QoL outcomes in alignment with HCBS Final Settings Rule and Access Rule. As a minimum, this measure will provide evidence for the level of participation in meaningful activities, including engagement in activities at home and in the community to the extent they desire, expansion of natural supports and community connections, and service quality in supporting engagement in meaningful activities, of their clients as indicators of Access and HCBS. There is currently a gap in measures available to provide reliable and valid data to service providers on HCBS outcomes, including the outcome of meaningful activity (UMN RTC/OM, 2020). The RTC/OM measure of Job Experience has the potential to fill this gap for providers to be able to report to CMS on their progress toward HCBS and Access. Most importantly, providers will be able to make informed judgments about the way people with disabilities receiving HCBS experience their services and their effectiveness to gaining and maintaining a job the participant desires.
In our work with PAVE in CA, the measure score will be used to validate a PAVE measure score related to having a job of choice to assess the quality of services as related to the Quality Incentive Program (QIP) for CA disability service providers. A pilot study is currently under way to collect data using RTC/OM and PAVE measures.
(A complete reference list is provided as a supplemental attachment in section 7.1.)
Unavailable since this is a new measure and no trends are currently available.
As with the use of any measure to increase outcomes, the use of the RTCOM Employment measure may have potential unintended consequences, particularly if performance on the measure is tied to incentive or value-based payment programs. Poor performance on the measure could lead to reduced resources or negatively influence future funding decisions for provider organizations. In response, entities (HCBS providers)may reallocate resources from other critical areas to focus narrowly on improving measured employment outcomes, potentially undermining other areas of service quality. This can include a tendency to over focus on the measured outcomes and decreasing focus on other important or related outcomes. Another potential consequence is that providers may be held accountable for outcomes that are sometimes outside of their control. Not all individuals served have funding for services that directly support employment, limiting the provider’s ability to influence the measured outcome. Additionally, implementing and reporting on the measure may require significant resources, especially for smaller or under-resourced providers. This may result in a reduction of resources for other performance measurement or quality improvement initiatives. To reduce the risk of these unintended consequences, the measure should be accompanied by clear guidance on appropriate use, interpretation, and its role within organization-wide and broader quality improvement strategies. It should not be used in isolation for value-based funding decisions and careful thought by users and policy makers should be put into equitable use of the measure in such initiatives. Potential unintended consequences for the beneficiary includes the potential for loss of service funding due to the increased income that could result from better employment outcomes. Organizations should carefully track outcomes for individuals to manage the complex relationship between increased income and service funding. If used carefully, the benefits of the RTCOM Employment measure such as improved focus on meaningful employment outcomes and enhanced service quality should outweigh the potential unintended consequences.
Comments
Staff Preliminary Assessment
CBE #5105-2 Staff Preliminary Assessment
Importance
Strengths
- A clear logic model is provided, depicting the relationships between inputs (e.g., funding, supportive policies, stakeholder input), activities (e.g., person-centered services, training/skill building), and desired outcomes (e.g., increased employment, decreased need for employment seeking services, and improved services quality.)
- The proposed measure addresses a healthcare need not sufficiently covered by existing measures, offering advantages in terms of including items that solicit information about how effectively the provider system supports an individual’s employment.
- Description of patient input supports the conclusion that the measure is important to patients, family members, caregivers, and other stakeholders. This input was gained through a national content validation study of the National Quality Forum’s Home- and Community-Based Services (HCBS) Outcome Measurement Framework using a participatory planning and decision making process, Technical Expert Panels (TEPs), Advisory Committees, and vetting by representatives of provider agencies. TEPs and the RTC/OM Center Advisory Committee include people with disabilities as well as other stakeholder groups.
Limitations
- The submission provides information on anticipated impact, but those references do not appear in the reference list and cannot be evaluated. This submission could be enhanced by providing more detail on the anticipated impact of these measures and tying this back to the logic model.
Rationale
- The new measure is rated as 'Not Met But Addressable' due to incomplete evidence, particularly related to anticipated impact of the measure.
Closing Care Gaps
The developer did not address this optional domain.
Feasibility Assessment
Strengths
- The developer described their feasibility assessments and how those informed the final measure specifications.
- There are no fees, licensing, or other requirements to use any aspect of the measure (e.g., value/code set, risk model, programming code, algorithm).
- The developer described how all required data elements can be collected without risk to patient confidentiality.
Limitations
- Data capture does not occur during the course of care and requires additional, disruptive steps to collect, further complicating its integration into clinical workflows.
- The developer described the costs and burden associated with data collection and data entry, validation, and analysis. They are in the process of creating some mitigation processes including developing an electronic system at University of Minnesota's Institute on Community Integration that houses this initiative.
Rationale
- The measure is rated ‘Not Met’ because data capture does not appear to occur during the course of care and requires additional steps. Burden outside of cost/staffing related to the survey was not described in sufficient detail.
- Submission would be strengthened with more detail on survey training, how and when the survey can be implemented (and where this is during routine care delivery), and whether there will be any implementation guidance provided. The committee should seek clarification regarding who would collect the data once the measure is implemented in HCBS agencies, when the data will be collected, and how.
Scientific Acceptability
Strengths
- The developer explained the internal consistency and test-retest used to assess data element reliability.
- Data used for testing were collected within the past five years.
- Developer reported internal consistency of 0.79 and test-retest reliability (Pearson correlation) of 0.92 which are above their respective thresholds.
Limitations
- It was unclear whether, for each patient, the same interviewer conducted the initial and follow-up interviews used for the test-retest reliability. If not, may need to provide some explanation for why that would not bias the results.
- It was unclear how many patients, and from how many entities, were included in the test-retest reliability testing. The R output provided states the number of subjects as 14 which seems low.
- The penultimate sentence from section 5.2.2 was unclear. It stated "The sample size used was the average number of complete responses across all items."
Rationale
- The developer reported data element-level reliability based on data collected within the last five years. Report internal consistency and test-retest reliability were above the thresholds of 0.7 and 0.5, respectively.
- Some clarification is needed about whether the same interviewer was used for test and retest interviews for a particular patient; the number of patients included for testing; and the penultimate sentence in section 5.2.2.
Strengths
- Construct validity of measure outcomes was evaluated with parallel analysis (PA) via scree plots as well as exploratory factor analysis (EFA). Results supported the instrument construction.
- The developer applied stratification to measure results by functional disability based on a conceptual model supported by literature and expert panels.
Limitations
- Statistical analysis noted no meaningful differences in measure score between stratified categories. Additional risk variables were considered and included in conceptual model, but not analyzed due to lack of data availability.
Rationale
- Validity testing results supported the instrument construction. Stratification was applied to manage potential differences due to patient characteristics, supported by scientific literature and TEP findings. As additional entity-level data become available, future risk analyses including functional disability and other proposed risk factors are warranted to demonstrate that stratification ensures fair comparisons and enhances measure accuracy.
Use and Usability
Strengths
- The measure is not currently in use, but the developer indicates a plan for use in Quality Improvement with Benchmarking (external benchmarking to multiple organizations) and Quality Improvement (Internal to the specific organization).
Limitations
- The developer argues that accountable entities can use the measure results to improve performance. However, the guidance on implementing these actions is vague and lacks specificity.
Rationale
- For initial endorsement, there is a plan for use in at least one accountability application. However, it is unclear how accountable entities can improve their performance.
- The logic model notes key activities, such as ensuring access to resources supporting employment success. The submission could be strengthened by describing how the performance scores could be used to inform such activities and how the entities would go about securing this
Committee Independent Review
5105-2 summary
Importance
See comments on 5105-1
Closing Care Gaps
New instrument - not addressed
Feasibility Assessment
See comments on 5105-1
Scientific Acceptability
Clarifications requested in staff preliminary review may be answered briefly by measure developer
Encourage further testing of measure structure; EFA did not support hypothesized structure
Use and Usability
See comments on 5105-1 - this is a new instrument - not yet being used. Are examples of improvement activities required for new instrument justifying not met rating in staff preliminary review?
Summary
New measure addressing important gap in HCBC measure set. See questions re staff preliminary assessment of feasibility, reliability and use and usability - same as with 5105-1
Measure needs additional evidence, data and analysis
Importance
Need to provide additional information about the importance of the measure for improving care and outcomes.
Closing Care Gaps
Need to provide information about how the measure would close gaps.
Feasibility Assessment
Need more detail about how the measure would be used and implemented in practice
Scientific Acceptability
Need more data supporting the necessity for this measure.
Need better analyses supporting the validity testing for this measure.
Use and Usability
Measure is not in use.
Summary
This measure should come back when it has additional evidence, data, usability data and stronger analyses.
Not met
Importance
agree with staff
Closing Care Gaps
Agree with staff
Feasibility Assessment
In my experience, the length of these surveys often exceeds the time and focus respondents can muster.
Scientific Acceptability
agree with staff
agree with staff
Use and Usability
In my experience as VP of Quality for an agency that manages these services, while the measures are important, the only useful results for us are those that score highest and lowest. We then held focus groups of direct care staff and clients to flesh out the details of the topic. The most valuable insights from those focus groups came from issues not asked about, which, when addressed, significantly impacted the other scores.
Summary
In my experience as VP of Quality for an agency that manages these services, while the measures are important, the only useful results for us are those that score highest and lowest. We then held focus groups of direct care staff and clients to flesh out the details of the topic. The most valuable insights from those focus groups came from issues not asked about, which, when addressed, significantly impacted the other scores.
Not met but addressable
Importance
Thorough logic model; process described is very comprehensive and patient centered. Fills a gap in existing measurement for disabled individuals at the individual and provider level
Closing Care Gaps
Not addressed
Feasibility Assessment
The developers state that administering the measure as an interview requires approximately 10 minutes per beneficiary; however there really is no efficient way to do this since the measure requires an interview. In addition, the responses need to be manually entered after the interview into a secure data system. Have the developers considerd any technology to assist?
Scientific Acceptability
Internal consistency (.79) and test-retest (.92) were acceptable
agree with staff assessment
Use and Usability
agree with staff assessment
Summary
It is clear that a lot of work has gone into the development of this measure and I commend the developers for their diligence.
5105-2
Importance
Disagree with staff assessment.
I am having difficulty understanding how the measure as developed addresses the patient (probably client) experience of care (experience of services).There are certainly items in the instrument which contribute to this measure and also address experience of care/service, but there are also aspects of the instrument items which address outcomes of service and still others which relate to the experience of working but are not necessarily related to experience of care or outcomes of care. I agree that the issue of experiences associated with the job are important to address. This measure has too much rolled into it that goes well beyond an experience of care measure.
Closing Care Gaps
optional, not submitted.
Feasibility Assessment
Agree with staff assessment, but also have other concerns.Given the nature of the instrument items and the developer's assertion that service recipients want to talk about their job experiences and be heard, it seems unlikely that the survey instrument can be completed in a 10 minute encounter.
Scientific Acceptability
agree with staff analysis, but concerns with whether this is truly an experience of care measure.
agree with staff analysis, but concerns with whether this is truly an experience of care measure.
Use and Usability
Agree with staff asssessment, but also have other concerns.
Because the measure is informed by instrument items which seem to go well beyond the patient/client's experience of care/services, it is not clear to me how service providers could use this measure to inform quality improvement efforts focused on experience of care, as some of the items informing the measure seem to be more related to outcomes of servce, and some seem to be even more distal than either experience of service/care or outcomes of service/care.
Summary
I do not understand how this is an experience of care measure. The 13 instrument items which inform the measure seem to relate not only to experience of care/service, but also to outcomes of care/service and to experiences which seem to go beyond either outcomes or experience of care/service. While the measure addresses an important issue, it would be more useful if the measure was informed specifically by instrument items which relate to experience of care/service alone.
Public Comments
No public comments received.
No public comments received.