RTCOM Instrument for Home- and Community-Based Services (HCBS) Recipients - Employment

CBE ID

5105

1.5 Project

Advanced Illness and Post-Acute Care

1.0 New or Maintenance

New

1.1 Measure Structure

Instrument + Derived Measure Set

1.1a Instrument or Derived Measure

Instrument

Is Under Review

Yes

Next Maintenance Cycle

Spring 2025

1.6 Measure Description

This instrument assesses an individual’s experiences in paid employment as well as their experiences searching for paid employment. The target population for this instrument is adults with disabilities who receive HCBS or HCBS-like services. This is a self-contained instrument that can be administered independently of other RTCOM instruments.

The instrument is administered through an in-person or video-conferencing interview where an interviewer guides an individual through a series of questions (i.e., items). The responses for each item are scored on a Likert scale. The available research supports measure administration every four to six months.

Measure Specs

General Information

1.7 Measure Type

Patient-reported Experience Performance Measure (PRE-PM)

1.8 Level of Analysis

Population or Geographic Area

Other

1.8a Population or Geographic Area Level of Analysis

Intellectual and/or Developmental Disability, Physical Disability, Psychiatric Disability

1.8b Other Level of Analysis

Home- and Community-Based Services Provider

1.9 Care Setting

Other

1.9b Other Care Setting

Home- and Community-Based Services Provider

1.10 Measure Rationale

Over the last decade, policy shifts have placed increased attention on integrated employment of people with disabilities. As a society, we have moved from sheltered workshops and day activity centers to supported employment, and most recently through the application of approaches to customized employment, to competitive integrated work opportunities for people with even the most intensive support needed, Yet despite a variety of policies, directives, and legislation, people with disabilities continue to experience limited participation in the labor market. (Almalky, 2020; Butterworth, et al, 2015a; Miethlich & Oldenburg, 2019). Recent estimate indicate that in the US, only one in three (34.9%) individuals with disabilities are employed compared to 76% of their counterparts without disabilities, and this disparity appears to be increasing over time (Bonaccio, et al 2020; Houtenville & Ruiz, 2012; Kraus,2017; Lauer & Houtenville, 2017).

Individuals with disabilities face challenges to employment beyond that experienced by their peers without disabilities. Not only are they less likely to be employed when they desire to be so (Andara et al, 2024) but the quality of their employment is significantly lower than their counterparts. Persons with disabilities consistently report lower levels employment quality than their counterparts without disabilities and are nearly twice as likely to report low-quality employment in the form of employment in which they feel trapped) or is insecure and unrewarding . This gap in employment quality is particularly pronounced for those who reported living with both a physical and mental/cognitive condition (Shahadi, et al, 2023). Given the relative stagnation that has occurred in disability employment over the past two decades, the emphasis in many states and among support providers is on raising overall employment rates. This, however, overlooks the problem that not all jobs are of equal quality, and some jobs offer few of the assumed benefits of paid employment Burgard & Lin, 2013; Burroway, 2017). Therefore, although participation in paid employment is an objective with the potential to support social and economic well-being, the long-term goal should be high-quality employment (Martin Ginis, et al, 2020; Tompa et al, 2022).

Specific external barriers to employment for individuals with disabilities that have been identified include a lack of job opportunities (Grant, 2008), insufficient supports to maintain employment (Shier, et al., 2009), and discrimination and stigmatization in the workplace (Winn & Hay, 2009). Individuals with mental health issues face similar difficulties finding and retaining employment, with an added burden of unique stigmatization due to perception of mental health and safety and when persons with disabilities do have jobs they tend to be underemployed (Stuart, 2006).
Beyond the financial benefits of the employment of people with disabilities, access to inclusive high quality employment that pays a living wage has been shown to be a key determinant of individual health and quality of life outcomes (Dean et al., 2018; Emerson et al., 2018; Randall et al., 2022; Robertson et al., 2019; Voermans et al., 2020), as well as being economically and socially beneficial for employers, communities, and broader society (Cimera, 2010; Taylor, et al., 2021).

Existing employment measures developed for use with people with disabilities typically assess a limited number of aspects of one’s work including rates of employment, type of employment, hours worked, and weekly or monthly earnings (Cimera & Burgess, 2011; Grigal, Hart, & Migliore, 2011). When considering the well-being of people with disabilities, earnings and hours worked are key outcome variables. However, this data only inform one about limited aspects of an individual’s employment experience (Nord, et al., 2013; Martin Ginis, et al, 2020; Migliorie et al., 2012; Tompa et al, 2022) and leave out a wide variety of outcomes strongly associated with both the quality of employment and the quality of life one leads. Second, only a few existing employment measures include items that solicit information about how effectively the provider system supports an individual’s employment. Just as important, few currently available instruments have demonstrated adequate reliability and validity and across different disability populations. A third limitation is that current measures do not account for the outcomes experienced by a large segment of the disability population, who although they desire to work, are not currently employed. Providers therefore have little information about the people they serve who are in the process of applying for work but have yet to be able to find employment or are in the process of applying for a new job. The submitted RTCOM employment measure concepts attempt to fill this gap in HCBS measurement.

The rationale for inclusion of a set of employment performance measures was based on:

(a) A systematic review of the disability employment research literature

(b) Use of a variety of approaches, including a participatory planning and decision-making process employed at a national level, to determine the extent to which stakeholders believed performance scores based on various components of the job experiences construct were important to include in the measure under development

(d) An examination of gaps or limitation inherent in current approaches to measuring outcomes and service quality in this area.

The Quality of Job Experiences and Experiences Seeking Employment measures developed as part of the Research and Training Center on HCBS Outcome Measurement (RTC/OM) were designed to target employment outcomes of people with disabilities at the agency level and be sufficiently sensitive to changes in policy and services that performance scores could be used to document improvement in quality of both services and HCBS recipient outcomes. This level of measurement needs to be more granular than that which is used to focus on compliance and requires measures able to demonstrate reliability, validity, accuracy, and sensitivity at the agency level with the specific groups of HCBS recipient organizations’ serve. The measure is constructed to be person-centered, taking into consideration personal preferences with respect to how one goes about a search for employment, the extent to which they desire support in applying for a job, and for those who work, the level and types of support they need on the job, the quality of interactions they have with co-workers and the general public, the number of hours worked, the extent to which these hours meet their needs, the degree to which person is looking for advancement in their employment, the types/forms of both supervision and job support they prefer as well as from whom, and the specific roles they serve for their employer.

These measure characteristics are necessary in order to ensure the provision of actionable data so that provider agencies can determine the effectiveness with which they are supporting individuals with disabilities to obtain employment, and not just survive in their jobs but thrive in them. Fortunately, the existing research base clearly indicates that people with intellectual and physical disabilities, autism, mental health challenges and TBI can all be matched to job positions in which they can be successful (Gustafsson, et al, 2013; Henry, et al 2019; Houtenville & Kalargyrou, 2012; Khayatzadeh-Mahani, et al 2020 Nevala et al, 2019; Suijkerbuijk, et al 2017) and a variety of approaches to support the employment of people with disabilities are now approaching or have met the criteria for consideration as evidenced-based or promising practices including supported (Wehman, 2023; Wehman et al, 2018) and customized (Inge, et al, 2018; Riesen et al, 2023) approaches to employment. As mentioned previously, however, it should be noted that employment quality is just as important an outcome to consider as the number of people with disabilities who are employed.

Information/data available based on performance scores on our two employment measures have the potential to provide support agencies with a variety of information that can be used to (a) document overall service quality and facilitate policy and/or programmatic changes needed as part of agency quality improvement efforts, (b) identify specific aspects of the Job Experiences subdomain where performance is less than desirable as well as those areas in which the agency is supporting exceptional outcomes, (c) longitudinally track changes that occur in service quality and job experiences, and provide families and persons with disabilities with information they can use to help make informed decisions as to which employment support agencies they desire to provide services to their family member with a disability.

(A complete reference list is provided as a supplemental attachment in section 7.1.)

1.11 Measure Webpage

http://example.com

1.20 Types of Data Sources

Patient-Reported Data and/or Survey Data

1.20c Format: Patient-Reported Data and/or Survey Data

Non-digital

1.25 Data Source Details

N/A

Measure Calculation

1.13a Attach Data Dictionary

RTCOM Codebook-and-Scoring-Scheme - Employment + ServicesSupports.pdf

1.16 Type of Score

Composite scale

1.17 Measure Score Interpretation

Better quality = Higher score

1.18 Calculation of Measure Score

The measure score for individual recipients is calculated by the sum of item responses on the instrument. See section 1.6 and the attached Data Codebook in 1.13 for the items and scoring codes on this instrument. Stratification does not modify the calculation of measure scores. Note that the measure score interpretation in section 1.17 is only applicable to this individual measure score.

Scores at the accountable-entity level are calculated using individual measure scores and incorporating the numerator and denominator criteria provided in sections 1.14/1.14a and 1.15/1.15a. For a calendar year, accountable entities will obtain individual measure scores for eligible HCBS recipients that they serve. The numerator is the number of individuals who obtained a score in the measure-derived interquartile range (IQR; see 1.14/1.14a). The denominator is the total number of eligible HCBS recipients that were assessed. The accountable entity score is this numerator/denominator ratio, which will be a proportion.

Each individual HCBS recipient should only be counted once in the numerator and denominator. In other words, an HCBS recipient with more than one assessment within a calendar year will still be only counted once for reporting purposes. For these individuals, use their most recent individual measure score that still falls within the calendar year.

A higher score indicates that there is a greater performance gap on measure outcomes for recipients at a given accountable entity. A higher proportion also suggests that the provider serves a wider population with greater variability in service needs. We have purposefully avoided an accountable entity score interpretation that is based on high (or the highest) individual measure scores as this would provide perverse incentives for those using our instruments. We also do not recommend higher or lower accountable entity scores be interpreted as “better” or “worse”. Rather, accountable entity scores should be accurate and informative.

1.21a Data Collection Tool URL(s)

http://example.com

1.21b Attach Data Collection Tool(s)

RTCOM Instruments - Employment + Demographics.pdf

1.22 Proxy Responses

1.23 Survey Respondent

Other

1.23a Other Survey Respondent

HCBS Recipient

1.24 Data Collection and Response Rate

The Employment Measure is reported by HCBS recipients and administered either in person or through a HIPAA-compliant remote video conferencing platform. Currently, the instrument is available only in English; however, an interpreter can be used during the interview to improve accessibility. Response rates may be improved by offering accommodations that increase accessibility, such as environmental modifications for individuals with disabilities and visual displays of response scales.

1.26 Minimum Sample Size

There is no minimum sample size that is needed to calculate a performance score at the individual HCBS recipient level. Performance scores for all of the IDMs submitted are based on instrument scores and it is these scores that are reported.

Given that the instrument was developed were intended to be used at the provider level to document service quality and HCBS recipient outcomes, it is unlikely that the measure would be used to represent the quality of services received and outcome experienced by a single individual. As a result, while performance scores could be calculated on the basis of a single individual they would have little meaning in that isolated context. Focus group results and other discussions we have had with HCBS providers indicate that the most likely future use of the measures will be to (a) document the extent to which providers and sites within provider networks meet established benchmarks, (b) determine the extent to which quality improvement efforts have had a significant impact on service quality and HCBS beneficiary outcomes in targeted areas, and (c) potentially be used in the establishment of “report cards” for agencies within and between provider networks that can be used by external funders and families to determine programmatic strengths and areas in need of improvement and to facilitate informed decision-making when people with disabilities and/or their family members are looking to contract for services.

When used in this manner, sample sizes deemed adequate for analysis and interpretation of the submitted measures will depend on a variety of factors including, but not limited to the questions one desires to answer, comfort levels in committing Type I (false positive) and Type II (false negative) errors, the sampling methods used (i.e., probability vs. non-probability sampling) , the target population and its attributes. Attributes of HCBS samples include a wide variety of factors that impact sample size and include client demographic variables such as disability type, intensity of support needs and their variation within the population of interest, geographic location, the size of the population, the effect size one expects and a variety of other features of clients and the environment that could be relevant for answering a provider’s evaluation questions.

The University of Minnesota’s Institute on Community Integration, home of the RTC/OM has long-term history of working with HCBS service providers on a wide variety of evaluation projects that have included the use of measures similar to the ones we are submitting for review. These experiences suggest that if a provider desires to use performance scores aggregated across individuals to determine changes within that provider organization, a representative sample (adjusted for population size) that includes an absolute minimum of 35-50% of HCBS beneficiaries receiving services is needed to draw valid conclusions regarding quality improvement efforts.

Importance

Evidence
Measure Impact

Evidence

2.1 Attach Logic Model

EM-Logic-Models-Employment_0.docx

2.2 Evidence of Measure Importance

Existing employment measures developed for use with people with disabilities typically assess a limited number of aspects of one’s work including rates of employment, type of employment, hours worked, and weekly or monthly earnings (Cimera & Burgess, 2011; Grigal, Hart, & Migliore, 2011). When considering the well-being of people with disabilities, earnings and hours worked are key outcome variables. However, this data only inform one about limited aspects of an individual’s employment experience (Nord, et al., 2013; Martin Ginis, et al, 2020; Migliorie et al., 2012; Tompa et al, 2022) and leave out a wide variety of outcomes closely associated with both the quality of employment and the quality of life one leads. Second, only a few existing employment measures include items that solicit information about how effectively the provider system supports an individual’s employment. Just as important, few currently available instruments have demonstrated adequate reliability and validity and across different disability populations. A third limitation is that current measures do not account for the outcomes experienced by a large segment of the disability population, who although they desire to work, are not currently employed. Providers therefore have little information about the people they serve who are in the process of applying for work but have yet to be able to find employment, or are in the process of applying for a new job. The submitted RTCOM employment measure concepts attempt to fill this gap in HCBS measurement.

The rationale for inclusion of a set of employment performance measures was based on:

· A systematic review of the disability employment research literature

· Use of a variety of processes, employed at a national level, to determine the extent to which stakeholders believed performance scores based on various components of the job experiences construct were important to include in the measure under development

· Input from several technical expert panels, and

· An examination of gaps or limitation inherent in current approaches to measuring outcomes and service quality in this area.

Of critical relevance related to the inclusion of employment experience measure is its Importance as defined by stakeholders. Importance was determined via three sources/processes. First, we began by considering all the domains and subdomains identified by the NQF Framework for Home & Community Based Services Outcome Measurement (2016). The framework for HCBS outcome measurement developed by the NQF mentioned employment but only included it as an element of the subdomain of Meaningful Activity which is itself a subdomain of the Community Inclusion domain.

University of Minnesota RTC/OM staff engaged stakeholders on whose lives HCBS has an impact in a series of participatory planning and decision-making groups. Stakeholders included: (a) people with disabilities (100 participants), including individuals with IDD, physical disabilities, traumatic brain injury, psychiatric disabilities, and age-related disabilities; (b) family members of people with the above noted disabilities (84 participants); (c) representatives of HCBS provider agencies (89 participants), and (d) state level HCBS program administrators/policy makers (47 participants) for a total of 320 participants in 58 small groups conducted nationally. Stakeholders took part in a participatory planning and decision-making (PPDM) process in which participants provided importance weights for each domain and subdomain of the NQF framework on a scale from 0-100 based on their perceived importance in determining the HCBS outcomes and quality. As part of the PPDM process, stakeholders are first asked to examine the basic domains and subdomains for the framework or theory under consideration and add or rearrange its components in order to refine the model. The large majority (83%) of PPDM groups across all stakeholder types reached consensus that employment was an outcome of sufficient importance as to assume the status of its own domain. Among stakeholder groups composed of with disabilities as well as groups representing families, over 90% of participants (people with disabilities = 94%; families = 91%) supporting this refinement of the NQF model with somewhat lower percentages of support among providers and program administrators. Because the existing NQF framework for HCBS outcome measurement did not include employment as a domain or subdomain but only included it as an element of the subdomain of Meaningful Activity no PPDM importance weightings of this domain can be provided.

As a third step in the process of developing a rationale for the inclusion of this measure, we solicited the input of measurement and content experts in disability-related fields through the use of technical expert panels that included people with disabilities. Two groups of experts were asked to rate the importance of all subdomains in the NQF framework in terms of their relevance, importance, and accuracy. Based on the 1-4 point Likert-type rating scales used for these measures, mean Job Experiences TEP scores for people with disabilities who were employed were as follows: Relevancy 3.50/4.00; Importance 3.53/4.00; and Accuracy 3.12/4.0. Mean scores for Experiences Seeking Employment (i.e., those unemployed but currently job seeking) were Relevancy 3.54/4.00; Importance 3.55/4.00; and Accuracy 3.00/4.0.

The Job Experiences and Experiences Seeking Employment measures developed as part of the Research and Training Center on HCBS Outcome Measurement (RTC/OM) were designed to target employment outcomes of people with disabilities at the agency level and be sufficiently sensitive to changes in policy and services that performance scores could be used to document improvement in quality of both services and HCBS recipient outcomes. This level of measurement needs to be more granular than that which is used to focus on compliance and requires measures able to demonstrate reliability, validity, accuracy and sensitivity at the agency level with the specific groups of HCBS recipient organizations’ serve. The measure is constructed to be person-centered, taking into consideration personal preferences with respect to personal preferences with respect to how one goes about a search for employment, the extent to which they desire support, the level and types of support they need, and for those currently employed he number of hours worked, the extent to which these hours meet their needs, the degree to which person is looking for advancement in their employment, the types/forms of both supervision and job support they prefer as well as from whom, and the specific roles they serve for their employer.

These measure characteristics are necessary in order to ensure the provision of actionable data so that provider agencies can determine the effectiveness with which they are supporting individuals with disabilities to obtain employment, and not just survive in their jobs but thrive in them. Fortunately, the existing research base clearly indicates that people with intellectual and physical disabilities, autism, mental health challenges and TBI can all be matched to job positions in which they can be successful (Gustafsson, et al, 2013; Henry, et al 2019; Houtenville & Kalargyrou, 2012; Khayatzadeh-Mahani, et al 2020 Nevala et al, 2019; Suijkerbuijk, et al 2017). A variety of approaches to support the employment of people with disabilities are now approaching or have met the criteria for consideration as evidenced-based or promising practices including supported (Wehman, 2023; Wehman et al, 2018) and customized (Inge, et al, 2018; Riesen et al, 2023) approaches to supporting employment success. As mentioned previously, however, it should be noted that employment quality is just as important an outcome to consider as the number of people with disabilities who are employed.

Information/data available based on performance scores on our two employment measures have the potential to provide support agencies with a variety of information that can be used to (a) document overall service quality and facilitate policy and/or programmatic changes needed as part of agency quality improvement efforts, (b) identify specific aspects of the Job Experiences subdomain where performance is less than desirable as well as those areas in which the agency is supporting exceptional outcomes, (c) longitudinally track changes that occur in service quality and job experiences, and provide families and persons with disabilities with information they can use to help make informed decisions as to which employment support agencies they desire to provide services to their family member with a disability.

(A complete reference list is provided as a supplemental attachment in section 7.1.)

Measure Impact

2.6 Meaningfulness to Target Population

Stakeholders, particularly those who have disabilities and their supporters, should be at the heart of measurement development. In order to ensure that HCBS outcome measures are of high quality, the measure development process must include input from stakeholders and most importantly the intended population with which the measures will be used. Furthermore, we contend that the unique challenges associated with measuring employment outcomes among members of diverse populations of HCBS recipients requires strong stakeholder involvement throughout all stages of development. This process has been affirmed by NIDILRR, ACL, and the Centers for Medicare and Medicaid Services (CMS, 2019). Using a sound HCBS outcome measurement framework that has evidence of content validity provided by stakeholders including people with disabilities; putting all measures developed through multiple expert panel reviews; undertaking cognitive testing with people with a variety of disabilities are necessary strategies or processes in which one must engage to ensure for quality measure development.

The target populations for the submitted measures of employment (Quality of Job Experience and Experiences Seeking Employment) include people with intellectual and developmental disabilities, psychiatric disabilities, physical disabilities, and TBI/ABI as well as age-related disabilities. Although the large majority of people with such disabilities possess the capacity to articulate their thoughts and feelings about the outcomes in question and their importance, it must be recognized that some do not. The intensity of their support needs may be such that they experience difficulty understanding questions and articulating their thoughts and feelings. Not only are such individuals at elevated risk for experiencing poor services and outcomes but most HCBS performance measure programs are not set up to reflect their experiences. The RTC/OM development team therefore instituted a process in which not only HCBS beneficiaries, but other relevant stakeholders, provided input into the measure development process. These consisted of family members of people with the above noted disabilities (who are often direct caregivers on either a part or full-time basis), representatives of support provider agencies, and HCBS program administrators who are responsible for using performance measure data to improve both services and outcomes. At multiple steps along the measure development process, these individuals were consulted and their input incorporated into the measure development process. All of the measures submitted for review were based on:

A systematic review of the disability employment literature
The results of a national content validation study of the National Quality Forum’s HCBS Outcome Measurement Framework using a Participatory Planning and Decision-Making (PPDM) process.
Utilization of multiple Technical Assistance Panels (TEPS) that included people with disabilities as well as members of other stakeholder groups
Input from an RTC/OM Center Advisory Committee composed of people with a variety of disabilities in addition to other stakeholder groups, and
Extensive cognitive testing of measure items and response options, and
Vetting of measures by representatives of provider agencies

A Systematic Review of Employment of People with Disabilities Literature. First, we began by considering the current literature which clearly indicates that people with disabilities desire to have meaningful employment and that work enriches their lives in a number of ways (Dean et al., 2018; Emerson et al., 2018; Randall et al., 2022; Robertson et al., 2019; Voermans et al., 2020), as well as being economically and socially beneficial for employers, communities, and broader society (Cimera, 2010; Taylor, et al., 2021).

PPDM Process. Of critical relevance related to ensuring that HCBS recipients for whom the measures under development were intended viewed them as important, we then examined the domains and subdomains identified by the NQF Framework for Home & Community Based Services Outcome Measurement (2016). The initial framework for HCBS outcome measurement developed by the NQF covered 11 domains as well as 2-7 subdomains within each domain. However, employment was not included in the original framework at either a domain or subdomain level. As a second step in determining importance, relevance, and meaningfulness of including employment measures, RTC/OM staff engaged stakeholders from 29 states on whose lives HCBS has an impact in a series of participatory planning and decision-making (PPDM) groups. Stakeholders included: (a) people representing with disabilities (100 participants), including individuals with IDD, physical disabilities, traumatic brain injury, psychiatric disabilities, and age-related disabilities; (b) family members of people with the above noted disabilities (84 participants); (c) representatives of HCBS provider agencies (89 participants), and (d) state level HCBS program administrators/policy makers (47 participants) for a total of 320 participants who took part in 58 small (4-6 person) PPDM groups.

The PPDM process initially included meeting with homogeneous (with respect to disability and stakeholder type) stakeholder groups and providing them with an opportunity to evaluate the original NQF framework. They were provided with the opportunity to add to it, remove domains and/or subdomains they believed were not important, and then stipulate which personal outcomes and service characteristics were most important to measure. Following stakeholders reaching consensus with respect to the domains and subdomains of the original framework they wanted to add or remove, members of each group took part in a process in which they first independently assigned importance weights for each domain and subdomain of the original NQF framework on a scale from 0-100 (or 0-10 for persons with cognitive challenges) based on their perceived importance in determining the HCBS outcomes and service quality. As part of the PPDM process, stakeholders then discussed their weightings first at the subdomain level and later at the domain level examining why people in their group assigned the importance weightings they did. Stakeholders were then given the opportunity to assign a second set of importance weightings taking into consideration what they had heard during their discussion.

Overall, results from PPDM groups indicated a high degree of stakeholder support for inclusion of a separate Employment Domain in the NQF HCBS Outcome Measurement Framework as well as support for the development of measures for both persons with disabilities currently employed as well as those seeking work but currently unemployed. Across all four stakeholder groups, measurement of employment outcomes and both access to and the quality of vocational supports was viewed as of high importance (People with Disabilities = 94% agreement; Family members = 96% agreement; Providers = 83% agreement, Program administrators = 91% agreement).

Technical Expert Panels. As a second step in the process of developing measures, we solicited the input of people with lived experience with disability (N = 9) as well as measurement and content experts in disability-related fields (N = 12) for a series of technical expert panels (TEPs). Four TEP groups were formed and initially asked to rate the importance of all subdomains in the NQF framework that received the highest importance weightings across stakeholder groups on a 1-5 point Likert-type scale with respect to their feasibility, usability/utility, and importance, as well as provide an overall score for the subdomain. Across stakeholder groups, the Employment domain received the following weigthing with respect to feasibility (Mean = 4.6/5.0), usability/utility (Mean = 4.5/5.0), and importance (Mean = 3.96/5.0), as well as provide an overall score for the subdomain (Mean = 4.3/5.0). This information, in combination with PPDM results, the systematic review of the literature, and analysis of existing HCBS outcome measures indicated both the meaningfulness of the employment construct and perceptions among stakeholders that it is important to measure.

At follow-up TEP meetings following the initial development of items for those subdomains that were selected to be a focus of RTC/OM measures related to employment (Job Experiences & Experiences Seeking Work), TEPs engaged in a similar process using 1-4 point Likert-type rating scales to rate the relevance, importance, accessibility/understandability, and accuracy of each item developed for the employment measures under construction. Mean scores for Quality of Job Experiences for employed people with disabilities were as follows: Relevance (Mean = 3.49/4.00), Importance (Mean = 3.49/4.00); Accessibility/Understandability (Mean = 3.37/4.00); and Accuracy (Mean = 3.12/4.00). Mean TEP scores for Experiences Seeking Work were Relevance (Mean = 3.53/4.00), Importance (Mean = 3.55/4.00); Accessibility/Understandability (Mean = 3.20/4.00); and Accuracy (Mean = 3.12/4.00).

RTC/OM National Advisory Group. In conjunction with the results of PPDM groups and TEPs, ongoing input was also solicited from a national advisory group of HCBS stakeholders composed of 14 individuals including 5 with lived experience with disability. The RTC/OM Center Advisory Committee provided valuable feedback not only with respect to the item content of the measures under development but measure administrator training content and the medium (live versus virtual using HIPAA compliant version of Zoom Meeting) through which interviews would be conducted and the approach to cognitive testing that would be used after item refinement. This group not only provided critical qualitative information stressing the importance of employment measures but stressed the need to also included recommendations for the development of measures associated with employment including people with disabilities having access to affordable and safe transportation, and the availability of employment within one’s community of choice.

Cognitive Testing. The terminology used to describe services or supports can vary based upon disability group membership, the settings in which supports are received, and/or the services and supports themselves. The terms work, job, and employment, for example, have positive, neutral, and sometimes even different meaning, when used in different contexts. For example, is a person with a disability living on their family’s farm and working forty hours/week for no pay but for food, shelter, and clothing “employed,” “a volunteer,” or working but not employed.” Thoroughly testing with people with disabilities, the language and terminology used in measure concepts, and how the measure would typically be implemented (e.g., interview, survey, etc.) is therefore essential to ensure items are universally understood across HCBS recipients.

Cognitive testing (CT) focused on obtaining direct input from respondents to verify their interpretation of items and the words of which they are composed to ensure that these matched the developer’s intent (Ericsson & Simon, 1980; Willis, 2005; Willis, et al., 1991) and used a cognitive testing strategy referred to as the “Think Aloud Method” to address the core cognitive components of item responding: comprehension, retrieval of information needed to answer the item, making a judgment, and reporting a response (Tourangeu, 1984; 2018). This approach provided yet another way to involve people with disabilities in the measure development process.

Focus Groups with Potential Measure Users. It is essential that people with disabilities have multiple opportunities to provide input into the measure development process and resulting measures. The employment measures under development, however, also needed to undergo vetting by representatives of employment provider and vocational rehabilitation agencies. After final performance measure refinement based on the results of piloting and field-testing the final measures under submission were presented to two separate groups of potential users utilizing a focus group format. Participants were recruited from a large human service organization in Minnesota that provides residential, home health, and employment services, and a statewide network of human services providers located in Michigan that provide residential, and employment supports (Total N = 23). Measures were initially shared with participants several days prior to scheduled focus groups. Groups were initiated with RTC/OM providing background information and answering questions about the measures themselves, their administration, analysis, and use. After facilitators were assured that participants’ questions had been answered group discussion focused on, (a) the importance of the measures, (b) their overall quality and comprehensiveness, (c) the feasibility of provider agencies using the measures, and (d) the utility of the performance measures developed and how providers could foresee using them. {Overall, the employment measures developed were viewed as important, high in quality, feasible to administer and capable of providing actionable information. The sole negative response to the measures was concern over with whom this information would be shared if the measures were employed on a widescale basis.

(A complete reference list is provided as a supplemental attachment in section 7.1.)

Feasibility

Feasibility
Proprietary Information

Feasibility

4.1a Data Structure and Availability

The composite scores (i.e., measure scores, see section 1.18) from both the Job Experiences and the Experiences Seeking Employment measure will be available through electronic platforms, such as Qualtrics or incorporated into existing electronic systems providers already use. It is important to understand that HCBS data are by definition not medical data and therefore require different electronic systems from the usual hospital managed system. The HCBS field is not as centralized at the provider level and their electronic systems are developing.

That said, in addition to working with the systems providers may already been using, we are in the process of developing an electronic system at the University of Minnesota’s Institute on Community Integration that houses this initiative. This system will offer providers an opportunity to utilize our electronic system and technical assistance to use the system to house their scores for a negotiated fee.

4.1b Implementation Costs and Burden

Administering the measure as an interview requires approximately 10 minutes per Home and Community-Based Services (HCBS) beneficiary. The primary human resource burden includes the time of a trained interviewer to administer the measure and a data entry specialist or the interviewer themselves to enter the responses into a secure data system. Minimal additional personnel time is needed for data validation, which may involve reviewing and cleaning inconsistent or incomplete data, and follow-up with interviewers to clarify discrepancies. While these tasks include some investment of staff time, they are essential to ensuring data quality and integrity. To support accurate and valid data collection, organizations may also use supplementary resources such as interviewer training sessions, standardized data entry protocols, and processes to conduct periodic checks of the data. Despite the staff requirements, the burden on respondents was minimal. During testing, the vast majority of HCBS recipients reported a positive experience with the interviews. Many beneficiaries expressed appreciation for the opportunity to share their perspectives and discuss their services and outcomes with interviewers.

4.1c Confidentiality

As part of one of the nation’s prominent research universities with a large academic health sciences program, the Institute on Community Integration and RTC/OM have access to a wide variety of servers/data storage systems that meet or exceed all HIPAA security requirements and can be used to ensure the privacy/confidentiality of personally identified information. All clinical and human subjects’ data collected as part of RTC/OM performance measures will be secured with a University of Minnesota-approved resource at all times for the full extent of their life. This is true even when data has been de-identified. University approved methods of storing, clinical and human subjects’ data that will be used by the implementer (RTC/OM) include servers exclusively devoted to and supported by the University’s Health Sciences Technology (HST) group. HST supports University departments that need to store Private Highly Restricted data. The HST Operations and Infrastructure team supports and manages applications and infrastructure to meet University standards regarding HIPAA compliance. HST will work with RTC/OM staff to identify onboarding and maintenance requirements necessary to comply with UIS standards for PHR systems and data. Operating systems include professionally managed RHEL (Linux) and Windows Server environments. HST offers multiple storage options for data, documents, and folders to meet a variety of clinical, research, and storage needs. Network storage (SMB/NFS) is available on the Twin Cities campus for Healthcare Component (HCC) departments. The University’s Health Information Privacy & Compliance Office (HIPCO) and its Institutional Review Board RB strongly encourage the use of limited datasets to maintain confidentiality when some identifiers are needed and this recommendation will be followed. A Limited Data Set is a dataset that contains a limited set of indirect identifiers, and it's only used within the University's Health Care Components or under a Data Use Agreement (DUA). This will allow implementers (RTC/OM) to access data with limited privacy risks while still being able to provide critical service quality and outcome data to RTC/OM measure users in a way that has a high degree of utility. Key aspects of creating a Limited Data Set will entail:

(a) Removing Direct Identifiers: All direct identifiers including names, street addresses, and telephone numbers will be removed.

(b) Including Limited Indirect Identifiers: Certain indirect identifiers, such as dates (e.g., date of birth, death, admission), and geographic information (city, state, zip code) will be included

(c) Creating Data Use Agreements (DUAs): When PHI in a Limited Data Set is shared with a third party, a Data Use Agreement (DUA) will first be established and approved by the UMN’s IRB, Sponsored Projects Administration and Office of General Counsel. This agreement will outline the permissible uses of the data and ensures compliance with HIPAA.

(d) Review and Approval Processes: The process for creating and sharing the Limited Data Set will include review by HIPCO, the IRB, and the Sponsored Projects Administration (SPA).

4.3 Feasibility Informed Final Measure

The final measure is a result of multiple feasibility assessments. First, we conducted Participatory Planning and Decision Making (PPDM) groups with people with disabilities, family members, and staff to weigh importance of the NQF domains and subdomains. This process assisted our team with the selection of measure concepts to prioritize for developing within HCBS. Second, the items and their response options from which the measure was composed was reviewed by a technical expert panel to test whether the measure reflected the intent behind the measure concept. Third, the measure underwent cognitive testing during which the items and their response options from which the measure was composed were tested by people with intellectual and developmental disabilities, aging needs, TBI, mental health needs, and physical disabilities. We wanted to assure that the measure was reflective of its measure concept as understood by people with different challenges. Fourth, the measure underwent pilot testing in two states (MN, PA). Feasibility was one of the main objectives of the pilot study during which we were able to identify items and response options that were either not providing us with information that reflected the measure concept accurately or those that did not contribute meaningfully to the measure. Based on all these stages to the feasibility assessment, we used an iterative process to refine the measure. The final version was used for psychometric testing in the RTC/OM National Field Study.

Scientific Acceptability

Testing Data

5.1.1 Data Used for Testing

The data presented in this submission were collected on a rolling basis during a multi-year field study of the RTC/OM measures between Spring 2021 and Spring 2024. This was a longitudinal data collection effort with three waves of data collection for each participant. However, only results from the first wave of data collection for each participant are presented unless otherwise noted. The first point of data collection for participants occurred between May 2021 and February 2024.

5.1.2 Differences in Data

To mitigate autocorrelation and other statistical artifacts during data analysis only the first wave of data collection is used for data analysis, with one exception. The one exception is the test-retest analysis which could have been collected during any wave. Test-retest responses were only collected once per participant.

5.1.3 Characteristics of Measured Entities

Providers in the sample were recruited through a national directory of HCBS providers maintained by Medicaid.gov, as well as through networks of known HCBS providers and contacts recruited by University Centers for Excellence in Developmental Disabilities and other organizations contracted to support recruitment and data collection in several states (e.g., Utah, Pennsylvania, Florida, Georgia, Kansas). Additional HCBS providers were referred by participants who responded to national recruitment efforts (e.g., website, newsletters). Providers were not recruited for the study in every state, attempts to expand the representation of the sample were made in every state. This resulted in 67 organizations formally participating in the study across the states of Minnesota, Kansas, Florida, New Jersey, Pennsylvania, Massachusetts, Georgia, Arizona, Kentucky, Iowa, California, and New York. The size of participating organizations, in terms of the number of beneficiaries served, ranged from 10 or fewer to several hundred, with a variety of sizes in between. In Kansas, three large Managed Care Organizations also participated, as HCBS in that state are administered through them. The types of HCBS provided included residential services, in-home supports, home health or skilled nursing, employment services, community access, financial assistance, transportation, and more.

5.1.4 Characteristics of Units of the Eligible Population

HCBS beneficiaries were recruited either directly by participating provider organizations through direct outreach or recruitment materials (e.g., flyers, videos), or through national-level postings and newsletters inviting participation. Each participant was screened to verify eligibility, including age and receipt of HCBS or HCBS-like services. All participants in the study who expressed interest in participating and met inclusionary criteria (age, currently receiving HCBS or HCBS-like services, able to provide consent/assent). Participant ability to understand the measure questions was first evaluated with the University of California, San Diego Brief Assessment of Capacity to Consent (UBACC). Capacity was also closely monitored by interviewers and if significant concerns about the validity of responses were raised, the participant’s data were excluded.

Participants reported their primary disability as Intellectual or Developmental Disability (181, 61.4%), Physical Disability (59, 20%), Traumatic Brain Injury (24, 8.1%), Psychiatric Disability (17, 5.8%), Age-related Disability (4, 1.4%), or Other (10, 3.4%). The age range of participants was between 19 and 76 years old. Participants between ages 18-34 made up 37.9% of the sample, 35-54 were 44.7% of the sample, and the remaining 17.4% of the sample were 55 or older. 160 participants (54.2%) identified as male, 133 participants (45.1%) identified as female, and 2 participants (<1%) identified as “other”. Participants identified their race as White (188, 63.7%), Black or African-American (66, 22.4%), Hispanic/Latino (12, 4.1%), Asian (2, <1%), or “Other race not listed” (5, 1.7%). No participants identified as solely American Indian or Alaska Native. Approximately 7.5% of participants identified with more than one race.

Reliability

5.2.1 Reliability Testing Conducted (instrument)

Person or encounter level (i.e., data element) (e.g., inter–abstractor reliability)

5.2.1a Why Testing Not Conducted

Reliability testing was conducted for each individual IDM.

5.2.2 Method(s) of Reliability Testing

Internal consistency reliability and test-retest reliability methods were used to assess the reliability of person-level outcomes. Internal consistency reliability is a way to test the generalizability of a set of items to the broad domain of items that could have been used on the test. This type of reliability is used to gauge the level of error in content sampling of the items as well as errors of measurement arising from sampling, administration, or other secular effects. On the other hand, test-retest reliability is an estimate of errors around an examinee's “true” score over a short time frame (Crocker & Algina, 2008).

Internal consistency reliability of measure responses was assessed with Cronbach’s alpha, a well-researched and widely-used method that was listed by the NQF as a method to demonstrate scientific acceptability (NQF, 2021). Test-retest reliability was evaluated using PPM correlations between participant composite scores across time points. Test-retest data were collected 10-14 days apart.

All calculations were performed in R (2025; version 4.4.0 or later) with the psych package (Revelle, 2025). A full data matrix of all respondents was loaded into R containing participant responses to all items on the instrument. For Cronbach’s alpha calculations, a correlation matrix from these responses was computed using matrix smoothing and full information maximum likelihood (FIML), the latter an optimal technique for handling missing data (Enders, 2010). The sample size used was the average number of complete responses across all items. For test-retest reliability, missing data were handled by pairwise deletion.

(A complete reference list is provided as a supplemental attachment in section 7.1.)

5.2.3 Reliability Testing Results

The attachment in 5.2.3a provides reliability testing results at the measure level. The document contains results for both Experiences Seeking Employment and Job Experiences measures, respectively. Standardized Cronbach’s alpha and test-retest correlation statistics are highlighted in yellow. The detailed report on internal consistency also includes 95% confidence intervals as well as leave-one-out item analysis. Alongside the test-retest correlation coefficient is also reported the number of subjects and items in the sample.

5.2.3a Attach Additional Reliability Testing Results

Reliability - Employment.pdf

5.2.4 Interpretation of Reliability Results

The Employment Instrument is evaluated at the encounter-level separately for the Experiences Seeking Employment and Job Experiences IDM/sub-measures.

Experiences Seeking Employment

Internal consistency (.71) and test-retest (.80) were adequate and good. This demonstrates this measure produces an acceptable level of measurement error for scores obtained from this portion of the instrument.

Job Experiences

Internal consistency (.79) and test-retest (.92) were good and excellent. This demonstrates this measure produces an low level of measurement error for scores obtained from this portion of the instrument.

Validity

5.3.1 Validity Testing Conducted (instrument)

Person or encounter level (i.e., data element) (e.g., sensitivity and specificity)

5.3.3 Method(s) of Validity Testing

There were two approaches used to perform validity testing at the encounter-level. Content validity and construct validity were the two approaches used. An in-depth discussion of the content validation process undertaken in Study 1 of the RTC/OM project was provided in section 2.6 and will not be repeated here.

Construct validity of measure outcomes was evaluated with parallel analysis (PA) via scree plots as well as exploratory factor analysis (EFA). Both were used to evaluate the factor structure of each measure. Similar to reliability analyses, all calculations were performed in R. First, to determine the number of factors to retain during EFA, parallel analyses were performed then compared with the theoretical structure proposed during measure development. This helped guide the number of factors that were fit during the EFA procedure.

EFA models were fit with the psych package using ordinary least squares . Oblique solutions were produced with Oblimin rotation. Missing data were handled via full information maximum likelihood (FIML) when computing the correlation from the full data matrix (see section 5.2.2).

5.3.4 Validity Testing Results

The attachment in 5.3.4a provides the results of parallel analyses and exploratory factor analyses for both the Experiences Seking Employment and Job Experiences IDMs, respectively. For both Experiences Seeking Employment and Job Experiences IDMs, three different EFA models were fit to the data and the model output is provided. Parallel analyses for both measures recommended retaining four factors in the model. The initial hypothesis for both measures was a one-factor model. To explore model adequacy, one-, two-, and three-factor models were fit to the data.

5.3.4a Attach Additional Validity Testing Results

Validity - Employment.pdf

5.3.5 Interpretation of Validity Results

Similar to Reliability results, the IDMs that make up this Instrument will be evaluated separately as to their outcome validity.

Experiences Seeking Employment

Parallel analysis suggested retaining four factors, although the hypothesized factor structure was a one-factor model. The presence of several factor loadings below .30 and subpar fit statistics (TLI = -0.03, RMSEA = 0.34) show that a higher-order model may be a better fit tot the data. This result is echoed by reliability results that are sufficient for basic research but do not show excellent results compared to other measures. Follow up analysis found that a three-factor model was a much better fit to the data (TLI = .73, RMSEA = .17).

Job Experiences

Parallel analysis suggested retaining four factors, and the hypothesized strcuture was a one-factor model. The best fit to the data was a three-factor EFA (TLI = .90, RMSEA = .06).

Use & Usability

Use
Usability

Use

6.1.1 Current Status

Not in use

6.1.2 Current or Planned Use(s)

Quality Improvement with Benchmarking (external benchmarking to multiple organizations)

Quality Improvement (Internal to the specific organization)

Usability

6.2.1 Actions of Measured Entities to Improve Performance

The measured entities are bound by CMS regulations to provide HCBS services to support QoL outcomes in alignment with HCBS Final Settings Rule and Access Rule. As a minimum, this measure will provide evidence for the level of participation in meaningful activities, including engagement in activities at home and in the community to the extent they desire, expansion of natural supports and community connections, and service quality in supporting engagement in meaningful activities, of their clients as indicators of Access and HCBS. There is currently a gap in measures available to provide reliable and valid data to service providers on HCBS outcomes, including the outcome of job seeking experiences (UMN RTC/OM, 2020). The RTC/OM measures of Job Experiences and Experiences with Seeking Employment have the potential to fill this gap for providers to be able to report to CMS on their progress toward HCBS and Access. Most importantly, providers will be able to make informed judgments about the way people with disabilities receiving HCBS experience their services and their effectiveness to successfully seek employment of their choice.

In our work with PAVE in CA, the measure score will be used to validate a PAVE measure score related to job seeking experineces to assess the quality of services as related to the Quality Incentive Program (QIP) for CA disability service providers. A pilot study is currently under way to collect data using RTC/OM and PAVE measures.

6.2.5a Potential Unintended Consequences

As with the use of any measure to increase outcomes, the use of the RTCOM Employment measure may have potential unintended consequences, particularly if performance on the measure is tied to incentive or value-based payment programs. Poor performance on the measure could lead to reduced resources or negatively influence future funding decisions for provider organizations. In response, entities (HCBS providers)may reallocate resources from other critical areas to focus narrowly on improving measured employment outcomes, potentially undermining other areas of service quality. This can include a tendency to over focus on the measured outcomes and decreasing focus on other important or related outcomes. Another potential consequence is that providers may be held accountable for outcomes that are sometimes outside of their control. Not all individuals served have funding for services that directly support employment, limiting the provider’s ability to influence the measured outcome. Additionally, implementing and reporting on the measure may require significant resources, especially for smaller or under-resourced providers. This may result in a reduction of resources for other performance measurement or quality improvement initiatives. To reduce the risk of these unintended consequences, the measure should be accompanied by clear guidance on appropriate use, interpretation, and its role within organization-wide and broader quality improvement strategies. It should not be used in isolation for value-based funding decisions and careful thought by users and policy makers should be put into equitable use of the measure in such initiatives. Potential unintended consequences for the beneficiary includes the potential for loss of service funding due to the increased income that could result from better employment outcomes. Organizations should carefully track outcomes for individuals to manage the complex relationship between increased income and service funding. If used carefully, the benefits of the RTCOM Employment measure such as improved focus on meaningful employment outcomes and enhanced service quality should outweigh the potential unintended consequences.

Breadcrumb

RTCOM Instrument for Home- and Community-Based Services (HCBS) Recipients - Employment