RTCOM Outcome Measure for HCBS Recipients [Instrument: Employment] - Experiences Seeking Employment

CBE ID

5105-1

1.5 Project

Advanced Illness and Post-Acute Care

1.0 New or Maintenance

New

1.1 Measure Structure

Instrument + Derived Measure Set

1.1a Instrument or Derived Measure

Derived Measure

1.2 Associated Instrument Lookup

RTCOM Instrument for Home- and Community-Based Services (HCBS) Recipients - Employment

Is Under Review

Yes

Next Maintenance Cycle

Spring 2025

1.6 Measure Description

This instrument assesses an individual’s experiences in paid employment as well as their experiences searching for paid employment. The target population for this instrument is adults with disabilities who receive HCBS or HCBS-like services. This is a self-contained instrument that can be administered independently of other RTCOM instruments.

The instrument is administered through an in-person or video-conferencing interview where an interviewer guides an individual through a series of questions (i.e., items). The responses for each item are scored on a Likert scale. The available research supports measure administration every four to six months.

1.6 Measure Description (derived)

The Experiences Seeking Employment measure is one of two proposed measures in the RTC/OM suite of measures focused on employment. This measure assesses individuals in their experiences looking for employment and is comprised of 13 items:

G1. You had bad experiences looking for paid work in the past.

G2. To keep a job (or the job you have), you would need more support than you currently have.

G3. You could find a job (or a different job) if you want one.

S1. Are you currently looking for a job (or a different job) that pays you?

S2. Have you given up looking because you could not find a job [if employed, different job]?

S3. You have had the training/education you need to find a paid job.

S4. In the past, you got the support you needed to keep working.

S5. [If Employed Ask:] You are able to get to work on time. [If Unemployed Ask:] You would be able to get to work on time if you had a job.

S6. [If employed, read:] You would like to continue to work for pay. [If unemployed, read:] You would like to work for pay.

S7. You do not have opportunities to get a job that pays you.

S8. You have opportunities to work at the type of paid jobs you would like.

S9. You might lose important services if you are paid for work.

S10. [If employed, read:] Too much of the money you earn is used to pay for your staff [If unemployed, read:] Too much of the money you earn would be used to pay for your staff.

The items “G1”, “S3”, “S4”, and “S5” are scored 0 to 3 on a frequency scale with response options:

“Never/Rarely”

“Sometimes”

“Often”

“Almost Always/Always”

The items “G2”, “G3”, and “S6” through “S10” are scored 0 to 3 on an agreement scale with response options:

“Strongly Disagree”

“Disagree”

“Agree”

“Strongly Agree”

Finally, the two items “S1” and “S2” are scored on a binary 0/1 scale.

For item “S1”, 1 = “Yes”

For item “S2”, 1 = “No”

Measure Specs

General Information

1.7 Measure Type

Patient-reported Experience Performance Measure (PRE-PM)

1.8 Level of Analysis

Population or Geographic Area

Other

1.8a Population or Geographic Area Level of Analysis

Intellectual and/or Developmental Disability, Physical Disability, Psychiatric Disability

1.8b Other Level of Analysis

Home- and Community-Based Services Provider

1.9 Care Setting

Other

1.9b Other Care Setting

Home- and Community-Based Services Provider

1.10 Measure Rationale

Over the last decade, policy shifts have placed increased attention on integrated employment of people with disabilities. As a society, we have moved from sheltered workshops and day activity centers to supported employment, and most recently through the application of approaches to customized employment, to competitive integrated work opportunities for people with even the most intensive support needed, Yet despite a variety of policies, directives, and legislation, people with disabilities continue to experience limited participation in the labor market. (Almalky, 2020; Butterworth, et al, 2015a; Miethlich & Oldenburg, 2019). Recent estimate indicate that in the US, only one in three (34.9%) individuals with disabilities are employed compared to 76% of their counterparts without disabilities, and this disparity appears to be increasing over time (Bonaccio, et al 2020; Houtenville & Ruiz, 2012; Kraus,2017; Lauer & Houtenville, 2017).

Individuals with disabilities face challenges to employment beyond that experienced by their peers without disabilities. Not only are they less likely to be employed when they desire to be so (Andara et al, 2024) but the quality of their employment is significantly lower than their counterparts. Persons with disabilities consistently report lower levels employment quality than their counterparts without disabilities and are nearly twice as likely to report low-quality employment in the form of employment in which they feel trapped) or is insecure and unrewarding . This gap in employment quality is particularly pronounced for those who reported living with both a physical and mental/cognitive condition (Shahadi, et al, 2023). Given the relative stagnation that has occurred in disability employment over the past two decades, the emphasis in many states and among support providers is on raising overall employment rates. This, however, overlooks the problem that not all jobs are of equal quality, and some jobs offer few of the assumed benefits of paid employment Burgard & Lin, 2013; Burroway, 2017). Therefore, although participation in paid employment is an objective with the potential to support social and economic well-being, the long-term goal should be high-quality employment (Martin Ginis, et al, 2020; Tompa et al, 2022).

Specific external barriers to employment for individuals with disabilities that have been identified include a lack of job opportunities (Grant, 2008), insufficient supports to maintain employment (Shier, et al., 2009), and discrimination and stigmatization in the workplace (Winn & Hay, 2009). Individuals with mental health issues face similar difficulties finding and retaining employment, with an added burden of unique stigmatization due to perception of mental health and safety and when persons with disabilities do have jobs they tend to be underemployed (Stuart, 2006).
Beyond the financial benefits of the employment of people with disabilities, access to inclusive high quality employment that pays a living wage has been shown to be a key determinant of individual health and quality of life outcomes (Dean et al., 2018; Emerson et al., 2018; Randall et al., 2022; Robertson et al., 2019; Voermans et al., 2020), as well as being economically and socially beneficial for employers, communities, and broader society (Cimera, 2010; Taylor, et al., 2021).

Existing employment measures developed for use with people with disabilities typically assess a limited number of aspects of one’s work including rates of employment, type of employment, hours worked, and weekly or monthly earnings (Cimera & Burgess, 2011; Grigal, Hart, & Migliore, 2011). When considering the well-being of people with disabilities, earnings and hours worked are key outcome variables. However, this data only inform one about limited aspects of an individual’s employment experience (Nord, et al., 2013; Martin Ginis, et al, 2020; Migliorie et al., 2012; Tompa et al, 2022) and leave out a wide variety of outcomes strongly associated with both the quality of employment and the quality of life one leads. Second, only a few existing employment measures include items that solicit information about how effectively the provider system supports an individual’s employment. Just as important, few currently available instruments have demonstrated adequate reliability and validity and across different disability populations. A third limitation is that current measures do not account for the outcomes experienced by a large segment of the disability population, who although they desire to work, are not currently employed. Providers therefore have little information about the people they serve who are in the process of applying for work but have yet to be able to find employment or are in the process of applying for a new job. The submitted RTCOM employment measure concepts attempt to fill this gap in HCBS measurement.

The rationale for inclusion of a set of employment performance measures was based on:

(a) A systematic review of the disability employment research literature

(b) Use of a variety of approaches, including a participatory planning and decision-making process employed at a national level, to determine the extent to which stakeholders believed performance scores based on various components of the job experiences construct were important to include in the measure under development

(d) An examination of gaps or limitation inherent in current approaches to measuring outcomes and service quality in this area.

The Quality of Job Experiences and Experiences Seeking Employment measures developed as part of the Research and Training Center on HCBS Outcome Measurement (RTC/OM) were designed to target employment outcomes of people with disabilities at the agency level and be sufficiently sensitive to changes in policy and services that performance scores could be used to document improvement in quality of both services and HCBS recipient outcomes. This level of measurement needs to be more granular than that which is used to focus on compliance and requires measures able to demonstrate reliability, validity, accuracy, and sensitivity at the agency level with the specific groups of HCBS recipient organizations’ serve. The measure is constructed to be person-centered, taking into consideration personal preferences with respect to how one goes about a search for employment, the extent to which they desire support in applying for a job, and for those who work, the level and types of support they need on the job, the quality of interactions they have with co-workers and the general public, the number of hours worked, the extent to which these hours meet their needs, the degree to which person is looking for advancement in their employment, the types/forms of both supervision and job support they prefer as well as from whom, and the specific roles they serve for their employer.

These measure characteristics are necessary in order to ensure the provision of actionable data so that provider agencies can determine the effectiveness with which they are supporting individuals with disabilities to obtain employment, and not just survive in their jobs but thrive in them. Fortunately, the existing research base clearly indicates that people with intellectual and physical disabilities, autism, mental health challenges and TBI can all be matched to job positions in which they can be successful (Gustafsson, et al, 2013; Henry, et al 2019; Houtenville & Kalargyrou, 2012; Khayatzadeh-Mahani, et al 2020 Nevala et al, 2019; Suijkerbuijk, et al 2017) and a variety of approaches to support the employment of people with disabilities are now approaching or have met the criteria for consideration as evidenced-based or promising practices including supported (Wehman, 2023; Wehman et al, 2018) and customized (Inge, et al, 2018; Riesen et al, 2023) approaches to employment. As mentioned previously, however, it should be noted that employment quality is just as important an outcome to consider as the number of people with disabilities who are employed.

Information/data available based on performance scores on our two employment measures have the potential to provide support agencies with a variety of information that can be used to (a) document overall service quality and facilitate policy and/or programmatic changes needed as part of agency quality improvement efforts, (b) identify specific aspects of the Job Experiences subdomain where performance is less than desirable as well as those areas in which the agency is supporting exceptional outcomes, (c) longitudinally track changes that occur in service quality and job experiences, and provide families and persons with disabilities with information they can use to help make informed decisions as to which employment support agencies they desire to provide services to their family member with a disability.

(A complete reference list is provided as a supplemental attachment in section 7.1.)

1.10 Measure Rationale (derived)

Employment: Experiences Seeking Employment

Despite a variety of policies, directives, and legislation, people with disabilities continue to experience limited participation in the labor market. (Almalky, 2020; Butterworth, et al, 2015a; Miethlich & Oldenburg, 2019). Recent estimates indicate that in the US, only one in three (34.9%) individuals with disabilities are employed compared to 76% of their counterparts without disabilities, and this disparity is increasing over time (Bonaccio, et al 2020; Houtenville & Ruiz, 2012; Kraus,2017; Lauer & Houtenville, 2017). Almost 60% of adults with disabilities are unemployed and most of these individuals desire to work. However, at this time, we have little information as to their experiences when they are seeking employment

Individuals with disabilities face challenges to employment beyond that experienced by their peers without disabilities. Barriers to success in seeking employment vary by individual characteristics such as age, gender, race, ethnicity, marital status, and educational attainment (Sevak, et al, 2015). The presence of a disability at older age poses an increased risk for unemployment (Ipsen, 2006) as older persons with a disability are more likely to be discriminated against in hiring and denied workplace accommodations (Cichy, et al., 2015). Lack of education or training can also negatively influence employment rates and wages earned (Fogg, et al., 2010; Kaye, 2010). However, most job seekers with disabilities perceive the nature of their disability as the primary barrier to employment (Kessler Foundation/National Organization on Disability, 2010; Kruse et al., 2010; Bureau of Labor Statistics, 2013).

Individual barriers, however, are not the only challenges faced by people with disabilities. Shifts in the labor market account for some of the employment disparities experienced by people with disabilities (Chan, et al 2010; 2013).

Companies, especially those of small size often view hiring of persons with disabilities as a risk - and are anxious of legal complications, loss of revenue, and costs associated with providing accommodations (Erickson, et al., 2014; Schur, et al., 2014). Questions of employers about finding people with disabilities who are qualified for open positions (Kaye et al 2011) and low quality job offerings that are of little interest to people with disabilities (Bonaccio, et al 2020; Hemphill & Kulik, 2016) also serve as barriers. Additionally, participation in SSDI or the Social Security Income (SSI) programs often serve as a disincentive for individuals with disabilities (Houtenville & Brucker, 2013) as the hours one can work without losing disability benefits often limit the jobs for which one can apply. Social barriers also play a role, as many employers still view people with disabilities as unable to carry out the functions of a job they are trying to fill due to ableism and the stigma associated with disability (Bureau of Labor Statistics, 2013; Erickson et al., 2014). Given the stagnation that has occurred in disability employment over the past two decades, the emphasis in many states and among support providers is on raising overall employment rates. This, however, overlooks the problem that people with disabilities experience greater difficulties finding employers willing to hire them.

When considering the outcomes experienced by people with disabilities seeking employment having a job, earnings and hours worked are key outcome variables. However, these data only inform one about limited aspects of an individual’s employment experience (Martin Ginis, et al, 2020; Migliorie et al., 2012; Tompa et al, 2022) and leave out a wide variety of outcomes strongly associated with the experiences one has while seeking a job and whether meaningful employment can be found. Only a few existing employment measures include items that solicit information about how effectively the vocational rehabilitation system supports an individual in getting a job including experiences identifying employment opportunities, conducting job searches, developing a resumé, preparing for job interviews, and with other aspects of the process one needs to use to find employment. Just as important, few currently available instruments have demonstrated adequate reliability and validity and across different disability populations. Providers therefore have little information about the people they serve who are in the process of applying for work but have yet to be able to find employment or are in the process of applying for a new job. The submitted RTCOM employment measure concepts attempt to fill this gap in HCBS measurement.

The rationale for inclusion of a Seeking Employment IDM in our submitted measures was based on:

(a) A systematic review of the disability employment research literature

(b) Use of a variety of processes, employed at a national level, to determine the extent to which stakeholders believed performance scores based on various components of the job experiences construct were important to include in the measure under development

(d) An examination of gaps or limitations inherent in current approaches to measuring outcomes and service quality in this area.

Of critical relevance related to the inclusion of employment experience measure is its Importance as defined by stakeholders. Importance was determined via three sources/processes. First, we began by considering all the domains and subdomains identified by the NQF Framework for Home & Community Based Services Outcome Measurement (2016). The framework for HCBS outcome measurement developed by the NQF mentioned employment but only included it as an element of the subdomain of Meaningful Activity which is itself a subdomain of the Community Inclusion domain. Second, RTC/OM staff engaged stakeholders on whose lives HCBS has an impact in a series of participatory planning and decision making (PPDM) process in which participants indicated a need for a seeking employment measure. As a third step in the process of developing a rationale for the inclusion of this measure, we solicited the input of measurement and content experts in disability-related fields through the use of technical expert panels that included people with disabilities. Two groups of experts were asked to rate the importance of a Seeking Employment Measure of its relevance, importance, and accuracy. Based on the 1–4-point Likert-type rating scales used for these measures, mean scores for Experiences Seeking Employment for people who are not currently employed but desire to be employed included: Relevancy 3.53/4.00; Importance 3.55/4.00; and Accuracy 3.12/4.0.

The Experiences Seeking Employment measure developed as part of the RTC/OM was designed to target outcomes of people with disabilities at the agency level and be sufficiently sensitive to changes in policy and services that could document improvement in quality of both services and HCBS recipient outcomes. This level of measurement needs to demonstrate reliability, validity, accuracy, and sensitivity at the agency level with specific groups of HCBS recipients each organization serves. The measure is also constructed to be person-centered, taking into consideration personal preferences with respect to how one goes about a search for employment and the extent to which they receive the level and types of support they need. These measure characteristics are necessary in order to ensure the provision of actionable data so that provider agencies can determine the effectiveness with which they are supporting individuals with disabilities to obtain employment, and not just survive in their jobs but thrive in them.

Information/data available based on performance scores on this IDM have the potential to provide employment support agencies with a variety of actionable information that can be used to (a) document overall service quality and facilitate policy and/or programmatic changes needed as part of quality improvement efforts, (b) identify specific aspects of the seeking employment subdomain where performance has fallen below benchmarks that have been established as well as those areas in which the agency is supporting exceptional outcomes, (c) longitudinally track changes that occur in service quality and employment outcomes, and provide families and persons with disabilities with information they can use to help make informed decisions as to which providers they desire to provide employment supports to their family members with a disability.

(A complete reference list is provided as a supplemental attachment in section 7.1.)

1.11 Measure Webpage

http://example.com

1.20 Types of Data Sources

Patient-Reported Data and/or Survey Data

1.20c Format: Patient-Reported Data and/or Survey Data

Non-digital

1.25 Data Source Details

N/A

Numerator

1.14 Numerator

The measure focus for the performance measure outcome is the number of individuals who have a composite score within the measure’s possible inter-quartile range (IQR). An individual's composite score is derived by summing the responses to 13 items on the Employment instrument. Items have response scales with either No/Yes binary response scale or a 4-point Likert scales. The scales are numerically coded 0/1 or 0 to 3, respectively. An individual's composite score can range from 0 to 35 where higher scores correspond to more positive experiences looking for employment opportunities. The IQR is 9 to 26 and the numerator is the number of individuals whose composite scores fall in this range.

1.14a Numerator Details

The numerator is calculated using a tabulation of composite scores from individual respondents. These individuals belong to the relevant population as defined in the denominator for this measure. Other details such as time period for data collection are equivalent to the denominator definitions and will be discussed in section 1.15a.

A composite score for an individual respondent is obtained by administering the Employment instrument to a respondent then calculating a sum score from the responses to 13 items on the instrument (section 1.6). This results in a composite score for one individual. See the attached data dictionary (1.13a) for a list of items, response options, and response scoring. An individual composite score (i.e., sum score) for the Experiences Seeking Employment measure can range between 0 to 35.

Included in the numerator are all individual respondents included in the denominator who have composite scores within the measure-derived inter-quartile range (IQR) for the measure. In other words, composite scores that fall within the middle 50% of possible scores that can be obtained for the measure. For the Experiences Seeking Employment measure this range is 9 to 26.

It is recommended that these calculations be performed prior to scaling of scores to be on a different, public-facing metric, e.g., T-scores.

Measure Calculation

1.13a Attach Data Dictionary

RTCOM Codebook-and-Scoring-Scheme - Employment + ServicesSupports.pdf

1.16 Type of Score

Composite scale

1.16.0 Same Type of Score? (derived)

Same as instrument

1.17 Measure Score Interpretation

Better quality = Higher score

1.17.0 Same Measure Score Interpretation? (derived)

Same as instrument

1.18 Calculation of Measure Score

The measure score for individual recipients is calculated by the sum of item responses on the instrument. See section 1.6 and the attached Data Codebook in 1.13 for the items and scoring codes on this instrument. Stratification does not modify the calculation of measure scores. Note that the measure score interpretation in section 1.17 is only applicable to this individual measure score.

Scores at the accountable-entity level are calculated using individual measure scores and incorporating the numerator and denominator criteria provided in sections 1.14/1.14a and 1.15/1.15a. For a calendar year, accountable entities will obtain individual measure scores for eligible HCBS recipients that they serve. The numerator is the number of individuals who obtained a score in the measure-derived interquartile range (IQR; see 1.14/1.14a). The denominator is the total number of eligible HCBS recipients that were assessed. The accountable entity score is this numerator/denominator ratio, which will be a proportion.

Each individual HCBS recipient should only be counted once in the numerator and denominator. In other words, an HCBS recipient with more than one assessment within a calendar year will still be only counted once for reporting purposes. For these individuals, use their most recent individual measure score that still falls within the calendar year.

A higher score indicates that there is a greater performance gap on measure outcomes for recipients at a given accountable entity. A higher proportion also suggests that the provider serves a wider population with greater variability in service needs. We have purposefully avoided an accountable entity score interpretation that is based on high (or the highest) individual measure scores as this would provide perverse incentives for those using our instruments. We also do not recommend higher or lower accountable entity scores be interpreted as “better” or “worse”. Rather, accountable entity scores should be accurate and informative.

1.18.0 Same Calculation of Measure Score? (derived)

Same as instrument

1.19 Measure Stratification Details

See data dictionary attachment.

The data dictionary contains item and scoring schemes for questions that ask respondents about their Services Needs. These questions are found on the RTC/OM’s Demographic survey. This survey is appended to the instrument information in section 1.13a Attach Data Dictionary.

The Service Needs questions are used to develop a classification variable of a respondent’s functional disability (see section 5.4.2). The functional disability variable is defined as the overall level of services and supports a respondent is currently receiving. During field testing of the RTC/OM instruments, this collection of items was found to be superior in identifying level of disability compared to items that directly asked participants about their functional difficulties (see question 8 in the Demographic survey, attached in section 1.13a). In essence, most participants indicated low levels in functional difficulties despite varying levels of service needs. This indicates that services are effective: they reduce or eliminate the difficulties a person experiences in various areas of their life. As such, level of current service needs is a better indicator of functional disability.

Scores on the service needs items are summed to create a composite score of service needs for a respondent. Higher composite scores indicate greater service needs. Cut points are used to group individuals into functional disability categories:

0 - 10 = Few or no services & supports
11 - 20 = Moderate services & supports
21+ = Intense services & supports

Our research team is cognizant that polychotomizing a numerical scale into a few discrete categories is not best practice, as it reduces the amount of information (i.e., variance explained) in the outcome of interest. However, we chose this categorical approach to make allowances for those who have less technical expertise and want an easy-to-understand display of the relationship between level of functional disability and service needs.

1.21a Data Collection Tool URL(s)

http://example.com

1.21b Attach Data Collection Tool(s)

RTCOM Instruments - Employment + Demographics.pdf

1.22 Proxy Responses

1.23 Survey Respondent

Other

1.23a Other Survey Respondent

HCBS Recipient

1.24 Data Collection and Response Rate

The Employment Measure is reported by HCBS recipients and administered either in person or through a HIPAA-compliant remote video conferencing platform. Currently, the instrument is available only in English; however, an interpreter can be used during the interview to improve accessibility. Response rates may be improved by offering accommodations that increase accessibility, such as environmental modifications for individuals with disabilities and visual displays of response scales.

1.26 Minimum Sample Size

There is no minimum sample size that is needed to calculate a performance score at the individual HCBS recipient level. Performance scores for all of the IDMs submitted are based on instrument scores and it is these scores that are reported.

Given that the instrument was developed were intended to be used at the provider level to document service quality and HCBS recipient outcomes, it is unlikely that the measure would be used to represent the quality of services received and outcome experienced by a single individual. As a result, while performance scores could be calculated on the basis of a single individual they would have little meaning in that isolated context. Focus group results and other discussions we have had with HCBS providers indicate that the most likely future use of the measures will be to (a) document the extent to which providers and sites within provider networks meet established benchmarks, (b) determine the extent to which quality improvement efforts have had a significant impact on service quality and HCBS beneficiary outcomes in targeted areas, and (c) potentially be used in the establishment of “report cards” for agencies within and between provider networks that can be used by external funders and families to determine programmatic strengths and areas in need of improvement and to facilitate informed decision-making when people with disabilities and/or their family members are looking to contract for services.

When used in this manner, sample sizes deemed adequate for analysis and interpretation of the submitted measures will depend on a variety of factors including, but not limited to the questions one desires to answer, comfort levels in committing Type I (false positive) and Type II (false negative) errors, the sampling methods used (i.e., probability vs. non-probability sampling) , the target population and its attributes. Attributes of HCBS samples include a wide variety of factors that impact sample size and include client demographic variables such as disability type, intensity of support needs and their variation within the population of interest, geographic location, the size of the population, the effect size one expects and a variety of other features of clients and the environment that could be relevant for answering a provider’s evaluation questions.

The University of Minnesota’s Institute on Community Integration, home of the RTC/OM has long-term history of working with HCBS service providers on a wide variety of evaluation projects that have included the use of measures similar to the ones we are submitting for review. These experiences suggest that if a provider desires to use performance scores aggregated across individuals to determine changes within that provider organization, a representative sample (adjusted for population size) that includes an absolute minimum of 35-50% of HCBS beneficiaries receiving services is needed to draw valid conclusions regarding quality improvement efforts.

1.26.0 Same Minimum Sample Size? (derived)

Same as instrument

Importance

Evidence

2.1 Attach Logic Model

EM-Logic-Models-Employment_0.docx

2.2 Evidence of Measure Importance

Existing employment measures developed for use with people with disabilities typically assess a limited number of aspects of one’s work including rates of employment, type of employment, hours worked, and weekly or monthly earnings (Cimera & Burgess, 2011; Grigal, Hart, & Migliore, 2011). When considering the well-being of people with disabilities, earnings and hours worked are key outcome variables. However, this data only inform one about limited aspects of an individual’s employment experience (Nord, et al., 2013; Martin Ginis, et al, 2020; Migliorie et al., 2012; Tompa et al, 2022) and leave out a wide variety of outcomes closely associated with both the quality of employment and the quality of life one leads. Second, only a few existing employment measures include items that solicit information about how effectively the provider system supports an individual’s employment. Just as important, few currently available instruments have demonstrated adequate reliability and validity and across different disability populations. A third limitation is that current measures do not account for the outcomes experienced by a large segment of the disability population, who although they desire to work, are not currently employed. Providers therefore have little information about the people they serve who are in the process of applying for work but have yet to be able to find employment, or are in the process of applying for a new job. The submitted RTCOM employment measure concepts attempt to fill this gap in HCBS measurement.

The rationale for inclusion of a set of employment performance measures was based on:

· A systematic review of the disability employment research literature

· Use of a variety of processes, employed at a national level, to determine the extent to which stakeholders believed performance scores based on various components of the job experiences construct were important to include in the measure under development

· Input from several technical expert panels, and

· An examination of gaps or limitation inherent in current approaches to measuring outcomes and service quality in this area.

Of critical relevance related to the inclusion of employment experience measure is its Importance as defined by stakeholders. Importance was determined via three sources/processes. First, we began by considering all the domains and subdomains identified by the NQF Framework for Home & Community Based Services Outcome Measurement (2016). The framework for HCBS outcome measurement developed by the NQF mentioned employment but only included it as an element of the subdomain of Meaningful Activity which is itself a subdomain of the Community Inclusion domain.

University of Minnesota RTC/OM staff engaged stakeholders on whose lives HCBS has an impact in a series of participatory planning and decision-making groups. Stakeholders included: (a) people with disabilities (100 participants), including individuals with IDD, physical disabilities, traumatic brain injury, psychiatric disabilities, and age-related disabilities; (b) family members of people with the above noted disabilities (84 participants); (c) representatives of HCBS provider agencies (89 participants), and (d) state level HCBS program administrators/policy makers (47 participants) for a total of 320 participants in 58 small groups conducted nationally. Stakeholders took part in a participatory planning and decision-making (PPDM) process in which participants provided importance weights for each domain and subdomain of the NQF framework on a scale from 0-100 based on their perceived importance in determining the HCBS outcomes and quality. As part of the PPDM process, stakeholders are first asked to examine the basic domains and subdomains for the framework or theory under consideration and add or rearrange its components in order to refine the model. The large majority (83%) of PPDM groups across all stakeholder types reached consensus that employment was an outcome of sufficient importance as to assume the status of its own domain. Among stakeholder groups composed of with disabilities as well as groups representing families, over 90% of participants (people with disabilities = 94%; families = 91%) supporting this refinement of the NQF model with somewhat lower percentages of support among providers and program administrators. Because the existing NQF framework for HCBS outcome measurement did not include employment as a domain or subdomain but only included it as an element of the subdomain of Meaningful Activity no PPDM importance weightings of this domain can be provided.

As a third step in the process of developing a rationale for the inclusion of this measure, we solicited the input of measurement and content experts in disability-related fields through the use of technical expert panels that included people with disabilities. Two groups of experts were asked to rate the importance of all subdomains in the NQF framework in terms of their relevance, importance, and accuracy. Based on the 1-4 point Likert-type rating scales used for these measures, mean Job Experiences TEP scores for people with disabilities who were employed were as follows: Relevancy 3.50/4.00; Importance 3.53/4.00; and Accuracy 3.12/4.0. Mean scores for Experiences Seeking Employment (i.e., those unemployed but currently job seeking) were Relevancy 3.54/4.00; Importance 3.55/4.00; and Accuracy 3.00/4.0.

The Job Experiences and Experiences Seeking Employment measures developed as part of the Research and Training Center on HCBS Outcome Measurement (RTC/OM) were designed to target employment outcomes of people with disabilities at the agency level and be sufficiently sensitive to changes in policy and services that performance scores could be used to document improvement in quality of both services and HCBS recipient outcomes. This level of measurement needs to be more granular than that which is used to focus on compliance and requires measures able to demonstrate reliability, validity, accuracy and sensitivity at the agency level with the specific groups of HCBS recipient organizations’ serve. The measure is constructed to be person-centered, taking into consideration personal preferences with respect to personal preferences with respect to how one goes about a search for employment, the extent to which they desire support, the level and types of support they need, and for those currently employed he number of hours worked, the extent to which these hours meet their needs, the degree to which person is looking for advancement in their employment, the types/forms of both supervision and job support they prefer as well as from whom, and the specific roles they serve for their employer.

These measure characteristics are necessary in order to ensure the provision of actionable data so that provider agencies can determine the effectiveness with which they are supporting individuals with disabilities to obtain employment, and not just survive in their jobs but thrive in them. Fortunately, the existing research base clearly indicates that people with intellectual and physical disabilities, autism, mental health challenges and TBI can all be matched to job positions in which they can be successful (Gustafsson, et al, 2013; Henry, et al 2019; Houtenville & Kalargyrou, 2012; Khayatzadeh-Mahani, et al 2020 Nevala et al, 2019; Suijkerbuijk, et al 2017). A variety of approaches to support the employment of people with disabilities are now approaching or have met the criteria for consideration as evidenced-based or promising practices including supported (Wehman, 2023; Wehman et al, 2018) and customized (Inge, et al, 2018; Riesen et al, 2023) approaches to supporting employment success. As mentioned previously, however, it should be noted that employment quality is just as important an outcome to consider as the number of people with disabilities who are employed.

Information/data available based on performance scores on our two employment measures have the potential to provide support agencies with a variety of information that can be used to (a) document overall service quality and facilitate policy and/or programmatic changes needed as part of agency quality improvement efforts, (b) identify specific aspects of the Job Experiences subdomain where performance is less than desirable as well as those areas in which the agency is supporting exceptional outcomes, (c) longitudinally track changes that occur in service quality and job experiences, and provide families and persons with disabilities with information they can use to help make informed decisions as to which employment support agencies they desire to provide services to their family member with a disability.

(A complete reference list is provided as a supplemental attachment in section 7.1.)

2.2 Evidence of Measure Importance (derived)

General Importance

Importance is defined as the relevance of measures to the lives of people with disabilities who receive HCBS and the potential of these measures to reveal important aspects of quality of life experienced by individuals across disability groups.

Importance was determined via three sources/processes. First, we began by considering all the domains and subdomains identified by the National Quality Forum’s (NQF) Framework for Home & Community Based Services Outcome Measurement (2016). The framework for HCBS outcome measurement was developed by a panel of NQF experts with the goal of identifying key areas to measure in order to be able to track the effectiveness of HCBS services. The final recommendation of the panel included 11 domains with 2-7 subdomains within each domain.

Next, we engaged stakeholders on whose lives HCBS has an impact in a series of feedback and planning groups. Stakeholders included: (1) Persons with disabilities (100 participants), including individuals with intellectual disability and developmental disabilities, physical disabilities, traumatic brain injury, mental illness, and age-related disabilities; (2) family members (84 participants); (3) providers (89 participants), and (4) program administrators/policy makers (47 participants) for a total of 320 participants in 58 small groups conducted nationally. Stakeholders took part in a participatory planning and decision-making (PPDM) process in which participants weighted each domain and subdomain of the NQF framework on a scale from 0-100 based on their perceived importance in determining the HCBS outcomes and quality.

As a third step in the process, we solicited the input of measurement experts in disability-related fields. Two groups of experts rated all subdomains, including the new subdomains of employment and transportation, in terms of feasibility, usability, and importance. These expert ratings were used in conjunction with stakeholder weighting from PPDM groups to narrow our development process to nine NQF subdomains on which to initially focus our work.

Specific Importance

Over the two decades, policy shifts have placed increased attention on integrated employment of people with disabilities. Yet despite a variety of policies, directives, and legislation, people with disabilities continue to experience limited participation in the labor market. (Boeltzig et al., 2009; Butterworth et al., 2015; U.S. Senate, 2013). People with disabilities face challenges to employment beyond that which is experienced by their peers without disabilities. In general, individuals with IDD are less likely to engage in paid employment than those without IDD status (Grant, 2008; Lemaire & Malik, 2008). Persons with physical disabilities are five times more likely than their nondisabled counterparts to be involuntarily unemployed (Turner & Turner, 2004). Specific external barriers to employment for individuals with IDD that have been identified include a lack of job opportunities (Grant, 2008), insufficient supports to maintain employment (Shier, et al., 2009), and discrimination and stigmatization in the workplace (Winn & Hay, 2009). For persons with traumatic brain injury, employment has typically been considered in terms of its stability (retention) post-injury. Only 40-44% of persons with TBI have been found to retain employment (Ponsford and Spitz, 2015; Ponsford et al., 2014). Individuals with mental health issues face similar difficulties finding and retaining employment, with an added burden of unique stigmatization due to perception of mental health and safety and when persons with disabilities do have jobs they tend to be underemployed (Stuart, 2006).

Distinguishing between the IDMs of Experiences Seeking Employment and Job Experiences is vital as those different aspects of employment represent different concepts support by research literature. Fyffe et al. (2022) investigated what specific factors are associated with not finding or not being able to find employment, including medical conditions, functional limitations, or disability rather than situational reasons associated with workplace engagement, such as “lack of job opportunities.” Assessing job seeking experiences is especially important for young adults with disabilities who are graduating from high schools or transition program (Lindsey, 2011).

During the iterative measure development process our stakeholders clearly indicated and literature supports this distinction that the employment tool needed to separated into two distinct measures: Experiences Seeking Employment and Job Experiences. In this document we are focusing on the former.

(A complete reference list is provided as a supplemental attachment in section 7.1.)

Measure Impact

2.3 Anticipated Impact

Our intent for developing this measure of Experiences of Seeking Employment was to assess the outcomes of experiences and success in finding desired employment (Meltzer et al., 2020); of reduction in employment service needs and in the use of day services (Drake et al., 2009); and of greater efficacy, timeliness, and quality of services to support employment seeking service delivery (Cheng et al., 2018).

(A complete reference list is provided as a supplemental attachment in section 7.1.)

2.5 Health Care Quality Landscape

Most of us, regardless of whether we have a disability, desire to live lives that are characterized as being of high quality and for many, but not all people, this includes employment. The extent to which people with disabilities are able to live the types of lives they desire is often dependent on their employment and the effectiveness of the paid and unpaid support they receive from others in both seeking and maintaining a meaningful job. The capacity to monitor the extent to which a person is able to obtain employment that reflects their personal goals is critical if we are to understand the extent to which community-based services including those intended to assist an individual in seeking employment, are doing what they are intended to do. At one level performance measures are needed to ensure compliance with federal and state regulations governing HCBS. Performance measures, however, are also needed that are person-centered, longitudinal to assess outcomes associated with various aspects of life as well as the quality of support service recipients receive. These measures need to be sufficiently sensitive to change that the impact of policy, funding, and programmatic changes on the outcomes people experience can be determined over time. They would also preferably have the capacity to be used with different disability populations who receive community supports.

As a result of the variety of vocational and employment services for which adult HCBS beneficiaries are eligible, measurement of the quality of supports they receive when seeking employment and the outcomes these individuals experience is far from a straightforward process. A nuanced approach needs to be taken that is responsive to a wide variety of personal and contextual factors. This process needs to be decidedly different than that currently used in medical/healthcare contexts due to the dissimilarities in the constructs measured. The individual/personal challenges a person with a disability faces when seeking employment, the ecological barriers they encounter and the quality of supports they receive in an attempt to acquire employment are both complex and difficult to assess. A second set of critical contextual factors are the policies and regulations under which HCBS employment supports are implemented which vary significantly between states. At the same time, in order to be confident that performance measures associated with HCBS recipients who are seeking employment adequately assess both the quality of the services they receive and the outcomes they experience, data are needed with respect to their reliability, validity, and sensitivity to change. Indicators of quality and unmet support needs as directly perceived by service recipients must be considered paramount when developing, administering, and interpreting results based on these measures.

Over the past twenty-years there has been great interest in assessing the outcomes of HCBS recipients and the quality of services they receive. During this period, CMS has championed the development and maintenance of the Home and Community-Based Services (HCBS) Quality Measure Set (QMS) - a set of nationally standardized quality measures for Medicaid-funded HCBS. The Human Services Research Institute’s National Core Indicators, the Center for Quality Leadership’s Personal Outcome Measures, and the CAHPS Home and Community-Based Services Survey have also been developed and are being used by both states and support providers as means through which to demonstrate compliance with CMS regulations, and in the case of the CQL-POM for quality assurance purposes. Each of these approaches has its strengths. However, all have significant limitations that lead to the need to develop new measurement approaches that address these shortcomings as they relate to people with disabilities seeking employment.

The QMS, developed through the support of CMS is intended as a resource for states and thus focuses on the compliance level. The approach taken, however, is medical in its orientation and as such, “performance measures” are most often conceptualized a single items. While this might be appropriate to measure discrete healthcare outcomes (e.g., number of falls or UTIs experienced by a patient), most psychometricians would argue that it is inadvisable to use when one is attempting to measure latent variables such as the experiences a person has while seeking employment.

The National Core Indicators (NCI/NCI-AD: HSRI https://www.nationalcoreindicators.org) is currently the most widely used tool in the U.S. for the assessment of outcomes associated with the receipt of HCBS and includes a number of employment-related outcomes including several related to job seeking outcomes. The instrument was developed and validated as a state-level compliance measure and does an excellent job when used at that level. It is not, however, intended to be used at the provider or individual level for quality improvement, service plan development, and/or outcome assessment. In addition, although the NCI includes indicators in a variety of areas, it is intended to be administered (and was validated) at the instrument level as opposed to on an indicator-by indicator basis. Users are therefore required to administer items related to all indicators as opposed to only those in which there is a specific interest. It should also be noted that although NCI and NCI-AD have been used with populations beyond those for which they were intended (i.e., people with IDD, physical, and age-related disabilities) these tools have only been validated for use with the limited disability groups noted.

CQL’s Personal Outcome Measures (CQL, 2017) is one of the better developed and validated HCBS Outcome tools and part of a commercially available system of assessment and quality improvement. However, the instrument is time consuming to fully administer limiting its feasibility for many providers.

A third approach to outcome assessment in the human services field that has been championed by CMS is the HCBS CAHPS Survey. The CAHPS is a questionnaire with sixty-nine core items developed for measuring the experiences of people with disabilities who are HCBS recipients. The CAHPS, unfortunately, currently has limited data available with respect to its validity or reliability including that related to both employment experience and experiences while seeking employment. (Nyce, Abery & Ticha, 2020).

In addition to the individual shortcomings of the most widely used HCBS outcome measures, there are additional limitations that cut across these instruments as they relate to outcomes people with disabilities experience when seeking employment. The first of these entails the small percentage of items associated with seeking employment that meet the criteria for person-centeredness. Persons of different ages with different disabilities may need and/or desire more or less support in a variety of different areas when they are looking for a job. They may reject certain forms of employment that just do not personally resonate with them and are likely to have widely different experiences based on a number of contextual factors including geographic area of residence. Current measures, in most cases fail to take these differences into account.

A second shortcoming that cuts across HCBS outcome measures associated with people seeking employment is the lack of evidence that they are sufficiently sensitive to change over time that they can be used in a longitudinal manner. Some developers, such as HSRI (NCI-ASC/NCI-AD) explicitly state that their measures are not intended to be used longitudinally. Others (e.g., CQL, CAHPS) have yet to provide sufficient evidence that, when used in a longitudinal manner, their measures are sufficiently sensitive to change that they can be used as evidence of the effectiveness/efficacy of quality improvement efforts, changes that take place in a HCBS recipient’s life, disability policy or funding or as part of value-based payment systems.

A third reason to think about the development of new outcome measures for employment emanates from the resources needed to administer measures at a time when the human services field is experiencing serious workforce shortages. All of the tools mentioned above are intended to be administered in their entirety as full instruments. They are neither modular in format allowing for administration focused on only one or a few indicators, nor tiered and able to provide both a quick general overview of indicators as well as a more in-depth assessment needed for having utility at the provider level. An HCBS provider that focuses on supporting people with disabilities to obtain employment needs a measure specifically related to that outcome that can support determination of the efficacy of supports and needed areas of quality improvement.

A final overarching rationale for considering the need and development of new measures is that the best developed and most well-researched measures that are currently available in the field are proprietary and part of measurement systems. States as well as large providers typically have the funding to pay for the use of these tools. Provider agencies, especially those that support employment which tend to be on the small side, however, often do not. As a result there is a need for measures at the provider level that are (a) able to be used a little to no cost, (b) person-centered, (c) of a composite nature with the ability to assess latent constructs, (d) based on recent theory and research pertaining to the employment of people with disabilities, (e) easily scored and interpreted, and (f) sufficiently sensitive to change over time so that they can be used longitudinally.

(A complete reference list is provided as a supplemental attachment in section 7.1.)

2.6 Meaningfulness to Target Population

Stakeholders, particularly those who have disabilities and their supporters, should be at the heart of measurement development. In order to ensure that HCBS outcome measures are of high quality, the measure development process must include input from stakeholders and most importantly the intended population with which the measures will be used. Furthermore, we contend that the unique challenges associated with measuring employment outcomes among members of diverse populations of HCBS recipients requires strong stakeholder involvement throughout all stages of development. This process has been affirmed by NIDILRR, ACL, and the Centers for Medicare and Medicaid Services (CMS, 2019). Using a sound HCBS outcome measurement framework that has evidence of content validity provided by stakeholders including people with disabilities; putting all measures developed through multiple expert panel reviews; undertaking cognitive testing with people with a variety of disabilities are necessary strategies or processes in which one must engage to ensure for quality measure development.

The target populations for the submitted measures of employment (Quality of Job Experience and Experiences Seeking Employment) include people with intellectual and developmental disabilities, psychiatric disabilities, physical disabilities, and TBI/ABI as well as age-related disabilities. Although the large majority of people with such disabilities possess the capacity to articulate their thoughts and feelings about the outcomes in question and their importance, it must be recognized that some do not. The intensity of their support needs may be such that they experience difficulty understanding questions and articulating their thoughts and feelings. Not only are such individuals at elevated risk for experiencing poor services and outcomes but most HCBS performance measure programs are not set up to reflect their experiences. The RTC/OM development team therefore instituted a process in which not only HCBS beneficiaries, but other relevant stakeholders, provided input into the measure development process. These consisted of family members of people with the above noted disabilities (who are often direct caregivers on either a part or full-time basis), representatives of support provider agencies, and HCBS program administrators who are responsible for using performance measure data to improve both services and outcomes. At multiple steps along the measure development process, these individuals were consulted and their input incorporated into the measure development process. All of the measures submitted for review were based on:

A systematic review of the disability employment literature
The results of a national content validation study of the National Quality Forum’s HCBS Outcome Measurement Framework using a Participatory Planning and Decision-Making (PPDM) process.
Utilization of multiple Technical Assistance Panels (TEPS) that included people with disabilities as well as members of other stakeholder groups
Input from an RTC/OM Center Advisory Committee composed of people with a variety of disabilities in addition to other stakeholder groups, and
Extensive cognitive testing of measure items and response options, and
Vetting of measures by representatives of provider agencies

A Systematic Review of Employment of People with Disabilities Literature. First, we began by considering the current literature which clearly indicates that people with disabilities desire to have meaningful employment and that work enriches their lives in a number of ways (Dean et al., 2018; Emerson et al., 2018; Randall et al., 2022; Robertson et al., 2019; Voermans et al., 2020), as well as being economically and socially beneficial for employers, communities, and broader society (Cimera, 2010; Taylor, et al., 2021).

PPDM Process. Of critical relevance related to ensuring that HCBS recipients for whom the measures under development were intended viewed them as important, we then examined the domains and subdomains identified by the NQF Framework for Home & Community Based Services Outcome Measurement (2016). The initial framework for HCBS outcome measurement developed by the NQF covered 11 domains as well as 2-7 subdomains within each domain. However, employment was not included in the original framework at either a domain or subdomain level. As a second step in determining importance, relevance, and meaningfulness of including employment measures, RTC/OM staff engaged stakeholders from 29 states on whose lives HCBS has an impact in a series of participatory planning and decision-making (PPDM) groups. Stakeholders included: (a) people representing with disabilities (100 participants), including individuals with IDD, physical disabilities, traumatic brain injury, psychiatric disabilities, and age-related disabilities; (b) family members of people with the above noted disabilities (84 participants); (c) representatives of HCBS provider agencies (89 participants), and (d) state level HCBS program administrators/policy makers (47 participants) for a total of 320 participants who took part in 58 small (4-6 person) PPDM groups.

The PPDM process initially included meeting with homogeneous (with respect to disability and stakeholder type) stakeholder groups and providing them with an opportunity to evaluate the original NQF framework. They were provided with the opportunity to add to it, remove domains and/or subdomains they believed were not important, and then stipulate which personal outcomes and service characteristics were most important to measure. Following stakeholders reaching consensus with respect to the domains and subdomains of the original framework they wanted to add or remove, members of each group took part in a process in which they first independently assigned importance weights for each domain and subdomain of the original NQF framework on a scale from 0-100 (or 0-10 for persons with cognitive challenges) based on their perceived importance in determining the HCBS outcomes and service quality. As part of the PPDM process, stakeholders then discussed their weightings first at the subdomain level and later at the domain level examining why people in their group assigned the importance weightings they did. Stakeholders were then given the opportunity to assign a second set of importance weightings taking into consideration what they had heard during their discussion.

Overall, results from PPDM groups indicated a high degree of stakeholder support for inclusion of a separate Employment Domain in the NQF HCBS Outcome Measurement Framework as well as support for the development of measures for both persons with disabilities currently employed as well as those seeking work but currently unemployed. Across all four stakeholder groups, measurement of employment outcomes and both access to and the quality of vocational supports was viewed as of high importance (People with Disabilities = 94% agreement; Family members = 96% agreement; Providers = 83% agreement, Program administrators = 91% agreement).

Technical Expert Panels. As a second step in the process of developing measures, we solicited the input of people with lived experience with disability (N = 9) as well as measurement and content experts in disability-related fields (N = 12) for a series of technical expert panels (TEPs). Four TEP groups were formed and initially asked to rate the importance of all subdomains in the NQF framework that received the highest importance weightings across stakeholder groups on a 1-5 point Likert-type scale with respect to their feasibility, usability/utility, and importance, as well as provide an overall score for the subdomain. Across stakeholder groups, the Employment domain received the following weigthing with respect to feasibility (Mean = 4.6/5.0), usability/utility (Mean = 4.5/5.0), and importance (Mean = 3.96/5.0), as well as provide an overall score for the subdomain (Mean = 4.3/5.0). This information, in combination with PPDM results, the systematic review of the literature, and analysis of existing HCBS outcome measures indicated both the meaningfulness of the employment construct and perceptions among stakeholders that it is important to measure.

At follow-up TEP meetings following the initial development of items for those subdomains that were selected to be a focus of RTC/OM measures related to employment (Job Experiences & Experiences Seeking Work), TEPs engaged in a similar process using 1-4 point Likert-type rating scales to rate the relevance, importance, accessibility/understandability, and accuracy of each item developed for the employment measures under construction. Mean scores for Quality of Job Experiences for employed people with disabilities were as follows: Relevance (Mean = 3.49/4.00), Importance (Mean = 3.49/4.00); Accessibility/Understandability (Mean = 3.37/4.00); and Accuracy (Mean = 3.12/4.00). Mean TEP scores for Experiences Seeking Work were Relevance (Mean = 3.53/4.00), Importance (Mean = 3.55/4.00); Accessibility/Understandability (Mean = 3.20/4.00); and Accuracy (Mean = 3.12/4.00).

RTC/OM National Advisory Group. In conjunction with the results of PPDM groups and TEPs, ongoing input was also solicited from a national advisory group of HCBS stakeholders composed of 14 individuals including 5 with lived experience with disability. The RTC/OM Center Advisory Committee provided valuable feedback not only with respect to the item content of the measures under development but measure administrator training content and the medium (live versus virtual using HIPAA compliant version of Zoom Meeting) through which interviews would be conducted and the approach to cognitive testing that would be used after item refinement. This group not only provided critical qualitative information stressing the importance of employment measures but stressed the need to also included recommendations for the development of measures associated with employment including people with disabilities having access to affordable and safe transportation, and the availability of employment within one’s community of choice.

Cognitive Testing. The terminology used to describe services or supports can vary based upon disability group membership, the settings in which supports are received, and/or the services and supports themselves. The terms work, job, and employment, for example, have positive, neutral, and sometimes even different meaning, when used in different contexts. For example, is a person with a disability living on their family’s farm and working forty hours/week for no pay but for food, shelter, and clothing “employed,” “a volunteer,” or working but not employed.” Thoroughly testing with people with disabilities, the language and terminology used in measure concepts, and how the measure would typically be implemented (e.g., interview, survey, etc.) is therefore essential to ensure items are universally understood across HCBS recipients.

Cognitive testing (CT) focused on obtaining direct input from respondents to verify their interpretation of items and the words of which they are composed to ensure that these matched the developer’s intent (Ericsson & Simon, 1980; Willis, 2005; Willis, et al., 1991) and used a cognitive testing strategy referred to as the “Think Aloud Method” to address the core cognitive components of item responding: comprehension, retrieval of information needed to answer the item, making a judgment, and reporting a response (Tourangeu, 1984; 2018). This approach provided yet another way to involve people with disabilities in the measure development process.

Focus Groups with Potential Measure Users. It is essential that people with disabilities have multiple opportunities to provide input into the measure development process and resulting measures. The employment measures under development, however, also needed to undergo vetting by representatives of employment provider and vocational rehabilitation agencies. After final performance measure refinement based on the results of piloting and field-testing the final measures under submission were presented to two separate groups of potential users utilizing a focus group format. Participants were recruited from a large human service organization in Minnesota that provides residential, home health, and employment services, and a statewide network of human services providers located in Michigan that provide residential, and employment supports (Total N = 23). Measures were initially shared with participants several days prior to scheduled focus groups. Groups were initiated with RTC/OM providing background information and answering questions about the measures themselves, their administration, analysis, and use. After facilitators were assured that participants’ questions had been answered group discussion focused on, (a) the importance of the measures, (b) their overall quality and comprehensiveness, (c) the feasibility of provider agencies using the measures, and (d) the utility of the performance measures developed and how providers could foresee using them. {Overall, the employment measures developed were viewed as important, high in quality, feasible to administer and capable of providing actionable information. The sole negative response to the measures was concern over with whom this information would be shared if the measures were employed on a widescale basis.

(A complete reference list is provided as a supplemental attachment in section 7.1.)

2.6 Meaningfulness to Target Population (derived)

Stakeholders, particularly those who have disabilities and their supporters, should be at the heart of measurement development. In order to ensure that home and community-based services (HCBS) outcome measures are of high quality, the measure development process must include input from stakeholders and most importantly the intended population with which the measures will be used. Furthermore, we contend that the unique challenges associated with measuring outcomes among members of diverse populations of HCBS recipients requires strong stakeholder involvement throughout all stages of development.

The target populations for the submitted measure for Experiences Seeking Employment includes people with IDD, psychiatric disabilities, physical disabilities, and TBI/ABI. The RTC/OM development team used a process in which not only HCBS recipients seeking employment provided input into the measure development process, but other stakeholders were included as well. These consisted of family members of people with the above noted disabilities, representatives from vocational rehabilitation and employment support agencies, and HCBS program administrators. At multiple steps along the measure development process, these individuals were consulted and their input incorporated into the measure development process. The employment measures submitted for review were based on:

The results of a national content validation study of the National Quality Forum’s HCBS Outcome Measurement Framework that utilized a Participatory Planning and Decision-Making (PPDM) process.
Utilization of multiple Technical Assistance Panels (TEPS) that included people with disabilities as well as members of other stakeholder groups
Input from an RTC/OM Center Advisory Committee composed of people with a variety of disabilities in addition to other stakeholder groups, and
Extensive cognitive testing of measure items and response options, and
Vetting of measures by representatives of provider agencies

Additional information about each of these processes and how they contributed to the development of the submitted Experiences Seeking Employment measure can be found in 5105.

NQF Framework and PPDM Results. Overall, results from PPDM groups indicated a high degree of stakeholder support for the inclusion for a stand-alone NQF domain for employment that included subdomains of Quality of Employment Experiences and Experiences Seeking Employment.

Technical Expert Panels. As a second step in the process of developing measures, we solicited the input of technical expert panels that using 1–4-point Likert-type scales rated Experiences Seeking Employment for people who are not currently employed as follows: Relevancy = 3.53/4.00; Importance = 3.55/4.00; and Accuracy = 3.12/4.0.

Cognitive Testing. The Experiences Seeking Employment measure was thoroughly tested with 23 people with disabilities who did not have a current job but were interested in one using the “Think Aloud Method” to address the core cognitive components of item responding as included in the Cognitive Aspects of Survey Methodology (CASM) model the language and terminology used in measure concepts, their completeness, and the extent to which response options provided a person the ability to accurately indicate their perspective on an item.

Focus Groups with Potential Measure Users. After final performance measure refinement based on the results of piloting and field-testing the final measures under submission were presented to two separate groups of potential users utilizing a focus group format. Participants were recruited from a large human service organization in Minnesota, and a statewide network of human services providers located in Michigan (Total N = 23). Group discussion focused on, (a) the importance of the measures, (b) their overall quality and comprehensiveness, (c) feasibility of measure use, and (d) measure utility.

(A complete reference list is provided as a supplemental attachment in section 7.1.)

Feasibility

Feasibility
Proprietary Information

Feasibility

4.1a Data Structure and Availability

The composite scores (i.e., measure scores, see section 1.18) from both the Job Experiences and the Experiences Seeking Employment measure will be available through electronic platforms, such as Qualtrics or incorporated into existing electronic systems providers already use. It is important to understand that HCBS data are by definition not medical data and therefore require different electronic systems from the usual hospital managed system. The HCBS field is not as centralized at the provider level and their electronic systems are developing.

That said, in addition to working with the systems providers may already been using, we are in the process of developing an electronic system at the University of Minnesota’s Institute on Community Integration that houses this initiative. This system will offer providers an opportunity to utilize our electronic system and technical assistance to use the system to house their scores for a negotiated fee.

4.1b Implementation Costs and Burden

Administering the measure as an interview requires approximately 10 minutes per Home and Community-Based Services (HCBS) beneficiary. The primary human resource burden includes the time of a trained interviewer to administer the measure and a data entry specialist or the interviewer themselves to enter the responses into a secure data system. Minimal additional personnel time is needed for data validation, which may involve reviewing and cleaning inconsistent or incomplete data, and follow-up with interviewers to clarify discrepancies. While these tasks include some investment of staff time, they are essential to ensuring data quality and integrity. To support accurate and valid data collection, organizations may also use supplementary resources such as interviewer training sessions, standardized data entry protocols, and processes to conduct periodic checks of the data. Despite the staff requirements, the burden on respondents was minimal. During testing, the vast majority of HCBS recipients reported a positive experience with the interviews. Many beneficiaries expressed appreciation for the opportunity to share their perspectives and discuss their services and outcomes with interviewers.

4.1c Confidentiality

As part of one of the nation’s prominent research universities with a large academic health sciences program, the Institute on Community Integration and RTC/OM have access to a wide variety of servers/data storage systems that meet or exceed all HIPAA security requirements and can be used to ensure the privacy/confidentiality of personally identified information. All clinical and human subjects’ data collected as part of RTC/OM performance measures will be secured with a University of Minnesota-approved resource at all times for the full extent of their life. This is true even when data has been de-identified. University approved methods of storing, clinical and human subjects’ data that will be used by the implementer (RTC/OM) include servers exclusively devoted to and supported by the University’s Health Sciences Technology (HST) group. HST supports University departments that need to store Private Highly Restricted data. The HST Operations and Infrastructure team supports and manages applications and infrastructure to meet University standards regarding HIPAA compliance. HST will work with RTC/OM staff to identify onboarding and maintenance requirements necessary to comply with UIS standards for PHR systems and data. Operating systems include professionally managed RHEL (Linux) and Windows Server environments. HST offers multiple storage options for data, documents, and folders to meet a variety of clinical, research, and storage needs. Network storage (SMB/NFS) is available on the Twin Cities campus for Healthcare Component (HCC) departments. The University’s Health Information Privacy & Compliance Office (HIPCO) and its Institutional Review Board RB strongly encourage the use of limited datasets to maintain confidentiality when some identifiers are needed and this recommendation will be followed. A Limited Data Set is a dataset that contains a limited set of indirect identifiers, and it's only used within the University's Health Care Components or under a Data Use Agreement (DUA). This will allow implementers (RTC/OM) to access data with limited privacy risks while still being able to provide critical service quality and outcome data to RTC/OM measure users in a way that has a high degree of utility. Key aspects of creating a Limited Data Set will entail:

(a) Removing Direct Identifiers: All direct identifiers including names, street addresses, and telephone numbers will be removed.

(b) Including Limited Indirect Identifiers: Certain indirect identifiers, such as dates (e.g., date of birth, death, admission), and geographic information (city, state, zip code) will be included

(c) Creating Data Use Agreements (DUAs): When PHI in a Limited Data Set is shared with a third party, a Data Use Agreement (DUA) will first be established and approved by the UMN’s IRB, Sponsored Projects Administration and Office of General Counsel. This agreement will outline the permissible uses of the data and ensures compliance with HIPAA.

(d) Review and Approval Processes: The process for creating and sharing the Limited Data Set will include review by HIPCO, the IRB, and the Sponsored Projects Administration (SPA).

4.3 Feasibility Informed Final Measure

The final measure is a result of multiple feasibility assessments. First, we conducted Participatory Planning and Decision Making (PPDM) groups with people with disabilities, family members, and staff to weigh importance of the NQF domains and subdomains. This process assisted our team with the selection of measure concepts to prioritize for developing within HCBS. Second, the items and their response options from which the measure was composed was reviewed by a technical expert panel to test whether the measure reflected the intent behind the measure concept. Third, the measure underwent cognitive testing during which the items and their response options from which the measure was composed were tested by people with intellectual and developmental disabilities, aging needs, TBI, mental health needs, and physical disabilities. We wanted to assure that the measure was reflective of its measure concept as understood by people with different challenges. Fourth, the measure underwent pilot testing in two states (MN, PA). Feasibility was one of the main objectives of the pilot study during which we were able to identify items and response options that were either not providing us with information that reflected the measure concept accurately or those that did not contribute meaningfully to the measure. Based on all these stages to the feasibility assessment, we used an iterative process to refine the measure. The final version was used for psychometric testing in the RTC/OM National Field Study.

Scientific Acceptability

Testing Data

5.1.1 Data Used for Testing

The data presented in this submission were collected on a rolling basis during a multi-year field study of the RTC/OM measures between Spring 2021 and Spring 2024. This was a longitudinal data collection effort with three waves of data collection for each participant. However, only results from the first wave of data collection for each participant are presented unless otherwise noted. The first point of data collection for participants occurred between May 2021 and February 2024.

5.1.2 Differences in Data

To mitigate autocorrelation and other statistical artifacts during data analysis only the first wave of data collection is used for data analysis, with one exception. The one exception is the test-retest analysis which could have been collected during any wave. Test-retest responses were only collected once per participant.

5.1.3 Characteristics of Measured Entities

Providers in the sample were recruited through a national directory of HCBS providers maintained by Medicaid.gov, as well as through networks of known HCBS providers and contacts recruited by University Centers for Excellence in Developmental Disabilities and other organizations contracted to support recruitment and data collection in several states (e.g., Utah, Pennsylvania, Florida, Georgia, Kansas). Additional HCBS providers were referred by participants who responded to national recruitment efforts (e.g., website, newsletters). Providers were not recruited for the study in every state, attempts to expand the representation of the sample were made in every state. This resulted in 67 organizations formally participating in the study across the states of Minnesota, Kansas, Florida, New Jersey, Pennsylvania, Massachusetts, Georgia, Arizona, Kentucky, Iowa, California, and New York. The size of participating organizations, in terms of the number of beneficiaries served, ranged from 10 or fewer to several hundred, with a variety of sizes in between. In Kansas, three large Managed Care Organizations also participated, as HCBS in that state are administered through them. The types of HCBS provided included residential services, in-home supports, home health or skilled nursing, employment services, community access, financial assistance, transportation, and more.

5.1.4 Characteristics of Units of the Eligible Population

HCBS beneficiaries were recruited either directly by participating provider organizations through direct outreach or recruitment materials (e.g., flyers, videos), or through national-level postings and newsletters inviting participation. Each participant was screened to verify eligibility, including age and receipt of HCBS or HCBS-like services. All participants in the study who expressed interest in participating and met inclusionary criteria (age, currently receiving HCBS or HCBS-like services, able to provide consent/assent). Participant ability to understand the measure questions was first evaluated with the University of California, San Diego Brief Assessment of Capacity to Consent (UBACC). Capacity was also closely monitored by interviewers and if significant concerns about the validity of responses were raised, the participant’s data were excluded.

Participants reported their primary disability as Intellectual or Developmental Disability (181, 61.4%), Physical Disability (59, 20%), Traumatic Brain Injury (24, 8.1%), Psychiatric Disability (17, 5.8%), Age-related Disability (4, 1.4%), or Other (10, 3.4%). The age range of participants was between 19 and 76 years old. Participants between ages 18-34 made up 37.9% of the sample, 35-54 were 44.7% of the sample, and the remaining 17.4% of the sample were 55 or older. 160 participants (54.2%) identified as male, 133 participants (45.1%) identified as female, and 2 participants (<1%) identified as “other”. Participants identified their race as White (188, 63.7%), Black or African-American (66, 22.4%), Hispanic/Latino (12, 4.1%), Asian (2, <1%), or “Other race not listed” (5, 1.7%). No participants identified as solely American Indian or Alaska Native. Approximately 7.5% of participants identified with more than one race.

Reliability

5.2.1 Reliability Testing Conducted (instrument)

Person or encounter level (i.e., data element) (e.g., inter–abstractor reliability)

5.2.1a Why Testing Not Conducted

Reliability testing was conducted for each individual IDM.

5.2.2 Method(s) of Reliability Testing

Internal consistency reliability and test-retest reliability methods were used to assess the reliability of person-level outcomes. Internal consistency reliability is a way to test the generalizability of a set of items to the broad domain of items that could have been used on the test. This type of reliability is used to gauge the level of error in content sampling of the items as well as errors of measurement arising from sampling, administration, or other secular effects. On the other hand, test-retest reliability is an estimate of errors around an examinee's “true” score over a short time frame (Crocker & Algina, 2008).

Internal consistency reliability of measure responses was assessed with Cronbach’s alpha, a well-researched and widely-used method that was listed by the NQF as a method to demonstrate scientific acceptability (NQF, 2021). Test-retest reliability was evaluated using PPM correlations between participant composite scores across time points. Test-retest data were collected 10-14 days apart.

All calculations were performed in R (2025; version 4.4.0 or later) with the psych package (Revelle, 2025). A full data matrix of all respondents was loaded into R containing participant responses to all items on the instrument. For Cronbach’s alpha calculations, a correlation matrix from these responses was computed using matrix smoothing and full information maximum likelihood (FIML), the latter an optimal technique for handling missing data (Enders, 2010). The sample size used was the average number of complete responses across all items. For test-retest reliability, missing data were handled by pairwise deletion.

(A complete reference list is provided as a supplemental attachment in section 7.1.)

5.2.3 Reliability Testing Results

The attachment in 5.2.3a provides reliability testing results at the measure level. The document contains results for both Experiences Seeking Employment and Job Experiences measures, respectively. Standardized Cronbach’s alpha and test-retest correlation statistics are highlighted in yellow. The detailed report on internal consistency also includes 95% confidence intervals as well as leave-one-out item analysis. Alongside the test-retest correlation coefficient is also reported the number of subjects and items in the sample.

5.2.3a Attach Additional Reliability Testing Results

Reliability - Employment.pdf

5.2.4 Interpretation of Reliability Results

The Employment Instrument is evaluated at the encounter-level separately for the Experiences Seeking Employment and Job Experiences IDM/sub-measures.

Experiences Seeking Employment

Internal consistency (.71) and test-retest (.80) were adequate and good. This demonstrates this measure produces an acceptable level of measurement error for scores obtained from this portion of the instrument.

Job Experiences

Internal consistency (.79) and test-retest (.92) were good and excellent. This demonstrates this measure produces an low level of measurement error for scores obtained from this portion of the instrument.

5.2.1 Accountable Entity Level Reliability Testing Conducted (derived)

Not applicable/accountable entity–level reliability testing not conducted

5.2.1a Why Testing Not Conducted (derived)

Reliability testing for this measure was restricted to encounter-level reliability analyis.

Validity

5.3.1 Validity Testing Conducted (instrument)

Person or encounter level (i.e., data element) (e.g., sensitivity and specificity)

5.3.3 Method(s) of Validity Testing

There were two approaches used to perform validity testing at the encounter-level. Content validity and construct validity were the two approaches used. An in-depth discussion of the content validation process undertaken in Study 1 of the RTC/OM project was provided in section 2.6 and will not be repeated here.

Construct validity of measure outcomes was evaluated with parallel analysis (PA) via scree plots as well as exploratory factor analysis (EFA). Both were used to evaluate the factor structure of each measure. Similar to reliability analyses, all calculations were performed in R. First, to determine the number of factors to retain during EFA, parallel analyses were performed then compared with the theoretical structure proposed during measure development. This helped guide the number of factors that were fit during the EFA procedure.

EFA models were fit with the psych package using ordinary least squares . Oblique solutions were produced with Oblimin rotation. Missing data were handled via full information maximum likelihood (FIML) when computing the correlation from the full data matrix (see section 5.2.2).

5.3.4 Validity Testing Results

The attachment in 5.3.4a provides the results of parallel analyses and exploratory factor analyses for both the Experiences Seking Employment and Job Experiences IDMs, respectively. For both Experiences Seeking Employment and Job Experiences IDMs, three different EFA models were fit to the data and the model output is provided. Parallel analyses for both measures recommended retaining four factors in the model. The initial hypothesis for both measures was a one-factor model. To explore model adequacy, one-, two-, and three-factor models were fit to the data.

5.3.4a Attach Additional Validity Testing Results

Validity - Employment.pdf

5.3.5 Interpretation of Validity Results

Similar to Reliability results, the IDMs that make up this Instrument will be evaluated separately as to their outcome validity.

Experiences Seeking Employment

Parallel analysis suggested retaining four factors, although the hypothesized factor structure was a one-factor model. The presence of several factor loadings below .30 and subpar fit statistics (TLI = -0.03, RMSEA = 0.34) show that a higher-order model may be a better fit tot the data. This result is echoed by reliability results that are sufficient for basic research but do not show excellent results compared to other measures. Follow up analysis found that a three-factor model was a much better fit to the data (TLI = .73, RMSEA = .17).

Job Experiences

Parallel analysis suggested retaining four factors, and the hypothesized strcuture was a one-factor model. The best fit to the data was a three-factor EFA (TLI = .90, RMSEA = .06).

5.3.2 Type of Accountable Entity Level Validity Testing Conducted (derived)

Systematic assessment of face validity of the measure’s performance score as an indicator of quality or resource use

5.3.3 Method(s) of Accountable Entity Level Validity Testing (derived)

At the accountable entity-level, the RTC/OM team led a focus group with six leaders of HCBS organizations across Minnesota to evaluate the face validity of our measures.

5.3.4 Accountable Entity Level Validity Testing Results (derived)

In 2024, a focus group was conducted with six leaders of Home and Community-Based Services (HCBS) organizations in Minnesota. During the session, participants received information about the measure under discussion, including the associated outcome area (Experiences Seeking Employment) and the method used to calculate the performance score. Participants were then asked to provide input on several aspects of the performance score using a scale ranging from 1-4:

Relevance: How relevant is the performance score to the HCBS services they provide? (Mean = _3.83_ )
Representation: To what extent does the score represent the employment outcomes it is intended to measure? (Mean = _3.25_ )
Accuracy: Does the score accurately reflect the outcomes experienced by their beneficiaries? (Mean = _3.2_ )
Feasibility: Would their organization have the capacity to collect the data needed for this measure? (Mean = _3.33_ )
Usefulness: How useful would the performance score be in helping the organization support its beneficiaries? (Mean = _3.4_ )

5.3.5 Interpretation of Accountable Entity Level Validity Results (derived)

These results indicate that, overall, potential users of the measure rated the performance score as having adequate face validity. This demonstrates that HCBS professionals support the idea that the measure adequately captures participant's experiences looking for work.

Risk Adjustment

5.4.1 Methods Used to Address Risk Factors

Stratification by risk factor category

5.4.2 Conceptual Model Rationale

The National Quality Forum (NQF) emphasizes the importance of risk adjustment in evaluating outcome measures to ensure that potential threats to validity are addressed. Sociodemographic factors such as income and race have been explored as possible elements for risk adjustment by the NQF, which aims to develop guidelines in this area. In 2017, the NQF reviewed 303 submitted measures to assess their applicability for adjusting social risk factors that could affect health outcomes. The NQF panel recommended that these social risk factors follow the same criteria as clinical and health-related risk factors, although it noted a lack of a conceptual framework for their inclusion (National Quality Forum, 2014).

In the context of risk adjustment for this study, the NQF panel advised that sociodemographic factors should: (1) have a conceptual link to the outcome,(2) show an empirical relationship to the outcome , (3) display variability, (4) exist prior to intervention or care, (5) remain unaffected by intervention or policy changes, (6) be resistant to change, (7) be based on data that can be easily collected, (8) uniquely explain variations in the outcome, (9) contribute to the overall model, and (10) be considered valid and acceptable (NQF, 2014). These guidelines help differentiate risk adjusters from other variables.

More recently, the NQF conducted a review and convened a technical expert panel to develop further guidance for developers of outcome measures (National Quality Forum, 2020). They found that social risk factors mostly emerged at the individual and community levels, derived from various socioeconomic and demographic indicators. Functional risk factors, however, were often specific to individuals and based on self-reported survey data, with fewer clear definitions available. Statistical methods such as regression analyses were frequently used, though other models like hierarchical linear modeling were also applied to accommodate a broader range of risk factors.

Other risk adjustment models reflect similar themes to the NQF’s recommendations. For example, the Centers for Medicare and Medicaid Services (CMS) sought expert input on risk adjustment, and the Department of Health and Human Services (HHS) reviewed and integrated those recommendations into 10 key principles for risk adjustment (Centers for Medicare and Medicaid Services, 2016). Five of these principles overlap with the NQF’s guidelines, including recommendations that risk adjusters should be clinically relevant, predictive of medical costs, based on adequate sample sizes, encourage specific coding, and maintain internal consistency.

Additional support for the NQF’s guidelines comes from the Research Agency for Healthcare Research and Quality (Velentgas, n.d.), which recommended that risk adjustment should not include variables affected by the outcomes, that variable selection should be based on prior knowledge of their relationship to outcomes, and that risk adjusters should have statistical ties to outcomes.

In our systematic review of studies involving risk adjustment for individuals with disabilities receiving home or community-based services, we categorized studies based on the type of risk adjusters used and their relation to specific outcomes. Panels helped prioritize the risk adjusters, and our findings suggest that four factors—chronic conditions, functional disability, mental health status, and cognitive functioning—may be recommended as candidate risk adjusters (Houseworth et al., 2022).

We have not currently collected data related to mental health status and cognitive functioning due to feasibility issues during the pilot study. We did collect data allowing us to stratify by functional disability/chronic conditions. Therefore, we are currently unable to determine if large difference between providers on those factors would impact performance scores, as the literature suggest. This could lead to some inappropriate conclusions.

5.4.2a Attach Conceptual Model

StratModESE.png

5.4.3 Variable Distribution Across Measured Entities

The attachment in 5.4.4a contains descriptive statistics across levels of functional disability for the measured outcomes. There are three levels of functional disability ranging from low to high service needs. Descriptive statistics reported for each level of functional disability are means, medians, standard deviations, minimum scores, and maximum scores.

5.4.4 Risk/Case-Mix Adjustment Modeling and/or Stratification Results

The attachment in 5.4.4a also contains an ANOVA analysis to determine if there were significant differences between the aforementioned functional disability categories on the measure outcome. No statistically-significant differences were found for the Experiences Seeking Employment measure outcome.

5.4.4a Attach Risk/Case-mix Adjustment Modeling and/or Stratification Specifications

Stratification - Experiences Seeking Employment.pdf

5.4.6 Interpretation of Risk/Case-mix Factor Findings

ANOVA results found no significant differences between level of functional disability and people's experiences looking for work. See section 5.4.2 as to why only functional disability was included in this analysis. This is a novel measure that looks at the experiences of people looking for a paid job, so we had no expectations as to the direction of the results.

5.4.7 Final Approach to Address Risk Factors

Stratification by risk factor category

Use & Usability

Use
Usability

Use

6.1.1 Current Status

Not in use

6.1.1 Current Status (derived)

In use

6.1.2 Current or Planned Use(s)

Quality Improvement with Benchmarking (external benchmarking to multiple organizations)

Quality Improvement (Internal to the specific organization)

6.1.2.0 Same Current or Planned Use(s)? (derived)

Same as instrument

Usability

6.2.1 Actions of Measured Entities to Improve Performance

The measured entities are bound by CMS regulations to provide HCBS services to support QoL outcomes in alignment with HCBS Final Settings Rule and Access Rule. As a minimum, this measure will provide evidence for the level of participation in meaningful activities, including engagement in activities at home and in the community to the extent they desire, expansion of natural supports and community connections, and service quality in supporting engagement in meaningful activities, of their clients as indicators of Access and HCBS. There is currently a gap in measures available to provide reliable and valid data to service providers on HCBS outcomes, including the outcome of job seeking experiences (UMN RTC/OM, 2020). The RTC/OM measures of Job Experiences and Experiences with Seeking Employment have the potential to fill this gap for providers to be able to report to CMS on their progress toward HCBS and Access. Most importantly, providers will be able to make informed judgments about the way people with disabilities receiving HCBS experience their services and their effectiveness to successfully seek employment of their choice.

In our work with PAVE in CA, the measure score will be used to validate a PAVE measure score related to job seeking experineces to assess the quality of services as related to the Quality Incentive Program (QIP) for CA disability service providers. A pilot study is currently under way to collect data using RTC/OM and PAVE measures.

6.2.1 Actions of Measured Entities to Improve Performance (derived)

The measured entities are bound by CMS regulations to provide HCBS services to support QoL outcomes in alignment with HCBS Final Settings Rule and Access Rule. As a minimum, this measure will provide evidence for the level of participation in meaningful activities, including engagement in activities at home and in the community to the extent they desire, expansion of natural supports and community connections, and service quality in supporting engagement in meaningful activities, of their clients as indicators of Access and HCBS. There is currently a gap in measures available to provide reliable and valid data to service providers on HCBS outcomes, including the outcome of job seeking experiences (UMN RTC/OM, 2020). The RTC/OM measure of Experiences with Seeking Employment has the potential to fill this gap for providers to be able to report to CMS on their progress toward HCBS and Access. Most importantly, providers will be able to make informed judgments about the way people with disabilities receiving HCBS experience their services and their effectiveness to successfully seek employment of their choice.

(A complete reference list is provided as a supplemental attachment in section 7.1.)

6.2.4 Progress on Improvement

Unavailable since this is a new measure and no trends are currently available.

6.2.5a Potential Unintended Consequences

As with the use of any measure to increase outcomes, the use of the RTCOM Employment measure may have potential unintended consequences, particularly if performance on the measure is tied to incentive or value-based payment programs. Poor performance on the measure could lead to reduced resources or negatively influence future funding decisions for provider organizations. In response, entities (HCBS providers)may reallocate resources from other critical areas to focus narrowly on improving measured employment outcomes, potentially undermining other areas of service quality. This can include a tendency to over focus on the measured outcomes and decreasing focus on other important or related outcomes. Another potential consequence is that providers may be held accountable for outcomes that are sometimes outside of their control. Not all individuals served have funding for services that directly support employment, limiting the provider’s ability to influence the measured outcome. Additionally, implementing and reporting on the measure may require significant resources, especially for smaller or under-resourced providers. This may result in a reduction of resources for other performance measurement or quality improvement initiatives. To reduce the risk of these unintended consequences, the measure should be accompanied by clear guidance on appropriate use, interpretation, and its role within organization-wide and broader quality improvement strategies. It should not be used in isolation for value-based funding decisions and careful thought by users and policy makers should be put into equitable use of the measure in such initiatives. Potential unintended consequences for the beneficiary includes the potential for loss of service funding due to the increased income that could result from better employment outcomes. Organizations should carefully track outcomes for individuals to manage the complex relationship between increased income and service funding. If used carefully, the benefits of the RTCOM Employment measure such as improved focus on meaningful employment outcomes and enhanced service quality should outweigh the potential unintended consequences.

RTCOM Outcome Measure for HCBS Recipients [Instrument: Employment] - Experiences Seeking Employment

Public Comment

PQM powered by Battelle

Breadcrumb

RTCOM Outcome Measure for HCBS Recipients [Instrument: Employment] - Experiences Seeking Employment

Public Comment