Home and Community-Based Services (HCBS) Consumer Assessment of Healthcare Providers and Systems (CAHPS®) Measures

Data Collection: 

• Despite a substantial amount of training and an extensive guide provided to survey vendors, all did not follow the data collection instructions exactly. These aspects can be reinforced when reviewing and modifying the materials for future administrations. 

• In addition, implementers will need to be thoroughly educated about skip patterns in the HCBS CAHPS survey instrument, applicability of questions to their programs, and how to explain this to data collectors and survey programmers (who will need to take these patterns into effect when analyzing the data). Some of these skip patterns may be adapted to specific states, in which case additional work will be required with survey vendors (e.g., to explain why the skip patterns were adapted and conduct additional review of the field disks to ensure the surveys were appropriately adapted). 

• It will be important for states to provide clear specifications about the nature of the work and realistic information about the context in which vendors will need to work. This is especially critical if they decide to use a survey vendor that is not familiar with the data collection instrument or HCBS populations.

Sampling: 

• We recommend screening the sample for deceased individuals to the greatest extent possible. 

Response Rate: 

• Many participants in Medicaid HCBS programs have guardians from whom consent for the person's participation in a survey must be secured. For many states, this information is not centrally or readily available, or not updated. Accessing this information prior to contact will help increase participation.

• The AAPOR response rates considers individuals who are deceased or who are physically or mentally unable to respond as eligible respondents resulting in lower response rates. An alternative is to calculate a response rate that does not include such individuals as eligible respondents. 

• To avoid alarming potential survey participants and to enhance the recruitment process, any pre-notification letters to the participant should clearly identify the primary survey vendor.  

• Programs should employ additional strategies for recruiting challenging populations, including using proxies.  Additional outreach can involve case/care managers, or states might enlist advocacy groups to communicate to beneficiaries the importance of participating in the survey. 

While field test response rates were less than optimal --22.0 percent, on average, they ranged from a high of 27.7 percent for participants from programs serving older adults, to a low of 9.8 percent for respondents in programs serving those with intellectual/developmental disabilities (I/DD).  The measure developer is confident that, as state use increases, the response rate will increase.  Our confidence is based on: 1) the use of proxy respondents; 2) enhanced program-specific recruitment efforts; and 3) historical experience of response rates associated with a long-standing survey targeting people with I/DD.  Taken together, these provide evidence of the feasibility of achieving sample sizes sufficient to discern variation and performance across programs.  Each of these reasons for expecting increased response rates in the future is addressed in more detail below.

1.             Use Of Proxy Respondents: A larger proportion of beneficiaries enrolled in programs serving people with I/DD tend to have guardians.  It was our experience in the field test that guardians tended to act as gate keepers, refusing access to the participants.  Many of these guardians said that the participant was not able to complete the survey but that as their guardian they wished to do so.   In the beginning of the field test, proxy respondents were not allowed; at the time the CAHPS Consortium did not allow proxy respondents and since CMS was seeking a CAHPS trademark for the survey, the survey team did not allow proxy respondents.  However, as field data started coming in, survey vendors began reporting that interviewers were allowing proxy respondents.   Subsequently, the measure developer decided to allow and document proxy respondents so there would have opportunity to assess their contributions. However, because proxy respondents were allowed only beginning in September 2014 and data collection occurred over the period of July 2014 through February 2015, it is not possible to make definitive statements about the effect of proxy respondents on the response rate. That said, there is suggestive evidence that the response rate for the I/DD population may be increased if proxy respondents are allowed. Our analyses show that if proxy responses for persons with I/DD are counted, the number of respondents increases by approximately 100 percent. 

The information below provides some additional information on proxy use during the field test for all populations, including information pertinent to proxy respondents for surveys targeting people with I/DD. Proxy respondents were most prevalent in programs serving people with I/DD. Nearly half of respondents for I/DD programs were proxy respondents whereas the proxy respondent rate was substantially less in programs serving people with other disabilities.

Proxy Respondents in the EoC Survey Field Test

Population             Proxy Complete (N)             Percent Proxy Complete Surveys      Range of State % Proxy Complete

I/DD              192         49.9%     36.1% - 85.7%

Older Adults          414         20.2%     5.2% -36.6%

Acquired Brain Injury         53           20.7%     5.7% - 39.3%

Behavioral Health and SUD         8              2.6%       0.0% - 5.1%

Overall    667         22.2%     0.0% - 85.7%

If proxy respondents are allowed in future administrations of the survey one would expect an increase in response rates for the I/DD population. As noted above, we found that guardians, family members and caregivers acted as gatekeepers, not allowing access to the participant. For example, 14 percent of all eligible I/DD sample members had a guardian who did not allow access to the participant because they were either “physically or mentally incompetent” (an AAPOR category for non-response). Converting at least some gate-keeper guardians to proxy respondents in the future should increase response rates substantially.

2.             Enhanced Program-Specific Recruitment Efforts: Since the conclusion of the field test data collection, the TEFT Demonstration state grantees have identified improvements that they intend to implement for the next round of data collection which they are conducting themselves. It is expected that some of these enhancements will result in improved response rates. They include:

•              Ensuring that pre-notification letters originate from the state agency operating the HCBS program being surveyed so that those receiving the letter have familiarity with the letterhead. 

•              Ensuring that participant contact information is accurate by requiring that case managers verify participant and guardian contact information for persons sampled. 

•              Ensuring that survey vendors have experience and specialized qualifications with the populations being surveyed so they are sensitized to particular considerations in interacting with people with certain types of disability. This is likely to increase rapport and result in improved recruitment.

•              Targeting survey mode to persons/groups more likely to respond to a certain mode (rather than randomization to mode as happened in the field test). 

•              Conducting outreach to relevant stakeholders about the survey. This includes case managers and providers so they can encourage beneficiaries to participate when they receive inquiries from sampled members about the legitimacy of the survey. It may also include family and caregiver support groups. Stakeholders are more likely to encourage survey participation if they understand who is sponsoring the survey, its purpose and benefits. 

•              Not fielding the survey during the winter holiday season.

•              Not fielding the survey during winter months in colder climates due to the risk of inclement weather prohibiting travel. 

3.             Response Rates from An Other Survey of People with I/DD:  Some of the TEFT Demonstration state grantees sponsor another survey – the National Core Indicators (NCI) survey -- that elicits feedback from people with I/DD. Some state I/DD agencies have conducted the NCI repeatedly over many years. Consequently beneficiaries, family members and guardians are very familiar with the survey. Also, the NCI allows proxy respondents. 

Four TEFT grantee states shared the response rates that they have attained in recent years for the adult NCI survey:

•              Arizona: 87 percent response rate

•              Kentucky: 94.5 percent response rate

•              Colorado: 39 percent response rate

•              Connecticut:  In order to complete 400 surveys, they pull a sample of upwards of 1,000.

In addition to the information provided by the TEFT states, the National Association for Directors of Developmental Disability Services (NASDDDS), one of the sponsors of the NCI survey for people with I/DD, reports that for their face-to-face surveys: “Most states interview about 500 people to get the 400 sample size number (and most have to pull about 800 names to get the sample size).” (http://www.nasddds.org/uploads/files/NCI_Description_and_Costs.pdf). While the NCI project does not report average response rates, this information from the NASDDDS website indicates the feasibility of achieving much higher response rates for the ID/DD subgroup than realized in the Experience of Care survey field test.

Timing of Data Collection: 

• States that experience snow/ice during the winter should be encouraged to schedule data collection in other seasons.

Administration time: 

Prospectively, for the PRA package submitted to OMB, the research team estimated 30 minutes per administration. This estimate was based on the length of the survey and CMS’s experience with previous CAHPS surveys of comparable length that were fielded with a similar, although not identical, population – the long-stay nursing home resident CAHPS.  The nursing home CAHPS reported an administration time of 20 minutes on average.  Since there are more items in the CAHPS  Home and Community-Based Services Survey than the Nursing Home CAHPS (96 items versus 45 items), the research team estimated 30 minutes for the PRA package.

Retrospective analysis post field test showed that on average, respondents answered 51 (out of 96 items). Skip patterns account for the majority of the discrepancy between total number of items and number answered. General guidance from a survey expert is that it takes a person can answer approximately 4 items per minute.  This estimate would put the time to administer the HCBS CAHPS in the vicinity of 13 minutes. Although field test data are not available on length of time to administer, we can safely assume that survey administration, on average, did not exceed 30 minutes based on these other sources. 

Routine use of satisfaction/experience surveys in HCBS programs: 

State Medicaid programs have been administering surveys to their HCBS participants for many years.  Many are home-grown and state-specific, others were developed with an intended wider audience; only some of these surveys have undergone limited testing. Participants and their caregivers are very accustomed to these surveys.  We know that at least 46 states currently administer a satisfaction/experience survey to their beneficiaries in at least one HCBS program.  However, these surveys are all disability-specific, unlike the HCBS CAHPS Survey, which was designed and tested as a cross-disability survey.