The ICH CAHPS Survey is designed to measure the experiences of people receiving in-center hemodialysis care from Medicare-certified dialysis centers. The survey is designed to meet the following three broad goals:
- Produce comparable data from the patient’s perspective that will allow objective and meaningful comparisons between dialysis centers on domains that are important to consumers.
- Create incentives for dialysis centers to improve their quality of care.
- Enhance public accountability in health care by increasing the transparency of the quality of care provided in return for public investment.
Specifically, the survey measures patients’ experiences on topics that are important from the perspective of patients and help them make more informed choices when selecting a dialysis center as well as helping dialysis centers improve the quality of dialysis care for their patients.
The survey is administered semiannually to patients who have received in-center hemodialysis for at least 3 months from Medicare-certified dialysis centers. Data collection for each survey period is 12 weeks. The survey is available in mail-only, phone-only, and mail with phone follow-up.
Survey results publicly reported include two ratings and two measures:
• Global rating of dialysis center staff
• Global rating of dialysis center
• Quality of Dialysis Center Care and Operations measure (QDCCO, calculated from 13 survey questions)
• Providing Information to Patients measure (PIP, calculated from 9 survey questions)
The global rating of the dialysis center staff captures the patient’s perspective on how they rate the staff at their dialysis center, on a scale of 0 (worst) to 10 (best).
This is Question 20 in the survey and the question and scale are as follows:
Using any number from 0 to 10, where 0 is the worst dialysis center staff possible and 10 is the best dialysis center staff possible, what number would you use to rate your dialysis center staff?
0 0 Worst dialysis center staff possible
1 1
2 2
3 3
4 4
5 5
6 6
7 7
8 8
9 9
10 10 Best dialysis center staff possible
Measure Specs
- General Information(active tab)
- Numerator
- Denominator
- Exclusions
- Measure Calculation
- Supplemental Attachment
- Point of Contact
General Information
One of the goals of the CMS National Quality Strategy is to foster engagement and to bring the voices of patients to the forefront. As part of fostering engagement, it is critical to hear the voice of individuals by obtaining feedback from them on in-center hemodialysis (ICH) facility performance and incorporating it as part of CMS’s comprehensive approach to quality. Patient-centeredness is a central goal of dialysis care and can be directly measured through surveys of dialysis patients. CMS created the In-Center Hemodialysis CAHPS® Survey, a component of the End-Stage Renal Disease (ESRD) Quality Incentive Program (QIP), to ensure that an assessment of the patient-centeredness of care would be included to monitor dialysis facility performance, promote quality improvement, and inform consumer decision making in the selection of a dialysis facility via public reporting of results. The ICH CAHPS Survey is a standardized survey instrument and data collection methodology for measuring ICH patients’ perspectives on their care in Medicare-certified dialysis centers. The survey is administered semiannually to patients who have received dialysis for at least 3 months from Medicare-certified dialysis centers.
The ICH CAHPS global rating of the Dialysis Center Staff captures the patient’s perspective on how they rate the staff at their dialysis center, on a scale of 0 (worst) to 10 (best).
This measure will be publicly reported on Medicare.gov Care Compare tool to help ICH facilities with quality improvement and help ESRD patients find high quality dialysis facilities.
In addition to survey responses from patients, we receive data from a CMS database. ICH facilities are required to enter dialysis information for all patients into the ESRD Quality Reporting System (EQRS). EQRS data are then used to create the samples for each ICH CAHPS Survey period. In addition, information such as sex and age are pulled from the EQRS data and merged with survey response data files for analysis purposes. We work with the EQRS team on a continuous basis to remain up to speed on the data, changes to the data or data format, and to mitigate any issues with the data.
Numerator
CMS calculates ICH CAHPS Survey measure scores using top-box scoring. The top-box score refers to the percentage of respondents that give the most positive response(s): the dialysis center staff rating measure use a “0-10” response scale where the top-box numerator is the number of respondents who answer “9 or 10.”
CMS calculates ICH CAHPS Survey measure scores using top-box scoring. The top-box score refers to the percentage of respondents that give the most positive response(s) (respondent answers: 9 and 10).
The dialysis center staff rating uses a “0-10” response scale where the top-box numerator is the number of respondents who answer “9 or 10.”
The survey item reads: Using any number from 0 to 10, where 0 is the worst dialysis center staff possible and 10 is the best dialysis center staff possible, what number would you use to rate your dialysis center staff?
Denominator
ICH CAHPS Survey respondents are adult eligible patients who received dialysis care from an in-center hemodialysis facility during the sampling window. A survey is defined as completed when at least 50 percent of the questions applicable to all patients are answered.
The denominator for the ICH CAHPS dialysis center staff rating measure is the number of respondents with completed surveys who answered the dialysis center staff rating question.
ICH CAHPS Survey respondents are adult patients who received dialysis care from an in-center hemodialysis facility during the sampling window. A survey is defined as completed when at least 50 percent of the questions applicable to all patients are answered.
The denominator for the ICH CAHPS dialysis center staff rating is the number of respondents with completed surveys who answered this specific rating question, over the 2 semiannual periods being publicly reported.
Exclusions
Cases are excluded from the rating measure denominator if:
- Patients are under 18 years of age
- Patients’ eligibility is unclear in mail survey
- Patients are not currently receiving dialysis
- Patients are deceased or receiving hospice
- Patients have not received ICH dialysis for at least 3 months
- Patients receive dialysis at a nursing home where they reside or at home
- Patients reside in jail or prison
- Patients are mentally or physically incapable
- Patients are no longer receiving care at sampled facility
- Patients have a language barrier
- Survey completed by a proxy
Denominator exclusions are based on final disposition codes from survey vendors, and approved during data cleaning activities. The cases with the following disposition codes assigned are excluded:
130 – Completed mail survey, survey eligibility unknown
140 – Ineligible: not currently receiving dialysis
150 - Deceased
160 – Ineligible: does not meet eligibility criteria (under 18, hospice, nursing home, less than 3 months, doesn’t receive dialysis at a center)
170 – Language barrier
180 – Mentally or physically incapacitated
190 – Ineligible: No longer receiving care a sampled facility
199 – Completed by proxy
Measure Calculation
The survey is administered semiannually to patients who have received in-center hemodialysis for at least 3 months from Medicare-certified dialysis centers. Data collection for each survey period is 12 weeks.
Survey results publicly reported include two ratings measures and two multi-item measures:
• Global rating of dialysis center staff
• Global rating of dialysis center
• Quality of Dialysis Center Care and Operations multi-item measure (QDCCO, calculated from 13 survey questions)
• Providing Information to Patients multi-item measure (PIP, calculated from 9 survey questions)
CMS calculates ICH CAHPS Survey measure scores using top-box scoring for completed surveys (a survey is defined as completed when at least 50% of the core questions applicable to all patients are answered). The top-box score refers to the percentage of respondents who give the most positive response(s).
ICH CAHPS Survey respondents are eligible adult patients who have received care from an ICH dialysis center for at least 3 months. The numerator for the multi-item measures is the number of most positive responses (9/10 for ratings and “Yes” or “Always” for multi-item measures). The denominator for the ICH CAHPS multi-item measures is the total number of respondents with completed surveys who answered at least one item within the multi-item measure.
Cases are excluded from the measure denominator if:
- Patients are under 18 years of age
- Patients’ eligibility is unclear in mail survey
- Patients are not currently receiving dialysis
- Patients are deceased or receiving hospice
- Patients have not received ICH dialysis for at least 3 months
- Patients receive dialysis at a nursing home where they reside or at home
- Patients reside in jail or prison
- Patients are mentally or physically incapable
- Patients are no longer receiving care at sampled facility
- Patients have a language barrier
- Survey completed by a proxy
More information on the risk adjustment and related calculations can be found in the two attachments submitted for 1.18a. One of the attachments is in Section 7 (supplemental).
The measure is not stratified.
The ICH CAHPS Survey is conducted semiannually each spring and fall. CMS-trained survey vendors can offer survey administration via mail-only, telephone-only, or mixed mode (mail with phone follow-up) to their client ICH facilities. The survey is available in English, Spanish, traditional and simplified Chinese, Samoan, and Vietnamese; the telephone interview is only available in English and Spanish. Vendors must administer the survey in English, but can choose whether to offer other languages.
We do not yet know response rates for the revised survey since it has not been implemented in the national implementation. Our overall response rate for the current ICH CAHPS Survey (not the revised survey) is around 30%; the mixed mode has the highest response rates of the 3 offered modes. Response rates are calculated for the instrument as a whole:
Response Rate = Total Number of completed surveys/(Total Number of Surveys Fielded - Total Number of Ineligible Surveys)
There is no minimum response rate requirement on ICH CAHPS. We are continuously working with survey vendors and ICH facilities to help improve response rates, by offering things such a flyers/posters and waiting room FAQs to place in facilities, training telephone interviewers on avoiding refusals, and using the CMS logo on mailing materials. We also have a dialysis patient page on our project website with survey FAQs and we reference this in the materials that are mailed to sample patients.
Every facility’s sample size differs, depending on the number of survey-eligible patients in the EQRS data, and the size of the facility. There is no minimum sample size per ICH facility. If a facility did not serve 30 survey-eligible patients in the preceding year, they are not required to participate in that year’s surveys. In order for ICH CAHPS scores (for the 2 ratings and 2 multi-item measures) to be publicly reported, a facility must have 30+ completed surveys across the two survey periods that are being reported during that Care Compare refresh on medicare.gov. For example, for the 2024 October Care Compare refresh period, a facility needed at least 30 completed surveys between the 2023 Spring and 2023 Fall Surveys, in order for the scores for the global ratings and QDCCO and PIP multi-item measures to be reported.
Supplemental Attachment
Point of Contact
Not applicable.
Elizabeth Goldstein
Woodlawn, MD
United States
Tracy Kline
RTI International
Research Triangle Park, NC
United States
Importance
Evidence
The Consumer Assessment of Healthcare Providers and Systems (CAHPS) set of patient experience surveys are well-established measures of healthcare quality. Each semiannual survey period, between 350,000 and 415,000 ICH patients are selected to receive the ICH CAHPS Survey. Public reporting of these survey results creates incentives for dialysis centers to improve their quality of care, directly impacting the patients who receive it. Because of this, it is important to ensure that the survey aligns with what patients believe constitutes high-quality care. Specifically, the survey measures patients’ experiences on topics that are important from the perspective of patients and where the patient is the best source of information. The results help dialysis patients make more informed choices when selecting a dialysis center as well as helping dialysis centers improve the quality of dialysis care for their patients.
In addition to a number of psychometric analyses, RTI conducted literature reviews, several rounds of focus groups and cognitive interviews with dialysis patients, conversations with ESRD stakeholders and Technical Expert Panels (TEP), and discussions with the CAHPS Consortium prior to finalizing a revised survey to test in a 2022 field test/mode experiment. Following the field test, results were discussed with the same groups of experts and all supported the revised survey.
During the focus groups, dialysis patients were asked about characteristics that were important to them in regards to high quality dialysis care. They were then asked about specific survey items identified for removal in the psychometric analysis. For each survey item, participants were asked how important questions were in rating and evaluating the care that they receive at their dialysis center. For the QDCCO measure, there was a mix of opinions on each question identified to be removed, but the consensus of the focus group participants was that if questions had to be removed from the measure, then these were the best to remove.
The 2020 and 2023 TEPs regarding survey revisions included 10 members each, consisting of dialysis patients, ESRD network representatives, a survey expert, dialysis patient advocates, and large dialysis organization representatives.
Data from the 2022 ICH CAHPS Survey field test/mode experiment do not allow for direct assessment of the relationship between survey measures and structures or processes. However, given that the modified instrument-derived multi-item measures (QDCCO and PIP) are the same or similar to current ICH CAHPS measures, CMS anticipates that these revised measures will exhibit similar relationships to those of the existing ICH CAHPS Survey measures. For example, an exercise comparing the current QDCCO multi-item measure (prior to item removal) to the revised QDCCO multi-item measure results in a near-perfect correlation (r=0.991) and illustrating the strong relationship between the current and revised state of the QDCCO multi-item measure.
The survey measures patients’ experiences on topics that patients have identified as important to measure the quality of care for facilities and for which the patient is the best source of this information. These survey measures help dialysis patients make more informed choices when selecting a dialysis center as well as helping dialysis centers improve the quality of dialysis care for their patients. The dialysis center staff rating asks the patient to rate their dialysis center staff on a scale of 0 (worst) to 10 (best). Focus groups/cognitive interviews with dialysis patients, conversations with ESRD stakeholders and Technical Expert Panels, and discussions with the CAHPS Consortium were conducted prior to the 2022 field test/mode experiment that tested the revised survey and after the field test/mode experiment results were finalized. All supported the revised survey, including keeping this existing rating question.
Summary of literature findings associated with this rating measure:
1. Hreńczuk M. Therapeutic relationship nurse–patient in hemodialysis therapy. Nurs Forum. 2021;56:579‐586. 10.1111/nuf.12590
- Hemodialysis patients noted that the nurse-patient relationship influences security, helps with coping, and increases quality of care.
- “In most cases, this relationship gave patients a sense of being understood by the other person (33.3%), gave them strength to fight the disease (27.3%), less often helped to restore hope (10.6%), helped them practice the ability to understand their problems by looking at them through the eyes of a different person (9.1%), helped to control helplessness (7.6%), and strengthened the positive aspects of health behaviors (6.1%).”
- Patient satisfaction encompasses various aspects of dialysis care, including interactions with staff.
2. Nair D, Wilson FP. Patient-reported outcome measures for adults with kidney disease: current measures, ongoing initiatives, and future opportunities for incorporation into patient-centered kidney care. Am J Kidney Dis. 2019;74(6):791–802. doi:10.1053/J.AJKD.2019.05.025
Measure Impact
A successful survey should be relevant to the target audience and produce meaningful results that can be used to inform decisions. For the ICH CAHPS Survey, this includes a survey that accurately measures characteristics of quality dialysis care from the patient’s perspective and produces results that ICH centers can use to improve their care.
Each survey period, between 80 and 100k sample patients respond to the current ICH CAHPS Survey, indicating that the survey is meaningful and that they feel that providing their information is valuable. The survey focuses on topics that ESRD patients have reported as being important in defining high-quality dialysis care; these patients are the only source of this information. During survey revisions, we met with 2 focus groups of 9 people who all noted that they felt the ICH CAHPS Survey is important and were happy to know that it was being reduced in length. CMS also receives a number of letters from dialysis patients noting that they believe the survey is valuable and that they appreciate being able to convey their thoughts and have their voices heard.
During the 2024 Pre-Rulemaking Measure Review (PRMR) process, the Patients for Patient Safety US organization commented the following: We certainly support this, because the experiences of patients is a valuable data source, and we're happy to see it included in instruments like CAHPS as well as other ways in which we use PROMs and PREMs. I will say that our organization Patients for Patient Safety US does see the CAHPS surveys as an incredibly important tool, not only for feeding back patient experience, but improving health equity, by bringing in people who are less likely to use these surveys by asking the right kinds of questions and including meaningful questions in them and I think this falls into that category.
Activities conducted by RTI during the survey revision process, such as focus groups, interviews, and TEP meetings, helped to ensure that the survey is relevant to the target audience, and that survey results are meaningful to both patients and the ICH facilities that provide care.
During initial TEP discussions, TEP members were asked if there were certain questions that should or should not be removed from the survey. The consensus was that the rating questions were important in giving sample patients a way to actually rate their care, noting that they felt patients needed this opportunity to quantify their opinion. This was confirmed with patients during focus groups, where they noted their appreciation for having such rating questions.
Performance Gap
We have provided the distribution of scores in Table 1 as requested. The information in the table is based on top-box calculated scores. The data used to calculate the deciles comes from the 2023 Fall and 2023 Spring National Implementation data. Due to the sampling design for the mode experiment, to have the least impact on public reporting, facility-level scores could not be calculated so national implementation was used as proxy. The facility-level performance scores for each survey period are created using top-box, and then combined to create 2023 performance scores. In total, 6700 facilities and 178,480 survey responses are represented in this data.
| Overall | Minimum | Decile_1 | Decile_2 | Decile_3 | Decile_4 | Decile_5 | Decile_6 | Decile_7 | Decile_8 | Decile_9 | Decile_10 | Maximum | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Mean Performance Score | 69.11 | 0 | 48.505 | 56.667 | 62.229 | 66.504 | 70.191 | 73.957 | 77.970 | 82.089 | 88.218 | 100 | 100 |
| N of Entities | 6,697 | 4 | 670 | 670 | 669 | 669 | 670 | 669 | 671 | 669 | 670 | 670 | 80 |
| N of Persons / Encounters / Episodes | 178,477 | 5 | 15,106 | 18,494 | 20,216 | 20,350 | 19,662 | 19,460 | 19,326 | 17,784 | 16,775 | 11,304 | 637 |
Equity
Equity
This domain is optional for Spring 2025.
Feasibility
Feasibility
As these patient-experience data are collected from patients based on their ICH dialysis care experiences, the structured data are not available in electronic sources outside of this data collection. Web-based data collection was tested in the 2022 ICH CAHPS Survey field test/mode experiment, but due to a lack of email addresses available in the sampling data, we had very few responses. We are working now to determine whether we can obtain additional email addresses to retest web data collection in a future mode experiment.
Proposed revisions to the current ICH CAHPS Survey instrument include shortening it and was developed through focus groups, cognitive interviews, and other instrument-development activities conducted for this revised instrument. The changes to the current ICH CAHPS survey were implemented to mitigate the challenges and barriers to responding to the current full survey. The shortened survey reduces patient burden and is expected to increase response rates.
The ICH CAHPS Survey is administered by independent survey vendors that are approved by CMS, and CMS’s implementation contractor provides oversight on a regular basis to ensure that the vendors are following established protocols. Additionally, the survey vendors are required to conduct regular review and monitoring of their own operational systems, whether the survey is administered by mail or telephone. Data is assessed semiannually for accuracy and missing data.
ICH CAHPS Survey results for the updated survey instrument will be fully publicly reported in October 2027 (2026 Spring + 2026 Fall data). However, because the April 2027 refresh would include a survey period that used the current survey (2025 Fall) and a survey period that used the revised survey (2026 Spring), we would plan to reanalyze the 2025 Fall data based on the revised survey measures and case-mix, then combine the reanalyzed data with the 2026 Spring data for public reporting in April 2027; therefore, we are not missing a refresh for ICH CAHPS data.
The following revisions were made to shorten the ICH CAHPS Survey:
The Nephrologist Communication and Caring (NCC) Measure was removed, which included the following questions:
- In the last 3 months, how often did your kidney doctors listen carefully to you?
- In the last 3 months, how often did your kidney doctors explain things in a way that was easy for you to understand?
- In the last 3 months, how often did your kidney doctors show respect for what you had to say?
- In the last 3 months, how often did your kidney doctors spend enough time with you?
- In the last 3 months, how often did you feel your kidney doctors really cared about you as a person?
- Using any number from 0 to 10, where 0 is the worst kidney doctors possible and 10 is the best kidney doctors possible, what number would you use to rate the kidney doctors you have now?
- Do your kidney doctors seem informed and up-to-date about the health care you receive from other doctors?
The following questions were removed from the QDCCO Measure:
- In the last 3 months, did dialysis center staff keep information about you and your health as private as possible from other patients?
- In the last 3 months, how often did dialysis center staff insert your needles with as little pain as possible?
- In the last 3 months, did dialysis center staff talk to you about what you should eat and drink?
- In the last 3 months, how often did you feel your kidney doctors really cared about you as a person?
The following questions were also removed from the survey; these items were not included in a measure:
- In the last 3 months, has anyone on the dialysis center staff asked you about how your kidney disease affects other parts of your life?
- Medicare and your State have special agencies that check the quality of care at this dialysis center. In the last 12 months, did you make a complaint to any of these agencies?
- Are you being treated for high blood pressure?
- Are you being treated for diabetes or high blood sugar?
- Are you being treated for heart disease or heart problems?
- Are you deaf or do you have serious difficulty hearing?
- Are you blind or do you have serious difficulty seeing, even when wearing glasses?
- Because of a physical, mental, or emotional condition, do you have serious difficulty concentrating, remembering, or making decisions?
- Do you have serious difficulty walking or climbing stairs?
- Do you have difficulty dressing or bathing?
- Because of a physical, mental, or emotional condition, do you have difficulty doing errands alone, such as visiting a doctor’s office or shopping?
- Who helped you complete this survey?
ICH facilities that served 30 or more survey-eligible patients in the preceding year are required by the ESRD PPS Rule to participate in the current calendar year’s two surveys. They are required to contract with one of the CMS-approved ICH CAHPS Survey vendors, who administers the survey on their behalf. Neither CMS nor RTI are involved in payment discussions between the vendors and facilities, and we have no cost information in terms of the facilities. The proposed revisions will not cause a significant burden to vendors or CMS’s implementation contractor, other than the modification of existing computer programs. Regarding burden to sample dialysis patients, the revised survey will take approximately 12 minutes of their time up to two times each year.
The sample is drawn from EQRS data, meaning that facilities are not involved and do not know who was selected to participate in the survey (unless a sample patient specifically tells them). Sample patients are asked to not ask for help from facility staff, and facilities are told not to assist their patients in completing the survey if asked and not to ask any patient whether they were sampled for the survey.
ICH CAHPS Survey data submitted to the ICH CAHPS Data Center (CMS’s implementation contractor) are de-identified. Case-level data are assigned a unique case ID, randomly generated so that there is no identifying information embedded in it. ICH CAHPS Survey results are aggregated at the facility level and publicly reported at the facility-, state- and national-level.
Vendors are allowed to provide unofficial aggregated results to their facility clients; however, there are rules on sharing such information when the number of responses is too low.
After the 2022 field test/mode experiment, and after the results were discussed with stakeholders (Technical Expert Panel members representing different ESRD organizations and large dialysis organizations, as well as the CAHPS Consortium), CMS approved the deletion of multiple items from the current nationally fielded ICH CAHPS Survey. The revised instrument is being submitted via this Instrument submission. For the 4 instrument-derived measures (2 ratings and 2 multi-item measures), 3 are the same as the current ICH CAHPS Survey and one is slightly modified.
Proprietary Information
Scientific Acceptability
Testing Data
The revised ICH CAHPS Survey was tested in fall 2022 during a field test/mode experiment. The total sample for the field test/mode experiment, based on EQRS data, was 24,354. Data were collected in English and Spanish, by mail, telephone, mail with telephone follow-up, and web with mail follow-up. Analyses are based on the field test/mode experiment data collected between October 2022 and January 2023, including 4605 respondents representing 3211 facilities. Field test/mode experiment data were used for the internal consistency analyses (Reliability – Section 5.2), factor analysis (Validity – Section 5.3), correlations among ratings and multi-item measures (Validity – Section 5.3), and risk analysis (IDM – Risk Adjustment – Section 5.4) analyses. For the signal-to-noise (IDM – Reliability – Section 5.2) and performance gap (IDM – Performance Gap – Section 2.4) analyses, national implementation data was used, which was collected during the 2023 Spring and 2023 Fall Surveys.
The national ICH CAHPS Survey is offered in six languages: English, Spanish, Chinese (simplified and traditional), Samoan, and Vietnamese. Additional translations will be made as needed.
There are no fees or licensing for use of the ICH CAHPS® Survey, training or oversight activities, or for accessing publicly reported ICH CAHPS® Survey measure scores or star ratings on the CMS Medicare.gov website.
None
For the field test/mode experiment that was conducted from October 2022 through January 2023, we took a sample of 24,354 ICH patients, using a stratified sampling method so as to have the least impact on public reporting. There were 6,736 facilities represented in the field test/mode experiment sample, located throughout the country. Facilities ranged in size from very small to very large. To have the least impact in public reporting, facilities that always meet the public reporting threshold (always have 30+ completed surveys for the two reporting periods) and facilities never meet the public reporting threshold (do not come close to the 30+ completes) were oversampled, and facilities that are normally right on the threshold line were undersampled. We randomly assigned sampled patients to one of the four data collection modes using the inverse of the estimated response rates to achieve the desired sample size in each mode. The final sample closely mirrored the national population of dialysis centers. Analyses were conducted using data from the 4605 respondents, who represented 3211 facilities.
For the 2023 Spring/Fall national implementation data used for facility-level analyses, all facilities registered on the ICH CAHPS website, by each dialysis facility that has determined they are required to participate in the surveys, were included in the sample if they had sample in the EQRS data. For the 2023 Spring/Fall Survey national implementation data, the following sample strategy was used:
Facilities with up to 240 patients: A census of all survey-eligible patients will be conducted for facilities with fewer than 240 survey-eligible patients at each semiannual sampling wave. Thus, patients at these ICH facilities may be sampled twice in a 12-month period. Facilities with 240 or more patients.
For dialysis centers with 240 or more survey-eligible ICH patients: A simple random sample will be selected for each sampling period, with the goal of obtaining 200 completed surveys per year while attempting to minimize the overlap of patients between subsequent semiannual waves of sampling.
The 2023 Spring Survey included 6675 facilities across the US and its territories, and the 2023 Fall Survey included 6674 facilities. Facility size differed from small to large; facilities were required to participate if they served 30+ survey-eligible patients in the prior calendar year.
For the field test/mode experiment, our sampling design randomly assigned sample patients to one of the four data collection modes, with a target of completing approximately 1,570 interviews for each mode. This design allowed us to measure mode effects related to nonresponse and measurement differences (e.g., because of social desirability) and also allowed for case-mix analyses. 24,354 ICH patients were sampled for the field test/mode experiment, using a stratified sampling method so as to have the least impact on public reporting. To have the least impact in public reporting, facilities that always meet the public reporting threshold (always have 30+ completed surveys for the two reporting periods) and facilities never meet the public reporting threshold (do not come close to the 30+ completes) were oversampled, and facilities that are normally right on the threshold line were undersampled. We randomly assigned sampled patients to one of the four data collection modes using the inverse of the estimated response rates to achieve the desired sample size in each mode. The final sample closely mirrored the national population of dialysis centers. Analyses were conducted using data from the 4605 respondents, who represented 3211 facilities.
For the 2023 Spring/Fall Survey national implementation data, the following sample strategy was used:
Facilities with up to 240 patients: A census of all survey-eligible patients will be conducted for facilities with fewer than 240 survey-eligible patients at each semiannual sampling wave. Thus, patients at these ICH facilities may be sampled twice in a 12-month period. Facilities with 240 or more patients.
For dialysis centers with 240 or more survey-eligible ICH patients: A simple random sample will be selected for each sampling period, with the goal of obtaining 200 completed surveys per year while attempting to minimize the overlap of patients between subsequent semiannual waves of sampling.
For the 2023 Spring, 378,513 patients were sampled. 91,914 respondents, representing 6609 facilities across the nation and its territories, were included in the analyses. For the 2023 Fall, 383,116 patients were sampled, 86,566 respondents representing 6595 facilities, were included in the analyses.
Eligibility criteria for the field test/mode experiment sample mirrored the criteria used for national implementation. Sample patients must:
- Be at least 18 years of age or older,
- Have received hemodialysis at an in-center facility for 3 months or longer,
- Be alive as of the last day of the sampling window, and
- Not be institutionalized or receiving hospice.
Attachment 5.1.4 with patient characteristics can be found in Supplemental 7.1 zip file.
Reliability
Psychometric testing at the instrument level includes a reliability assessment using the Cronbach’s alpha estimate, from the internal consistency analysis of measurement error. The reliability analysis was conducted at the item and patient level, using SAS’s PROC CORR procedure with ALPHA notation specified, which employs listwise deletion of missing data. Available response data from the revised survey field test/mode experiment was included as input for the internal consistency analysis.
Output from the PROC CORR was evaluated against psychometric thresholds of acceptable internal consistency. In general, Cronbach’s alpha estimates of above 0.9 were evaluated as good internal consistency and 0.7 was considered a minimum threshold for acceptability in this analysis (Nunnally & Bernstein, 1994).
Citation:
Nunnally J, Bernstein L. Psychometric theory. New York: McGraw-Hill Higher, INC; 1994.
The attachment provides the internal consistency results for the Quality of Dialysis Center Care and Operations (QDCCO) and Providing Information to Patients (PIP) item sets. The item-total correlations are provided in the second column from the right, and the alpha estimate for the set (if a particular item is removed) is in the far right. In summary, the 13 QDCCO items have a standardized Cronbach’s alpha estimate of 0.930 which exceeds the threshold for good internal consistency. In addition, the standardized item-total correlations range from 0.54 to 0.78 and none of the alpha estimates improve substantially with any item removed.
The 9 PIP items have a standardized Cronbach’s alpha estimate of 0.743 which is just above the minimal acceptability threshold. The standardized item-total correlation range for the PIP item set has a minimum of 0.29 and a maximum of 0.55, and again, none of the alpha estimates improve substantially with any item removed.
The internal consistency estimates for the QDCCO item set (α = 0.930), and PIP item set (α = 0.743) are above the minimal acceptability threshold (Nunnally & Bernstein, 1994). In addition,
most of the item-total correlations across multi-item measures are above 0.5, the remaining primarily above 0.3, with one value at 0.29. Therefore, the items assigned to each multi-items measure (QDCCO and PIP) function well together and we conclude that the items perform consistent measurement of the representative constructs.
The signal-to-noise analysis, termed inter-unit in this documentation, was conducted at the facility and derived measure level using the CAHPS 5.0 macro. For the overall estimate, 2023 Spring and Fall survey data was subset to facilities that had at least 30 completed cases between the two survey periods, to match the Public Reporting standards. 2023 Spring and Fall National Implementation survey data was used as a proxy due to the design of the 2022 field test/mode experiment, which was sampled in a way to ensure that the impact on Public Reporting data would be minimal and therefore not suitable for this analysis.
The CAHPS 5.0 macro calculates the global F-test to determine differences among entities on measures or items. F-statistics are transformed into reliability such that if the F-statistic is around 1, the reliability will be close to zero and indicate differences are due to random variation. The larger the real differences among facilities, the larger the F-statistic, and the higher the reliability estimate (approaching 1.0) and reliabilities above 0.7 are considered acceptable.
CITATION: Keller S, O’Malley AJ, Hays RD, Matthew RA, Zaslavsky AM, Hepner KA, Cleary PD. Methods Used to Streamline the CAHPS® Hospital Survey. Health Services Research, 2005, 40, 2057-2077. PMIDp: 16316438.
The Dialysis Center Staff Rating’s overall inter-unit reliability, using the 2023 Spring and Fall National Implementation survey data, is 0.75 (F=3.93, p<0.001). The information in Table 2 provides overall information as well as reliability by decile. Deciles for this table are based on psychometrics done within the CAHPS Macro; and the “mean performance score” row in this table is showing variation in the mean value by sample size. Deciles were specified on facilities that have 30 or more completes to match the Public Reporting standards. The extreme specifications included facilities with the smallest possible (1 or 2 completes) and close to the maximum number (100 completes or more) were specified as such to provide enough facilities to conduct the analysis.
| | Overall | Minimum | Decile_1 | Decile_2 | Decile_3 | Decile_4 | Decile_5 | Decile_6 | Decile_7 | Decile_8 | Decile_9 | Decile_10 | Maximum |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Reliability | 0.75 | 0.61 | 0.76 | 0.71 | 0.70 | 0.68 | 0.68 | 0.73 | 0.77 | 0.79 | 0.80 | 0.74 | 0.90 |
| Mean Performance Score | 8.64 | 8.32 | 8.64 | 8.68 | 8.61 | 8.64 | 8.63 | 8.60 | 8.65 | 8.72 | 8.59 | 8.67 | 8.74 |
| N of Entities | 2319 | 74 | 441 | 243 | 328 | 239 | 215 | 225 | 206 | 153 | 162 | 107 | 22 |
| N of Persons / Encounters / Episodes | 100828 | 115 | 13652 | 8118 | 11766 | 9312 | 9009 | 10236 | 10301 | 8468 | 10385 | 9581 | 2564 |
The inter-unit reliability assessment for the ICH CAHPS Dialysis Center Staff Rating indicates that overall, and at nearly all deciles of interest, the measure maintains acceptable performance (e.g., greater than 0.7) in finding real variation among facilities relative to random variation. In the instances where the inter-unit reliability is lower than 0.7, the estimate maintains sufficient strength to meet the requirement for acceptability (≥ 0.6) in the E&M Guidebook evaluation rubric (version 2.1).
Validity
Changes to the items within the QDCCO multi-item measure were assessed with face validity. Next, structural and convergent validity were used to determine whether the QDCCO and PIP item sets measure their respective construct as intended.
Confirmatory factor analysis (CFA) was used to determine structural validity of the QDCCO and PIP item sets through model fit statistics compared to acceptability thresholds. The thresholds for model acceptability are as follows:
RMSEA of 0.05 or less (Browne & Cudeck, 1993)
CFI of 0.90 at a minimum (Hu & Bentler, 1999)
TLI of 0.90 at a minimum historically, 0.95 indicates good fit
Convergent validity was also assessed through CFA by evaluating factor loadings for each item. CFA analyses were conducted using the Mplus analytic software (Version 8.11; 2023) using the Weighted Least Squares Mean and Variance adjusted (WLSMV) estimator to account for categorical data with pairwise deletion for missing data. In addition, correlations, conducted using SAS PROC CORR, evaluating the two multi-item measures from the item sets (QDCCO and PIP) with ratings of the Dialysis Center and Dialysis Staff, were provided for convergent validity.
Citations:
Browne, M. W., & Cudeck, R. (1993). Alternative ways of assessing model fit. In K. A. Bollen & J. S. Long (Eds.), Testing structural equation models (pp. 136–162). Thousand Oaks, CA: Sage.
Hu, L., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6(1), 1–55. http://dx.doi.org/10.1080/10705519909540118
Muthén, L. K., & Muthén, B. O. (1998-2023). Mplus (Version 8.11) [Computer software]. Los Angeles, CA: Muthén & Muthén
Glossary:
RMSEA - root mean square error of approximation
CFI - comparative fit index
TLI - Tucker-Lewis Index
All experts confirmed face validity and agreed that the remaining items still provided construct representation of the quality of a facility’s care and operations. In addition, psychometric results showed that the removal of 4 questions from the QDCCO did not negatively impact the measure’s validity.
Confirmatory factor analysis (CFA) model fit to assess structural validity produced acceptable values, consistent with a well-fitting model. These estimates are:
RMSEA=0.038 (95% CI = 0.036 0.040)
CFI=0.987
TLI=0.985
The attachment provides the estimates for each standardized factor loading on the representative items to assess convergent validity. Correlative information on the factors is also provided, and moderate. All factor loadings for the QDCCO and PIP item sets are above 0.4, with the majority above 0.5, and all are statistically significant (p < 0.001).
The final assessment of convergent validity through correlation information results in the relationship between each averaged measure, and the Dialysis Staff Rating (rQDCCO = 0. 0.757, rPIP = 0.434) and the Dialysis Center Rating (rQDCCO = 0.737, rPIP = 0.403), all of which are statistically significant (p<0.001).
The QDCCO and PIP item sets continue to effectively measure consistent constructs. Confirmatory factor analysis (CFA) produced appropriate model fit indices and statistically significant factor loadings. Correlational analyses showed moderate relationships to patient ratings of the overall facility and the staff. The findings confirm the valid measurement of the QDCCO and PIP constructs.
Validity was assessed at the measure and facility level using correlational methods and SAS’s PROC CORR procedure, which employs pairwise deletion of missing data. Output from the PROC CORR analysis was evaluated against standard correlational thresholds. In general, a correlation of 0.50 shows a moderate relationship and estimates higher are indicative of stronger relationships.
Available ratings from each facility were drawn from the 2023 Spring and Fall National Implementation survey data. The National Implementation data was used as a proxy due to the design of the 2022 field test/mode experiment, which was sampled in a way to ensure minimal Public Reporting data impact.
The Dialysis Center Staff Rating is comprised from a single item; therefore, convergent validity was assessed through correlations at the rating and facility level and examined by the relationship between the Dialysis Center Staff Rating, with the PIP and QDCCO multi-item measures, and then with the Dialysis Center Rating.
CITATION
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillside, NJ: Lawrence Erlbaum Associates
Convergent validity results, using the 2023 Spring and Fall National Implementation survey data, show that the relationship between the Dialysis Center Staff Rating, and the QDCCO multi-item measure (rRStaff = 0. 824), the PIP multi-item measure (rRStaff = 0. 488), and the rating of the Dialysis Center (rRStaff = 0. 888) are approaching, or stronger, than the established moderate threshold (0.50) and statistically significant (p<0.001).
The Dialysis Center Staff Rating shows moderate relationships to the derived QDCCO and PIP multi-item measures as well as the patient rating of the dialysis center. The findings confirm that the ICH CAHPS items result in the valid measurement, specifically of the Dialysis Center Staff Rating, without replicating other information within the survey.
Risk Adjustment
All case-mix/risk adjustment analyses were conducted using the data from the 2022 field test/mode experiment. The independent variables that were included in the mode and case-mix models are those hypothesized to affect response tendencies, and the results of the variables included in the two multi-item measures and two ratings. These independent variables are used as case-mix adjusters for ICH CAHPS scores for ICH facilities. These independent variables are not under the control of the facility, meaning a facility can’t change these descriptors of their patients (e.g., a facility can’t control the demographic make-up of their population with regards to age, sex, overall health, overall mental health, etc.). Adjusting for this helps to ensure that variations in case-mix do not affect the CAHPS scores calculated and publicly reported for ICH facilities because these factors impact response tendencies and are outside the control of the ICH facilities.
These independent variables came from four sources: (1) the ICH CAHPS Survey mode experiment implementation; (2) the ICH CAHPS Survey mode experiment as patients’ self-reported characteristics; (3) the Common Medicare Environment (CME) database; and (4) the EQRS database.
Using the case-mix model, the facility’s case-mix adjusted scores are adjusted for differences between a facility’s patient composition according to the ICH CAHPS case-mix characteristics and the overall national composition of ICH patients on these same characteristics. This adjustment allows consumers to compare different ICH facilities based on the same overall patient composition, thus allowing for a comparison that isn’t affected by the given facility’s patient composition.
We have attached a file showing the descriptive statistics, submitted under 7.1 other attachments.
We conducted the multivariate regression analysis assessing the candidate case-mix variables. We estimated 50 multivariate regression models for the mode and case-mix analysis, including one regression model for each question comprising the two multi-item measures and two ratings for the top-box outcomes. When estimating the multivariate regression models, we used the individual patient as the unit of analysis. For the top-box outcomes the dependent variables for all the multivariate regression models were converted to a binary score with the most positive response set to 1 and all other nonmissing responses set to 0.
We used ordinary least squares models to investigate the candidate case-mix variables which included:
• Mode of survey administration.
• Language survey was completed. Binary yes/no response categories for whether the respondent completed the survey in English or Spanish.
Patients’ Self-Reported Characteristics Independent Variables
• Overall health. Including five categories for self-reported overall physical health status.
• Overall mental health. Including five categories for self-reported mental or emotional health status.
• Education. Including six response categories for self-reported education.
• Did someone help you complete this survey. Including binary yes/no response categories for whether the respondent had someone else help them to complete the ICH CAHPS Survey.
• Low-income subsidy or dually enrolled. This variable is calculated from variables indicating if the respondent had a low-income subsidy or was dually enrolled in both Medicare and Medicaid at any time during the survey period.
• Age. This variable was calculated from the respondent’s date of birth. We created five categories for a categorical age variable.
• Sex. Binary male/female categories.
• Total years on dialysis. This variable includes six response categories for different time periods for the respondent since they originally started dialysis treatment. This variable is calculated from the EQRS variable for the date dialysis started for this patient.
• Was diabetes the primary cause of ESRD. This variable includes binary yes/no response categories for whether diabetes was the cause of ESRD for this patient. This variable is calculated from the EQRS variable for the cause of ESRD.
The facility indicator variable was not included in the models for this analysis. The number of respondents from each of the facilities ranged from only one to eight with the majority having only one respondent, thus the cluster effects were not important. Generally, the linear form of the multivariate regression models was:
Dependent variable = sum of (coefficients*mode indicators) + sum of (coefficients*patient characteristic indicators)
When there were categorical patient characteristics for an independent variable, such as age groups, one group was used as the reference category from the set of categories included in the regression model. That group is the reference to which the effects of the other categories for that variable are compared, which simplifies the interpretation of the regression coefficients.
RTI randomly assigned patients to mode.
Sequential Modeling
We estimated each of the regression models using the independent case-mix variables previously noted. All of the independent variables were statistically significant for at least one of the regression models.
The first set of multivariate regression models run on the 25 dependent variables included as independent variables all of the case-mix factors in their original form. We refer to this first set of models as the full model or Model 1. In the second set of multivariate regression models (Model 2), we recategorized the mental health question by rolling together the fair and poor categories to reduce the VIFs for this variable (a finding from the multicollinearity analysis). In the third and final set of regression models (Model 3), we dropped the low-income subsidy or dually enrolled variable due to the difficulty of obtaining this variable and the high level of missing values (13.18%) for this variable. Model 3 was our final model.
To determine the recommended regression model, we calculated the patient-level case-mix adjusted scores from the three models investigated. Then we calculated the two scores by averaging the patient-level case-mix adjusted scores for each survey item comprising a given multi-item measure. We used the results of the patient-level case-mix adjusted measure and global rating scores from the three models and the unadjusted scores to conduct an impact analysis. The goal of the impact analysis was to understand how the results from the three case-mix regression models changed compared to the unadjusted scores and to each other.
Using the three sets of adjusted patient-level scores and the unadjusted scores, we calculated the absolute value of the difference between each score considering all possible combinations (e.g., unadjusted vs. Model 1, unadjusted vs. Model 2, unadjusted vs. Model 3). The mean of these percentage point differences is found in Table 1of the attachment in 5.4.5a.
For the top-box scores, the absolute values of the percentage point differences range from 0.95 for QDCCO patient-mix adjusted top-box score using Model 2 compared to the case-mix adjusted top-box score using Model 1 to 10.43 for the global rating of the dialysis center unadjusted top-box score compared to patient-mix adjusted top-box score using Model 2. In general, the percentage point differences between the case-mix adjusted scores for the three models is small with the differences between Model 2 and Model 3 being the smallest.
Using the three sets of adjusted patient-level scores and the unadjusted scores, we calculated the Spearman’s rank correlation coefficient. Spearman’s rank correlation quantifies the strength and direction of association between two ranked variables (i.e., how similar they are). A high correlation (close to 1) indicates that the two variables are strongly related and a low correlation (close to 0) indicates that the two variables have a weak or no relationship. Negative values (close to −1) indicate that the two variables are strongly inversely related.
In this analysis, correlations close to 1 indicate that the different sets of patient-mix models are adjusting the scores similarly. The Spearman’s Rank Correlations are found in Table 2of the attachment in 5.4.5a for the two global ratings and two measure outcome scores. These correlations demonstrate the strength of the relationship of the global ratings and outcome measures between the adjusted patient-level scores using the three patient-mix models and the unadjusted scores. For each measure the lowest correlations existed when comparing to the unadjusted scores. For example, for the top-box global rating of the dialysis center, the correlation for the unadjusted scores and Model 1 was 0.81. The same correlation was found for the unadjusted and Model 2, and for unadjusted and Model 3.
For the two ratings and two multi-item measures the pairwise Spearman’s Rank Correlations for the adjusted scores using Model 2 compared to Model 3 were either 0.98 or 0.99 and the correlations using Model 1 to Model 2 ranged from 0.97 to 0.99. Given that all the correlations between the adjusted scores were so close to 1, we concluded that there are very few differences in the patient-level adjusted scores when using the different sets of patient-mix variables in the multiple regression models.
Final Case-Mix Regression Model
The three sets of multiple regression models tested performed very similarly for top-box. The Spearman’s Rank Correlations were all close to 1 demonstrating that the ranks of the adjusted scores were very similar. This suggests very small differences in the results of the adjusted scores for the different sets of case-mix models. Furthermore, the percentage point differences between the three models were less than 2 for all measures, hence we recommend the simplest model without the low-income subsidy or dually enrolled variable. This is Model 3 with the following case-mix variables:
overall health,
overall mental health (with fair and poor categories rolled together),
patient age,
patient sex,
patient education,
language survey was conducted (English or Spanish),
help completing the survey,
total years on dialysis, and
was diabetes primary cause of ESRD
Use & Usability
Use
Dialysis facility
Usability
Dialysis center scores will increase as they target these measures, therefore improving patients’ perspective of their dialysis care. Each dialysis center may have different training mechanisms or internal procedures which can help improve the overall quality of the services they deliver.
In an attempt to increase survey response rates (performance on the survey overall) to have enough data to be publicly reported, ICH facilities are encouraged to hang posters/flyers and waiting room FAQs in their facilities, so that patients understand that the ICH CAHPS survey is legitimate.
ICH facility scores will increase as facilities target this rating measure, which asks for patients’ perception of their dialysis center staff. Each ICH center may have different training mechanisms or internal procedures which can help improve the overall quality of the services they deliver.
The revised instrument has not been implemented beyond the field test stage; however, we did receive positive feedback during the Pre-Rulemaking Measure Review (PRMR) process. The PRMR hospital committee recommended the revised survey. The PRMR committee recognized the importance of patient experience of care data and supported efforts to reduce the length of the survey while maintaining scientific acceptability. The CAHPS Consortium noted that we should remove one additional question (which was done after more analyses) and then move forward with the revised survey; they felt that it would help survey burden and was a great start at reducing a very long survey. In addition, feedback was obtained from several large dialysis organizations (LDOs) during TEP meetings and email correspondence. They noted that this was a good start at reducing the survey but would like to see it reduced even more in the future. The consensus from the LDOs was that the revised survey didn’t remove any major questions that would raise alarm.
The proposed dialysis center staff rating (same as the current rating used in national implementation) will be one of four ICH CAHPS publicly reported.
Feedback from ESRD stakeholders and the CAHPS Consortium, as well as patients during focus groups and cognitive interviews, was positive for keeping the rating measure as is. TEP members noted that giving patients the opportunity to rate their center staff is important and therefore should be kept in the revised survey; it gives patients a way to quantify the care they receive.
Although this revised survey has not been implemented yet, final decisions were based on feedback from many stakeholders, including representatives from our LDOs. During Technical Expert Panel (TEP) meetings and via email discussions, feedback was provided on which questions could safely be dropped from the current survey, to create a revised survey to reduce patient burden. The TEP supported efforts to shorten the survey to reduce burden on respondents. They noted that there were no major concerns with the questions removed and appreciated the effort to make this a little easier on the patients responding to the survey.
Although this revised survey has not been implemented yet, final decisions were based on feedback from many stakeholders, including representatives from our LDOs. During TEP meetings and via email discussions, feedback was provided on the dialysis center staff rating and it was decided to keep it as is.
The revised survey has not been implemented yet so we have no improvement data based on this as of yet, but this rating is not changing.
The scores for the current Rating of the Dialysis Center Staff measure, for facilities with at least 20 responses from 2015 thru 2023, have increased over time. The biggest increases were seen from 2015 to 2020. In 2015 the median score for this rating was 61.8 and in 2020 the median score was 65.6. The median score in 2023 was 65.8. The other deciles exhibited a similar pattern.
The revised survey has not been implemented beyond the field test/mode experiment phase. No unexpected findings were observed during the field test/mode experiment. Respondents appreciated the shorter survey.
There have been no unexpected findings of the dialysis center staff rating based on current data, which is based on the current ICH CAHPS Survey (not the revised survey). We have not implemented the revised survey yet; however, this rating has not changed and we do not anticipate any unexpected findings.
Public Comments
Public Comment