The ICH CAHPS Survey is designed to measure the experiences of people receiving in-center hemodialysis care from Medicare-certified dialysis centers. The survey is designed to meet the following three broad goals:
- Produce comparable data from the patient’s perspective that will allow objective and meaningful comparisons between dialysis centers on domains that are important to consumers.
- Create incentives for dialysis centers to improve their quality of care.
- Enhance public accountability in health care by increasing the transparency of the quality of care provided in return for public investment.
Specifically, the survey measures patients’ experiences on topics that are important from the perspective of patients and help them make more informed choices when selecting a dialysis center as well as helping dialysis centers improve the quality of dialysis care for their patients.
The survey is administered semiannually to patients who have received in-center hemodialysis for at least 3 months from Medicare-certified dialysis centers. Data collection for each survey period is 12 weeks. The survey is available in mail-only, phone-only, and mail with phone follow-up.
Survey results publicly reported include two ratings and two measures:
• Global rating of dialysis center staff
• Global rating of dialysis center
• Quality of Dialysis Center Care and Operations measure (QDCCO, calculated from 13 survey questions)
• Providing Information to Patients measure (PIP, calculated from 9 survey questions)
The ICH CAHPS Providing Information to Patients (PIP) measure captures whether patients feel that their kidney doctors and dialysis center staff gave them the information they needed to take care of their health. For example, how to care for a graft, fistula, or catheter, patient rights, right treatment options, and how to get off a machine if there is an emergency.
The measure includes the following questions:
Q10: The dialysis center staff can connect you to the dialysis machine through a graft, fistula, or catheter. Do you know how to take care of your graft, fistula, or catheter?
Q17: As a patient you have certain rights. For example, you have the right to be treated with respect and the right to privacy. Did this dialysis center ever give you any written information about your rights as a patient?
Q18: Did dialysis center staff at this center ever review your rights as a patient with you?
Q19: Has dialysis center staff ever told you what to do if you experience a health problem at home?
Q20: Has any dialysis center staff ever told you how to get off the machine if there is an emergency at the center?
Q25: You can treat kidney disease with dialysis at a center, a kidney transplant, or with dialysis at home. In the last 12 months, did your kidney doctors or dialysis center staff talk to you as much as you wanted about which treatment is right for you?
Q27: In the last 12 months, has a doctor or dialysis center staff explained to you why you are not eligible for a kidney transplant?
Q28: Peritoneal dialysis is dialysis given through the belly and is usually done at home. In the last 12 months, did either your kidney doctors or dialysis center staff talk to you about peritoneal dialysis?
Q29: In the last 12 months, were you as involved as much as you wanted in choosing the treatment for kidney disease that is right for you?
Measure Specs
- General Information(active tab)
- Numerator
- Denominator
- Exclusions
- Measure Calculation
- Supplemental Attachment
- Point of Contact
General Information
One of the goals of the CMS National Quality Strategy is to foster engagement and to bring the voices of patients to the forefront. As part of fostering engagement, it is critical to hear the voice of individuals by obtaining feedback from them on in-center hemodialysis (ICH) facility performance and incorporating it as part of CMS’s comprehensive approach to quality. Patient-centeredness is a central goal of dialysis care and can be directly measured through surveys of dialysis patients. CMS created the In-Center Hemodialysis CAHPS® Survey, a component of the End-Stage Renal Disease (ESRD) Quality Incentive Program (QIP), to ensure that an assessment of the patient-centeredness of care would be included to monitor dialysis facility performance, promote quality improvement, and inform consumer decision making in the selection of a dialysis facility via public reporting of results. The ICH CAHPS Survey is a standardized survey instrument and data collection methodology for measuring ICH patients’ perspectives on their care in Medicare-certified dialysis centers. The survey is administered semiannually to patients who have received dialysis for at least 3 months from Medicare-certified dialysis centers.
The ICH CAHPS Providing Information to Patients (PIP) multi-item measure captures whether patients feel that their kidney doctors and dialysis center staff gave them the information they needed to take care of their health.
This measure reflects patient experiences with their dialysis facility across a variety of domains that have been identified as important to patients and stakeholders based on focus groups and cognitive interviews during the survey revision development stage. This measure will be publicly reported on Medicare.gov Care Compare tool to help ICH facilities with quality improvement and help ESRD patients find high quality dialysis facilities.
In addition to survey responses from patients, we receive data from a CMS database. ICH facilities are required to enter dialysis information for all patients into the ESRD Quality Reporting System (EQRS). EQRS data are then used to create the samples for each ICH CAHPS Survey period. In addition, information such as sex and age are pulled from the EQRS data and merged with survey response data files for analysis purposes. We work with the EQRS team on a continuous basis to remain up to speed on the data, changes to the data or data format, and to mitigate any issues with the data.
Numerator
CMS calculates ICH CAHPS Survey measure scores using top-box scoring. The top-box score refers to the percentage of respondents that give the most positive response(s): 9 items in the ICH CAHPS PIP measure use a “Yes/No” response scale where the top-box numerator is the number of respondents who answer “Yes .”
CMS calculates ICH CAHPS Survey multi-item measure scores using top-box scoring. The top-box score refers to the percentage of respondents that give the most positive response(s) (answers: Yes).
9 items in the ICH CAHPS PIP measure use a “Yes/No” response scale where the top-box numerator is the number of respondents who answer “Yes.”
The PIP measure is composed of the following survey items:
- The dialysis center staff can connect you to the dialysis machine through a graft, fistula, or catheter. Do you know how to take care of your graft, fistula or catheter?
- As a patient you have certain rights. For example, you have the right to be treated with respect and the right to privacy. Did this dialysis center ever give you any written information about your rights as a patient?
- Did dialysis center staff at this center ever review your rights as a patient with you?
- Has dialysis center staff ever told you what to do if you experience a health problem at home?
- Has any dialysis center staff ever told you how to get off the machine if there is an emergency at the center?
- You can treat kidney disease with dialysis, kidney transplant or with dialysis at home. In the last 12 months, did your kidney doctors or dialysis center staff talk to you as much as you wanted about which treatment is right for you?
- In the last 12 months, has a doctor or dialysis center staff explained to you why you are not eligible for a kidney transplant?
- Peritoneal dialysis is dialysis given through the belly and is usually done at home. In the last 12 months, did either your kidney doctors or dialysis center staff talk to you about peritoneal dialysis?
- In the last 12 months, were you as involved as much as you wanted in choosing the treatment that is right for you?
Denominator
ICH CAHPS Survey respondents are adult eligible patients who received dialysis care from an in-center hemodialysis facility during the sampling window. A survey is defined as completed when at least 50 percent of the questions applicable to all patients are answered.
The denominator for the ICH CAHPS PIP measure is the number of respondents with completed surveys who answer at least one item within the multi-item measure over the 2 semiannual periods being publicly reported.
ICH CAHPS Survey respondents are adult patients who received dialysis care from an in-center hemodialysis facility during the sampling window. A survey is defined as completed when at least 50 percent of the questions applicable to all patients are answered.
The denominator for the ICH CAHPS PIP multi-item measure is the number of respondents with completed surveys who answer at least one item within the multi-item measure over the 2 semiannual periods being publicly reported.
Exclusions
Cases are excluded from the multi-item measure denominator if:
- Patients are under 18 years of age
- Patients’ eligibility is unclear in mail survey
- Patients are not currently receiving dialysis
- Patients are deceased or receiving hospice
- Patients have not received ICH dialysis for at least 3 months
- Patients receive dialysis at a nursing home where they reside or at home
- Patients reside in jail or prison
- Patients are mentally or physically incapable
- Patients are no longer receiving care at sampled facility
- Patients have a language barrier
- Survey completed by a proxy
Denominator exclusions are based on final disposition codes from survey vendors, and approved during data cleaning activities. The cases with the following disposition codes assigned are excluded:
130 – Completed mail survey, survey eligibility unknown
140 – Ineligible: not currently receiving dialysis
150 - Deceased
160 – Ineligible: does not meet eligibility criteria (under 18, hospice, nursing home, less than 3 months, doesn’t receive dialysis at a center)
170 – Language barrier
180 – Mentally or physically incapacitated
190 – Ineligible: No longer receiving care a sampled facility
199 – Completed by proxy
Measure Calculation
The survey is administered semiannually to patients who have received in-center hemodialysis for at least 3 months from Medicare-certified dialysis centers. Data collection for each survey period is 12 weeks.
Survey results publicly reported include two ratings measures and two multi-item measures:
• Global rating of dialysis center staff
• Global rating of dialysis center
• Quality of Dialysis Center Care and Operations multi-item measure (QDCCO, calculated from 13 survey questions)
• Providing Information to Patients multi-item measure (PIP, calculated from 9 survey questions)
CMS calculates ICH CAHPS Survey measure scores using top-box scoring for completed surveys (a survey is defined as completed when at least 50% of the core questions applicable to all patients are answered). The top-box score refers to the percentage of respondents who give the most positive response(s).
ICH CAHPS Survey respondents are eligible adult patients who have received care from an ICH dialysis center for at least 3 months. The numerator for the multi-item measures is the number of most positive responses (9/10 for ratings and “Yes” or “Always” for multi-item measures). The denominator for the ICH CAHPS multi-item measures is the total number of respondents with completed surveys who answered at least one item within the multi-item measure.
Cases are excluded from the measure denominator if:
- Patients are under 18 years of age
- Patients’ eligibility is unclear in mail survey
- Patients are not currently receiving dialysis
- Patients are deceased or receiving hospice
- Patients have not received ICH dialysis for at least 3 months
- Patients receive dialysis at a nursing home where they reside or at home
- Patients reside in jail or prison
- Patients are mentally or physically incapable
- Patients are no longer receiving care at sampled facility
- Patients have a language barrier
- Survey completed by a proxy
More information on the risk adjustment and related calculations can be found in the two attachments submitted for 1.18a. One of the attachments is in Section 7 (supplemental).
The measure is not stratified.
The ICH CAHPS Survey is conducted semiannually each spring and fall. CMS-trained survey vendors can offer survey administration via mail-only, telephone-only, or mixed mode (mail with phone follow-up) to their client ICH facilities. The survey is available in English, Spanish, traditional and simplified Chinese, Samoan, and Vietnamese; the telephone interview is only available in English and Spanish. Vendors must administer the survey in English, but can choose whether to offer other languages.
We do not yet know response rates for the revised survey since it has not been implemented in the national implementation. Our overall response rate for the current ICH CAHPS Survey (not the revised survey) is around 30%; the mixed mode has the highest response rates of the 3 offered modes. Response rates are calculated for the instrument as a whole:
Response Rate = Total Number of completed surveys/(Total Number of Surveys Fielded - Total Number of Ineligible Surveys)
There is no minimum response rate requirement on ICH CAHPS. We are continuously working with survey vendors and ICH facilities to help improve response rates, by offering things such a flyers/posters and waiting room FAQs to place in facilities, training telephone interviewers on avoiding refusals, and using the CMS logo on mailing materials. We also have a dialysis patient page on our project website with survey FAQs and we reference this in the materials that are mailed to sample patients.
Every facility’s sample size differs, depending on the number of survey-eligible patients in the EQRS data, and the size of the facility. There is no minimum sample size per ICH facility. If a facility did not serve 30 survey-eligible patients in the preceding year, they are not required to participate in that year’s surveys. In order for ICH CAHPS scores (for the 2 ratings and 2 multi-item measures) to be publicly reported, a facility must have 30+ completed surveys across the two survey periods that are being reported during that Care Compare refresh on medicare.gov. For example, for the 2024 October Care Compare refresh period, a facility needed at least 30 completed surveys between the 2023 Spring and 2023 Fall Surveys, in order for the scores for the global ratings and QDCCO and PIP multi-item measures to be reported.
Supplemental Attachment
Point of Contact
Not applicable.
Elizabeth Goldstein
Woodlawn, MD
United States
Tracy Kline
RTI International
Research Triangle Park, NC
United States
Importance
Evidence
The Consumer Assessment of Healthcare Providers and Systems (CAHPS) set of patient experience surveys are well-established measures of healthcare quality. Each semiannual survey period, between 350,000 and 415,000 ICH patients are selected to receive the ICH CAHPS Survey. Public reporting of these survey results creates incentives for dialysis centers to improve their quality of care, directly impacting the patients who receive it. Because of this, it is important to ensure that the survey aligns with what patients believe constitutes high-quality care. Specifically, the survey measures patients’ experiences on topics that are important from the perspective of patients and where the patient is the best source of information. The results help dialysis patients make more informed choices when selecting a dialysis center as well as helping dialysis centers improve the quality of dialysis care for their patients.
In addition to a number of psychometric analyses, RTI conducted literature reviews, several rounds of focus groups and cognitive interviews with dialysis patients, conversations with ESRD stakeholders and Technical Expert Panels (TEP), and discussions with the CAHPS Consortium prior to finalizing a revised survey to test in a 2022 field test/mode experiment. Following the field test, results were discussed with the same groups of experts and all supported the revised survey.
During the focus groups, dialysis patients were asked about characteristics that were important to them in regards to high quality dialysis care. They were then asked about specific survey items identified for removal in the psychometric analysis. For each survey item, participants were asked how important questions were in rating and evaluating the care that they receive at their dialysis center. For the QDCCO measure, there was a mix of opinions on each question identified to be removed, but the consensus of the focus group participants was that if questions had to be removed from the measure, then these were the best to remove.
The 2020 and 2023 TEPs regarding survey revisions included 10 members each, consisting of dialysis patients, ESRD network representatives, a survey expert, dialysis patient advocates, and large dialysis organization representatives.
Data from the 2022 ICH CAHPS Survey field test/mode experiment do not allow for direct assessment of the relationship between survey measures and structures or processes. However, given that the modified instrument-derived multi-item measures (QDCCO and PIP) are the same or similar to current ICH CAHPS measures, CMS anticipates that these revised measures will exhibit similar relationships to those of the existing ICH CAHPS Survey measures. For example, an exercise comparing the current QDCCO multi-item measure (prior to item removal) to the revised QDCCO multi-item measure results in a near-perfect correlation (r=0.991) and illustrating the strong relationship between the current and revised state of the QDCCO multi-item measure.
The survey measures patients’ experiences on topics that patients have identified as important to measure the quality of care for facilities and for which the patient is the best source of this information. These surveys measures help dialysis patients make more informed choices when selecting a dialysis center as well as helping dialysis centers improve the quality of dialysis care for their patients. The PIP multi-item measure asks questions regarding whether patients are provided the information they need about their dialysis care and health. Focus groups/cognitive interviews with dialysis patients, conversations with ESRD stakeholders and Technical Expert Panels, and discussions with the CAHPS Consortium were conducted prior to the 2022 field test/mode experiment that tested the revised survey and after the field test/mode experiment results were finalized. All supported the revised survey, including keeping the PIP multi-item measure as is, with no changes.
Summary of literature findings associated with this measure:
1. Wael F. Hussein, Paul N. Bennett, Graham Abra, Emily Watson, Brigitte Schiller (2022). “Integrating Patient Activation into Dialysis Care,” American Journal of Kidney Diseases. Volume 79, Issue 1,2022, Pages 105-112, ISSN 0272-6386,https://doi.org/10.1053/j.ajkd.2021.07.015.
- Patients who are partaking in their care at higher degree, and to a degree they wish, have less complications, including hospitalizations and emergency department visits, improved outcomes, and overall, less burden on the care system.
- Patients with greater knowledge of their dialysis care, report increased confidence and less anxiety associated with their care.
- Patients with more knowledge and confidence this leads to improved outcomes and overall greater satisfaction with the care they are receiving.
- Patients receiving ICH report lower quality of life, for both mental and physical health.
- Patients want to improve quality of life, and some may find that with a different mode of dialysis.
- Important for physicians and dialysis facility to discuss all aspect of care and different modes of dialysis care, including Peritoneal dialysis at home.
- Giving more choice and switching dialysis care method can improve patient quality of life and increase independence.
- Every individual patient has specific priorities and needs related to the dialysis treatment they receive. Nephrology community should better understand patient needs and communicate information and management of dialysis care that is right for patient.
- Patients who have discussions and more information leads to better satisfaction and perceived quality of care.
2. Antoun, J., Brown, D.J., Jones, D.J.W., Clarkson, B.G., Shepherd, A.I., Sangala, N.C. et al. (2023) Exploring patients' experiences of the impact of dialysis therapies on quality of life and wellbeing. Journal of Renal Care, 49, 15–23. https://doi.org/10.1111/jorc.12416
3. Tommel J, Evers AWM, van Hamersvelt HW, et al. “What matters to you?”: The relevance of patient priorities in dialysis care for assessment and clinical practice. Semin Dial. 2023; 36(2): 131-141. doi:10.1111/sdi.13080
Measure Impact
A successful survey should be relevant to the target audience and produce meaningful results that can be used to inform decisions. For the ICH CAHPS Survey, this includes a survey that accurately measures characteristics of quality dialysis care from the patient’s perspective and produces results that ICH centers can use to improve their care.
Each survey period, between 80 and 100k sample patients respond to the current ICH CAHPS Survey, indicating that the survey is meaningful and that they feel that providing their information is valuable. The survey focuses on topics that ESRD patients have reported as being important in defining high-quality dialysis care; these patients are the only source of this information. During survey revisions, we met with 2 focus groups of 9 people who all noted that they felt the ICH CAHPS Survey is important and were happy to know that it was being reduced in length. CMS also receives a number of letters from dialysis patients noting that they believe the survey is valuable and that they appreciate being able to convey their thoughts and have their voices heard.
During the 2024 Pre-Rulemaking Measure Review (PRMR) process, the Patients for Patient Safety US organization commented the following: We certainly support this, because the experiences of patients is a valuable data source, and we're happy to see it included in instruments like CAHPS as well as other ways in which we use PROMs and PREMs. I will say that our organization Patients for Patient Safety US does see the CAHPS surveys as an incredibly important tool, not only for feeding back patient experience, but improving health equity, by bringing in people who are less likely to use these surveys by asking the right kinds of questions and including meaningful questions in them and I think this falls into that category.
Activities conducted by RTI during the survey revision process, such as focus groups, interviews, and TEP meetings, helped to ensure that the survey is relevant to the target audience, and that survey results are meaningful to both patients and the ICH facilities that provide care.
The PIP measure is the only ICH CAHPS multi-item measure that has internal consistency and inter-unit reliability estimates slightly exceeding the threshold value of 0.7. Therefore, the PIP measure meets or just slightly exceeds the standard acceptability threshold. Because of this, psychometric analyses showed that removing questions from this measure would most likely tip this measure into the realm of not meeting the threshold. The TEP and CAHPS Consortium agreed that no questions should be removed from this measure at this time; no questions were presented to patients (during focus groups) for removal from this measure.
Performance Gap
We have provided the distribution of scores in Table 1 as requested. The information in the table is based on top-box calculated scores. The data used to calculate the deciles comes from the 2023 Spring and 2023 Fall National Implementation data. Due to the sampling design for the mode experiment, to have the least impact on public reporting, facility-level scores could not be calculated so national implementation was used as proxy. The facility-level performance scores for each survey period are created using top-box, and then combined to create the 2023 performance scores. In total, 6700 facilities and 178,480 survey responses are represented in this data.
| Overall | Minimum | Decile_1 | Decile_2 | Decile_3 | Decile_4 | Decile_5 | Decile_6 | Decile_7 | Decile_8 | Decile_9 | Decile_10 | Maximum | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Mean Performance Score | 77.98 | 0 | 68.321 | 71.976 | 74.503 | 76.580 | 78.464 | 80.398 | 82.246 | 84.381 | 87.436 | 100 | 100 |
| N of Entities | 6,700 | 1 | 670 | 670 | 670 | 670 | 670 | 670 | 670 | 670 | 670 | 670 | 3 |
| N of Persons / Encounters / Episodes | 178,480 | 1 | 15,696 | 19,544 | 20,545 | 19,852 | 20,080 | 18,628 | 18,553 | 17,327 | 16,490 | 11,765 | 3 |
Equity
Equity
This domain is optional for Spring 2025.
Feasibility
Feasibility
As these patient-experience data are collected from patients based on their ICH dialysis care experiences, the structured data are not available in electronic sources outside of this data collection. Web-based data collection was tested in the 2022 ICH CAHPS Survey field test/mode experiment, but due to a lack of email addresses available in the sampling data, we had very few responses. We are working now to determine whether we can obtain additional email addresses to retest web data collection in a future mode experiment.
Proposed revisions to the current ICH CAHPS Survey instrument include shortening it and was developed through focus groups, cognitive interviews, and other instrument-development activities conducted for this revised instrument. The changes to the current ICH CAHPS survey were implemented to mitigate the challenges and barriers to responding to the current full survey. The shortened survey reduces patient burden and is expected to increase response rates.
The ICH CAHPS Survey is administered by independent survey vendors that are approved by CMS, and CMS’s implementation contractor provides oversight on a regular basis to ensure that the vendors are following established protocols. Additionally, the survey vendors are required to conduct regular review and monitoring of their own operational systems, whether the survey is administered by mail or telephone. Data is assessed semiannually for accuracy and missing data.
ICH CAHPS Survey results for the updated survey instrument will be fully publicly reported in October 2027 (2026 Spring + 2026 Fall data). However, because the April 2027 refresh would include a survey period that used the current survey (2025 Fall) and a survey period that used the revised survey (2026 Spring), we would plan to reanalyze the 2025 Fall data based on the revised survey measures and case-mix, then combine the reanalyzed data with the 2026 Spring data for public reporting in April 2027; therefore, we are not missing a refresh for ICH CAHPS data.
The following revisions were made to shorten the ICH CAHPS Survey:
The Nephrologist Communication and Caring (NCC) Measure was removed, which included the following questions:
- In the last 3 months, how often did your kidney doctors listen carefully to you?
- In the last 3 months, how often did your kidney doctors explain things in a way that was easy for you to understand?
- In the last 3 months, how often did your kidney doctors show respect for what you had to say?
- In the last 3 months, how often did your kidney doctors spend enough time with you?
- In the last 3 months, how often did you feel your kidney doctors really cared about you as a person?
- Using any number from 0 to 10, where 0 is the worst kidney doctors possible and 10 is the best kidney doctors possible, what number would you use to rate the kidney doctors you have now?
- Do your kidney doctors seem informed and up-to-date about the health care you receive from other doctors?
The following questions were removed from the QDCCO Measure:
- In the last 3 months, did dialysis center staff keep information about you and your health as private as possible from other patients?
- In the last 3 months, how often did dialysis center staff insert your needles with as little pain as possible?
- In the last 3 months, did dialysis center staff talk to you about what you should eat and drink?
- In the last 3 months, how often did you feel your kidney doctors really cared about you as a person?
The following questions were also removed from the survey; these items were not included in a measure:
- In the last 3 months, has anyone on the dialysis center staff asked you about how your kidney disease affects other parts of your life?
- Medicare and your State have special agencies that check the quality of care at this dialysis center. In the last 12 months, did you make a complaint to any of these agencies?
- Are you being treated for high blood pressure?
- Are you being treated for diabetes or high blood sugar?
- Are you being treated for heart disease or heart problems?
- Are you deaf or do you have serious difficulty hearing?
- Are you blind or do you have serious difficulty seeing, even when wearing glasses?
- Because of a physical, mental, or emotional condition, do you have serious difficulty concentrating, remembering, or making decisions?
- Do you have serious difficulty walking or climbing stairs?
- Do you have difficulty dressing or bathing?
- Because of a physical, mental, or emotional condition, do you have difficulty doing errands alone, such as visiting a doctor’s office or shopping?
- Who helped you complete this survey?
ICH facilities that served 30 or more survey-eligible patients in the preceding year are required by the ESRD PPS Rule to participate in the current calendar year’s two surveys. They are required to contract with one of the CMS-approved ICH CAHPS Survey vendors, who administers the survey on their behalf. Neither CMS nor RTI are involved in payment discussions between the vendors and facilities, and we have no cost information in terms of the facilities. The proposed revisions will not cause a significant burden to vendors or CMS’s implementation contractor, other than the modification of existing computer programs. Regarding burden to sample dialysis patients, the revised survey will take approximately 12 minutes of their time up to two times each year.
The sample is drawn from EQRS data, meaning that facilities are not involved and do not know who was selected to participate in the survey (unless a sample patient specifically tells them). Sample patients are asked to not ask for help from facility staff, and facilities are told not to assist their patients in completing the survey if asked and not to ask any patient whether they were sampled for the survey.
ICH CAHPS Survey data submitted to the ICH CAHPS Data Center (CMS’s implementation contractor) are de-identified. Case-level data are assigned a unique case ID, randomly generated so that there is no identifying information embedded in it. ICH CAHPS Survey results are aggregated at the facility level and publicly reported at the facility-, state- and national-level.
Vendors are allowed to provide unofficial aggregated results to their facility clients; however, there are rules on sharing such information when the number of responses is too low.
After the 2022 field test/mode experiment, and after the results were discussed with stakeholders (Technical Expert Panel members representing different ESRD organizations and large dialysis organizations, as well as the CAHPS Consortium), CMS approved the deletion of multiple items from the current nationally fielded ICH CAHPS Survey. The revised instrument is being submitted via this Instrument submission. For the 4 instrument-derived measures (2 ratings and 2 multi-item measures), 3 are the same as the current ICH CAHPS Survey and one is slightly modified.
Proprietary Information
Scientific Acceptability
Testing Data
The revised ICH CAHPS Survey was tested in fall 2022 during a field test/mode experiment. The total sample for the field test/mode experiment, based on EQRS data, was 24,354. Data were collected in English and Spanish, by mail, telephone, mail with telephone follow-up, and web with mail follow-up. Analyses are based on the field test/mode experiment data collected between October 2022 and January 2023, including 4605 respondents representing 3211 facilities. Field test/mode experiment data were used for the internal consistency analyses (Reliability – Section 5.2), factor analysis (Validity – Section 5.3), correlations among ratings and multi-item measures (Validity – Section 5.3), and risk analysis (IDM – Risk Adjustment – Section 5.4) analyses. For the signal-to-noise (IDM – Reliability – Section 5.2) and performance gap (IDM – Performance Gap – Section 2.4) analyses, national implementation data was used, which was collected during the 2023 Spring and 2023 Fall Surveys.
The national ICH CAHPS Survey is offered in six languages: English, Spanish, Chinese (simplified and traditional), Samoan, and Vietnamese. Additional translations will be made as needed.
There are no fees or licensing for use of the ICH CAHPS® Survey, training or oversight activities, or for accessing publicly reported ICH CAHPS® Survey measure scores or star ratings on the CMS Medicare.gov website.
None
For the field test/mode experiment that was conducted from October 2022 through January 2023, we took a sample of 24,354 ICH patients, using a stratified sampling method so as to have the least impact on public reporting. There were 6,736 facilities represented in the field test/mode experiment sample, located throughout the country. Facilities ranged in size from very small to very large. To have the least impact in public reporting, facilities that always meet the public reporting threshold (always have 30+ completed surveys for the two reporting periods) and facilities never meet the public reporting threshold (do not come close to the 30+ completes) were oversampled, and facilities that are normally right on the threshold line were undersampled. We randomly assigned sampled patients to one of the four data collection modes using the inverse of the estimated response rates to achieve the desired sample size in each mode. The final sample closely mirrored the national population of dialysis centers. Analyses were conducted using data from the 4605 respondents, who represented 3211 facilities.
For the 2023 Spring/Fall national implementation data used for facility-level analyses, all facilities registered on the ICH CAHPS website, by each dialysis facility that has determined they are required to participate in the surveys, were included in the sample if they had sample in the EQRS data. For the 2023 Spring/Fall Survey national implementation data, the following sample strategy was used:
Facilities with up to 240 patients: A census of all survey-eligible patients will be conducted for facilities with fewer than 240 survey-eligible patients at each semiannual sampling wave. Thus, patients at these ICH facilities may be sampled twice in a 12-month period. Facilities with 240 or more patients.
For dialysis centers with 240 or more survey-eligible ICH patients: A simple random sample will be selected for each sampling period, with the goal of obtaining 200 completed surveys per year while attempting to minimize the overlap of patients between subsequent semiannual waves of sampling.
The 2023 Spring Survey included 6675 facilities across the US and its territories, and the 2023 Fall Survey included 6674 facilities. Facility size differed from small to large; facilities were required to participate if they served 30+ survey-eligible patients in the prior calendar year.
For the field test/mode experiment, our sampling design randomly assigned sample patients to one of the four data collection modes, with a target of completing approximately 1,570 interviews for each mode. This design allowed us to measure mode effects related to nonresponse and measurement differences (e.g., because of social desirability) and also allowed for case-mix analyses. 24,354 ICH patients were sampled for the field test/mode experiment, using a stratified sampling method so as to have the least impact on public reporting. To have the least impact in public reporting, facilities that always meet the public reporting threshold (always have 30+ completed surveys for the two reporting periods) and facilities never meet the public reporting threshold (do not come close to the 30+ completes) were oversampled, and facilities that are normally right on the threshold line were undersampled. We randomly assigned sampled patients to one of the four data collection modes using the inverse of the estimated response rates to achieve the desired sample size in each mode. The final sample closely mirrored the national population of dialysis centers. Analyses were conducted using data from the 4605 respondents, who represented 3211 facilities.
For the 2023 Spring/Fall Survey national implementation data, the following sample strategy was used:
Facilities with up to 240 patients: A census of all survey-eligible patients will be conducted for facilities with fewer than 240 survey-eligible patients at each semiannual sampling wave. Thus, patients at these ICH facilities may be sampled twice in a 12-month period. Facilities with 240 or more patients.
For dialysis centers with 240 or more survey-eligible ICH patients: A simple random sample will be selected for each sampling period, with the goal of obtaining 200 completed surveys per year while attempting to minimize the overlap of patients between subsequent semiannual waves of sampling.
For the 2023 Spring, 378,513 patients were sampled. 91,914 respondents, representing 6609 facilities across the nation and its territories, were included in the analyses. For the 2023 Fall, 383,116 patients were sampled, 86,566 respondents representing 6595 facilities, were included in the analyses.
Eligibility criteria for the field test/mode experiment sample mirrored the criteria used for national implementation. Sample patients must:
- Be at least 18 years of age or older,
- Have received hemodialysis at an in-center facility for 3 months or longer,
- Be alive as of the last day of the sampling window, and
- Not be institutionalized or receiving hospice.
Attachment 5.1.4 with patient characteristics can be found in Supplemental 7.1 zip file.
Reliability
Psychometric testing at the instrument level includes a reliability assessment using the Cronbach’s alpha estimate, from the internal consistency analysis of measurement error. The reliability analysis was conducted at the item and patient level, using SAS’s PROC CORR procedure with ALPHA notation specified, which employs listwise deletion of missing data. Available response data from the revised survey field test/mode experiment was included as input for the internal consistency analysis.
Output from the PROC CORR was evaluated against psychometric thresholds of acceptable internal consistency. In general, Cronbach’s alpha estimates of above 0.9 were evaluated as good internal consistency and 0.7 was considered a minimum threshold for acceptability in this analysis (Nunnally & Bernstein, 1994).
Citation:
Nunnally J, Bernstein L. Psychometric theory. New York: McGraw-Hill Higher, INC; 1994.
The attachment provides the internal consistency results for the Quality of Dialysis Center Care and Operations (QDCCO) and Providing Information to Patients (PIP) item sets. The item-total correlations are provided in the second column from the right, and the alpha estimate for the set (if a particular item is removed) is in the far right. In summary, the 13 QDCCO items have a standardized Cronbach’s alpha estimate of 0.930 which exceeds the threshold for good internal consistency. In addition, the standardized item-total correlations range from 0.54 to 0.78 and none of the alpha estimates improve substantially with any item removed.
The 9 PIP items have a standardized Cronbach’s alpha estimate of 0.743 which is just above the minimal acceptability threshold. The standardized item-total correlation range for the PIP item set has a minimum of 0.29 and a maximum of 0.55, and again, none of the alpha estimates improve substantially with any item removed.
The internal consistency estimates for the QDCCO item set (α = 0.930), and PIP item set (α = 0.743) are above the minimal acceptability threshold (Nunnally & Bernstein, 1994). In addition,
most of the item-total correlations across multi-item measures are above 0.5, the remaining primarily above 0.3, with one value at 0.29. Therefore, the items assigned to each multi-items measure (QDCCO and PIP) function well together and we conclude that the items perform consistent measurement of the representative constructs.
The signal-to-noise analysis, termed inter-unit in this documentation, was conducted at the facility and derived measure level using the CAHPS 5.0 macro. For the overall estimate, 2023 Spring and Fall survey data was subset to facilities that had at least 30 completed cases between the two survey periods, to match the Public Reporting standards. 2023 Spring and Fall National Implementation survey data was used as a proxy due to the design of the 2022 field test/mode experiment, which was sampled in a way to ensure that the impact on Public Reporting data would be minimal and therefore not suitable for this analysis.
The CAHPS 5.0 macro calculates the global F-test to determine differences among entities on measures or items. F-statistics are transformed into reliability such that if the F-statistic is around 1, the reliability will be close to zero and indicate differences are due to random variation. The larger the real differences among facilities, the larger the F-statistic, and the higher the reliability estimate (approaching 1.0) and reliabilities above 0.7 are considered acceptable.
CITATION: Keller S, O’Malley AJ, Hays RD, Matthew RA, Zaslavsky AM, Hepner KA, Cleary PD. Methods Used to Streamline the CAHPS® Hospital Survey. Health Services Research, 2005, 40, 2057-2077. PMIDp: 16316438.
The PIP multi-item measure’s overall inter-unit reliability, using the 2023 Spring and Fall National Implementation survey data, is 0.71 (F= 3.39, p<0.001). The information in Table 2 provides overall information as well as reliability by decile. Deciles for this table are based on psychometrics done within the CAHPS Macro; and the “mean performance score” row in this table is showing variation in the mean value by sample size. Deciles were specified on facilities that have 30 or more completes to match the Public Reporting standards. The extreme specifications included facilities with the smallest possible (1 or 2 completes) and close to the maximum number (100 completes or more) were specified as such to provide enough facilities to conduct the analysis.
| | Overall | Minimum | Decile_1 | Decile_2 | Decile_3 | Decile_4 | Decile_5 | Decile_6 | Decile_7 | Decile_8 | Decile_9 | Decile_10 | Maximum |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Reliability | 0.71 | 0.90 | 0.69 | 0.72 | 0.66 | 0.66 | 0.65 | 0.65 | 0.65 | 0.80 | 0.78 | 0.79 | 0.78 |
| Mean Performance Score | 0.78 | 0.75 | 0.78 | 0.78 | 0.78 | 0.78 | 0.77 | 0.78 | 0.78 | 0.79 | 0.77 | 0.78 | 0.79 |
| N of Entities | 2319 | 74 | 441 | 243 | 328 | 239 | 215 | 225 | 206 | 153 | 162 | 107 | 22 |
| N of Persons / Encounters / Episodes | 100828 | 115 | 13652 | 8118 | 11766 | 9312 | 9009 | 10236 | 10301 | 8468 | 10385 | 9581 | 2564 |
The inter-unit reliability assessment for the ICH CAHPS PIP multi-item measure indicates that overall, and at several deciles of interest, the measure maintains close to, or above acceptable performance limits (e.g., greater than 0.7) in finding real variation among facilities relative to random variation. In the instances where the inter-unit reliability is lower than 0.7, the estimate maintains sufficient strength to meet the requirement for acceptability (≥ 0.6) in the E&M Guidebook evaluation rubric (version 2.1).
Validity
Changes to the items within the QDCCO multi-item measure were assessed with face validity. Next, structural and convergent validity were used to determine whether the QDCCO and PIP item sets measure their respective construct as intended.
Confirmatory factor analysis (CFA) was used to determine structural validity of the QDCCO and PIP item sets through model fit statistics compared to acceptability thresholds. The thresholds for model acceptability are as follows:
RMSEA of 0.05 or less (Browne & Cudeck, 1993)
CFI of 0.90 at a minimum (Hu & Bentler, 1999)
TLI of 0.90 at a minimum historically, 0.95 indicates good fit
Convergent validity was also assessed through CFA by evaluating factor loadings for each item. CFA analyses were conducted using the Mplus analytic software (Version 8.11; 2023) using the Weighted Least Squares Mean and Variance adjusted (WLSMV) estimator to account for categorical data with pairwise deletion for missing data. In addition, correlations, conducted using SAS PROC CORR, evaluating the two multi-item measures from the item sets (QDCCO and PIP) with ratings of the Dialysis Center and Dialysis Staff, were provided for convergent validity.
Citations:
Browne, M. W., & Cudeck, R. (1993). Alternative ways of assessing model fit. In K. A. Bollen & J. S. Long (Eds.), Testing structural equation models (pp. 136–162). Thousand Oaks, CA: Sage.
Hu, L., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6(1), 1–55. http://dx.doi.org/10.1080/10705519909540118
Muthén, L. K., & Muthén, B. O. (1998-2023). Mplus (Version 8.11) [Computer software]. Los Angeles, CA: Muthén & Muthén
Glossary:
RMSEA - root mean square error of approximation
CFI - comparative fit index
TLI - Tucker-Lewis Index
All experts confirmed face validity and agreed that the remaining items still provided construct representation of the quality of a facility’s care and operations. In addition, psychometric results showed that the removal of 4 questions from the QDCCO did not negatively impact the measure’s validity.
Confirmatory factor analysis (CFA) model fit to assess structural validity produced acceptable values, consistent with a well-fitting model. These estimates are:
RMSEA=0.038 (95% CI = 0.036 0.040)
CFI=0.987
TLI=0.985
The attachment provides the estimates for each standardized factor loading on the representative items to assess convergent validity. Correlative information on the factors is also provided, and moderate. All factor loadings for the QDCCO and PIP item sets are above 0.4, with the majority above 0.5, and all are statistically significant (p < 0.001).
The final assessment of convergent validity through correlation information results in the relationship between each averaged measure, and the Dialysis Staff Rating (rQDCCO = 0. 0.757, rPIP = 0.434) and the Dialysis Center Rating (rQDCCO = 0.737, rPIP = 0.403), all of which are statistically significant (p<0.001).
The QDCCO and PIP item sets continue to effectively measure consistent constructs. Confirmatory factor analysis (CFA) produced appropriate model fit indices and statistically significant factor loadings. Correlational analyses showed moderate relationships to patient ratings of the overall facility and the staff. The findings confirm the valid measurement of the QDCCO and PIP constructs.
Validity was assessed at the measure and facility level using correlational methods and SAS’s PROC CORR procedure, which employs pairwise deletion of missing data. Available measure scores from each facility were derived from the National Implementation survey data. The 2023 Spring and Fall National Implementation data was used as a proxy due to the design of the 2022 field test/mode experiment, which was sampled in a way to ensure minimal Public Reporting data impact. Output from the PROC CORR analysis was evaluated against standard correlational thresholds. In general, a correlation of 0.50 shows a moderate relationship and estimates higher are indicative of stronger relationships.
Discriminant validity was assessed at the facility level by examining correlations among the PIP items and the derived measure for both the QDCCO and PIP item sets. The expectation is that an item’s correlation with the PIP measure (without the item of interest) will be higher than the correlation with QDCCO measure.
Convergent validity was also assessed through correlations at the multi-item measure and facility level and examined the relationship between the PIP measure and rating of the Dialysis Center, and then again with the rating of Dialysis Center Staff.
CITATION
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillside, NJ: Lawrence Erlbaum Associates
Discriminant validity results, using the 2023 Spring and Fall National Implementation survey data, are presented in the attached table. As expected, the PIP items show stronger relationships with the PIP multi-item measure than the QDCCO multi-item measure. In addition, convergent validity results show that the relationship between the PIP measure, and the Dialysis Center Staff Rating (rPIP = 0.488) and the Dialysis Center Rating (rPIP = 0.457) are approaching the established moderate threshold (0.50) and statistically significant (p<0.001).
The PIP multi-item measure represents the construct of patients’ expectation of information. Correlational analyses showed smaller relationships to alternative measures not intended to measure provided patient information and relationships approaching moderate strength to patient ratings of the overall center and the center staff. The findings confirm that the ICH CAHPS items result in the valid measurement, specifically of the PIP multi-item measure variable, without replicating other information within the survey.
Risk Adjustment
All case-mix/risk adjustment analyses were conducted using the data from the 2022 field test/mode experiment. The independent variables that were included in the mode and case-mix models are those hypothesized to affect response tendencies, and the results of the variables included in the two multi-item measures and two ratings. These independent variables are used as case-mix adjusters for ICH CAHPS scores for ICH facilities. These independent variables are not under the control of the facility, meaning a facility can’t change these descriptors of their patients (e.g., a facility can’t control the demographic make-up of their population with regards to age, sex, overall health, overall mental health, etc.). Adjusting for this helps to ensure that variations in case-mix do not affect the CAHPS scores calculated and publicly reported for ICH facilities because these factors impact response tendencies and are outside the control of the ICH facilities.
These independent variables came from four sources: (1) the ICH CAHPS Survey mode experiment implementation; (2) the ICH CAHPS Survey mode experiment as patients’ self-reported characteristics; (3) the Common Medicare Environment (CME) database; and (4) the EQRS database.
Using the case-mix model, the facility’s case-mix adjusted scores are adjusted for differences between a facility’s patient composition according to the ICH CAHPS case-mix characteristics and the overall national composition of ICH patients on these same characteristics. This adjustment allows consumers to compare different ICH facilities based on the same overall patient composition, thus allowing for a comparison that isn’t affected by the given facility’s patient composition.
We have attached a file showing the descriptive statistics, submitted under 7.1 other attachments.
We conducted the multivariate regression analysis assessing the candidate case-mix variables. We estimated 50 multivariate regression models for the mode and case-mix analysis, including one regression model for each question comprising the two multi-item measures and two ratings for the top-box outcomes. When estimating the multivariate regression models, we used the individual patient as the unit of analysis. For the top-box outcomes the dependent variables for all the multivariate regression models were converted to a binary score with the most positive response set to 1 and all other nonmissing responses set to 0.
We used ordinary least squares models to investigate the candidate case-mix variables which included:
• Mode of survey administration.
• Language survey was completed. Binary yes/no response categories for whether the respondent completed the survey in English or Spanish.
Patients’ Self-Reported Characteristics Independent Variables
• Overall health. Including five categories for self-reported overall physical health status.
• Overall mental health. Including five categories for self-reported mental or emotional health status.
• Education. Including six response categories for self-reported education.
• Did someone help you complete this survey. Including binary yes/no response categories for whether the respondent had someone else help them to complete the ICH CAHPS Survey.
• Low-income subsidy or dually enrolled. This variable is calculated from variables indicating if the respondent had a low-income subsidy or was dually enrolled in both Medicare and Medicaid at any time during the survey period.
• Age. This variable was calculated from the respondent’s date of birth. We created five categories for a categorical age variable.
• Sex. Binary male/female categories.
• Total years on dialysis. This variable includes six response categories for different time periods for the respondent since they originally started dialysis treatment. This variable is calculated from the EQRS variable for the date dialysis started for this patient.
• Was diabetes the primary cause of ESRD. This variable includes binary yes/no response categories for whether diabetes was the cause of ESRD for this patient. This variable is calculated from the EQRS variable for the cause of ESRD.
The facility indicator variable was not included in the models for this analysis. The number of respondents from each of the facilities ranged from only one to eight with the majority having only one respondent, thus the cluster effects were not important. Generally, the linear form of the multivariate regression models was:
Dependent variable = sum of (coefficients*mode indicators) + sum of (coefficients*patient characteristic indicators)
When there were categorical patient characteristics for an independent variable, such as age groups, one group was used as the reference category from the set of categories included in the regression model. That group is the reference to which the effects of the other categories for that variable are compared, which simplifies the interpretation of the regression coefficients.
RTI randomly assigned patients to mode.
Sequential Modeling
We estimated each of the regression models using the independent case-mix variables previously noted. All of the independent variables were statistically significant for at least one of the regression models.
The first set of multivariate regression models run on the 25 dependent variables included as independent variables all of the case-mix factors in their original form. We refer to this first set of models as the full model or Model 1. In the second set of multivariate regression models (Model 2), we recategorized the mental health question by rolling together the fair and poor categories to reduce the VIFs for this variable (a finding from the multicollinearity analysis). In the third and final set of regression models (Model 3), we dropped the low-income subsidy or dually enrolled variable due to the difficulty of obtaining this variable and the high level of missing values (13.18%) for this variable. Model 3 was our final model.
To determine the recommended regression model, we calculated the patient-level case-mix adjusted scores from the three models investigated. Then we calculated the two scores by averaging the patient-level case-mix adjusted scores for each survey item comprising a given multi-item measure. We used the results of the patient-level case-mix adjusted measure and global rating scores from the three models and the unadjusted scores to conduct an impact analysis. The goal of the impact analysis was to understand how the results from the three case-mix regression models changed compared to the unadjusted scores and to each other.
Using the three sets of adjusted patient-level scores and the unadjusted scores, we calculated the absolute value of the difference between each score considering all possible combinations (e.g., unadjusted vs. Model 1, unadjusted vs. Model 2, unadjusted vs. Model 3). The mean of these percentage point differences is found in Table 1of the attachment in 5.4.5a.
For the top-box scores, the absolute values of the percentage point differences range from 0.95 for QDCCO patient-mix adjusted top-box score using Model 2 compared to the case-mix adjusted top-box score using Model 1 to 10.43 for the global rating of the dialysis center unadjusted top-box score compared to patient-mix adjusted top-box score using Model 2. In general, the percentage point differences between the case-mix adjusted scores for the three models is small with the differences between Model 2 and Model 3 being the smallest.
Using the three sets of adjusted patient-level scores and the unadjusted scores, we calculated the Spearman’s rank correlation coefficient. Spearman’s rank correlation quantifies the strength and direction of association between two ranked variables (i.e., how similar they are). A high correlation (close to 1) indicates that the two variables are strongly related and a low correlation (close to 0) indicates that the two variables have a weak or no relationship. Negative values (close to −1) indicate that the two variables are strongly inversely related.
In this analysis, correlations close to 1 indicate that the different sets of patient-mix models are adjusting the scores similarly. The Spearman’s Rank Correlations are found in Table 2of the attachment in 5.4.5a for the two global ratings and two multi-item measure outcome scores. These correlations demonstrate the strength of the relationship of the global ratings and multi-item measure outcome measures between the adjusted patient-level scores using the three patient-mix models and the unadjusted scores. For each measure the lowest correlations existed when comparing to the unadjusted scores. For example, for the top-box global rating of the dialysis center, the correlation for the unadjusted scores and Model 1 was 0.81. The same correlation was found for the unadjusted and Model 2, and for unadjusted and Model 3.
For the two ratings and two multi-item measure the pairwise Spearman’s Rank Correlations for the adjusted scores using Model 2 compared to Model 3 were either 0.98 or 0.99 and the correlations using Model 1 to Model 2 ranged from 0.97 to 0.99. Given that all the correlations between the adjusted scores were so close to 1, we concluded that there are very few differences in the patient-level adjusted scores when using the different sets of patient-mix variables in the multiple regression models.
Final Case-Mix Regression Model
The three sets of multiple regression models tested performed very similarly for top-box. The Spearman’s Rank Correlations were all close to 1 demonstrating that the ranks of the adjusted scores were very similar. This suggests very small differences in the results of the adjusted scores for the different sets of case-mix models. Furthermore, the percentage point differences between the three models were less than 2 for all measures, hence we recommend the simplest model without the low-income subsidy or dually enrolled variable. This is Model 3 with the following case-mix variables:
overall health,
overall mental health (with fair and poor categories rolled together),
patient age,
patient sex,
patient education,
language survey was conducted (English or Spanish),
help completing the survey,
total years on dialysis, and
was diabetes primary cause of ESRD
Use & Usability
Use
Dialysis facility
Usability
Dialysis center scores will increase as they target these measures, therefore improving patients’ perspective of their dialysis care. Each dialysis center may have different training mechanisms or internal procedures which can help improve the overall quality of the services they deliver.
In an attempt to increase survey response rates (performance on the survey overall) to have enough data to be publicly reported, ICH facilities are encouraged to hang posters/flyers and waiting room FAQs in their facilities, so that patients understand that the ICH CAHPS survey is legitimate.
ICH facility scores will increase as facilities target the PIP measure, which asks for patients’ perception of whether their kidney doctors and dialysis center staff gave them the information they needed to take care of their health. Each ICH center may have different training mechanisms or internal procedures which can help improve the overall quality of the services they deliver.
The revised instrument has not been implemented beyond the field test stage; however, we did receive positive feedback during the Pre-Rulemaking Measure Review (PRMR) process. The PRMR hospital committee recommended the revised survey. The PRMR committee recognized the importance of patient experience of care data and supported efforts to reduce the length of the survey while maintaining scientific acceptability. The CAHPS Consortium noted that we should remove one additional question (which was done after more analyses) and then move forward with the revised survey; they felt that it would help survey burden and was a great start at reducing a very long survey. In addition, feedback was obtained from several large dialysis organizations (LDOs) during TEP meetings and email correspondence. They noted that this was a good start at reducing the survey but would like to see it reduced even more in the future. The consensus from the LDOs was that the revised survey didn’t remove any major questions that would raise alarm.
The proposed PIP multi-item measure (same as the current PIP measure used in national implementation) will be one of four ICH CAHPS publicly reported.
Feedback from ESRD stakeholders and the CAHPS Consortium, as well as patients during focus groups and cognitive interviews, was positive for keeping the PIP measure as is. TEP members noted that giving patients the opportunity to answer the questions associated with the PIP measure is important and therefore should be kept in the revised survey; it gives patients a way to quantify the care they receive.
Although this revised survey has not been implemented yet, final decisions were based on feedback from many stakeholders, including representatives from our LDOs. During Technical Expert Panel (TEP) meetings and via email discussions, feedback was provided on which questions could safely be dropped from the current survey, to create a revised survey to reduce patient burden. The TEP supported efforts to shorten the survey to reduce burden on respondents. They noted that there were no major concerns with the questions removed and appreciated the effort to make this a little easier on the patients responding to the survey.
Although this revised survey has not been implemented yet, final decisions were based on feedback from many stakeholders, including representatives from our LDOs. During TEP meetings and via email discussions, feedback was provided on the PIP measure and it was decided to keep it as is.
The revised survey has not been implemented yet so we have no improvement data based on this measure as of yet; but this measure is not changing.
The scores for the current PIP multi-item measure, for facilities with at least 20 responses from 2015 thru 2023, have remained fairly steady, dropping very slightly in 2022. The median score for the PIP measure was 80.2 in 2015, 80.3 in 2021, 79.4 in 2022, and bouncing back up in 2023 to 79.7. The other deciles exhibited a similar pattern.
The revised survey has not been implemented beyond the field test/mode experiment phase. No unexpected findings were observed during the field test/mode experiment. Respondents appreciated the shorter survey.
There have been no unexpected findings of the PIP multi-item measure based on current data, which is based on the current ICH CAHPS Survey (not the revised survey). We have not implemented the revised survey yet; however, the PIP measure has not changed and we do not anticipate any unexpected findings.
Comments
Staff Preliminary Assessment
CBE #0258-3 Staff Preliminary Assessment
Importance
Strengths
- A logic model is provided, in which the composite scale 'Providing Information to Patients measure (PIP)' measure is positioned as an outcome, with inputs that include ‘staffing and resources,’ and impacts such as ‘selecting and engaging patient in dialysis treatment best for them’ and ‘Patient Driven Dialysis Center.’ The developer cited one review article, reporting that patients with greater knowledge of their dialysis care reported improved outcomes, perceived higher quality of care, and better satisfaction with their care.
- National data from the 2023 Fall and 2023 Spring surveys show a performance gap, with decile ranges from 68.3% in decile 1 to 100% in decile 10 (mean 78.0%), indicating variation in measure performance across the target population. This measure has not been revised.
- The developer’s description of patient input sought for revision of the instrument overall supports the general conclusion that this measured patient-reported experience is meaningful with at least moderate certainty. Patients were not asked to provide input specifically on revision of the PIP measure, as psychometric testing results did not support reducing the items in the scale.
Limitations
- The logic model provided is a starting point for demonstrating how the measure's implementation will lead to the anticipated outcomes, and inputs and activities are limited to those required to report the measure. The logic model could be improved by specifying common clinical quality activities hemodialysis centers could use to improve on this measure focus.
- The submission does not directly address the scope of the population the measure covers, e.g., the estimated number of U.S. adults receiving in-center hemodialysis care, and could be strengthened by providing a few details about this population.
- The developer’s evidence review is focused primarily on their efforts to shorten the survey by identifying the content that is most meaningful to patients. The literature review provided that explores relationships between the topic addressed by this measure and material outcomes is minimal. The developer did not provide details for the cited review article, such as its scope and methodology or the types of studies included in the review. Two empirical studies are cited but not discussed. The submission could be strengthened by expanding the literature review and discussing the quality, quantity, and consistency of existing studies, or explicitly stating that this area is a gap in the available literature and explaining how they determined this.
Rationale
- This maintenance measure is rated as 'Not Met but Addressable' due to sparse review of evidence and a vague logic model. Enhancements, including a more complete discussion of available literature and more complete logic model, could elevate its importance.
Closing Care Gaps
The developer did not address this optional domain.
Feasibility Assessment
Strengths
- Required data elements are collected via patient survey. The developer tested web survey collection in the 2022 field test and cited hemodialysis centers’ lack of patients’ email addresses as barriers to online collection.
- The developer reported finding no feasibility issues resulting from the revised survey. The revisions to the survey are clearly described.
- As the survey is collected directly from patients, there is no burden on health care providers. The developer noted that the survey has been shortened, reducing burden for patients.
- The developer explained that hemodialysis centers are not involved in sampling patients and that data submitted to the ICH CAHPS data center are deidentified. Vendors may share aggregate data with clients, but rules prohibit sharing when responses are “too low.”
- There are no fees, licensing, or other requirements to use any aspect of the measure (e.g., value/code set, risk model, programming code, algorithm).
Limitations
- The developer did not provide typical or average costs associated with contracting with a survey vendor.
Rationale
- This maintenance measure meets all criteria for 'Met' due to its well-documented feasibility assessment, clear and implementable data collection strategy, and transparent handling of patient confidentiality, burden, licensing, and fees. These factors collectively ensure that the measure can be implemented effectively and sustainably in a real-world healthcare setting.
Scientific Acceptability
Strengths
- The developer performed the required reliability testing for this maintenance measure, namely, they conducted accountable entity-level (“measure score”) reliability testing at the level for which the measure is specified. They also provided evidence of person/encounter-level (“data element”) reliability testing for all critical data elements. Data sources used for reliability analysis are adequately described.
- For the accountable entity-level testing national implementation data collected during the 2023 Spring and Fall Surveys was used. The entities included in the analysis were characterized by accountable entities with at least 30 completes. There were 100,828 encounters across 2,319 qualifying entities. The data for the data element reliability analyses were from the field test/mode experiment data collected 11/22-1/23 and consisted of 4,605 encounters across 3,211 facilities.
- The developer conducted test-retest reliability testing at the accountable entity-level, however the decile table isn't quite correct. Decile sizes range from 107 to 441 entities. An overall IUR of 0.71 is reported and a rough estimation based on the performance gap table suggests that more than 70% of accountable entities are likely meeting the expected threshold of 0.6 at the levels for which the measure is specified, but only for entities with at least 30 completes (about 35% of the entities in the importance table).
- The developer also conducted internal consistency analysis of the 9 Providing Information to Patients (PIP) items.
- The internal consistency estimates for the PIP item set (α = 0.743) is above the expected threshold of 0.7.
Limitations
- The reliability decile table isn't quite correct, each decile should include approximately the same number of entities. Decile sizes range from 107 to 441 entities. The performance gap tables have not been limited to a minimum of 30 completes and include nearly three times as many entities than the reliability tables but with only about 75% more of the total number of persons.
Rationale
- The developer performed the required reliability testing for this maintenance measure and the results suggest sufficient reliability at data element and accountable entity levels.
Strengths
- The developer provides an Importance Table and Logic Model, providing a plausible causal association between the entity response to the measure and the measure focus. Empirical support for ruling out confounders includes adequate reliability (although the method IUR is not recommended), adequate risk-adjustment, and a correlation with a related experience measures with construct overlap (Provision of Information by Staff and Kidney Doctors, Dialysis Center Staff Training and Internal Procedures, Patient-Centered Communication, Promoting Patient Activation). Specifically, the argument is that the individual items are more highly correlated with PIP (because of the construct overlap) than with QDCCO (where there is less construct overlap). Similar, the results suggest that the construct overlap (the common causal factors) across the ICH measures is significant, perhaps 50% or more (r=0.4884 for staff rating; r=0.457 for center rating).
- Developer conducted statistical case-mix adjustment, based on a conceptual model, selecting risk factors that have a significant correlation with the outcome. The developer also explored social factors, including dual eligibility/low-income subsidy. The developer did not include these factors in the final models due to the minimal impact these social case-mix indicators have on the measure scores and to the model overall.
Limitations
- The main limitation is that the nature of the construct overlap among the related experience measures, and the nature of the responsible mechanisms that make up that construct, are not explicitly stated, apart from the logic model, or supported by empirical studies (apart from the reliability and risk adjustment). Residual risk for confounders not ruled out includes: Health literacy or cognitive status, Demographic and social characteristics of patients, and Patients’ prior healthcare experiences or expectations. Residual risk for a responsible mechanism includes the potential counter-acting mechanisms (none are explicitly stated).
- The developer does not provide model testing results that indicate the impact on provider scores at the high or low extremes of case-mix is due to the presence of case-mix factors.
Rationale
- The measure developer provides some support for the causal claim that the entity response to the measure is causally related to the measure focus. The developer provides empirical support for ruling out confounders (always with some residual risk of unstated or unexamined confounders) and for ruling in responsible mechanisms (always with some residual risk that the explicit mechanisms are only partially responsible for the measure focus).
- Opportunities to strengthen the inference of validity, that do not impact the domain rating, include: a more explicit consideration of potential confounders and ruling out such confounders as causal explanations of the variation among entities, groups, and over time. Conditions might also reflect a more explicit consideration of responsible mechanisms and ruling in such mechanisms as causal explanations of the variation among entities, groups, and over time.
- The case-mix adjustment methods are appropriate and demonstrate that variation in the prevalence of case-mix indicators across facilities contribute to unique variation in the outcome.
The impact of patient-level factors on provider scores at high or low extremes of case-mix is unclear in testing data.
Use and Usability
Strengths
- This measure is currently in use in the End Stage Renal Disease (ESRD) Quality Incentive Program.
- The developer referenced input from the TEP regarding recommended revisions to the survey and described positive feedback on the revised instrument offered by stakeholder groups such as the CAHPS Consortium and the Pre-Rulemaking Measure Review (PRMR) Hospital Committee.
- The developer reported no unexpected or adverse findings related to the currently implemented measure, and indicated they do not expect any for this unchanged measure once the new survey is implemented.
Limitations
- The developer argued that accountable entities can use the measure results to improve performance; however, no material actions that entities can take to improve performance were suggested.
- Feedback from measured entities on the currently implemented measure is not described.
- The developer reported only minimal change in the median performance score between 2015 (80.2%) and 2023 (79.7%), and no rationale for this lack of improvement was provided.
Rationale
- The rating for this maintenance measure is ‘Not Met.' The measure is actively used in at least one accountability application. However, the developer does not address actions that accountable entities can use to improve performance, and does not provide an explanation for the lack of improvement in this measure since 2015.
Public Comments
No public comments received.
No public comments received.