This measure assesses how well facilities provide clear, personalized discharge instructions to patients aged 18 years or older who had a surgery or procedure at an outpatient facility. It uses a 9-item survey to obtain patient’s feedback on 3 domains: applicability; medications; and daily activities. Facility scores are calculated by averaging the individual patient scores for each facility. Individual patient scores are calculated using a top-box approach, measuring the percentage of the total number items given the most favorable responses ("Yes" or "Very Clear") out of the total number of relevant items.
-
-
1.5 Measure Type1.6 Composite MeasureNo1.7 Electronic Clinical Quality Measure (eCQM)1.8 Level Of Analysis1.9 Care Setting1.10 Measure Rationale
Not applicable. For the URL link below, since this is a new measure, a webpage does not exist. We had to use a placeholder to bypass this question on the form.
1.11 Measure Webpage1.20 Testing Data Sources1.25 Data SourcesWe developed a nonproprietary, novel, 9-item instrument and piloted it in both English and Spanish. During pilot testing in hospital outpatient departments (HOPDs), survey administration, collection, and submission were facilitated via a third-party vendor. The survey was piloted using a web-based mode, in which patients provided the facility with their phone number and email, as well as permission to contact them via text and/or email. Patients were sent a link to the survey using the contact method(s) they opted into. All patients that failed to respond to the survey within 7 days were sent a single reminder. Responses were recorded on the platform and could be downloaded as a CSV file.
-
1.14 Numerator
The numerator is the sum of all individual patient scores a facility received from eligible respondents.
1.14a Numerator DetailsThe numerator is the sum of all individual scores a facility received from eligible respondents. An individual score is calculated for each respondent by taking the sum of items for which the respondent gave the most positive response (“Yes” or “Very Clear”) and dividing by the number of items the respondent deemed applicable to their procedure or surgery. Applicable items are calculated by subtracting the sum of items for which the respondent selected “Does not apply” from the total number of items (nine).
-
1.15 Denominator
The denominator is the total number of eligible respondents for the facility. Respondents are eligible if they are 18 years or older, had a procedure or surgery, were discharged alive with a stay less than two midnights.
1.15a Denominator DetailsThe denominator is the total number of eligible respondents for the facility. Respondents are eligible if they are 18 years or older, had a procedure or surgery, and were discharged alive with a stay less than two midnights.
-
1.15b Denominator Exclusions
None
1.15c Denominator Exclusions DetailsNone
-
OLD 1.12 MAT output not attachedAttached1.13 Attach Data Dictionary1.13a Data dictionary not attachedYes1.16 Type of Score1.16a Other Scoring Method
Continuous variable, e.g., average
1.17 Measure Score InterpretationBetter quality = Higher score1.18 Calculation of Measure Score(1) The target population is identified by having an age >=18, who had a procedure or surgery, and a length of stay less than 2 midnights. This is calculated by subtracting the date of discharge from the date of admission and dividing it by 24 hours. Patients discharged alive is defined as not expired at disposition. This population is sent the survey.
(2) Determine the eligible respondents by removing any incomplete surveys. This is the denominator.
(3) Calculate individual scores for patients in the denominator. The individual score is calculated for each respondent by summing the items to which they responded positively with "Yes" or "Very Clear" and then dividing this sum by the total number of items that respondents found applicable to their procedure or surgery. Applicable items are determined by subtracting the sum of items marked as "Does not apply" from the total number of items, which is 9 for this instrument. The sum of these scores is the numerator.
(4) The facility's measure score is the arithmetic mean of all individual scores calculated by dividing the numerator by the denominator.
1.19 Measure Stratification DetailsThere is no stratification for this measure.
1.21b Attach Data Collection Tool(s)1.21a Data Source URL(s) (if applicable)1.22 Are proxy responses allowed?Yes1.23 Survey Respondent1.24 Data Collection and Response RateThis measure has not yet been implemented but could be implemented through a third party or self-administered by a hospital. The measure was tested in two pilots.
During the pilots, survey administration, data collection, and submission were facilitated by a third-party vendor. A hospital could also choose to perform these activities itself. The survey is designed to be electronically distributed to batches of patients on a rolling basis using a web-based platform. Patients can choose to receive an SMS text and/or email with a survey invitation, followed by a reminder after 7 days. A minimum of 100 responses are needed for reporting. Response rates are calculated by dividing respondents by the total number of individuals who were sent a survey, regardless of whether the survey bounced back or failed to send due to missing contact information.
1.26 Minimum Sample SizeA minimum of 100 survey responses are necessary to calculate the measure.
-
Most Recent Endorsement ActivityManagement of Acute Events, Chronic Disease, Surgery, and Behavioral Health Fall 2023Last Updated
-
StewardCenters for Medicare & Medicaid ServicesSteward Organization POC EmailSteward Organization URLSteward Organization Copyright
Not applicable
Measure Developer Secondary Point Of ContactOscar Gonzalez
Acumen LLC
500 Airport Blvd., Suite 100
Burlingame, CA 94010
United StatesMeasure Developer Secondary Point Of Contact Email
-
-
-
2.1 Attach Logic Model2.2 Evidence of Measure Importance
The number of outpatient surgeries and procedures have been steadily rising since 20091-3. During the COVID-19 pandemic, the percentage of outpatient procedures such as lumpectomy, mastectomy, and cholecystectomy rose significantly.3 As the scale and complexity of outpatient surgical procedures increase, so does the concern that the patient sent home after undergoing general anesthetic may not have full understanding of the information they received.
A study comparing inpatient and outpatient surgery procedures found that inpatient providers were better at communicating discharge instructions to patients more frequently, including continuing medication names and instructions (96% vs. 40%); new medication names and instructions (99% vs. 29%); and pending diagnostic test names and instructions (90% vs. 61%).4 A lack of consistently written documentation in the outpatient setting is associated with worse patient understanding and lower patient activation, defined as a measure of an individual’s understanding, competence, and willingness to participate in care decisions, during their recovery.4-6 As a result, information that is simpler to read and more complete has been associated with fewer follow-up calls to providers as well as less frequent hospital readmissions.7-9
The strongest evidence that providers can improve patient understanding of discharge information comes from a systematic review of 58 studies and 5,721 participants discharged after an inpatient surgical procedure, which found patients had a greater understanding of self-care and better symptom experience after receiving education in which the content was individualized and given in a combination of media on an individual basis and in more than one session.10
There is also evidence that the outcome this PRO-PM focuses on is tied to clinical outcomes. Several studies show decreased readmissions in patients who received enhanced, clear discharge instructions. Receipt of discharge instructions that were easier to read following inpatient admission for surgery was associated with a lower proportion of patients calling after hospital discharge versus usual care that did not include discharge instructions with improved readability (9.0% v 21.9%; P<.0001).11-13
Citations:
1.) DelSole EM, Makanji HS, Kurd MF. Current trends in ambulatory spine surgery: a systematic review. J Spine Surg. 2019;5(Suppl 2):S124-S132. Doi:10.21037/jss.2019.04.12
2.) Kondamuri NS, Miller AL, Rathi VK, et al. Trends in Ambulatory Surgery Center Utilization for Otolaryngologic Procedures among Medicare Beneficiaries, 2010-2017. Otolaryngol Head Neck Surg. 2020;162(6):873-880. Doi:10.1177/0194599820914298
3.) Shariq OA, Bews KA, Etzioni DA, Kendrick ML, Habermann EB, Thiels CA. Performance of General Surgical Procedures in Outpatient Settings Before and After Onset of the COVID-19 Pandemic. JAMA Netw Open. 2023;6(3):e231198. Doi:10.1001/jamanetworkopen.2023.1198
4.) Downey E, Olds DM. Comparison of Documentation on Inpatient Discharge and Ambulatory End-of-Visit Summaries. J Healthc Qual. 2021;43(3):e43-e52.
5.) Hoek AE, Anker SCP, van Beeck EF, Burdorf A, Rood PPM, Haagsma JA. Patient Discharge Instructions in the Emergency Department and Their Effects on Comprehension and Recall of Discharge Instructions: A Systematic Review and Meta-analysis. Ann Emerg Med. 2020;75(3):435-444.
6.) Kang E, Gillespie BM, Tobiano G, Chaboyer W. Discharge education delivered to general surgical patients in their management of recovery post discharge: A systematic mixed studies review. Int J Nurs Stud. 2018;87:1-13.
7.) Choudhry AJ, Younis M, Ray-Zack MD, et al. Enhanced readability of discharge summaries decreases provider telephone calls and patient readmissions in the posthospital setting. Surgery. 2019;165(4):789-794.
8.) Mitchell JP. Association of provider communication and discharge instructions on lower readmissions. J Healthc Qual. 2015;37(1):33-40.
9.) VanSuch M, Naessens JM, Stroebel RJ, Huddleston JM, Williams AR. Effect of discharge instructions on readmission of hospitalised patients with heart failure: do all of the Joint Commission on Accreditation of Healthcare Organizations heart failure core measures reflect better care? Qual Saf Health Care. 2006;15(6):414-417.
10.) Fredericks S, Guruge S, Sidani S, Wan T. Postoperative patient education: a systematic review. Clin Nurs Res. 2010;19(2):144-164. doi:10.1177/1054773810365994
11.) Choudhry AJ, Younis M, Ray-Zack MD, et al. Enhanced readability of discharge summaries decreases provider telephone calls and patient readmissions in the posthospital setting. Surgery. 2019;165(4):789-794.
12.) Mitchell JP. Association of provider communication and discharge instructions on lower readmissions. J Healthc Qual. 2015;37(1):33-40.
13.) VanSuch M, Naessens JM, Stroebel RJ, Huddleston JM, Williams AR. Effect of discharge instructions on readmission of hospitalized patients with heart failure: do all of the Joint Commission on Accreditation of Healthcare Organizations heart failure core measures reflect better care? Qual Saf Health Care. 2006;15(6):414-417.
-
2.3 Anticipated Impact
If implemented, the expected effect of the Information Transfer PRO-PM is outpatient providers and facilities enhancing patient education around post-discharge self-care instructions. Patients prefer discharge education that provides relevant, concise, and personalized information.1 Patients with improved understanding have demonstrated having greater activation, making fewer calls to facilities, and having lower readmissions.
Citation
1.) Newnham H, Barker A, Ritchie E, Hitchcock K, Gibbs H, Holton S. Discharge communication practices and healthcare provider and patient preferences, satisfaction and comprehension: A systematic review. Int J Qual Health Care. 2017;29(6):752-768
2.5 Health Care Quality LandscapeThe Information Transfer PRO-PM is unique in that it quantifies patients’ perceived understanding in 3 key domains of post-operative self-care that have been identified in the literature and by patients and experts as fundamental. This measure is similar to, but distinct from, the OAS CAHPS 37 item survey that deals with a variety of patient related experiences, including a global assessment of patient-provider communication. Our survey provides feedback to providers on what elements of discharge instruction could be improved.
2.6 Meaningfulness to Target PopulationOur four-person Patient Working Group (PWG) was surveyed regarding the meaningfulness of the performance score. All four PWG members found the Information Transfer PRO-PM hospital mean score conveyed important information AND could help improve the clarity of self-care instructions for all patients undergoing an outpatient surgery or procedure.
We interviewed 13 patients in second pilot who took the survey. Patients interviewed stated they took the survey because they wanted to improve the hospital or share what they did well. Most patients felt the survey length was appropriate and brief enough for them to complete; they also felt that the pertinency of this type of healthcare survey was a large part of the reason for completing it and additionally agreeing to complete an interview on their experience. Furthermore, several interviewees had completed the OAS CAHPS survey and found this survey to be much more reasonable in length and the complexity of the questions. All interviewees found the survey language to be clear, direct, and easy to comprehend. Several patients had poor experiences with care and felt strongly about advocating to improve such experiences for patients in the future; some of the topics they mentioned included poor communication around wound care for specific procedures, medication changes, and generally a poor perception of care from their healthcare team. Other interviewees had profoundly great experiences in care, in the communication from their provider and nursing staff. Overall, patients with differing opinions and experiences found this survey to be of great importance to safe, quality care, especially as more complicated procedures they experienced are performed in the outpatient setting.
-
-
-
3.1 Feasibility Assessment
Feasibility Assessment
After the survey was completed, we interviewed QM staff responsible for gathering data. All facility staff interviewed felt that survey administration with a vendor was burdenless on front end staff and providers and required minimal effort from quality improvement staff.
During the survey administration period, we monitored key performance indices of each facility, such as the monthly volume of a site, and organizational indices, such as the survey batch response rate, the survey batch failure rate, missingness of patient data elements, and the lag time (the time between the surgery or and procedure and the survey response date). This monitoring identified some implementation challenges.
The challenges encountered during the second pilot survey administration can be effectively addressed with the following strategies:
Email Capture in Facilities:
Issue: Six facilities belonging to one organization had a 50% failure rate of the survey due to front line staff not capturing patient emails in their records.
Overcoming: Implement a systematic approach to capture patient email addresses during the registration process. This can include training staff to ask for email addresses, introducing digital forms for patient intake, or partnering with IT to integrate email capture into the electronic health records system.
Facility Eligibility Criteria:
Issue: Some facilities with the participating organizations were eliminated due to low monthly case volumes and high failure rates resulting in an inability to reach the minimum number of responses.
Overcoming: Expanding the data capture period beyond the period used in the pilot could increase the number of facilities that can achieve the minimum survey response necessary for the measure. Survey administrators can monitor the batch failure rate and monthly case volumes to determine if a site needs to adjust their front-end procedures.
OAS CAHPS Participation and Special Permissions:
Issue: Nine facilities faced delays in survey administration to patients due to their participation in OAS CAHPS, which requires special permission to administer a survey with overlapping survey items to same or similar population. Patients received the survey well outside the optimal time window of 2-7 days.
Overcoming: We harmonized the measure to eliminate overlapping items, reduced the total number of items to further reduce patient burden, allowing the 9-item survey to be administered in conjunction with OAS CAHPS.By addressing these issues encountered in the second pilot survey administration future survey administration is expected to be more efficient and effective.
Data element feasibility
Patient demographics, such as the first name, last name, race, birth sex, date of birth, ethnicity, preferred language, email address, and phone number, and procedure were all required data elements in the USCD v1, encounter type, encounter location, and time were required data elements in USCD v2. The current ONC certification required EHR or HIT to support the capture and exchange of all elements used for administering the survey. With the exception of race for procedures which had a high degree of missingness, this information was universally available in all participating organizations. The measure calculation is based solely on the information provided by patients responding to the instrument.
3.3 Feasibility Informed Final MeasureIn order to reduce the overlap between our instrument and OAS CAHPS and to reduce patient burden, the original 21-item survey used in the first pilot was reduced to eliminate any overlapping questions. It was then further shortened using an empirical approach to produce the shortest survey with the highest internal reliability/consistency. Lastly, we dropped all questions in the about-you section of the survey as the measure is not risk adjusted. This resulted in a streamlined 9-item survey instrument.
-
3.4a Fees, Licensing, or Other Requirements
Not applicable, not a proprietary measure and no proprietary components.
3.4 Proprietary InformationNot a proprietary measure and no proprietary components
-
-
-
4.1.3 Characteristics of Measured Entities
We required facilities included in measure testing to have a minimum monthly case volume of 250 outpatient procedures to facilitate reaching the required response rate within the testing period. The survey was administered and tested in 26 facilities in the Western or Northeastern regions of the United States. Of these, 19 facilities were collocated within a HCUP-defined medium-to-large urban teaching medical center. The remaining 7 facilities that were not located on the premises of a hospital, all shared a single CCN with one of the 19 facilities. All 8 were grouped together as a single facility for analysis, yielding a final facility count of 19. See Table 1 of tables/figures attachment.
4.1.1 Data Used for TestingThe survey was administered between 08-01-2022 - 03-01-2023 at 26 facilities. Seven facilities and one hospital shared a single CCN and were grouped together as a single facility for analysis, resulting in a final count of 19 measured entities or HOPDs.
Data from these HOPDs are described in the patient demographic table and were used to assess the reliability of the instrument; however, the reliability of the performance score is reported for HOPDs with a minimum of 100 respondents.
4.1.4 Characteristics of Units of the Eligible PopulationAll patient-level analysis was conducted on the full sample of respondents from all 19 HOPDs. However, all accountable-entity level analysis for performance score calculations are reported for facilities with the minimum number of survey responses (100). (See the demographic Table 2 for patient level analysis).
4.1.2 Differences in DataThe data sources are the same, the count or sample varies as above.
-
4.2.1 Level(s) of Reliability Testing Conducted4.2.2 Method(s) of Reliability Testing
Data elements of PRO Reliability
We measured the instrument or data element reliability using the Cronbach Alpha score, which assesses the internal consistency of the 9-item scale, or the extent to which the 9 items within the scale reliably measure the same underlying construct.
Accountable Entity Level
We used signal-to-noise ratio to assess the reliability of the performance score for 15 facilities that met the completed 100-survey threshold. We used a mixed-effect intercept only model to estimate the variance among facilities and facility specific errors, based on which we then calculated the reliability scores for all facilities.
4.2.3 Reliability Testing ResultsData Element-PRO Reliability
We examined the internal consistency of the entire survey instrument (9 items), the Cronbach alpha tests the internal consistency or agreement of patient responses of items in the survey. An alpha score of 0.80 is considered very good. The overall alpha score of the 9-item survey was 0.8941, which indicates that all measure the same concept, global quality (clarity and applicability) of discharge instructions.
Accountable Entity Level Signal to Noise Ratio
Here, and in Table 3 in the attachment, we provide the distribution of signal-to-noise reliability among facilities with at least 100 completed surveys. Reliability scores range from a minimum of 0.572 to a maximum of 0.817, with a mean of 0.689. The median signal to noise reliability was 0.697 and the interquartile range (IQR, or 25th percentile to the 75th percentile) was 0.603-0.767. We were unable to split the reliability results into deciles due to the small number of hospitals in our test dataset (15 hospitals with at least 100 completed surveys). Please see Table 3 on tables/figures attachment.
4.2.4 Interpretation of Reliability ResultsData Element-PRO Reliability Interpretation
An alpha score of 0.80 is considered very good. The overall alpha score of the 9-item survey was 0.8941, which indicates that all measure the same concept, global quality (clarity and applicability) of discharge instructions. Our instrument is unidimensional.
Accountable Entity Level Signal to Noise Ration Interpretation
Mean signal to noise reliability of 0.689 indicates moderate reliability with 68.9% of the variance of the mean score due to between hospital difference. A value less than 0.5 would be considered poor reliability, a value of 0.5 to 0.75 would be considered moderate reliability.
-
4.3.1 Level(s) of Validity Testing Conducted4.3.2 Type of accountable entity-level validity testing conducted4.3.3 Method(s) of Validity Testing
Empirical validity testing of the measure score:
To validate the performance score, our team compared hospital performance on the 9-item instrument to their performance on the OAS CAHPS. OAS CAHPS is a previously validated survey of outpatient surgery patients' experience with care they received following a surgery or procedure. The OAS CAHPS has 37 items and includes several questions in a domain titled “communication about your procedure.” We evaluated the criterion validity using a Pearson’s correlation coefficient to assess the strength and direction of the linear relationship between the Information Transfer PRO-PM hospital mean score to the OAS CAHPS linear mean score for the domain “communication about your procedure” for 9 of the 15 HOPDs. Empiric validity testing of the performance score was tested for these 9 facilities as they were the only facilities with publicly available OAS CAHPS linear mean scores for “communication about your procedure” (using data from patients surveyed between January 2021 – December 2021).
Systematic assessment of face validity:
Face validity was captured by administering specific questions to the TEP (n=10), requesting their vote on the measure’s ability to distinguish between good and poor quality of care at measured facilities.
We polled the TEP on two separate questions:
1) “The unadjusted Information Transfer PRO-PM, as specified, will provide valid assessment of the transfer of key information to patients at discharge from the facility,” and
2) “The unadjusted Information Transfer PRO-PM, as specified, can be used to distinguish between better and worse quality care at measured facilities”
The possible answer categories were: Strongly disagree, moderately disagree, somewhat disagree, somewhat agree, moderately agree, strongly agree.
Analysis of missing data
We identified the extent and distribution of missing items among respondents. We evaluated the pattern of missingness to assess if items were missing completely at random. Finally, we conducted a complete case analysis.
Assessment of Non-Response bias
We carefully examined if there were systematic differences in the patient-characteristics of the respondents versus the non-respondents, by comparing their age and sex, however CPT codes, race/ethnicity/language/insurance data was not consistently available and/or standardized across entities to make accurate comparisons
4.3.4 Validity Testing ResultsEmpiric Validity assessment
Our hypothesis was that the Info Transfer PRO-PM rates hospital's ability clearly communicate key information about recovery is similarly related to the OAS CAHPS domain about global communication around a patient's outpatient surgical procedure. A Pearsons correlation of 0.64 with p-value 0.06, indicates that while the Information Transfer PRO-PM's 9-item survey instrument mean score appeared moderately positively correlated with the previously validated OAS CAHPS provider communication domain, the results were not statistically significant. We failed to reject the null hypothesis. Only 9 out of the 15 facilities had publicly available CAHPS data.
Systematic assessment of face validity
In responses to the first TEP question (“The unadjusted Information Transfer PRO-PM, as specified, will provide valid assessment of the transfer of key information to patients at discharge from the facility”), six of 10 TEP members (60 percent) agreed that the measure was a valid assessment of the transfer of key information to patients at discharge from the facility.
Specifically, TEP responses to the first question were:
Strongly agree: 2
Moderately agree: 4
Somewhat agree: 2
Somewhat disagree: 1
Moderately disagree: 1
Strongly disagree: 0
In response to the second TEP question, eight of 10 TEP members (80 percent) agreed that the measure could be used to distinguish between better and worse quality of care at facilities.
Specifically, TEP responses to the second question were:
Strongly agree: 1
Moderately agree: 5
Somewhat agree: 2
Somewhat disagree: 1
Moderately disagree: 1
Strongly disagree: 0
Analysis of missing data
Item-missingness was evaluated across all respondents, graphing the missingness as bar plots for each item. Items in the same domain had similar percents missingness, and the percentage of missingness increased with each subsequent domain. Approximately 12.2% of all respondents skipped questions about applicability, 12.9% of respondents skipped questions about medications, and 13.7% skipped questions about activity. The data pattern suggests patients skipped entire domains, rather than skipping specific items. The minimum number of items skipped was 4, corresponding to questions in the last domain activity.
Non-response
The team evaluated non-response by assessing differences in patient characteristics between respondents and non-respondents and whether such differences were associated with the performance scores. While there were statistically significant differences between respondents and non-respondents by age, and sex, however these patient risk factors were not associated with performance score (see Table 4 on table/figures attachment). The age and sex are not associated with outcome, with p-value 0.734 and 0.128, respectively. Due to issues with data quality, it was not possible to examine differences in race, ethnicity, or insurance status (see Table 5 on table/figures attachment).
4.3.5 Interpretation of Validity ResultsWith this submission we provide two independent sources of evidence in support of the validity of the Information Transfer PRO-PM. First, our TEP face validity assessment supports that the measure can be used to distinguish between better- and worse-performing facilities. Second, our empiric validity testing shows, as we hypothesized, a strong association in the correct direction (positive correlation) between the Information Transfer PRO-PM measure score and the OAS CAHPS domain about global communication around a patient's outpatient surgical procedure. Taken together, these results support the validity of the Information Transfer PRO-PM.
-
4.4.1 Methods used to address risk factors4.4.1b If an outcome or resource use measure is not risk adjusted or stratified
Following extensive discussions with clinicians and patient stakeholders, the measure developers and CMS reached a consensus that Hospital Outpatient Departments (HOPD) should tailor discharge instructions to accommodate patients' unique characteristics, such as age, education, health status, and prior experience with surgeries.
It is essential for clinicians to connect with patients on their level and provide clear, personalized guidance for post-discharge self-care. This is a measure of patient centered communication. Risk adjusting for patient-level social factors that can be influenced by the HOPD discharge process could unintentionally encourage different/ disparate outcomes between social disadvantaged patient care. Therefore, the unadjusted measure effectively captures the average patient's assessment of the clarity of discharge instructions provided by HOPD.
Furthermore, we analyzed data collected from the second pilot to evaluate the potential impact of adjusting the performance measure for patient factors identified in our exploration of the literature. These candidate variables, including self-reported race, ethnicity, age, education, self-reported health status, and history of procedures, were identified in an environmental scan and literature review as having an association with perceived and assessed understanding of medical and surgical patients discharge instructions. However, we found that there were no statistically significant differences between the scores obtained from the adjusted and unadjusted measures. Figure 2 on tables/figures attachment illustrates this relationship, showing a strong correlation with a coefficient of 0.992.
Our analysis of the pilot data revealed that patients with higher education levels tended to be less satisfied with the discharge information they received. However, there is no evidence to suggest that hospitals would limit access to socially advantaged group (those with higher education) as a consequence of publicly reporting an unadjusted PRO-PM. Ultimately, the performance differences between the adjusted and unadjusted measures were minimal, and both exhibited moderate reliability.
Risk adjustment approachOffRisk adjustment approachOffConceptual model for risk adjustmentOffConceptual model for risk adjustmentOff
-
-
-
5.1 Contributions Towards Advancing Health Equity
We did not select that we would provide information on Equity in our intent to submit form.
-
-
-
6.1.2 Current or Planned Use(s)6.1.4 Program Details
-
6.2.1 Actions of Measured Entities to Improve Performance
This measure is currently not in use for performance improvement; however, the results may inform providers and facilities about the quality of discharge/post-procedure information provided to patients receiving an outpatient procedure or surgery. Specifically, the survey allows for tracking and improvement on discharge instructions related to information specific to the patient, medication, and daily activities, which is a gap in measured quality. We interviewed quality officers of the organizations that participated in the pilot about the meaningfulness of the performance score. They indicated that the scores aligned with known issues raised by patients in an open-ended survey and accurately quantify known issues with the discharge process.
Each organization's quality officer was interested in their overall score, its correlation with OAS CAHPS, and in learning how to improve their process of communicating discharge instructions based on feedback of the survey. The body of literature referenced in the evidence section demonstrates that providers can effectively improve patient’s understanding of instructions for self-care by giving clear, personalized instruction via more than one medium, such as conversation, written instruction, or video. Discharge instructions are better retained and understood by patients when they receive the same instructions on different occasions. Providers would need to ask patients about their home environment and ask patients about their preferences concerning pain medication, physical therapy, and other medical problems, before advising them accordingly.
-
-
-
Measure Comments
OrganizationAmerican Occupational Therapy AssociationPublic Comments from Pre-Rulemaking Measure Review (PRMR)
CBE #4210 -Patient Understanding of Key Information Related to Recovery After a Facility-Based Outpatient Procedure is also a measure under consideration for potential inclusion in the Hospital Outpatient Quality Reporting Program as MUC2023-172 and is currently undergoing review by the Pre-Rulemaking Measure Review (PRMR) committees. Prior to its review, the measure was posted for PRMR public comment, and received one comment, which can be found here: https://p4qm.org/sites/default/files/2024-01/Compiled-MUC-List-Public-Comment-Posting.xlsx. Please review and consider this PRMR comment for MUC2023-172 in addition to any submitted within the public comment section of this measure’s webpage. If there are no comments listed in the public comment section of this webpage, then none were submitted.
-
CBE #4210 Staff Assessment
Importance
ImportanceStrengths:
- The developer provides a logic model depicting that an increase in patient understanding of post-operative care instructions will improve care coordination, patient self-care ability, and subsequent patient outcomes, such as reduced unplanned emergency department (ED) visits, reduced medication errors, and reduced risk of injury from engaging in daily activities too early. For this patient-reported outcome performance measure (PRO-PM), the developer provides supportive evidence indicating a lack of consistency in communicating discharge instructions to patients in the outpatient setting, which leads to worse patient understanding and willingness to participate in care decisions during their recovery.
- The developer also cites evidence to support the logic model and business case for the measure, in which studies have shown the measured outcome can decrease hospital readmissions and the need to call the hospital after discharge.
- The developer notes a similar measure, OAS CAHPS, which is a 37-item survey that garners feedback on a variety of patient experiences with care, including provider communication. However, the developer posits that this measure is unique, as it provides facilities with feedback on what elements of discharge instruction and communication could be improved. The developer also provides evidence indicating a gap in providing discharge communication instructions to patients in the outpatient setting, compared with the inpatient setting, including medication and diagnostic information and instructions.
- The developer states that its Patient Working Group of four members found the measure “conveyed important information and could help improve the clarity of self-care instructions for all patients undergoing an outpatient surgery or procedure.” In addition, the developer interviewed 13 patients who were administered the survey. Overall, the 13 patients found that survey to be of great importance for safe, quality care.
Limitations:
- None
Rationale:
- The developer provides a logic model depicting that an increase in patient understanding of post-operative care instructions will improve care coordination, patient self-care ability, and subsequent patient outcomes, such as reduced unplanned emergency department (ED) visits, reduced medication errors, and reduced risk of injury from engaging in daily activities too early. The developer cites studies showing the measured outcome can decrease hospital readmissions and the need to call the hospital after discharge. The developer posits that this measure is unique and that the target population within the measure find the survey to be of great importance for safe, quality care.
Feasibility Acceptance
Feasibility AcceptanceStrengths:
- The developer interviewed staff responsible for administering the survey, which found the survey administration to be burdenless on providers. During its second pilot testing, the developer reported 50% survey failure rates amongst six facilities under one organization due to the front-line staff not capturing patient emails. To mitigate this, the developer proposes implementing a systematic approach to capturing patient emails. There were also some facility participation challenges due to low case volumes, however, the developer proposes expanding the data capture period beyond what was used for pilot testing. Lastly, the developer reports nine facilities administered the survey well beyond the optimal time window of 2-7 days. The delays were due to the facilities’ participation in OAS CAHPS, which requires special permission to administer a survey with overlapping survey items to same or similar population.
- To overcome this, the developer states it eliminated overlapping items, reduced the total number of items to further reduce patient burden, allowing the 9-item survey to be administered in conjunction with OAS CAHPS. The developer states that the measure calculation is based solely on the information provided by patients responding to the instrument. Patient demographics were required data elements in the USCD v1, and encounter type, encounter location, and time were required data elements in USCD v2. The current ONC certification required EHR or HIT to support the capture and exchange of all elements used for administering the survey. With the exception of race for procedures, which had a high degree of missingness, the developer states this information was available in all participating organizations. Lastly, the developers states this measure is not proprietary, nor has any proprietary components.
- After the measure was submitted to Battelle, the developer clarified that USCD refers to the United States Core Data for Interoperability (USCDI), which is an ONC product that establishes a set of standard data classes and elements that improve the exchange of interoperable health information from certified health information technology systems, including EHR. Version 1 and 2 of USCDI contain all the data elements necessary for administering the survey to patients.
Limitations:
- None
Rationale:
- This measure is not proprietary nor has any proprietary components. The developer interviewed staff responsible for administering the survey, which found the survey administration to be burdenless on providers. The developer encountered administration challenges amongst several test sites due to lack of patient email capture, lack of adequate case volume, and OAS CAHPS permissions. The developer proposed mitigation strategies for each. Lastly, the developer states that the measure calculation is based solely on the information provided by patients responding to the instrument. All data elements are required under the USCDI v1 and v2 and current ONC certification required EHR or HIT to support the capture and exchange of all elements used for administering the survey. However, it is unclear what USCDI v1 and v2 are.
Scientific Acceptability
Scientific Acceptability ReliabilityStrengths:
- The measure is well-defined and precisely specified.
- For data element reliability, the internal consistency alpha score was 0.8941, which is above the threshold of 0.7.
- For accountable entity level reliability, almost all of the facilities had a signal-to-noise reliability above the threshold of 0.6 with an IQR of 0.603-0.767 and median of 0.697.
- A minimum of 100 survey responses are necessary to calculate the measure.
- The survey was administered between 08-01-2022 and 03-01-2023.
Limitations:
- Criteria for patients to be included in denominator are inconsistent. The Denominator and Denominator Details section do not make mention of removing incomplete surveys but the Calculation of Measure Score section says "Determine the eligible respondents by removing any incomplete surveys. This is the denominator."
- The definition of midnights, in relation to the denominator definition, is confusing. It says in Calculation of Measure Score section says the number of midnights is calculated by subtracting the date of discharge from the date of admission and dividing it by 24 hours. How do you subtract dates? Do they mean taking the difference in hours between admission and discharge? Also the hour of admission is important because if you are admitted just before midnight then you would have two midnights if you stayed for a little over 24 hours.
- The sample size is somewhat small with only 15 facilities meeting the completed 100-survey threshold. Because of the small sample size, the developer was not able to provide reliability by decile.
- At least one facility (minimum of 0.572) has a signal-to-noise reliability below the threshold of 0.6.
Rationale:
Reliability testing of data elements and accountable entity-level reliability were performed. The vast majority of facilities have a reliability which exceeds the accepted threshold of 0.6 with only one facility below the threshold (minimum of 0.572). Sample size for each year and accountable entity level analyzed is sufficient. The developer may consider expanding its testing in the future, and consider mitigation approaches to address small case volumes. Some possible mitigation strategies to improve these estimates could be:
- Empirical approaches outlined in the report, MAP 2019 Recommendations from the Rural Health Technical Expert Panel Final Report, https://www.qualityforum.org/WorkArea/linkit.aspx?LinkIdentifier=id&ItemID=89673
- Extend the time frame.
- Focus on applying mitigation at the lower volume providers.
Scientific Acceptability ValidityStrengths:
- The developer conducted both empirical validity testing and face validity testing of the measure score (i.e., accountable entity-level).
- For the empiric testing, the developer evaluated criterion validity using a Pearson’s correlation coefficient to assess the strength and direction of the linear relationship between the hospital mean score of the proposed measure to the OAS CAHPS linear mean score for the domain “communication about your procedure” for nine of the 15 facilities. The developer adds that the empiric validity testing was performed for these nine facilities “as they were the only facilities with publicly available OAS CAHPS linear mean scores for ‘communication about your procedure’ (using data from patients surveyed between January 2021 – December 2021).”
- The developer hypothesized that the proposed measure rates are similarly related to the OAS CAHPS domain about global communication around a patient's outpatient surgical procedure. The developer found a Pearsons correlation of 0.64, indicating moderate positive correlation; however, the results were not significant (p-value 0.06).
- For face validity, the developer convened a 10-person technical expert panel, of which, six of 10 TEP members (60%) agreed that the measure was a valid assessment of the transfer of key information to patients at discharge from the facility, and eight of 10 TEP members (80%) agreed that the measure could be used to distinguish between better and worse quality of care at facilities.
- The developer reports data missingness, ranging from 12.2% - 13.7%, for questions about applicability, medications, and activity. The developer states that this “data pattern suggests patients skipped entire domains, rather than skipping specific items.” However, the developer does not discuss any mitigation approaches to improve this missingness.
- This measure is also not risk-adjusted nor stratified. The developer examined the potential impact of risk adjustment and found the “performance differences between the adjusted and unadjusted measures were minimal, and both exhibited moderate reliability.” The developer states that facilities should tailor discharge instructions to accommodate patients' unique characteristics, such as age, education, health status, and prior experience with surgeries.
Limitations:
- The developer does not provide data element testing of the survey instrument.
- For score-level testing, the developer did not find statistical significance for the criterion validity testing and did not provide a rationale as to why. For face validity, the developer did not disclose who was on the TEP and why members of the TEP disagreed with the questions asked.
Rationale:
- The developer does not provide data element testing of the survey instrument. The committee should consider whether the survey instrument was developed using a credible approach.
- The developer conducted both empiric and face validity testing of the measure score (i.e., accountable entity-level). For face validity, the developer convened a 10-person technical expert panel (TEP), of which six of 10 TEP members (60%) agreed that the measure was a valid assessment of the transfer of key information to patients at discharge from the facility, and eight of 10 TEP members (80%) agreed that the measure could be used to distinguish between better and worse quality of care at facilities. However, the developer does not disclose who was on the TEP, nor the rationale from the TEP members that did not agree.
- For the empiric testing, the developer did find moderately positive correlation to the OAS CAHPS domain about global communication around a patient's outpatient surgical procedure, which it hypothesized. However, this correlation was not statistically significant (p=0.06), and the developer did not discuss why. Perhaps the developer can speak to the face and empiric validity testing during the endorsement meeting.
- Lastly, this measure is not risk-adjusted nor stratified due to minimal impact on the performance scores. The developer states that that facilities should tailor discharge instructions to accommodate patients' unique characteristics, such as age, education, health status, and prior experience with surgeries.
Equity
EquityStrengths:
- N/A
Limitations:
- Developer did not address this optional criterion.
Rationale:
- Developer did not address this optional criterion.
Use and Usability
Use and UsabilityStrengths:
- Developer indicates that the measure is planned for use in public reporting and internal/external QI.
- After the measure was submitted to Battelle, the developer added more information in their comments: The measure is intended for initial rollout in CMS's HOQR program.
- Developer asserts that findings from the PRO-PM survey instrument provide the information entities need to improve quality and that the PRO-PM aligns with known issues with the discharge process, and cite literature finding that patient's understanding of instructions for self-care can be enhanced by giving personalized instruction in more than one medium and on more than one occasion.
Limitations:
- None
Rationale:
- The measure is planned for use in public reporting and internal and external QI initiatives, including initial rollout in CMS's HOQR program.
- Developer claims that the survey instrument itself provides entities with information on which areas need improvement, and cites literature finding that patients' understanding of instructions for self-care can be enhanced by giving personalized instruction in more than one medium and on more than one occasion.
Summary
N/A
-
could be an important measure
Importance
Importanceagree with staff assessment.
Feasibility Acceptance
Feasibility Acceptanceagree with staff assessment. Reducing the number of questions should increase response rate.
Scientific Acceptability
Scientific Acceptability Reliabilityagree with the staff assessment.
Scientific Acceptability Validityagree with staff assessment.
Equity
Equityhowever, the developer indicated that this would not be addressed in the intent to submit form.
Use and Usability
Use and Usabilityagree with staff assessment.
Summary
I believe this could be an important measure to determine quality of care and allow HOPDs to evaluate opportunities for improvement with patient discharge instructions.
Measure Summary
Importance
ImportanceAgree with staff assessment. I appreciate the connection between readmissions, ED visits, and adverse events following patient discharge. It looks at a more broad picture of patient care.
Feasibility Acceptance
Feasibility AcceptanceAgree with staff assessment. Mitigation strategies included are key for organizations with low volume.
Scientific Acceptability
Scientific Acceptability ReliabilityAgree with staff assessment. Addressing and allowing for mitigation strategies for low volume is essential for rural providers.
Scientific Acceptability ValidityAgree with staff assessment.
Equity
EquityDeveloper did not address this.
Use and Usability
Use and UsabilityAgree with staff assessment.
Summary
Overall, this measure supports improved quality outcomes and provides a holistic approach for patients undergoing outpatient surgery. Discharge education and instructions are crucial components of a patients success following surgery.
important PROM of patient understanding
Importance
ImportanceAgree with staff assessment. Logic model shows evidence for low patient understanding of post-operative care instructions in outpatient settings and downstream negative outcomes. Unique and distinct from existing OAS CAHPS measure and supported by patients.
Feasibility Acceptance
Feasibility AcceptanceAgree with staff assessment. Interviews with facility staff support the administration of the measure, reduced items lower patient burden, and multiple implementation approaches.
Scientific Acceptability
Scientific Acceptability ReliabilityAgree with staff assessment. The sample size is somewhat small with only 15 facilities meeting the completed 100-survey threshold. Because of the small sample size, the developer was not able to provide reliability by decile. At least one facility (minimum of 0.572) has a signal-to-noise reliability below the threshold of 0.6. Unclear if there is a min # of items needed to calculate the score and impact of missingness given small # of items. Need clarification on denominator.
Scientific Acceptability ValidityAgree with staff assessment. Concerns with developers stating that the minimum number of items skipped was 4, when it is only a 9 item measure. Unclear why some members of the TEP disagreed with face validity questions. Mean score appeared moderately positively correlated with the previously validated OAS CAHPS provider communication domain, the results were not statistically significant
Equity
EquityWas not a focus but noting test sample was predominately white and English-speaking. Further validation works needs to be completed to ensure constructs are valid in more diverse populations.
Use and Usability
Use and UsabilityAgree with staff assessment. Survey is for use in public reporting and allows for tracking and improvement on discharge instructions post-operative care in outpatient settings. Quality officers commented that the measure aligned with known issues raised by patients in an open-ended survey and accurately quantify known issues with the discharge process.
Summary
Will help outpatient settings identify and mitigate an important patient literacy issue
Useful measure
Importance
ImportanceAgree with the provided assessment. Reducing re-adminssions and ED visits should be a focus as we see increased wait times, boarding numbers, and longer hospital lengths of stay from complications which may be prevented. Patient education is an important piece of autonomy and improving self-care.
Feasibility Acceptance
Feasibility AcceptanceAgree with the provided assessment. The shortened survey should lend to better patient response rates. There appears to be little to no added work to the institution to administer the survey. There will be an added component of education and counseling that will need to take place and accounted for.
Scientific Acceptability
Scientific Acceptability ReliabilityAgree with the assessment, there is a gap in institutions who will be unable to reach the sample size. Incomplete survey rate should be compared to current practice, and uniformly excluded or included when calculating the total sample (denominator).
Scientific Acceptability ValidityAgree with the assessment, however I was confused regarding the panel and selection criteria of the panelists. Further explanation/rationale for the panelists vote would be useful information.
Equity
EquityThis was not addressed.
Use and Usability
Use and UsabilityThere is a clear objective for which this metric will be used, and has benefit for both the institution as an improvement tool and for the patient as a decision support tool when comparing available options for treatment.
Summary
This appears to be a useful measure that will enhance patient education and provide specific recommendations for post-surgical self-care. Potential impact on reducing hospital admissions and ED visits make this relevant to current practice.
An issue that needs to be solved to improve patient outcomes
Importance
ImportanceI agree with the staff assessment. I appreciate the attention given to ED visits, hospital readmissions, medication use issues and other adverse patient outcome.
Feasibility Acceptance
Feasibility AcceptanceI agree with staff assessment. Reduction of number of survey questions increased responses to survey. It also doesn’t add a burden to staff.
Scientific Acceptability
Scientific Acceptability ReliabilityI agree with the staff assessment. There needs to find a way to address low volume sites because rural areas need to have a way to assess discharge for outpatient procedures.
Scientific Acceptability ValidityI agree with the staff assessment. On order to pass this measure the committee must be 75% in agreement but the TEP was about 60% which doesn’t seem to be a good score to move a measure forward. Also, the make up the TEP wasn’t shared with us. Did it have patients on it?
Equity
EquityNot required or addressed. In the future this might be important area to assess as you survey your patients.
Use and Usability
Use and UsabilityI agree with staff assessment. This measure will benefit the patient and the healthcare organization by improving patient outcomes. Patients will have a better understanding of discharge and in the end it will also reduce provider burnout because they will have less patient follow up phone calls.
Summary
NA
see below
Importance
ImportanceAgree with staff. Underlying motivation that inpatient ranks better than outpatient is potentially confounded (ie different types of procedures inpatient vs outpatient).
Feasibility Acceptance
Feasibility AcceptanceAgree with staff
Scientific Acceptability
Scientific Acceptability ReliabilityAgree with staff
Scientific Acceptability ValidityAgree with staff
Equity
EquityAgree with staff
Use and Usability
Use and UsabilityAgree with staff
Summary
Agree with staff, see my comment re motivation/rationale for measure
N/A
Importance
ImportanceAgree with staff assessment. Increase in patient understanding of post-operative care instructions will improve care coordination, patient self-care ability, and can decrease hospital readmissions and the need to call the hospital after discharge.
Feasibility Acceptance
Feasibility AcceptanceAgree with staff assessment.
Scientific Acceptability
Scientific Acceptability ReliabilityAgree with staff assessment.
Scientific Acceptability ValidityAgree with staff assessment.
Equity
EquityDeveloper did not address this optional criterion.
Use and Usability
Use and UsabilityAgree with staff assessment.
Summary
N/A
Summary
Importance
ImportanceOutpatient surgeries have been steadily rising, so this measure is timely. Providing clear discharge instructions is tied to improved patient outcomes.
Feasibility Acceptance
Feasibility AcceptanceThe measure is not proprietary. Although there are some implementation challenges, the developer has addressed them.
Scientific Acceptability
Scientific Acceptability ReliabilityThe Denominator and Denominator Details section do not make mention of removing incomplete surveys but the Calculation of Measure Score section says "Determine the eligible respondents by removing any incomplete surveys. This is the denominator." The definition of midnights, in relation to the denominator definition, is confusing. The sample size is small.
Scientific Acceptability ValidityThe developer does not provide data element testing of the survey instrument. For score-level testing, the developer did not find statistical significance for the criterion validity testing and did not provide a rationale as to why. There is no risk adjustment.
Equity
EquityOptional criterion not addressed.
Use and Usability
Use and UsabilityThe measure is planned for use in public reporting and internal and external QI initiatives, including initial rollout in CMS's HOQR program.
Summary
Although there are some issues that need to be resolved, this is a promising and much needed measure.
Patient Understanding of Key Information Related to Recovery Aft
Importance
ImportanceThere is a clear description for the rationale behind needed better information at the point of discharge, and how this assessment may be used at assess information provision/receipt/understanding.
Feasibility Acceptance
Feasibility AcceptanceThe pilot studies found likely intuitive barriers to implementation (timing, approval) but these are anticipated to be overcome.
Scientific Acceptability
Scientific Acceptability ReliabilityThere needs to be further documentation of validity in regards to the larger measures, and less 'missing' data/better scale and scope across populations and locations.
Scientific Acceptability ValidityThere needs to be further documentation of validity in regards to the larger measures, and less 'missing' data/better scale and scope across populations and locations.
Equity
EquityThere are no issues with equity in measure utility or implementation.
Use and Usability
Use and UsabilityThere is a clear case made, and supported by evidence and reviewer comments (patients and board members) that this will be useful.
Summary
A clear new instrument with suitable validity, feasibility, and acceptability.
Summary
Importance
ImportanceProviding additional innovative ways to measure how well patients are ready for discharge is important and aligned with the NQS.
Feasibility Acceptance
Feasibility AcceptanceWhile the developer did address the potential burden on the providers, I feel the burden on the facilities while appears minimal it's not insignificant. Thoughtfully establishing strong processes that add to the duties of nursing and other personnel need to be measured to truly estimate the burden on the facility administering the survey to patients.
Scientific Acceptability
Scientific Acceptability ReliabilityAgree with staff observations related to the definitions of the denominator and the calculation of '2 midnights'.
Scientific Acceptability ValidityConcerns about missing data from a relatively small test. Concerns regarding the lack of risk adjustment. There are PRO measures in existance that provide method for risk adjustment that could be considered.
Equity
Equitynot provided. If the measure was risk stratified, perhaps the developed can address ways to use the measure in this space.
Use and Usability
Use and UsabilityAgree wtih staff comments
Summary
Agree that adding to the measures that approach quality and safety from the patients own experience is an important area for development.
The developers have…
Importance
ImportanceThe developer has demonstrated the importance of the measure on patient outcomes and that there are actions facilities can take that will improve measure scores.
Feasibility Acceptance
Feasibility AcceptanceAlthough the developers address the addition of the instrument with the already required OAS CAHPS, it seems to be a significant barrier to overcome.
Scientific Acceptability
Scientific Acceptability ReliabilityReliability testing is strong.
Scientific Acceptability ValidityWhile the testing data supports the validity of the measure, I have concerns with the specifications. In particular, the survey instructions allow proxy caregivers to respond to the survey. However, responses do not indicate if the respondent was a proxy. That may affect the results and may not be indicative of the patient's responses. Perhaps the measure should exclude patients who cannot complete the survey themselves.
Equity
EquityNot addressed, but also not required.
Use and Usability
Use and UsabilityI understand that this measure will have an initial rollout in HORP. I would recommend that it not be publicly reported.
Summary
The developers have demonstrated that the measure is important and methodologically sound. I noted a few concerns regarding the measure specifications and feasibility.
N/A
Importance
ImportanceI agree with the Staff's assessment.
Feasibility Acceptance
Feasibility AcceptanceI agree with the Staff's assessment.
Scientific Acceptability
Scientific Acceptability ReliabilityI agree with the Staff's assessment.
Scientific Acceptability ValidityI agree with the Staff's assessment.
Equity
EquityTopic was not addressed.
Use and Usability
Use and UsabilityI agree with the Staff's assessment.
Summary
N/A
This measure carries strong…
Importance
ImportanceAgree with staff assessment.
Feasibility Acceptance
Feasibility AcceptanceAgree with staff.
Scientific Acceptability
Scientific Acceptability ReliabilityAs with staff, share concerns around the clarity of the denominator. Also concerned that patients who deem multiple survey questions non-applicable will have an outsized influence on measure performance.
Scientific Acceptability ValidityAgree with staff assessment.
Equity
EquityNot included.
Use and Usability
Use and UsabilityAgree with staff.
Summary
This measure carries strong potential to improve the post-operative outcomes and reduce downstream expenditures by identifying opportunities to educate and engage patients in their care after an ambulatory surgical service.
This appears to be a QM…
Importance
ImportanceThe literature review was not rigorous. The developer identifies need for consistent information in outpatient setting following surgical procedures. Cites one review paper a systematic review of 58 studies and 5,721 participants discharged after an inpatient surgical procedure, which found patients had a greater understanding of self-care and better symptom experience—which seems broad and subjective. Was this restricted to only inpatient surgery? It appears that few studies to support association with improved outcomes but these are also based on inpatient admissions?
Feasibility Acceptance
Feasibility AcceptanceThe larger contextual issue is that this QM requires patient survey data. The study team oversaw data collection, but it’s not clear if hospitals will be able to collect data, analyze and report the data. For the pilot study, data were collected in batches. Lessons learned from CAHPS include that patient survey data is challenging for some Medicaid agencies to report. Even during this pilot, only 9 HPODs (out of 19? 26?) had CAHPS data. Why propose a type of data source that at least some Medicaid agencies have difficulties reporting on?
It may be premature to assess the measure for endorsement. The development of the survey included two pilot studies. The original survey was 21-items. It was reduced to nine. Questions that overlapped with CAHPS and gathered self-report on socio’s were deleted.
Without the self-report on the socio’s, the team was left relying on EHR data that had missing data, such as race.
The 9-item version was used for the second pilot test. Without the self-report on the socio’s, the team was left relying on EHR data that had missing data, such as race. Without the self-report on the socio’s, the team was left relying on EHR data that had missing data, such as race.
The survey was administered between 08-01-2022 - 03-01-2023 at 26 facilities in Western or Northeastern US regions. Seven facilities and one hospital shared a single CCN (CMS Certification number) and were grouped together as a single facility for analysis, resulting in a final count of 19 measured entities or HOPDs. It appears that the “19 facilities were collocated within a HCUP-defined medium-to-large urban teaching medical center.” Perhaps only academic medical centers in urban areas were only included? This is an issue if CMS wants to use this measure for healthcare systems that may be safety-net hospitals that serve more Medicaid beneficiaries? There is no mention of risk adjustment at the facility level. Also the severity of the procedure likely varied at the individual level?
Facility eligibility criteria were a minimum monthly case volume of 250 outpatient procedures.
Response rates are conservatively estimated but not reported, likely because data were batched? They were calculated by dividing respondents by the total number of individuals who were sent a survey, regardless of whether the survey bounced back or failed to send due to missing contact information.
The second pilot identified areas for improvement that were not addressed. It’s possible that funding ended, making it not possible to further refine before being considered for endorsement. Six facilities belonging to one organization had a 50% failure rate of the survey due to front line staff not capturing patient emails in their records. Some facilities with the participating organizations were eliminated due to low monthly case volumes and high failure rates resulting in an inability to reach the minimum number of responses (n=100). It is reported later in application that 15 facilities had min number of responses. It appears that about 79% (15/19) of the facilities did report back data on at least 100 surveys?
Patients also rejected some questions: “Approximately 12.2% of all respondents skipped questions about applicability, 12.9% of respondents skipped questions about medications, and 13.7% skipped questions about activity.” Note: there were no questions about applicability. However, there were response options for items 3-7 that allowed for reporting not applicable.
The denominator is the total number of eligible respondents for the facility. Respondents are eligible if they are 18 years or older, had a procedure or surgery, and were discharged alive with a stay less than two midnights. The “two midnights” is awkward. Patient discharged within 24 hours after procedure? The developer proposes to subtract the date of discharge from the date of admission and dividing it by 24 hours. Maybe the research team can do this, but can Medicaid agencies easily do this? Did the developer only have dates of admission and discharge and not time-stamped data for these?
Scientific Acceptability
Scientific Acceptability ReliabilityThe measure specifications have some challenges related to how the survey is scored. I appreciate the developer sharing the actual survey and the additional time spent by the Battelle to obtain it and help address additional questions.
Individual Score:
Questions for the Developer:
1. The numerator description states, “An individual score is calculated for each respondent by taking the sum of items for which the respondent gave the most positive response (“Yes” or “Very Clear”) and dividing by the number of items the respondent deemed applicable to their procedure or surgery. Applicable items are calculated by subtracting the sum of items for which the respondent selected “Does not apply” from the total number of items (nine).
However, in looking over the instrument,, the survey has nine items but the response options differ between items 1, 2 and 3-9. it appears that only 7 of the 9 survey questions have the response options that include does not apply. The first two do not include this option.
Q: Was there an error in the measure description and team meant to report only 7 items for allowing to opt out of score if not applicable?
2. The measure description states, “It uses a 9-item survey to obtain patient’s feedback on 3 domains: applicability; medications; and daily activities”.
Applicability is mentioned as one of the three domains. But it appears that the heading for the first two items is “Information Took Into Account My Needs.”
Items 3-9 include the response option of not applicable. The other ratings are Very Clear, Somewhat, No. The domains are: 1) information about new medication (why, SE’s, when to stop) (n=3); and 2) daily activities (change diet, change physical activity, when return to work, when could drive) (n=4). The respondent is asked to rate whether the information considered broadly defined “health needs” and “personal situation” (yes, no). Personal situation examples include lack of transportation, insurance coverage. There is no option for “not applicable” for this rating.
Q: Was the first domain something more global like, extent information took into account my needs?
Q: What if a person answers NO to personal situation because they did not have major barriers that had to be addressed, such as lack of transportation, lack of insurance? Could there be situations where a commercially insured patient, fully employed, with good social support reports “no” but states no because “not applicable” and could unfairly lower the score?
3. Individual scores. The measure description states, “An individual score is calculated for each respondent by taking the sum of items for which the respondent gave the most positive response (“Yes” or “Very Clear”) and dividing by the number of items the respondent deemed applicable to their procedure or surgery. Individual patient scores are calculated using a top-box approach, measuring the percentage of the total number items given the most favorable responses ("Yes" or "Very Clear") out of the total number of relevant items.”
Q: To double check, the individual score will include a point for Yes among items 1 and 2, and then a point for Very Clear for items 3-9. The denominator will vary depending on final number (out of 7) of items that are rated as not applicable. There is a possibility that all seven items could be identified as not applicable, even though the pilot data does not suggest this.
Thus, if a sum, is it possible that a person could report scores ranging from 0-9? Persons with more applicable survey items may be more likely to receiving a higher sum score? This could be confounded by type of procedure, which is not addressed.
Q: If the individual score is the % of items that had a yes or very clear, can a patient rate the provider as 100% if yes for only the first two items if 3-7 not applicable?. A patient may rate a provider as 100%, if yes first two and Very Clear only one item that was deemed applicable? A patient may rate a provider as much lower, if Yes on health needs, somewhat on personal situation and somewhat on items 3-9?
4. Facility level. The measure description states, “The numerator is the sum of all individual scores a facility received from eligible respondents.” “Facility scores are calculated by averaging the individual patient scores for each facility.”
Q: Is the individual score a %, or simply sum of all items that are rated yes (items 1,2) and among items 3-9 that are deemed applicable, the items that are rated Very Clear?
Q: If a sum, a person could report scores ranging from 0-9? Persons with more applicable survey items will be more likely to receiving a higher sum score?
Q: Is this the average of the sum of points using the top-notch approach or averaging the % score for the individual?
Q: If average, is the standard deviation also reported to help the facility interpret this rating?
Q: Was the team envisioning using the % to provide feedback to a provider (like 100%, 80%), but use sums for facility score?
5. A strength of the survey is that “Our survey provides feedback to providers on what elements of discharge instruction could be improved.”
Q: Did the pilot study obtain any data to support the feasibility of providers (not facilities which is the unit of analysis) being given back the responses of their patients within x period of time?
Q: Or is there argument that by using this survey, there is potential for facilities to stratify these data by provider and the assumption is that facilities have the resources to provide this feedback to the providers from their patients in relatively real time? That providers have protected time to review this information?
Reliability testing of the survey items is limited to using the Cronbach Alpha score and was found to be 0.8941. I assume this is at the individual level.
At the facility level (n=15), reliability was assessed using signal to noise ratio. This approach is also used at the health plan level (e.g., NCQA). Reliability scores range from a minimum of 0.572 to a maximum of 0.817, with a mean of 0.689.
Overall, the survey is a bit unique because the patient can judge whether up to 9 of the items are applicable. This adds a little more complexity to the programming of these data to subtracting the sum of items for which the respondent selected “Does not apply” from the total number of items (nine). Is it possible that a patient may report on only 1-2 of the questions and if 1-2 are rated highly and divided by 1-2, they get 100% for “yes” or “very clear” but respondent burden is less, more likely to complete?
The denominator is the total number of eligible respondents for the facility. Respondents are eligible if they are 18 years or older, had a procedure or surgery, and were discharged alive with a stay less than two midnights. The “two midnights” is awkward. Patient discharged within 24 hours after procedure? The developer proposes to subtract the date of discharge from the date of admission and dividing it by 24 hours. Maybe the research team can do this, but can Medicaid agencies easily do this? Did the developer only have dates of admission and discharge and not time-stamped data for these?
Scientific Acceptability ValidityThe title of the measure is quite ambitious, “Patient Understanding of Key Information Related to Recovery After a Facility-Based Outpatient Procedure or Surgery, Patient Reported Outcome-Based Performance Measure.” It implies that patient understanding of key information is measured but this is per patient report and the number of items that the patient deems applicable could vary from 0-9? Key information is not operationally defined---is this what the patient judges as applicable? Or is key information what is clinically necessary to have? Achieving these two requirements is assumed to associated with improved recovery. During the pilot testing, neither the assessment of the target mechanism nor proposed outcome are assessed. It is not clear if the pilot testing was restricted to patients following outpatient surgery or also included patients with “other outpatient procedures.”
The measure description reports that the measure assesses how well facilities provide clear, personalized discharge instructions to patients. Again, how was clarity and how the extent discharge instructions were personalized or tailored to the patient’s characteristics and preferences is not well specified.
Validity testing was not robust, but consistent with approach by NCQA. This raises the usual question of whether we keep standards for scientific acceptability for QM’s relatively low to be consistent with earlier QM’s or raise the bar for new ones.
Face validity (judgement call) using a TEP (n=10), they were asked to rate using a Likert scale their level of agreement with only two statements:
1) “The unadjusted Information Transfer PRO-PM, as specified, will provide valid assessment of the transfer of key information to patients at discharge from the facility,” only 60% agreed
2) “The unadjusted Information Transfer PRO-PM, as specified, can be used to distinguish between better and worse quality care at measured facilities” 80% agreed.
Criterion validity of the 9-item survey measure was assessed by comparing hospital performance on the 9-item instrument to their performance on the OAS CAHPS. Only 9 HPOD’s had CAHPS survey data. The team compared results with the results from the item on communication about your procedure. A positive correlation reported.
The final version of the survey is 9 items is available in Spanish and English. There is no mention in how the different versions performed during reliability and validity testing.
Equity
EquityData on race were missing, stratification not feasible.
The argument for not adjusting for patient level social factors seemed weak. The developer states there was consensus that Hospital Outpatient Departments (HOPD) should tailor discharge instructions to accommodate patients' unique characteristics, such as age, education, health status, and prior experience with surgeries. However, tailoring was not tested.
The developer states, that “it is essential for clinicians to connect with patients on their level and provide clear, personalized guidance for post-discharge self-care. This is a measure of patient centered communication.” However, the pilot test did not assess the quality of provider-patient communication or how personalized the guidance presented was.
I did not understand this, “Risk adjusting for patient-level social factors that can be influenced by the HOPD discharge process could unintentionally encourage different/ disparate outcomes between social disadvantaged patient care. Therefore, the unadjusted measure effectively captures the average patient's assessment of the clarity of discharge instructions provided by HOPD.” Why would the discharge process differ for persons?
There was no mention of developing methods to risk adjust at the facility level. This could be important if facilities include safety-net hospitals.
Use and Usability
Use and UsabilityNot in use. Plans for use not provided. Not ready for prime time.
The developer reports, “Our survey provides feedback to providers on what elements of discharge instruction could be improved.” How? Was this assessed?
Did the pilot study obtain any data to support the feasibility of providers (not facilities which is the unit of analysis) being given back the responses of their patients within x period of time?
Or is the argument that by using this survey, there is potential for facilities to stratify these data by provider and the assumption is that facilities have the resources to provide this feedback to the providers from their patients in relatively real time? That providers have protected time to review this information?
Summary
This appears to be a QM developed by Yale (referred to as “third party vendor”) which was contracted by CMS to develop the measure. Questions are raised about differences in how the individual and facility scores are calculated. Applicability is identified as one of the domains, but applicability is a response option for only items 3-9 in the survey. There are general concerns about feasibility of patient survey data for use by state Medicaid agencies and whether there has been sufficient pilot testing to further refine and examine use for possible version 3. Concerns about measure specifications are detailed under reliability. Approach for validity testing is somewhat consistent with NCQA measures using data at only the health plan level, but do we continue to maintain this standard for new measures? Without self-reported sociodemographic characteristics in the survey, the team must rely on other patient-level data sources that increase risk for missing data. Also, it’s not clear how this capacity could vary by facility, which could differ by data infrastructure. These concerns raise questions about whether this measure has the capacity to identify disparities. The measure is not in use and there no plan reported.
N/A
Importance
ImportanceAgree with staff assessment. Understanding post-op care instructions will help patient's with their recovery appropriately while also impacting ED visits, readmissions, etc.
Feasibility Acceptance
Feasibility AcceptanceAgree with staff assessment. Elements are required under USCDI v1 and v2, and are required by ONC within certified EHRs and data is provided directly from patients.
Scientific Acceptability
Scientific Acceptability ReliabilityAgree with staff assessment.
Scientific Acceptability ValidityAgree with staff assessment
Equity
EquityDeveloper did not address. It should be evaluated in future versions of the measure to provide additional information about gaps and educational opportunities within stratified areas.
Use and Usability
Use and UsabilityAgree with staff assessment.
Summary
This is an important area where patients can provide insight into their understanding, adherence, and personalized care.
N/a
Importance
ImportanceI find support for this business case both in the information presented and experientially in my work.
Feasibility Acceptance
Feasibility AcceptanceI appreciate the work done here by the developers; however, EHR and email systems are notoriously diverse in what clinicians are and are not able to see. This measure may be well defined but practice behaviors are such that these activities are not always prioritized, especially in light of labor shortages in the industry
Scientific Acceptability
Scientific Acceptability ReliabilityI am in agreement with staff here
Scientific Acceptability Validityin agreement w/ staff here as well.
Equity
EquityNo representation of participation in this optional domain
Use and Usability
Use and UsabilityAs designed, the measure is met; however, it remains to be seen how well it performs in this next cycle if approved.
Summary
N/A
Overall, I think this…
Importance
ImportanceAgree with staff assessment. The measure addresses an important issue surrounding appropriate education of patients prior to discharge.
Feasibility Acceptance
Feasibility AcceptanceSeems to be feasible. One potential barrier is the ability of hospitals to obtain necessary patient information to send out the survey (i.e., email). Training of staff should assit with this.
Scientific Acceptability
Scientific Acceptability ReliabilityDevelopers should provide further clarity regarding the denominator.
Scientific Acceptability ValidityThere appears to be no risk adjustment.
Equity
EquityDid not provide.
Use and Usability
Use and UsabilityThis will be a publicly reported measure, which seems appropriate.
Summary
Overall, I think this measure addresses an important issue surrounding patient education at time of discharge following surgery. Issues which are relevant to this measure include the potential difficulty with reporting for rural or low volume centers (developers note there needs to be at least 100 survey responses to calculate measure), and the ability to send patients surveys (obtaining patient emails, potential need to train staff to address this). The developers could take it a step further and provide more clarity regarding what else we can do with the outcome of this measure.
A good starting point with future work necessary
Importance
ImportanceThe literature review although comprehensive does not clearly connect increased pt understanding and activation associated with easier to understand instructions for surgical pt to improved pt outcomes for surgical outpts. Many of the studies increasing improved outcomes are associated with medical inpatient discharges. Although one maybe able to extrapolate it would be important to either identify the literature supporting this measure in surgical outpts or note the lack of literature.
Feasibility Acceptance
Feasibility AcceptanceI agree with staff that the measure can be met since the denominator is respondents not all patients. The capture of emails will not address those who do not have emails or access to electronic communication. It seems texting is another option for implementation of the survey. The addition of mail may capture more.
Scientific Acceptability
Scientific Acceptability ReliabilityI agree with staff assessment
Scientific Acceptability ValidityI agree with staff assessment
Equity
EquityThis is optional and indicated that it was not going to be addressed in this measure.
I think there should be some recognition that people who are unable to read and respond to written surveys in English or Spanish may experience a greater risk of negative post-operative outcomes which could be mitigated by providing the improved discharge instructions in a format in which they can engage.
Use and Usability
Use and UsabilityI agree with staff assessment though I would add that the measure is not generalizable to people who are illiterate or non-English or non-Spanish speaking this could potentially impact the use of this measure for facilities providing services to a population with a higher proportion of people with low English or Spanish proficiency. In turn those populations may also not have a means to inform the quality of discharge information provided to them since this measure would be inaccessible to them.
Summary
I think there should be some recognition that people who are unable to read and respond to written surveys in English or Spanish may experience a greater risk of negative post-operative outcomes which could be mitigated by providing the improved discharge instructions in a format in which they can engage. I believe this is an appropriate start with future work improving the capacity of measuring the quality provided to patients who fall into this category
Final Comment
Importance
ImportanceAgree with the need to receive feedback that discharge instruction are being understood by the patient or caregiver at the facility level.
Feasibility Acceptance
Feasibility AcceptanceAgree with the staff assessment. Would like to see a text options vs email as I would think that would increase participation.
Scientific Acceptability
Scientific Acceptability ReliabilityAgree with the staff assessment. I would emphasize confusion with the term two midnights and argue that is too short. Would hope that would be expanded in future submission if approved.
Scientific Acceptability ValidityAgree with the staff assessment. Appreciate no risk-adjustment for this type of measure.
Equity
EquityIt was mentioned that the survey was available in English and Spanish. Would hope this could be expanded to more languages.
Use and Usability
Use and UsabilityThis measure would be beneficial to support QI programs at facilities. It would also be interesting to see this measure related to readmissions.
Summary
I think the patient community and employers who direct self-funded plans would welcome this measure as a means to identify high quality health care facilities.
Not Yet
Importance
ImportanceThe measure makes sense on its face, but the cursory literature review could have been more robust
Feasibility Acceptance
Feasibility AcceptanceI have concerns about the feasibility of achieving reliable scores in low-income populations. Mail surveys in this population frequently suffer from low-response rates and may be biased towards those who are unsatisfied. If this is so brief and unobtrusive a measure, why can't it be administered after the patient has received discharge insytructions but before leaving the facility? This would enable the faiclity to address concerns before the patient leaves so iterative improvement on d/c instructions should occur.
Scientific Acceptability
Scientific Acceptability ReliabilityAgree with staff assessment
Scientific Acceptability ValidityAgree with staff assessment
Equity
EquityI don't believe this was addressed at all
Use and Usability
Use and UsabilityThere are too many issues with this measure for it to be included as a publicly reported measure. Authors note it could be used for a varity of things, but provide no additional information. I don't think this is ready for approval as is.
Summary
This is a really important issue but the measure as currently designed is not ready for roll out
CBE#4210
Importance
ImportanceAgree with staff assessment.
Feasibility Acceptance
Feasibility AcceptanceAgree with staff assessment
Scientific Acceptability
Scientific Acceptability ReliabilityAgree with staff assessment
Scientific Acceptability ValidityAgree with staff assessment
Equity
EquityThis would be a valuable incorporation in the measure
Use and Usability
Use and UsabilityAgree with staff assessment
Summary
Agree completely with staff assessment. The developer shows what elements of discharge instruction and communication could be improved. The developer clearly showed the importance of the measure.
Important measure, with potential to drive positive system chang
Importance
ImportanceAgree with staff assessment.
Feasibility Acceptance
Feasibility AcceptanceAgree with staff assessment.
Scientific Acceptability
Scientific Acceptability ReliabilityAgree with staff assessment.
Scientific Acceptability ValidityAgree with staff assessment.
Equity
EquityIn that this measure directly addresses patient reported quality of discharge education- emphasizing personalized discharge instruction, I am concerned the developers have chosen not to acknowledge the Health Equity section.
Elsewhere in the submission, developers acknowledge potential consequence to those with higher levels of education, as based on pilot data they "tended to be less satisfied with the discharge information they received. However, there is no evidence to suggest that hospitals would limit access to socially advantaged group (those with higher education) as a consequence of publicly reporting an unadjusted PRO-PM."
After reviewing pilot study descriptive statistics, the current data are based on study population of 93% English speaking patients.
I am curious how this tool would perform in a study sample with greater representation of non-English speaking populations. Delivering "personalized, clear discharge instructions," requires greater system level resource investment when language barrier is present (multi-modal translation services, clinician time) and variability between translators can lead to inconsistency in education received by patients.
I would like to see developers expand upon health equity considerations for this measure, as I think there should be consideration for how this may impact non-English speaking populations and accessibility of care.
Use and Usability
Use and UsabilityAgree with staff assessment.
Summary
Overall, my concern relates to lack of consideration for how this measure may impact patients who are non-English speaking, the HOPD that serve non-English speaking communities, and potential implications for accessibility for this already vulnerable group.
-
AOTA supports adding the Patient Understanding of Key Information Related to Recovery After a Facility-Based Outpatient Procedure or Surgery, Patient Reported Outcome-Based Performance Measure. Providing personalized, clear discharge instruction is important to compliance and follow through with medical recommendations. Several factors can influence a patient’s understanding of discharge instructions including health literacy level and functional cognition.