When the measure returns for maintenance, the committee would like to see:
- Explore methods to enable the evaluation of improvement over time; and
- Consider additional approaches for the reliability assessment that inform the reliability-validity (e.g. shrinkage) and reliability-usability (e.g. stability) tradeoffs
This measure was developed to improve the quality of care delivered to patients undergoing orthopedic procedures in an ambulatory surgical center (ASC). To assess quality, the measure calculates the risk-standardized rate of acute, unplanned hospital visits within seven days of qualified orthopedic surgeries or procedures performed at an ASC among Medicare fee-for-service (FFS) patients aged 65 years and older. An unplanned hospital visit is defined as an emergency department (ED) visit, observation stay, or unplanned inpatient admission.
Measure Specs
- General Information(active tab)
- Numerator
- Denominator
- Exclusions
- Measure Calculation
- Supplemental Attachment
- Point of Contact
General Information
The rationale for public reporting of the ASC Orthopedic Surgery remains imperative, given the significant growth in ASC utilization; in 2018, there were more than 23 million ambulatory surgeries performed at ASCs in the United States (Young et al., 2021). As ASCs increasingly become the preferred choice for lower-risk surgeries, including orthopedic procedures, evaluating postoperative outcomes, such as unplanned hospital visits, becomes pivotal for ensuring quality care and patient safety. This rising trend in orthopedic procedures at ASCs underscores the importance of incentivizing ASCs to address preventable complications and acute care needs, thus fostering continuous quality improvement in outpatient surgical care (Lopez et. al., 2021).
The ASC Orthopedic Surgery measure captures unplanned hospital visits 7 days after an outpatient orthopedic procedure. The measure focuses on the outcome of unplanned hospital visits because this is a broad, patient-centered outcome that captures the full range of hospital visits resulting from adverse events or poor care coordination following outpatient surgery. By providing ASCs with detailed information about patients who have an unplanned hospital visit, this measure supports quality improvement at ASCs, and through public reporting the measure provides through assessment and illumination of the variation in risk-adjusted hospital visits following outpatient orthopedic procedures.
As described in Section 2.2, efforts to reduce post-discharge hospital visits by addressing surgical site infections and medication reconciliation are pivotal in enhancing patient outcomes, exemplified by initiatives like the AHRQ's quality improvement collaborative for ambulatory surgery (Davis et al., 2019). Improved data availability is crucial for accurately assessing outcomes and facilitating post-discharge patient follow-up. Better measurement and understanding of hospital visits post-surgery are essential for driving quality improvement initiatives and ensuring patient-centered care in outpatient settings. Hence, interventions aimed at enhancing care quality, including optimized patient selection, advanced surgical techniques, pain management protocols, patient education, and medication reconciliation, have the potential to mitigate unplanned hospital visits and improve overall surgical outcomes (Erhun et. al., 2016). Please see Section 6.2.1 for more information on how ASCs can improve performance.
Improving the quality of care provided at ASCs is a key priority in the context of growth in the number of ASCs and procedures performed in this setting. Orthopedic procedures conducted in ambulatory settings have historically been minor in nature. However, advancements in surgical and anesthetic techniques, cost-effectiveness, and lower complication rates now enable the performance of more complex procedures on an outpatient basis; CMS now allows both total knee and total hip arthroplasty to be performed in the ASC setting.
References
Davis KK, Mahishi V, Singal R, Urman RD, Miller MA, Cooke M, Berry WR. Quality Improvement in Ambulatory Surgery Centers: A Major National Effort Aimed at Reducing Infections and Other Surgical Complications. J Clin Med Res. 2019 Jan;11(1):7-14. doi: 10.14740/jocmr3603w. Epub 2018 Dec 3. PMID: 30627272; PMCID: PMC6306128.
Erhun F, Malcolm E, Kalani M, Brayton K, Nguyen C, Asch SM, Platchek T, Milstein A. Opportunities to improve the value of outpatient surgical care. Am J Manag Care. 2016 Sep 1;22(9):e329-35. PMID: 27662397.
Lopez, C. D., Boddapati, V., Schweppe, E. A., Levine, W. N., Lehman, R. A., & Lenke, L. G. (2021). Recent Trends in Medicare Utilization and Reimbursement for Orthopaedic Procedures Performed at Ambulatory Surgery Centers. The Journal of bone and joint surgery. American volume, 103(15), 1383–1391. https://doi.org/10.2106/JBJS.20.01105
Young S, Pollard RJ, Shapiro FE. Pushing the Envelope: New Patients, Procedures, and Personal Protective Equipment in the Ambulatory Surgical Center for the COVID-19 Era. Adv Anesth. 2021 Dec;39:97-112. doi: 10.1016/j.aan.2021.07.006. Epub 2021 Jul 27. PMID: 34715983; PMCID: PMC8313519.
Medicare Fee-For-Service (FFS) claims and administrative data. Please see section 4.2.2 for details.
Numerator
The measure defines the outcome as any (one or more), all-cause, unplanned hospital visit within seven days of an outpatient orthopedic procedure; a hospital visit includes any emergency department (ED) visit, observation stay, or unplanned inpatient admission occurring within seven days after the ASC procedure.
Unplanned Hospital Visits
The measure defines the outcome as any unplanned hospital visit within seven days of an outpatient ASC orthopedic procedure; a hospital visit includes any emergency department (ED) visit, observation stay, or unplanned inpatient admission. The measure focuses on the outcome of unplanned hospital visits because this is a broad, patient-centered outcome that captures the full range of hospital visits resulting from adverse events or poor care coordination following outpatient surgery. This measure’s goal is to support quality improvement through the assessment and illumination of the variation in risk-adjusted hospital visits following surgery.
ED visits and observation stays are defined using revenue center codes identified in Medicare Part B outpatient hospital claims. The 2023 Orthopedic Measure Data Dictionary tab “ASC Ortho Outcome ED Obs” provides the specific codes used to identify ED visits and observation stays.
Seven Day Time Frame
The measure limits the outcome to seven days, as existing literature suggests that most adverse events after outpatient surgery occur within the first seven days following the surgery and our empirical analyses during measure development indicated that the highest rates of hospital visits were within seven days of outpatient orthopedic procedure. Thus, based on existing literature and empirical findings, as well as input from the Technical Expert Panel (TEP) and orthopedic consultants, the measure development team concluded that unplanned hospital visits within seven days is the optimal outcome to ensure capture of surgery-related adverse events and to minimize capture of hospital visits unrelated to the surgery.
Planned Hospital Visits
For inpatient admissions, the measure outcome includes hospital visits within the first 7 days following the procedure, unless that inpatient admission is deemed a “planned” admission as defined by the measures Planned Admissions Algorithm (PAA). The algorithm is a set of criteria for classifying readmissions as planned or unplanned using Medicare claims. The Centers for Medicare & Medicaid Services (CMS) seeks to count only unplanned admissions in the measure outcome, because variation in “planned” admissions does not reflect quality differences. The measure does not consider observation stays or ED visits as planned.
In brief, the algorithm identifies admissions that are typically planned and may occur after the patient’s index event. The algorithm always considers a few specific, limited types of care planned (e.g., major organ transplant, rehabilitation, or maintenance chemotherapy). Otherwise, the algorithm defines a planned admission as a non-acute admission for a scheduled procedure (e.g., total hip replacement or cholecystectomy), and the algorithm never considers admissions for acute illness or for complications of care planned. For example, the algorithm considers hip replacement unplanned if hip fracture (an acute condition) is the discharge diagnosis but planned if osteoarthritis (a non-acute condition) is the discharge diagnosis. The algorithm considers admissions that include potentially planned procedures with acute diagnoses or that might represent complications of a procedure unplanned and thus counts these admissions in the measure outcome.
For more information about the PAA, please see the attached 2023 Orthopedic Surgery Measure Updates and Specification Report. Also see sheets ‘PAA1 Always Planned Px’, ‘PAA2 Always Planned Dx’, ‘PAA3 Post Planned Px’, and ‘PAA4 Acute Dx’ in the attached Data Dictionary for the most up-to-date sets of codes in the algorithm for ‘always planned procedures,’ ‘always planned diagnoses,’ ‘potentially planned procedures, and ‘acute’ diagnoses.
Denominator
The target population for this measure is Medicare FFS patients aged 65 years and older undergoing selected outpatient orthopedic surgeries, typically performed by an orthopedist, at ASCs.
See Figure 1 (in attachment) for a flow chart that outlines the cohort construction.
The target population for this measure is Medicare FFS patients aged 65 years and older, undergoing outpatient orthopedic procedures in ASCs that are routinely performed by orthopedists. We limit the measure to patients who have been enrolled in Medicare FFS Parts A and B for the 12 months prior to the date of surgery to ensure that we have adequate data for identifying comorbidities for risk adjustment. See the attached Data Dictionary, “ASC Ortho Cohort” for a complete list of all CPT procedure codes included in the measure cohort.
We identify eligible outpatient orthopedic surgeries using Medicare’s ASC list of covered procedures, which includes procedures for which ASCs can be reimbursed under the ASC payment system. The January 2023 list of surgeries is publicly available at: https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Payment/ASCPayment/11_Addenda_Updates.html (refer to Addendum AA). Surgeries on the ASC list of covered procedures do not involve or require major or prolonged invasion of body cavities, extensive blood loss, major blood vessels, or care that is either emergent or life-threatening. The ASC list is publicly available, is annually reviewed and updated by Medicare, and includes a transparent public comment submission and review process for addition and/or removal of procedure codes. Using an existing, defined list of surgeries, rather than defining surgeries de novo, is useful for long-term measure maintenance. Procedures listed in Medicare’s list of covered ASC procedures are defined using Healthcare Common Procedure Coding System (HCPCS) and Common Procedural Terminology (CPT) codes.
Ambulatory surgeries include a heterogeneous mix of non-surgical procedures, minor surgeries, and more substantive surgeries. The measure is not intended to include very low-risk (minor) surgeries or non-surgical procedures. Therefore, we further limit the list of covered ASC procedures to “major” and “minor” procedures defined using Medicare’s Global Surgical Package (https://www.cms.gov/files/document/mln907166-global-surgery-booklet.pdf). Specifically, we identify “major” and “minor” surgeries using the global surgery indicator (GSI) values of 090 and 010, respectively, which correspond to the number of post-operative days included in Medicare’s global surgery payment for the procedure. The measure does not include minor/non-surgical procedures identified using the GSI code 000.
Finally, to define the orthopedic procedures cohort, we use the Clinical Classifications Software (CCS) developed by the Agency for Healthcare Research and Quality (AHRQ). The CCS is a tool for clustering procedures into clinically meaningful categories using CPT codes by operation site. We include all procedures defined by the CCS as “operations on the musculoskeletal system” and retained all of those typically performed by orthopedic surgeons. Examples of orthopedic procedures include treatment of toe deformities, arthroscopic knee procedures, therapeutic procedures on muscles, tendons, joints, and bones, and treatment of fractures. The measure does not, however, include CCS 144 (“Procedures to treat a facial fracture or dislocation”) because our experts indicated that these procedures are typically performed by plastic surgeons; ear, nose, and throat surgeons; and oral maxillofacial surgeons. See the accompanying 2023 Orthopedic Measure Data Dictionary for a complete listing of all Current Procedural Terminology (CPT®) procedure codes included in the measure cohort.
Exclusions
The measure excludes surgeries for patients without 7 or more days of continuous enrollment in Medicare FFS Parts A and B after the surgery. The measure excludes these patients to ensure all patients have full data available for outcome assessment.
Please see Figure 1 (attachment) for a flow chart that diagrams the cohort construction (inclusion and exclusion).
Lack of 7 or more days of continuous enrollment in Medicare FFS after the ASC surgery is determined by patient enrollment status in FFS Parts A and B using the Medicare Enrollment Database (unless lack of enrollment was due to death). The procedure must be 7 or more days from the end of the month, or the enrollment indicators must be appropriately marked for the month that falls within 7 days of the procedure date (unless disenrollment is due to death); otherwise, the procedure is excluded.
Measure Calculation
The measure uses a two-level hierarchical logistic regression model to estimate ASC-level risk-standardized hospital visit rates (RSHVRs). This approach accounts for the clustering of patients within ASCs and variation in sample size across ASCs The RSHVR is calculated as the ratio of the predicted to the expected number of post-surgical unplanned hospital visits among an ASC’s patients, multiplied by the national observed rate of unplanned hospital visits. For each ASC, the numerator of the ratio is the number of hospital visits predicted for the ASC’s patients, accounting for its observed rate, the number and complexity of orthopedic procedures performed at the ASC, and the case mix. The denominator is the number of hospital visits expected nationally for the ASC’s case/procedure mix. As noted above, to calculate an ASC’s predicted-to-expected (P/E) ratio, the measure uses a two-level hierarchical logistic regression model. The log-odds of the outcome for an index procedure is modeled as a function of the patient demographics, comorbidities, procedure characteristics, and a random ASC-specific intercept. A ratio greater than one indicates that the ASC’s patients have more post-surgical hospital visits than expected, compared to an average ASC with similar patient and procedural complexity. A ratio less than one indicates that the ASC’s patients have fewer post-surgical hospital visits than expected, compared to an average ASC with similar patient and procedural complexity. An ASC’s P/E ratio is then multiplied by the overall national rate of unplanned hospital visits to calculate the ASC-level RSHVR. This approach is analogous to an observed-to-expected ratio, but accounts for within-facility correlation of the observed outcome and sample size differences, accommodates the assumption that underlying differences in quality across ASCs lead to systematic differences in outcomes, and is tailored to and appropriate for a publicly reported outcome measure as articulated in published scientific guidelines (Normand and Shahian, 2007; Krumholtz et al., 2007).
Please see the measure updates document for a detailed description of the hierarchical model and measure score calculation: https://qualitynet.cms.gov/files/651b5f9970a30f001c388001?filename=2023_ASC_MeasureUpdatesRpt.pdf
Below we provide the individual steps to calculate the measure score:
- Identify surgeries meeting the inclusion criteria described in the denominator section above, and in Tab 2, “ASC_Ortho_Cohort,” of the data dictionary.
- Exclude procedures meeting any of the exclusion criteria described in the exclusion section above.
- Identify a binary flag for an unplanned hospital visit within 7 days of index procedures.
- Use patients’ historical and index procedure claims data to create risk-adjustment variables.
- Fit a hierarchical generalized linear model (HGLM) and calculate the ratio of the number of “predicted” hospital visits to the number of “expected” hospital visits for each facility, given its case/procedure mix using the results. This is the risk-standardized hospital visit ratio. The HGLM is adjusted for age, clinical risk factors, and procedure RVU and body system that vary across patient populations, are unrelated to quality, and influence the outcome. Details about the risk-adjustment model can be found in the original measure development methodology report: Facility-Level 7-Day Hospital Visits after General Surgery Procedures Performed at Ambulatory Surgical Centers under archived resources at https://qualitynet.cms.gov/asc/measures/surgery/resources#tab2
- Multiply the hospital visit ratio by the observed national 7-day hospital visit rate to obtain a risk-standardized hospital visit rate (RSHVR) for each facility.
- Use statistical bootstrapping to construct a 95% confidence interval estimate for each facility’s RSHV rate. For more information about the measure methodology, please see Appendix A in the most recent 2023 Measure Updates and Specifications Report posted here: https://qualitynet.cms.gov/files/651b5f9970a30f001c388001?filename=2023_ASC_MeasureUpdatesRpt.pdf
References
Krumholz HM, Brindis RG, Brush JE, et al. Standards for Statistical Models Used for Public Reporting of Health Outcomes An American Heart Association Scientific Statement From the Quality of Care and Outcomes Research Interdisciplinary Writing Group: Cosponsored by the Council on Epidemiology and Prevention and the Stroke Council Endorsed by the American College of Cardiology Foundation. Circulation. 2006;113(3):456-462.
Normand S-LT, Shahian DM. Statistical and clinical aspects of hospital outcomes profiling. Statistical Science. 2007;22(2):206-226.
This measure is not stratified.
Not applicable. This measure is not based on a sample.
Supplemental Attachment
Point of Contact
Not applicable
Marsha Hertzberg
Centers for Medicare & Medicaid Services (CMS)
7500 Security Boulevard
Windsor Mill , MD 21244
United States
Oscar Gonzalez
Acumen LLC
500 Airport Blvd., Suite 100
Burlingame, CA 94010
United States
Importance
Evidence
Unforeseen hospital visits following outpatient surgeries are often preventable and indicative of quality of care in outpatient surgical settings. These visits commonly occur due to complications such as infection, bleeding, urinary retention, nausea, vomiting, and pain, highlighting the significance of factors like patient selection, surgical technique, and post-discharge planning in healthcare delivery. Various demographic and clinical factors, including age and comorbidities, influence the risk of post-procedure hospital visits, underscoring the need for comprehensive preoperative assessment and care coordination (Heilbronn, Lin, & Bhattacharyya, 2020; Rothenberg et al., 2019; Alder et al., 2023). Additionally, technical aspects of surgery, such as anesthetic technique and duration, alongside adherence to clinical pathways and guideline-concordant care, significantly impact outcome rates and patient safety (Bongiovanni et al., 2021).
Efforts to address surgical site infections and medication reconciliation are pivotal in enhancing patient outcomes, exemplified by initiatives like the AHRQ's quality improvement collaborative for ambulatory surgery (Davis et al., 2019). Improved data availability is crucial for accurately assessing outcomes and facilitating post-discharge patient follow-up. Better measurement and understanding of hospital visits post-surgery are essential for driving quality improvement initiatives and ensuring patient-centered care in outpatient settings. Hence, interventions aimed at enhancing care quality, including optimized patient selection, advanced surgical techniques, pain management protocols, patient education, and medication reconciliation, have the potential to mitigate unplanned hospital visits and improve overall surgical outcomes (Erhun et al., 2016).
As healthcare moves towards emphasizing value-based practices and more orthopedic procedures are conducted on an outpatient basis (including total hip and total knee arthroplasty), it becomes increasingly important to assess postoperative healthcare resource utilization. Decision-making regarding operative settings requires thorough clinical evaluation to mitigate risks and ensure safe patient care delivery. Understanding predictive risk factors for unplanned healthcare utilization following outpatient surgeries, along with associated costs, is vital as healthcare reimbursement shifts towards value-based payment models (Williams et al., 2021).
References
Alder, C., Bronsert, M. R., Meguid, R. A., Stuart, C. M., Dyas, A. R., Colborn, K. L., & Henderson, W. G. (2023). Preoperative risk factors and postoperative complications associated with mortality after outpatient surgery in a broad surgical population: An analysis of 2.8 million ACS-NSQIP patients. Surgery, 174(3), 631-637. https://doi.org/10.1016/j.surg.2023.04.048
Bongiovanni, T., Parzynski, C., Ranasinghe, I., Steinman, M. A., & Ross, J. S. (2021). Unplanned hospital visits after ambulatory surgical care. PloS one, 16(7), e0254039.
Davis KK, Mahishi V, Singal R, Urman RD, Miller MA, Cooke M, Berry WR. Quality Improvement in Ambulatory Surgery Centers: A Major National Effort Aimed at Reducing Infections and Other Surgical Complications. J Clin Med Res. 2019 Jan;11(1):7-14. doi: 10.14740/jocmr3603w. Epub 2018 Dec 3. PMID: 30627272; PMCID: PMC6306128.
Erhun F, Malcolm E, Kalani M, Brayton K, Nguyen C, Asch SM, Platchek T, Milstein A. Opportunities to improve the value of outpatient surgical care. Am J Manag Care. 2016 Sep 1;22(9):e329-35. PMID: 27662397.
Heilbronn, C., Lin, H., & Bhattacharyya, N. (2020). Adult ambulatory otologic surgery: Unplanned revisits and complications. The Laryngoscope, 130(7), 1788-1791. https://doi.org/10.1002/lary.28346
Rothenberg, K. A., Stern, J. R., George, E. L., et al. (2019). Association of frailty and postoperative complications with unplanned readmissions after elective outpatient surgery. JAMA Network Open, 2(5), Article e194330. https://doi.org/10.1001/jamanetworkopen.2019.4330
Williams BR, Smith LC, Only AJ, Parikh HR, Swiontkowski MF, Cunningham BP. Unplanned Emergency and Urgent Care Visits After Outpatient Orthopaedic Surgery. J Am Acad Orthop Surg Glob Res Rev. 2021 Sep 20;5(9):e21.00209. doi: 10.5435/JAAOSGlobal-D-21-00209. PMID: 34543235; PMCID: PMC8454905.
Measure Impact
Improving the quality of care provided at ASCs is a key priority in the context of growth in the number of ASCs and procedures performed in this setting. Ambulatory care now accounts for the majority of elective surgical care in the United States. From both the patient and provider perspective, several factors have fueled this growth. ASCs provide greater convenience for patients and families (due to advances in surgical technology), as well as lower financial costs for both patients and healthcare providers. As such, as more complex procedures migrate to the ASC setting, and ASCs serve and higher-risk patients there is an increased need to monitor the quality of care at ASCs (Munich et al., 2014; Goldfarb et al., 2017).
A hospital visit following same-day surgery, including orthopedic surgery, is an unexpected and potentially preventable outcome for patients scheduled for elective same-day surgeries that have a low anticipated risk. As ASC providers may be unaware of their patients’ hospital visits after surgery, they may underestimate adverse event rates, suggesting the need for better measurement to drive quality improvement. Therefore, both patients and providers benefit from outcome measures of hospital visits – a broad, patient-centered outcome that reflects the full range of reasons leading to hospitalization among patients undergoing same-day surgery.
Patients express concern about post-operative complications, in particular, pain, and say that they want to be informed about potential complications prior to the procedure. For example, in comments submitted to researchers following a survey of 800 patients on the topic of post-operative complications (Rendell et al., 2020), patients stated: “Make sure patients know all possible complications before their surgery,” and patients said they wanted to know what could any patients wanted happen to them after an operation; for example one patient wrote, “I wish someone had warned me of this [complication] before the surgery as it is quite common I now know.” Public reporting of post-operative hospital visit rates following orthopedic procedures (described below) is an act of transparency that can help patients better understand those post-operative risks.
The ASC Orthopedic Surgery measure is part of the Ambulatory Surgical Center Quality Reporting (ASCQR) Program, a pay-for-reporting program. The measure went through a “dry run” in 2018 to help educate measure entities on data collection and interpretation, familiarize them with facility-specific reports (FSRs), and for CMS to collect feedback on the measure specifications. This measure was first publicly reported January 2022. Currently, there are no other publicly available quality reports related to these elective orthopedic procedures (that now include total knee and total hip arthroplasty), which underscores the measurement gap that would exist without this measure. Thus, this measure addresses an important quality measurement area and enhances the information available to patients choosing among ASCs that provide same-day outpatient surgery. Furthermore, providing outcome rates to ASCs makes visible to clinicians and hospitals meaningful quality differences and incentivizes improvement.
References
Goldfarb CA, Bansal A, Brophy RH. Ambulatory surgical centers: a review of complications and adverse events. JAAOS-Journal of the American Academy of Orthopaedic Surgeons. 2017 Jan 1;25(1):12-22.
Munnich EL, Parente ST. Procedures take less time at ambulatory surgery centers, keeping costs down and ability to meet demand up. Health Affairs. 2014 May 1;33(5):764-9.
Rendell, V. R., Siy, A. B., Stafford, L. M. C., Schmocker, R. K., Leverson, G. E., & Winslow, E. R. (2020). Severity of postoperative complications from the perspective of the patient. Journal of Patient Experience, 7(6), 1568–1576. https://doi.org/10.1177/2374373519893199
Performance Gap
There remains variation in measure scores among facilities. Below we provide the distribution of measures scores and evidence of variation through the median odds ratio.
Measure score distribution
Table 1 and Table 2 (see attachment) show the distribution of measure scores, the risk-standardized hospital visit rate, or RSHVR, using the most recent testing data (2024 EM Dataset; January 1, 2021-December 31, 2022).
RSHVRs range from 1.01% to 3.68%; the median is 2.18%; the 25th percentile is 2.12% and the 75th percentile is 2.26%. The best performer (1.01%) is performing about 54 percent better than the median, and the worst performer (3.68%) is performing about 70 percent worse than the median.
Median Odds Ratio
We provide further evidence of variation by calculating and interpreting the median odds ratio (MOR) (Merlo, et al., 2006). The MOR represents the median increase in odds of a hospital visit if a procedure on a single patient was performed at a higher risk facility compared to a lower risk facility. It is calculated by taking all possible combinations of facilities, always comparing the higher risk facility to the lower risk facility. The MOR is interpreted as a traditional odds ratio would be.
The median odds ratio for the ASC Orthopedic Surgery Measure is 1.27. The median odds ratio suggests a meaningful increase in the risk of a hospital visit if an orthopedic procedure was performed at a higher risk facility compared to a lower risk facility. A value of 1.27 indicates that a patient has a 27 percent increase in the odds of a hospital visit if the orthopedic procedure was performed at higher risk facility compared to a lower risk facility indicating the impact of quality on the outcome rate is meaningful.
References
Merlo J, Chaix B, Ohlsson H, Beckman A, Johnell K, Hjerpe P, Råstam L, Larsen K. (2006) A brief conceptual tutorial of multilevel analysis in social epidemiology: Using measures of clustering in multilevel logistic regression to investigate contextual phenomena. J Epidemiol Community Health, 60(4):290-7.
| Overall | Minimum | Decile_1 | Decile_2 | Decile_3 | Decile_4 | Decile_5 | Decile_6 | Decile_7 | Decile_8 | Decile_9 | Decile_10 | Maximum | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Mean Performance Score | 2.20 | 1.01 | 1.88 | 2.05 | 2.12 | 2.15 | 2.17 | 2.18 | 2.20 | 2.27 | 2.35 | 2.62 | 3.68 |
| N of Entities | 2,961 | 1 | 296 | 296 | 296 | 296 | 296 | 297 | 296 | 296 | 296 | 296 | 1 |
| N of Persons / Encounters / Episodes | 479,525 | 1,630 | 130,182 | 62,806 | 30,971 | 23,194 | 12,331 | 7,742 | 31,979 | 37,052 | 48,494 | 94,774 | 568 |
Equity
Equity
As shown above, and described in the conceptual model, ASCs do not serve a high proportion of patients with social risk factors. The median facility proportion of patients with the DE variable for the ASC Orthopedic measure is 1% (count of 1 patient); and for high ADI, 2% (count of 1 patient). Therefore, we would expect, and show below, that adjusting this measure for social risk factors would have minimal impact on measure scores. Furthermore, when CMS attempted to stratify ASC measures for patients with DE, there were insufficient numbers of patients to support the stratification approach.
Nonetheless, we performed two analyses to explore the impact of adding either of two social risk factors (DE, and ADI) to the risk model, on measure scores. We found that adding either social risk factor to the model did not result in major impacts on measure scores, even for providers with the highest proportion of patients with either social risk factor (the top quartile, in our analyses). Therefore, in this pay-for-reporting program, providers will not be unfairly profiled when assessed using this measure. We describe the analyses and results below.
To examine the impact of social risk factors on measure scores, we first examined correlations (Pearsons) between measure scores calculated with and without either social risk factor in the risk model, and found that correlations were 0.999, and 1.000, for the DE and high ADI variables, respectively (Figure 5 and 6). Second, we examined the association (Spearman) between the facility proportion of patients with each social risk factor and measure scores for all facilities (Figures 7 and 8) focusing on the quartile of facilities with the highest proportion of patients with social risk factors (top quartile). We found that there is a very weak, but negative, significant correlation (r=-0.02, p=.05) between the measures score and the proportion of patients with the DE variable, meaning that facilities with a higher proportion of DE patients actually have slightly lower (better) measure scores; we see similar results for the high ADI variable, but the association is not significant (r=0.07, p>.05). Taken together this suggests there is no reason to adjust for social risk factors for the ASC Orthopedic Surgery measure.
Feasibility
Feasibility
This is a claims-based measure and as mentioned above, the measure score is calculated automatically from claims data which are routinely generated during the delivery of care. No data are collected by facilities; therefore, this measure imposes no burden on measured entities, and no implementation effort. CMS monitors feedback from the public and measured entities through CMS’s Q&A portal on QualityNet; there have been no concerns about burden related to this measure. There are no concerns about patient confidentiality because the measure is based on claims data submitted by facilities to CMS, and CMS then uses that data for both payment and calculation of the measure score.
We did not perform an analysis of missing data for the measure because it is based on a 100% sample of paid, final action claims submitted by facilities for payment. To ensure complete claims, we allow at least 3 months of time between accessing the data and the end of the performance period.
Because this is a claims-based measure there is no burden on the facility; rates are automatically calculated by CMS based on claims data submitted by facilities for payment.
Proprietary Information
Scientific Acceptability
Testing Data
We use paid, final action Medicare claims to identify orthopedic procedures performed in the outpatient setting at Ambulatory Surgical Centers (ASCs), including subsequent hospital visits. In addition, we use the Center for Medicare and Medicaid Services (CMS) enrollment and demographic data to determine inclusion and exclusion criteria. Patient history is assessed using claims data collected in the 12 months prior to the procedure. Please see the numerator and denominator details sections for additional information. Please see the sections below for details on the different datasets used for testing (original development, and this endorsement maintenance submission).
The measure requires a data source that allows us to link patient data across care settings in order to identify appropriate surgical procedures for inclusion, comorbidities for risk adjustment, and the outcome of hospital visits. Therefore, we used claims data, as they support these linkages and were available for the population of interest.
The datasets, dates, number of measured entities, number of orthopedic procedures, and demographic profile for the patients used in each type of testing are indicated below:
1. Medicare FFS FY 2015 Dataset
-Dates: October 1, 2014 – September 30, 2015
-Number of facilities: 2,523 ASCs
-Number of orthopedic procedures: 189,806
-Demographic characteristics: average age of 73.1 years; 60.6% female
-Dataset used for: Original measure development including selection of risk variables, face validity testing
2. Development Sample and Validation Sample
The Development and Validation Samples were random samples of the Medicare FFS FY 2015 Dataset. The Development Sample included 70% of the orthopedic ASC procedures in the Medicare FFS FY 2015 Dataset, and the Validation Sample included 30% of the orthopedic ASC procedures in the Medicare FFS FY 2015 Dataset.
Development Sample
-Dates: October 1, 2014 – September 30, 2015
-Number of facilities: 2,450 ASCs
-Number of orthopedic procedures: 132,865
-Demographic characteristics: average age of 73.1 years; 60.7% female
-Dataset used for: testing the patient-level risk-adjustment model (overfitting provided in this submission)
Validation Sample
-Dates: October 1, 2014 – September 30, 2015
-Number of facilities: 2,269 ASCs
-Number of orthopedic procedures: 56,941
-Demographic characteristics: average age of 73.1 years; 60% female
-Dataset used for: testing and validating the patient-level risk adjustment model (overfitting provided in this submission)
3. 2024 Endorsement Maintenance Dataset
Dates: January 1, 2021-December 31, 2022
Number of facilities (with at least one procedure): 2,961 (100%)
Number of patients: 479,525 (100%)
Male: 193,724 (40.4%)
Female : 285,801 (59.6%)
Dual eligible 9,472 (2.0%)
High ADI: 31,288 (6.5%)
Number of facilities with at least 35 procedures: 1,754 (59.2%)
Number of patients (within facilities >=35 procedures): 465,902 (97.2%)
Dataset use for: all updated measure testing, including reliability, empiric validity, updated risk model testing, testing for impact of social risk factors.
The number of measured entities differs by testing type and dataset. Please see details in section 4.1.2.
Patient characteristics differ by testing type and dataset. Please see details in section 4.1.2.
Reliability
We tested facility-level measure score reliability using the signal-to-noise method, using the formula presented by Adams and colleagues (Adams et al., 2010; Yu et al., 2013). Specifically, for each facility we calculate the reliability as:
Reliability=(σ_(facility-to-facility)^2)/(σ_(facility-to-facility)^2+ (σ_(facility error variance)^2)/n)
Where facility-to-facility variance is estimated from the hierarchical logistic regression model, n is equal to each facility’s observed case size, and the facility error variance is estimated using the variance of the logistic distribution (pi^2/3).
Signal-to-noise reliability scores can range from 0 to 1. A reliability of zero implies that all the variability in a measure is attributable to measurement error. A reliability of one implies that all the variability is attributable to real difference in performance.
We calculated the measure score reliability for all facilities, and for facilities with a volume cutoff of 35 procedures. Our rationale for this is described below.
In general, CMS sets the volume cutoff for publicly reporting facility measures scores based on two considerations. CMS considers the empiric results of reliability testing conducted on the dataset used for public reporting. CMS also considers the volume cutoff for score reporting used for related measures (for example, Facility 7-Day Risk-Standardized Hospital Visit Rate after Outpatient Colonoscopy, or CBE #3366, Hospital Visits after Urology Ambulatory Surgical Center Procedures, also under CBE review in this cycle). Regardless of the score reporting volume cutoff, all facilities and their cases are used in calculating the measure scores. The minimum sample size for public reporting is a policy choice that balances considerations such as the facility-level reliability testing results on the reporting data and consistency across measures for consumers.
References
Adams J, Mehrota, A, Thoman J, McGlynn, E. (2010). Physician cost profiling – reliability and risk of misclassification. NEJM, 362(11): 1014-1021.
Yu, H, Mehrota, A, Adams J. (2013). Reliability of utilization measures for primary care physician profiling. Healthcare, 1, 22-29.
Measure score reliability (signal-to-noise reliability) is shown in Table 3 (see attachment) for all facilities and facilities with at least 35 cases, and in Table 4 for facilities with at least 35 cases, the public reporting threshold described previously. For facilities with at least 35 procedures, signal-to-noise reliability ranged from 0.398-0.982 with a median of 0.759.
Using two years of performance data, the median facility-level reliability score is 0.759 (IQR, 0.605 - 0.869) for ASCs with at least 35 procedures (the public reporting threshold) representing moderate reliability.
At the current public reporting threshold, most facilities (about 75 percent of facilities with at least 35 cases) fall above the 0.6 minimum threshold stated in Battelle’s current CBE guidebook. If CMS were to increase the case volume minimum so that all facilities exceeded this threshold, it would remove publicly available information from about 25 percent of facilities that are currently publicly reported.
We believe that median reliability of 0.6 (signal to noise) is sufficiently high for a facility-level publicly reported measure in a pay for reporting program. Increasing the minimum case volume has the tradeoff of removing more than half of facilities (with at least one procedure) from reporting to the public.
| | Overall | Minimum | Decile_1 | Decile_2 | Decile_3 | Decile_4 | Decile_5 | Decile_6 | Decile_7 | Decile_8 | Decile_9 | Decile_10 | Maximum |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Reliability | 0.731 | 0.398 | 0.445 | 0.527 | 0.527 | 0.672 | 0.730 | 0.785 | 0.831 | 0.869 | 0.903 | 0.941 | 0.982 |
| Mean Performance Score | 2.21 | 2.15 | 2.22 | 2.19 | 2.19 | 2.22 | 2.23 | 2.24 | 2.20 | 2.19 | 2.19 | 2.17 | 1.62 |
| N of Entities | 1,754 | 4 | 175 | 176 | 176 | 175 | 173 | 180 | 174 | 174 | 177 | 175 | 1 |
| N of Persons / Encounters / Episodes | 465,902 | 140 | 7,454 | 10,455 | 14,261 | 19,043 | 24,863 | 34,940 | 45,576 | 61,468 | 88,313 | 159,529 | 2,898 |
Validity
For this endorsement maintenance submission, we provide evidence of validity through three approaches: face validity conducted during measure development, validation of the outcome, and empiric validity through associations with volume. Below we describe the methodology; the results are provided in the subsequent section.
Face validity
We assessed measure validity through systematic assessment of measure face validity by a technical expert panel (TEP) of national experts and stakeholder organizations.
To convene the TEP, we released a public call for nominations and selected individuals to represent a range of perspectives including clinicians, patients, and individuals with experience in quality improvement, performance measurement, and healthcare disparities. We held three structured TEP conference calls consisting of presentation of key issues, our proposed approach, and relevant data, followed by open discussion among TEP members.
The TEP, made up of 15 members including patient representatives, expert clinicians, methodologists, researchers, and providers, were asked to formally assess the measure’s face validity. We systematically assessed the face validity of the measure score as an indicator of quality by soliciting TEP members’ agreement with the following statement (via an online survey):
- “The risk-standardized hospital visit rates obtained from the ‘Hospital Visits after Orthopedic Ambulatory Surgical Center Procedures’ measure as specified are valid and useful measures of ASC orthopedic surgical quality of care.”
- “The risk-standardized hospital visit rates obtained from the ‘Hospital Visits after Orthopedic Ambulatory Surgical Center Procedures’ measure as specified will provide ASCs with information that can be used to improve their quality of care.”
The survey offered participants six response options ranging from “strongly disagree” to “strongly agree” on a six-point scale:
1=Strongly disagree
2=Moderately disagree
3=Somewhat disagree
4=Somewhat agree
5=Moderately agree
6=Strongly agree
List of TEP Members
1) Robin Blomberg, BA, MA – National Forum of End-Stage Renal Disease, Network 16 (Representative for Kidney Patient Advisory Council); Seattle, WA
2) Kirk Campbell, MD – New York University Hospital for Joint Diseases (Clinical Assistant Professor of Orthopedic Surgery); New York, NY
3) Gary Culbertson, MD, FACS – Iris Surgery Center (Surgeon; Medical Director); Sumter, SC
4) Martha Deed, PhD – Consumers Union Safe Patient Project (Patient Safety Advocate); Austin, TX
5) James Dupree, MD, MPH – University of Michigan (Urologist; Health Services Researcher); Ann Arbor, MI
6) Nester Esnaola, MD, MPH, MBA – Fox Chase Cancer Center (Professor of Surgery; Associate Director for Cancer Health Disparities and Community Engagement); Philadelphia, PA
7) John Gore, MD, MS – University of Washington (Associate Professor of Urology); Seattle, WA
8) Lisa Ishii, MD, MHS – Johns Hopkins School of Medicine (Associate Professor); American Academy of Otolaryngology-Head and Neck Surgery (Coordinator for Research and Quality); Baltimore, MD; Alexandria, VA
9) Atul Kamath, MD – Perelman School of Medicine, University of Pennsylvania (Assistant Professor and Clinical Educator Director of Orthopedic Surgery); Hospital of the University of Pennsylvania (Attending Surgeon); Philadelphia, PA
10) Tricia Meyer, PharmD, MS, FASHP – Scott & White Medical Center (Regional Director of Pharmacy); Texas A&M University College of Medicine (Associate Professor of Anesthesiology); Temple, TX
11) Linda Radach, BA – Consumers Union Safe Patient Project (Patient Safety Advocate); Austin, TX
12) Amita Rastogi, MD, MHA, CHE, MS – Health Care Incentives Improvement Institute (Chief Medical Officer); Newtown, CT
13) Donna Slosburg, RN, BSN, LHRM, CASC – ASC Quality Collaboration (Executive Director); St. Pete Beach, FL
14) Thomas Tsai, MD, MPH – Brigham and Women’s Hospital (General Surgeon); Harvard School of Public Health (Research Associate); Boston, MA
15) Katherine Wilson, RN, BA, MHA – AMSURG Corp (Vice President of Quality); Nashville, TN
Validation of the Outcome
The outcome of an unplanned hospital visit following an outpatient elective orthopedic surgery is intended to capture adverse events that occur as part of the care received before, during, and after the procedure. To validate the outcome, we identified the most commonly occurring principal discharge diagnosis codes associated with the post-procedure hospital visit. (For any hospitalization captured as part of the outcome, a claim for the hospital visit is submitted to CMS that indicates the main reason for the hospitalization; there is only one such main reason, called the “principal diagnosis code” that is used to capture this information.) Based on previous research to validate the outcome during measure development, we know that the most frequent reasons for a hospital visit following elective orthopedic surgery are complications from the procedure. We updated this analysis for this endorsement maintenance submission.
External Empiric Validity
One approach for assessing the validity of a quality measure is to show that performance on the test measure is associated with another quality measure in the same causal pathway. To do this, we needed to identify a comparator measure, however, as summarized below, we did not identify a suitable measure with currently publicly available data.
We first considered CMS’s two related CBE-endorsed measures, Facility-Level 7-Day Hospital Visits after General Surgery Procedures Performed at ASCs (ASC General Surgery), and the ASC Urology measure. The outcome of both measures is nearly identical to that of the ASC Orthopedic Surgery measure; an unplanned hospital visit is defined as an emergency department (ED) visit, observation stay, or unplanned inpatient admission. Hence, the measures target the same quality domains. The patient cohort is also somewhat similar in that the measures target Medicare Fee-For-Service (FFS) patients aged 65 years and older. The cohorts (in terms or procedures included in the measure) for the ASC General Surgery and ASC Urology measures do not, however, overlap with the ASC Urology measure. Furthermore, the clinicians performing the procedures across the different cohorts are unlikely to be the same individuals, and in addition, most ASCs (about 70% as of 2022) subspecialize (MedPAC, 2024).
Association with volume
Studies have shown that there is a volume outcome relationship between volume and some (but not all) outcomes for some orthopedic procedures (Clarify Health Institute, 2023; Levaillant et al., 2021). We therefore assessed if higher facility volume was associated with the ASC Orthopedic Surgery measure score. We hypothesized that there would be a weak to moderate, negative relationship between facility procedural volume and ASC Orthopedic Surgery measure scores, with higher procedural volumes being associated with better (lower) ASC Orthopedic Surgery measure scores. We limited this analysis to facilities with at least 200 procedures based on literature suggesting this relationship would be evident at higher volumes (Kugler et al., 2022).
References
Clarify Health Institute. (2023). High-Volume Orthopedic Surgeons Generate Better Outcomes at Lower Costs. Retrieved from https://clarifyhealth.com/insights/institute/briefs/pump-up-the-volume/
Kugler, C. M., Goossen, K., Rombey, T., De Santis, K. K., Mathes, T., Breuing, J., Hess, S., Burchard, R., & Pieper, D. (2022). Hospital volume–outcome relationship in total knee arthroplasty: A systematic review and dose–response meta-analysis. Knee Surgery, Sports Traumatology, Arthroscopy, 30(8), 2862–2877. https://doi.org/10.1007/s00167-021-06692-8
Levaillant, M., Marcilly, R., Levaillant, L., et al. (2021). Assessing the hospital volume-outcome relationship in surgery: A scoping review. BMC Medical Research Methodology, 21(204). https://doi.org/10.1186/s12874-021-01396-6
MedPAC’s Report to Congress, Chapter 5, Ambulatory Surgical Center Services, March 2024. https://www.medpac.gov/wpcontent/uploads/2024/03/Mar24_MedPAC_Report_To_Congress_SEC.pdf
Face Validity as Determined by the Technical Expert Panel (TEP)
Validity was assessed by the TEP. The TEP provided input on the cohort, risk model, and outcome to strengthen the measure; the TEP supported the final measure specifications with high agreement. A total of 14 TEP members completed the face validity survey. 13 of the 14 TEP members responded to the survey of face validity.
Of the 13 respondents:
-12 TEP members (92.3%) indicated that they agreed (6 strongly agreed, 5 moderately agreed, and 1 somewhat agreed), and 1 TEP member moderately disagreed, with the following statement: “The risk-standardized hospital visit rates obtained from the ‘Hospital Visits after Orthopedic Ambulatory Surgical Center Procedures’ measure as specified are valid and useful measures of ASC orthopedic surgical quality of care.”
-12 TEP members (92.3%) indicated that they agreed (6 strongly agreed, 5 moderately agreed, and 1 somewhat agreed), and 1 TEP member moderately disagreed, with the following statement: “The risk-standardized hospital visit rates obtained from the ‘Hospital Visits after Orthopedic Ambulatory Surgical Center Procedures’ measure as specified will provide ASCs with information that can be used to improve their quality of care.”
These validity testing results demonstrate TEP agreement with the overall face validity of the measure.
Validation of the Outcome
Table 5 (see attachment) shows the most frequent (the top 25) principal diagnosis codes associated with the outcome (unplanned hospital visits within 7 days of a qualifying procedure); many of these codes indicate a complication from the procedure. For example, the top four principal diagnoses are “other acute postprocedural pain,” “syncope and collapse,” “retention of urine, unspecified,” and “constipation, unspecified”. Many of the complications in Table 5 are preventable, and/or could be managed in a non-acute care setting.
External Empiric Validity
As noted above, we did not identify any measures that were suitable for comparison with the ASC Orthopedic Surgery measure, therefore we examined the association of the measure score with facility volume. Pearson’s correlation coefficient for the association between measure scores and facility procedural was -0.133 (p< .001) or for facilities with at least 200 procedures.
Our results, shown above, demonstrate validity of the ASC Orthopedic Surgery measure at several levels: face validity as assessed by experts (with over 92 percent agreement); validity of the outcome, shown by analysis of ICD-10 codes associated with the outcome, and validity of the measure score as shown by association (in the expected direction) with procedural volume.
Risk Adjustment
Our approach to risk adjustment is tailored to, and appropriate for, a publicly reported outcome measure as articulated in published scientific guidelines (Krumholz et al., 2006; Normand & Shahian, 2007). For example, we only adjust for risk factors that are present at the start of care. We do not risk adjust for conditions that are possible adverse events of care and that are only recorded at the time of the surgery (see Data Dictionary, Tab 5, “ASC_Ortho_CoC CCs”). We do not adjust for factors related to the delivery of care that may reflect care quality.
The measure employs a hierarchical logistic regression model (a form of hierarchical generalized linear model [HGLM]) to create an ASC-level 7-day RSHVR. This approach to modeling appropriately accounts for the structure of the data (patients clustered within facilities), the underlying risk due to patients’ procedures/comorbidities, and sample size at a given ASC when estimating hospital visit rates. In brief, the approach simultaneously models two levels (patient and facility) to account for the variance in patient outcomes within and between facilities (Normand & Shahian, 2007). At the patient level, the model adjusts the log-odds of hospital visits within 7 days after the procedure for selected demographic, clinical, and procedure risk variables. The second level models the facility-specific intercepts as arising from a normal distribution. The facility intercept, or facility-specific effect, represents the ASC contribution to the risk of 7-day hospital visits, after accounting for patient risk and sample size, and can be inferred as a measure of quality. If there were no differences among ASCs, then after adjusting for patient risk, the facility intercepts would be identical across all ASCs.
Candidate Risk-Adjustment Variables
The measure adjusts for differences across facilities in patient demographic and clinical factors and in procedure-related risk. Potential candidate risk factors were identified from related quality measures and the literature; a preliminary list of risk factors was developed and then revised based on TEP and expert clinical input.
To define the candidate risk factors, during measure development we defined the clinical risk factors in claims data using Version 22 of the CCs from CMS’s Hierarchical Condition Categories (HCC) grouper, which classified thousands of diagnosis codes into clinically coherent and mutually exclusive groups of codes, or condition categories (Pope et al., 2000). In some cases (for example, morbid obesity), individual codes were used to define the risk factor. The measure does not apply the hierarchical logic of the HCC. Based on prior validation work conducted for similar measures, we have confidence that model variables defined using the CCs are reasonable proxies for clinical conditions. CMS has validated similar risk-adjustment models that use the CCs against models that use chart-abstracted data for risk adjustment. Note that the measure was later re-specified for use in ICD-10.
To address surgical procedural complexity, we used the work RVUs of the procedure, an approach employed by the American College of Surgeons National Surgical Quality Improvement Program (Raval et al., 2010).
We then reviewed the list of candidate risk variables with TEP members. None of the clinical experts suggested removing any of the candidate risk factors from the list. One of the TEP members suggested considering additional clinical risk factors that the American Association of Hip and Knee Surgeons (AAHKS) has recommended for risk adjusting inpatient hip and knee arthroplasty outcome measures, including smoking, chronic anticoagulant use, previous intra-articular infection, congenital hip deformity, angular knee deformity greater than 15 degrees, and previous open reduction and internal fixation of hip and knee. The AAHKS list also included workers’ compensation status; however, we did not consider it. Although it may be correlated with the outcome, the relationship is affected by a number of factors that we may not want to adjust for in this quality measure, including variation in eligibility for workers’ compensation by state.
Finally, to consolidate similar risk factors, we checked the bivariate direction and strength of association of the individual risk factors defined by CCs or ICD-10 codes and then combined risk factor diagnoses into clinically coherent comorbidity variables. For example, a “cancer” variable was created that combined several individual cancer diagnoses.
Variable Selection
To select the final set of variables to include in the risk-adjustment model, all risk variables were entered into logistic regression analyses predicting the outcome of hospital visits within 7 days in the Development Sample. To develop a parsimonious risk model, non-significant variables (at the 0.05 level) were iteratively removed from the model using a stepwise purposeful selection approach described by Hosmer and Lemeshow (Hosmer & Lemeshow, 2000). Our goal was to minimize the number of variables in the model while preserving model performance (as measured by the c-statistic). All variables significant at p<0.05 were retained in the final model. In addition, we retained in the model two variables (tobacco use disorder and morbid obesity) because experts advised that these were important risk predictors and expressed a strong preference for including them in the model.
Social Risk Factors for Supplemental Disparities Analyses
We selected variables representing social risk factors based on a review of literature, conceptual pathways, and feasibility. Below, we describe the pathways by which social risk factors may influence risk of hospital visits following outpatient surgical procedures.
Our conceptualization of the pathways by which patients’ social risk factors may affect the outcome is informed by the literature (Bhattacharyya, 2015; Jha, Orav, & Epstein, 2011; Menachemi et al., 2007; Reames et al., 2014; Skinner et al., 2005; Trivedi et al., 2014) and IMPACT Act–funded work by the National Academy of Science, Engineering and Medicine (NASEM) and the Department of Health and Human Services Assistant Secretary for Policy and Evaluation (ASPE) (Department of Health and Human Services, 2016; National Academies of Sciences, Engineering, and Medicine, 2016a; National Academies of Sciences, Engineering, and Medicine, 2016b).
Literature Review: Social Risk Variables and Ambulatory Surgery Post-Procedure Hospital Visits
To examine the relationship between social risk factors and risk of hospital visits following outpatient surgical procedures, a literature search was performed with the following exclusion criteria: non-English language articles, articles published prior to 2012,articles without primary data, articles focused on pediatric patient population, and articles not explicitly focused on social risk factors and hospital visits after ambulatory surgery. A total of 176 studies were reviewed by title and abstract, and all but two studies were excluded from full-text review based on the above criteria. The two studies indicated that Black and Hispanic patients and patients from lower-income households were at increased risk of post-procedure hospital visits in the ambulatory surgery setting (Bhattacharyya, 2015; Hosmer & Lemeshow, 2000). No studies were found that suggested that variation in patients’ social risk affected variation in outcome risk across facilities performing ambulatory surgical procedures. A 2024 update to the literature search (going back 5 years) did not identify any new studies that focused on ASCs or orthopedic procedures, but we find additional evidence for the impact of social risk factors on outcomes for patients undergoing surgery in general (Azap et al., 2020; Ghirimoldi et al., 2021; Paro et al., 2021). For example, a 2019 study found that while patients with low income undergoing colectomy had higher rates of surgical-site infections compared with higher-income patients, there was no difference in surgical-site infection rates based on income for patients undergoing hysterectomy (Qi et al., 2019). A 2023 study in cancer patients undergoing surgery found that patients with psychosocial risk factors were more likely to experience complications following surgery (Leeds et al., 2019). Finally, a 2021 study found that for some procedures, people living in counties with high social vulnerability (SVI) were more likely to experience complications compared with patients who live in low SVI counties (Diaz et al., 2021).
We note that compared to the patient mix for elective outpatient procedures at hospital outpatient departments (HOPDs), ASCs serve a low proportion of patients with social risk factors therefore there are disparities in access to care at ASCs, with Black patients and patients with public insurance being less likely to receive care at an ASC compared with others without those social risk factors (Janeway et al., 2020). A recent study found that there are disparities in the geographic distribution of ASCs, with counties with higher socioeconomic status having more ASCs per capita compared with counties with lower socioeconomic status (Chatterjee, Amen, & Khormaee, 2022).
Conceptual Pathways for Social Risk Factors
Although there is limited literature linking social risk factors and adverse outcomes for ambulatory surgery performed at ASCs, potential pathways may include:
- Differential care within an ASC. One pathway by which social risk factors may contribute to hospital visit risk is that patients may not receive equivalent care within a facility. For example, in the hospital setting, Black patients may experience differential, lower quality, or discriminatory care within a given facility (Menachemi et al., 2007). Alternatively, patients with social risk factors, such as lower education, may require differentiated care – e.g., provision of information at a lower health literacy level – that they do not receive.
- Use of lower-quality facilities. Patients of lower income, lower education, or unstable housing may not have equitable access to high-quality facilities because such facilities are less likely to be found in geographic areas with large populations of poor patients; thus, patients with low income may be more likely to be seen in lower-quality facilities, which can contribute to increased risk of post-procedure adverse outcomes (Jha, Orav, & Epstein, 2011; Trivedi et al., 2014). Similarly, Black patients have been shown to have less access to high-quality facilities compared with white patients (Reames et al., 2014). However, as described above, patients with social risk factors are less likely to have access to care at ASCs, in general (Chatterjee, Amen, & Khormaee, 2022; Janeway et al., 2020).
- Influence of social risk factors on hospital visit risk outside of ASC quality. Some social risk factors, such as income or wealth, may affect the likelihood of post-procedure hospital visits without directly being associated with the quality of care received at the ASC. For instance, while an ASC may make appropriate care decisions and provide tailored care and education, a lower-income patient may have a worse outcome post-procedure due to competing economic priorities or a lack of access to care outside of the facility.
We developed and used the conceptual framework described below to identify potential social risk factors. We analyzed two well-studied social risk factors that could best be operationalized in data, outlined below. We note that this measure already adjusts for age and note that the risk model already accounts for patient comorbidities which may differ among patients with social risk factors.
1. Medicare-Medicaid dual-eligibility status
Dual eligibility (DE) for Medicare and Medicaid is available at the patient level in the Medicare Master Beneficiary Summary File. The eligibility threshold for over 65-year-old Medicare patients considers both income and assets. For the dual-eligible indicator, there is a body of literature demonstrating differential health care and health outcomes among beneficiaries, therefore the DE indicator allows us to examine some of the pathways of interest (Department of Health and Human Services, 2016).
2. Area Deprivation index (ADI)
The ADI, initially developed by Health Resources & Services Administration (HRSA), is based 17 measures across four domains: income, education, employment, and housing quality (Kind et al., 2018; Singh, 2003).
The 17 components are listed below:
Population aged ≥ 25 y with < 9 y of education, %
Population aged ≥ 25 y with at least a high school diploma, %
Employed persons aged ≥ 16 y in white-collar occupations, %
Median family income, $
Income disparity
Median home value, $
Median gross rent, $
Median monthly mortgage, $
Owner-occupied housing units, % (home ownership rate)
Civilian labor force population aged ≥16 y unemployed, % (unemployment rate)
Families below poverty level, %
Population below 150% of the poverty threshold, %
Single-parent households with children aged < 18 y, %
Households without a motor vehicle, %
Households without a telephone, %
Occupied housing units without complete plumbing, % (log)
Households with more than 1 person per room, % (crowding)
ADI scores were derived using beneficiary’s 9-digit ZIP Code of residence, which is obtained from the Master Beneficiary Summary File, and is linked to 2017-2021 US Census/American Community Survey (ACS) data. In accordance with the ADI developers’ methodology, an ADI score is calculated for the census block group corresponding to the beneficiary’s 9-digit ZIP Code using 17 weighted Census indicators. Raw ADI scores were then transformed into a national percentile ranking ranging from 1 to 100, with lower scores indicating lower levels of disadvantage and higher scores indicating higher levels of disadvantage. Percentile thresholds established by the ADI developers were then applied to ADI percentile to dichotomize neighborhoods into more disadvantaged (high ADI areas=ranking equal to or greater than 85) or less disadvantaged areas (Low ADI areas= ranking of less than 85).
References
Azap, R. A., Paredes, A. Z., Diaz, A., Hyer, J. M., & Pawlik, T. M. (2020). The association of neighborhood social vulnerability with surgical textbook outcomes among patients undergoing hepatopancreatic surgery. Surgery, 168(5), 868-875.
Bhattacharyya, N. (2015). Healthcare disparities in revisits for complications after adult tonsillectomy. American Journal of Otolaryngology, 36(2), 249-253.
Chatterjee, A., Amen, T. B., & Khormaee, S. (2022). Trends in geographic disparities in access to ambulatory surgery centers in New York, 2010 to 2018. JAMA Health Forum, 3(10), Article e223608. https://doi.org/10.1001/jamahealthforum.2022.3608
Department of Health and Human Services, Office of the Assistant Secretary of Planning and Evaluation. (2016). Report to Congress: Social risk factors and performance under Medicare’s value-based payment programs. Retrieved from https://aspe.hhs.gov/pdf-report/report-congress-social-risk-factors-and-performance-under-medicares-value-based-purchasing-programs
Diaz, A., Hyer, J. M., Barmash, E., Azap, R., Paredes, A. Z., & Pawlik, T. M. (2021). County-level social vulnerability is associated with worse surgical outcomes especially among minority patients. Annals of Surgery, 274(6), 881–891. https://doi.org/10.1097/SLA.0000000000004691
Forefront Group. (2023). The Area Deprivation Index is the most scientifically validated social exposome tool available for policies advancing health equity. https://doi.org/10.1377/forefront.20230714.676093Ghirimoldi, F. M., Schmidt, S., Simon, R. C.,
Hosmer, D. W., & Lemeshow, S. (2000). Applied Logistic Regression. New York: Wiley.
Janeway, M. G., Sanchez, S. E., Chen, Q., Nofal, M. R., Wang, N., Rosen, A., & Dechert, T. A. (2020). Association of race, health insurance status, and household income with location and outcomes of ambulatory surgery among adult patients in 2 US states. JAMA Surgery. https://doi.org/10.1001/jamasurg.2020.3318
Jha, A. K., Orav, E. J., & Epstein, A. M. (2011). Low-quality, high-cost hospitals, mainly in South, care for sharply higher shares of elderly black, Hispanic, and Medicaid patients. Health Affairs, 30, 1904-1911.
Krumholz, H. M., Brindis, R. G., Brush, J. E., et al. (2006). Standards for statistical models used for public reporting of health outcomes: An American Heart Association scientific statement from the Quality of Care and Outcomes Research Interdisciplinary Writing Group: cosponsored by the Council on Epidemiology and Prevention and the Stroke Council endorsed by the American College of Cardiology Foundation. Circulation, 113(3), 456-462.
Leeds, I. L., Meyers, P. M., Zachary Obinna Enumah, He, J., Burkhart, R. A., Haut, E. R., Efron, J. E., & Johnston, F. M. (2019). Psychosocial risks are independently associated with cancer surgery outcomes in medically comorbid patients. Annals of Surgical Oncology, 26(4), 936–944. https://doi.org/10.1245/s10434-018-07136-3
Menachemi, N., Chukmaitov, A., Brown, L. S., et al. (2007). Quality of care differs by patient characteristics: outcome disparities after ambulatory surgical procedures. American Journal of Medical Quality, 22(6), 395-401.
National Academies of Sciences, Engineering, and Medicine (NASEM); (2016a). Accounting for Social Risk Factors in Medicare Payment: Identifying Social Risk Factors. Washington DC: National Academies Press.
National Academies of Sciences, Engineering, and Medicine (NASEM); (2016b). Accounting for Social Risk Factors in Medicare Payment: Data. Washington DC: National Academies Press.
Normand, S.-L. T., & Shahian, D. M. (2007). Statistical and clinical aspects of hospital outcomes profiling. Statistical Science, 22(2), 206-226.
Paro, A., Hyer, J. M., Diaz, A., Tsilimigras, D. I., & Pawlik, T. M. (2021). Profiles in social vulnerability: the association of social determinants of health with postoperative surgical outcomes. Surgery, 170(6), 1777-1784.
Pope, G., et al. (2000). Principal inpatient diagnostic cost group models for Medicare risk adjustment. Health Care Financing Review, 21(3), 26.
Qi, A. C., Peacock, K., Luke, A. A., Barker, A., Olsen, M. A., & Joynt Maddox, K. E. (2019). Associations between social risk factors and surgical site infections after colectomy and abdominal hysterectomy. JAMA Network Open, 2(10), Article e1912339. https://doi.org/10.1001/jamanetworkopen.2019.12339
Raval, M. V., Cohen, M. E., Ingraham, A. M., et al. (2010). Improving American College of Surgeons National Surgical Quality Improvement Program risk adjustment: Incorporation of a novel procedure risk score. Journal of the American College of Surgeons, 211(6), 715-723.
Reames, B. N., Birkmeyer, N. J., Dimick, J. B., et al. (2014). Socioeconomic disparities in mortality after cancer surgery: Failure to rescue. JAMA Surgery, 149, 475-481.
Singh, G. K. (2003). Area deprivation and widening inequalities in US mortality, 1969–1998. American Journal of Public Health, 93(7), 1137–1143. https://doi.org/10.2105/ajph.93.7.1137
Skinner, J., Chandra, A., Staiger, D., et al. (2005). Mortality after acute myocardial infarction in hospitals that disproportionately treat black patients. Circulation, 112, 2634-2641.
Trivedi, A. N., Nsa, W., Hausmann, L. R., et al. (2014). Quality and equity of care in U.S. hospitals. New England Journal of Medicine, 371, 2298-2308.
Wang, C. P., Wang, Z., Brimhall, B. B., ... & Shireman, P. K. (2021). Association of socioeconomic area deprivation index with hospital readmissions after colon and rectal surgery. Journal of Gastrointestinal Surgery, 25(3), 795-808.
The measure uses a two-level hierarchical logistic regression model to estimate facility-level RSHVRs. This approach accounts for the clustering of patients within ASCs and variation in sample size across ASCs.
The risk-adjustment model includes 29 variables (age, 27 comorbidity variables, and one surgical complexity variable). Work RVUs are assigned to each Current Procedural Terminology (CPT®) procedure code and approximate surgical procedural complexity by incorporating elements of physician time and effort. For patients with multiple concurrent CPT procedure codes, we risk adjust for the CPT code with the highest work RVU value. With the exception of morbid obesity and tobacco use disorder, which we define using individual (ICD-10) diagnosis codes, we define comorbidity variables using CMS Condition Categories (CCs), which are clinically meaningful groupings of thousands of ICD-10 diagnosis codes.
Certain CCs are considered possible complications of care, and thus are not risk-adjusted for if they are coded only at the time of surgery. Please see the attached Data Dictionary for CCs that are considered possible complications of care and are not risk-adjusted for if they occur only at the surgery (Tab 5: “ASC_Ortho_CoC CCs”)
Table 6 in the attachment includes the risk variable frequencies for each risk variable in the final risk model.
Table 7 in the attachment shows the distribution of social risk factors identified in the conceptual model. The facility median proportion of patients with the DE and high ADI variables is 1% and 2%, respectively; the median count of patients is 1 for each variable.
As described previously, during risk variable selection, we iteratively removed non-significant variables from the initial model using a step-wise purposeful selection approach until only statistically significant (p<0.05), assessed using a likelihood ratio test) variables remained in the model. In addition, we retained in the model two variables (tobacco use disorder and morbid obesity) because experts advised that these were important risk predictors and expressed a strong preference for including them in the model. The final risk variables in the model are listed below, and are also shown in the data dictionary, and the attachment (Tables 6 and 8) that include risk variable frequencies and odds ratios.
Final Model Variables:
1. Age
2. Work Relative Value Units
3. Tobacco use disorder
4. Morbid obesity (CC 22)
5. Chronic ulcers (CC 157, 158, 159, 160, 161)
6. Minor symptoms, signs, findings (CC 179)
7. Major symptoms, abnormalities (CC 178)
8. Psychiatric disorders (CC 57, 58, 59, 60, 61, 62, 63)
9. Dementia (CC 51, 52, 53)
10. Multiple sclerosis (CC 77)
11. Major traumatic fracture or internal injury (CC 170, 171, 172)
12. Other gastrointestinal disorders (CC 38)
13. Chronic lung disease (CC 111, 112, 113)
14. Bone/joint/muscle infections/necrosis (CC 39)
15. Cancer (CC 8, 9, 10, 11, 12, 13, 14)
16. Congestive heart failure (CC 85)
17. Chronic anticoagulant use
18. Chronic renal disease (CC 132 , 134, 135, 136, 137, 138, 139, 140)
19. Disorders of fluid/electrolyte/acid-base (CC 23, 24)
20. Hypertension and hypertensive disease (CC 94, 95)
21. Ischemic heart disease (CC 86, 87, 88, 89)
22. Other respiratory disorders (CC 118)
23. Pneumonia (CC 114, 115, 116)
24. Head injury (CC 166, 167, 168)
25. Rheumatoid and osteoarthritis (CC 40, 41, 42)
26. Seizure disorders and convulsions (CC 79)
27. Stroke (CC 99, 100)
28.Vascular disease (CC 106, 107, 108, 109)
29. Opioid abuse
Risk Model Calibration and Discrimination
CORE’s measures undergo an annual measure reevaluation process, which ensures that the risk-standardized models are continually assessed and remain valid, given possible changes in clinical practice and coding standards over time. Modifications made to measure cohorts, risk models, and outcomes are informed by a review of the most recent literature related to measure conditions or outcomes, feedback from stakeholders, and empirical analyses, including assessment of coding trends that reveal shifts in clinical practice or billing patterns. Input is solicited from a workgroup composed of up to 20 clinical and measure experts, inclusive of internal and external consultants and subcontractors.
To assess model performance, we computed three summary statistics for the ASC Orthopedic Surgery measure: two discrimination statistics (the C-statistic, predictive ability) and one calibration statistic (overfitting) (Harrell et al, 2001). In addition, we provide risk-decile plots.
Discrimination Statistics
(1) Area under the receiver operating characteristic (ROC) curve (c-statistic)
The c-statistic is the probability that predicting the outcome is better than chance, which is a measure of how accurately a statistical model is able to distinguish between a patient with and without an outcome.
C-statistic (2024 EM dataset): 0.675
The c-statistic of 0.675 indicates good model discrimination.
(2) Predictive ability
Discrimination in predictive ability measures the ability to distinguish high-risk subjects from low-risk subjects; therefore, for a model with good predictive ability we would expect to see a wide range in hospital visit ratios between the lowest decile and highest decile. To calculate the predictive ability, we calculated the range of observed hospital visit ratios between the lowest and highest predicted deciles.
Predictive Ability, % (lowest decile - highest decile): 0.77-5.59
The model continues to show an acceptable range between the lowest decile and highest decile, indicating the ability to distinguish high-risk subjects from low-risk subjects.
Calibration Statistics (from original measure development)
(3) Over-fitting indices
Over-fitting refers to the phenomenon in which a model accurately describes the relationship
between predictive variables and outcomes in the development dataset but fails to provide valid predictions in new patients. Estimated calibration values of γ0 far from 0 and estimated values of γ1 far from 1 provide evidence of over-fitting. We used Dataset #1 for this analysis. Our results, shown below, show a calibration value of close to 0 at one end and close to 1 to the other end indicating good calibration of the model.
Development Sample results:
Calibration: (0, 1)
FY 2015 Validation Sample results:
Calibration: (-0.04, 0.99)
CORE notes that after initial measure development we do not re-test our risk models for overfitting using a dataset that is external to the testing sample. In our risk models, coefficients are updated each time the measure is calculated; we refit the model with new data each time the measure is calculated. Therefore, random statistical fluctuations in model coefficients across repeated reporting cycles are part of the overall random error in the facility performance estimates.
Risk Decile Plots
We provide updated risk decile plots for all patients, and for patients with DE and high ADI in Figures 4 (see attachment).
Higher deciles of the predicted outcomes are associated with higher observed outcomes, which continue to show good calibration of the model. The risk decile plot indicates continued good discrimination of the model and good predictive ability, for all patients, and for patients with DE and high ADI, separately.
We describe the approach to risk variable selection in Section 4.4.2, and analysis and rationale for not including social risk factors in the final model in Section 5.1. In this section we provide the interpretation of the risk model testing results described in section 4.4.5.
The following results demonstrate that the risk-adjustment model adequately controls for differences in patient characteristics:
Discrimination Statistics
The calculated c-statistic was 0.675, which indicates good model discrimination. The model also predicted a wide range between the lowest decile and highest decile, indicating the ability to distinguish high-risk subjects from low-risk subjects.
Calibration Statistics
The calibration value which was consistently close to 0 at one end and close to 1 to the other, indicating good calibration of the model.
Risk Decile Plot
Higher deciles of the predicted outcomes are associated with higher observed outcomes, which show a good calibration of the model.
Overall Interpretation
Interpreted together with information provided in the aforementioned sections, our diagnostic results demonstrate the risk-adjustment model adequately controls for differences in patient characteristics (case mix).
Use & Usability
Use
The level of analysis is facility, and the care setting is ambulatory surgery center.
Usability
While ASCs continue to show promising results of reduced complications post-surgery in comparison to hospital settings, the possibility of an unplanned hospital visit due to adverse events post-surgery still exists for patients (Munnich et. al., 2014; Schwartz, 2020). These complications can include surgical site infections, postoperative pain management issues, and other complications, as shown in the literature and in the empirical results shared with this submission (Owens et. al., 2014; Erhun et. al., 2016; Rajput et. al., 2021). This measure provides the opportunity for improvement in reducing unplanned visits due to complications after surgery in the ASC setting.
Evidence-based interventions focusing on preoperative assessment, enhancing postoperative monitoring, and improving patient education can further reduce complications and rates of unplanned hospital visits (Erhun et. al., 2016; York et. al., 2019). For example, ASCs can opt to provide patient education pre- and post-procedure to facilitate the recovery process and reduce complications. Patient education improves transparency while allowing providers to clarify patient responsibilities and expectations. Continued and systemic monitoring of symptoms after surgery using validated assessments can lead to early detection of symptoms by providers post-surgery and can help in planning a course of action before complications exacerbate (Girish, 2021). A 2020 pilot study tested an electronic postoperative symptom-tracking platform to determine its clinical usefulness in the first week after minimally invasive ambulatory surgery. Responses above a defined threshold on the symptom instrument triggered an alert to the healthcare provider. The authors found that for majority of the patients presenting symptoms, a simple consultation phone call and adjustment of medications was sufficient in controlling complications (Zivanovic et. al., 2022). The adoption of evidence-based quality improvement systems in ASC settings can overall advance patient care and sustain a culture of improvement among facilities. Employing a methodological system of quality improvement focused on improving certain performance domains of quality provides guidelines for ASCs to follow and offsets the burden from clinical staff to develop their own plans for quality control (Rakover 2020, et. al).
Finally, specific to this measure, CMS shares several reports with ASCs to support quality improvement, including:
- Facility- Specific Reports (FSRs): These reports include measure results benchmarked against the state and the nation. For the reporting period covered for this data submission, facilities received one FSR, released in fall of 2023 on procedures performed January 1, 2021, through December 31, 2022. (Click here to view a mock FSR.)
- Claims-Detail Reports (CDRs): These reports provide claim-level details (including the principal diagnosis associated with the post-procedure visit) for each claim that meets numerator and denominator criteria. CDRs provide facilities with an opportunity to improve the quality of care provided to patients undergoing procedures prior to final measure calculation and public reporting of measure results. For this reporting period, covered for this data, facilities received two reports (one in September 2022, and one in March 2023). (Click here to view a mock CDR.)
References
Munnich EL, Parente ST. Procedures take less time at ambulatory surgery centers, keeping costs down and ability to meet demand up. Health Affairs. 2014 May 1;33(5):764-9.
Schwartz JC. Mastectomy and prepectoral reconstruction in an ambulatory surgery center reduces major infectious complication rates. Plastic and Reconstructive Surgery–Global Open. 2020 Jul 1;8(7):e2960.
Owens PL, Barrett ML, Raetzman S, Maggard-Gibbons M, Steiner CA. Surgical site infections following ambulatory surgery procedures. Jama. 2014 Feb 19;311(7):709-16.
Erhun F, Malcolm E, Kalani M, Brayton K, Nguyen C, Asch SM, Platchek T, Milstein A. Opportunities to improve the value of outpatient surgical care. Am J Manag Care. 2016 Sep 1;22(9):e329-35. PMID: 27662397.
Rajput K, Vadivelu N, Kaye AD, Shah RV, editors. Pain Control in Ambulatory Surgery Centers. Springer International Publishing; 2021 Mar.
York PJ, Gang CH, Qureshi SA. Patient education in an ambulatory surgical center setting. J Spine Surg. 2019 Sep;5(Suppl 2):S206-S211. doi: 10.21037/jss.2019.04.07. PMID: 31656877; PMCID: PMC6790809.
Joshi GP. Putting patients first: ambulatory surgery facilitates patient-centered care. Current Opinion in Anesthesiology. 2021 Dec 1;34(6):667-71.
Zivanovic O, Chen LY, Vickers A, Straubhar A, Baser R, Veith M, Aiken N, Carter J, Curran K, Simon B, Mueller J, Jewell E, Chi DS, Sonoda Y, Abu-Rustum NR, Leitao MM Jr. Electronic patient-reported symptom monitoring in patients recovering from ambulatory minimally invasive gynecologic surgery: A prospective pilot study. Gynecol Oncol. 2020 Oct;159(1):187-194. doi: 10.1016/j.ygyno.2020.07.004. Epub 2020 Jul 24. PMID: 32718730; PMCID: PMC7380930.
Rakover J, Little K, Scoville R, Holder B. Implementing daily management systems to support sustained quality improvement in ambulatory surgery centers. AORN journal. 2020 Apr;111(4):415-22.
Stakeholders can submit questions and issues to CMS through an online tool (Q&A tool) available to the public on QualityNet. CMS responds to each question submitted by stakeholders. During the time between the last maintenance review and re-endorsement of this measure (2020), we have received no major reports of issues with this measure. Through the Q&A tool, stakeholders have asked for assistance with their questions including: interpreting their facility’s patient-level data, understanding measure specifications (inclusion, exclusion, risk adjustment), interpretation/clarification of results.
Each year, we review and consider issues raised through the Q&A or in the literature related to this measure, and those issues are considered by measure and clinical experts. Any issues that warrant additional analytic work due to potential changes in the measure specifications are addressed as a part of annual measure reevaluation. If small changes are indicated after additional analytic work is complete, those changes are usually incorporated into the measure in the next measurement period. If the changes are substantial CMS may propose the changes through rulemaking and adopt the changes only after CMS received public comment on the changes and finalizes those changes in rulemaking.
Each year we also review and consider changes to HCPCS and ICD-10 codes that are then incorporated into the measure. Those code set files are made available to the public on QualityNet.
During the time between the last maintenance submission and the measure’s re-endorsement in 2020 and this current submission we have made no major changes to the ASC Orthopedic Surgery measure based on stakeholder feedback.
This measure captures an ever-changing mix of procedures that have been added (or removed) from the ASC-covered procedure list, which is the basis for inclusion of procedures for this measure. For example, Total Knee Arthroplasty (TKA) was removed from the CMS inpatient-only (IPO) list in January 2018, allowing TKAs to be performed in the inpatient or outpatient setting; TKA was added to the Ambulatory Surgery Center (ASC) Covered Procedures List in January 2020 and THA was added in January 2021. Outpatient volumes of THA/TKA procedures have been steadily increasing; after the onset of COVID, the proportion of THA/TKA procedures performed (for Medicare FFS patients) in the outpatient setting exceeded those performed in the inpatient setting (data not shown); other than this ASC Orthopedic Surgery measure, there are currently no active/implemented outpatient performance measures that capture complications following THA/TKA procedures performed in the ASC setting. (CMS has, however, finalized the adoption of the THA/TKA PRO-PM for the HOPD and ASC settings (Federal Register, November 2023).
There have been no unexpected findings during implementation.
Comments
Staff Preliminary Assessment
CBE #3470 Staff Assessment
Importance
Strengths:
- The developer posits that this measure provides an opportunity for improvement in reducing unplanned hospital visits due to complications after orthopedic surgery in the ambulatory surgical center setting.
The developer cites evidence from the primary literature interventions that ASCs can implement to improve hospitalizations post-surgery, such as pre-operative assessments, enhancing post-operative monitoring, and improving patient education. The developer further suggests facilities can implement a system of quality improvement and use CMS facility-based and claims-based reports for quality improvement efforts.
Using data from Jan 1, 2021 through Dec 31, 2022, the developer reported a distribution of scores with an overall score of 2.20 (mean facility-level rate) and a decile range of 1.88 – 2.62. For this measure, a lower score is better. The distribution shows ~30% of ASCs are higher than the facility-level mean.
The developer cites evidence, in which patients expressed concern over post-operative complications. The developer argues that due to the increase in procedures in ASCs there is a need for this measure to inform ASC providers and providers of the outcome of hospital visits. Further, by providing reports of this measure to providers will incentivize quality improvement efforts.
Limitations:
- None identified.
Rationale:
- The developer posits that this measure provides an opportunity for improvement in reducing unplanned hospital visits due to complications after orthopedic surgery in the ambulatory surgical center setting.
The developer cites evidence from the primary literature interventions that ASCs can implement to improve hospitalizations post-surgery, such as pre-operative assessments, enhancing post-operative monitoring, and improving patient education. The developer further suggests facilities can implement a system of quality improvement and use CMS facility-based and claims-based reports for quality improvement efforts.
Using data from Jan 1, 2021 through Dec 31, 2022, the developer reported a distribution of scores with an overall score of 2.20 (mean facility level rate) and a decile range of 1.88 – 2.62. For this measure, a lower score is better. The distribution shows ~30% of ASCs are higher than the national average
The developer cites evidence, in which patients expressed concern over post-operative complications. The developer argues that due to the increase in procedures in ASCs there is a need for this measure to inform ASC providers and providers of the outcome of hospital visits. Further, by providing reports of this measure to providers will incentivize quality improvement efforts.
Feasibility Acceptance
Strengths:
- The developer did not conduct a feasibility assessment of an assessment of missing data for this measure, as it used a 100% sample of paid Medicare claims. The developer further states that since this is an claims-based measure, there is no added burden of reporting as all required data elements are routinely generated. The measure does not include any proprietary information.
Limitations:
- None identified.
Rationale:
- The developer did not conduct a feasibility assessment of an assessment of missing data for this measure, as it used a 100% sample of paid Medicare claims. The developer further states that since this is an claims-based measure, there is no added burden of reporting as all required data elements are routinely generated. The measure does not include any proprietary information.
Scientific Acceptability
Strengths:
- The measure is clear and well defined.
Entity level reliability (signal-to-noise) is calculated on 2 years of data (1/21-12/22) for the risk adjusted measure across 1754 entities and is >0.6 for about 75% of the entities. (The estimated lower quartile is 0.605.)
Limitations:
- About 25% of the entities have a reliability below 0.6.
Rationale:
- The measure is well defined. Reliability was assessed at the entity level. Reliability statistics are above the established thresholds for about 75% of the entities.
Strengths:
- The developer provided three sources of evidence in support of measure validity (1) face validity of the measure score, (2) association with procedural volume, and (3) validity of the outcome based on an analysis of diagnosis codes. For face validity, the developer convened a technical expert panel (TEP) during the measure’s initial development. 12/13 (92.3%) of TEP respondents agreed that the measure is valid and useful and can improve the quality of care. Second, in a claims-based analysis of the outcome, the developer showed that the most frequent (the top 25) principal diagnosis codes associated with the outcome (unplanned hospital visits within 7 days of a qualifying procedure), many of which indicate a complication from the procedure. For example, the top four principal diagnoses are “other acute postprocedural pain,” “syncope and collapse,” “retention of urine, unspecified,” and “constipation, unspecified”. Lastly, the developer considered conducting a correlation analysis to another quality measure but did not find a measure that “was suitable for comparison.” With respect to the association by facility volume, the developer hypothesized a weak-to-moderate, negative relationship meaning that higher volume is associated with lower (better) measure scores. The developer based this hypothesis on prior literature, which found a volume-outcome relationship in “some (but not all) outcomes” for orthopedic procedures when considering procedural volume. It is unclear based on the developer summary if those outcomes include hospital visits. The developer reported a statistically significant correlation of -0.133, consistent with their hypothesis. However, the committee should consider if this is an acceptable form of validity for this measure (see limitations).
The developer also conducted statistical risk adjustment, based on their conceptual model, and selecting factors that had a significant correlation to the outcome. The developer also explored social risk factors, such as dual eligibility and the Area Deprivation Index. The developer did not include these in the final models due to the minimal impact these social risk factors have on the measure scores and to the model overall. Further, the developer reported that when stratifying the measure by dual eligibility, there were insufficient numbers of patients to support the stratification approach. The developer reported a c-statistic of 0.675, indicating good model discrimination.
Limitations:
- The developer conducted accountable entity-level validity testing by associating measure score to facility volume. The developer based this testing on prior literature, which found a volume-outcome relationship in “some (but not all) outcomes” for orthopedic procedures when considering procedural volume. It is unclear based on the developer summary if those outcomes include hospital visits. The developer reported a statistically significant correlation of -0.133, consistent with their hypothesis. However, it is unclear, based on the evidence provided, how facility procedural volume supports the validity of the measure, including the quality construct. The developer did not identify procedural volume within the measure's logic model. Volume may indicate higher number of surgeons or increased quality control and resources, but this is not clearly articulated in the submission. In addition, some facilities may not be able to control procedural volume, thus based on this analysis, facilities with low volumes will have worse outcomes. Overall, it is not clear how procedural volume is an appropriate covariate for validity analysis.
Rationale:
- The developer provided three sources of evidence in support of measure validity (1) face validity of the measure score, (2) association with procedural volume, and (3) validity of the outcome based on an analysis of diagnosis codes. For face validity, the developer convened a technical expert panel (TEP) during the measure’s initial development. 12/13 (92.3%) of TEP respondents agreed that the measure is valid and useful and can improve the quality of care. Second, in a claims-based analysis of the outcome, the developer showed that the most frequent (the top 25) principal diagnosis codes associated with the outcome (unplanned hospital visits within 7 days of a qualifying procedure), many of which indicate a complication from the procedure. Lastly, the developer conducted accountable entity-level validity testing by associating measure score to facility volume. The developer considered conducting a correlation analysis to another quality measure but did not find any that “was suitable for comparison.” With respect to the association by facility volume, the developer hypothesized a weak-to-moderate, negative relationship meaning that higher volume is associated with lower (better) measure scores. The developer based this hypothesis on prior literature, which found a volume-outcome relationship in “some (but not all) outcomes” for orthopedic procedures when considering procedural volume. It is unclear based on the developer summary if those outcomes include hospital visits. The developer reported a statistically significant correlation of -0.133, consistent with their hypothesis. However, it is unclear, based on the evidence provided, how facility procedural volume supports the validity of the measure, including the quality construct. The developer did not identify procedural volume within the measure's logic model. Volume may indicate higher number of surgeons or increased quality control and resources, but this is not clearly articulated in the submission. In addition, some facilities may not be able to control procedural volume, thus based on this analysis, facilities with low volumes will have worse outcomes. Overall, it is not clear how procedural volume is an appropriate covariate for validity analysis. The developer may be able to speak to this during the endorsement meeting.
The developer also conducted statistical risk adjustment, based on their conceptual model, and selecting factors that had a significant correlation to the outcome. The developer also explored social risk factors, such as dual eligibility and the Area Deprivation Index. The developer did not include these in the final models due to the minimal impact these social risk factors have on the measure scores and to the model overall. Further, the developer reported that when stratifying the measure by dual eligibility, there were insufficient numbers of patients to support the stratification approach. The developer reported a c-statistic of 0.675, indicating good model discrimination.
Equity
Strengths:
- The developer reports that ASCs do not serve a high proportion of patients with social risk factors with the median facility proportion of patients with the dual eligibility variable being 1% (count of 1 patient) and for the high ADI, 2% (count of 1 patient).
Limitations:
- The developer did not identify whether the measure rates are different across other patient populations or how this measure may help to reduce disparities. Rather, for this section, the developer provided a rationale for why dual eligibility and Area Deprivation Index were not included in the risk model. For future submissions, the developer may consider whether the measure can identify differences in performance across different patient groups by their social and/or economic status.
Rationale:
- The developer reports that ASCs do not serve a high proportion of patients with social risk factors with the median facility proportion of patients with the dual eligibility variable being 1% (count of 1 patient) and for the high ADI, 2% (count of 1 patient). The developer did not identify whether the measure rates are different across other patient populations or how this measure may help to reduce disparities. Rather, for this section, the developer provided a rationale for why dual eligibility and Area Deprivation Index were not included in the risk model. For future submissions, the developer may consider whether the measure can identify differences in performance across different patient groups by their social and/or economic status.
Use and Usability
Strengths:
- The measure is currently used in the ASC Reporting Program
The developer cites evidence of interventions that can improve hospital visits, as noted previously. The developer further suggests facilities can implement a system of quality improvement and use CMS benchmarking reports, which include the principal diagnosis on admission, to tie quality improvement efforts to complications that are occurring
The developer summarized that stakeholders may submit questions about the measure to CMS via an online tool. CMS then responds to those questions. The developer notes that since re-endorsement in 2020 there have been no major reports or issues with the measure. In addition, no major changes to the measure have been conducted as a result of literature reviews and feedback received via the online Q&A tool.
The developer reports no unexpected findings.
Limitations:
- The developer did not report any findings on the progress on improvement since the procedures are “ever-changing,” making it challenging to accurately assess year-over-year performance.
Rationale:
- The measure is currently used in the ASC Reporting Program
The developer cites evidence of interventions that can improve hospital visits, as noted previously. The developer further suggests facilities can implement a system of quality improvement and use CMS benchmarking reports, which include the principal diagnosis on admission, to tie quality improvement efforts to complications that are occurring
The developer summarized that stakeholders may submit questions about the measure to CMS via an online tool. CMS then responds to those questions. The developer notes that since re-endorsement in 2020 there have been no major reports or issues with the measure. In addition, no major changes to the measure have been conducted as a result of literature reviews and feedback received via the online Q&A tool.
The developer did not report any findings on the progress on improvement since the procedures are “ever-changing,” making it challenging to accurately assess year-over-year performance.
The developer reports no unexpected findings.
Committee Independent Review
Do not support
Importance
agree with staff
Feasibility Acceptance
agree with staff
Scientific Acceptability
agree with staff
agree with staff
Equity
Patients and their caregivers might find this measure more respectful if more settings of follow-up were considered, such as home care, satellite clinics, portal communication, and telehealth, as well as transportation and appointment access.
Use and Usability
Patients and their caregivers might find this measure more meaningful if it considered the corporatization of healthcare with associated increased out-of-pocket expenses and the shift from solo to group practices; the increase in minimally invasive procedures, imaging advancements, telemedicine, the impact of the COVID-19 pandemic, and included data from Medicare Advantage Plans.
Summary
Patients and their caregivers might find this measure more meaningful if it considered the corporatization of healthcare with associated increased out-of-pocket expenses and the shift from solo to group practices; the increase in minimally invasive procedures, imaging advancements, telemedicine, the impact of the COVID-19 pandemic, and included data from Medicare Advantage Plans.
Patients and their caregivers might find this measure more respectful if more settings of follow-up were considered, such as home care, satellite clinics, portal communication, and telehealth, as well as transportation and appointment access.
Support
Importance
agree with staff assessment
Feasibility Acceptance
agree with staff assessment
Scientific Acceptability
agree with staff assessment
agree with staff assessment. would explore whether narrowing the list of admission diagnoses improves validity and reliability.
Equity
Surprised that the sample size with social risk factors was inadequate to allow reporting. Would explore alternative approaches to measuring ASC performance across these measures.
Use and Usability
Considering that this is an existing measure, would expect some evidence of its impact on ASC performance, even if highlighting quality improvement activities triggered by reporting. Should improve with adoption of the measure.
Summary
Important measure to support quality of common procedures in the ASC setting.
Some areas need further…
Importance
Reduction of unplanned hospital visits due to complications after orthopedic surgery in the ambulatory surgical center setting is shown in the literature. Since there is an increase in procedures in ASCs there is a need for this measure to inform ASC providers and providers of the outcome of hospital visits.
Feasibility Acceptance
Uses 100% Medicare claims and developer did not conduct feasibility.
Scientific Acceptability
Reliability is above the threshold of 75%.
Not clear on the procedural volumes and how used in validity analysis.
Equity
The developer reports that ASCs do not serve a high proportion of patients with social risk factors—the developer may want to consider analysis to look at differences by patient group.
Use and Usability
The developer did not report any findings on the progress improvement to be able to accurately assess performance over time.
Summary
Some areas need further clarification from the developer as noted above.
agree with staff assessment
Importance
agree with staff assessment
Feasibility Acceptance
agree with staff assessment
Scientific Acceptability
agree with staff assessment
agree with staff assessment
Equity
agree with staff assessment
Use and Usability
agree with staff assessment
Summary
no additional comments
No additional comments
Importance
The developer effectively highlights the importance of the measure by linking it to significant outcomes. The measure targets reducing unplanned hospital visits following orthopedic surgeries at ambulatory surgical centers among elderly Medicare patients. By focusing on preventable complications such as infections and pain, the developer shows the measure's potential impact on improving patient safety and outcomes. This aligns with the literature, which emphasizes the critical need for enhanced care practices and comprehensive preoperative assessments to mitigate risks associated with outpatient surgeries.
Feasibility Acceptance
The developer has effectively ensured feasibility for the measure by utilizing claims data generated as part of routine care delivery. This approach imposes no additional burden on facilities, as CMS automatically calculates the measure's score from data already submitted for payment. Patient confidentiality remains intact since the measure relies solely on aggregated claims data, not individual patient details.
Scientific Acceptability
Agree with staff comments
Agree with staff comments
Equity
The developer did not specifically analyze whether there are variations in measure rates among different patient populations or discuss how the measure might contribute to reducing disparities. This approach could provide insights into disparities in healthcare outcomes and guide efforts to promote equity through improved measure design and implementation strategies.
Use and Usability
The measure is confirmed to be in use within the ASC Reporting Program, meeting the criteria of use.
Usability Considerations: The developer has outlined strategies for improving hospital visits and implementing quality improvement systems, indicating usability. However, while they haven't reported major issues or changes, they acknowledge challenges in assessing year-over-year performance due to evolving procedures.
Summary
No additional comments
Support with limitation
Importance
Agree w staff assessment
Feasibility Acceptance
Agree w staff assessment
Scientific Acceptability
Agree w staff assessment
Agree w staff assessment
Equity
Agree w staff assessment
Use and Usability
Agree w staff assessment
Summary
Support pending addressable items
Support
Importance
Agree with staff assessment.
Feasibility Acceptance
Agree with staff assessment.
Scientific Acceptability
Agree with staff assessment.
As per facilities with a reliability less than 0.60, we may consider the utility of publicly reporting data that does not meet minimum reliability thresholds.
Agree with staff assessment.
Equity
Agree with staff assessment.
Use and Usability
Agree with staff assessment.
The developer cites an "ever changing" list of covered ASC procedure preventing assessment of improvement over time. This will be a consistent issue for this measure and the developer should explore strategies to mitigate this issue.
Summary
As an increasing number of surgeries move from the inpatient to ASC setting, it is important to track this outcome. This is overall a robust measure but could be strengthened with methods to address performance assessment in the face of the evolving list of covered ASC procedures. Additional information is needed in regards to stratified equity analysis, validity assessment of the quality-volume relationship, and public reporting for ASCs with reliability lower than the minimum threshold.
Public Comments
Public Comment Shared During May 29 Listening Session
I have 3 joint replacements myself, and the recovery on both knees, I never fully recovered to 100%. I think there's a lot that's missing in the process after someone leaves from an orthopedic center and the follow-up that is not there, that really needs to be. So, I'm excited to see this kind of measure go through in hopes that a lot of these procedures have better outcomes for those of us who have to go through them.
Public Comment Shared During May 29 Listening Session
So, these [CBE 3470, 3357, 2539, and 3366] are all ambulatory, which means you were not admitted to the hospital to have the surgery. I had to go into a hospital after a general surgery, but it was orthopedic, right. Mine was within that 7 days. But really any hospital visit after any kind of surgery is important. I totally recognize these and think it's very important, particularly when there's sort of a push to get people in and out of the hospital. So, thank you.
Yeah, I mean, they need to watch. There's a risk standardized hospital visit. They all are about some hospital visit, right? And that does not mean that actually means hospital. Either a stay or observational. I'm assuming. So it's important. And these are all very important. Because sometimes it's just check the box, but it’s important to know why the person ends up back in the hospital after some sort of procedure. Thank you.
Public Comment Shared During May 29 Listening Session
Well, Hello! So my question would just be regarding the age, the 65 years and older for the Medicare patients Fee-for-Service. I don't know if the previous maintenance [review] was still in the same age category, but if the age could be expanded for Medicare Fee-for-Service patients that are younger as well, too. Just widening the age range for these Medicare patients. I know they're looking at older patients, but I don't know if they're looking at older patients being able to bounce back after the surgery, but not because they have to be re-admitted or have an ED visit or an unplanned inpatient admission. So, to just kind of like understand whether it's the age-related readmission, or if it's the procedural readmission, or if they're having complications from the actual surgical event that they experienced. So, I just have a question on why do they [the developer] have the age at 65, and not maybe in the fifties, or just to kind of exclude whether it's age related or procedure related.
Endorsement Should Be Removed From This Measure
CBE ID: 3470
Hospital Visits After Orthopedic Ambulatory Surgical Center Procedures
Endorsement should be removed from this measure. It is not a valid or reliable measure of the quality or efficiency of outpatient orthopedic procedures due to problems with the numerator and the risk adjustment methodology. Continued public reporting of the results could mislead patients about where they should receive such procedures.
Problems with the Numerator
The stated goal of the measure is to improve the quality of care delivered to patients undergoing orthopedic procedures in an ambulatory surgery center (ASC). However, the numerator includes events that have nothing to do with the procedure, while excluding events that represent undesirable outcomes and avoidable costs.
The Measure Information Form provides a table showing the diagnosis codes associated with post-surgical visits, but there is no analysis as to whether these diagnoses are likely to represent complications of the type of procedure the patient received. Some of the most frequent diagnoses were “COVID-19” and “primary hypertension,” which are not complications of outpatient surgeries. Others, such as “retention of urine,” could well have represented chronic conditions that existed prior to the surgery.
Other studies have found that a large fraction of post-surgical visits are not related to the index surgery:
The inclusion of so many visits unrelated to the procedure creates two different problems.
Problems with the Risk Adjustment Methodology
The actual visit rate is modified using a hierarchical logistical regression model that is intended to adjust for differences in the characteristics of the patients receiving procedures that affect outcomes. However, the model fails to adjust for differences in the propensity of patients to visit a hospital for other health problems.
For example, a patient may come to the ED for an acute condition that is unrelated to the orthopedic procedure (e.g., a viral infection, such as flu or COVID-19). An ASC in a community experiencing high rates of such illnesses could have higher rates of post-procedure visits, and this measure would make it appear that the ASC delivers lower quality care. The risk adjustment used in the measure controls for the presence of chronic conditions, but it does not control for the frequency of new acute conditions.
In addition, patients with more severe chronic conditions will have higher rates of ED visits for exacerbations of their conditions, and the rates of ED visits for chronic disease exacerbations will be higher in communities where chronic disease management services are less available and for patients who have lower incomes or lack of family support that make it more difficult for them to access the services that do exist. In these types of communities and for these types of patients, there will likely be more post-procedure ED visits for chronic disease exacerbations that are unrelated to outpatient procedures. The risk adjustment methodology only controls for the presence of a chronic condition; it does not control for either the severity of the condition or for differences in the care patients receive for the chronic condition. Care for the chronic condition is not the responsibility of the physicians or ASC staff who delivered the outpatient procedure, but under this measure, a higher rate of ED visits for problems related to chronic conditions will be interpreted as poor quality care for the outpatient procedure.
One of the earliest studies of outcomes following outpatient surgery adjusted for this by including a factor measuring a patient’s hospital utilization prior to surgery. (Fleisher LA, et al. “Inpatient Hospital Admission and Death After Outpatient Surgery in Elderly Patients,” Archives of Surgery 139:67-72.) However, no such adjustment is included in the risk adjustment model for this measure. As a result, an ASC that has more patients with poorly managed chronic conditions will inappropriately appear to be delivering lower-quality care. Other patient characteristics that could affect the likelihood of post-procedure hospital visits include whether the patient is in a Skilled Nursing Facility or is receiving home health care, but no adjustment is made for these factors and there is no indication in the Measure Information Form that they were even examined.
Poor Measure Reliability and an Inappropriately Low Minimum Case Threshold
The measure is inappropriately being used for ASCs that perform as few as 35 procedures. With that few cases, a single post-discharge visit results in a 3% post-discharge visit rate. Since the overall rate is 2.2%, a single visit at a small ASC could make the difference between having a “low” or “high” visit rate. Moreover, because of the wide range of procedures that are included in the measure, variation in outcomes could easily occur simply because of differences in the specific procedures performed and patients treated during a particular year.
Because of the low volumes and high variability of post-discharge visits, this measure is not sufficiently reliable to use for public reporting. The developers report that the signal-to-noise reliability was below 0.6 for 20% of ASCs, below 0.7 for 40% of ASCs, and below 0.8 for most ASCs. It does not appear that any effort was made to assess the reliability of the measure over time.
Lack of Business Case for Using the Measure and Undesirable Effects of Doing So
The measure identified only 16 out of 3,000 ASCs as being either better or worse than the national average. Over one-third were classified as “number of cases too small” and the rest were classified as “no different than the national average.” Moreover, it does not appear that the developers made any effort to examine the 8 facilities identified as “better than the national rate” or the 8 facilities identified as “worse than the national rate” to assess whether this was really a valid classification based on the number of cases, the reasons for post-discharge visits, and the consistency of the classification over time.
Since only a very small number of ASCs are being classified as anything other than “no different than expected,” this measure is unlikely to encourage improvements in the quality of care delivery. On the other hand, since the weaknesses in the measure methodology could cause some ASCs to be inappropriately labeled as “worse than expected” because of the kinds of patients they treat, use of the measure creates an undesirable incentive for ASCs to avoid treating patients with characteristics that are likely to result in higher numbers of visits, such as patients with multiple comorbidities and patients who do not have good access to primary care and chronic disease management services. It could also cause patients to inappropriately avoid obtaining a needed procedure, or to try and obtain the procedure in a facility that is not adequately equipped to address their needs.
3470 Hospital Visits after Orthopedic Ambulatory Surgical Center
The American Medical Association (AMA) questions whether this measure produces scores that can be considered sufficiently reliable. Specifically, the minimum reliability with at least 35 cases was 0.398, which falls well below what the AMA considers to be acceptable. We believe that measures must achieve a minimum reliability of at least 0.7 or higher in order to be achieve endorsement.