Primary Measure - Most and Moderately Effective Contraceptive Provision or Use: Percentage of patients ages 15-44 and assigned female at birth who received a most or moderately effective contraceptive, or were documented to use a most or moderately effective contraceptive method in the measurement period. The primary measure captures new provision as well as current use of most and moderately effective contraceptive methods to accurately capture contraceptive utilization even if provided in a different calendar year or a different health care site.
Sub-Measure - LARC-SINC: Percentage of patients ages 15-44 and assigned female at birth who received LARC in the measurement period. The sub-measure captures LARC provision to ensure access to these methods by identifying low provision rates (i.e., below 2%).
For both measures: to focus on the population of non-postpartum patients interested in contraceptive services, the denominator excludes those individuals who did not receive or have documented use of a method if they indicated during the year that they did not want these services, as well as those who are eligible for postpartum contraceptive services during the measurement period.
Measure Specs
- General Information(active tab)
- Numerator
- Denominator
- Exclusions
- Measure Calculation
- Supplemental Attachment
- Point of Contact
General Information
Supporting patients to prevent pregnancy when they wish to do so has social and health benefits for individuals and their families (1,2). Achieving people’s reproductive goals depends on being able to achieve or prevent pregnancy when and how they want to (3). However, in 2015, based on National Survey of Family Growth (NSFG) data, only 47.9% of pregnancies were categorized as occurring at the desired time for the individual (4). In order to help patients achieve their reproductive goals, facilities where individuals receive care must ensure that contraceptive needs are assessed and met. This includes ensuring that the most effective long-acting reversible methods of contraception (LARCs), e.g. intrauterine devices (IUDs) and implants, are available in a timely fashion. Multiple commentaries have detailed how the use of performance measures related to contraceptive provision can improve health care quality and promote positive reproductive health outcomes (3–5). The University of California, San Francisco (UCSF) designed the Contraceptive Use electronic clinical quality measure (eCQM) (CU-SINC, Non-Postpartum; CBE #3699e) to give health care organizations and facilities the opportunity to measure contraceptive use among patients who want contraceptive services. Specified for use with electronic health record (EHR) system data, CBE #3699e can be calculated in a wide array of health care settings, including systems that do not rely on administrative claims and thus fills gaps of extant contraceptive provision measures (CBE #2902, #2903, and #2904) that rely on claims data. Specifically, administrative claims data have limitations affecting measure implementation in different care settings as well as assessment of previous contraceptive services received and patient preferences for contraception. The claims-based measures are designed for calculation in service delivery systems with a fee-for-services model. Thus, entities that use prospective payment systems, such as Federally Qualified Health Centers (FQHCs), cannot easily employ CBE #2902, #2903, and #2904 to evaluate contraceptive services quality. Additionally, these measures of contraceptive provision do not always accurately identify which contraceptive method a woman is using following a visit (particularly LARC methods and sterilization, which are not captured in administrative claims if provided prior to the latest health care visit or during a previous measurement period). Furthermore, patient preferences for contraceptive services are not available in administrative data, and the claims-based measures cannot accurately parse which women need or want contraceptive services.
ECQMs offer a way to measure reproductive health care quality by utilizing EHR system data (6). Unlike administrative claims, EHR systems can capture patient need for contraceptive and other health services and are utilized in a wider array of health care settings. To focus the measure on the population of women interested in contraceptive services, CBE #3699e, i.e. CU-SINC, Non-Postpartum uses the Self-Identified Need for Contraception (SINC) data element to remove people who are not interested in contraception from the measure denominator. This more accurately captures the construct of interest – whether people who want contraception are able to access it – and helps guard against the possibility of directive or coercive counseling towards contraception that may be an unintentional result of use of a contraceptive use performance measure (7,8). CU-SINC, Non-Postpartum is structured to have a primary measure that is the proportion of those who desire contraception who are documented to have those needs met across all methods. It intentionally includes methods that may have been provided in previous calendar years, such as IUD and implants, and methods provided at other sites, in order to capture an overall assessment of how well people’s needs are being met. Recognizing that there are unique barriers to provision of IUDs and implants, including procedural training, availability of medical equipment, stocking of methods, and implementation of billing practices, the measure includes the LARC-SINC submeasure that captures provision of these methods at the actual site. Designed as a floor measure, this submeasure assesses whether these methods are available to those who want them.
In summary, CU-SINC, Non-Postpartum can be used in settings that cannot use the claims-based contraceptive provision measures and provides improved measurement of whether patient’s contraceptive needs are being fulfilled compared to claims-based measures. CBE #3699e will inform quality improvement initiatives that help health care organizations better meet patients’ needs by increasing patient-centered access to contraception, a step towards the goal of reproductive autonomy and well-being for all. Moreover, improvement in the quality of contraceptive care has been shown to improve people’s ability to identify methods that they can use over time and to promote engagement with health care across the reproductive life course, which will improve people’s reproductive outcomes and therefore would also be expected to have a positive impact on health care costs.
REFERENCES
1. Conde-Agudelo A, Rosas-Bermúdez A, Kafury-Goeta AC. Birth Spacing and Risk of Adverse Perinatal Outcomes: A Meta-analysis. JAMA. 2006 Apr 19;295(15):1809.
2. Conde-Agudelo A, Rosas-Bermúdez A, Kafury-Goeta AC. Effects of birth spacing on maternal health: a systematic review. American Journal of Obstetrics and Gynecology. 2007 Apr;196(4):297–308.
3. Gavin L, Frederiksen B, Robbins C, Pazol K, Moskosky S. New clinical performance measures for contraceptive care: their importance to healthcare quality. Contraception. 2017 Sep;96(3):149–57.
4. Gavin LE, Ahrens KA, Dehlendorf C, Frederiksen BN, Decker E, Moskosky S. Future directions in performance measures for contraceptive care: a proposed framework. Contraception. 2017 Sep 1;96(3):138–44.
5. Moniz MH, Gavin LE, Dalton VK. Performance Measures for Contraceptive Care: A New Tool to Enhance Access to Contraception. Obstetrics & Gynecology. 2017 Nov;130(5):1121–5.
6. Trussell J, Aiken ARA, Micks, E, Guthrie K. Efficacy, safety, and personal considerations. In: Contraceptive Technology. 21st ed. Ayer Company Publishers, Inc.; 2018. p. 95–128.
7. Person-Centered Reproductive Health Program [Internet]. [cited 2024 Jul 18]. Self-Identified Need for Contraception (SINC). Available from: https://pcrhp.ucsf.edu/sinc
8. Dehlendorf C, Perry JC, Borrero S, Callegari L, Fuentes L, Perritt J. Meeting people’s pregnancy prevention needs: Let’s not force people to state an “Intention.” Contraception. 2024 Jul;135:110400.
CU-SINC, Non-Postpartum uses electronically extracted data from structured fields within EHR systems, after data are collected from ambulatory, outpatient clinical encounters and entered into EHR structured fields.
We implemented and tested the CU-SINC, Non-Postpartum in primary care settings through a quality improvement learning collaborative among FQHCs. All value sets utilized in our measure rely on standardized coding systems and are published on VSAC.
For more information on the feasibility of the CU-SINC, Non-Postpartum, see Section 4 Feasibility. To review our reliability and validity analyses methods and results, see Section 5, Scientific Acceptability.
Numerator
Primary measure: Of patients in the denominator, those who received or were documented to be using a most (i.e., sterilization, implants, intrauterine devices or systems (IUD/IUS) or moderately effective (i.e., injectables, oral pills, patch, or ring) contraceptive method.
Sub-measure: Of patients in the denominator, those who were provided a long-acting reversible method (IUD or implant).
The numerator of the primary measure includes patients in the denominator who were provided or were documented to be using a most (i.e., sterilization, implants, intrauterine devices or systems) or moderately effective (i.e., injectables, oral pills, patch, or ring) method during the measurement period.
The numerator of the submeasure includes patients in the denominator who were provided a long-acting reversible method (i.e., IUD or implant) during the measurement period.
The measurement period is a single calendar year, and this measure uses two consecutive calendar years of data: the year prior to the measurement period, and the measurement period. The measure is patient-based and calculated at the facility and clinician group/practice level of analysis.
All data elements available to calculate the primary measure numerator are defined within the active value sets and codes in the Value Set Authority Center (VSAC) and included in the measure specification. For the list of value sets to calculate the primary measure numerator, see item 1.13a, Data Dictionary Attachment (3699e VSAC Value Sets_508.xlsx), worksheet named “Value Set List” with “Numerator (Primary Measure)” shown in Column B. The measure’s specification was developed in CMS’ Measure Authoring Development Integrated Environment (MADiE) system and uses direct reference codes for the SINC Screening question and its multiple response options. This eCQM specification utilizes Clinical Quality Language (CQL) and Quality Data Model (QDM 5.6). The primary measure assesses patients who were provided or were documented to be using most and moderately effective contraception; thus, the value sets include codes indicating surveillance of these contraceptive methods.
All data elements available to calculate the submeasure numerator are defined within the active value sets and codes in the VSAC and included in the measure specification. For the list of value sets to calculate the submeasure numerator, see item 1.13a, Data Dictionary Attachment (3699e VSAC Value Sets_508.xlsx) worksheet named “Value Set List” with “Numerator (Submeasure)” shown in Column B. The submeasure assesses patients who were provided LARC methods and are not already using them; thus, the submeasure value sets exclude codes related to surveillance of LARC methods.
Denominator
Non-postpartum patients assigned female at birth aged 15-44 with a qualifying encounter in the measurement period.
The measure denominator includes patients assigned female at birth aged 15-44 who had a qualifying encounter (QE) during the measurement period. Age is calculated with the start of the measurement period as an anchor date.
The measurement period is a single calendar year, and this measure uses two consecutive calendar years of data: the year prior to the measurement period, and the measurement period. The measure is patient-based and calculated at the facility and clinician group/practice level of analysis.
All data elements available to calculate this denominator are defined within the active value sets in the VSAC and included in the measure specification. For a list of value sets to calculate the denominator, see item 1.13a, Data Dictionary Attachment (3699e VSAC Value Sets_508.xlsx) worksheet named “Value Set List” with “Denominator” shown in Column B.
Note that a single code from any of the value sets defining QE that is documented during the measurement period counts as a QE.
Exclusions
Patients are excluded from the denominator if they had a live birth and are eligible for postpartum contraception during the measurement period (i.e., had a prenatal care visit plus a documented live birth delivery date or documented estimated delivery date without a non-live birth event between three months prior to the measurement period and nine months into the measurement period), are anatomically infecund (i.e., due to removal of uterus and/or both ovaries), or those who did not want to discuss their contraceptive needs from among those who were not provided nor were documented to be using a most or moderately effective method throughout the measurement period.
The measurement period is one calendar year, and this measure uses two consecutive calendar years of data: the year prior to the measurement period, and the measurement period. The measure is patient-based and calculated at the facility and clinician group/practice level of analysis.
Patients who are excluded from the denominator meet one of the following criteria:
- Had a live birth making them eligible for postpartum contraception in the measurement period: Those who had a prenatal care visit in the year prior to the measurement period through the first nine months of the measurement period (i.e., 1/1/XX-1 through 9/30/XX) with a documented live birth delivery date, or a documented estimated delivery date (EDD) between three months prior to the measurement period and nine months into the measurement period (i.e., 10/1/XX-1 through 9/30/XX), provided that those with EDD only did not have a documented ectopic pregnancy, intrauterine fetal demise, early pregnancy loss, or abortion (i.e., non-live birth event)
- Are anatomically infecund: those with documentation of anatomical infecundity due to removal of uterus and/or bilateral ovaries during year prior to the measurement period and through the measurement period (i.e., 1/1/XX-1 through 12/31/XX)
- Answered “No” to all instances of the SINC question and were not provided nor were documented to be using a most or moderately effective method throughout the measurement period.
All data elements available to calculate the denominator exclusions are defined within value sets active in the VSAC and included in the measure specification. For a list of value sets to calculate the denominator exclusions, see item 1.13a, Data Dictionary Attachment (3699e VSAC Value Sets_508.xlsx) worksheet named “Value Set List” with “Denominator Exclusions” shown in Column B.
Measure Calculation
The measurement period is one calendar year, and this measure uses two consecutive calendar years of data: the year prior to the measurement period, and the measurement period. The measure is patient-based and calculated at the facility and clinician group/practice level of analysis.
Step 1. Identify all patients assigned female at birth ages 15-44 years who had a qualifying encounter at the specified facility or clinician group/practice.
Step 2. Define the denominator by excluding patients who:
- Had a live birth making them eligible for postpartum contraception in the measurement period: Those who had a prenatal care visit in the year prior to the measurement period through the first nine months of the measurement period with a documented live birth delivery date, or a documented EDD between three months prior to the measurement period and nine months into the measurement period (i.e., 10/1/XX-1 through 9/30/XX), provided that those with EDD only did not have a documented ectopic pregnancy, intrauterine fetal demise, early pregnancy loss, or abortion (i.e., non-live birth event)
- Are anatomically infecund: those with documentation of anatomical infecundity due to removal of uterus and/or bilateral ovaries during year prior to the measurement period and through the measurement period (i.e., 1/1/XX-1 through 12/31/XX)
- Answered “No” to all instances of the SINC question and were not provided nor were documented to be using a most or moderately effective method throughout the measurement period.
Step 3a. Define numerator 1 by using codes to identify patients in the denominator who were provided or continued use of a most or moderately effective method (primary measure)
Step 3b. Define numerator 2 by using codes to identify patients in the denominator who were provided a long-acting reversible method of contraception (LARC, i.e., intrauterine device or subcutaneous implant, submeasure).
Step 4a: Calculate the primary measure rate by dividing numerator 1 by the denominator
Step 4b: Calculate the submeasure rate by dividing numerator 2 by the denominator
In this application, we present CU-SINC, Non-Postpartum scores stratified by the following age groups: ages 15-20 years and ages 21-44 years in addition to the full age range of the initial population, ages 15-44 years. These age groups align with the endorsed claims-based Contraceptive Care measures (CBE #2903 and #2904) stewarded by the HHS Office of Population Affairs (OPA) and allow for measure rates among adolescents (ages 15-20 years) to be examined separately from adult (ages 21-44 years) patients for the purposes of quality improvement. Age is calculated using the start of the measurement period or calendar year as the anchor date; no value sets are required. See also Section 5.4.1 for more details.
Although we do not specify that the measure must be calculated with a minimum sample size, we suggest using (and present in this application) a minimum of at least 50 eligible patients per reporting unit. Patient denominator minimums ensure that practices/facilities are large enough to have an adequate volume of patients across the year for consistent reporting. This minimum sample size also fields sufficient reliability as described in section 5.2.3.
Supplemental Attachment
Point of Contact
Not applicable
Christine Dehlendorf
San Francisco, CA
United States
Christine Dehlendorf
University of California, San Francisco
san francisco , CA
United States
Importance
Evidence
Providing contraceptive care to those who wish to prevent pregnancy supports individuals and their families to meet their reproductive goals, including delaying or preventing pregnancy and achieving desired birth spacing (1,2). Achieving people’s reproductive goals depends on being able to achieve or prevent pregnancy when and how they want to (3). However, in 2015, based on National Survey of Family Growth (NSFG) data, only 47.9% of pregnancies were categorized as occurring as desired at the time for the individual (4). Contraception is a highly effective clinical preventive service that can assist women in reaching their reproductive health goals (5,6). While most and moderately effective contraceptive methods have a failure rate of 1-23%, not using any method at all has a failure rate of 85% (6). The most commonly used and most effective contraceptive methods in the United States require contact with a health care provider (7). Moreover, access to sexual and reproductive healthcare has been associated with use of prescription contraceptive methods and contraceptive counseling has been linked to ongoing contraceptive use (8,9). Therefore, to support patients in achieving their reproductive goals, facilities where individuals receive their care must ensure that their contraceptive needs are assessed and met. This includes ensuring that long-acting reversible methods of contraception (LARC) - intrauterine devices (IUDs) and implants – are available. The Women’s Preventive Services Guidelines, issued by the American College of Obstetricians and Gynecologists and the Health Resources and Service Administration, recommend unhindered and affordable access to a full suite of contraceptive methods (10,11).
Barriers to access to contraceptive care exist. Despite this need for contraceptive services, many people who do not want to become pregnant do not use contraception. Data from the NSFG 2011-2017, for example, found that of those at risk of an unintended pregnancy and who were sexually active, 18% were not using any form of contraception, and, of those, 84% did not want to become pregnant in the following two years (12). One contributor to these statistics is lack of access to contraceptive care. NSFG analyses of 2011-2013 data found that only 46% of women at risk of unintended pregnancy receive contraceptive services in a year (13). Further, among ambulatory encounters with women of reproductive age in the United States, only 14% include any reproductive health services, including contraception (14). Among those who wanted to prevent pregnancy for at least 5 years, 19% reported that they were not using contraception because they could not access a method (12).
Gaps in care access are not equitably experienced and are increasing. One NSFG analysis of the 2006-2017 waves of data collection found that overall 18.3% of women of reproductive age received contraceptive counseling, and when looking at subgroups, found the Black and Latina heterosexual women had lower odds of receiving contraceptive counseling compared to white heterosexual women (with Latina sexual minority women having the lowest odds) (15). A separate analysis of 2011-2013 NSFG data reported disparities in receipt of contraceptive services by both race/ethnicity and age (with younger patients being less likely to receive contraceptive services) (13). Additionally, changing policy environments have led to a decrease in contraceptive care access and increased barriers to care. A recent study used four waves of cross-sectional study data to look at changes in contraceptive care access in four states between 2021 and 2023 and found a four-percentage point increase in reported barriers to accessing contraceptive care between the time points (16). Moreover, ten percent of respondents who were not using a preferred contraceptive method named access barriers, including difficulty accessing a facility, as a reason (17). These barriers infringe upon individuals’ ability to access and use contraception, compromising their ability to achieve their reproductive goals.
Contraceptive performance measures are an important tool to address gaps in access. Multiple commentaries have detailed how the use of performance measures related to contraceptive provision can improve health care quality and promote positive reproductive health outcomes (18–20). This impact can occur, for example, through encouraging providers and systems to prioritize reproductive health services, including in quality improvement initiatives, and follow national clinical recommendations related to this care and offer a full range of methods.
Existing claims-based measures of contraceptive provision have several limitations, including that they cannot be used in settings with prospective payment systems, such as Federally Qualified Health Centers (FQHCs), which are critical for contraceptive access. The National Quality Forum (NQF) endorsed the first clinical performance measures focused on contraception in October 2016, empowering health care organizations to assess contraceptive services to improve quality of care. Stewarded by the U.S. Health and Human Services Office of Population Affairs and specified for calculation in administrative claims, the Contraceptive Care measures (Consensus-Based Entity [CBE] #2902, #2903, and #2904) estimate the percentage of women ages 15-44 years provided a most or moderately effective method of contraception in two populations in this age range: postpartum women and all fecund women. These CBE-endorsed measures also evaluate access to LARCs, which is a subset of most and moderately effective methods, by focusing on low (i.e., less than 2%) rates of use as a proxy for access (18–20) to these specific methods.
The Contraceptive Care measures provide reliable and valid metrics for health entities to evaluate the proportion of women receiving prescription contraceptive methods, but administrative claims data have limitations affecting measure implementation in different care settings as well as assessment of previous contraceptive services received and patient preferences for contraception. They are designed for calculation in service delivery systems with a fee-for-services model, which rely on claims. Thus, entities that use prospective payment systems, such as FQHCs, which are community-based health care providers that receive federal funds to provide primary care services in underserved areas, cannot easily employ CBE #2902, #2903, and #2904 to evaluate contraceptive services quality. These measures also do not always accurately identify which contraceptive method a woman is using following a visit (particularly LARC methods and sterilization, which are not captured in administrative claims if provided at a different site or prior to the measurement period). Furthermore, these claims-based measures cannot account for whether a person wants to be using contraception since this information is not available in administrative claims.
Electronic clinical quality measures (eCQMs) offer a way to measure contraceptive care quality by utilizing electronic health record (EHR) system data (6). Unlike administrative claims, EHR systems can capture patient need for contraceptive and other health services and can be utilized in care settings where claims-based measures are not feasible. EHR systems can also identify ongoing use of a method (e.g. IUD, implant, or sterilization) even if not provided in the measurement period or at the specific clinical site. Ideally, eCQMs are calculated with data captured in structured form during the process of patient care. CBE #3699e, UCSF’s Contraceptive Use eCQM (CU-SINC), Non-Postpartum aims to document both contraceptive use and provision while defining the population in need of contraceptive services for the denominator more accurately through encounter-level EHR data.
CU-SINC, Non-Postpartum promotes person-centered contraceptive care screening practices and better specifies the population in need of services to align with equity goals. To focus the measure on the population of women interested in contraceptive services, UCSF’s Person-Centered Reproductive Health Program (PCRHP) created the Self-Identified Need for Contraception (SINC) data element (21). SINC consists of a standardized question and response options in the LOINC code system. SINC acts as the basis of the process measure CBE #4655e (aka Contraceptive Care Screening eCQM) and serves as an exclusion criterion for the CBE #3699e denominator. Before SINC, no measure of patient desire for contraceptive services existed for consistent implementation across EHR systems (note that One Key Question® (22), a proprietary question that assesses desire for pregnancy in the next year, does not fulfill this need, in that it assesses future desires, rather than immediate need for services). Developed through our engagement with Reproductive Justice Consultants, patients, and industry stakeholders, this screening question asks patients for their desire for contraceptive services on the day of their visit. SINC helps refine the CBE #3699e denominator to exclude those individuals who did not receive or have documented use of a prescription contraceptive method if they indicated no desire for contraceptive services (23). Use of this novel data element helps guard against the possibility of directive or coercive counseling towards contraception that may be an unintentional result of implementation of a contraceptive use performance measure. This is particularly important given the (ongoing) history of reproductive oppression, contraceptive coercion, and biased counseling in the United States directed at women of color and low-income women (24–30).
CU-SINC, Non-Postpartum encourages provision of contraceptive services. Like the currently endorsed measure of most and moderately effective contraceptive methods that relies on claims data (CBE #2903), the CBE #3699e primary measure is designed to encourage provision of these contraceptive methods to those who desire them. We recognize that some patients will prefer to use non-prescription methods that do not qualify as most or moderately effective methods, even when provided with full counseling. Currently this measure is limited to prescription methods because non-prescription methods are not routinely or consistently documented in EHRs. As a result, we do not have a currently identified benchmark for the CBE #3699e primary measure and do not expect scores to reach 100%. We hope for, and will continue to work toward, consistent EHR documentation of non-prescription contraceptive methods. At that point, we will re-assess the specifications and interpretation guidance of CU-SINC, Non-Postpartum. The goal of the CBE #3699e submeasure related to IUD and implant provision (LARC-SINC submeasure) is to ensure access to these methods, and will be interpreted similarly to CBE #2904, in which the goal is to identify low rates of provision (i.e., below 2%) as an indication of barriers to access. We emphasize that it is important that contraceptive services are provided in a patient-centered manner that treats each person as a unique individual with respect, empathy, and understanding, providing accurate, easy-to-understand information based on the patient’s self-identified needs, goals, preferences, and values (31).
In summary, CU-SINC, Non-Postpartum can support enhanced access to patient-centered contraceptive care, can be used in settings that cannot use the claims-based contraceptive provision measures, and provides improved measurement of whether patient’s contraceptive needs are being fulfilled compared to existing claims-based measures. By specifying the denominator as people who self-identify as needing contraceptive services, CBE #3699e shifts the focus to people’s reproductive health needs as they define them. Implementing CBE #3699e will result in quality improvement initiatives that help health care organizations better meet patients’ needs by increasing patient-centered access to contraception, a step towards the goal of reproductive autonomy and well-being for all.
REFERENCES
1. Conde-Agudelo A, Rosas-Bermúdez A, Kafury-Goeta AC. Effects of birth spacing on maternal health: a systematic review. American Journal of Obstetrics and Gynecology. 2007 Apr;196(4):297–308.
2. Congdon JL, Trope LA, Bruce JS, Chung PJ, Dehlendorf C, Chamberlain LJ. Meeting the Needs of Postpartum Women With and Without a Recent Preterm Birth: Perceptions of Maternal Family Planning in Pediatrics. Matern Child Health J. 2020 Mar;24(3):378–88.
3. Dehlendorf C, Akers AY, Borrero S, Callegari LS, Cadena D, Gomez AM, et al. Evolving the Preconception Health Framework: A Call for Reproductive and Sexual Health Equity. Obstet Gynecol. 2021 Feb 1;137(2):234–9.
4. Kost K, Zolna M, Murro R. Pregnancies in the United States by Desire for Pregnancy: Estimates for 2009, 2011, 2013, and 2015. Demography. 2023 Jun 1;60(3):837–63.
5. Mansour D, Inki P, Gemzell-Danielsson K. Efficacy of contraceptive methods: A review of the literature. The European Journal of Contraception & Reproductive Health Care. 2010 Feb;15(1):4–16.
6. Trussell J, Aiken ARA, Micks, E, Guthrie K. Efficacy, safety, and personal considerations. In: Contraceptive Technology. 21st ed. )Ayer Company Publishers, Inc.; 2018. p. 95–128.
7. Daniels K, Abma JC. Current Contraceptive Status Among Women Aged 15-49: United States, 2017-2019. NCHS Data Brief. 2020 Oct;(388):1–8.
8. Lee JK, Parisi SM, Akers AY, Borrerro S, Schwarz EB. The Impact of Contraceptive Counseling in Primary Care on Contraceptive Use. J GEN INTERN MED. 2011 Jul;26(7):731–6.
9. Kavanaugh ML, Pliskin E. Use of contraception among reproductive-aged women in the United States, 2014 and 2016. F&S Reports. 2020 Sep;1(2):83–93.
10. Committee opinion no. 615: Access to contraception. Obstet Gynecol. 2015 Jan;125(1):250–5.
11. Health Resouces and Services Administration and ACOG. Women’s Preventive Services Guidelines [Internet]. U.S. Department of Health and Human Services, Health Resources and Services Administration; 2019 Dec. Available from: https://www.hrsa.gov/womens-guidelines/index.html
12. Frederiksen BN, Ahrens K. Understanding the extent of contraceptive non-use among women at risk of unintended pregnancy, National Survey of Family Growth 2011-2017. Contracept X. 2020;2:100033.
13. Pazol K, Robbins CL, Black LI, Ahrens KA, Daniels K, Chandra A, et al. Receipt of Selected Preventive Health Services for Women and Men of Reproductive Age - United States, 2011-2013. MMWR Surveill Summ. 2017 Oct 27;66(20):1–31.
14. Bello JK, Rao G, Stulberg DB. Trends in contraceptive and preconception care in United States ambulatory practices. Fam Med. 2015 Apr;47(4):264–71.
15. Agénor M, Pérez AE, Wilhoit A, Almeda F, Charlton BM, Evans ML, et al. Contraceptive Care Disparities Among Sexual Orientation Identity and Racial/Ethnic Subgroups of U.S. Women: A National Probability Sample Study. J Womens Health (Larchmt). 2021 Oct;30(10):1406–15.
16. Kavanaugh ML, Friedrich-Karnik A. Has the fall of Roe changed contraceptive access and use? New research from four US states offers critical insights. Health Affairs Scholar. 2024 Feb 1;2(2):qxae016.
17. Kavanaugh ML, Hussain R, Little AC. Unfulfilled and method‐specific contraceptive preferences among reproductive‐aged contraceptive users in Arizona, Iowa, New Jersey, and Wisconsin. Health Services Research. 2024 Jun;59(3):e14297.
18. Gavin L, Frederiksen B, Robbins C, Pazol K, Moskosky S. New clinical performance measures for contraceptive care: their importance to healthcare quality. Contraception. 2017 Sep;96(3):149–57.
19. Gavin LE, Ahrens KA, Dehlendorf C, Frederiksen BN, Decker E, Moskosky S. Future directions in performance measures for contraceptive care: a proposed framework. Contraception. 2017 Sep;96(3):138–44.
20. Moniz MH, Gavin LE, Dalton VK. Performance Measures for Contraceptive Care: A New Tool to Enhance Access to Contraception. Obstetrics & Gynecology. 2017 Nov;130(5):1121–5.
21. Person-Centered Reproductive Health Program [Internet]. [cited 2024 Jul 18]. Self-Identified Need for Contraception (SINC). Available from: https://pcrhp.ucsf.edu/sinc
22. Power to Decide. One Key Question online [Internet]. Available from: https://powertodecide.org/one-key-question
23. Person-Centered Reproductive Health Program. Assessing the Need for Contraceptive Services [Internet]. University of California, San Francisco; [cited 2023 Jul 14]. Available from: https://pcrhp.ucsf.edu/sites/g/files/tkssra4126/f/Implementation%20Guid…
24. Stern AM. Sterilized in the name of public health: race, immigration, and reproductive control in modern California. Am J Public Health. 2005 Jul;95(7):1128–38.
25. Downing RA, LaVeist TA, Bullock HE. Intersections of Ethnicity and Social Class in Provider Advice Regarding Reproductive Health. Am J Public Health. 2007 Oct;97(10):1803–7.
26. Becker D, Tsui AO. Reproductive Health Service Preferences And Perceptions of Quality Among Low-Income Women: Racial, Ethnic and Language Group Differences. Perspectives on Sexual and Reproductive Health. 2008 Dec;40(4):202–11.
27. Borrero S, Schwarz EB, Creinin M, Ibrahim S. The Impact of Race and Ethnicity on Receipt of Family Planning Services in the United States. Journal of Women’s Health. 2009 Jan;18(1):91–6.
28. Gomez AM, Wapman M. Under (implicit) pressure: young Black and Latina women’s perceptions of contraceptive care. Contraception. 2017 Oct;96(4):221–6.
29. Gomez AM, Fuentes L, Allina A. Women or LARC First? Reproductive Autonomy And the Promotion of Long-Acting Reversible Contraceptive Methods. Perspect Sex Repro H. 2014 Sep;46(3):171–5.
30. Gubrium AC, Mann ES, Borrero S, Dehlendorf C, Fields J, Geronimus AT, et al. Realizing Reproductive Health Equity Needs More Than Long-Acting Reversible Contraception (LARC). Am J Public Health. 2016 Jan;106(1):18–9.
31. Romer SE, Blum J, Borrero S, Crowley JM, Hart J, Magee MM, et al. Providing Quality Family Planning Services in the United States: Recommendations of the U.S. Office of Population Affairs (Revised 2024). American Journal of Preventive Medicine. 2024 Dec;67(6):S41–86.
Measure Impact
Contraception is a reproductive health technology relevant to many individuals and families throughout the life course. It can help enable individuals to build families and exercise reproductive autonomy aligned with their desires, preferences, values, and circumstances. Further, use of this technology is widespread; many patients who want to prevent pregnancy use contraceptive services to do so. Research shows desire to avoid pregnancy is strongly associated with contraceptive use (1–3) and use is prevalent; from 2015-2019, 26 million women received a contraceptive service in the United States (4). The most commonly used contraceptive methods in the United States require contact with a health care provider (5). Increased access to and use of preferred contraceptive methods (intermediate outcome) will better meet patients’ needs by increasing patient-centered access to contraception, a step towards the goal of reproductive autonomy and well-being for all (impact). In our development of our contraceptive performance measures, UCSF has worked closely with a patient stakeholder group (PSG) consisting of nine reproductive-aged individuals assigned female at birth, who endorsed the importance of access to patient-centered contraceptive care and guided our approach to measurement.
CBE #3699e uses SINC to define its denominator. This intentional inclusion serves to better center patient preferences for contraceptive care, by limiting the denominator to only those who expressed interest in contraception or pregnancy prevention in a calendar year. SINC was designed as a patient-centered approach responsive to patient preferences of how they would like to be asked about reproductive health service needs. Studies have highlighted that patients prefer to be asked about their reproductive health needs through service-bound questions. In one qualitative study, patients desired contraceptive counseling availability in primary care, provided that their care team engaged in a manner that respects their autonomy and reproductive desires. To achieve this, they preferred a contraceptive care screening question that was open-ended, inclusive, and promoted autonomy, in particular a service-bound question akin to SINC (6). This finding was further tested and reinforced in a survey of over 1,000 federally qualified health center patients in New York (7). SINC was developed through our engagement with two Reproductive Justice thought leaders, nine patient advisors, and three industry stakeholders (the Coalition to Expand Contraceptive Access, the National Association of Community Health Centers, and the National Family Planning and Reproductive Health Association) to ensure the phrasing and approach resonated with patients and would not inadvertently pressure patients (8). The screening question language and response options were crafted intentionally and iteratively with patient and community input. These groups overall supported the development and utilization of this measure in clinical care and provided critical input for optimization of the SINC. PCRHP’s PSG uplifted the importance of the ability to opt out of receiving counseling through intentional inclusion of answer options such as “I am here for something else”. Further, they supported efforts to ensure the phrasing was succinct and interpretable for patients and provided key input on the appropriate frequency of asking SINC that would prevent undue pressure, leading to our decision to make this an annual screening question and the specification for the denominator of CBE #3699e to be any “Yes” response to SINC in a calendar year. Similarly, review by community groups, including the National Birth Equity Collaborative, provided input into wording and implementation guidance. Ultimately, the SINC screening tool takes a person-centered, service-bound approach that does not rely on assumptions about why patients are seeking contraception. Its role in defining the denominator of CBE #3699e serves to center patient priorities for their contraceptive care services.
REFERENCES
1. Samari G, Foster DG, Ralph LJ, Rocca CH. Pregnancy preferences and contraceptive use among US women. Contraception. 2020 Feb;101(2):79–85.
2. Rocca CH, Smith MG, Hale NL, Khoury AJ. Ranges of pregnancy preferences and contraceptive use: Results from a population‐based survey in the southeast United States. Perspect Sexual Reproductive. 2022 Sep;54(3):90–8.
3. Stulberg DB, Datta A, White VanGompel E, Schueler K, Rocca CH. One Key Question® and the Desire to Avoid Pregnancy Scale: A comparison of two approaches to asking about pregnancy preferences. Contraception. 2020 Apr;101(4):231–6.
4. Frost JJ, Mueller J, Pleasure ZH. Trends and Differentials in Receipt of Sexual and Reproductive Health Services in the United States: Services Received and Sources of Care, 2006–2019 [Internet]. Guttmacher Institute; 2021 Jun [cited 2024 Sep 10]. Available from: https://www.guttmacher.org/report/sexual-reproductive-health-services-i…
5. Daniels K, Abma JC. Current Contraceptive Status Among Women Aged 15-49: United States, 2017-2019. NCHS Data Brief. 2020 Oct;(388):1–8.
6. Manze MG, Romero DR, Sumberg A, Gagnon M, Roberts L, Jones H. Women’s Perspectives on Reproductive Health Services in Primary Care. Fam Med. 2020 Feb 7;52(2):112–9.
7. Jones HE, Calixte C, Manze M, Perlman M, Rubin S, Roberts L, et al. Primary care patients’ preferences for reproductive health service needs assessment and service availability in New York Federally Qualified Health Centers. Contraception. 2020 Apr;101(4):226–30.
8. Dehlendorf C, Perry JC, Borrero S, Callegari L, Fuentes L, Perritt J. Meeting people’s pregnancy prevention needs: Let’s not force people to state an “Intention.” Contraception. 2024 Jul;135:110400.
Performance Gap
To assess variation of our proposed eCQM in the ambulatory primary care setting, we implemented and tested the CU-SINC, Non-Postpartum primary measure and LARC-SINC submeasure in 207 Community Health Centers (CHCs) within two health care-controlled networks (HCCNs): HCCN 1 and HealthEfficient. HCCN 1 CHCs operate in states in the Midwest and West coast of the United States, while HealthEfficient CHCs are in the Northeast region of the country. We used the following data sets from each HCCN to provide measure scores and descriptive statistics for the scores at the facility (CHC) and clinician group/practice (site) levels of analysis in the most recent measurement period (2023).
- HCCN 1. The HCCN 1 dataset comprised all female patients aged 15-44 years from 1,466 outpatient sites nested in 204 CHCs in 2023.
- HealthEfficient. The HealthEfficient dataset comprised all female patients aged 15-44 years from 42 outpatient sites nested in three CHCs in 2023.
For the purposes of this application, UCSF suggests that the CHC and outpatient site be considered proxies for the facility and clinician group/practice levels of analysis, respectively, since the outpatient sites operate as different locations of the same CHC.
When calculating performance scores, we excluded facilities and clinician groups/practices who had fewer than 50 patients in the #3699e denominator. As described in section 1.26, patient denominator minimums ensure that practices/facilities are large enough to have an adequate volume of patients across the year for consistent reporting. After the exclusion, HCCN 1 has 193 facilities (991 clinician groups/practices) and HealthEfficient has three facilities (15 clinician groups/practices) included in the performance score calculation. See Tables 1a – 1h in Section 2.4a for performance gap results.
Most or Moderately Effective Methods (CU-SINC, Non-Postpartum, Primary Measure)
The primary measure of #3699e captures new provision as well as current use of most and moderately effective contraceptive methods to accurately capture contraceptive utilization even if provided in a different calendar year or a different health care site. Across the 193 facilities within HCCN 1 (N of Persons=703,688), the overall performance score for most or moderately effective methods (primary measure) was 25.0%. The highest performing facility provided or documented use of most or moderately effective contraceptive methods in 81.5% of its eligible patients; the lowest performing facility had a measure score of 1.6% during the measurement period. The overall performance score among the three facilities within HealthEfficient (N of Persons=7,003) was 32.3%, while the percentage of eligible patients with most or moderately contraceptive method provision or documented utilization ranged from 22.4% to 44.7%.
At the clinician group/practice (nested within CHC) level of analysis in HCCN 1, the lowest performing group/practice had a measure score of 0% during the measurement period, whereas the highest performing HCCN 1 clinician group/practice had a score of 94.2%. For HealthEfficient, the range of scores was 9.2% to 46.2%. The overall clinician group/practice-level performance score for the primary measure was 24.6% in HCCN 1 and 25.4% in HealthEfficient.
Long-Acting Reversible Contraception (LARC-SINC Submeasure)
The submeasure of #3699e is a floor measure that captures provision of LARC methods (e.g., intrauterine devices or systems, subcutaneous implants) to ensure access to these methods by identifying low provision rates (i.e., below 2%). This measure therefore will not have a benchmark encouraging high rates of use. In addition, utilization in pay-for-performance or similar programs or setting a high benchmark will be explicitly defined as inappropriate, as doing so may incentivize coercive practices.
Within HCCN 1, the overall performance score for provision of LARC was 5.2% at the facility level. The highest performing facility had a LARC provision rate of 18.5% among its eligible patients; the lowest performing facility did not provide LARC methods to any eligible patients (0%) during the measurement period. The overall performance score among the three facilities within HealthEfficient was 7.8%, while the percentage of eligible patients with LARC provision ranged from 4.5% to 14.2%.
At the clinician group/practice (nested within facility) level in HCCN 1 and HealthEfficient, the lowest performing group/practice had a performance score of 0% and 2.0%, respectively, during the measurement period. The highest performing HCCN 1 and HealthEfficient group/practice provided LARC methods to 29.1% and 15.2% of eligible patients, respectively. The overall clinician group/practice-level performance score for the submeasure was 5.1% in HCCN 1 and 7.2% in HealthEfficient.
Our implementation and testing of the CU-SINC, Non-Postpartum in these HCCNs provide evidence of an existing performance gap for the primary measure in the primary care setting, given the variation in performance scores across facilities and clinician groups/practices within each network. The submeasure also identified some areas within our partner HCCNs in which patients experience lower access to LARC methods (i.e., submeasure rate below 2%).
Equity
Equity
Gaps in equitable contraceptive care access are well documented. One NSFG analysis of the 2006-2017 waves of data collection found that overall 18.3% of women of reproductive age received contraceptive counseling, and when looking at subgroups, found the Black and Latina heterosexual women had lower odds of receiving contraceptive counseling compared to white heterosexual women (with Latina sexual minority women having the lowest odds) (1). A separate analysis of these NSFG data reported disparities in receipt of contraceptive services by both race/ethnicity and age (with younger patients being less likely to receive contraceptive services) (2). Examining Contraceptive Use electronic clinical quality measure (CU-SINC), Non-Postpartum (CBE #3699e) scores overall and by subgroups is a means to evaluate for equitable access to contraceptive care.
In addition, when reviewing CU-SINC, Non-Postpartum scores, attention must also be given to higher rates of contraceptive provision, particularly long-acting reversible contraceptive (LARC) methods, to racial and ethnic minoritized patients and patients living on low incomes. Evidence has documented disparities in contraceptive counseling, including provider pressuring low-income and racially minoritized groups to use contraception, or to specifically use a LARC method, even when that is not their preference (3–6). This is tied to a long history of coercive reproductive healthcare practices grounded in structural racism and classism (7–9). As higher rates of LARC provision are not inherently positive and can in fact reflect problematic counseling, the LARC-SINC submeasure of CBE #3699e, aligned with guidance for the claims-based measure CBE #2904 (Contraceptive Care – Access to LARC), is intended to be used a floor measure, monitoring for very low rates of provision (below 2%) that could indicate barriers to access. We can also examine the CBE #3699e LARC-SINC submeasure by sub-groups to identify both whether there is sufficient access to LARC methods across groups (>=2%) and for signs of potential overprovision to racially minoritized and low-income patients.
METHODS: To evaluate differences in CU-SINC, Non-Postpartum rates by demographic subgroups (age, race, and ethnicity), we first calculated measure rates at the facility (health center) level for each of our two data partners (Health Center Control Network [HCCN] 1 and HealthEfficient). Results are shown in Tables 1, 3, and 4 (Supplemental attachment). Then, using aggregated data at the facility level, we analyzed the differences in rates among subgroups across facilities using a multilevel regression model (10), controlling for nesting of subgroups in facilities. Age, race, and ethnicity groups are each analyzed in a separate model. Regression results are shown in Tables 2 and 5 (File located in Section 7 - Supplemental Information).
RESULTS:
Most or moderately effective contraceptive methods primary measure:
Among 204 facilities in HCCN 1, (Tables 1 & 2):
- Patients aged 15-20 years had a significantly higher rate (29.3%) compared to patients aged 21-44 years (24.1%, p<0.001).
- Black/African American patients had a significantly lower rate (22.7%) compared to White patients (25.2%, p=0.002).
- Compared to non-Hispanic patients (23.9%), Hispanic patients had a significantly lower rate (20.4%) and those who did not report ethnicity had a significantly higher rate (26.7%). However, due to the large number of patients in the “Unreported/Refused to report” ethnicity groups, these results may not represent reliable differences by ethnicity categories.
Among three HealthEfficient facilities (Tables 3 & 5), there were descriptive differences, however these were not statistically significant. The non-significant results may be due to the small number of facilities (n=3) that limited the statistical power to detect statistically significant differences:
- Patients in the 15–20y age group had a higher rate (39.9%) compared to the 21-44y age group (34.9%),
- Those in the non-White race groups had lower rates compared to the White group,
- Hispanic patients had a higher rate (42.5%) compared to the non-Hispanic group (34.6%).
LARC-SINC submeasure (LARC provision)
Among HCCN1 facilities (Tables 1 & 2):
Patients who did not report ethnicity had a significantly higher rate (5.6%) compared to non-Hispanic patients (4.9%, p=0.047). Due to the large number of patients in the “Unreported/Refused to report” ethnicity groups, these results may not represent reliable differences by ethnicity categories.
Among HealthEfficient facilities (Tables 4 & 5):
- Black/African American patients had a significantly lower rate (5.9%) compared to White patients (9.4%, p=0.003).
- Compared to non-Hispanic patients (7.5%), Hispanic patients have a significantly higher rate (11.9%, p=0.004). We note that due to the large number of “Unreported” patients for ethnicity in health center 2 and 3 and a 0 count of Hispanic patients in these health centers, these results may not represent reliable differences by ethnic categories.
INTERPRETATION AND ANTICIPATED IMPACT:
We found significantly lower rates of most or moderately effective contraceptive use among Black patients compared to white patients within the HCCN 1 cohort, and lower rates (nonsignificant) among non-white patients compared to white patients within the HealthEfficient facilities. This may indicate an issue with inequitable access, with Black patients not able to obtain contraception at the same rate as white patients to support them in meeting their reproductive goals as they themselves define it. This may result in a higher number of undesired pregnancies among non-white patients compared to white patients (11). It is also possible that this reflects differences in preferences for prescription vs. non-prescription methods by race. Assessing the LARC-SINC submeasure, all groups surpass the floor 2% at the HCCN level, suggesting suitable access across groups. However, despite the unreliability of the estimate, results may suggest an overprovision of LARC methods to Hispanic patients, or alternatively differences in preferences. Accountable entities could utilize this finding as an indicator of an equity issue that would warrant further exploration to determine whether there are differential counseling practices between ethnic groups, such as through use of the visit-specific Person-Centered Contraceptive Counseling measure (CBE #3543). The differences by age in HCCN 1 may indicate higher levels of access to desired contraception among younger individuals as compared to those over the age of 20. Alternative explanations include that providers may be more directive with adolescents, as concern about their potential fertility may motivate less patient-centered care, or that there may be differences in preferences by age group. Ensuring that patients of all ages who have the potential for pregnancy have their needs assessed and met in a patient-centered manner can be facilitated by examining CU-SINC, Non-Postpartum data and assessing for differential practices.
REFERENCES
1. Agénor M, Pérez AE, Wilhoit A, Almeda F, Charlton BM, Evans ML, et al. Contraceptive Care Disparities Among Sexual Orientation Identity and Racial/Ethnic Subgroups of U.S. Women: A National Probability Sample Study. J Womens Health (Larchmt). 2021 Oct;30(10):1406–15.
2. Pazol K, Robbins CL, Black LI, Ahrens KA, Daniels K, Chandra A, et al. Receipt of Selected Preventive Health Services for Women and Men of Reproductive Age - United States, 2011-2013. MMWR Surveill Summ. 2017 Oct 27;66(20):1–31.
3. Gomez AM, Wapman M. Under (implicit) pressure: young Black and Latina women’s perceptions of contraceptive care. Contraception. 2017 Oct;96(4):221–6.
4. Dehlendorf C, Ruskin R, Grumbach K, Vittinghoff E, Bibbins-Domingo K, Schillinger D, et al. Recommendations for intrauterine contraception: a randomized trial of the effects of patients’ race/ethnicity and socioeconomic status. Am J Obstet Gynecol. 2010 Oct;203(4):319.e1-8.
5. Downing RA, LaVeist TA, Bullock HE. Intersections of Ethnicity and Social Class in Provider Advice Regarding Reproductive Health. Am J Public Health. 2007 Oct;97(10):1803–7.
6. Borrero S, Schwarz EB, Creinin M, Ibrahim S. The Impact of Race and Ethnicity on Receipt of Family Planning Services in the United States. Journal of Women’s Health. 2009 Jan;18(1):91–6.
7. Stern AM. Sterilized in the name of public health: race, immigration, and reproductive control in modern California. Am J Public Health. 2005 Jul;95(7):1128–38.
8. Roberts D. Killing the Black Body. New York, NY: Penguin Random House; 1998.
9. Brandi K, Fuentes L. The history of tiered-effectiveness contraceptive counseling and the importance of patient-centered family planning care. American Journal of Obstetrics and Gynecology. 2020 Apr;222(4):S873–7.
10. Leyland AH, Groenewegen PP. Multilevel modelling and public health policy. Scand J Public Health. 2003 Aug;31(4):267–74.
11. Samari G, Foster DG, Ralph LJ, Rocca CH. Pregnancy preferences and contraceptive use among US women. Contraception. 2020 Feb;101(2):79–85.
Feasibility
Feasibility
Background
UCSF implemented and tested the Contraceptive Use electronic clinical quality measure (eCQM) (CU-SINC), Non-Postpartum (CBE #3699e) primary measure and LARC-SINC submeasure in 1,508 locations within 207 Community Health Centers (CHCs), nested within two health care-controlled networks (HCCNs). Thus, CBE #3699e was tested and implemented in two different EHR systems: HCCN 1 utilizes its customized version of Epic, and HealthEfficient uses eClinicalWorks (eCW).
The first facilities to implement CBE #3699e were the nine facilities within two HCCNs included in our analysis that also participated in Innovating Contraceptive Care in Community Health Centers (ICC in CHCs). ICC in CHCs was a 2020-2023 quality improvement (QI) initiative to improve the CHCs’ contraceptive services through the implementation of evidence-based strategies to incorporate person-centered contraceptive care. This project provided participating facilities and clinician group/practices with resources to effectively incorporate CBE #3699e primary measure and submeasure, including the Self-Identified Need for Contraception (SINC) question. SINC allows CHCs to ask patients about their contraceptive needs at their visit. Our QI initiative also included one additional CHC that implemented the data element into their AthenaOne EHR system. We excluded this additional CHC from measure testing because we do not have access to their data.
Availability of data elements
UCSF recommends implementation of SINC to center contraceptive care on the patients and their stated desire for contraceptive services, but CBE #3699e can still be calculated when SINC is not asked or implemented.
To assist our data partners to implement the SINC data element, UCSF first worked with ten CHCs using three different EHR systems (eCW, Epic, AthenaOne) for the ICC in CHCs QI initiative. In these instances, EHR changes were able to be implemented in about three months in collaboration with clinical and vendor teams. The implementation burden depends on the change process for the EHR at the health system to embed the SINC data element into the relevant EHR template so that care teams can use SINC. HCCN 1 uses its customized version of Epic, a centrally managed EHR system, and easily implemented the SINC question, first across the ICC-participating CHCs and then for all remaining CHCs. In HealthEfficient, each CHC uses eCW and initially implemented SINC at the individual ICC-participating clinician group/practices before incorporating SINC at the non-ICC clinician group/practices. This de-centralized process required more coordination and time. During the final meeting for ICC in CHCs, one participating CHC using Epic stated, “(HCCN 1) did a phenomenal job with our SINC question (and) made it so user-friendly for our clinicians”. An ICC-participating CHC in HealthEfficient at the same meeting reported, “It was challenging to get the SINC implemented into our E[H]R; we use eCW and also train the staff, and (it was) more so a resource challenge, finding the time and staff to distribute the information about the changes.” In contrast, a different participating CHC in HealthEfficient, noted that implementing SINC screening into their EHR system facilitated their QI activities and stated, “I think that what made it easy for me … is that [the HIT staff] had already incorporated the [SINC] questions into our eCW. So it was something that was already running.” Most systems, from our experience, are able to implement the SINC EHR template changes in about three months, with some systems requiring less time. We note that clinical sites across the country have begun to implement and use the SINC data element since it was made available as a LOINC code in August 2021, demonstrating the feasibility of this implementation. These clinical sites include the Planned Parenthood Federation of America, which includes SINC as its recommended approach in its Medical Standard and Guidelines to assessing reproductive needs, and sites affiliated with Upstream USA, including Cambridge Health Alliance. In 2024, HHS Health Resources Services Administration (HRSA) released guidance about reporting of reproductive needs screening in Uniform Data System (UDS) that highlighted SINC. Since that time, UCSF has received several requests for technical assistance from EHR vendors that work with FQHCs who are interested in incorporating this data element.
Maintenance updates to CBE #3699e specification impact on data structures and availability
University of California, San Francisco’s (UCSF’s) updates to CBE #3699e include the following revisions: adding a new, active code set for live birth delivery procedures to replace the inactive, not maintained set used in the specification that was approved for trial use and; updating the value sets that we author on Value Set Authority Center (VSAC) to reflect annual code system edits to keep the code sets current. We also added and deleted codes from our VSAC code sets to ensure the code sets align with how we define the CBE #3699e numerator, denominator, and denominator exclusions. For example, we added one ICD-10-CM code for female sterilization surveillance so that CU-SINC, Non-Postpartum captures patients who use sterilization for contraception.
These small changes did not affect the data structures and availability of eCQM elements in our HCCN partners’ EHR systems.
Feasibility scorecard results
Our HCCN data partners completed the feasibility scorecards for the CU-SINC, Non-Postpartum to provide feedback on the measure and its calculation in their respective EHR systems (see the attached file named “3699e_eCQM-Feasibility-Scorecard_508.xlsx”). The HCCNs evaluated all data elements specified by value sets or direct reference codes (n=87) utilized in CBE #3699e in the feasibility scorecard to ensure a thorough assessment, but CBE #3699e does not require all 87 elements or value sets for calculation due to components of the measure (e.g. use of specific contraceptive methods) being captured through multiple overlapping terminologies.
HealthEfficient indicated that 25% of the eCQM data elements scored 0 in the Data Availability feasibility domain (i.e., data are readily available in a structured format), 28% scored 0 in the Data Accuracy domain (i.e., information contained in the data is correct), 33% scored 0 in the Data Standards domain (i.e., data element is coded using a nationally accepted terminology standard and mapped to the QDM), and 28% scored 0 in the Workflow domain (i.e., extent to which data element capture impacts the workflow for the user). In contrast, HCCN 1 reported that 14% of eCQM data elements scored 0 for availability, 8% scored 0 for accuracy, 16% scored 0 for standards, and 17% scored 0 for workflow. The HCCN data partners provided their solutions to address the CU-SINC, Non-Postpartum elements that did not achieve 100% across the four feasibility domains (see the tabs named, “Feasibility Plan” in the attached scorecard).
The 0 scores for Data Availability, Data Standards, and Data Workflow for both partners were reported mostly because neither EHR system uses all the code systems available in the Centers for Medicaid and Medicare (CMS) Measure Authoring Development Integrated Environment (MADiE) and National Library of Medicine (NLM) VSAC web-based systems. HealthEfficient had 24 data elements scoring 0 for the Workflow domain while HCCN 1 had 15 elements scoring 0 for Workflow, indicating that these elements are not routinely collected during clinical care because both partners’ systems do not use SNOMED CT. Neither HealthEfficient’s nor HCCN 1’s system uses SNOMED or LOINC to define most or moderately effective contraceptive methods; however, this does not negatively impact the ability to calculate the measure, as the specifications take into account the fact that different EHR systems use multiple combinations of code systems. As a result, the CU-SINC, Non-Postpartum includes several terminologies that define the measure’s data elements to facilitate measure use and calculation, even if one or more code systems are excluded. In the case of contraceptive methods, HealthEfficient and HCCN 1 accomplish this using ICD-10-CM codes, which are included in our specifications and stated in the CBE #3699e feasibility scorecard plans to address the absence of SNOMED CT codes. One example of this redundancy is that we use six value sets to assess utilization of oral contraceptive pills in #3699e; these value sets contain codes from ICD-10-CM, CPT, HCPCS, RXNORM, LOINC, and SNOMED CT code systems. Patients in the CBE #3699e denominator who have at least one code from any of these six oral contraceptive pill value sets for a visit in the appropriate date range are counted in the CU-SINC, Non-Postpartum numerator.
Another example of this flexibility is the “Prenatal Care Visits Group” definition that refines the CBE #3699e denominator. The value sets that define prenatal care visits include codes from ICD-10-CM, CPT, HCPCS, and SNOMED CT. If a patient’s EHR record contains the prenatal care visit codes from ICD-10-CM and CPT, then that record can count as a prenatal visit, even if that EHR system does not use SNOMED CT codes. Furthermore, both HCCNs had data elements that scored 0 in the Data Accuracy domain because the participating CHCs do not routinely offer services related to the data elements for live birth delivery, even if the partners’ EHR systems contain structured fields in which the information can reside. UCSF specified the CU-SINC, Non-Postpartum so that the measure allows for use of Estimated Delivery Date (EDD) when Live Birth Delivery Date is unavailable or missing from a patient’s record. Although our partners reported in the feasibility scorecard that Live Birth Delivery Date may not often be reported (because CHCs do not offer labor and delivery services), the EDD data element will be recorded for patients visiting CHCs for prenatal care services, making it available in both HCCNs’ EHR systems and for calculation of the primary measure and submeasure.
Finally, the new SINC data element was initially not integrated into the EHR systems of the nine CHCs participating in the ICC in CHCs QI initiative, but they all ultimately implemented it in order to both assess contraceptive needs and calculate CBE #3699e. Unlike HCCN 1, which easily implemented SINC in its centrally-managed EHR system across their CHCs, HealthEfficient had to implement the SINC question and response options in their EHR for each participating CHC as separate processes over a longer time period. While one HealthEfficient CHC reported a smooth implementation of SINC use, the other CHCs found it more challenging to add the new element. Both HCCN 1 and HealthEfficient CHCs have maintained their use of SINC in primary care visits after the end of ICC in CHCs and expanded SINC use to additional clinician groups/practices that did not originally participate in our QI initiative.
Estimates of the costs or burden of CU-SINC, Non-Postpartum implementation
To implement SINC in partner EHR systems, our partner HCCNs changed their primary care EHR template to include the SINC question and its response options as structured fields. The EHR system programmers and analysts bore the burden of mapping EHR data required for the measure to the eCQM concepts according to the CHCs’ regular clinical workflows for the purposes of calculating and reporting the measure rates. After implementation, we estimate that data entry by clinical staff will require a minute or less to conduct. The SINC question and its response options are also specified in the LOINC code system and can be mapped as such for EHR systems that include LOINC.
With respect to clinical implementation, UCSF advocates for SINC to be included as part of a patient-centered primary care workflow, enhancing access to reproductive health services for those who need it. Therefore, the added time burden on the clinical workflow will be offset by enhanced quality and efficiency of care. The SINC Screening question is estimated to require a minute or less to implement within the primary care workflow.
Data abstraction, measure calculation, and reporting
In order to calculate the measure, it is necessary to perform a data extraction of all the required elements for measure calculation for all encounters in two consecutive calendar years for patients assigned female at birth. The burden of this process will depend on the structure of the clinical datasets and the experience of the EHR programmers. During the ICC in CHCs project period, HCCN 1 and HealthEfficient performed these data extracts, and we provided quality assurance (QA) checks of their datasets to calculate the CU-SINC, Non-Postpartum. Our QA involved confirming that the analysis datasets included both the required elements for measure calculation and all encounters occurring in two consecutive calendar years for patients assigned female at birth, ages 15-44 years. When questions about calculation arose, UCSF and Far Harbor collaborated with the HCCN EHR system programmers and analysts to investigate and agree on a solution. As with the data extraction process, the experience of measure users and EHR programmers will determine the time and resources needed to ensure accurate measure calculations. When a health system first calculates CBE #3699e, the measure users and EHR programmers may require technical assistance and additional QA checks to assess the accuracy of their measure scores. As they gain familiarity with calculating the measure, the time and resources needed for calculation should be reduced. Furthermore, as health systems adopt the use of CU-SINC, Non-Postpartum, these experienced measure users and EHR programmers could answer questions or assist other similar health systems who want to implement SINC and calculate CBE #3699e. One partner implemented a new analytics and population health reporting platform in CHCs working on the QI initiative; in our QA, we helped the partner troubleshoot this new query system to obtain accurate encounter records to calculate the CU-SINC, Non-Postpartum and assess the impact of their QI strategies on contraceptive use and LARC provision.
The data elements comprising CBE #3699e, including the SINC data element, reside within the clinical agency’s secure EHR system server. Each HCCN upholds the normal requirements to ensure that its EHR data elements, including SINC and the elements defining the CU-SINC, Non-Postpartum, are shared only with entities that have active data use agreements and the infrastructure to securely exchange Protected Health Information / Potentially Identifiable Information (PHI/PII).
When calculating CBE #3699e performance scores, the UCSF and Far Harbor team excluded facilities and clinician groups/practices who served fewer than 50 patients during the measurement period. We recommend this cutoff to measure users as well, who will calculate and report CBE #3699e rates. Patient denominator minimums ensure that practices/facilities are large enough to have an adequate volume of patients across the year for consistent reporting. Reporting measure rates only for units with 50 or more patients helps to protect patient confidentiality. After the exclusion, HCCN 1 has 193 facilities (991 clinician groups/practices) and HealthEfficient has three facilities (15 clinician groups/practices) included in the performance score calculation presented in this application.
Adjustments to CBE #3699e in response to feasibility assessment
UCSF first started developing CBE #3699e in 2019 and successfully obtained trial use approval for the eCQM from the previous Consensus-Based Entity (CBE). During that initial development phase, UCSF heard from its potential HCCN data partners that their respective EHR systems contained only some code systems used to specify eCQMs. In anticipation of this challenge to calculate the measure in different EHR systems, we designed CBE #3699e and our other eCQMs to be flexible and usable across EHR systems regardless of the different combinations of code systems that each EHR system can employ. Recognizing that different EHR systems use multiple combinations of code systems, CU-SINC, Non-Postpartum includes several redundant terminologies that define the measure’s components to facilitate measure use and calculation, even if one code system is excluded. For example, as discussed in Section 4.1a, contraceptive use and LARC provision for the primary measure and submeasure numerators is specified in both ICD-10-CM and SNOMED. So even though HealthEfficient’s and HCCN 1’s EHR systems do not contain SNOMED, they can still use ICD-10-CM codes to assess contraceptive use and LARC provision. This adjustment is also stated in the CBE #3699e feasibility scorecard plans for HealthEfficient and HCCN 1 respectively.
Proprietary Information
Scientific Acceptability
Testing Data
Electronic health records (EHR) data from two networks were used for reliability and validity testing:
- Health Center Control Network (HCCN) 1. The HCCN 1 uses its customized Epic system and the dataset comprised all female patients aged 15-44 years from 1,466 outpatient sites (clinician groups/practices) nested in 204 Community Health Centers (facilities) located in Washington, Ohio, California, and Oregon in 2023.
- HealthEfficient. The HealthEfficient uses the eCW system and the dataset comprised all female patients aged 15-44 years from 42 outpatient sites (clinician groups/practices) nested in three Community Health Centers (facilities) located in Connecticut, New York, and New Jersey in 2023.
- HCCN 1. For accountable entity-level reliability and validity testing, the full dataset was used. For data element validity testing, we utilized data from a random sample of 150 female patients ages 15-44 who visited eight sites in calendar year 2023. Due to the low volume of patients who had a birth in HCCN 1, we oversampled eight additional women who had an estimated delivery date, making a total of 158 women for this element. As it was not feasible for HCCN1 to do data element validity testing at all sites, these eight sites were randomly selected to provide an overall assessment of the data validity in the network. The age distribution is comparable between the sample and the full dataset.
- HealthEfficient. For accountable entity-level reliability and validity testing, the full dataset was used. For data element validity testing, we utilized data from a random sample of 144 female patients ages 15-44 who visited five sites in calendar year 2023. As it was not feasible for HealthEfficient to do data element validity testing at all sites, these five sites were randomly selected to provide an overall assessment of the data validity in the network. The age distribution is comparable between the sample and the full dataset.
Accountable Entity-Level Reliability and Validity
The measure was tested at the clinician group/practice-level and the facility-level for both data partners. Accountable entity-level reliability and validity testing used full denominator population in the datasets from HCCN 1 and HealthEfficient networks described above.
HCCN 1 includes 1,466 outpatient sites (clinician groups/practices) nested in 204 Community Health Centers (facilities) located in Washington, Ohio, California, and Oregon. HealthEfficient includes 42 outpatient sites (clinician groups/practices) nested in 3 Community Health Centers (facilities) located in Connecticut, New York, and New Jersey.
Data Element Validity
Eight sites from HCCN 1 and five sites from HealthEfficient provided data, and the analysis was conducted using aggregated numbers across all sites for each network. A sample of 150 patients were randomly selected from the eight sites in HCCN 1 (158 patients were selected for estimated delivery date), and 144 patients were randomly selected from the five sites in HealthEfficient.
See file “5.2.3a. Characteristics and reliability full tables and methods_NPP.docx” in section 5.2.3a, Tables 1-4 for tables of characteristics of units of the eligible population.
Reliability
Several methods have been suggested to assess the reliability of provider-level performance measures (1-3). These methods may focus on different facets of reliability such as consistency across time, consistency across raters or units, or variability at different levels of aggregation. Partnership for Quality Measurement (PQM) has suggested a signal-to-noise approach as one way to evaluate measure reliability (4). For this application, reliability was estimated from a Beta-binomial model using parametric empirical Bayes methods. This method is better able to produce reliability estimates when the performance measures fall in the extreme ranges of the distribution (below 10% or above 90%), whereas the signal-to-noise approach may produce unstable results under those conditions, especially when the patient size is small. Two distributional shape parameters (alpha and beta) were estimated from the observed quality scores, and reliability was then calculated as a function of alpha, beta, and total patient count for each unit of analysis. Overall reliability in this context represents the ability of the proposed measure to confidently distinguish the performance of one entity (e.g., site) from another. A detailed description of this method is demonstrated in Section 5.2.3a Appendix, where we lay out the formulation of the method and describe how it improves upon the Beta-binomial approach applied in previous studies (4-8). This method is currently under revise and resubmit process for journal publication. A threshold of 0.6 is used based on the PQM Endorsement and Maintenance Guidebook.
Measure developers frequently recommended setting a minimum patient size for performance measurement when estimating at the facility or provider level because patient size has a large impact on reliability (9, 10). In this analysis, we tested reliability using 50 as a cutoff of total patients served at each unit of analysis to show how such a threshold impacts reliability.
REFERENCES
1. Adams, J. L., Mehrotra, A., Thomas, J. W., & McGlynn, E. A. (2010). Physician cost profiling--reliability and risk of misclassification. The New England journal of medicine, 362(11), 1014–1021. https://doi.org/10.1056/NEJMsa0906323
2. Scholle, S. H., Roski, J., Adams, J. L., Dunn, D. L., Kerr, E. A., Dugan, D. P., & Jensen, R. E. (2008). Benchmarking physician performance: reliability of individual and composite measures. The American journal of managed care, 14(12), 833–838.
3. Fung, V., Schmittdiel, J. A., Fireman, B., Meer, A., Thomas, S., Smider, N., Hsu, J., & Selby, J. V. (2010). Meaningful variation in performance: a systematic literature review. Medical care, 48(2), 140–148. https://doi.org/10.1097/MLR.0b013e3181bd4dc3
4. Adams, John L. (2009) The Reliability of Provider Profiling: A Tutorial. Santa Monica, CA: RAND Corporation. https://www.rand.org/pubs/technical_reports/TR653.html.
5. Adams, J. L., & Paddock, S. M. (2017). Misclassification Risk of Tier-Based Physician Quality Performance Systems. Health services research, 52(4), 1277–1296. https://doi.org/10.1111/1475-6773.12561
6. Blair, R., Liu, J., Rosenau, M., Brannan, M., Hazelwood, N., Gray, K. F. & Schmitt, A. (2015). Development of Quality Measures for Inpatient Psychiatric Facilities. Development, 2, 04.
7. Kazis, L. E., Rogers, W. H., Rothendler, J., Qian, S., Selim, A., Edelen, M. O.& Butcher, E. (2017). Outcome performance measure development for persons with multiple chronic conditions. RAND.
8. Staggs, V. S., & Cramer, E. (2016). Reliability of Pressure Ulcer Rates: How Precisely Can We Differentiate Among Hospital Units, and Does the Standard Signal-Noise Reliability Measure Reflect This Precision?. Research in nursing & health, 39(4), 298–305. https://doi.org/10.1002/nur.21727
9. HEDIS, N. (2007). Technical Specifications for Physician Measurement. Washington DC: National Committee for Quality Assurance.
10. Safran, D. G., Karp, M., Coltin, K., Chang, H., Li, A., Ogren, J., & Rogers, W. H. (2006). Measuring patients' experiences with individual primary care physicians: results of a statewide demonstration project. Journal of General Internal Medicine, 21(1), 13-21.
Reliability analyses were conducted at two levels (facility and clinician group/practice), stratified by three age categories (15-20, 21-44, and 15-44 years). More detailed information including reliability estimates for each unit at each level for HealthEfficient can be found in Section 5.2.3a, Tables 5-10. Detailed reliability results at the unit level for HCCN 1 are not included due to the large number of units.
Our tested reliability is consistently greater than 0.60 at the facility and clinician group/practice levels when using the patient size threshold of 50, showing adequate reliability at these levels.
For the most or moderately effective methods, reliability ranges from 0.854 to 1 for HCCN 1 and 0.985 to 0.994 for HealthEfficient for the 15-44 years age group at the facility level, when using the threshold of 50 patients per unit. At the clinician group/practice level, reliability ranges from 0.849 to 0.999 for HCCN 1 and 0.817 to 0.991 for HealthEfficient for the 15-44 years age group. HealthEfficient has a small number of units (n=3) at the facility level.
For the long-acting reversible contraceptive (LARC-SINC) submeasure, reliability ranges from 0.647 to 0.999 for HCCN 1 and 0.971 to 0.988 for HealthEfficient for the 15-44 years age group at the facility level, when using the threshold of 50 patients per unit. At the clinician group/practice level, reliability ranges from 0.640 to 0.996 for HCCN 1 and 0.598 to 0.975 for HealthEfficient for the 15-44 years age group.
Our results also show that with the minimum threshold of 50 patients, reliability improves greatly at the clinician group/practice level, compared to not using a threshold. This is because at this level patient size can be small and likelihood of units less than 50 is high. Measure developers frequently recommend the minimum patient size approach for performance and our analysis suggests that a minimum of 50 patients yields sufficient reliability for our measure.
See Section 5.2.3a Tables 11-18 for reliability tables by decile.
Our tested reliability is consistently greater than 0.70 at the facility and clinician group/practice levels for all age groups, showing adequate reliability at these levels. Overall, testing results showed that the Contraceptive Use electronic clinical quality measure (eCQM) (CU-SINC), Non-Postpartum measure, as currently specified, can distinguish the true contraceptive use performance in facilities and clinician groups/practices from one entity to another.
Validity
Data element level validity
For HCCN 1, a total of 150 female patients aged 15-44 years in 2023 were randomly sampled from eight sites. Due to the low volume of patients who had a birth in HCCN 1, we oversampled eight additional women who had an estimated delivery date documented in their medical record, making a total of 158 women for this element. For HealthEfficient, a total of 144 female patients aged 15-44 years in 2023 were randomly sampled from five sites. For each of these patients, data elements used for the CU-SINC, Non-Postpartum calculation were compared between the EHR records and the patient charts, and agreement numbers were summarized in a two by two table (yes/yes, yes/no, no/yes, and no/no) for each element. We compared 14 different data composites/fields reflecting combinations of multiple data elements to capture core constructs of the measures CU-SINC, Non-Postpartum, such as use of contraceptive methods and exclusion conditions: female sterilization, implants, intrauterine devices, injectables, contraceptive pills, contraceptive patch, vaginal ring, infecundity, had a qualifying encounter, SINC response, had a prenatal visit, had a live birth delivery date, had an estimated delivery date, and had a non-live birth. Using the patient chart as the authoritative source, we calculated sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) for each data element.
Empirical validity testing
We tested validity of the CU-SINC, Non-Postpartum by exploring whether they were correlated with other similar quality measures listed below:
- Cervical cancer screening (CMS124v12): Percentage of women ages 21 to 44 years who were screened for cervical cancer using either of the following criteria:
- Women ages 21 to 44 who had cervical cytology performed within the last three years;
- Women ages 30 to 44 who had cervical human papillomavirus (HPV) testing performed within the last three years.
- Encounter for contraceptive counseling: Percentage of women ages 15 to 44 years who received any contraceptive counseling during the measurement year.
The original cervical cancer screening specification includes women ages 21 to 64 years. We restricted the calculation of this measure to be among women ages 21 to 44 years in order to match with the age range of the CU-SINC, Non-Postpartum. The original specification also included women who had HPV test during the last five years (instead of three years). For both our data partners, we only had three years of data and thus were not able to include the look back period of five years as originally specified. The measure numerator only included women who received service during the three years.
Our analysis of correlation with cervical cancer screening is based on the hypothesis that facilities/groups that perform well on meeting patients’ contraceptive need should perform well on other reproductive health services. As cervical cancer screening is a core reproductive health need for women of reproductive age, measures of this screening should be positively correlated to the CU-SINC, Non-Postpartum. We hypothesize that the magnitude of correlations will be weak to moderate due to the difference in recommended screening frequency and target population for this measure compared to the CU-SINC, Non-Postpartum. Our analysis of correlation with contraceptive counseling is based on the hypothesis that provision of this counseling is more common among facilities/groups that are generally meeting their patients’ contraceptive needs, as captured in the CU-SINC, Non-Postpartum. We expect a moderate and positive correlation between CU-SINC, Non-Postpartum and contraceptive counseling.
We performed correlation analyses at the facility and clinician group/practice levels for HCCN 1. For HealthEfficient, due to their small number of facilities (n=3), we only conducted the analysis at the clinician group/practice level. In addition, we used a cutoff of 50 eligible patients per unit to exclude facilities and clinician groups/practices who served only a small number of patients during the measurement year. As described in section 1.26, patient denominator minimums ensure that practices/facilities are large enough to have an adequate volume of patients across the year for consistent reporting. We are not reporting results for the 15-20 years age group for HealthEfficient at the clinician group/practice level due to their small number of units after applying the cutoff of 50 patients per unit (n=9).
To test these correlations, we used two different approaches.
In the first approach, we used a Pearson’s correlation test. This test estimates the strength of the linear association between two continuous variables. The correlation coefficient ranges from -1 to +1. A value of 1 indicates a perfect positive linear correlation between two variables. A value of 0 indicates no linear association. A value of -1 indicates a perfect negative linear relationship between two variables. We used a threshold of p < .05 to evaluate the statistical significance of test results.
Even though Pearson’s correlation test is widely used to evaluate the correlation between two measures, it is only optimal in cases where linearity can be assumed. Crucially, the bounded nature of the variation in the proportion of CU-SINC Non-Postpartum (i.e., 0 and 1) means that estimates of association that assume linearity on the measure rates may be biased. This is a particular concern when the count of service events is either very high or very low relative to the total number of patients in a cluster. In addition, the correlations captured by the Pearson correlation matrix are averaged over the “true” and error variances. As a result, Pearson's correlation could downwardly bias the correlation substantially in cases when the clusters are small with few patients and where the measurement error is high.
Given these limitations with Pearson’s correlation test we present a novel alternative approach. We employ a multilevel correlation estimation method to test the relationship between the CU-SINC, Non-Postpartum and the related measures. The model is based on a multivariate generalized linear mixed model framework (1). By employing a logit transformation of the binomial proportions, the model relaxes the linearity assumption on the original measurement scale. In addition, it analytically separates “true” score variance from measurement error by presenting measurement error as a random, binomial deviate, conditional on each cluster’s “true” quality measure. Thus, the multilevel correlation estimation approach captures the correlation more accurately when the cluster size is small.
In the present analyses, the parameters of the multilevel model were estimated using a hierarchical SAS 9.4 GLIMMIX procedure with a log link function and fully unstructured residual error. Parameters were estimated by pseudo-maximum-likelihood using the Laplace method. The error structure was reported as correlation coefficients and variances. We are also able to provide 95% confidence limits for the estimates using likelihood bounds, which is far more informative than the single p-value for statistical significance. Rather than estimating all possible pairwise associations simultaneously, we estimated each pairwise association in a separate model to speed up and improve model convergence. In the appendix of the application, we provide a detailed description of the model with example statistical programing code.
REFERENCE
1. Coull, B. A., & Agresti, A. (2000). Random effects modeling of multiple binomial responses using the multivariate binomial logit‐normal distribution. Biometrics, 56(1), 73-80.
Results of the data element validity analyses are shown in Section 5.3.4a Tables 1-2. Results of the empirical validity testing are shown in Section 5.3.4a Tables 3-8.
Data element level validity
Among both HCCNs, sensitivity, specificity, PPV, and NPV were above 0.7 for the majority of the data elements.
For both HCCNs, we found that the patient charts captured a small number of sterilization cases (five in HCCN 1 and three in HealthEfficient), whereas standardized medical codes for sterilization were not found in the EHR data extract. This is largely because the CHCs from where the data was extracted are outpatient centers and do not provide sterilization procedure themselves. When documenting sterilization that has occurred in other sites, this can be documented in the surgical history section of their EHR systems using unstructured fields (as opposed to using ICD-10 codes to document this in the problem list).
For HealthEfficient, we found one vaginal ring that was observed in the patient charts but not in the EHR data extract, resulting in sensitivity of 0 for this element. Given the low use of vaginal ring, this validity testing result for this element is likely unreliable and we don’t expect it to have a meaningful impact on our measure calculation.
For HCCN 1, we also found three live birth dates and 1 non-live birth that were observed in the patient charts but not in the EHR data extract. This is due to the same reason as sterilization in that pregnancy outcome information was often entered into the system as history records, instead of standardized medical terminologies such as CPT or ICD codes. This level of discrepancy for these elements is expected in outpatient settings.
To compensate for the possibility of missing live birth date information in the outpatient settings, we specify our measure to use either live birth date or estimated delivery date to capture live births. For HCCN 1, even though the sensitivity for live birth date was low (0.4), estimated delivery date had a higher sensitivity (0.73). There were three estimated delivery dates that were observed in the patient charts and not in the EHR data extract, but two out of these three patients did not have any prenatal care visits. The measure specification requires the presence of both prenatal care visit and a live birth date/estimated delivery date for a woman to be excluded from the denominator. Therefore, these two mismatched estimated delivery date records would not have impacted the measure calculation; only one out of the 158 sampled records had a true mismatch on estimated delivery date that would have impacted the measure score to a small degree. The combined use of these two elements allows us to more reliably capture women who have had a live birth.
Overall, our data provide fairly strong evidence for validity of CU-SINC, Non-Postpartum at the data element level.
Empirical validity testing
Coefficients with absolute values of less than 0.3 are generally considered indicative of weak associations whereas absolute values of 0.3 or higher denote moderate to strong associations (1). Using the multilevel correlation estimation method, we observed weak to moderate positive correlations between the CU-SINC, Non-Postpartum and cervical cancer screening for HCCN 1 at both facility and clinician group/practice levels for most or moderately effective methods, while for HealthEfficient this correlation was positive but not significant. For contraceptive counseling, we found moderate to strong positive correlations for most or moderately effective methods at both levels for all age groups in both HCCN 1 and HealthEfficient. Pearson’s correlation test showed similar positive correlations but often at weaker magnitudes and were non-significant for some sub-age groups. The LARC submeasure had significant, if less strong, association with contraceptive counseling in both HCCN 1 and HealthEfficient, with weak associations with cervical cancer screening in HCCN1 and no association in HealthEfficient. This lack of association with HealthEfficient is possibly due to the small number of units at this level for this data partner that limited the capacity to capture true correlations.
The magnitude of correlation was generally weaker using Pearson’s correlation, as expected, because the distributional assumptions of this method are a poor fit to binary outcomes, resulting in underestimation. We demonstrate that our generalized linear multilevel estimation more closely captures the “true” correlation between two measures and is better suited for binary outcomes.
Overall, we observed positive correlations between the CU-SINC, Non-Postpartum and those services that (in theory) should be related (cervical cancer screening and contraceptive counseling). These were highly consistent with our hypotheses and provided good evidence for validity of the CU-SINC, Non-Postpartum at the score level.
REFERENCE
1. Ratner, B. (2009) The correlation coefficient: Its values range between +1/−1, or do they?. J Target Meas Anal Mark 17, 139–142. https://doi.org/10.1057/jt.2009.5
Risk Adjustment
We do not believe that risk adjustment is justified. Our measure is explicitly designed to measure contraceptive use among those who have an interest in this use, as determined by the use of the SINC question to refine the denominator. Therefore, variations in contraceptive use by socio-demographic characteristics in this population will exist as a result of modifiable clinical and programmatic considerations, and not patient preferences. These differences will be reduced if contraceptive services are offered in a patient-centered manner, as promoted by Centers for Disease Control and Prevention (CDC) and the Health and Human Services Office of Population Affairs (OPA) (1-4).
In this application, we present CU-SINC, Non-Postpartum scores by age group so that adolescents (ages 15-20 years) can be examined separately from adult (ages 21-44 years) patients for the purposes of quality improvement. Though their current clinical guidelines report that most and moderately effective contraceptive methods are safe and recommended for teen and nulliparous populations who wish to use them, the American Academy of Pediatrics (AAP), ACOG, CDC, and OPA note that it can still be difficult for these populations to access these contraceptive methods (1-6); thus, it is important for facilities and clinician group/practices that want to improve the quality of their contraceptive services for adolescents and track progress to calculate the CU-SINC, Non-Postpartum measure rates by age group. We also utilize age groups that align with the OPA’s claims-based Contraceptive Care measures (CBE #2903 and #2904).
REFERENCES
- Gavin, L., Moskosky, S., Carter, M., Curtis, K., Glass, E., Godfrey, E., Marcell, A., Mautone-Smith, N., Pazol, K., Tepper, N., Zapata, L., & Centers for Disease Control and Prevention (CDC) (2014). Providing quality family planning services: Recommendations of CDC and the U.S. Office of Population Affairs. MMWR. Recommendations and reports: Morbidity and mortality weekly report. Recommendations and reports, 63(RR-04), 1–54.
- Gavin, L., & Pazol, K. (2016). Update: Providing Quality Family Planning Services - Recommendations from CDC and the U.S. Office of Population Affairs, 2015. MMWR. Morbidity and mortality weekly report, 65(9), 231–234. https://doi.org/10.15585/mmwr.mm6509a3
- Gavin, L., Pazol, K., & Ahrens, K. (2017). Update: Providing Quality Family Planning Services - Recommendations from CDC and the U.S. Office of Population Affairs, 2017. MMWR. Morbidity and mortality weekly report, 66(50), 1383–1385. https://doi.org/10.15585/mmwr.mm6650a4
- Romer, S. E., Blum, J., Borrero, S., Crowley, J. M., Hart, J., Magee, M. M., Manzer, J. L., & Stern, L. (2024). Providing Quality Family Planning Services in the United States: Recommendations of the U.S. Office of Population Affairs (Revised 2024). American journal of preventive medicine, 67(6S), S41–S86. https://doi.org/10.1016/j.amepre.2024.09.007
- Committee Opinion No. 710: Counseling Adolescents About Contraception. (2017). Obstetrics and gynecology, 130(2), e74–e80. https://doi.org/10.1097/AOG.0000000000002234
- Chung, R. J., Lee, J. B., Hackell, J. M., Alderman, E. M., Committee on Adolescence, & Committee on Practice and Ambulatory Medicine (2024). Confidentiality in the Care of Adolescents: Policy Statement. Pediatrics, 153(5), e2024066326. https://doi.org/10.1542/peds.2024-066326
Use & Usability
Use
Usability
Health center agencies and networks can use the measure to assess whether eligible patients of reproductive age who want contraception are accessing methods. We tested usability in a pilot application of this measure in ten community health centers (CHCs) across the United States. This pilot occurred in the context of an intervention to increase access to person-centered contraceptive care. Contraceptive Use electronical clinical quality measure (eCQM) (CU-SINC), Non-Postpartum scores were calculated before and after the intervention. In this context, participating CHCs did report an increase in scores tied to implementation of evidence-based strategies to improve the quality of contraceptive services. For example, one CHC saw a relative increase in scores of 18.0% (23.3% to 27.5%). This was associated with quality improvement efforts to expand contraceptive care screening and provision into Primary Care, as opposed to only in on the Women’s Health department. Another CHC reported a relative increase in scores of 61.1% (from 18.5% to 29.8%) by standardizing the Self-Identified Need for Contraception (SINC) screening into their primary care workflows and requiring providers to participate in a standardized trainings on person-centered contraceptive counseling and provision. Barriers to improvement included staff and clinician turnover, suboptimal integration of SINC into the electronic health record (EHR) and clinical workflows, change in EHR vendors and platforms, provider lack of training or comfort with contraceptive counseling and resistance to use of SINC due to competing priorities.
Extrapolating from these examples, enhancing performance on this measure includes two key components: 1) optimizing the denominator by standardizing screening for contraceptive need, which will limit the denominator to the desired population of eligible patients who want contraception, while ensuring that patients are being asked about their needs, and 2) optimizing surveillance (i.e. documentation of use) and provision of the full range of contraceptive methods. These two areas represent two intervention points along the reproductive health care pathway for patients.
To enhance screening for contraceptive need, entities would develop clear and standard workflows adapted to optimize SINC screening in the health care setting, including how to identify eligible patients for routine screening, identifying at which point and by whom SINC will be asked, and working with EHR vendor to optimize SINC location in the health record. As part of the aforementioned intervention with CHCs, sites were trained by experts the National Family Planning and Reproductive Health Association (NFPRHA) in how to integrate SINC into clinical workflows. During this training, a NFPHRA expert provided quality improvement strategies and workflow development support. They also provided participants with several templates and worksheets to help sites more easily integrate contraceptive screening into their clinical workflow, including sample workflow diagrams, process mapping flowcharts, tally sheets, and policy and procedure templates. Sites were able to utilize these templates to standardize their procedures; example activities from sites include integrating SINC into the rooming tab of their EHR, training medical assistants to record SINC responses during rooming, and programming SINC into check-in modules. We received feedback from sites that having these materials and training greatly helped teams seamlessly integrate SINC into their existing workflows, which will improve performance on the CU-SINC, Non-Postpartum (CBE #3699e) through both refining the denominator and improving identification of patients need for contraception. This is also aligned with performance enhancement for the Contraceptive Care Screening eCQM (CBE #4655e).
To enhance performance on this measure in regard to contraceptive provision and surveillance, entities would prioritize comprehensive clinician and staff training both on the importance of routine contraceptive care and contraceptive counseling skills as appropriate. CHCs that participated in our pilot project who identified deficits in provider comfort and training provided on-site trainings both on person-centered contraceptive counseling practices and specific skill trainings, such as intrauterine device insertion and removal. To enhance EHR documentation of patient contraceptive use, particularly among patients who have received contraception from another healthcare location, health care settings can standardize procedures and fields for collection of patient-reported contraceptive use. These strategies are further described on our quality improvement website (1).
REFERENCES
1. Person-Centered Reproductive Health Program. Innovating Contraceptive Care in Community Health Centers [Internet]. 2022. Available from: https://innovatingcontraceptivecare.ucsf.edu/
To develop CU-SINC, Non-Postpartum (CBE #3699e), UCSF solicited a wide range of expert input, including convening three stakeholder panels to discuss how to optimize the measure specifications to capture the desired measure of the extent to which patient’s contraceptive needs are being addressed. In addition, we had a series of meetings with Dr. Joia Crear-Perry from National Birth Equity Collaborative and Dr. Jamila Perritt from Physicians for Reproductive Health to provide a Reproductive Justice and race-equity informed perspective on the measure development. Their consultation particularly informed the development and wording of SINC as a means to refine the denominator of the measure and decrease the potential to incentivize directive counseling towards individuals who are not interested in receiving contraceptive care. The inclusion of SINC in CBE #3699e increases the patient-centeredness of our measure and contributes to patient-centered workflows to identify and meet patients’ reproductive health needs. Given the documented disparities in reproductive health counseling experienced by patients of color (1–4), utilization of the SINC data element is also consistent with attention to race equity and health care disparities.
In addition, we participated in expert workgroup meetings to discuss measurement of contraceptive provision and care with measure users. This includes meetings held by the Health and Human Services Office of Population Affairs (OPA) every other year and a semi-annual independent National Contraceptive Quality Measures Workgroup convened by the Coalition to Expand Contraceptive Access (CECA), Planned Parenthood Federation of America (PPFA) and NFPRHA. These workgroups bring together representatives from the Centers for Medicare and Medicaid Services (CMS), OPA, and PPFA, as well as relevant stakeholders and subject matter experts, including direct care providers, who could use CU-SINC, Non-Postpartum to assess contraceptive use and long-acting reversible contraception (LARC) provision in their respective health systems. In these meetings, we received widespread support for the measure and input into its specifications, including around the necessity to refine the denominator.
We continue to participate in, and solicit feedback from, these expert working groups. Feedback on CU-SINC, Non-Postpartum received from these meetings focus primarily on score interpretation, ensuring that the performance measure does not inadvertently incentivize practices that compromise reproductive autonomy and health equity goals. For example, at each meeting we discuss ensuring that measure guidance highlights that the primary and sub- measure should not be used in the context of pay-for-performance.
CBE #3699e was approved for Trial Use in 2022. However, validity and reliability testing were necessary prior to moving toward full implementation and reporting in health care systems across the United States, including by federal agencies. Thus, there has not been widespread uptake of CBE #3699e at this time. We anticipate wider uptake of CU-SINC, Non-Postpartum, and thus, additional feedback by the next maintenance cycle. We piloted the use of CBE #3699e in ten CHCs across two Health Center Controlled Networks (HCCNs) through a quality improvement learning collaborative. Feedback on measures from CHC partners was focused on SINC integration into clinic workflows and were overall supportive of SINC being used to refine the denominator of CBE #3699e to better capture contraceptive access among the appropriate population. Regarding the measure specifications, EHR programmers and analysts at our HCCN partners requested clarification of the codes and data sources to include to generate the measure as part of our technical assistance. This included, for example, ensuring records of contraception provided at external health centers was captured and included in structured EHR fields. With this additional support, the networks were able to generate the requested codes and calculate the measure.
REFERENCES
1. Downing RA, LaVeist TA, Bullock HE. Intersections of Ethnicity and Social Class in Provider Advice Regarding Reproductive Health. Am J Public Health. 2007 Oct;97(10):1803–7.
2. Becker D, Klassen AC, Koenig MA, LaVeist TA, Sonenstein FL, Tsui AO. Women’s Perspectives on Family Planning Service Quality: An Exploration of Differences by Race, Ethnicity and Language. Perspectives on Sexual and Reproductive Health. 2009 Sep;41(3):158–65.
3. Borrero S, Schwarz EB, Creinin M, Ibrahim S. The Impact of Race and Ethnicity on Receipt of Family Planning Services in the United States. Journal of Women’s Health. 2009 Jan;18(1):91–6.
4. Dehlendorf C, Ruskin R, Grumbach K, Vittinghoff E, Bibbins-Domingo K, Schillinger D, et al. Recommendations for intrauterine contraception: a randomized trial of the effects of patients’ race/ethnicity and socioeconomic status. Am J Obstet Gynecol. 2010 Oct;203(4):319.e1-8.
Through our participation in the CECA Technical Expert Panels, National Contraceptive Quality Measures Workgroup, and OPA Expert Work Group, UCSF solicited and received input on how to incorporate patient choice more directly into an eCQM of contraceptive use and LARC provision. The feedback obtained from these workgroups contributed to UCSF finalizing the following substantive decisions regarding our eCQM specifications prior to submitting the measure for CBE Approval for Trial Use endorsement:
- Incorporation of the SINC data element to account for patient choice in the CU-SINC, Non-Postpartum. We developed SINC in the context of routine care and considered the use of One Key Question® (OKQ®) as an alternative. OKQ® asks patients whether they wish to get pregnant in the next year and has been proposed as a means of identifying patients in need of contraceptive services by excluding those desiring pregnancy (1). However, through discussion with stakeholders, we determined OKQ® was not optimal because it does not identify patients who desire contraception and/or pregnancy prevention at the current time (which is not incompatible with desiring pregnancy in a year).
- Definition of the postpartum period to 90 days after live birth delivery (as opposed to 60 days) for an exclusion criterion aligned with the American College of Obstetrician & Gynecologists (ACOG) postpartum care guidelines (2). A 90-day window also provides a greater amount of time to meet a patient’s contraceptive needs in the postpartum period. As CBE #3699e excludes the postpartum population (i.e., those included in CBE #3682e), this expansion of the postpartum period needed to be integrated into the specifications.
- Utilization of both live birth delivery date and estimated delivery date (EDD) as exclusion criteria for CBE #3699e. Because delivery date is not always available in the electronic health record of the site of prenatal care, CBE #3699e also uses EDD as entered into the prenatal care record to exclude postpartum patients from the CU-SINC, Non-Postpartum denominator.
Feedback from experts and health center partners during and after being approved for Trial Use and piloting the measures led to the following updates to implementation guidance and specifications:
- Clarification in specifications that all records of active contraceptive use should be included in the measure, even in the case of self-report from a patient where contraceptive method has been obtained elsewhere.
- Options for modifying SINC implementation guidance for specific populations, including young people and use in sexual and reproductive healthcare-specific settings.
- Addition of surveillance codes for sterilization to the measure specifications as a means of capturing previous sterilization as a method of contraceptive use for inclusion in both the numerator and denominator of the measure. This serves to optimize the measure, ensuring that ongoing reliance on sterilization for pregnancy prevention as a contraceptive method in use.
- Extension of timeframe for capture of infecundity. The infecundity codes used for exclusion were originally only included if notated in the measurement year. This was expanded to include the year prior to the measurement year to better capture surveillance of documented infecundity.
Additional specification updates made since receiving Approval for Trial Use designation to optimize and improve the measure:
- Updated the timeframe of data elements that determine pregnancy-related exclusions (live birth delivery date, EDD, prenatal care visits, pregnancy-related diagnosis) from three months prior through the end of the calendar year of the measurement period to three months prior through the first nine months of the calendar year. This serves to ensure that patients will not be double counted between measurement periods.
- Two additional value sets were added to the definition of a qualifying encounter to align with current CMS eCQM specifications that use qualifying encounters.
REFERENCES
1. Power to Decide. One Key Question online [Internet]. Available from: https://powertodecide.org/one-key-question
2. ACOG Committee Opinion No. 736: Optimizing Postpartum Care. Obstet Gynecol. 2018 May;131(5):e140–50.
As widespread uptake of the measure has not yet occurred due to our previous Approval for Trial Use designation, our experience is limited to our pilot sites. We piloted CU-SINC, Non-Postpartum with nine pilot CHCs from seven states across three geographic regions, who utilized CU-SINC, Non-Postpartum in the context of a year-long quality improvement learning collaborative to advance person-centered contraceptive care. Most provided primary care services, with one CHC focused specifically on sexual and reproductive healthcare. Median baseline contraceptive use percentage was 23.3% (range: 12.9% - 72.5%) (number of patients represented: 42,560). At endline, the median score increased to 29.8% (range: 13.00% - 76.8%) (number of patients represented: 20,896).
As widespread uptake of the measure has not yet occurred due to our previous Approval for Trial Use designation, our experience is limited to our pilot sites. From this limited data, we do not have any unexpected findings.
Comments
Staff Preliminary Assessment
CBE #3699e Staff Preliminary Assessment
Importance
Strengths
- A clear logic model is provided, depicting the relationships between inputs (e.g., integrating Self-Identified Need for Contraception [SINC] into electronic health records, training protocols for clinical care team on contraceptive counseling), activities (e.g., analysis of contraceptive use and Long-Acting Reversible Contraception [LARC] provision among patients interested in contraceptive counseling during the calendar year), and desired outcomes (e.g., increased rates of contraceptive counseling with patients and increased access to contraceptive services and use of preferred methods by patient). This model demonstrates how the measure’s implementation will lead to the anticipated outcomes.
- The problem this measure addresses presents a significant burden for patients. The developer cites survey data to support the claim that less than half of women at risk of unintended pregnancy (46%) do not use any form of contraception, with 19% of women reporting that they do not use contraception because they could not access a method. The developer also highlights, from the same survey data from 2015, only 47.9% of pregnancies were categorized as occurring as desired at the time for the individual.
- The proposed measure addresses a healthcare need not sufficiently covered by existing measures (e.g., CBE #2902, #2903, and #2904 that use claims data), offering advantages in terms of expanding health care settings and using electronic health record data.
- At the facility-level, deciles for the primary measure (most or moderately effective methods) ranged from 8.4% to 50.1% and from 0.9% to 12.7% for the sub-measure (LARC methods) based on 2023 data for 193 facilities in health care-controlled network (HCCN) 1. Data from three facilities in a second network did not include decile information, but reported a performance score range from 22.4% to 44.7% for the primary measure and 4.5% to 14.2% for the sub-measure.
- At the clinician group/practice level, deciles ranged from 5.8% to 57.3% for the primary measure and from 0.1% to 15.0% for the sub-measure based on 2023 data from 991 clinician group/practices. Decile data from 15 clinician group/practices in a second network ranged from 9.2% to 42.2% for the primary measure and from 8.9% to 9.6% for the sub-measure.
- Performance gap data for the primary measure shows variation and less than optimal performance across the target population, indicating a gap in care.
Limitations
- The measure submission could be strengthened with discussion of the evidence showing interventions entities can implement to improve on the measure focus.
- The submission does not specify if the American College of Obstetricians and Gynecologists (ACOG) clinical practice guidelines referenced are graded, and the guidelines are not cited or discussed with sufficient quantity or specificity to support the claim that access to contraceptive methods supports patients in achieving reproductive goals.
Rationale
- This maintenance measure is rated as 'Not Met But Addressable' due to incomplete evidence and insufficient information regarding the ACOG guideline. Enhancements, including review of evidence that explores actions entities can take to improve (such as those listed in the logic model) and more complete information about the guideline could elevate its importance.
Closing Care Gaps
Strengths
- The developer provided evidence of gaps in care related to the measure focus for subgroups, including qualitative studies.
- The measure’s performance was empirically tested across all of the identified subgroup variables including race, ethnicity, and age.
- The analysis revealed significant differences in performance scores by race and ethnicity. For example, Black/African American patients had a significantly lower measure rates compared to White patients. And compared to non-Hispanic patients, Hispanic patients had a significantly lower measure rates for the primary measure.
Limitations
- The submission could be strengthened with more detail about the recommended actions entities can take to close care gaps, such as targeted quality improvement initiatives or policy changes.
- The developer recognized the limitations of the ethnicity subcategories in their trial due to the large number of patients in the “unreported/refused to report” groups.
Rationale
- The measure sufficiently assesses gaps in care by race, ethnicity, and age, and provides examples of how accountable entities can use this measure to improve differences in care for these subgroups. This includes using the findings as an indicator of an equity issue which warrants further exploration of possible differential contraceptive counseling practices towards different racial, ethnic, and age groups and ensuring that all patients who have the potential for pregnancy have their contraceptive needs assessed.
Feasibility Assessment
Strengths
- The developer completed an eCQM Scorecard showing the measure was tested in at least two EHRs, eClinicalWorks and EPIC.
- The developer described the costs and burden associated with data collection and data entry, validation, and analysis. They discussed barriers that were encountered in implementing or reporting the measure, which include EHRs using varying coding systems and partners changing their EHR templates to incorporate SINC screening questions into the primary care workflow. They also noted mitigation facilitators, such as using multiple overlapping terminologies to ensure comprehensive data capture and providing technical assistance to overcome the burden identified.
- The developer described how all required data elements can be collected without risk to patient confidentiality, including the housing of data elements within a secure EHR system server, and implementing patient denominator minimums (50 or more patients) to protect patient confidentiality.
- There are no fees, licensing, or other requirements to use any aspect of the measure (e.g., value/code set, risk model, programming code, algorithm).
Limitations
- The data required for the measure are partially available in a structured format within the tested EHR systems (eClinicalWorks and EPIC); however, across the four domains up to 33% of data elements score 0 in eClinicalWorks and up to 17% scored 0 in EPIC. For example, the developer reported that HealthEfficient (using eClinicalWorks) had 24 data elements scoring 0 for the Workflow domain while HCCN 1 (using EPIC) had 15 elements scoring 0 for Workflow, suggesting that these elements are not routinely collected during clinical care as the systems do not use SNOMED CT. The developer indicated that the unavailable data elements do not impact the ability to calculate the measure as the measure specifications were developed to account for the fact that different EHR systems use multiple combinations of code systems. They provided a feasibility plan detailing how to account for all data elements that scored 0 (e.g., capturing the same information through ICD-10-CM codes).
Rationale
- This maintenance measure meets all criteria for “Met” due to its well-documented feasibility assessment, clear and implementable data collection strategy, a description of adjustments made to specifications to account for varying EHR systems and availability of codes, and transparent handling of patient confidentiality. The eCQM Feasibility Scorecard results indicate that the measure is mostly supported by the current capabilities of the tested EHR systems and provides a plan to readdress missing data elements. These factors collectively ensure that the measure can be implemented effectively and sustainably in a real-world healthcare setting.
Scientific Acceptability
Strengths
- The developer clearly explains their rationale for using what they describe as a modified Empirical Bayes approach to estimate accountable entity-level reliability. The method they describe is equivalent to the Nieser-Harris adjustment to the Adams (2009) signal-to-noise method. Data used for testing were collected over calendar year 2023.
For the clinician group/practice and facility-level, the reported reliability was greater than the 0.6 threshold for all but one entity that met the minimum sample size of 50 for both measures of interest.
The developer reported reliability for all entities and for entities that met the minimum sample size of 50. Reliability was reported as much higher for entities that met the minimum sample size requirement. This demonstrates the need for the minimum sample size to get reliable results.
Limitations
- Of the 42 entities in the clinician group/practice level from the HealthEfficient network, 14 had more patients than the minimum sample size of 50. Fifteen other entities each had ten or fewer patients.
Rationale
- The developer performed accountable-entity level reliability testing using data collected within the last two years. Testing was done at the clinician group/practice and facility-level and all but one entity that met the minimum sample size of 50 had a reported reliability estimate above the threshold of 0.6 for both measures of interest.
Strengths
- Validity: The developer provides an Importance Table and Logic Model, providing a plausible causal association between the entity response to the measure and the measure focus. Empirical support for ruling out confounders includes adequate reliability (N>50) and a correlation with a related process measure with construct overlap (Cervical cancer screening, Contraceptive Counseling). The developer provides a conceptual rationale for the construct overlap, and most importantly for the absence of overlap: Empirical support for ruling-in responsible mechanisms includes several empirical studies and reports (e.g. Use of Person-Centered Screening (SINC), Provision of a Full Range of Methods, Standardized Workflow Integration, Provider Training on Person-Centered Counseling, Community Engagement and Co-Design, Standardization of Documentation and Surveillance, Performance Feedback and Technical Assistance).
Limitations
- Validity: Residual risk for confounders includes lack of a entity level reliability testing that cannot rule-out change as a confounder. There is also no risk adjustment for an intermediate outcome. Residual risk for confounders includes Patient preference for non-prescription methods, Previous receipt of contraceptive care at other sites or time points, Mismatch between patient-reported need and documentation practices, Differences in population composition across sites, and Varying patient visit types and clinical priorities. Residual risk for a responsible mechanism includes the potential counter-acting mechanisms (Integration of the SINC question into EHR workflows, Availability of the full method mix (including LARC), Use of standardized templates and clinical workflows, Participation in technical assistance and QI collaboratives, EHR system capability to capture structured data).
- Risk Adjustment: The developer did not conduct risk or case-mix adjustment, providing the rationale that variations in contraceptive use by socio-demographic characteristics will exist as a result of modifiable clinical and programmatic considerations and not patient preferences. The developer did not address potential impact of access factors such as insurance status on the measure score. The developer states that measure scores are stratified by age group in the submission due to differences in access to contraceptive methods, but it is unclear in the submission language if the measure is intended to be stratified in use.
Rationale
- The measure developer provides some support for the causal claim that the entity response to the measure is causally related to the measure focus. The developer provides empirical support for ruling out confounders (always with some residual risk of unstated or unexamined confounders) and for ruling in responsible mechanisms (always with some residual risk that the explicit mechanisms are only partially responsible for the measure focus). Conditions might include a more quantitative comparison of the overlap in mechanisms involved in the related process measures, and how the degree of commonality helps to explain the degree of correlation.
- While this domain is rated 'Met', the following limitations for risk adjustment were noted: The developer did not conduct risk/case-mix adjustment; developer provided a rationale to address social and demographic characteristics but not potential risks due to access. The developer states the importance of calculating measure score by age group, but it is unclear in current submission language if the measure is intended to be stratified by age group in use.
Use and Usability
Strengths
- The developer provides a summary of how accountable entities can use the measure results to improve performance. Specifically, 1) optimizing the denominator by standardizing screening for contraceptive need and 2) optimizing surveillance and provision of the full range of contraceptive methods. The latter example is reflected in the measure’s logic model.
- The developer collects feedback via stakeholder panels, expert workgroup meetings, consultation with physicians, and from the community health centers (CHCs) after trial use. The developer incorporated the feedback received into the measure submission (e.g. expanding the postpartum period from 60 to 90 days, giving more time to meet a patient’s contraceptive needs and using both live birth delivery date and estimated delivery date as exclusion criteria).
Limitations
- The measure is not currently in use, but the developer indicated a plan for use in the Health Resources & Services Administration (HRSA)’s Uniform Data System (UDS) reporting program within one year.
- Although the measure is not currently in use, the developer provided a progress update from nine pilot community health centers who used CU-SINC Non-Postpartum for a year-long quality improvement learning collaborative. They shared an increase of contraceptive use from 23.3% to 29.8% (number of patients: 20,896).
- Since the measure is not in use, the limited data from the pilot sites are not sufficient to describe any unexpected findings.
Rationale
- The measure is not currently in use, however there is a plan for use in at least one accountability application. They provide a limited, preliminary positive trend in performance from their pilot sites; however, they could not expand on unexpected findings. Despite the measure not being currently in use, the developer gathered feedback from trial use and other stakeholder groups and incorporated that feedback into the measure specifications and guidance. They also specify actions accountable entities can do to improve performance, based on the trial implementation.
Committee Independent Review
#3699e
Importance
The logic model could be enhanced by addressing issues like cultural sensitivities and health literacy, which are crucial for effective contraceptive provision and use. An overemphasis on LARC (Long-Acting Reversible Contraception) might inadvertently detract from a patient-centered counseling approach. The model could benefit from more detailed discussions regarding the types of QI interventions that can be effectively utilized.
Closing Care Gaps
There was data from the measure indicating disparities in effective contraceptive use. More details around what specifically could be done to improve this measure score, and with attention to cultural sensitivities would be beneficial, as well as perhaps addressing concerns around the political climate.
Feasibility Assessment
While the data did indicate testing in two EHR system, there do seem to be some challenges related to how easy it was for some of the health centers to implement this measure that the developer outlined such as lack of all the code systems, and challenges with adding a new data element into the EHR. There was a feasibility assessment that helped account for these issues.
Scientific Acceptability
Data from 13 total sites were used, and the reliability ranges met the threshold at both the facility and CGP level. Of note, the sample size was lower at some of the entities.
While conceptual validity appears to exist with the related process measure, potential confounders may not be adequately accounted for. Furthermore, access issues, such as insurance, require attention.
Use and Usability
The measure is not currently in use. More data from the pilots would have been helpful.
Summary
There are some areas where this measure could be improved. The pilots did indicate that there are some barriers to implementation. More detail on the types of interventions resulting from low performance would have been helpful as well as thoughts around cultural sensitivities and insurance status that may pose barriers.
Support
Importance
Criteria met
Closing Care Gaps
Criteria Met
Feasibility Assessment
Not all data elements are in structured fields.
Scientific Acceptability
Criteria met
Criteria met
Use and Usability
The measure is not in use. No data from users has been provided for feedback. The measure was approved for trial use in 2022. Significant details regarding criterion required for a maintenance measure are not provided in the submission.
Summary
I support the measure
A potentially important…
Importance
This is an important measure for women's health, although I agree with the ACOG's comment that it does not reflect a patient's preferences which is clearly more important rather than whether they received a contraceptive method.
Closing Care Gaps
Based on data provided, there is variability in achievement of the measure assessed across several health systems.
Feasibility Assessment
The measures was implemented in 2 different EHRs.
Scientific Acceptability
Data is provided about reliability
Data is provided about validity
Use and Usability
The challenge for health systems is the resources required to implement the measure into the EHR and the effort that may be required to abstract the data to calculate the measure. In addition, it may require changes to workflows for the clinic. The above changes may be more challenging for some clinics/health systems than others and also depedent on their priorities.
Summary
A potentially important measure but would require commitment by clinics and health systems to implement the measure into the EHR and potentially revise work flows to incorporate systematic assessment.
Summary Recommendation
Importance
The measure addresses a significant public health priority, reducing unintended pregnancies by ensuring access to contraceptive services for patients who state they have a need. The topic is meaningful to patients, especially those at higher risk for reproductive health inequities.
Closing Care Gaps
The measure partially supports closing care gaps by focusing only on patients who self-report a need for contraception through screening. The subgroup analysis provides valuable insight into disparities in contraceptive access by race, ethnicity, and age, confirming that Black and Hispanic patients have significantly lower rates of receiving most or moderately effective methods. However, its current use lacks standardized stratification in routine reporting, and it does not explicitly identify or require actions for closing equity gaps. The measure does not yet incorporate safeguards to monitor and reduce disparaties. These gaps could be addressed through measure updates that require stratified reporting and tie performance to person-centered counseling standards.
Feasibility Assessment
The measure is feasible in organizations with structured EHR systems and dedicated clinical informatics teams, but implementation of the SINC tool introduces cost and variability. Health centers report workflow disruptions during onboarding, variation in EHR templates, and time/resource requirements for training and template standardization. While the measure itself uses routinely available data, the infrastructure needed to support consistent screening and coding presents a barrier to scalability. These challenges are addressable but must be actively supported with technical guidance and investment, particularly in smaller or under-resourced settings.
Scientific Acceptability
The measure has been rigorously tested for reliability across multiple EHR systems and large patient panels. Beta-binomial models indicate that reliability consistently exceeds the acceptable 0.7 threshold at both clinician and facility levels. Minimum denominator thresholds are established to ensure score stability, and testing has accounted for multiple age and racial strata, supporting consistency across populations.
Validity testing demonstrates that the measure accurately reflects contraceptive provision among patients who express a need for it. Face validity is supported by alignment with clinical best practices and person-centered care models. Chart reviews, sensitivity/specificity testing, and correlation with related outcomes (e.g., cervical cancer screening) further affirm the measure’s integrity.
Use and Usability
The measure has demonstrated successful pilot implementations and shows promise as a quality improvement tool, but it is not yet in widespread accountability use. Feedback loops for implementers, usability guidance, and structured improvement data are limited. Additionally, the equity-oriented findings highlight a need for clearer instructions on stratified use, ethical interpretation of LARC provision rates, and integration with person-centered counseling measures (e.g., CBE #3543). These limitations are addressable and should be prioritized if the measure is to scale in value-based care environments.
Summary
The Contraceptive Care (Non-Postpartum, SINC-Based) measure identifies whether patients who say they want contraception actually receive it. It focuses on patient preference, which is a strength, and it is backed by strong reliability and validity testing. New equity data show disparities by race, ethnicity, and age, highlighting the need for better monitoring. However, it doesn’t currently require reporting by subgroups or track whether those gaps are closed. The measure is promising but harder to implement in some settings due to workflow and tech challenges. With targeted improvements, it could be a strong tool for person-centered, equitable contraceptive care.
Public Comments
CU-SINC, Non-Postpartum Measure Public Comment
On behalf of the American College of Obstetricians and Gynecologists (ACOG), representing more than 62,000 physicians and partners dedicated to advancing women’s health and the health of individuals seeking obstetric and gynecologic care, we greatly appreciate the opportunity to provide feedback on the Contraceptive Use SINC, Non-Postpartum eCQM (CBE ID 3699e).
The benefits of contraception, named as one of the 10 great public health achievements of the 20th century by the Centers for Disease Control and Prevention, are widely recognized and include improved health and well-being, reduced global maternal mortality, health benefits of pregnancy spacing for maternal and child health, female engagement in the work force, and economic self-sufficiency for women. ACOG supports access to comprehensive contraceptive care and methods as an integral component of women’s health care. We are also committed to encouraging and upholding policies and actions that ensure the availability of affordable and accessible contraceptive care and methods. Furthermore, ACOG advocates for continual improvements in the quality of care provided to our member’s patients and smooth transitions for our members to better engage in the space of quality improvement.
While contraceptive care access measures exist in the United States, a focus on self-identified need for contraception is a primarily preferred route of capturing quality in this space. However, we are concerned with aspects of the specifications that could lead to unrealized potential in this measure:
Ultimately, ACOG recommends maintaining the Contraceptive Use SINC, Non-Postpartum eCQM (CBE ID 3699e) for trial use only until these concerns are addressed.
Developer response to ACOG comment - thank you for your comment
Thank you for submitting a comment and raising important issues related to our measure. See our responses to particular concerns below.
Prioritizing patient preferences and limitations of the Self-Identified Need for Contraception (SINC): We agree that patient preference should be a focus of any metrics related to quality for contraceptive care and that person-centered counseling is key. However, we also want to ensure that people who want contraception are actually getting access to methods. This measure improves on the claims-based provision measures in this regard in that it 1) utilizes SINC to define the denominator, further specifying the population of interest to those who express an interest in discussing contraception and 2) encompassing “use”, not just “provision” to more accurately reflect whether patients are getting their needs met. We agree that a patient indicating that they want to discuss contraception does not necessarily mean they will want to use a method. For this reason, implementation guidance for this measure highlight that it is not a goal to reach 100% nor is any specific benchmark set. Moreover, we encourage utilization of the Person-Centered Contraception measure (PCCC; CBE # 3543) alongside this measure to ensure that patients are receiving contraceptive care in a way that respects their choice and autonomy.
Use of the term “most or moderately effective contraceptive methods” and lack of inclusion of non-prescription methods: We agree patients should be counseled on all methods based on their needs, preferences, and values. As an eCQM, this measure is limited by what is captured in electronic health records, and currently non-prescription methods are not well or consistently documented. That said, we plan to advocate for better capture of non-prescription methods. With better capture of non-prescription methods, we would update specifications in future maintenance applications to be inclusive of all methods patients report using. The use of the term “most or moderately effective” methods is used for consistency with the claims-based measures and with Contraceptive Technology.
Omission of CPT code 11981: We reviewed our value set files submitted for #3682e and #3699e and identified that CPT code 11981 was left off because the files contained an incomplete list of procedure codes for contraceptive implants. However, the MADiE specifications submitted for both #3682e and #3699e include 11981 because these two measures use the Contraceptive Implant Provision Procedures value set, which contains 11981, in their terminologies (see https://p4qm.org/measures/3682e and https://p4qm.org/measures/3699e). This value set is also published on VSAC as OID 2.16.840.1.113762.1.4.1166.202 (login required to view). We will provide PQM with updated Excel files for #3682e and #3699e to show the full list of Contraceptive Implant Provision Procedure codes utilized in our measures, including CPT code 11981. Thank you for bringing this matter to our attention.
Comment in support of full endorsement of CBE#3699e
Upstream appreciates the opportunity to comment on CBE #3699e: The annual percentage of most or moderately effective contraceptive method users, among patients assigned female at birth ages 15-44, excluding those who are postpartum or did not want to discuss their contraceptive needs. We support the endorsement of this measure and believe it will facilitate the ability of healthcare organizations to monitor and measure access to and provision of essential contraceptive services in their patient populations.
Upstream is a nonprofit organization working to expand access to contraception by providing high-quality, patient-centered training and technical assistance to healthcare organizations. Since our founding in 2014, Upstream has partnered with over 150 health care organizations across more than 30 states to integrate high-quality, patient-centered contraceptive care into primary care. A foundational component of our work is to support health care teams in configuring their EHR system to support a contraceptive care workflow that begins with screening patients for their reproductive health and/or contraceptive needs. An increasing number of our partners are choosing to implement the Self-Identified Need for Contraception (or SINC) screening question in their workflows. In addition, SINC is included in two annual federal reporting requirements: the HRSA Health Center Program’s Uniform Data System and the Title X Family Planning Annual Report, so we believe it is feasible that a growing number of healthcare organizations and researchers in the field may soon have enough of the required data to calculate the SINC-based denominator and leverage this eCQM.
Incorporating the SINC into the denominator of this measure improves upon the claims-based version of this measure (CBE #2903) by capturing the population of patients who indicate that they want to discuss contraceptive services with their care team. By honing the denominator in this way, the measure seeks to reduce potential coercive practices (i.e., care teams wanting to “improve their scores” on the measure by discussing contraception whether or not the patient requests the conversation). Implementing quality measures focused on contraceptive services is a nuanced and specialized topic. Additional implementation guidance from the measure stewards and researchers in the field would help to strengthen use and ensure proper interpretation of this eCQM.
Thank you Upstream!
Our team at the Person-Centered Reproductive Health Program (PCRHP) thanks our colleagues at Upstream for their enthusiastic endorsement of CBE #3699e. We wholeheartedly agree on the importance of measuring access to and provision of contraceptive care. Thank you for taking the time to comment.