Proposed E&M Enhancements

I support these options (written in descending order of preference - although I think they both should be done so as to provide a mechanism to re-endorse)-

• Voting Approach Refinement: Align with the American National Standards Institute (ANSI) standards, by collecting reasons and suggested resolutions from members voting "Do Not Endorse/Remove Endorsement." This supports committee recommendations and ensures concerns are captured.
   
• Ad Hoc Measure Review Process: Measures affected by these changes will be reconsidered and endorsement decisions updated. The process includes:

I support these options (written in descending order of preference - although I think they both should be done so as to provide a mechanism to re-endorse)-

• Voting Approach Refinement: Align with the American National Standards Institute (ANSI) standards, by collecting reasons and suggested resolutions from members voting "Do Not Endorse/Remove Endorsement." This supports committee recommendations and ensures concerns are captured.
   
• Ad Hoc Measure Review Process: Measures affected by these changes will be reconsidered and endorsement decisions updated. The process includes:

My vote is for -

Voting Approach Refinement followed by Ad Hoc Measure Review Process

So if you have many strong minded skilled people in the committee then you must drop the threshold as great minds DO NOT THINK alike.  

But if you have a bags of different kinds (with respect to their level of expertise) of people, then you should increase the threshold.

It’s the concept we in machine learning use when doing ensemble methods/group based decision making.

I would like to share my concerns about the enhancement proposal on the different threshold percentages.  In my opinion, the percentages should be set to just one level, i.e., 70% or 75%, regardless of the size. In addition to creating differing standards, such percentage differences tend to go against simple logic. In my opinion, it will create different standards for measures simply because of size and not because of their merits for endorsement. 

In truth, this proposal proposes to change the decision on just one vote (14 (70% of 20) 15 (75% of 20).  I am not sure this endorsement should be adopted.

The proposed enhancements look good to me. 

On behalf of the American Heart Association (AHA), including the American Stroke Association (ASA) and more than 35 million volunteers and supporters, we appreciate the Partnership for Quality Measurement’s (PQM) effort to improve the efficiency and validity of the current Consensus Based Entity (CBE) measure endorsement and maintenance (E&M) process. 

The AHA supports the lower endorsement threshold for smaller workgroups. 

The AHA supports the continued endorsement of measures previously endorsed when a consensus is not reached in the current review.

The AHA supports an opportunity for voter comments when voting to not endorse or remove endorsement.

The AHA suggests greater clarity on impacted timeframe for the retroactive application of the above changes if approved. Specifically, it would be beneficial to clarify if this reconsideration could apply to all measures since PQM assumed oversight of the CBE process or a narrower timeframe. 

The American Academy of Family Physicians (AAFP) appreciates the opportunity to provide feedback on proposed revisions to the Endorsement & Maintenance (E&M) voting process and thresholds. Our comments on the revisions are below and aligned with those submitted by the American Medical Association and American College of Physicians:  

1. Consensus Threshold Adjustment: The American Academy of Family Physicians (AAFP) does not support this proposal. Adjusting the consensus threshold down to 70% for groups less than 20 will impact the recommendation (voting) group within each E&M committee and make it easier for measures to reach consensus. This contradicts the original intent of a rigorous measure review and endorsement process. According to current PQM E&M Guidebook, Battelle “established the 75% threshold of consensus to be consistent with the goal of adding rigor to all aspects of the consensus development process. The threshold is based on an evidence-based index (Appendix F). Analogous to inter-rater reliability statistics, the evidence-based index assesses the degree of disagreement (or lack of consensus) amongst the independent committee reviews and the committee endorsement votes…”

The larger problem and more important issue to address is a lack of participation among committee members. PQM should facilitate informed, active participation and voting by all committee members. This could help ensure that all stakeholder groups and viewpoints are heard. For example, PQM could explore measure reviews that are divided into shorter calls (e.g., one to two hours in length) over the review timeframe to allow adequate time for discussion rather than the current strategy of one six-hour meeting. This approach may increase the number of voting members participating and allow for a richer discussion.

2. Interpretation on Non-Consensus Decisions: The AAFP does not support this proposed revision. If a maintenance measure fails to reach consensus, then there are legitimate reasons behind the lack of consensus. These measures should not automatically maintain their endorsement. Instead, the committee members should further review and discuss until they can reach consensus one way or the other. If not, then PQM should clearly outline a different process for what happens to maintenance measures that do not reach consensus. 

If this proposal is implemented, then a maintenance measure that has 74% vote of “do not endorse,” would maintain its endorsement. This is problematic and concerning. Maintenance measures should be held to the same high standards as new measures. 

Given the current measurement landscape and the plethora of challenges that come with the addition of new measures and without the retirement of less impactful, possibly outdated, and burdensome existing measures, it is imperative for Battelle/PQM to maintain only the highest standards and scientific rigor in its measure review processes. 

3. Voting Approach Refinement: The AAFP supports this proposal. It is important to collect and document reasons and suggested resolutions from members voting "Do Not Endorse/Remove Endorsement."

4. Ad Hoc Measure Review Process: This proposal is highly concerning; the AAFP does not support it given all the issues noted above in #1-2.

While we appreciate the opportunity to provide feedback, we believe that PQM should provide a clear, more detailed rationale for what it is trying to accomplish by making these proposed revisions to the voting process and thresholds. Additionally, we would appreciate transparency regarding which organization and/or organizations suggested these revisions and why. 

Additionally, we offer the following general concerns about the current E&M review process: 

• The “endorsed with conditions” voting option remains problematic. It is likely that many of the “endorse with conditions” votes should actually be “do not endorse” votes until all conditions are addressed. Currently, there is no guarantee that measure developers and stewards actually address the conditions before the measure is used. How can PQM enforce that the committee’s conditions are addressed BEFORE it is implemented? One possible solution to this problem is to remove the “endorse with conditions” voting option.
• Tendency to vote “yes” rather than “no” – There appears to be a tendency for committees to vote “yes” on measures rather than “no” even in situations when the discussion has primarily focused on significant concerns about the measure. Affirmative votes should not be encouraged by PQM or committee members, particularly when there has been open discussion about numerous measure concerns. Instead, the default should be “no” until the measure has proven to meet all the E&M criteria outlined in the PQM “Measure Evaluation Rubric.” 

The American College of Physicians (ACP) appreciates the opportunity to comment on the proposed enhancements to the endorsement and maintenance (E&M) voting procedures. Our comments are aligned with those submitted by the American Academy of Family Physicians and the American Medical Association.

While we appreciate brevity when appropriate, the web page that briefly lists the 4 enhancements fails to provide any information on the rationale for the modifications. Sharing this background would allow us to understand the full context as we reviewed the modifications.

We have a number of concerns with the proposed enhancements.

1. We do not support the “Consensus Threshold Adjustment.” In the initial version of the E&M guidebook, the Measure of Consensus was determined to be 75% regardless of how many votes were recorded. In fact, the guidebook states, “Battelle established the 75% threshold of consensus to be consistent with the goal of adding rigor to all aspects of the consensus development process.” We believe lowering the threshold would be in conflict with those principles and may threaten the rigor and integrity of the E&M process.
2. We do not support the “Interpretation of Non-Consensus Decisions.” Maintenance measures should not be treated differently than new measures. In fact, maintenance measures are and should be held to a higher standard as data has been collected from the use of the measure. In particular, the measure developer should have data documenting the performance gap as well as whether the implementation of the measure has led to improvements over time. The purpose of reviewing existing measures for continued endorsement is so that they can be adjudicated just as new measures and to ensure that they remain important, evidence-based, and methodologically sound. The consensus not reached outcome should result in the same fate.
3. We support the “Voting Approach Refinement.” This approach would allow for transparency and enable voting members to identify their concerns with the measure and recommend improvements for the future.
4. We are opposed to the “Ad Hoc Measure Review Process.” If the proposed improvements to the voting procedures move forward, the modifications should be applied prospectively and not retroactively. Additionally, we are concerned that this retroactive review would appear to be done at the PQM staff level and that the ad hoc review would not involve the recommendations group. This falls in the face of the process in place.

Thank you for the opportunity to provide feedback.

The American Society of Clinical Oncology (ASCO) appreciates the opportunity to respond to PQM’s call for comments on its proposed enhancements to the E&M voting procedures.

With respect to the consensus threshold adjustment, generally ASCO believes the threshold should be consistent across different-sized groups. The current methodology states that consensus is based on an evidence-based index, “measure of consensus,” and a measure of consensus of 95 percent corresponds to a consensus threshold of 75 percent of votes. Per the E&M guidebook, the measure of consensus states that “this approach is advantageous compared to other metrics based on variance, in that it takes into consideration the different sizes of the voting groups and different rating options,” implying that the 75 percent threshold already accounts for smaller group sizes. However, if lowering the threshold to 70 percent continues to correspond to an appropriate measure of consensus, i.e., 95 percent, ASCO supports this proposal with a note that impact may be minimal as it takes solely one vote, within a group of less than 20, to go from “consensus reached” to “consensus not reached.”

ASCO supports PQM’s proposal to retain the endorsement status for maintenance measures with a “consensus not reached” vote. And lastly ASCO supports aligning with the American National Standards Institute standards by collecting reasons and resolutions from members voting “do not endorse / remove endorsement.” Measure stewards can take into consideration formal feedback and recommendations as it makes future measure updates.

Thank you for the opportunity to submit comments and we look forward to continuing to work with PQM.

The American Medical Association (AMA) appreciates the opportunity to comment on the proposed changes to endorsement and maintenance voting procedures. Our comments on the revisions are below and aligned with those submitted by the American Academy of Family Physicians and American College of Physicians. While we support continued refinement of the processes, the Partnership for Quality Measurement (PQM) must ensure that they reflect true consensus and are transparent. For stakeholders to meaningfully evaluate and respond to changes, PQM should include detailed justifications for proposed revisions, including examples of any challenges encountered under the existing process. Without this context, it is difficult to assess whether the proposed solutions are appropriate to the problems they aim to solve. Insufficient details were provided on the issues or concerns that the changes attempt to address and a clearer rationale for the proposed modifications should be distributed with information on other alternative options that were explored but ultimately not selected and why. 

Consensus threshold adjustment: 

If the true concern is how to ensure that consensus is achieved and opposing viewpoints are aired, then we do not believe that lowering the threshold to 70% in instances where less than 20 voting members are present is an acceptable solution. Rather, PQM should identify alternative ways in which participation from a larger number of members can be facilitated and ensure that all stakeholder groups and viewpoints are heard. For example, PQM could explore measure reviews that are divided into shorter calls (e.g., one to two hours in length) over the review timeframe to allow adequate time for discussion rather than the current strategy of one six-hour meeting. This approach may increase the number of voting members participating and allow for a richer discussion.

Endorsement should be a high bar to achieve and lowering the threshold should not be the solution for limited attendance. PQM should require active participation and voting by all recommendation group members and achieving consensus should not be compromised. In addition, lowering the threshold may inadvertently incentivize strategic non-participation by stakeholders who oppose a measure, ultimately distorting the consensus process. 

Interpretation of non-consensus decisions:

Again, it is not clear why it is acceptable for a measure for which consensus could not be reached to maintain its endorsement status, particularly when the policy does not indicate when the measure would be reconsidered. Additional detail on next steps (e.g., automatically review the measure during the next cycle) would be needed before the AMA can consider supporting this proposed change. Alternatively, PQM should consider a process that allows members to review and discuss the measure until all questions and concerns are addressed even if it requires delaying a decision on a measure until the next review cycle. 

Voting approach refinement:

The AMA supports this change as it is critical that opposing viewpoints and concerns are captured as a part of any endorsement and maintenance review. 

Ad hoc measure review process:

We do not support revising the process to lower the voting threshold for certain measures and as a result, believe that endorsement decisions should be final. The PQM has an appeals process through which measure developers can request reconsideration of decisions. Adding an additional step to an already complicated process does not seem reasonable or necessary. 

Thank you for the opportunity to comment. 

I believe the organization is slowly biasing the process to maintain recommendations and not addressing flaws in measure proposals.  When consensus (which is getting watered down) is not reached the measure maintains endorsement--so really, this is a stacked process.  What should be a dichotomous decision (Yes, maintain endorsement, or No, don't maintain endorsement) is effectively biasing decisions toward maintaining the status quo.  And to remove endorsement requires yet another step.  

Rather than fostering positive measure improvement, these changes, and the overall process, promote stagnation.

The concept is straightforward, but it might help to briefly explain why a lower threshold is better suited for smaller groups.

Upholding the integrity of the Measure of Consensus—this is a good point. It might be beneficial to briefly elaborate on how lowering the threshold  allows for more decisive action while maintaining  collaboration.

What is the rationale for why endorsed measures retain their status despite a "consensus not reached" vote?

Will there be a specific timeframe for re-evaluating these measures? What will trigger a need for reassessment if measure after it didn’t  reach consensus initially.

I totally agree on that voting approach refinement.

It might also be useful to specify who is eligible to submit these requests (e.g., developers, stewards, stakeholders) for clarity.

 What qualifies as an "immediate issue." Can you give an  example  to help  understand the types of concerns that would warrant an ad hoc review.

The proposed enhancements seem reasonable and I appreciate the 75% threshold for voting members representing groups of more than 20 members vs 70% for groups of 20 or less. Regarding consenus not reached and maintenance measures I expect that discussions have been taken into account where reasoning has been documented/ recorded. ANSI alignment seems appropriate.

Consensus Threshold Adjustment:

I think the Measure of Consensus threshold should be the same.....the issue is not the percent, but having enough people that committed to participation showing up for the meeting and voting.