Committee Engagement Enhancements for Endorsement & Maintenance (E&M) Spring 2024 Cycle

I support these changes.

I support the proposed changes for engaging the Advisory Group members in a more meaningful way.   The proposed changes will make for a better experience.

My only "concern" with the proposed changes is eliminating the step of committee members submitting ratings in advance of the meeting.  I, personally, find that exercise to be important.  It forces me to review and evaluate each measure in a systematic way.   My fear is if no pre-work is required, committee members may not take the time to review each measure carefully before the actual committee call. 
 

I strongly oppose the proposal to limit voting on measures to the members of the Recommendations Group, rather than the full E&M Committee.  Merely increasing the proportion of the Committee who serve on the Recommendation Group does not assure that the full range of knowledge and perspectives represented on the Committee are reflected in the votes on measures. 

The biggest barrier to achieving a quorum at the last meeting was not the size of the Committee, but the way that Battelle scheduled and structured the meeting.  Battelle should survey the Committee members in advance of the meeting and then select a date that will work for the largest number of Committee members, rather than simply announcing a date that is convenient for Battelle but is not workable for many Committee members.  The date of the meeting should be established well in advance in order to avoid having Committee members unknowingly schedule conflicting meetings, and also to give them time to try to rearrange any existing conflicts.  If an Advisory Committee member can only be available for a portion of the meeting day, Battelle should help that Committee member determine when the voting portions of the meeting will be held, rather than forcing them to sit quietly on Zoom for the entire day waiting for the brief periods of time when the votes are taken.  In addition, if a member of the Recommendations Group cannot participate in the meeting, a member of the Advisory Group should be recruited to replace them. 

If it turns out that the meeting falls a few votes short of a quorum, there is nothing wrong with contacting the unavailable members after the meeting to obtain their votes.  Obtaining a few votes offline is far preferable to excluding a large subset of the Committee members from voting at all.

It is a good idea to have a meeting of the E&M Committee members to discuss the measures in advance of the endorsement meeting.  However, it is a bad idea to design the meeting primarily for the “Advisory Group” members, and to merely “encourage” the members of the “Recommendations Group” to attend.  Participation by all of the Committee members will enable all of their questions and concerns to be identified prior to the endorsement meeting so that the developer, staff, or other Committee members can address them.  This will likely lead to endorsement meetings that are shorter and therefore more likely to achieve a quorum of the full Committee.  In addition, this could make it unnecessary to have a “Recommendations Group” at all; since most of the questions and concerns would have been identified and addressed during the pre-meeting, the full Committee could participate in the discussion at the meeting,

All questions raised by the Committee members at the pre-meeting should be recorded and sent to the measure developer for response, not just the “frequently asked” questions.  In addition, any public comments on the measures should also be given to the developer for response prior to the endorsement meeting, rather than simply forwarding the comments to the Committee and having the developer respond to them during the endorsement meeting.  Battelle staff should then review the developer responses and provide comments and additional information to the Committee members prior to the endorsement meeting to help them determine how to vote on the measure. 

I oppose preventing the Committee members from submitting ratings of measures prior to the endorsement meeting.  Rather than eliminating these ratings, Battelle should add an additional category for “Unsure.”  The ratings submitted prior to the meeting can then provide a useful “straw poll” to help guide the discussion at the endorsement meeting.

If there is going to be a category for “Endorse With Conditions,” then there needs to be a more organized process of identifying the options for the conditions and voting on them.  At the last meeting, measure #2687 was “Endorsed with Conditions” but there was no process at the meeting for determining what the conditions should be.  Battelle unilaterally decided to reclassify the measure as “Endorsed,” which is not what the Committee voted for.  Changes need to be made to the process in order to avoid this problem in the future.

Hi,

I am on the Clinician Advisory Committee.  With the previous vendor I was on what would be the current Recommendations Group.

My experience for the first go round was not that great.  I am a specialist in Behavioral Health (mental health, substance use and intellectual disabilities), however, almost all of the measures (I believe 2 were behavioral health related) that were presented for review were for medical conditions.  Although I can review and make recommendations on the process and review validity and reliability and other factors, it was quite difficult to make medical domain recommendations.  I did not have the basic knowledge or understanding of the developer's work actually being accurate in the medical domain.  e.g. if the measure was calling for specific values I did not have the fund of knowledge to know if those values were even relevant to the measure.  If I had the ability to question medical personnel for some of those answers that would have been helpful.  It seems that the change for Spring 2024 will accommodate this and provide that information and have developers available 2-3 months before Recommendation Group review.

The larger problem is that my expertise in behavioral health is not being applied to all behavioral health measures.  I later found, through Battelle email announcements, that other groups were reviewing other behavioral health measures that were then approved.  I am unclear why all measures did not come before the Advisory Group.  IT would be helpful when reviewing some domains to actually have domain experts involved in all of the measures that are being presented for that domain.  I could apply my expertise to all behavioral health measures.

Thanks for providing the opportunity for input.  I will try to convey the same message in the upcoming onsite meeting in April.

Here are my thoughts regarding the changes:

1. I think this is a valuable change. Many of those new to the process do not understand the verbiage of proposals. By having these meetings, they have a chance to contribute more in a session where nothing else is scheduled. Obviously having enough people participating is important to the validity of the whole process especially with the other proposed changes.
2. This is obviously the most controversial.  Whether this erodes participation in the overall E & M process depends in how successful the advisory group meetings are.  As an advisory member, I noticed that perhaps 2 or 3 advisory group members contributed original ideas in their comments prior to the meeting. These members are important contributors so perhaps there is a way to encourage their continued participation informally. 
3.  I support this change.
4.  I support this change.
5.  I found the format of the pre meeting comments a bit cumbersome. Nevertheless I think it is important to have ideas from committee members in advance of the meeting. I am wondering if other recommendation group members took time to review those documents but I learned much from them. So I would support creating a document that  summarizes the advisory group meetings to be provided at least a week before the recommendation meeting.  To create this document, someone will need to go through the meeting transcript which is an extra burden. So perhaps the entire transcript could be made available to the chairs.  In some way, the recommendations committee needs to have something that distills the comments of those meetings. I doubt many will attend both of the other meetings.

If you plan to enhance engagement and participation. Will the compensation be changed? As for the other two things. I don't see any problems

Thank you for your responsiveness to committee feedback and your rapid cycle change approach.  I would like to offer a few suggestions

1.Build the feedback from the Advisory Committee into the formal review process - The proposed changes in the Advisory Committee  review make attending these discussions and review of their feedback and questions optional for the Recommendations Committee and Measure Developers.  Recommendation Committee members and Measure Developers will be encouraged to attend the discussion, but there does not appear to be any other incentive for them to consider or respond to substantive feedback.  It looks like the summary provided to the Recommendations Committee will focus on FAQ only and will not include critical analysis or commentary. In previous E & M review processes, feedback from groups like the Scientific Review group and Public were built into the review process.  Committees with voting responsibility were required to respond to these reviews thus assuring they were read and considered.  Without a mechanism to demonstrate value and use of the Advisory Committee's feedback, Advisory Committee members may find limited incentive to do the reviews or remain engaged. 

2. Along with Suggestion #1, consider having the Advisory Committee focus on specific aspects of E & M that will be meaningful to Advisory Committee members.  Perhaps consider having Recommendations Committee Assignments available by the time the Advisory Committee meets so that members can plan to attend the discussions most relevant to their reviews. 

3.  Consider renaming the Public Comment Listening Session- my first reactions were 1) this seems like a passive title and 2) it's not immediately clear who's listening to whom. If the goal here is to "enhance the engagement and participation... of members of the public", name the time in a way that makes it sound like it's important.  Having the opportunity for the public to respond verbally as well as in writing seems like a good idea to try.  Again, consider building public comments into the formal review process - in the current changes, it all sounds optional whether committee members attend these sessions or review the comments. 

If an Advisory Board member asks for additional materials from the developer during her review process, can these materials also be provided to the other members of the Advisory Board and E&M Committee?  This issue came up when preparing for the review of Patient Understanding of Key Information Related to Recovery After a Facility-Based Outpatient Procedure or Surgery, Patient Reported Outcome-Based Performance Measure.  I found some discrepancies between the application and actual survey measure that raised questions for the developer during deliberation. This influenced my rating, but the E&M Committee did not appear to have the actual survey when they were reviewing and voting on the measure?   As always, I appreciate your team's strong efforts to continuously improve the process. 

1. Regarding coordination of the different Advisory and Recommendations groups, I do not believe that the changes would result in better results - i.e., consensus. Agree with other commenters that Batelle should increase engagement with both groups prior to the voting meetings in order to collect feedback/concerns and navigate towards consensus (including specific conditions for a conditional approval).
2.  Voting - no issues.
3.  Recommendation Group Size - no issues.
4. Public Comment Listening Sessions - no issues.
5.  Committee Independent Review Ratings - As mentioned by other commenters, I would recommend keeping the ratings, as it ensures that the committee members carefully apply the rubric and commit to a position.

We appreciate the opportunity to comment on the proposed Committee Engagement Enhancements for the upcoming cycle of the E&M review.

While we support the goal of ensuring that the quality measurement review process is reliable, transparent, attainable, equitable, and meaningful, we have concerns with the enhancements to the E&M process. 

We appreciate the value of aligning the roles of the Advisory Group and Recommendations Group across the PRMR and E&M processes.  However, we have seen that the Advisory Group has had a limited impact on the current PRMR process and aligning the two processes will similarly reduce the influence of the Advisory Group in the E&M process.  Seeking feedback during Advisory Group meetings can be a useful tool for discussions among members and generate additional comments.  The manner in which the Advisory Group’s input is shared with the Recommendations Group is critical to the success of the new approach.  We believe there should be dedicated time during the Recommendations Group meetings for staff to verbally summarize the Advisory Group discussion for each measure.

Additionally, we understand that the increase in the number of Recommendations Group members intends to mitigate the minimized role of Advisory Committee members.  However, we urge Battelle to carefully consider the challenges of facilitating such a large group, particularly in terms of engagement and productivity.

Finally, we hope the Battelle team considers the extent of information reviewed and collected from committee members.  The PRMR process required many hours of examining measure materials and more hours to input all requested feedback.  The E&M process was considerably less time-consuming.  If the intent is to have the E&M process mirror the PRMR process, we caution Battelle against asking for such detailed feedback as it is onerous. The feedback requested from Advisory Group members in the PRMR process is nowhere near the level of detail given to Recommendations Group members.  We hope that this process is not repeated with the E&M process.

P4QM is soliciting feedback from the public on proposed enhancements to its committee process for the Spring 2024 cycle. Vizient appreciates the opportunity to provide feedback and supports many of the proposed changes, as it would allow the E&M Committee a better opportunity to offer more tailored feedback on the measures. 

Vizient is concerned about the proposal that advisory group members would not vote during the E&M committee meetings. When read in conjunction with the other proposals, it is unclear what role advisory committee members would serve in the evaluation of these measures, as there is also no voting in the proposed advisory committee meetings, and P4QM is proposing to remove some of the evaluation criteria from the independent reviews. While Vizient understands that this proposal is primarily to ease burdens associated with quorum on the E&M committee meetings, we remain concerned that advisory group members will not have the opportunity to provide meaningful feedback or votes through any avenue. Vizient recommends that P4QM develop a better process for capturing the votes of advisory group members to ensure a diverse perspective of opinions. 

Thank you for the opportunity to participate in this public comment opportunity. Our team has reviewed the proposed enhancements and would like to offer the following comments: 

Our understanding is that developers can attend the early Advisory Group meeting and Public Comment Listening Sessions in listen-only mode. For the Advisory Group meetings, developers are expected to respond to comments after the meeting via the FAQ document. For the Public Comment Listening Sessions, it doesn’t appear that developers have a chance to respond to comments. To ensure accuracy of measure discussions, we recommend allowing developers to respond to at least some questions in real time. Allowing brief clarification by developers may avoid unnecessary speculation for a more informed and streamlined discussion.

The American Medical Association (AMA) appreciates the opportunity to comment on the Partnership for Quality Measurement (PQM) Endorsement and Maintenance process. We support PQM’s commitment to evaluating and reviewing the process and strongly encourage PQM to weigh whether these new changes could impede transparency of the process as well as increase its complexity. 

Advisory Group Meetings, Voting, and Recommendations Group Size: The AMA is concerned to see the roles and responsibilities of the Advisory and Recommendations Group divided even further, leaving us to question the usefulness of the two separate groups. Based on our experience with the Pre-Rulemaking Measure Review (PRMR), the degree to which an Advisory Group is useful when members are not allowed to vote and are only able to ask questions is unclear. In particular, if they are unable to provide any input on how the measures should be evaluated against the measure evaluation criteria, this group will likely only add to the complexity of the process. We also do not believe that increasing the number of members on the Recommendation Group ensures that the broad perspectives and expertise are represented, which was the point of including the Advisory Group initially. We also oppose limiting the voting responsibility to the Recommendations Group. We encourage PQM to reconsider these proposals and rather focus on how to further enhance the input that the Advisory Group may have during the process. In addition, efforts should be made to schedule Committee reviews based on member availability, potentially parsing out measure reviews to meetings that require smaller time commitments (i.e., 1 or 2 hour calls), or other strategies should be employed that might facilitate increased member participation. 

Public Comment Listening Sessions: We support every effort to obtain input from multiple stakeholders throughout this process but encourage PQM to evaluate whether these sessions will provide any additional value. Because the Endorsement and Maintenance process overlaps with other activities such as PRMR or rulemaking comment periods, it is not clear whether stakeholders will be able to participate in another activity due to competing demands. The AMA again urges PQM to consider staggering the Endorsement and Maintenance projects with these other activities to avoid the continued overlap. 

Committee Independent Review Ratings: The AMA does not support removing the ratings of measures prior to the endorsement meetings as we found it particularly useful to understand the areas of concern or where clarification was needed. The release of these ratings also increased transparency of the overall evaluation process. This level of information would have been valuable to see during the PRMR process and we encourage PQM to consider adding it to the PRMR process rather than remove it from the Endorsement and Maintenance process. 

We appreciate the opportunity to comment.

PQA commends Battelle on their continued commitment to a rigorous and effective consensus-based endorsement process. PQA agrees that a clear and impactful role for Advisory Group members is important to facilitate strong engagement. PQA also agrees that managing the number of voting members is necessary.

However, PQA notes that holding a separate call for the Advisory Group, during which the Recommendation Group members are not necessarily in attendance and during which no voting occurs, may obfuscate rather than clarify the role and impact of the Advisory Group. If the only concrete output of the Advisory Group is an FAQ document assembled by Battelle staff and containing developer responses, Battelle will need to ensure that document receives significant attention by Recommendation Group members prior to and during E&M voting meetings. 

With more qualitative material (public comment summaries, FAQs, etc.) available to Recommendation Group members and a reduction in quantitative information (via moving away from initial committee reviews), it will be critical that Battelle staff allow Recommendation Group members sufficient time to familiarize themselves with material. Additionally, Battelle staff should continue to ensure that meeting discussion and decision-making are focused on measure criteria. 

Public Comment Listening Sessions & Advisory Group Meetings
- Will measure developers need to be prepared to respond/answer questions during Public Comment Listening Sessions and/or Advisory Group Meetings?
- Will materials be shared in advance for the Public Comment Listening Sessions and/or Advisory Group Meetings? If so, when can we expect to see those materials?
- Will measure developers need to respond to any of the listening session summaries?
- When will public comment be posted for review by measure developers?

E&M Committee Meetings
- If E&M committees don't submit ratings anymore, will measure developers have any indication of questions/comments that may come up during Recommendations Group Meetings?
- How far in advance will materials be sent to measure developers? Or will materials only be posted to the events page? If so, when can we expect to see these posted?


PQM Staff Internal Review
- If/when measure developers respond to items flagged during the PQM staff review, when can the developers expect to see those updates made to the PQM staff review?

Dear PQM team,

We at the American Urological Association are pleased with many of the changes in the Endorsement and Maintenance (E&M) process to-date and we appreciate your desire for continuous improvement.  Moreover, we understand the complexity of the process and understand the effort it takes to come up with solutions to unanticipated problems.  We appreciate the opportunity to offer the following comments on the proposed changes to the Endorsement and Maintenance (E&M) process that are to be implemented for the Spring 2024 cycle and beyond. 

#1:  Adding an Advisory Group Meeting

We support changes to the process that would result in more effective capture of Advisory Group members’ questions and concerns.  More specifically, we support allowing Advisory Group members to discuss measures verbally rather than limiting their participation to “chat” during the E&M meetings.  However, we are concerned with the proposed timing of this meeting. Rather than 2-3 months prior to the endorsement meeting, we believe the Advisory Group meeting should be held after the preliminary assessments by Battelle staff have been distributed.  In addition, we believe that the Advisory Group Meeting must be of sufficient duration to allow full discussion of the measures by the Advisory Group.  We agree that attendance at such an Advisory Group meeting by Recommendation Group members would be ideal, although this would add burden for that group.  Therefore, if PQM moves forward with the proposed change to hold Advisory Group meetings, we recommend holding long-duration calls (as needed) shortly before the endorsement meeting (e.g., 6 weeks or less). Also, in addition to providing a summarized FAQ and developer/steward responses, we recommend that Battelle provide transcripts of the Advisory Group meetings, to ensure that the full context and nuance of the discussion is conveyed.

#2:  Limiting voting to the Recommendations Group only

In the spirit of seating separate “recommendations” and “advisory” groups, it makes sense that only the “recommendations” group would vote (otherwise, the need for two groups is less apparent).  Thus, we support this proposed change. 

#3:  Increasing the size of the Recommendations Group

We support this proposed change.

#4.  Adding public listening sessions

We are somewhat ambivalent about this proposed change but generally do not support it.  While we favor efforts to accommodate those who wish to make public comment, this change would increase the number of meetings for both measure developers/stewards, as well as for some E&M committee members and Battelle staff.  We question whether additional opportunities for public comment are necessary and are concerned that this change might sway public commenters from offering written comments (which we prefer over oral comments that must be summarized by others).  We would be more supportive of this proposed change if Battelle provided transcripts of these calls rather than (or in addition to) summaries of the comments. 

#5:  Pre-meeting ratings no longer required

We do not support this proposed change.  We found the preliminary ratings by E&M committees to be a useful tool: it promotes accountability to conduct a thorough review in a timely manner, it provides food for thought when others call out different concerns and/or vote differently, and it provides transparency about which issues require in-depth discussion during the E&M meeting.  Rather than abandoning the pre-meeting ratings, we recommend that Battelle provide examples of how to write informative rationale statements or otherwise share concerns with fellow committee members, so that the ratings themselves are more informative for all audiences.

Thank you again for the opportunity to comment. 

Sincerely,

Karen Johnson, PhD

Director, Quality and Measurement

American Urological Association

Thank you for your attention and responsiveness to feedback following PQM’s first E&M cycle.

I wish to clarify if the survey developers will have the opportunity to make changes/update their submission in response to questions/comments raised by the Advisory Group.

Thank you for the opportunity to provide feedback on the endorsement and management (E&M) process for measures seeking initial endorsement or endorsement maintenance. My feedback reflects the experience of a measure developer who defended a claims-based measure evaluated during the fall 2023 cycle and attended numerous E&M meetings convened by the prior consensus-based entity.

First, I appreciate the thought that went into proposing updates for the E&M process beginning with the spring 2024 cycle. Battelle has made a number of improvements for E&M of measure evaluation based on lessons learned from the prior consensus-based entity. Some of the improvements highlighted in the document proposing changes to the process for spring 2024 are good, including separating work led by the Advisory Committee and the Recommendations Committee, and soliciting feedback from the public in a structured manner.

That said, I do think there are additional ways E&M could be improved further; these include:

• Engagement of measure stewards and developers in the E&M process: It makes sense for the Advisory Committee and the Recommendations Committee to meet separately. That said, participation by developers in those meetings would be helpful for us to begin preparing to defend measures seeking initial or maintenance endorsement. Participation, in this instance, may or may not include a speaking role; even listening to feedback earlier in the evaluation process would allow us to run additional analyses, revise materials (if a clarification is needed), and author drafts of comments to questions we anticipate being raised by the Recommendations Committee.

• Feedback provided in advance of Advisory and Recommendations Committee meetings: Developers would be happy to provide written feedback to the Advisory Committee or Recommendations Committee in advance of calls for these groups. Because what Committees review is a summary of detailed qualitative and quantitative testing efforts, sharing additional context or exhibits may improve understanding of Full Measure Submission forms in advance of discussing and voting on suitability for endorsement.

• Voting prior to Advisory and Recommendations Committee meetings: Similar to other commenters, I would encourage Battelle to retain initial votes on the evaluation criteria that are used to determine suitability of endorsement. I use the written comments and votes, by criterion, to prepare talking points and update measure stewards about any points that require clarification or opportunities to improve measures’ technical specifications. Removing the preliminary vote from initial review of the Full Measure Submission forms takes key data away from developers who need substantial time to prepare for E&M evaluation.

• Voting by the Recommendations Committee: While I do support voting prior to the Recommendations Committee meeting, it is members of the Recommendations Committee from whom prospective votes would be most appreciated. From the measure my team brought for fall 2023 E&M review, the disparity between votes of the Advisory Committee and those from the Recommendations Committee were substantial. Collecting feedback from the body who will vote on suitability for endorsement would be more helpful to prepare for the discussion. Members of the Advisory Committee did highlight key opportunities for improvement of measure technical specifications, so collecting their input as a springboard to begin iterating on updates to specifications would be appreciated.

• Engagement of subject-matter experts in review of Full Measure Submission forms: Minimizing the number of Committees used to discuss and evaluate measures was a smart decision made by Battelle when revising the E&M process. However, the new, larger Advisory Committee and Recommendations Committee rosters make discussion of topics specific to each measure’s focus difficult—subject-matter expertise necessary to discuss and vote on some measures that are analytically complex, not clinically focused, or built in a novel way may not be available on a given Committee. I would recommend identifying a lead discussant for each measure who brings substantial training, employment, or lived experience that relates to the measure’s construct to facilitate discussion by the full Advisory Committee or Recommendations Committee; doing so could help to focus and drive the discussion. This person may or may not be on the specific Advisory Committee and Recommendations Committee from which written feedback, verbal comments, and votes will be obtained; they could act in a non-voting capacity to facilitate conversation and identify questions that are most proximal to the measure’s focus and outcome.

I look forward to bringing measures for initial and maintenance endorsement to Battelle during its spring 2024 E&M cycle and am excited to see what changes will be implemented to improve the process. Thank you.

Thank you for the opportunity to comment on the proposed changes to the CBE process. We appreciate Battelle’s openness to discussion and continued improvement of the CBE process, and we greatly appreciate the overall responsiveness and efforts of Battelle staff.  We comment on two of the recommendations below. 

Recommendation #1: Advisory Group Meetings: For the pre-meeting, the proposed change says that developers are welcome to attend and “listen” to the discussion. We wanted to clarify that developers would be allowed to answer questions at the meeting rather than just listen and respond to an FAQ after the meeting.   We would also suggest “flipping” the process, and having Committee members pose their questions that require clarification (without discussion) before the pre-meeting, allowing developers to respond to the questions (or prepare to respond to them at the meeting) and then attend the pre-meeting and provide clarifications. Then in the next step, the Committee could make their preliminary recommendations while already having their questions clarified. We also request that developers be permitted to respond to public comment before the Committee discussion at the measure evaluation meeting. 

Recommendation #5: Committee Independent Review Ratings: We recommend that Battelle maintain the existing process and have Committee members enter a rating (Met/Not Met) prior to the measure review meeting. This allows developers to understand the concerns of the Committee to prepare properly for the discussion. We also request that Battelle begin a public comment on the evaluation criteria and rubric for input by developers.  

Thank you!