DR CTQS TEP Meeting Meeting Date: 4/20/2020 Meeting Time: 9:00am-10:30am Pacific Meeting Location: Virtual Conference via Zoom Approval Date: May 11th, 2020 Recorded by: UCSF Team   Meeting Minutes:

Project Overview:

The Centers for Medicare & Medicaid Services (CMS) has granted an award to the University of California San Francisco (UCSF) to develop a measure of computed tomography (CT) image quality and radiation safety. The project is a part of CMS’s Medicare Access & CHIP Reauthorization Act (MACRA)/Measure Development for the Quality Payment Program. The project title is “DR CTQS: Defining and Rewarding Computed Tomography Quality and Safety”. The Cooperative Agreement number is 1V1CMS331638-02-00. As part of its measure development process, UCSF convened groups of stakeholders and experts who contributed direction and thoughtful input to the measure developer during measure development and maintenance.

Project Objectives:

The goal of the project is to create a quality measure for CT to ensure image quality standards are preserved and harmful effects of radiation used to perform the tests are minimized. Radiation doses delivered by CT are far higher than conventional radiographs (x-rays), the doses are in the range known to be carcinogenic, and there is a significant performance gap across health care organizations and clinicians which has consequences for patients. The goal of the measure is to provide a framework where health care organizations and clinicians can assess their doses, compare them to benchmarks, and take corrective action to lower them while preserving the quality of images so that they are useful to support clinical practice. The measure will be electronically specified using procedural and diagnostic codes in billing data as well as image and electronic data stored with CT scans, typically stored within the Picture Archiving and Communication Systems (PACS) - the computerized systems for reviewing and storing imaging data or Radiology Information Systems (RIS).

TEP Objectives:

In its role as a measure developer, the University of California San Francisco is obtaining input from a broad group of stakeholders to develop a set of recommendations to develop a radiology quality and safety measure. The proposed measure will be developed with the close collaboration of the leadership from diverse medical societies as well as payers, health care organizations, experts in safety and accreditation, and patient advocates. A well-balanced representation of stakeholders on the TEP is intended to ensure the consideration of key perspectives and obtain balanced input.

Scope of Responsibilities:

The TEP’s role is to provide input and advice to the measure developer (University of California San Francisco) related to a series of planned steps throughout the 3-year project. The specific steps will include developing and testing a risk-adjusted measure which can be used to monitor CT image quality in the context of minimizing radiation doses while maintaining acceptable image quality. The TEP will assist UCSF in conceptualizing the measure and any appropriate risk adjustment of it. The TEP will assist UCSF with identifying barriers to implementing the proposed measure and test sites in which the developer can assess the feasibility and performance of its use. The TEP will assist UCSF with interpreting results obtained from the test sites and in suggesting modifications of the measure prior to it being incorporated into a software tool which will be made available to providers to enable them to report and monitor their performance. The TEP will provide input and advice to UCSF regarding the software tool to ensure that it is valuable for a wide range of stakeholders and CMS.

Guiding Principles:
Participation on the TEP is voluntary. Individuals participating on the TEP understand that their input will be recorded in the meeting minutes. Proceedings of the TEP will be summarized in a report that may be disclosed to the general public. If a participant has disclosed private, personal data by his or her own choice, then that material and those communications are not deemed to be covered by patient-provider confidentiality. Questions about confidentiality will be answered by the TEP organizers.

All potential TEP members must disclose any significant financial interest or other relationships that may influence their perceptions or judgment. It is unethical to conceal (or fail to disclose) conflicts of interest. However, the disclosure requirement is not intended to prevent individuals with particular perspectives or strong points of view from serving on the TEP. The intent of full disclosure is to inform the TEP organizers, other TEP members and CMS about the source of TEP members’ perspectives and how that might affect discussions or recommendations.

All potential TEP members should be able to commit to the anticipated time frame needed to perform the functions of the TEP.

Estimated Number and Frequency of Meetings:

TEP is expected to meet three times per year, either in-person or via a webinar.

This meeting was originally set to occur in-person, but was changed to a virtual meeting as mandated by federal social distancing measures and state-wide Shelter-in-Place orders.

Table 1. TEP Member Name, Title, and Affiliation

 

Name	Title	Organization
Attendees
Niall Brennan, MPP	CEO	Health Care Cost Institute
Helen Burstin, MD, MPH, FACP	Executive Vice President	Council of Medical Specialty Societies
Mythreyi Bhargavan Chatfield, PhD	Executive Vice President	American College of Radiology
Jay Bronner, MD	President and Chief Medical Officer	Radiology Partners
Missy Danforth	Vice President of Health Care Ratings	The Leapfrog Group
Tricia Elliot, MBA, CPHQ	Director, Quality Measurement	Joint Commission
Jeph Herrin, PhD	Adjunct Assistant Professor	Yale University
Name	Title	Organization
Attendees
Hedvig Hricak, MD, PhD	Radiology Chair	Memorial Sloan Kettering Cancer Center
Leonard Lichtenfeld, MD, MACP	Interim Chief Medical Officer	American Cancer Society, Inc.
Matthew Nielsen, MD, MS	Professor	UNC Gillings School of Global Public Health
Debra Ritzwoller, PhD	Patient	Patient Representative
Lewis Sandy, MD	Executive Vice President, Clinical Advancement	UnitedHealth Group
Mary Suzanne Schrandt, JD	Patient	Patient Representative
Anthony “Tony” Seibert, PhD	Professor	University of California, Davis
Todd Villines, MD, FSCCT	Professor and Director of Cardiovascular Research and Cardiac CT Programs	University of Virginia
Kenneth Wang, MD, PhD	Adjunct Assistant Professor	University of Maryland, Baltimore
Not in Attendance
Arjun Venkatesh, MD, MBA, MHS	Assistant Professor	Yale School of Medicine

 

Ex Officio TEP
Mary White, ScD	Chief, Epidemiology and Applied Research Branch	Centers for Disease Control and Prevention
Not in Attendance
Amy Berrington de Gonzalez, DPhil	Branch Chief & Senior Investigator	National Cancer Institute; Division of Cancer Epidemiology & Genetics, Radiation Epidemiology Branch
CMS & CATA Representatives
Janis Grady	Project Officer	Centers for Medicare & Medicaid Services
Marie Hall	CATA Team	Health Services Advisory Group
UC Team
Rebecca Smith-Bindman, MD	Principal Investigator	University of California, San Francisco
Andrew Bindman, MD	Principal Investigator	University of California, San Francisco
Patrick Romano, MD, MPH	Co-Investigator	University of California, Davis
Naomi López-Solano, CCRP	Project Manager	University of California, San Francisco
Diana Ly, MPH	Project Manager	University of California, San Francisco

 

Technical Expert Panel Meeting

Prior to the meeting, TEP members received a copy of the agenda, presentation slides, link to DR-CTQS study website which contains minutes from the prior TEP meetings, honorarium documentation, and a conflict of interest form. The meeting was conducted with the use of PowerPoint slides.

9:00 AM           Call meeting to order by TEP Chair                                Dr. Helen Burstin

Dr. Helen Burstin called the meeting to order. She noted that the meeting will last for 1.5 hours and will include a discussion period after each presentation.

Part 2 of TEP #4 will be conducted at a later date, as response to the COVID-19 pandemic allows.

9:05 AM           Roll Call and Updated Conflicts                                       Dr. Helen Burstin

TEP members and Ex Officio members attendance listed above.

Conflict of interest defined as you, your spouse, your registered domestic partner, and/or your dependent children:

1. received income or payment as an employee, consultant or in some other role for services or activities related to diagnostic imaging

2. currently own, or have held in the past 12 months, an equity interest in any health care related company which includes diagnostic imaging as a part of its business

3. hold a patent, copyright, license or other intellectual property interest related to diagnostic imaging

4. hold a management or leadership position (i.e., Board of Directors, Scientific Advisory Board, officer, partner, trustee, etc.) in an entity with an interest in diagnostic imaging

5. received and cash or non-cash gifts from organizations or entities with an interest in diagnostic imaging

6. received any loans from organizations or entities with an interest in diagnostic imaging

7. received any paid or reimbursed travel from organizations or entities with an interest in diagnostic imaging

COIs were disclosed to UCSF prior to this TEP meeting via paperwork. No members had new financial conflicts that precluded their participation. TEP members were also asked to verbally disclose any COIs when introducing themselves for the purpose of group transparency. TEP members re-stated their affiliations and any existing conflicts. Dr. Helen Burstin stated her affiliation as the CEO of the Council of Medical Specialty Societies, and her status as a faculty member at the George Washington University Medical School. She is now on the board of the Society to Improve Diagnosis in Medicine, although this is not a conflict of interest. Dr. Jay Bronner stated his relationship with Radiology Partners, and had no conflicts of interest. Dr. Jeph Herrin stated his affiliation with Yale University, and no new conflicts of interest. Dr. Matthew Nielsen reported his affiliation with the University of North Carolina. He noted he is the Quality Improvement Chair at the American Urological Association, however this association is not directly related to imaging. Dr. Debra Ritzwoller stated her affiliation with Kaiser Permanente Colorado and as a patient/guardian stakeholder. Dr. Kenneth Wang noted his affiliation with the Veterans Administration in Baltimore and University of Maryland. Of note, he is participating on his personal time not representing government. His conflicts include a small start-up and occasional reimbursements from Radiology Society of North America. He also has a patent pending in the area of ultrasound imaging. Niall Brennan stated that he had no new conflicts and that he is currently the President and CEO of the Health Care Cost Institute. Dr. Hedvig Hricak is currently the Chair of the Memorial Sloan Kettering Cancer Center Department of Radiology. She disclosed her current conflict as a board member of IBA. Dr. Mythreyi Chatfield stated her affiliation with the American College of Radiology, as the Executive Vice President of Quality and Safety, and had no new conflicts of interest to disclose. Tricia Elliot restated her role as the Director of Quality Measurement at The Joint Commission, and no new conflicts of interest. Dr. Leonard Lichtenfeld reminded the panel of his role as the Interim Chief Medical Scientific Officer of the American Cancer Society. He did not have any conflicts but mentioned his stock ownership in Google and noted that they have some interest in using augmented intelligence in radiology analytics. Dr. Lewis Sandy stated his affiliation with UnitedHealth Group as the Executive Vice President of Clinical Advancement and had no new conflicts of interest to disclose. Suzanne Schrandt restated her role as the Director of Patient Engagement at the Arthritis Foundation. She also disclosed her new relationship as the Senior Patient Engagement Advisor for the Society to Improve Diagnosis in Medicine. Dr. Anthony Seibert stated his role as a medical physicist at UC Davis Health, and had no conflicts of interest to declare. Dr. Todd Villines stated his role as a cardiologist at the University of Virginia, he disclosed his changes in conflicts of interest to the TEP; he no longer has any relationships with industry stakeholders, and he is the editor in chief of the Journal of Cardiovascular CT, and he is a non-voting board member of the Society of Cardiovascular CT. Finally, Missy Danforth restated her role as the Vice President of Health Care Ratings at the Leapfrog Group, and had no new conflicts to declare. Dr. Mary White reported her affiliation with the Centers for Disease Control & Prevention, and had no new conflicts of interest.

 

9:15 AM: Method for Automating the Categorization of CT Scans, Dr. Rebecca Smith-Bindman

Dr. Smith-Bindman began with a review of measure concept, which is to identify diagnostic CT scans that are performed in an unsafe manner, either because they utilize excessive radiation doses (given the clinical indications for imaging) or because they have low image quality, undermining their diagnostic value. This also includes a balancing measure to ensure that indiscriminate efforts to reduce radiation dose do not compromise image quality. She reminded the TEP that the measure will evaluate at the level of each individual CT scan, and the level of analysis will be the practitioner or practitioner group. Each CT will be put into a category for the anatomic area and indication (CT-Cat) based on why the CT was obtained. Within each CT-Cat, a CT scan will then be assessed for “failure” on two criteria: 1) is the radiation dose too high for that category? and 2) is the image quality too low for that category? The second criterion will not be discussed as a part of the meeting today but will be the focus of discussion during the second part of the fourth TEP meeting (TEP#4, part 2).

The categories of the CT-Cat were established through a combination of literature review, empirical data from the UCSF International CT Dose Registry, and input from TEP members. Dr. Smith-Bindman displayed a graph (slide #14 of presentation) which illustrated the percentage of CT scans that fall within the 19 proposed CT categories of the CT-Cat. The categories that contained the highest proportion of scans were: Abdomen Routine (25%), Head Routine (24%), and Chest Routine (20%). The remaining 16 categories each accounted for 1-7% of CT scans.

One of challenges in implementing the measure will be to put the CT scans into the CT-Cat categories in an automated fashion. The UC project team has been using data from the UCSF health system to test (Alpha 2) the accuracy of the automated approach to determine and validate the accuracy of determining the indications for CT exams. The project team has developed two approaches for automated assignment of CT-Cat. The first approach uses electronic health record (EHR) data (i.e. the diagnostic codes associated with a test order) in combination with electronic billing codes. An expert coder mapped specific procedure codes (Current Procedural Terminology/Healthcare Common Procedure Coding System (CPT/HCPCS)) and diagnostic codes (International Classification of Diseases, Tenth revision, Clinical Modification (ICD-10CM)) to each CT-Cat.

The second method uses the DICOM data stored with each CT radiology record in combination with billing codes. Some of the DICOM fields are standaradized but the reason for the scan, protocol name, and study description are free text. The UC team has applied natural language processing (NLP) to these fields.

Dr. Smith-Bindman discussed the tradeoffs of both approaches, mainly; EHR data is more difficult to obtain and may be incomplete, and DICOM data is not fully standardized and potentially gameable. The Alpha-2 Testing was performed on 4,153 UCSF patients who received a CT scan. The UC project team developed a “gold-standard approach” wherein the CT-Category was determined via a detailed chart review, and then compared to the assignment to a CT-Cat based on the EHR and DICOM approaches. An assessment was made of the sensitivity, specificity, and overall accuracy of each of the two automated approaches.

When creating automated rules with the EHR and DICOM data for assigning a CT scan to a particular CT-category, the UC project team aimed to minimize cases in which radiologists might mistakenly be penalized for using higher doses. This was operationalized by maximizing the sensitivity for high dose categories and maximizing the specificity low dose categories. Also for CT scans where there was more than one indication, the CT scan was defaulted to the higher radiation dose category.

Based on the alpha testing, the DICOM approach accurately categorized 92% of CT scans. The EHR derived algorithm, accurately categorized 80% of scans.

Dr. Smith-Bindman displayed a table (slide #20 of presentation) that showed the sensitivity, specificity, and likelihood ratio of the accuracy of the DICOM data approach for each category of the CT-cat. Sensitivity ranged from 0.79 to1.0, across categories, and specificity ranged from 0.92 to1.0 across categories. The positive and negative likelihood ratios were indicative of a high level of accuracy, especially for the the DICOM data approach.

The next steps for validating CT Categories were then discussed. Thus far, the plan is to go forward with testing the two approaches for determining CT-Cat, but the UC project team is currently leaning towards the DICOM method because of the greater accuracy in this approach. The team plans to validate this approach among different groups of physicians such as: University of California health systems at Davis, San Diego, and Irvine, a private practice with imaging centers in Austin, Texas , and large academic hospital-based practices at Mt Sinai Hospital in New York, New York, and Henry Ford Hospital in Detroit, Michigan. These locations reflect diversity in practice, EHR, types of CT scanners, and geographic location.

 

9:35 AM Discussion: Method for Automating the Categorization of CT Scans, Dr. Burstin

Dr. Burstin opened up the meeting to discussion of these topics. (Dr. Wang) began with a clarifying question regarding use of the claims data. Dr. Smith-Bindman clarified that the billing codes contribute to both approaches for determining CT-cat (EHR and DICOM.)

(Dr. Chatfield) The ACR representative suggested that the American College of Radiology (ACR), Dose-Index Registry data might provide another place in which to test the accuracy of CT-Cat. .

(Dr. Sandy) suggested that there may need to be a process to enable a practitioner or medical group the opportunity to review the automated assignment as a way to re-assure that assessments of the radiation dose and image quality are being done on the proper CT-Cat. (Dr. Burstin) backed this suggestion, as such processes can help build the reliability of the measure over time.

(Dr. Villines) indicated that he was positively impressed by the sensitivity and specificity numbers that were shared in this portion of the presentation. He asked whether the project team anticipated any issues with applying the automation rules to different EHRs across or different PACS Systems?

Dr. Smith-Bindman explained that the team has chosen sites that vary in the types of CT scanners, EHRs and PACS systems. The project team also plans to collect data using different approaches aside from the UCSF developed software tool. This would include data reporting from dose-management software companies and by the CT manufacturers. The different data collection methods will be compared.

(Dr. Burstin) expressed confusion about DICOM not being reliable, as it was her understanding that this is an international standard. She also expressed that she felt that “game-ability” of the measure is more likely to occur via the billing claims data.

Dr. Smith-Bindman clarified that while some DICOM fields are standardized the ones used for CT-Cat are not. The NLP approach to these data developed at UCSF will be tested at the other sites. Furthermore there are shifts underway which will likely make the “reason for study” field in the DICOM data more consistent across sites. ACR has developed decision support software that is likely to be widely adopted, and that standardizes the indication for imaging. This field will be used to populate this ‘reason for study” field. Because radiologists influence the recording of the reason for scan in the DICOM data there is the potential that they could alter what is recored in anticipation of how they might be judged on a quality measure. The UC team believes that this is unlikely to be a problem prior to the implementation of the measure and that systems could be put in place to monitor this over time.

Dr. Bindman pointed out that one potential benefit of the EHR data approach is that the information is entered by the practitioner who orders the test not the practioner that performs the test. This makes it less gameable.

(Dr. Burstin) suggested that there may be a difference in the accuracy of CT-Cat between large health care system versus single radiologist practices. It may be difficult to get a reasonable reliability estimate for an individual doctor with much smaller sample sizes. Dr. Burstin requested that future meetings provide information to help form a judgment of whether it will be practical and valid to have this measure apply to invidual practitioners.  Dr. Smith-Bindman agreed to provide such data at future TEP meetings.

(Dr. Siebert) asked a clarifying question about, in terms of the CT Categories what is considered abdomen and what is considered pelvis; i.e. where does one end and another begin? Dr. Smith-Bindman responded that for the purposes of this work, the abdomen category is defined as including any abdomen and any pelvis. They are combined together. This is based on her previously published research that the two categories are almost indistinguishable when looking at a CT scan.

In response to this, (Dr. Hricak) mentioned that there is an opportunity to evaluate when CT scans inappropriately include certain anatomical areas that are not clinically indicated. Dr. Smith-Bindman acknowledged the value of such a measure but that it was beyond the scope of the current work measuring CT radiation doses for the scans as performed.

 

9:50 AM: Method for Setting the Upper Radiation Dose Threshold,   Dr. Andrew Bindman

Dr. Bindman began with review of the purpose of an upper radiation dose threshold. This measure requires that a radiation dose threshold be established for each CT scan, above which a scan will be rated as failed. The threshold will be specific for each CT-Cat. For example, the upper limit for high dose abdomen will be greater than the upper limit for routine abdominal scans. The goal is to set an upper threshold as low as possible to support safety, but not so low that it risks the quality of the image. Because larger patients require greater dose, the measure will be adjusted within each CT-Cat for patient size.

To set the upper radiation dose threshold the UC project team combined information from the UC International CT Dose Registry (UC Dose Registry) with data collected in a study of radiologists who rated the image quality of a test set CT scans. The UC Dose Registry provided empirical data on the distribution of radiation doses used within each CT-Cat from the 151 participationg institutions. These data were combined with the assessments from the 125 radiologists who each rated the image quality of 200 of the test cases. These test cases (N=740) were sampled from actual cases from UC Dose Registry and were selected to represent the four largest CT manufacturers. These test cases were sampled across the range of radiation doses within a CT-Cat and were intentionally slightly oversampled at the low end of dose where it was assumed that most of the image quality issues would arise. The radiologists were asked to rate the images on a 4-point scale defined as follows: Excellent (“images provide the needed information”), Adequate (“image quality is acceptable but not excellent. You would re-scan and change the parameters for a higher quality image if it is easy to repeat, but if not, this is good enough for what you need.”), Marginally Acceptable (“image quality is less than ideal and may compromise diagnostic quality. If the patient cannot easily be re-scanned you will interpret this but would change parameters for future scans of this type.”), and Poor (“image quality is not adequate for diagnosis and should be repeated.”).

The majority of cases were rated as having sufficient image quality (Excellent= 49%, Adequate= 40%, Marginally Acceptable= 8%, Poor= 3%). For most CT-Cats, the percentage of readings in which the ratings were adequate or excellent increased with dose, but the percentage change relative to the dose distribution was small. For a few CT-Cats there was no association between radiologists’ assessment of quality and dose. The number of cases rated as poor or marginal varied among radiologists. Recognizing these differences, the data were adjusted for how hard of a grader a radiologist was. A graph was displayed of the proportion of interpretations into each of the 4 categories by radiologist reader (slide #32).

The UC project team used the radiation doses on the test cases within a CT-Cat superimposed on the distrigution of doses within that CT-Cat from the UC Dose Registry. This allowed the UC project team to identify when increases in radiation doses did not meaningfully contribute to a higher proportion of radiologists rating the image quality as excellent, adequate or marginally acceptable. The table demonstrating the CT Scan image quality rating by observed dose can be found on slide #33.

Dr. Bindman then proposed a rule for setting the upper radiation dose threshold. The rule would set the upper limit of acceptable (non failing) where at least 98% of radiologists assess images as being excellent or adequate or marginally acceptable AND at least 90% of physicians rate the dose as excellent or adequate. He then demonstrated how this rule would play out in the UCSF data across a range of CT-Cats (figure found on slide #36).

If these proposed rules were applied, on average, approximately 20% of CT scans would be considered above threshold in 13 CT-Cats. For two categories, where even the lowest observed dose satisfies the criteria and for the four categories in which there is no association between radiation dose and image quality, the UC project team proposes that the upper threshold is set at the average reduction of the 13 CT-Cats.

 

10:10 AM Discussion: Method for Setting the Upper Radiation Dose Threshold, Dr. Burstin

(Dr. Villines) expressed his approval for the approach that has been developed thus far. In terms of the categories of physician assessment of image quality, he noted that in his clinical practice, “Excellent” is usually defined as textbook quality, if not too much radiation dose, while “Adequate” is usually defined as typical diagnostic quality. He also brought up if the complicating factor of image noise will be brought into the analysis and shared with the TEP in this presentation. Dr. Bindman replied that the question of image noise will be one topic of discussion during Part 2 of TEP #4.

(Dr. Bronner) asked if the clinical history associated with the CT exams used in the Image Quality Sub-Study was known to the radiologist readers. Dr. Bindman replied that a simplified history was provided to the participants that would have been sufficient for them to be oriented to the proper CT-Cat..

(Dr. Sandy) had a question about where to set the threshold. He posited a simpler rule: setting the threshold at the modal point of every scan that is studied. He felt that the UC projet team approach was too lenient. (Dr. Chatfield) from the ACR said that she agreed that we were being too lenient and pointed out the ACR’s convention is to set the 75^th percentile as the upper limit.

Dr. Bindman asked the TEP if anyone thought that the current construction of the thresholds was too aggressive. The TEP responded no, although some members thought that there is a level of complexity that may make it difficult for radiologists to interpret their results.

 

10:25 AM       Wrap Up and Next Steps                                                     Dr. Bindman1                                                                                                                                                                                                                  

Dr. Bindman thanked the TEP for their advice and noted that the UC project team would reflect on the input and incorporate it into discussions with CMS. He stated that the agenda for Part 2 of TEP #4 would include presentations on the proposed method for automating evaluation of image quality to ensure that doses below upper threshold are not reduced so much as to undermine image quality.

He reminded the TEP that information about this meeting and future meetings will be posted at ctqualitymeasure.ucsf.edu, as well as reminded the members of the possibility to receive an honorarium for their participation.

This meeting will occur via webinar in late May or early June. Members of the UCSF team will reach out to TEP members to begin scheduling this follow-up meeting.

 

10:30 AM       Adjourn                                                                                             Dr. Burstin