Meeting Date: 8/5/2020 Meeting Time: 11:00am-1:30pm PDT Meeting Location: Virtual Conference via Zoom Approval Date: 8/27/2020 Recorded by: UCSF Team

MEETING MINUTES:

Project Overview:

The Centers for Medicare & Medicaid Services (CMS) has granted an award to the University of California San Francisco (UCSF) to develop a measure of computed tomography (CT) image quality and radiation safety. The project is a part of CMS’s Medicare Access & CHIP Reauthorization Act (MACRA)/Measure Development for the Quality Payment Program. The project title is “DR CTQS: Defining and Rewarding Computed Tomography Quality and Safety”. The Cooperative Agreement number is 1V1CMS331638-02-00. As part of its measure development process, UCSF convened groups of stakeholders and experts who contributed direction and thoughtful input to the measure developer during measure development and maintenance.

Project Objectives:

The goal of the project is to create a quality measure for CT to ensure image quality standards are preserved and harmful effects of radiation used to perform the tests are minimized. Radiation doses delivered by CT are far higher than conventional radiographs (x-rays), the doses are in the range known to be carcinogenic, and there is a significant performance gap across health care organizations and clinicians which has consequences for patients. The goal of the measure is to provide a framework where health care organizations and clinicians can assess their doses, compare them to benchmarks, and take corrective action to lower them while preserving the quality of images so that they are useful to support clinical practice. The measure will be electronically specified using procedural and diagnostic codes in billing data as well as image and electronic data stored with CT scans, typically stored within the Picture Archiving and Communication Systems (PACS) - the computerized systems for reviewing and storing imaging data or Radiology Information Systems (RIS).

TEP Objectives:

In its role as a measure developer, the University of California San Francisco is obtaining input from a broad group of stakeholders to develop a set of recommendations to develop a radiology quality and safety measure. The proposed measure will be developed with the close collaboration of the leadership from diverse medical societies as well as payers, health care organizations, experts in safety and accreditation, and patient advocates. A well-balanced representation of stakeholders on the TEP is intended to ensure the consideration of key perspectives and obtain balanced input.

Scope of Responsibilities:

The TEP’s role is to provide input and advice to the measure developer (University of California San Francisco) related to a series of planned steps throughout the 3-year project. The specific steps will include developing and testing a risk-adjusted measure which can be used to monitor CT image quality in the context of minimizing radiation doses while maintaining acceptable image quality. The TEP will assist UCSF in conceptualizing the measure and any appropriate risk adjustment of it. The TEP will assist UCSF with identifying barriers to implementing the proposed measure and test sites in which the developer can assess the feasibility and performance of its use. The TEP will assist UCSF with interpreting results obtained from the test sites and in suggesting modifications of the measure prior to it being incorporated into a software tool which will be made available to providers to enable them to report and monitor their performance. The TEP will provide input and advice to UCSF regarding the software tool to ensure that it is valuable for a wide range of stakeholders and CMS.

Guiding Principles:

Participation on the TEP is voluntary. Individuals participating on the TEP understand that their input will be recorded in the meeting minutes. Proceedings of the TEP will be summarized in a report that may be disclosed to the general public. If a participant has disclosed private, personal data by his or her own choice, then that material and those communications are not deemed to be covered by patient-provider confidentiality. Questions about confidentiality will be answered by the TEP organizers.

All potential TEP members must disclose any significant financial interest or other relationships that may influence their perceptions or judgment. It is unethical to conceal (or fail to disclose) conflicts of interest. However, the disclosure requirement is not intended to prevent individuals with particular perspectives or strong points of view from serving on the TEP. The intent of full disclosure is to inform the TEP organizers, other TEP members and CMS about the source of TEP members’ perspectives and how that might affect discussions or recommendations.

All potential TEP members should be able to commit to the anticipated time frame needed to perform the functions of the TEP.

Estimated Number and Frequency of Meetings:

TEP is expected to meet three times per year, either in-person or via a webinar.

This meeting was originally set to occur in-person, but was changed to a virtual meeting as mandated by federal social distancing measures and state-wide Shelter-in-Place orders.

Table 1. TEP Member Name, Title, and Affiliation

Name	Title	Organization
Attendees
Mythreyi Bhargavan Chatfield, PhD	Executive Vice President	American College of Radiology
Niall Brennan, MPP	CEO	Health Care Cost Institute
Jay Bronner, MD	President and Chief Medical Officer	Radiology Partners
Helen Burstin, MD, MPH, FACP	Executive Vice President	Council of Medical Specialty Societies
Tricia Elliot, MBA, CPHQ	Director, Quality Measurement	Joint Commission
Jeph Herrin, PhD	Adjunct Assistant Professor	Yale University
Hedvig Hricak, MD, PhD	Radiology Chair	Memorial Sloan Kettering Cancer Center
Name	Title	Organization
Attendees
Leonard Lichtenfeld, MD, MACP	Interim Chief Medical Officer	American Cancer Society, Inc.
Matthew Nielsen, MD, MS	Professor	UNC Gillings School of Global Public Health
Debra Ritzwoller, PhD	Patient	Patient Representative
Lewis Sandy, MD	Executive Vice President, Clinical Advancement	UnitedHealth Group
Mary Suzanne Schrandt, JD	Patient	Patient Representative
Anthony “Tony” Seibert, PhD	Professor	University of California, Davis
Kenneth Wang, MD, PhD	Adjunct Assistant Professor	University of Maryland, Baltimore
Not in Attendance
Missy Danforth	Vice President of Health Care Ratings	The Leapfrog Group
Arjun Venkatesh, MD, MBA, MHS	Assistant Professor	Yale School of Medicine
Todd Villines, MD, FSCCT	Professor and Director of Cardiovascular Research and Cardiac CT Programs	University of Virginia

 

Ex Officio TEP
Amy Berrington de Gonzalez, DPhil	Branch Chief & Senior Investigator	National Cancer Institute; Division of Cancer Epidemiology & Genetics, Radiation Epidemiology Branch
Mary White, ScD	Chief, Epidemiology and Applied Research Branch	Centers for Disease Control and Prevention
CMS & CATA Representatives
Janis Grady	Project Officer	Centers for Medicare & Medicaid Services
Not in Attendance
Marie Hall	CATA Team	Health Services Advisory Group
UC Team
Rebecca Smith-Bindman, MD	Principal Investigator	University of California, San Francisco
Andrew Bindman, MD	Principal Investigator	University of California, San Francisco
Patrick Romano, MD, MPH	Co-Investigator	University of California, Davis

Prior to the meeting, TEP members received a copy of the agenda, presentation slides, link to DR-CTQS study website which contains minutes from the prior TEP meetings, honorarium documentation, and a conflict of interest form. The meeting was conducted with the use of PowerPoint slides.

11:00 AM:      Call meeting to order by TEP Chair                                  Dr. Helen Burstin

Dr. Helen Burstin called the meeting to order. She noted that the meeting will last for 2.5 hours and will include a discussion period after each presentation.

11:05 AM:      Roll Call and Updated Conflicts                                                     Dr. Burstin

TEP members and Ex Officio members attendance listed above.

Conflict of interest defined as you, your spouse, your registered domestic partner, and/or your dependent children:

1. received income or payment as an employee, consultant or in some other role for services or activities related to diagnostic imaging

2. currently own, or have held in the past 12 months, an equity interest in any health care related company which includes diagnostic imaging as a part of its business

3. hold a patent, copyright, license or other intellectual property interest related to diagnostic imaging

4. hold a management or leadership position (i.e., Board of Directors, Scientific Advisory Board, officer, partner, trustee, etc.) in an entity with an interest in diagnostic imaging

5. received and cash or non-cash gifts from organizations or entities with an interest in diagnostic imaging

6. received any loans from organizations or entities with an interest in diagnostic imaging

7. received any paid or reimbursed travel from organizations or entities with an interest in diagnostic imaging

COIs were disclosed to UCSF prior to this TEP meeting via paperwork. No members had new financial conflicts that precluded their participation. TEP members were also asked to verbally disclose any COIs when introducing themselves for the purpose of group transparency. TEP members re-stated their affiliations and any existing conflicts. Dr. Helen Burstin stated her affiliation as the CEO of the Council of Medical Specialty Societies and had no new conflicts of interest. Dr. Mythreyi Chatfield stated her affiliation with the American College of Radiology (ACR), and had no new conflicts of interest to disclose. Niall Brennan stated that he had no new conflicts and that he is currently the President and CEO of the Health Care Cost Institute. Dr. Jay Bronner stated his relationship with Radiology Partners, and had no conflicts of interest. Tricia Elliot restated her role as the Director of Quality Measurement at The Joint Commission, and no new conflicts of interest. Dr. Jeph Herrin stated his affiliation with Yale University, and no new conflicts of interest. Dr. Leonard Lichtenfeld reminded the panel of his role as the Interim Chief Medical Scientific Officer of the American Cancer Society. He did not have any conflicts but mentioned his stock ownership in Google and noted that they have some interest in using augmented intelligence in radiology analytics. Dr. Matthew Nielsen reported his affiliation with the University of North Carolina. Dr. Debra Ritzwoller stated her affiliation with Kaiser Permanente Colorado and as a patient/guardian stakeholder, and had no new conflicts. Dr. Kenneth Wang noted his affiliation with the Veterans Administration in Baltimore and University of Maryland. Of note, he is participating on his personal time and not representing government, and stated no new conflicts. Dr. Mary White reported her affiliation with the Centers for Disease Control & Prevention, and had no new conflicts of interest. Dr. Hedvig Hricak is currently the Chair of the Memorial Sloan Kettering Cancer Center Department of Radiology. She disclosed her current conflict as a board member of IBA. Dr. Lewis Sandy stated his affiliation with UnitedHealth Group as the Executive Vice President of Clinical Advancement and had no new conflicts of interest to disclose. Suzanne Schrandt restated her role as the Director of Patient Engagement at the Arthritis Foundation, and reported no new conflicts. Dr. Anthony Seibert stated his role as a medical physicist at UC Davis Health, and had no conflicts of interest to declare.

11:15 AM:      Updates                                                                               Dr. Andrew Bindman

Dr. Bindman provided the TEP with updates related to; (1) progress on the timeline, and (2) the potential for having the developed measure adopted in other CMS programs in addition to MIPS.

Dr. Bindman informed the TEP that UCSF is 22 months into a 36-month long project. UCSF has provided CMS with an update on progress to date as a part of a process to request the third year of funding. UCSF is hopeful that CMS will agree that the project is on track and merits the on-going support.

Dr. Bindman updated the TEP about the impact of COVID-19 on the project. He reported that the core research team at UCSF, UC Davis and associated collaborators at Duke University and the University of Maryland have been able continue to work together online without a major interruption. The major way in which COVID-19 has impacted the project is in working with external testing sites. While those sites, for the most part, have remained very committed to the idea of working with us on the project, the IT partners at these sites have faced competing demands, such as; getting telehealth operating to enable their health care system to remain functional.This has created some delays in terms of when data will be collected from those testing sites. Recognizing this, adjustments to the testing plan have been made to ensure that all the necessary results will be collected in order to complete the project on time. Dr. Smith-Bindman then provided a description of a schematic showing how testing will be conducted at six different healthcare systems. There are three different parts of testing. Beta one focuses on the Computed Tomography (CT) categories and our assessment of the upper dose threshold. Beta two focuses on the quality measure that is the primary focus of todays TEP meeting. Beta three will include an assessment of the burden of data collection on providers. While the original plan was to move all testing sites together through these different phases, UCSF now plans to move sites through the different beta testing phases on a rolling basis. UCSF believes that learning from the process of onboarding the first sites can be used to accelerate the testing process with the sites that are delayed.

Dr. Bindman thanked the TEP for the prior feedback that identified that a challenge related to using the measure in the Merit-based Incentive Payment System (MIPS) Program is that physicians who perform these tests in the context of a hospital setting may have difficulty gaining access to the data that are needed to submit to the MIPS Program.This is because hospitals typically own and operate the IT system. If the hospital was not part of any incentive structure related to the reporting, hospital leaders may not prioritize making the data available to the radiologist. With the help of our CMS project officer, Janis Grady, Dr. Bindman and Dr. Smith-Bindman have had several conversations with leaders within the CMS quality programs. In response, CMS encouraged UCSF to submit the CT measure to the Measures Under Consideration (MUC) list for the Inpatient Quality Reporting Program (IQR) program as well as the Outpatient Quality Reporting Program (OQR) and the Critical Access Hospital (CAH) programs. While it may prove difficult to complete testing in time for the measure to be adopted by CMS for 2021, and there are some important differences in the reporting of how these programs work, UCSF regards the potential widespread adoption of the measure across differerent CMS quality programs as a positive development. The IQR, OQR and CAH programs do not use a registry-based way of reporting, the same way that there is available for physicians. UCSF is exploring the possibility of developing an Electronic Clinical Quaity Measure (eCQM),  which is a digital way of reporting information from electronic record reporting systems. While there are many hurdles to overcome, UCSF is committed to working with CMS to make this happen, due to the team’s belief that having the measure implemented in hospital and physician-based quality programs would help to align incentives for improving quality of care. 

Dr. Bindman solicited feedback from the TEP on the project updates. UCSF received a positive response to the possibility of having the measure adopted in hospital as well as physician-based quality programs. A question was raised about the timing of the measure moving forward on the MUC list, Dr. Bindman replied that this was still being discussed with CMS. Another question was asked as to whether UCSF still planned to rely on a QCDR for the MIPS measure, and Dr. Bindman confirmed that was still the plan. While the hospital programs do not make use of QCDRs, Dr. Bindman said that UCSF would try to make the data collection process of hospitals  via an eCQM similar to what the QCDR would do in the MIPS program. Another question was asked about whether the data fields for making the measure an eCQM existed. Dr. Bindman said that while the digital data necessary for the measure existed in DICOM standardized data fields within electronic health records, it had not yet been mapped for the purposes of quality measure development, but that CMS was in conversations with the National Library of Medicine on UCSF’s behalf to make this possible. The TEP representative from the Joint Commission not only endorsed the idea of a hospital based measure, but expressed interest in working with UCSF to help make sure that any problems in the data fields could be addressed to help get the measure implemented. Dr. Smith-Bindman acknowledged that some data fields in the Digital Imaging and Communitcaitons in Medicine (DICOM) may not be reliable but that she believes that the ones needed for the measure, such as the reporting of radiation dose, are done in a valid and standardized way across hospitals. The representative from the ACR raised a question about whether data from DICOM are accurately stored in electronic health records. Dr. Smith-Bindman stated that as a part of Beta testing, UCSF is testing whether it can extract data diretly from the DICOM and combine that with other data in the electronic record, such as that on the clinical indication for the CT exam, as well as the billing type, as a way to accurately categorize CT scans and assess the radiation dose. The TEP will have an opportunity to review these testing data at upcoming TEP meetings.

11:35 AM:      Approach to Setting Upper Dose Thresholds                                Dr. Bindman

Dr. Bindman reviewed the strategy discussed at the prior TEP meeting (TEP #4 Part I) for setting the upper radiation dose threshold. The plan is to place CT scans into dose range categories based on anatomical area and clinical indications. Then, using the ratings 125 physicians made in assessing 740 test cases in these categories with varying levels of radiation, the following rule was proposed: the upper dose threshold should be no lower than where at least 98% of the physicians assess the images as being excellent or adequate or marginally acceptable, which is the same as saying fewer than 2% rate the images as poor, and that at least 90% of the physicians are saying that the dose is excellent or adequate, which is the same as saying 10% rating the images as poor or marginally acceptable. The dose would be adjusted for patient size. The challenge of this definition was that in some categories there was no dose in the range tested that did not meet the criteria, meaning every observed dose would be considered acceptable. UCSF did not believe setting the lowest possible dose in those categories as the upper dose threshold was reasonable, and had proposed therefore to lower the upper dose by the percentage change in the categories where an upper threshold using the defined criteria was acceptable. However, the feedback of the TEP was that this approach was too conservative. The recommendation was to incorporate median doses into our decision rules for categories in which even the lowest observed dose satisfied the criteria. UCSF created a modified rule with the following criteria: at least 90% of the physicians believe the dose is excellent or adequate, and for categories where even the lowest observed dose met this criteria, that the median dose would be considered the upper dose threshold. UCSF modelled this in the UCSF Dose Registry and found that applying this modification to the rule would result in 38% of the cases in being categorized as out of range. The UCSF team requested TEP input on this modification. At the prior TEP meeting,  some members also objected to the use of the term “failure” when applied to scans that exceed the dose threshold. UCSF has proposed to define these exams as “out of range”.

12:00PM:       Discussion; Approach to Setting Upper Dose Thresholds               Dr. Burstin

A couple of the TEP members raised the concern that using the median could be problematic because over time the median would change as doses come down. UCSF clarified that the median would only be used for determing the dose for setting a threshold at the outset and that the median would not be re-calculated each year. It would simply be the way of determining the upper threshold for those categories in which even the lowest observed doses among the test cases fulfilled the criteria. A question was asked by Dr. Siebert as to whether, in addition to patient size, the assessment should be assessed for type of CT machine and their varying capabilities. Dr. Smith-Bindman answered by saying that UCSF has found when they have studied it in data from the UCSF Dose Registry, that actually the imaging machine used, the manufacturer, and the model, are relatively small predictors of dose. They do matter, but they don't matter very much, and that the much more important predictor is how the scanner is being used. These findings were published in the British Medical Journal (BMJ) last year. Furthermore, Dr. Smith-Bindman stated that in discussing this with four leading manufacturers at the Radiological Society of North America (RSNA) annual convention, they all confirmed from their own analysis that almost every scanner out there, the brand new ones or the old ones have the ability to use lower doses. The manufacturers acknowledged that they have struggled with how to get their physicians who buy their equipment to use it most effectively. Dr. Siebert stated after Dr. Smith-Bindman’s answer that he fully agreed with her. Dr. Bindman then read from the chat that Dr. Sandy and Dr. Bronner both wrote that they thought the revised threshold relying on the median looked about right. Dr. Burstin raised that she wondered if 38% of cases being out of range is too high and in particular wondered if it was falling disproportionately on some categories. The representative from the ACR also raised concerns about the percentage of scans that might be rated as out of range. Dr. Bindman reminded the TEP that even if that many scans were rated as out of range that the assessment of an individual clinician or medical group would still be done along a curve based on the percentage of scans that are out of range. A patient representative asked if there were clinical indications that might be a reason for higher doses. Dr. Bindman reminded the TEP that clinical indications are included in setting dose ranges within anatomical areas. Dr. Smith-Bindman elaborated on that by pointing out that bias in setting up these categories was to allow clinicians to have certain scans that ordinarily could be done with lower doses to be placed in the higher dose categories if there was anything in the clinical indications that might suggest a need for it. Another question was asked as to whether the UC Dose Registry was sufficiently representative for setting the thresholds. Dr. Smith-Bindman said that the UCSF Dose Registry has scans from more than 150 imaging facilities and that comparisons with ACR’s registry has revealed very similar dose ranges by anatomical areas. For example, the distribution in dose for kidney stones studies in the UCSF Dose Registry is identical to the dose in the ACR registry. The dose in the UCSF Dose Registry for patients who undergo non-contrast abdomen or multi-phase contrast are the same as in the ACR registry. In addition, as a part of this project, UCSF will be building a QCDR which would ultimately be used for reporting to CMS. This QCDR would provide another means of assessing dose ranges used in practice and provide a point of comparison with the UCSF Dose Registry.

While there was no formal vote on adopting the term “out of range” instead of failure or the application of the median dose as the upper dose threshold  for those CT categories in which there is no dose at which at least 90% of radiologists rate the images as excellent or adequate, the TEP members acknowledged that UCSF had been responsive to the earlier feedback and had proposed a viable plan to proceed.

12:15PM:       Approach to Assessing Image Quality                 Dr. Rebecca Smith-Bindman

Dr. Smith-Bindman provided a presentation on how the project team is approaching the assessment of image quality. The point of this work is not to identify distinctions at the high end of quality, but to protect against the risk that as payment is used to incentivize the use of lower radiation doses, that the doses that are used are not too low and thereby undermine sufficient image quality to make accurate diagnoses. The purpose of this quality component is not to maximize image quality, it is to ensure there is adequate image quality to make diagnosis. UCSF does not enter into this work with the belief that image quality is currently a significant issue, and this is supported by the results of the image quality study that UCSF conducted. In rating the test case images from across a wide range of CT categories and doses, 49% of all radiologists’ ratings were excellent, 40% were adequate, 8% were marginally acceptable and only 3% were rated as poor. For most CT categories, the percentage of ratings as adequate or excellent increase with dose. For most categories, as there is more dose use, there were a higher percentage of excellent and adequate examinations, although the changes were relatively small. There were some categories where there was no association between the rating of image quality and radiation dose.

The concept behind the image quality study is to develop a radiologist concensus of a gold standard of inadequate image quality. Inadequate image quality was defined as those scans rated by a significant number of radiologists as marginally acceptable or poor. Using that gold-standard, UCSF undertook a process to automate the way in which CT scans with inadequate image quality could be identified using radiomics and machine learning. Radiomics describes measurements from radiology images that are stored data as part of the image. The four assessed measures are noise, noise texture, resolution, and contrast. Noise measures level of fluctuation in the images that's not due to the anatomical changes, these anatomical changes between lung and liver, but these are the fluctuations that don't depend on change in the tissue composition. Noise texture has to do with the average visual texture of those fluctuations through how radiologists will see those differences. Resolution reflects how sharp the images are, and a radiologist’s capacity to see differences between smaller and smaller lesions. Contrast reflects the average level of signal differentiation represented on images.The project team used machine-learning algorithms that learn from the data to figure out how these radiomic measures collectively relate to our radiologists’ gold standard scores of image quality. For machine learning of image quality, the algorithm is trained on a subset of the 740 cases that include the radiologist's gold standard assessment of image quality for each case, and the algorithm used artificial intelligence (AI) to learn from the radiologist's assessments, and then the algorithm is tested on a separate subset of these examinations blinded to the radiologists’ interpretation.

The cases were labeled as inadequate based on a predetermined threshold where 25% of radiologists interpreted the scan as marginal or poor quality. This meant at this threshold that  25% of radiologists thought the images were marginal or poor, which UCSF interpreted as meaning that the quality of the exam was not good enough. If the proportion were set lower, that would result in a large number of out-of-range scans. If the proportion were set higher, to identify scans that are an even worse quality, then it becomes an extremely rare event and almost impossible to find. For most of the categories, there would  not be a sufficient number of cases that would be characterized as out of range when there is a requirement for a substantially higher number of radiologists rating these cases as marginal or poor quality. 

To test how well the automated AI approach works to identify inadequate images the UCSF project team calculated the area under an Receiver Operating Characteristic (ROC) curve plotting the sensitivity by the false-positive rate using the radiologists’ reading as the gold standard. The higher the area under the curve the better the automated approach is working. An area under the curve of 0.7 to 0.8 is considered acceptable, 0.8 to 0.9 is considered excellent and greater than 0.9 is considered outstanding. Combining all of the categories, the generated Area Under Curve (AUC), using the automated approach is 89%. Similarly, the AUC values for the head categories, for the chest categories, and the abdomen categories, were all in a range considered excellent or outstanding.

While the ROC curve analysis can provide an estimate of overall performance it does not address the sensitivity and specificity at single point along the ROC curve. To understand the performance of the automated approach when applied in clinical practice, UCSF evaluated the positive predictive value. This describes how many cases identified as inadequate are truly inadequate. The positive predictive value is influenced by the prevalence. UCSF modelled a false-positive rate at 5%, and then at 10%. UCSF thought that a 10% false positive rate would be as high as the practicing community would tolerate.

For example, for head CT, if UCSF set the false-positive rate as being 5%, 23% of truly inadequate cases would be identified. If the false-positive rate is 10%, then the sensitivity for finding inadequate cases increases to 40%. The corresponding positive predictive value with a 5% false-positive rate is 46%, while for a 10% false-positive rate, the positive predictive value is 36%. The positive predictive value is better at a 5% false-positive rate, but the sensitivity is better with the 10% false-positive rate.

With a false-positive rate of 5%, the positive predictive values for different anatomical areas range from 31% to 54%. These results are preliminary and UCSF is in the process of replicating them with an independent team. To put these results in context, the overall positive predictive value of screening mammography nationally is only 4% with a false positive rate of 11% . The proposed image quality measure will perform better than this, but there will still be some images falsely labelled as inadequate. This is likely to be random in which case it will create some background noise in the assessment of radiologists, but should not skew their performance relative to one another. UCSF will be exploring potential confounders to try to ensure that the false positives are random and not systematically associated with certain practitioners. For example, in preliminary analysis UCSF did not find that patient size was associated with false positives.

1:00PM:         Discussion; Approach to Assessing Image Quality                           Dr. Burstin

For the discussion, TEP members were asked to comment on whether a 25% threshold of physician ratings of a scan as inadequate seemed appropriate, and whether they believed a 5% or 10% false-positive rate was an acceptable rate when trying to identify the truly abnormal quality exams. Dr. Siebert asked a clarifying question as to whether images rated as inadequate quality could be due to things such as patient movement or other artifacts rather than radiation dose. UCSF team explained that the test cases were curated to eliminate those that would be rated as inadequate because of patient movement or other artifacts, so as to isolate the impact of the radiation dose in the assessment. Dr. Siebert went on to comment that while he would prefer a 5% rather than a 10% false positive rate he recognized the impact on the positive predictive value and could see if needed the application of a 10% false positive rate. A patient representative acknowledged that this was a rather technical question, but that she hopes to get input of a larger cross-section of the patient community to weigh in on this in the future. In considering the sensitivity of finding poor quality images, Dr. Bindman made the point that it may be less critical to find all of them, and more critical to have a reliable way of finding at least some of them so that there is a signal back to radiologists that going too low with doses does carry some risk of being penalized. Dr. Bindman said this was akin to knowing that audits occur on some tax returns, and that this acts as a deterrent for more than just those who receive the audit. Dr. Wang endorsed this way of thinking and suggested this might allow for a lower false positive rate. He also raised a question about whether the 25% threshold was high enough because still 75% of radiologists were saying image was adequate or excellent. Other TEP members thought this had merit. Dr. Smith-Bindman said she would look into a higher threshold, but raised the concern that such rare cases might be even harder to identify using an automated approach. TEP members appreciated the care the UCSF team has taken to identify poor quality images, but also suggested that simplifying the approach would make it easier for the community to accept it. The ACR representative suggested that, like the proposed hospital measures, perhaps the image quality component should be dropped and only focus on the radiation dose. Others suggest that low radiation dose might be used to define an inadequate scan. Dr. Smith-Bindman agreed to look into that further, but suggested that she did not think it would work as well as the radiomics used in making the assessment.A question was asked if the images would still be needed if the image quality assessment were dropped. Dr. Smith-Bindman suggested they would still be needed to assess patient size. Dr. Burstin also raised whether changing from a balancing measure to a paired set of measures might be a better approach. Dr. Smith-Bindman said that this was the original idea, but that CMS had pushed for the balancing measure.

Following the discussion, UCSF indicated it would explore how well the automated approach for assessing image quality would work if the threshold for characterizing a scan as inadequate depended on more than 25% of radiologists rating a scan as mariginal or poor.

UCSF also indicated that it would continue to work to simplify the measure by exploring whether low radiation dose can be a surrogate for inadequate image quality.

1:25PM:         Wrap Up and Next Steps                                                                 Dr. Bindman

Dr. Bindman summarized the feedback from the TEP that members concur with notion of using a measure of image quality as a backstop. The goal is not to necessarily identify each scan with questionably inadequate image quality but to create a deterrent as a check against radiologists being too aggressive in lowering dose.  To the degree possible, UCSF should try to simplify the approach for measuring imagae quality and consider how well low radiation dose can be an effective surrogate for inadequate image quality by, for example, looking at how well low radiation dose predicts inadequate exams. UCSF will also explore a higher threshold than 25% of what is considered an inadequate image. At the next TEP meeting, UCSF will provide updates on these analyses as well as results from beta testing in settings independent of the UC Dose Registry. Two of the testing sites will be providing UCSF, not just with electronic data, but their entire CT scans. UCSF is building in a phase in which it will look at any of the scans that are judged to be out of range, and then independently go through those scans and make sure whether to see if those are true positives or false positives, and so forth. This will enhance what has been done with the test sample and the UC Dose Registry.

1:30PM:         Adjourn                                                                                                Dr. Burstin

Dr. Burstin closed the meeting by congratulating the UCSF team for its impressive work and thanked the TEP for its rich discussion of the results.