8:30

Meeting Called to Order by TEP Chair (Dr. Helen Burstin)

8:35

Roll Call

TEP member attendance listed above, other attendees below.

8:40

TEP Charter discussed and approved without amendment (unanimous vote) (appendix pp 19-20)

8:50

Each TEP member made a statement of introduction and any relevant financial conflicts.

Conflict of interest defined as you, your spouse, your registered domestic partner, and/or your dependent children:

1. received income or payment as an employee, consultant or in some other role for services or activities related to diagnostic imaging

2. currently own, or have held in the past 12 months, an equity interest in any health care related company which includes diagnostic imaging as a part of its business

3. hold a patent, copyright, license or other intellectual property interest related to diagnostic imaging

4. hold a management or leadership position (i.e., Board of Directors, Scientific Advisory Board, officer, partner, trustee, etc.) in an entity with an interest in diagnostic imaging

5. received and cash or non-cash gifts from organizations or entities with an interest in diagnostic imaging

6. received any loans from organizations or entities with an interest in diagnostic imaging

7. received any paid or reimbursed travel from organizations or entities with an interest in diagnostic imaging

COIs disclosed to UCSF prior to TEP meeting via paperwork. TEP members were also asked to verbally disclose any COIs when introducing themselves for the sake of group transparency.

No members had financial conflicts which precluded their participation.  

9:00

Presentation on radiation and risk (Dr. Amy Berrington de Gonzalez) (appendix pp 27-30)

The presentation focused on the evidence base for ionizing radiation as a risk factor for cancer.

10:00

Presentation on Medicare Quality Payment Program and cooperative agreements (Dr. Reena Duseja) (appendix pp 21-26)

The presentation described the status of Medicare’s Quality Payment Program and how this project as one of the funded cooperative agreements with CMS is expected to interface with that program. It was noted that CMS call for proposals for these cooperative agreements had identified a need at Medicare for radiology quality measures.

11:00

Presentation on variation in CT radiation doses (Dr. Rebecca Smith-Bindman) (appendix pp 30-34)

The presentation focused on the research evidence demonstrating the amount of radiation used in performing CT scans, the variability in the radiation doses used to perform these scans, and the success of interventions to lower doses

11:30

Project overview (Dr. Andrew Bindman) (appendix pp 34-37)

Members of the University of California project team where identified and introduced to TEP member panel.

Project concept:

Develop a composite quality measure which would be linked with financial incentives, as devised by CMS, intended to encourage the minimal use of radiation in the performance of CT scans while also ensuring that CT scans produce high quality images for diagnosis. This measure is focused on an adult population because the Medicare population is primarily comprised of adults. The intention is for this measure to be implemented as part of the Medicare quality payment program, either MIPS or APM.

Project Approach:

To create a standardized method associated with performance of CT scans, focused on the following anatomic areas: head, neck, chest, abdominal, pelvis, spine, extremities, and combinations thereof. To develop standardized method for assessing if a CT exceeds a minimum threshold of image quality, as well as a establishing a maximum risk-adjusted radiation dose standard, which considers patient characteristics, the scanned anatomical region, and scan indication. To provide software for capturing and recording calculated risk adjusted radiation dose in image quality assessments. The idea is to capture the information made available to CMS, but also provide feedback to radiologists so they can learn and respond to their doses.

Project Principles:

• To apply assessment to all eligible CT scans, not sampled
• To reduce clinician burden by creating a measure based on electronic data
• To apply measure to all specialties that perform CT scans
• To apply measure to all settings where CT scans are performed (hospital, emergency, ambulatory surgical centers, etc.)
• To allow clinicians to generate data for self-assessment and benchmarking related to their own performance

Data Source and Generation of Measure:

When fully operational, the University of California team expects that providers will be able to directly generate and report their data to CMS, or to work in collaboration with a registry such as one that is maintained by the American College of Radiology to submit the data to CMS on their behalf. The concept will first be explored using the UCSF dose registry which is a Patient Centered Outcomes Research Initiative (PCORI) funded repository of over 6 million CT scans from 151 institutions worldwide.  It will later be tested in sites that vary in size, type (academic vs community), clinical location (inpatient, outpatient, emergency department) and geographic location (urban vs rural).

TEP members raised a question about whether the study would focus only on the radiation dose of a particular scan of if it would create a cumulative assessment. It was noted that patients with greater morbidity would be likely to have more scans. The UC team acknowledged the importance of the issue but pointed out that the complexity of creating a patient level longitudinal record was beyond the capabilities of record keeping today.  The UC team reiterated that the focus of the proposed measure is on the radiation dose associated with a particular scan (cross-sectional rather than longitudinal).

Another TEP member raised a question about the level of analysis this measure is meant to capture: individual practitioner or practice level.  The UC team responded that it will develop and test how the measure performs at each level. Within the UC dose registry project, UCSF has previously found that providers are most motivated by feedback relating to how they compare to other radiologists in their practice.

A TEP member pointed out that sites which vary by geography (e.g. urban vs rural) or by the degree to which they care for low income patients may vary in their ability to optimize radiation doses. Careful consideration must also be made to not unduly penalize rural institutions, so as not to cause the one provider within several hundred miles to shut down. The TEP member recommended and the UC team agreed that when testing the measure that sites be included which vary by these characteristics so as to be able to assess how radiation doses vary across them.

11:45

Radiation dose simulation program: demonstrates how radiation dose can be manipulated in the conducting of CT exam. (appendix p 38)

Dr. Smith-Bindman’s research team at UCSF has previously created a simulation program to help users to understand the ways that CT operators can alter the radiation dose and how this may influence the quality of the image. TEP members were introduced and encouraged to try using the simulation to improve their own understanding of the variables that affect radiation dose and image quality. The simulation program can be accessed at https://ctqualitymeasure.ucsf.edu/external-links.

12:30

Working Lunch 

Presentation on options to quantify radiation dose (Dr. Rebecca Smith-Bindman) (appendix pp 37-40)

Two potential primary measures of radiation dose were presented:  Dose length product (DLP) versus effective dose as the primary measure of dose. Definitions of measures and explanation of differences between them provided in linked presentation. TEP members were asked their opinion about which one should be used for the proposed measure.

The TEP discussion raised questions about whether the focus should be on a measure which is most useful to the provider or to the patient. It was noted by TEP members that DLP is universally used by professionals in the field, but effective dose may be more intuitive for patients.

TEP members were generally inclined to support DLP as the measure of radiation dose if a valid strategy can be created to combine assessments across anatomic areas and there is a way to create a way to make the results understandable for patients.  It was acknowledged by TEP members and the UC research team that different doses are appropriate for different anatomical regions and for certain indications because of differences in the nature of the tissue which radiation must penetrate to generate an image.  It was noted that effective dose is easier to combine because it is already weighted by anatomical area but the downside is that the weighting is subject to change over time. 

Action: At a subsequent TEP meeting, the UC team will provide additional data and suggestions on how to quantify radiation dose using DLP in a way that can be combined across anatomical areas to support the establishment of a single measure regardless of the distribution of the type of CT scans a provider performs.

1:15

Presentation on options to risk adjust radiation dose (Dr. Patrick Romano) (appendix pp 40-44)

Dr. Romano presented on the indications for risk adjustment how it is done. Descriptions are provided in the linked presentation. As a part of the presentation several potential demographic and clinical variables were raised as potential risk factors that could be used to adjust assessments of radiation dose. These were discussed with TEP members who were asked to indicate their support for the outlined approach and to raise questions about other potential risk factors. For example, Dr. Romano noted that the size of the patient should be taken into consideration as the radiation dose is appropriately increased with larger patients. TEP members endorsed this concept.

There was a discussion about how clinical indication (cancer screening, cancer surveillance) could introduce appropriate differences in dose which may need to be accounted for in assessing radiation dose. This was widely acknowledged with the recognized understanding that there are limitations in the valid extraction of indication from available electronic data. The UC team offered to look into whether CT protocols could be used to determine clinical indications in some cases. However, Dr. Romano and the UC team pointed out that process measures can reflect discretionary choices related to quality and that in general relying on protocols to risk adjust could be problematic. For example, the choice of using single phase versus multi-phase studies can have a large influence on the radiation dose used in a study but it may not always be an appropriate decision based on the indication for the scan.  Adjusting for protocol could therefore in some cases result in an “over-adjustment” that prevents observing true differences in quality and safety.

TEP members raised a question about whether variability in dosing associated with CT manufacturer should be used in risk adjustment. The UC team responded that based on prior research work which can be shared with the TEP, the DLP variability is primarily driven by patient diameter and not the manufacturer of the CT. For this reason, the UC team expects to risk-adjust measurements of radiation dose by patient size but not CT manufacturer.

Discussion questioning whether there is any basis to adjust the assessment of radiation doses by age or sex. The UC team noted that there are observable differences in the radiation doses between men and women of the same size but unless there are clear indications specifying why this is appropriate it may not be appropriate to adjust for these differences. The UC team intends to further explore this issue and take input from the TEP about whether age and sex adjustment is warranted.  

Discussion about whether in addition to reporting an overall risk-adjusted radiation dose there should also be a report of the number and percentage of CT imaging studies that exceed a threshold which constitute “never doses” with or without risk-adjustment. The discussion seemed to favor both a “standard approach” which would be submitted to NQF and a “never dose approach” which might be treated as a serious reportable event.

2:45

Presentation on measure of image quality (Dr. Smith-Bindman) (appendix pp 44-46)

The presentation available in the linked slides focused on the role of image quality as a balancing measure for a measure designed to encourage lower use of radiation in the performance of CT scans. The point was made that in an attempt to lower doses there is a risk that this could undermine image quality needed to make accurate diagnoses.

The UC team described its ideas for creating a study to test the validity of two potential ways of measuring scan quality in comparison to a gold standard assessment by radiologists. In the first case, the calculated “noise” factor produced by the CT machine will be used as a potential measure of image quality. In the second case, artificial intelligence will be deployed to judge image quality. In each case these potential approaches which could be automated and scaled for operational purposes will be compared with the assessments on a test sample of CT scans read by radiologists who are blinded to the dose used to create the image. The UC team suggested that radiologists would make their judgments using a graded scale of image quality as a way to create more robust data. However, the TEP members thought that this would be overly complex for the radiologists and that it would not reflect the actual clinical experience in which radiologists simply judge whether the image quality is adequate or not.

Decision: The UC team accepted this feedback and indicated that it would plan to redesign its approach to use radiologists’ assessments as the gold standard by having them simply judge the adequacy of the images. It was suggested that this might require the test sample to include more images but that this would be discussed at a subsequent meeting of the TEP.

Another TEP member raised the point that if it could be demonstrated in the validation study that radiologists did not systematically rate lower dose scans as inadequate quality that it may be unnecessary to move ahead with the balancing measure but to simply focus on the radiation dose component of the measure. The TEP members agreed to return to this question at a later point in which there would be empirical data to better assess this from the validation study.

3:10

Dr. Andrew Bindman, Future Plans (appendix p 46)

At the next TEP meeting, anticipated for June 2019, TEP members will assist the UC project team in:

1. Refining the approach for controlling for the appropriate use of different radiation doses based on the anatomical areas being scanned and that physicians vary in the distribution of CT scans they perform by anatomical area
2. Refining the component of the project in which radiologists’ assessments of image quality will be used as a gold-standard against which to judge the validity of automated assessments derived either from the CT output (“image noise”) or based on computer generated readings using artificial intelligence (AI).

3:15

Meeting Adjourned