Measure Testing

The measure was tested in diverse settings and populations, including:  

  • Nearly 50,000 CT exams 
  • 16 hospitals (including inpatient, outpatient, and ED settings) across Alabama, California, Michigan, and New York; 4 of these hospitals were designated as “safety net hospitals”  
  • 1 large ambulatory imaging practice with 17 locations in Texas 
  • 16 clinician group practices 
  • 606 individual clinicians.

Measure results varied slightly by the level of analysis (hospital, clinician group, individual clinician), but on average:  

  • Average measure score (reflecting the % of exams that were “out of range,” i.e. failed, the measure) = 30%  
  • Standard deviation = 18% 
  • Range = 2% - 100% out of range 

Virtually all exams failed the measure due to excessive radiation doses, not for inadequate image quality. Out of range scores based on image quality (noise) alone were <1% at every single testing site.  

Reliability testing – which assesses whether a measure will consistently produce the same result when tested under the same conditions – achieved an intraclass correlation coefficient (ICC) of 0.99, reflecting excellent reliability. 

Validity testing – which assesses whether a measure is measuring what it intends to measure and thus is an accurate reflection of quality – was tested by a poll of our diverse Technical Expert Panel, which unanimously voted that: 

  • Radiation dose and image noise, as specified by this measure, are relevant metrics of CT quality; and  
  • Performance on this measure is an accurate representation of quality.

Testing of the eCQM’s calculations against manual review and scoring of data showed 100% agreement. 

Feasibility testing showed all data elements were consistently: 

  • Available in structured fields in electronic systems
  • Accurate as they are captured by providers or billing specialists through routine care
  • Standardized using nationally accepted vocabularies

Burden testing: We assessed burden via interviews with representatives from all testing sites, including site PIs, PACS administrators, and IT and radiology-IT staff. In these interviews, we explored the burden to physicians and staff in terms of hours, cost, complexity, and changes in workflow. On average, sites spent around 50 hours to set up the data extraction software, link required systems, and extract testing data. All testing sites reported that if the testing were repeated, the hours required would be lower in subsequent rounds. Testing was completed at an average cost of $2,600-3,250 per testing site. This level of implementation effort is similar to the burden for other eCQMs, and generally less than the effort involved in participating in national registries.