In Review...

John A. Swets, Tulips to Thresholds

From Tulips To Thresholds

 


backToTheFuturePostIt.png

Click here to revisit our

our RSNA 2008 Education Exhibit

"Common Deficiencies in ROC Studies"

certOfMerit.png

Log in


Forgot your password?
New user?
 
You are here: Home

Introduction

Metz ROC Software at The University of Chicago

Development of methodology for evaluation of diagnostic performance has been an essential part of our department’s research for many decades. The primary focus of this work has been on Receiver Operating Characteristic (ROC) analysis, which describes the accuracy of a diagnostic modality, for a particular clinical task, in terms of its trade-off between sensitivity and specificity [2-4].  Three early insights of our group [5-9] were that novel ROC methodology was needed to meet the needs of medical image evaluation; that newly proposed statistical methods are of little value without careful validation of their behavior using both simulated and real datasets; and that development and free distribution of convenient, reliable software for ROC analysis could substantially encourage acceptance of the methodology. Currently, software developed in accord with this theme (ROCFIT, LABROC4, PROPROC, CORROC, INDROC, ROCKIT and LABMRMC) or in collaboration with others (DBM-MRMC, with the University of Iowa) is used by thousands of registered users. We developed and released a Java-based nearly platform-independent software package for ROC analysis that includes many alternative ROC analysis approaches at the beginning of December of 2010 and can be downloaded from the software section of this web site.


Our recent progress has included the addition of many non-parametric methods for analysis of ROC data [10] and a review of the medical-imaging literature to determine the spectrum of ROC methods used in published observer performance studies [11]. Current research efforts that eventually will be incorporated into our released software include development of “proper” ROC models for analysis of both partially- and fully-paired datasets (to be released soon), investigation of Bayesian approaches to ROC curve fitting, and development of quantitative methods that relate different test-result scales to each other and to individual operating points on ROC curves. Generalization of conventional ROC analysis to evaluate diagnostic tasks that involve more than two decision alternatives is under investigation [12,13], and we will release software capable of analyzing such data as soon as possible.


References

1. Goodenough DJ, Rossmann K, Lusted LB. Radiographic applications of signal detection theory. Radiology 105: 199-200, 1972.
2. Metz CE. Basic principles of ROC analysis. Seminars in Nuclear Medicine 8: 283-298, 1978.
3. Metz CE. ROC methodology in radiologic imaging. Investigative Radiology 21: 720-733, 1986.
4. Wagner, RF, Metz, CE, Campbell, G. Assessment of medial imaging systems and computer aids: a tutorial review. Academic Radiology 14: 7238, 2007.
5. Metz CE, Kronman HB. Statistical significance tests for binormal ROC curves. Journal of Mathematical Psychology 22: 218-243, 1980.
6. Metz CE, Herman BA, Shen J-H. Maximum-likelihood estimation of ROC curves from continuously-distributed data. Statistics in Medicine 17: 1033-1053, 1998.
7. Metz CE, Herman BA, Roe CA. Statistical comparison of two ROC-curve estimates obtained from partially-paired datasets. Medical Decision Making 18: 110-121, 1998.
8. Metz CE, Pan X. Proper binormal ROC curves: theory and maximum-likelihood estimation. Journal of Mathematical Psychology 43: 1-33, 1999.
9. Pesce LL, Metz CE. Reliable and computationally efficient maximum-likelihood estimation of “proper” binormal ROC curves. Academic Radiology 14: 814-829, 2007.
10. Gallas B, Pesce LL. Comparison of ROC methods for partially-paired data. Proc. SPIE, Vol. 7263, 72630V (2009)
11. Shiraishi J, Pesce LL, Metz CE, Doi K. On Experimental Design and Data Analysis in Receiver Operating Characteristic (ROC) Studies: Lessons Learned from Papers Published in RADIOLOGY from 1997 to 2006. Radiology; 253: 822-830, 2009
12. Edwards DC, Metz CE, Kupinski MA. Ideal observers and optimal ROC hypersurfaces in N-class classification. IEEE Transactions on Medical Imaging 23: 891-895, 2004.
13. Edwards DC, Metz CE. Analysis of proposed three-class classification decision rules in terms of the ideal observer decision rule. Journal of Mathematical Psychology 50: 478-487, 2006