Introduction
Metz ROC Software at The University of Chicago
Our recent progress has included the addition of many non-parametric methods for analysis of ROC data [10] and a review of the medical-imaging literature to determine the spectrum of ROC methods used in published observer performance studies [11]. Current research efforts that eventually will be incorporated into our released software include development of “proper” ROC models for analysis of both partially- and fully-paired datasets (to be released soon), investigation of Bayesian approaches to ROC curve fitting, and development of quantitative methods that relate different test-result scales to each other and to individual operating points on ROC curves. Generalization of conventional ROC analysis to evaluate diagnostic tasks that involve more than two decision alternatives is under investigation [12,13], and we will release software capable of analyzing such data as soon as possible.
References
1. Goodenough DJ, Rossmann K, Lusted LB. Radiographic applications of signal detection theory. Radiology 105: 199-200, 1972.
2. Metz CE. Basic principles of ROC analysis. Seminars in Nuclear Medicine 8: 283-298, 1978.
3. Metz CE. ROC methodology in radiologic imaging. Investigative Radiology 21: 720-733, 1986.
4. Wagner, RF, Metz, CE, Campbell, G. Assessment of medial imaging systems and computer aids: a tutorial review. Academic Radiology 14: 7238, 2007.
5. Metz CE, Kronman HB. Statistical significance tests for binormal ROC curves. Journal of Mathematical Psychology 22: 218-243, 1980.
6. Metz CE, Herman BA, Shen J-H. Maximum-likelihood estimation of ROC curves from continuously-distributed data. Statistics in Medicine 17: 1033-1053, 1998.
7. Metz CE, Herman BA, Roe CA. Statistical comparison of two ROC-curve estimates obtained from partially-paired datasets. Medical Decision Making 18: 110-121, 1998.
8. Metz CE, Pan X. Proper binormal ROC curves: theory and maximum-likelihood estimation. Journal of Mathematical Psychology 43: 1-33, 1999.
9. Pesce LL, Metz CE. Reliable and computationally efficient maximum-likelihood estimation of “proper” binormal ROC curves. Academic Radiology 14: 814-829, 2007.
10. Gallas B, Pesce LL. Comparison of ROC methods for partially-paired data. Proc. SPIE, Vol. 7263, 72630V (2009)
11. Shiraishi J, Pesce LL, Metz CE, Doi K. On Experimental Design and Data Analysis in Receiver Operating Characteristic (ROC) Studies: Lessons Learned from Papers Published in RADIOLOGY from 1997 to 2006. Radiology; 253: 822-830, 2009
12. Edwards DC, Metz CE, Kupinski MA. Ideal observers and optimal ROC hypersurfaces in N-class classification. IEEE Transactions on Medical Imaging 23: 891-895, 2004.
13. Edwards DC, Metz CE. Analysis of proposed three-class classification decision rules in terms of the ideal observer decision rule. Journal of Mathematical Psychology 50: 478-487, 2006