Describing Three-class Task Performance: Three-class Volume under ROC Surface (VUS) and Three-class Hotelling Trace (3-HT) as Figures-of-Merit

X. He and E.C. Frey

 

Abstract - In this work, we compared two figures-of-merit (FOMs) as performance measures for classification tasks that involve three diagnostic alternatives, the three-class volume under ROC surface (VUS) and 3-class Hotelling trace (3-HT). A simple three-class signal classification task was used. Each class contains two images; each image has one possible signal to represent one property of the class. Thus, by changing the magnitude and the signal to noise background ratio on both images independently, we controlled the ease of the classification task and calculated the corresponding VUS and 3-HT. When only one property of the class was adjusted, VUS and 3-HT provide monotonic relationship. However, the 3-HT is unbounded, while the VUS is bounded as predicted. When both of the properties of the class were adjusted, VUS and 3-HT failed to provide monotonic relationship. This is due to the fact that 3-HT cannot distinguish the cases where it can perfectly make all classification from the case where it can perfectly classify only one of the classes. This study shows that VUS results are reasonable, while the 3-HT fails to predict observer performance in some cases, suggesting there are limitations in using 3-HT as a FOM in evaluating three-class classification tasks.