Friday, January 05, 2007

Model Performance Measurement

A wide variety of model performance measures have been devised. Despite the popularity of mean squared error for numeric models and simple accuracy for classification models, there are many other choices. For my part, I generally prefer mean absolute error for numeric models and the AUC (for class separation) and informational loss (for probability assessment) for classification models.

This log entry is pretty much just a quick giveaway: I have constructed a generic performance calculation MATLAB routine, SampleError.m. Its operation is straightforward, but I find it handy to contain all of these measures in one routine, with the ability to switch among them as a simple parameter change. The use of this routine is simple and is explained by help SampleError and it makes a great building block for modeling routines.

I update many of my MATLAB routines from time to time, and this one is no exception. Presently, though, the following performance measures are supported:

'L-1' (mean absolute error)
'L-2' (mean squared error)
'RMS' (root mean squared error)
'AUC' (requires tiedrank() from Statistics Toolbox)
'Conditional Entropy'
'Informational Loss'
'Median Squared Error'
'Worst 10%'
'Worst 20%'

Note: I still need to verify the Cross-Entropy measure. The last two are classification performance measures, being the proportion of the target class found in the predicted most likely 10% and 20%, respectively.

Incidentally, I'd appreciate any feedback on any of the code in this Web log, whether it be about typos, outright coding errors of efficiency issues. Also, please send suggestions for additional measures.


Anonymous said...

excellent work!

Unknown said...

hi sir
i want to use AUC but i do not know how it work. i have 20 population and for each one based on ground truth i got sensitivity, specificity and accuracy. now i want to plot ROC curve for them.yes i can make use from TPR against FPR but how? TPR is a continuous number >= 0.7 and FPR also a continuous number but in range of >=0.01 and <= 0.04. iread in literatures they change the threshold to get ROC but i did not understand if they change the threshold and get another result how to combine it and plot ROC ?!
please support me with that.
thank you so much.