How to compare different models

Hello,

I would like to compare different machine learning models ( SVM, random forests, Neural networks) for a multiclass database. I cannot use the ROC curve for this problem as I have more than 2 classes. I have from the scorer the confusion matrix and the Accuracy and the Cohen Kappa. There is also the F-measure, sensitivity, recall.. for every class.

Can I say that the best model is the model with the highest accuracy or I should take in to account the other parameters. Is it appropriate to take the average of the F-measure, recall ..over the different classes?

How can I measure the out-of-bag error for my model and how can I apply it for my models comparison?