Observational studies and experiments in medicine, pharmacology and agronomy often look at whether different methods/advisors produce similar values over time when measuring a quantitative variable. This article aims to describe the lcc statistical package, which can be used to estimate the extent of the agreement between two (or more) methods over time, and to illustrate the methodology developed using three real examples. The Kappa Agreement Index (KIA) will tell you how much better or worse your class is than you would expect by chance. If you assigned random requests to classes (i.e. some kind of terribly informed binder), you would get something just by chance. Therefore, you will always find that the kappa value is less than the overall accuracy. However, the Kappa index is considered a conservative measure such as the overall accuracy of the classification. Your KIA value basically tells you that your class is about 66% better than a random case ranking in different classes. That`s not bad! The output of the summary function () includes the values of the Akaike Information Criterion (AIC) (Akaike, 1974), bayes information criterion (BIC) (Black, 1978), log-likelihood value, and a gof adjustment measurement quality calculated from the correlation coefficient (Lin, 1989) between adjusted values extracted from the mixed effects model and observed values.
This measure can be used with caution to describe the general agreement between the observed values and the adjusted values, with a value of 1 which is a perfect match between them, zero without conformity and perfect match. It is obvious that a high model performance is related to a high positive gof value (usually between 0.8 and 1). Hi Jason, in the case of cross-validation k-folds, which should be used as the accuracy of a model, which is shown in the variable «model» or in confusionMatrix? I guess it`s the first one, but I wanted to confirm. Thank you. I have a small set of models (120 or something like that, with 20 «positive» cases or something like that). I use logistic regression and cross-validation (cv – 10). Do I need a holdhout set that I can test to really determine accuracy? It would be worth mentioning that the Bland-Altman method (Bland-Altman, 1986) is frequently used as a naïve analysis to calculate the average difference between two methods (as a «bias» measure), with 95% of compliance (loA) in the analysis of repeated measurements (including longitudinal data).