MCNEMAR_LNK(1)

NAME
mcnemar_lnk - compute mcnemar's test to compare error files for two classifiers

SYNOPSIS
mcnemar_lnk -fa fa.err -fb fb.err

OPTIONS
LNKnet error file for classifier A. LNKnet error file for classifier B. This classifier must be tested using the same test data as classifier A. Prints man page describing this function

DESCRIPTION
mcnemar_lnk compares two LNKnet error files produced by two classifiers (labeled A and B) to determine if the error rates of these classifiers are statistically different. The classifiers must be tested using the same test data for this test to be applied.

Mcnemar's test is more sensitive than simply comparing binomial standard deviations because it focuses on patterns where one classifier was correct and the other was wrong. A chi-square value is computed that depends on a confusion matrix for the two classifiers. The rows of the 2x2 matrix correspond to classifier A and the columns correspond to classifier B. The diagonal entries count the number of patterns where both classifiers were correct and where both were wrong. The off-diagonal entries count the number of patterns where the classifier decisions differ. For these patterns one classifier was wrong and the other was right. A chi-square value is computed based on the difference between the off-diagonal counts in this matrix to determine if the error rates of the classifiers differ. A high-value for this normalized squared difference (above 3.84) means the classifiers are significantly different at the 0.05 level.

To use this program, first train two classifiers using LNKnet and test them on the same test data files. If you used the LNKnet "test" files, the error files will be named *.err.test". If you used the LNKnet "eval" files the error files will be name "*.err.eval". For example if you trained an mlp classifier with the default experiment number and tested it with the eval data you would create an error file named X1mlp.err.eval. If you then trained a bintree classifier as above you would create an error file named X1bintree.err.eval. To compare these files type

     mcnemars X1mlp.err.eval X1bintree.err.eval
This script can be used for error files created using cross-validation training/testing. Make sure the button in the LNKnet main window cross validation box labeled "Randomize Patterns Before Assigning to Folds" IS NOT CHECKED.

EXAMPLE RUN AND OUTPUT
  mcnemar_lnk -fa X1svm.err.eval -fb X1mlp.err.eval
  Classifier A X1svm.err.eval  18.67% error
  Classifier B X1mlp.err.eval  18.07% error
            Bwrong  Bright
  Awrong     27      4
  Aright      3    132
  total patterns 166
  off-diagonal difference 1
  chi-square value  0.571
  Sorry, classifiers are not different at the .05 level

LIMITATIONS
Make sure the test file used with both classifiers is the same. The program will terminate if test files differ. This script can be used for error files created using cross-validation training/testing. Make sure the button in the LNKnet main window cross validation box labeled "Randomize Patterns Before Assigning to Folds" IS NOT CHECKED.

The first line of the mcnemar_lnk file is the path to perl on your system. It sometimes needs to be changed by hand to point to where perl is located on different systems.

AUTHOR
Richard Lippmann - 1 Sept 2003 - MIT Lincoln Laboratory