PLOT_PROB(1)

NAME
plot_prob - Plot predicted versus actual posterior class probabilities for a LNKnet classifier.

SYNOPSIS
Show how closely continuous classifier outputs approximate the posterior probabilities for patterns in a given class.

plot_prob [-bin_plot] [-target n] [-nbins n] [-var_bin]
[-bin_ends x[,x]...] [-min_bin_count n] [-chi_square]
[-pathexp directory] [-ferror file] [-fplot file]
-noutputs n [-npatterns n] [-xmin x] [-xmax x]
[-xstep x] [-ymin x] [-ymax x] [-ystep x]
[-title string] [-verbose n] [-sum_to_one] [-mac]
[-fmaclines file] [-no_graphics]

DESCRIPTION
It has been shown that the outputs of some classifiers approximate Bayesian posterior probabilities. plot_prob shows how close that approximation is. To generate the plot, patterns are sorted into bins by the value of the target class output. A Chi-square fit is used to compare the average value in each bin with the fraction of the binned patterns that are actually in the target class. A table is generated which shows the predicted and observed posterior probability values.

When the plot is displayed, the observed posterior probabilities are represented by filled circles with error bars stretching up and down. The error bars show +/- two standard deviations of the posterior probabilities. The actual number of patterns in the bin and in the target class are shown above each error bar. There are two versions of the plot. In one version, posterior probability versus average target output is plotted and a line along the diagonal is used to indicate where perfect posterior probability outputs show lie. In the second version of the plot, the bins are evenly spaced along the horizontal axis. The average target output for each bin is shown as an X.

The plot is generated using data from the error file. The error file verbosity flag verror, must be greater than or equal to 2 during testing to generate the plot. Sufficient storage capacity is required to store the error file until plot_prob is run.

EXAMPLE
This example tests a Multi-Layer Perceptron classifier using test data and creates a posterior probability plot.

mlp -train -create -pathdata $LNKHOME/data/class \
-finput vowel.train -fdescribe vowel.defaults \
-npatterns 338 -fparam mlp.param -nodes 2,25,10 -epochs 100

mlp -pathdata $LNKHOME/data/class -finput vowel.test \
-fdescribe vowel.defaults -npatterns 167 \
-ferror mlp.err.test -verror 2 -fparam mlp.param

plot_prob -pathexp ./ -ferror mlp.err.test \
-fplot prob.plot -noutputs 10

OPTIONS

--  Prints a list of all flags and arguments.

-bin_plot
    Evenly space the observed posterior probability values along the horizontal axis of the plot. Show average output value of patterns in the bin as an X. The default is to use the average bin output as the value along the horizontal axis for observed posterior probability values.

-target n
    The number of the target class. n goes from 0 to the number of classes minus one.

-nbins n
    If the -var_bins flag is not set, the number of evenly spaced bins running from an output value of 0 to 1. Values which are less than 0 are put in the first bin. Values which are greater than 1 are put in the last bin.

-var_bin
    Use the setting of the -bin_ends flag to determine the range of each plot bin.

-bin_ends x[,x]...
    A sorted list of numbers from 0 to 100 giving the ends of the plot bins. For example -bin_ends 0,5,10,20,50,100 will produce 5 bins. For a target output of x,
    if x < 0.05 the pattern is assigned to bin 0,
    if 0.05 <= x < 0.1, the pattern is assigned to bin 1,
    if 0.1 <= x < .2, the pattern is assigned to bin 2,
    if 0.2 <= x < .5, the pattern is assigned to bin 3,
    if 0.5 <= x, the pattern is assigned to bin 4.

-min_bin_count n
    For the chi squared fit to be valid, there must be a minimum of 5 or more patterns in each bin. Bins which do not have enough patterns are merged with neighboring bins until all remaining bins have the minimum number of patterns.

-chi_square
    Display a small table showing the results of the chi squared fit.

-pathexp directory
    Directory where ferror and fplot are stored.

-ferror file
    File where the trial-by-trail classification test results are stored. The error file verbosity flag verror, must be greater than or equal to 2 during testing to generate this error file.

-fplot file
    File for storing plot.

-noutputs n
    There are outputs for n classes stored in the error file.

-npatterns n
    Read results for the first n patterns from the error file. If -npatterns is set to 0, read all the patterns in the file.

-xmin x
    When -bin_plot is not set, the lower limit of the average target output axis.

-xmax x
    When -bin_plot is not set, the upper limit of the average target output axis.

-xstep x
    When -bin_plot is not set, the space between tic marks on the average target output axis.

-ymin x
    The lower limit of the observed posterior probability axis.

-ymax x
    The upper limit of the observed posterior probability axis.

-ystep x
    The space between tic marks on the observed posterior probability axis.

-title string
    Title for plot.

-verbose n
    Controls amount of printout.

    0 - Nothing printed to standard output

    1 - Print a table of the predicted and observed posterior probabilities for each bin.

-sum_to_one
    Normalize the outputs for each pattern to sum to one before assigning them to bins. This is especially necessary when the outputs are generated by a likelihood classifier. The likelihood classifiers are gauss, gmix, histogram, and parzen.

-mac
    If this flag is set, the points in the plot will be written to the file -fmaclines.

-fmaclines file
    This is an ASCII file containing the data used to create the posterior probability plot. Each line contains plot values for one plot bin. The first field is the average classifier output for patterns in the bin. The next field is the observed posterior probability. The two fields are the plus and minus two standard deviation limits for the posterior probability. The last two fields are the number of patterns in the bin and the number of bin patterns in the target class. This file can be used to generate plots on a MacIntosh or other PC using a spread-sheet graphing program such as Excel or DeltaGraph.

-no_graphics
    Generate the plot file but do not display it in an olxplot window.

SEE ALSO
classifier(1), irbf_lnk(1), rbf_lnk(1), mlp_lnk(1), gauss_lnk(1), gmix_lnk(1), histogram_lnk(1), parzen_lnk(1)

HISTORY
First written 19 April 1995.

BUGS
Send bug reports to Linda Kukolich (kukolich@sst.ll.mit.edu)