MLP_LNK(1)

NAME
mlp_lnk - Multi-Layer Perceptron Classifier

SYNOPSIS
A program for training and testing a multi-layer perceptron with up to 8 hidden layers.

mlp_lnk -nodes n,n[,n]... [ -alpha x ] [ -etta x |
-use_etta_list -etta_list x[,x]...]
[ -etta_change_type n ] [ -etta_nepochs x ]
[ -epsilon x ] [ -kappa x ] [ -decay x ]
[ -tolerance x ] [ -hfunction n ] [ -ofunction n ]
[ -sigmoid_param x |
-use_sig_param_list -sig_param_list ] [ -cost_func n ]
[ -cost_param x ] [ -epochs n ] [ -batch n,n,n ]
[ -init_mag x ] [ -classifier_options]

EXAMPLE
mlp_lnk -epochs 100 -train -create -nodes 2,25,10 -etta .1 \
-pathdata $LNKHOME/data/class -finput vowel.train \
-fdescribe vowel.defaults -npatterns 338 -random \
-fparam mlp.param

mlp_lnk -pathdata $LNKHOME -finput vowel.train \
-fdescribe vowel.defaults -fparam mlp.param -npatterns 166

DESCRIPTION
In testing, a multi-layer perceptron computes several weighted sums of features in the input pattern. These sums are passed through sigmoid functions and the results are used as the input to the next layer in the network. Input values are fed forward this way until the output layer of the network is reached. The class corresponding to the highest output node value is selected as the class of the pattern.

In training, the weights connecting each pair of adjacent layers are adjusted to minimize the error in the outputs of each input pattern. A gradient descent method is used for weight training, with the changes needed propagating backwards through the network.

OPTIONS
-- Prints a list of all flags and arguments.

-nodes n,n[,n]...
    The number of nodes in each layer of the perceptron. The first number in the list is the number of input features for each pattern. The last is the number of classes. Any numbers in the middle are the number of nodes in each hidden layer. There is a maximum of 10 layers, or 8 hidden layers.

-alpha x
    x is the momentum of the weight training. 0 <= alpha < 1.

-etta x
    x is the step size when training weights.
    weight_inc(t+1) = etta*(gradient for this weight) + alpha*weight_inc(t)

-use_etta_list
    When this flag is set, initialize the step sizes for each layer to the values in etta_list.

-etta_list x[,x]...
    If -use_etta_list is set, initialize the step size in each layer to the corresponding value in the list. Because there is one fewer layer of weights than network layers, -etta_list should have one fewer entries than -nodes has.

-etta_change_type n
    n determines whether to change the weight step sizes during training. This flag only has effect when both -train and -create are set.

    0   Do not change step sizes during training. To change step sizes, restart training without the -create flag and change the -etta or -etta_list flag.

    1   Hold the step sizes approximately constant for -etta_nepochs epochs of training (etta_nepochs is a float), then reduce the step size. This should not be done when performing weight updates in batches.

    2   Use the Multiple Adaptive Step size algorithm for adjusting the step size for each weight. This flag only has effect when doing weight updates in batches.

-etta_nepochs x
    If the etta_change_type flag is set to 1, when training, hold the step size approximately constant for x epochs before reducing it. Note that the number of epochs is counted from when the model was first created with the -train and -create flags both set.

-epsilon x
    When the direction of change for a weight switches from one batch to the next, decrease the step size for that weight.
    etta(t+1) = (1 - epsilon)*etta(t)
    0 <= epsilon <= 1
    This is true when doing batch training of the weights and the etta_change_type flag is set to 2.

-kappa x
    When the direction of change for a weight is the same from one batch to the next, increase the step size for that weight.
    etta(t+1) = etta(t) + kappa
    This is true when doing batch training of the weights and the etta_change_type flag is set to 2.

-decay x
    Multiply each weight by (1 - x) whenever any weight is updated.
    0 <= decay < 1

-tolerance x
    If the value of an output node is less than x do not change the weights of the connections to that node. If the node corresponds to the correct class, do not change its weights if the value is greater than 1-x.

-hfunction n
    n determines the function to be used on the outputs of all the hidden layer nodes.

    0 sigmoid
        output = 1/(1 + exp(- param * input))
        0 < output < 1

    1 symmetric sigmoid (-1,1)
        output = 2/(1 + exp(- param * input)) - 1
        -1 < output < 1

    4 linear
        output = input

-ofunction n
    n determines the function to be used as the sigmoid in the output layer. When using the squared error error function type, the allowable values for ofunction are the same as those for hfunction. When using the Cross Entropy cost function, ofunction should be 0. When using the Maximum Likelihood cost function, the output function is
    [exp( param * input)]/[sum_over_outputs(exp( param * input))].

-sigmoid_param x
    x is a scaling factor for the sigmoid functions.

-use_sig_param_list
    Set the sigmoid function steepness factors in each network layer to the values in -sig_param_list.

-sig_param_list
    If -use_sig_param_list is set, set the sigmoid function scaling factors for each network layer according the this list. Because there is no sigmoide function in the input layer, sig_param_list should have one fewer entries than the -nodes network structure list.

-cost_func n
    n determines the output cost function.

    0 Squared Error

    1 Cross Entropy

    2 Maximum Likelihood

    3 Perceptron Convergence Procedure
        (Single Layer Perceptron)

    4 Top Two Difference

-cost_param x
    Initialization parameter for the Top Two Difference error criterion. This parameter is the steepness of the sigmoid the difference between the correct and next highest outputs are passed through. (0 <= x <= 1).

-epochs n
    n is the number of times to run through all of the data in the input file during training. This flag has no effect during testing.

-batch interval,max,inc
    This flag permits batch training with variable numbers of trials per batch. The argument to batch must be a list of three integers. The first number, interval, is the number of patterns shown in the first batch. The second number, max, is the maximum number of patterns in a batch. The third number, inc is the number of patterns to add to each batch after each epoch.
    Some Examples:
    batch 1,1,0
    Update the weights after each pattern is presented.

    batch 5,5,0
    Update the weights after each fifth pattern is presented.

    batch 2,20,2
    Update the weights after every second pattern during the first epoch. During the second epoch, update the weights after every fourth pattern. During the third epoch, update the weights after every sixth pattern. During the tenth epoch and any later epochs, update the weights after every twentieth pattern.

-init_mag x
    x is the maximum allowed magnitude of the initial random connection weights.

-classifier_options
    mlp_lnk accepts the general classifier arguments described on the classifier(1) manual page. Certain arguments are not used by the program, and so are unavailable.

    -ninputs, -noutputs
        These values are taken from the argument -nodes.

REFERENCES
Introduction: Richard P Lippmann, An Introduction to Computing with Neural Nets, IEEE ASSP Magazine, April 1987. Pages 15-18.

Cost Functions: Michael D Richard and Richard P Lippmann, Neural Network Classifiers Estimate Bayesian a posteriori Probabilities, Neural Computation 3,461-483, 1991, MIT. Pages 463-469

Weight Update Equations: Yuchun Lee, Classifiers: Adaptive Modules in Pattern Recognition Systems, Bachelor's and Master's Thesis, Massachusetts Institute of Technology, Department of Electrical Engineering and Computer Science, Cambridge, MA, May 1989. Pages 45-48.

SEE ALSO
classifier(1), mlp2c(1), mlp_feat_sel(1), bintree2mlp(1), plot_mlp(1), mlp_plot_bound(1)

HISTORY
First written 26 May 1992 by Linda Kukolich
Modified 22 June 1994, added sig_param_list and etta_list, added etta_change_type

BUGS
Send bug reports to Linda Kukolich (kukolich@sst.ll.mit.edu)