SVM_LNK(1)

NAME
svm_lnk - Support Vector Machine Classifier

SYNOPSIS
Multi-class linear and nonlinear support vector machine classifier.

svm_lnk [ -cbound x ] [ -mclass_type n ]
[ -kernel_type n ] [ -kernel_sigma x ]
[ -kernel_order n ] [ -kernel_poly_divisor x ]
[ -sigmoid_fit ] [ -kkt_tolerance x ]
[ -alpha_tolerance x ] [ -classifier_options]

EXAMPLE
svm_lnk -train -create \
-mclass_type 1 -cbound 10 -kernel_type 0 \
-pathdata $LNKHOME/data/class -finput vowel.train \
-fdescribe vowel.defaults \
-fparam svm.param

svm_lnk -pathdata $LNKHOME/data/class -finput vowel.test \
-fdescribe vowel.defaults -fparam svm.param

DESCRIPTION
This program classifies two or more classes using one or more linear or nonlinear two-class support vector machine classifiers. Support vector machine (SVM) classifiers are similar to perceptrons. They separate patterns in two classes using a hyperplane. SVM's, however, position the separating hyperplane to maximize the margin, where the margin is the minimum distance from the separating hyperplane to patterns in the two classes. Training involves attempting to satisfy Karush-Kuhn-Tucker (KKT) conditions that specify the quadratic minimization problem that defines the SVM. A linear SVM performs this minimization in the space of the original input features while a nonlinear SVM performs this minimization in an higher-order space implicitly generated using nonlinear kernels. See reference [3] below for an introduction to support vector machines.

In practice, it is often impossible to satisfy strict KKT conditions and separate all patterns. An upper bound (cbound) is used to set the maximum cost incurred for violation of the KKT conditions. This allows the output for support vectors with non-zero Lagrange multipliers to deviate from +/- 1.0. Higher cbound values lead to more complex classifiers that try to correctly classify each training pattern. Lower cbound values lead to simpler classifiers that allow misclassifications and violations of strict KKT conditions. The value of cbound for a particular problem must be selected empirically using cross-validation. This upper bound is set using the -cbound x flag.

The kernel type determines whether an SVM is a simple linear discriminator or whether it maps the inputs to a higher-order space. This software supports linear kernels, Gaussian kernels, polynomial kernels (x*y)^n, and inhomogeneous polynomial kernels (x*y + 1)^n. The kernel type is selected using the -kernel_type n flag. Some kernels have free parameters. The standard deviation has to be selected for the Gaussian kernel and the order has to be selected for the polynomial kernels. These parameters are selected using the -kernel_sigma x and -kernel_order n flags. In addition, the inner terms in the polynomial and inhomogeneous kernels can be divided by a scale factor before being raised to power. This improves numerical stability if there are many input features. For example you could divide by 256 if there were 256 input features and the data was normalized to a mean of zero and standard deviation of one. This scale factor is selected using the -kernel_poly_divisor x flag.

Classical SVM classifiers only discriminate between two classes. This software implements two approaches for multi-class problems with M classes. These approaches are selected using the -mclass_type n flag. The first approach, -mclass_type 0, constructs M component binary classifiers which separate each class from all the remaining classes. During testing, the classification decision corresponds to the class of the component classifier with the highest output (before the clipping nonlinearity). The second approach, -mclass_type 1, constructs many more simpler binary classifiers that separate all possible combinations of classes taken two at a time. This results in M*(M-1)/2 simple classifiers. During testing, the class with the most votes across all binary component classifiers is selected. In the case of ties, outputs (before the clipping nonlinearity) for each class are scanned across all pairwise classifiers that include that class, and the minimum is found. These minimum values are compared to find the class with the highest minimum value. The final classification decision corresponds to that class. The second pairwise approach sometimes provides better performance. Although it requires many more classifiers, they are simpler, and overall training time is often similar across both approaches.

Classical SVM classifiers provide zero/one outputs that indicate only whether the input pattern belongs to class A or B. They do not provide posterior probabilities that can be used to adjust differences in prior probabilities between training and testing, assign costs to different types of errors, reject patterns, and form complete ROC curves. This software approximates posterior probabilities using an approach motivated by [4] but simplified to use only training data. The output of each component SVM (before the clipping nonlinearity) is fed into a sigmoid function with an output ranging from 0 to 1.0 and constrained to produce an output of 0.5 when the input is at the decision region boundary (input = 0.0). This constraint preserves the error rate for component binary classifiers when errors have equal costs. The slope of the sigmoid is selected during training to minimize the mean squared error between the sigmoid output and desired outputs of zero and one for the two classes. Training patterns with unclipped outputs near +/- 1 (mainly support vectors) are weighted much less in this minimization because internal parameters in the classifier have been tuned to produce outputs of +/- 1 for these patterns. For multi-class problems, posterior probabilities are computed from the component classifiers. When M classifiers are generated for an M-class problem, posterior probabilities are the M outputs for each class from the M component classifiers. When M*(M-1)/2 pairwise classifiers are generated, the posterior probability for each class is the minimum posterior probability output for that class across all pairwise classifiers. A sigmoid is always fit to the output of every component classifier. This fit can be used or ignored depending on the -sigmoid_fit flag. This flag should normally be used to provide an output that approximates posterior probabilities.

This implementation of SVM's uses an efficient, fast algorithm that scales well to problems with many features and many training patterns. It uses John Platt's Sequential Minimal Optimization (SMO) algorithm [1] as improved by Keerthi, Shevade, Bhattacharyya and Murthy [2]. The core algorithm examines pairs of patterns (one from each class) and modifies Lagrange multipliers using an analytic solution when patterns violate KKT conditions. Training involves two-pass sweeps. In the first pass of a sweep, all patterns are examined one at a time to find violations of KKT conditions. Lagrange multipliers are adapted when a violation is found. In the second pass, the subset of patterns found in the first pass that violate KTT conditions are examined repetitively and their Lagrange multipliers are adjusted until such patterns satisfy KKT conditions. Adjustments always involve pairs of patterns that do not satisfy KKT conditions. Another examination of all patterns to find KKT condition violations begins the next sweep. Training stops when all patterns satisfy KKT conditions. During training, two bias values (the bias for the hyperplane) are maintained and used by the algorithm. These high and low bias values initially differ, and then converge to be similar after convergence. After the algorithm completes, a final independent check is made to make sure the solution satisfies all KKT conditions. A warning is printed along with diagnostics if the solution does not satisfy KKT conditions.

During training, information is printed out during each pass of every sweep when the verbosity flag is greater than zero. The following is an example of a table for a component classifier which separates classes 9 and 8 (digits "9" versus "8") for the ocrdigit data base. A linear kernel was used, cbound was 1.0, and there were 120 training patterns.

NSweeps Changed TChanged KernelEvals UnBounded AtUpper HiBias LoBias DeltaBias
1       58      58       3140        53        0       1.73   0.07   1.655375
1       2       60       3357        50        0       1.77   0.29   1.478128
2       64      124      9528        53        0       1.45   0.80   0.644528
2       0       124      9528        53        0       1.45   0.80   0.644528
3       62      186      16061       47        0       1.45   1.07   0.379679
3       18      204      17594       37        0       1.36   1.19   0.175545
4       42      246      20861       40        0       1.36   1.24   0.122306
4       409     655      50742       33        0       1.27   1.27   0.001997
5       0       655      50742       33        0       1.27   1.27   0.001997

The first column indicates the sweep number. As noted above, there are two passes per sweep. The first pass examines all training patterns and the second examines only the subset of patterns found in the first pass that violates the KKT conditions. The second column indicates the number of patterns in a pass that violate KKT conditions. For example, on the first pass through all 120 training patterns, 58 patterns violated the KKT conditions. Lagrange multipliers for these patterns are all updated or "changed". The third column indicates the cumulative number of patterns where Lagrange multipliers were adapted or the total patterns with "changed" Lagrange multipliers. For example, on the first past, 58 adaptations occurred and a total of 655 adaptations were required to complete training. The Fourth column shows the cumulative number of kernel evaluations required during training. When the number of input features is large, most of the computation in this algorithm involves kernel evaluations. For this problem, more than 50,000 kernel evaluations were required to complete training. The fifth column shows the number of support vectors that have non-zero Lagrange multipliers that are below cbound. After training is complete, there are 33 non-zero support support vectors below cbound and none at the upper bound. All support vectors (unbounded and at the upper bound) must be stored and used for classification. Support vectors at the upper bound correspond to patterns where outputs are not +/- 1.0. These patterns may or may not be misclassified. The final three columns show the lower bias bound, the upper bias bound, and the difference between these bounds. See [3] for a descriptions of these bounds and how they are computed. After training is complete, the difference between these bias bounds should be small and less than the KKT tolerance.

Any implementation of SVMs must address numerical precision limitations and the desired accuracy of fit to KKT conditions. This software is designed for input features that have been normalized to have zero mean and unit variance. This is achieved in LNKnet using the simple normalization in the Feature Normalization window. In addition, the accuracy desired for KKT conditions can be adjusted. KKT conditions specify that the unclipped component classifier output for non-zero support vectors below cbound must be +/- 1. In practice, exactly producing outputs of +/- 1 may take excessively long and have little effect on classification performance. The tolerance (absolute difference between actual and desired outputs) allowed around desired outputs of +/- 1 is specified using the -kkt_tolerance x flag. This value defaults to 0.001. It can be increased, for example to 0.01, to reduce convergence time. Another flag -alpha_tolerance x controls a lower limit on Lagrange multipliers. Lagrange multipliers below this limit are set to zero. This defaults to 1e-08. It can be lowered when data normalization and/or and the kernel_poly_divisor scale factor can not be adjusted properly and Lagrange multiplier adjustments are small and below the threshold. Evidence of small Lagrange multiplier adjustments below below this limit is that the algorithm converges rapidly to a bad solution that doesn't satisfy KKT conditions without changing Lagrange multipliers on any training patterns. A warning will be printed with recommended changes if this occurs. This tolerance can also be increased if there are too many small Lagrange multipliers.

This algorithm converges (usually rapidly) to a good solution. Good solutions are found for a wide range of parameter values. Extensive error checking is performed to verify the final solution and warnings and corrective suggestions are suggested if KKT conditions are not satisfied. This only occurs if numerical precision problems occur. Such problems usually don't occur if (1) The data is normalized to zero mean unit variance using simple normalization, (2) If the Gaussian kernel standard deviation isn't too large compared to the number of input features, (3) The polynomial kernel divisor is roughly equal to the number of input features, and (4) There are no severe outlier data patterns that are far away from other patters of the same class but among patterns of some other class. When KKT conditions can't be satisfied, the algorithm will still converge and warnings will be printed out stating why KKT conditions weren't met and how serious this is. These warnings can sometimes be ignored because when they occur, classifiers are created and they typically work reasonably well. The extent of KKT violation is printed out and small violations of KKT conditions don't affect classification performance significantly. If warnings are printed out and KKT violations are substantial, try a different classifier (e.g. Gaussian or polynomial instead of linear), try increasing cbound, try -mclass_type 1 instead of -mclass type 0, increase kernel_poly_divisor, or decrease the standard deviation of Gaussian kernels. In addition try searching for obvious extreme outlier patterns that might be due to mislabeled data. For high-order polynomial kernels and Gaussian kernels with large standard deviations, lowering alpha_tolerance may help. For multi-class problems, warnings are printed out for each component classifier and the total number of warnings is printed out when training is complete. Search for the string "WARNING" in the training log file.

This software has been successfully applied to large problems with many input features and many training patterns. Memory requirements increase roughly linearly in the number of input features and number of training patterns.

OPTIONS
-- Prints a list of all flags and arguments.

-cound x
    The upper bound for support vector Lagrange multipliers. This represents the cost assigned to a training pattern that violates KKT conditions. Higher values result in more complex classifiers that try to satisfy KKT conditions for all patterns and classify every pattern correctly. Lower values result in simpler classifiers that allow violations of KKT conditions and misclassifications.

-mclass_type [0,1]
    Specifies the approach used to classify patterns for multi-class problems. The first "one-versus-all-other" approach is selected with -mclass_type 0. This option constructs M component binary classifiers that separate each class from all the remaining classes. During testing, the classification decision corresponds to the class of the component classifier with the highest output (before the clipping nonlinearity). The second "all-combinations" approach is selected with -mclass_type 1. This option constructs M*(M-1)/2 component binary classifiers that separate all possible combinations of classes taken two at a time. This results in M*(M-1)/2 simple classifiers. During testing, the class with the most votes across all binary component classifiers is selected. In the case of ties, outputs (before the clipping nonlinearity) for each class are scanned across all pairwise classifiers that include that class and the minimum is found. These minimum values are compared to find the class with the highest minimum value. Although this approach creates more component classifiers, each one is simpler, and overall performance is often better.

-kernel_type [0,1,2,3,4]
    Select the kernel type. The value supplied should be 0 for linear kernels, 1 for Gaussian kernels, 2 for polynomial kernels (x*x)^n, 3 for inhomogeneous polynomial kernels (x*x +1)^n, and 4 for linear kernels where support vectors are stored instead of the simpler weight vector. Option 4 was provided for pedagogical purposes. Normally, only weights are stored and used for classification with linear support vector machines. Option 4 forces all support vectors to be stored for linear support vector machines. These support vectors can be plotted on decision region plots (when internals are plotted) to illustrate how support vectors are selected.

-kernel_sigma x
    The standard deviation used for Gaussian kernels.

-kernel_order n
    The order or power for polynomial kernels.

-kernel_poly_divisor x
    The inner term for polynomial kernels is divided by this factor before being raised to a power. This inner term is either x*x for polynomial kernels or (x*x +1) for inhomogeneous polynomial kernels. This divisor should be adjusted such that the inner term is roughly one. If the inputs are normalized to zero mean unit variance, this should be the number of input features.

-sigmoid_fit
    Use a sigmoid to approximate posterior probabilities at the output of the classifier. A sigmoid fit is always made and this flag uses the fit.

-kkt_tolerance x
    KKT conditions are taken as satisfied if the absolute difference between outputs of component classifiers and +/- 1 is less than this value for non-zero support vectors below the upper cbound. This defaults to .001 and can be increased (e.g. to .01) to speed convergence.

-alpha_tolerance x
    Lagrange multipliers below this threshold are set to zero. This defaults to 1e-08 but can be decreased if the data is not normalized or if there are many small Lagrange multipliers.

REFERENCES
1. J. Platt, Fast Training of Support Vector Machines Using Sequential Minimal Optimization, in "Advances in Kernel Methods - Support Vector Learning", in B. Scholkopf, C. Burges, and A. Smola, eds. MIT Press, 1998, http://www.research.microsoft.com/~jplatt/smo.html.

2. S.S. Keerthi, S.K. Shevade, C. Bhattacharvva, K.R.K. Murthy, Improvements to PLatt's SMO Algorithm for SVM Classifier Design, Technical Report CD-99-14, National University of Singapore, 1999, http://guppy.mpe.nus.edu.sg/~mpessk/svm/svm.html.

3. N. Cristianini and J. Shawe-Taylor, An Introduction to Support Vector Machines (and Other Kernel-Based Learning Methods), Cambridge University Press, 2000, http://www.suport-vector.net.

4. J. Platt, Probabilistic Outputs for Support Vector Machines and Comparisons to Regularized Likelihood Methods, in Advances in Large Margin Classifiers, A. Smola, P. Bartlett, B. Scholkopf, D. Schuumans, eds.,2000, hhtp://www.research.microsoft.com/users/jplatt/SVMprob.ps.gz.

SEE ALSO
classifier(1), svm_feat_sel(1), svm_plot_bound(1)

HISTORY
First written 2 Feb 2001 by Richard Lippmann
Modifed 30 October 2001 by Richard Lippmann
Modified 18 July 2002 by Richard Lippmann

BUGS
Send bug reports to Linda Kukolich (kukolich@sst.ll.mit.edu)