Evaluation

java.lang.Object
- weka.classifiers.Evaluation

All Implemented Interfaces:

java.io.Serializable, RevisionHandler, Summarizable

Direct Known Subclasses:

AggregateableEvaluation
```
public class Evaluation
extends java.lang.Object
implements java.io.Serializable, Summarizable, RevisionHandler
```
Class for evaluating machine learning models. Delegates to the actual implementation in weka.classifiers.evaluation.Evaluation.
-------------------------------------------------------------------
General options when evaluating a learning scheme from the command-line:
-t filename
Name of the file with the training data. (required)
-T filename
Name of the file with the test data. If missing a cross-validation is performed.
-c index
Index of the class attribute (1, 2, ...; default: last).
-x number
The number of folds for the cross-validation (default: 10).
-no-cv
No cross validation. If no test file is provided, no evaluation is done.
-split-percentage percentage
Sets the percentage for the train/test set split, e.g., 66.
-preserve-order
Preserves the order in the percentage split instead of randomizing the data first with the seed value ('-s').
-s seed
Random number seed for the cross-validation and percentage split (default: 1).
-m filename
The name of a file containing a cost matrix.
-l filename
Loads classifier from the given file. In case the filename ends with ".xml", a PMML file is loaded or, if that fails, options are loaded from XML.
-d filename
Saves classifier built from the training data into the given file. In case the filename ends with ".xml" the options are saved XML, not the model.
-v
Outputs no statistics for the training data.
-o
Outputs statistics only, not the classifier.
-i
Outputs information-retrieval statistics per class.
-k
Outputs information-theoretic statistics.
-classifications "weka.classifiers.evaluation.output.prediction.AbstractOutput + options"
Uses the specified class for generating the classification output. E.g.: weka.classifiers.evaluation.output.prediction.PlainText or : weka.classifiers.evaluation.output.prediction.CSV -p range
Outputs predictions for test instances (or the train instances if no test instances provided and -no-cv is used), along with the attributes in the specified range (and nothing else). Use '-p 0' if no attributes are desired.
Deprecated: use "-classifications ..." instead.
-distribution
Outputs the distribution instead of only the prediction in conjunction with the '-p' option (only nominal classes).
Deprecated: use "-classifications ..." instead.
-no-predictions
Turns off the collection of predictions in order to conserve memory.
-r
Outputs cumulative margin distribution (and nothing else).
-g
Only for classifiers that implement "Graphable." Outputs the graph representation of the classifier (and nothing else).
-xml filename | xml-string
Retrieves the options from the XML-data instead of the command line.
-threshold-file file
The file to save the threshold data to. The format is determined by the extensions, e.g., '.arff' for ARFF format or '.csv' for CSV.
-threshold-label label
The class label to determine the threshold data for (default is the first label)
-------------------------------------------------------------------
Example usage as the main of a classifier (called FunkyClassifier):
```
 public static void main(String [] args) {
   runClassifier(new FunkyClassifier(), args);
 }
 
```
------------------------------------------------------------------
Example usage from within an application:
```
 Instances trainInstances = ... instances got from somewhere
 Instances testInstances = ... instances got from somewhere
 Classifier scheme = ... scheme got from somewhere
 
 Evaluation evaluation = new Evaluation(trainInstances);
 evaluation.evaluateModel(scheme, testInstances);
 System.out.println(evaluation.toSummaryString());
 
```
Version:

$Revision: 14450 $

Author:

Eibe Frank (eibe@cs.waikato.ac.nz), Len Trigg (trigg@cs.waikato.ac.nz)

See Also:

Serialized Form

Field Summary

Fields
Modifier and Type Field and Description

static java.lang.String[] BUILT_IN_EVAL_METRICS

Fields
Modifier and Type	Field and Description
`static java.lang.String[]`	`BUILT_IN_EVAL_METRICS`

Constructor Summary

Constructors
Constructor and Description

Evaluation(Instances data)

Evaluation(Instances data, CostMatrix costMatrix)

Constructors
Constructor and Description
`Evaluation(Instances data)`
`Evaluation(Instances data, CostMatrix costMatrix)`

Method Summary

All Methods Static Methods Instance Methods Concrete Methods
Modifier and Type	Method and Description
`double`	`areaUnderPRC(int classIndex)` Returns the area under precision-recall curve (AUPRC) for those predictions that have been collected in the evaluateClassifier(Classifier, Instances) method.
`double`	`areaUnderROC(int classIndex)` Returns the area under ROC for those predictions that have been collected in the evaluateClassifier(Classifier, Instances) method.
`double`	`avgCost()` Gets the average cost, that is, total cost of misclassifications (incorrect plus unclassified) over the total number of instances.
`double[][]`	`confusionMatrix()` Returns a copy of the confusion matrix.
`double`	`correct()` Gets the number of instances correctly classified (that is, for which a correct prediction was made).
`double`	`correlationCoefficient()` Returns the correlation coefficient if the class is numeric.
`double`	`coverageOfTestCasesByPredictedRegions()` Gets the coverage of the test cases by the predicted regions at the confidence level specified when evaluation was performed.
`void`	`crossValidateModel(Classifier classifier, Instances data, int numFolds, java.util.Random random)` Performs a (stratified if class is nominal) cross-validation for a classifier on a set of instances.
`void`	`crossValidateModel(Classifier classifier, Instances data, int numFolds, java.util.Random random, java.lang.Object... forPredictionsPrinting)` Performs a (stratified if class is nominal) cross-validation for a classifier on a set of instances.
`void`	`crossValidateModel(java.lang.String classifierString, Instances data, int numFolds, java.lang.String[] options, java.util.Random random)` Performs a (stratified if class is nominal) cross-validation for a classifier on a set of instances.
`boolean`	`equals(java.lang.Object obj)` Tests whether the current evaluation object is equal to another evaluation object.
`double`	`errorRate()` Returns the estimated error rate or the root mean squared error (if the class is numeric).
`double[]`	`evaluateModel(Classifier classifier, Instances data, java.lang.Object... forPredictionsPrinting)` Evaluates the classifier on a given set of instances.
`static java.lang.String`	`evaluateModel(Classifier classifier, java.lang.String[] options)` Evaluates a classifier with the options given in an array of strings.
`static java.lang.String`	`evaluateModel(java.lang.String classifierString, java.lang.String[] options)` Evaluates a classifier with the options given in an array of strings.
`double`	`evaluateModelOnce(Classifier classifier, Instance instance)` Evaluates the classifier on a single instance.
`double`	`evaluateModelOnce(double[] dist, Instance instance)` Evaluates the supplied distribution on a single instance.
`void`	`evaluateModelOnce(double prediction, Instance instance)` Evaluates the supplied prediction on a single instance.
`double`	`evaluateModelOnceAndRecordPrediction(Classifier classifier, Instance instance)` Evaluates the classifier on a single instance and records the prediction.
`double`	`evaluateModelOnceAndRecordPrediction(double[] dist, Instance instance)` Evaluates the supplied distribution on a single instance.
`double`	`evaluationForSingleInstance(double[] dist, Instance instance, boolean storePredictions)` Evaluates the supplied distribution on a single instance.
`double`	`falseNegativeRate(int classIndex)` Calculate the false negative rate with respect to a particular class.
`double`	`falsePositiveRate(int classIndex)` Calculate the false positive rate with respect to a particular class.
`double`	`fMeasure(int classIndex)` Calculate the F-Measure with respect to a particular class.
`static java.util.List<java.lang.String>`	`getAllEvaluationMetricNames()` Utility method to get a list of the names of all built-in and plugin evaluation metrics
`double[]`	`getClassPriors()` Get the current weighted class counts.
`boolean`	`getDiscardPredictions()` Returns whether predictions are not recorded at all, in order to conserve memory.
`Instances`	`getHeader()` Returns the header of the underlying dataset.
`java.util.List<java.lang.String>`	`getMetricsToDisplay()` Get a list of the names of metrics to have appear in the output The default is to display all built in metrics and plugin metrics that haven't been globally disabled.
`AbstractEvaluationMetric`	`getPluginMetric(java.lang.String name)` Get the named plugin evaluation metric
`java.util.List<AbstractEvaluationMetric>`	`getPluginMetrics()` Returns the list of plugin metrics in use (or null if there are none)
`java.lang.String`	`getRevision()` Returns the revision string.
`double`	`incorrect()` Gets the number of instances incorrectly classified (that is, for which an incorrect prediction was made).
`double`	`kappa()` Returns value of kappa statistic if class is nominal.
`double`	`KBInformation()` Return the total Kononenko & Bratko Information score in bits.
`double`	`KBMeanInformation()` Return the Kononenko & Bratko Information score in bits per instance.
`double`	`KBRelativeInformation()` Return the Kononenko & Bratko Relative Information score.
`static void`	`main(java.lang.String[] args)` A test method for this class.
`double`	`matthewsCorrelationCoefficient(int classIndex)` Calculates the matthews correlation coefficient (sometimes called phi coefficient) for the supplied class
`double`	`meanAbsoluteError()` Returns the mean absolute error.
`double`	`meanPriorAbsoluteError()` Returns the mean absolute error of the prior.
`double`	`numFalseNegatives(int classIndex)` Calculate number of false negatives with respect to a particular class.
`double`	`numFalsePositives(int classIndex)` Calculate number of false positives with respect to a particular class.
`double`	`numInstances()` Gets the number of test instances that had a known class value (actually the sum of the weights of test instances with known class value).
`double`	`numTrueNegatives(int classIndex)` Calculate the number of true negatives with respect to a particular class.
`double`	`numTruePositives(int classIndex)` Calculate the number of true positives with respect to a particular class.
`double`	`pctCorrect()` Gets the percentage of instances correctly classified (that is, for which a correct prediction was made).
`double`	`pctIncorrect()` Gets the percentage of instances incorrectly classified (that is, for which an incorrect prediction was made).
`double`	`pctUnclassified()` Gets the percentage of instances not classified (that is, for which no prediction was made by the classifier).
`double`	`precision(int classIndex)` Calculate the precision with respect to a particular class.
`java.util.ArrayList<Prediction>`	`predictions()` Returns the predictions that have been collected.
`double`	`priorEntropy()` Calculate the entropy of the prior distribution.
`double`	`recall(int classIndex)` Calculate the recall with respect to a particular class.
`double`	`relativeAbsoluteError()` Returns the relative absolute error.
`double`	`rootMeanPriorSquaredError()` Returns the root mean prior squared error.
`double`	`rootMeanSquaredError()` Returns the root mean squared error.
`double`	`rootRelativeSquaredError()` Returns the root relative squared error if the class is numeric.
`void`	`setDiscardPredictions(boolean value)` Sets whether to discard predictions, ie, not storing them for future reference via predictions() method in order to conserve memory.
`void`	`setMetricsToDisplay(java.util.List<java.lang.String> display)` Set a list of the names of metrics to have appear in the output.
`void`	`setPriors(Instances train)` Sets the class prior probabilities.
`double`	`SFEntropyGain()` Returns the total SF, which is the null model entropy minus the scheme entropy.
`double`	`SFMeanEntropyGain()` Returns the SF per instance, which is the null model entropy minus the scheme entropy, per instance.
`double`	`SFMeanPriorEntropy()` Returns the entropy per instance for the null model.
`double`	`SFMeanSchemeEntropy()` Returns the entropy per instance for the scheme.
`double`	`SFPriorEntropy()` Returns the total entropy for the null model.
`double`	`SFSchemeEntropy()` Returns the total entropy for the scheme.
`double`	`sizeOfPredictedRegions()` Gets the average size of the predicted regions, relative to the range of the target in the training data, at the confidence level specified when evaluation was performed.
`java.lang.String`	`toClassDetailsString()` Generates a breakdown of the accuracy for each class (with default title), incorporating various information-retrieval statistics, such as true/false positive rate, precision/recall/F-Measure.
`java.lang.String`	`toClassDetailsString(java.lang.String title)` Generates a breakdown of the accuracy for each class, incorporating various information-retrieval statistics, such as true/false positive rate, precision/recall/F-Measure.
`java.lang.String`	`toCumulativeMarginDistributionString()` Output the cumulative margin distribution as a string suitable for input for gnuplot or similar package.
`void`	`toggleEvalMetrics(java.util.List<java.lang.String> metricsToToggle)` Toggle the output of the metrics specified in the supplied list.
`java.lang.String`	`toMatrixString()` Calls toMatrixString() with a default title.
`java.lang.String`	`toMatrixString(java.lang.String title)` Outputs the performance statistics as a classification confusion matrix.
`java.lang.String`	`toSummaryString()` Calls toSummaryString() with no title and no complexity stats.
`java.lang.String`	`toSummaryString(boolean printComplexityStatistics)` Calls toSummaryString() with a default title.
`java.lang.String`	`toSummaryString(java.lang.String title, boolean printComplexityStatistics)` Outputs the performance statistics in summary form.
`double`	`totalCost()` Gets the total cost, that is, the cost of each prediction times the weight of the instance, summed over all instances.
`double`	`trueNegativeRate(int classIndex)` Calculate the true negative rate with respect to a particular class.
`double`	`truePositiveRate(int classIndex)` Calculate the true positive rate with respect to a particular class.
`double`	`unclassified()` Gets the number of instances not classified (that is, for which no prediction was made by the classifier).
`double`	`unweightedMacroFmeasure()` Unweighted macro-averaged F-measure.
`double`	`unweightedMicroFmeasure()` Unweighted micro-averaged F-measure.
`void`	`updatePriors(Instance instance)` Updates the class prior probabilities or the mean respectively (when incrementally training).
`void`	`useNoPriors()` disables the use of priors, e.g., in case of de-serialized schemes that have no access to the original training set, but are evaluated on a set set.
`double`	`weightedAreaUnderPRC()` Calculates the weighted (by class size) AUPRC.
`double`	`weightedAreaUnderROC()` Calculates the weighted (by class size) AUC.
`double`	`weightedFalseNegativeRate()` Calculates the weighted (by class size) false negative rate.
`double`	`weightedFalsePositiveRate()` Calculates the weighted (by class size) false positive rate.
`double`	`weightedFMeasure()` Calculates the macro weighted (by class size) average F-Measure.
`double`	`weightedMatthewsCorrelation()` Calculates the weighted (by class size) matthews correlation coefficient.
`double`	`weightedPrecision()` Calculates the weighted (by class size) precision.
`double`	`weightedRecall()` Calculates the weighted (by class size) recall.
`double`	`weightedTrueNegativeRate()` Calculates the weighted (by class size) true negative rate.
`double`	`weightedTruePositiveRate()` Calculates the weighted (by class size) true positive rate.
`static java.lang.String`	`wekaStaticWrapper(Sourcable classifier, java.lang.String className)` Wraps a static classifier in enough source to test using the weka class libraries.

Methods inherited from class java.lang.Object
getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Field Detail
  - BUILT_IN_EVAL_METRICS
```
public static final java.lang.String[] BUILT_IN_EVAL_METRICS
```
- Constructor Detail
  - Evaluation
```
public Evaluation(Instances data)
           throws java.lang.Exception
```
    Throws:
    
    java.lang.Exception
  - Evaluation
```
public Evaluation(Instances data,
                  CostMatrix costMatrix)
           throws java.lang.Exception
```
    Throws:
    
    java.lang.Exception
- Method Detail
  - getAllEvaluationMetricNames
```
public static java.util.List<java.lang.String> getAllEvaluationMetricNames()
```
    Utility method to get a list of the names of all built-in and plugin evaluation metrics
    
    Returns:
    
    the complete list of available evaluation metrics
  - getHeader
```
public Instances getHeader()
```
    Returns the header of the underlying dataset.
    
    Returns:
    
    the header information
  - getPluginMetrics
```
public java.util.List<AbstractEvaluationMetric> getPluginMetrics()
```
    Returns the list of plugin metrics in use (or null if there are none)
    
    Returns:
    
    the list of plugin metrics
  - getPluginMetric
```
public AbstractEvaluationMetric getPluginMetric(java.lang.String name)
```
    Get the named plugin evaluation metric
    
    Parameters:
    
    name - the name of the metric (as returned by AbstractEvaluationMetric.getName()) or the fully qualified class name of the metric to find
    
    Returns:
    
    the metric or null if the metric is not in the list of plugin metrics
  - setMetricsToDisplay
```
public void setMetricsToDisplay(java.util.List<java.lang.String> display)
```
    Set a list of the names of metrics to have appear in the output. The default is to display all built in metrics and plugin metrics that haven't been globally disabled.
    
    Parameters:
    
    display - a list of metric names to have appear in the output
  - getMetricsToDisplay
```
public java.util.List<java.lang.String> getMetricsToDisplay()
```
    Get a list of the names of metrics to have appear in the output The default is to display all built in metrics and plugin metrics that haven't been globally disabled.
    
    Returns:
    
    a list of metric names to have appear in the output
  - toggleEvalMetrics
```
public void toggleEvalMetrics(java.util.List<java.lang.String> metricsToToggle)
```
    Toggle the output of the metrics specified in the supplied list.
    
    Parameters:
    
    metricsToToggle - a list of metrics to toggle
  - setDiscardPredictions
```
public void setDiscardPredictions(boolean value)
```
    Sets whether to discard predictions, ie, not storing them for future reference via predictions() method in order to conserve memory.
    
    Parameters:
    
    value - true if to discard the predictions
    
    See Also:
    
    predictions()
  - getDiscardPredictions
```
public boolean getDiscardPredictions()
```
    Returns whether predictions are not recorded at all, in order to conserve memory.
    
    Returns:
    
    true if predictions are not recorded
    
    See Also:
    
    predictions()
  - areaUnderROC
```
public double areaUnderROC(int classIndex)
```
    Returns the area under ROC for those predictions that have been collected in the evaluateClassifier(Classifier, Instances) method. Returns Utils.missingValue() if the area is not available.
    
    Parameters:
    
    classIndex - the index of the class to consider as "positive"
    
    Returns:
    
    the area under the ROC curve or not a number
  - weightedAreaUnderROC
```
public double weightedAreaUnderROC()
```
    Calculates the weighted (by class size) AUC.
    
    Returns:
    
    the weighted AUC.
  - areaUnderPRC
```
public double areaUnderPRC(int classIndex)
```
    Returns the area under precision-recall curve (AUPRC) for those predictions that have been collected in the evaluateClassifier(Classifier, Instances) method. Returns Utils.missingValue() if the area is not available.
    
    Parameters:
    
    classIndex - the index of the class to consider as "positive"
    
    Returns:
    
    the area under the precision-recall curve or not a number
  - weightedAreaUnderPRC
```
public double weightedAreaUnderPRC()
```
    Calculates the weighted (by class size) AUPRC.
    
    Returns:
    
    the weighted AUPRC.
  - confusionMatrix
```
public double[][] confusionMatrix()
```
    Returns a copy of the confusion matrix.
    
    Returns:
    
    a copy of the confusion matrix as a two-dimensional array
  - crossValidateModel
```
public void crossValidateModel(Classifier classifier,
                               Instances data,
                               int numFolds,
                               java.util.Random random)
                        throws java.lang.Exception
```
    Performs a (stratified if class is nominal) cross-validation for a classifier on a set of instances. Now performs a deep copy of the classifier before each call to buildClassifier() (just in case the classifier is not initialized properly).
    
    Parameters:
    
    classifier - the classifier with any options set.
    
    data - the data on which the cross-validation is to be performed
    
    numFolds - the number of folds for the cross-validation
    
    random - random number generator for randomization
    
    Throws:
    
    java.lang.Exception - if a classifier could not be generated successfully or the class is not defined
  - crossValidateModel
```
public void crossValidateModel(Classifier classifier,
                               Instances data,
                               int numFolds,
                               java.util.Random random,
                               java.lang.Object... forPredictionsPrinting)
                        throws java.lang.Exception
```
    Performs a (stratified if class is nominal) cross-validation for a classifier on a set of instances. Now performs a deep copy of the classifier before each call to buildClassifier() (just in case the classifier is not initialized properly).
    
    Parameters:
    
    classifier - the classifier with any options set.
    
    data - the data on which the cross-validation is to be performed
    
    numFolds - the number of folds for the cross-validation
    
    random - random number generator for randomization
    
    forPredictionsPrinting - varargs parameter that, if supplied, is expected to hold a weka.classifiers.evaluation.output.prediction.AbstractOutput object
    
    Throws:
    
    java.lang.Exception - if a classifier could not be generated successfully or the class is not defined
  - crossValidateModel
```
public void crossValidateModel(java.lang.String classifierString,
                               Instances data,
                               int numFolds,
                               java.lang.String[] options,
                               java.util.Random random)
                        throws java.lang.Exception
```
    Performs a (stratified if class is nominal) cross-validation for a classifier on a set of instances.
    
    Parameters:
    
    classifierString - a string naming the class of the classifier
    
    data - the data on which the cross-validation is to be performed
    
    numFolds - the number of folds for the cross-validation
    
    options - the options to the classifier. Any options
    
    random - the random number generator for randomizing the data accepted by the classifier will be removed from this array.
    
    Throws:
    
    java.lang.Exception - if a classifier could not be generated successfully or the class is not defined
  - evaluateModel
```
public static java.lang.String evaluateModel(java.lang.String classifierString,
                                             java.lang.String[] options)
                                      throws java.lang.Exception
```
    Evaluates a classifier with the options given in an array of strings.
    Valid options are:
    -t filename
    Name of the file with the training data. (required)
    -T filename
    Name of the file with the test data. If missing a cross-validation is performed.
    -c index
    Index of the class attribute (1, 2, ...; default: last).
    -x number
    The number of folds for the cross-validation (default: 10).
    -no-cv
    No cross validation. If no test file is provided, no evaluation is done.
    -split-percentage percentage
    Sets the percentage for the train/test set split, e.g., 66.
    -preserve-order
    Preserves the order in the percentage split instead of randomizing the data first with the seed value ('-s').
    -s seed
    Random number seed for the cross-validation and percentage split (default: 1).
    -m filename
    The name of a file containing a cost matrix.
    -l filename
    Loads classifier from the given file. In case the filename ends with ".xml",a PMML file is loaded or, if that fails, options are loaded from XML.
    -d filename
    Saves classifier built from the training data into the given file. In case the filename ends with ".xml" the options are saved XML, not the model.
    -v
    Outputs no statistics for the training data.
    -o
    Outputs statistics only, not the classifier.
    -i
    Outputs detailed information-retrieval statistics per class.
    -k
    Outputs information-theoretic statistics.
    -classifications "weka.classifiers.evaluation.output.prediction.AbstractOutput + options"
    Uses the specified class for generating the classification output. E.g.: weka.classifiers.evaluation.output.prediction.PlainText or : weka.classifiers.evaluation.output.prediction.CSV -p range
    Outputs predictions for test instances (or the train instances if no test instances provided and -no-cv is used), along with the attributes in the specified range (and nothing else). Use '-p 0' if no attributes are desired.
    Deprecated: use "-classifications ..." instead.
    -distribution
    Outputs the distribution instead of only the prediction in conjunction with the '-p' option (only nominal classes).
    Deprecated: use "-classifications ..." instead.
    -no-predictions
    Turns off the collection of predictions in order to conserve memory.
    -r
    Outputs cumulative margin distribution (and nothing else).
    -g
    Only for classifiers that implement "Graphable." Outputs the graph representation of the classifier (and nothing else).
    -xml filename | xml-string
    Retrieves the options from the XML-data instead of the command line.
    -threshold-file file
    The file to save the threshold data to. The format is determined by the extensions, e.g., '.arff' for ARFF format or '.csv' for CSV.
    -threshold-label label
    The class label to determine the threshold data for (default is the first label)
    
    Parameters:
    
    classifierString - class of machine learning classifier as a string
    
    options - the array of string containing the options
    
    Returns:
    
    a string describing the results
    
    Throws:
    
    java.lang.Exception - if model could not be evaluated successfully
  - evaluateModel
```
public static java.lang.String evaluateModel(Classifier classifier,
                                             java.lang.String[] options)
                                      throws java.lang.Exception
```
    Evaluates a classifier with the options given in an array of strings.
    Valid options are:
    -t name of training file
    Name of the file with the training data. (required)
    -T name of test file
    Name of the file with the test data. If missing a cross-validation is performed.
    -c class index
    Index of the class attribute (1, 2, ...; default: last).
    -x number of folds
    The number of folds for the cross-validation (default: 10).
    -no-cv
    No cross validation. If no test file is provided, no evaluation is done.
    -split-percentage percentage
    Sets the percentage for the train/test set split, e.g., 66.
    -preserve-order
    Preserves the order in the percentage split instead of randomizing the data first with the seed value ('-s').
    -s seed
    Random number seed for the cross-validation and percentage split (default: 1).
    -m file with cost matrix
    The name of a file containing a cost matrix.
    -l filename
    Loads classifier from the given file. In case the filename ends with ".xml",a PMML file is loaded or, if that fails, options are loaded from XML.
    -d filename
    Saves classifier built from the training data into the given file. In case the filename ends with ".xml" the options are saved XML, not the model.
    -v
    Outputs no statistics for the training data.
    -o
    Outputs statistics only, not the classifier.
    -i
    Outputs detailed information-retrieval statistics per class.
    -k
    Outputs information-theoretic statistics.
    -classifications "weka.classifiers.evaluation.output.prediction.AbstractOutput + options"
    Uses the specified class for generating the classification output. E.g.: weka.classifiers.evaluation.output.prediction.PlainText or : weka.classifiers.evaluation.output.prediction.CSV -p range
    Outputs predictions for test instances (or the train instances if no test instances provided and -no-cv is used), along with the attributes in the specified range (and nothing else). Use '-p 0' if no attributes are desired.
    Deprecated: use "-classifications ..." instead.
    -distribution
    Outputs the distribution instead of only the prediction in conjunction with the '-p' option (only nominal classes).
    Deprecated: use "-classifications ..." instead.
    -no-predictions
    Turns off the collection of predictions in order to conserve memory.
    -r
    Outputs cumulative margin distribution (and nothing else).
    -g
    Only for classifiers that implement "Graphable." Outputs the graph representation of the classifier (and nothing else).
    -xml filename | xml-string
    Retrieves the options from the XML-data instead of the command line.
    
    Parameters:
    
    classifier - machine learning classifier
    
    options - the array of string containing the options
    
    Returns:
    
    a string describing the results
    
    Throws:
    
    java.lang.Exception - if model could not be evaluated successfully
  - evaluateModel
```
public double[] evaluateModel(Classifier classifier,
                              Instances data,
                              java.lang.Object... forPredictionsPrinting)
                       throws java.lang.Exception
```
    Evaluates the classifier on a given set of instances. Note that the data must have exactly the same format (e.g. order of attributes) as the data used to train the classifier! Otherwise the results will generally be meaningless.
    
    Parameters:
    
    classifier - machine learning classifier
    
    data - set of test instances for evaluation
    
    forPredictionsPrinting - varargs parameter that, if supplied, is expected to hold a weka.classifiers.evaluation.output.prediction.AbstractOutput object
    
    Returns:
    
    the predictions
    
    Throws:
    
    java.lang.Exception - if model could not be evaluated successfully
  - evaluationForSingleInstance
```
public double evaluationForSingleInstance(double[] dist,
                                          Instance instance,
                                          boolean storePredictions)
                                   throws java.lang.Exception
```
    Evaluates the supplied distribution on a single instance.
    
    Parameters:
    
    dist - the supplied distribution
    
    instance - the test instance to be classified
    
    storePredictions - whether to store predictions for nominal classifier
    
    Returns:
    
    the prediction
    
    Throws:
    
    java.lang.Exception - if model could not be evaluated successfully
  - evaluateModelOnceAndRecordPrediction
```
public double evaluateModelOnceAndRecordPrediction(Classifier classifier,
                                                   Instance instance)
                                            throws java.lang.Exception
```
    Evaluates the classifier on a single instance and records the prediction.
    
    Parameters:
    
    classifier - machine learning classifier
    
    instance - the test instance to be classified
    
    Returns:
    
    the prediction made by the clasifier
    
    Throws:
    
    java.lang.Exception - if model could not be evaluated successfully or the data contains string attributes
  - evaluateModelOnce
```
public double evaluateModelOnce(Classifier classifier,
                                Instance instance)
                         throws java.lang.Exception
```
    Evaluates the classifier on a single instance.
    
    Parameters:
    
    classifier - machine learning classifier
    
    instance - the test instance to be classified
    
    Returns:
    
    the prediction made by the clasifier
    
    Throws:
    
    java.lang.Exception - if model could not be evaluated successfully or the data contains string attributes
  - evaluateModelOnce
```
public double evaluateModelOnce(double[] dist,
                                Instance instance)
                         throws java.lang.Exception
```
    Evaluates the supplied distribution on a single instance.
    
    Parameters:
    
    dist - the supplied distribution
    
    instance - the test instance to be classified
    
    Returns:
    
    the prediction
    
    Throws:
    
    java.lang.Exception - if model could not be evaluated successfully
  - evaluateModelOnceAndRecordPrediction
```
public double evaluateModelOnceAndRecordPrediction(double[] dist,
                                                   Instance instance)
                                            throws java.lang.Exception
```
    Evaluates the supplied distribution on a single instance.
    
    Parameters:
    
    dist - the supplied distribution
    
    instance - the test instance to be classified
    
    Returns:
    
    the prediction
    
    Throws:
    
    java.lang.Exception - if model could not be evaluated successfully
  - evaluateModelOnce
```
public void evaluateModelOnce(double prediction,
                              Instance instance)
                       throws java.lang.Exception
```
    Evaluates the supplied prediction on a single instance.
    
    Parameters:
    
    prediction - the supplied prediction
    
    instance - the test instance to be classified
    
    Throws:
    
    java.lang.Exception - if model could not be evaluated successfully
  - predictions
```
public java.util.ArrayList<Prediction> predictions()
```
    Returns the predictions that have been collected.
    
    Returns:
    
    a reference to the FastVector containing the predictions that have been collected. This should be null if no predictions have been collected.
  - wekaStaticWrapper
```
public static java.lang.String wekaStaticWrapper(Sourcable classifier,
                                                 java.lang.String className)
                                          throws java.lang.Exception
```
    Wraps a static classifier in enough source to test using the weka class libraries.
    
    Parameters:
    
    classifier - a Sourcable Classifier
    
    className - the name to give to the source code class
    
    Returns:
    
    the source for a static classifier that can be tested with weka libraries.
    
    Throws:
    
    java.lang.Exception - if code-generation fails
  - numInstances
```
public final double numInstances()
```
    Gets the number of test instances that had a known class value (actually the sum of the weights of test instances with known class value).
    
    Returns:
    
    the number of test instances with known class
  - coverageOfTestCasesByPredictedRegions
```
public final double coverageOfTestCasesByPredictedRegions()
```
    Gets the coverage of the test cases by the predicted regions at the confidence level specified when evaluation was performed.
    
    Returns:
    
    the coverage of the test cases by the predicted regions
  - sizeOfPredictedRegions
```
public final double sizeOfPredictedRegions()
```
    Gets the average size of the predicted regions, relative to the range of the target in the training data, at the confidence level specified when evaluation was performed.
    
    Returns:
    
    the average size of the predicted regions
  - incorrect
```
public final double incorrect()
```
    Gets the number of instances incorrectly classified (that is, for which an incorrect prediction was made). (Actually the sum of the weights of these instances)
    
    Returns:
    
    the number of incorrectly classified instances
  - pctIncorrect
```
public final double pctIncorrect()
```
    Gets the percentage of instances incorrectly classified (that is, for which an incorrect prediction was made).
    
    Returns:
    
    the percent of incorrectly classified instances (between 0 and 100)
  - totalCost
```
public final double totalCost()
```
    Gets the total cost, that is, the cost of each prediction times the weight of the instance, summed over all instances.
    
    Returns:
    
    the total cost
  - avgCost
```
public final double avgCost()
```
    Gets the average cost, that is, total cost of misclassifications (incorrect plus unclassified) over the total number of instances.
    
    Returns:
    
    the average cost.
  - correct
```
public final double correct()
```
    Gets the number of instances correctly classified (that is, for which a correct prediction was made). (Actually the sum of the weights of these instances)
    
    Returns:
    
    the number of correctly classified instances
  - pctCorrect
```
public final double pctCorrect()
```
    Gets the percentage of instances correctly classified (that is, for which a correct prediction was made).
    
    Returns:
    
    the percent of correctly classified instances (between 0 and 100)
  - unclassified
```
public final double unclassified()
```
    Gets the number of instances not classified (that is, for which no prediction was made by the classifier). (Actually the sum of the weights of these instances)
    
    Returns:
    
    the number of unclassified instances
  - pctUnclassified
```
public final double pctUnclassified()
```
    Gets the percentage of instances not classified (that is, for which no prediction was made by the classifier).
    
    Returns:
    
    the percent of unclassified instances (between 0 and 100)
  - errorRate
```
public final double errorRate()
```
    Returns the estimated error rate or the root mean squared error (if the class is numeric). If a cost matrix was given this error rate gives the average cost.
    
    Returns:
    
    the estimated error rate (between 0 and 1, or between 0 and maximum cost)
  - kappa
```
public final double kappa()
```
    Returns value of kappa statistic if class is nominal.
    
    Returns:
    
    the value of the kappa statistic
  - getRevision
```
public java.lang.String getRevision()
```
    Description copied from interface: RevisionHandler
    
    Returns the revision string.
    
    Specified by:
    
    getRevision in interface RevisionHandler
    
    Returns:
    
    the revision
  - correlationCoefficient
```
public final double correlationCoefficient()
                                    throws java.lang.Exception
```
    Returns the correlation coefficient if the class is numeric.
    
    Returns:
    
    the correlation coefficient
    
    Throws:
    
    java.lang.Exception - if class is not numeric
  - meanAbsoluteError
```
public final double meanAbsoluteError()
```
    Returns the mean absolute error. Refers to the error of the predicted values for numeric classes, and the error of the predicted probability distribution for nominal classes.
    
    Returns:
    
    the mean absolute error
  - meanPriorAbsoluteError
```
public final double meanPriorAbsoluteError()
```
    Returns the mean absolute error of the prior.
    
    Returns:
    
    the mean absolute error
  - relativeAbsoluteError
```
public final double relativeAbsoluteError()
                                   throws java.lang.Exception
```
    Returns the relative absolute error.
    
    Returns:
    
    the relative absolute error
    
    Throws:
    
    java.lang.Exception - if it can't be computed
  - rootMeanSquaredError
```
public final double rootMeanSquaredError()
```
    Returns the root mean squared error.
    
    Returns:
    
    the root mean squared error
  - rootMeanPriorSquaredError
```
public final double rootMeanPriorSquaredError()
```
    Returns the root mean prior squared error.
    
    Returns:
    
    the root mean prior squared error
  - rootRelativeSquaredError
```
public final double rootRelativeSquaredError()
```
    Returns the root relative squared error if the class is numeric.
    
    Returns:
    
    the root relative squared error
  - priorEntropy
```
public final double priorEntropy()
                          throws java.lang.Exception
```
    Calculate the entropy of the prior distribution.
    
    Returns:
    
    the entropy of the prior distribution
    
    Throws:
    
    java.lang.Exception - if the class is not nominal
  - KBInformation
```
public final double KBInformation()
                           throws java.lang.Exception
```
    Return the total Kononenko & Bratko Information score in bits.
    
    Returns:
    
    the K&B information score
    
    Throws:
    
    java.lang.Exception - if the class is not nominal
  - KBMeanInformation
```
public final double KBMeanInformation()
                               throws java.lang.Exception
```
    Return the Kononenko & Bratko Information score in bits per instance.
    
    Returns:
    
    the K&B information score
    
    Throws:
    
    java.lang.Exception - if the class is not nominal
  - KBRelativeInformation
```
public final double KBRelativeInformation()
                                   throws java.lang.Exception
```
    Return the Kononenko & Bratko Relative Information score.
    
    Returns:
    
    the K&B relative information score
    
    Throws:
    
    java.lang.Exception - if the class is not nominal
  - SFPriorEntropy
```
public final double SFPriorEntropy()
```
    Returns the total entropy for the null model.
    
    Returns:
    
    the total null model entropy
  - SFMeanPriorEntropy
```
public final double SFMeanPriorEntropy()
```
    Returns the entropy per instance for the null model.
    
    Returns:
    
    the null model entropy per instance
  - SFSchemeEntropy
```
public final double SFSchemeEntropy()
```
    Returns the total entropy for the scheme.
    
    Returns:
    
    the total scheme entropy
  - SFMeanSchemeEntropy
```
public final double SFMeanSchemeEntropy()
```
    Returns the entropy per instance for the scheme.
    
    Returns:
    
    the scheme entropy per instance
  - SFEntropyGain
```
public final double SFEntropyGain()
```
    Returns the total SF, which is the null model entropy minus the scheme entropy.
    
    Returns:
    
    the total SF
  - SFMeanEntropyGain
```
public final double SFMeanEntropyGain()
```
    Returns the SF per instance, which is the null model entropy minus the scheme entropy, per instance.
    
    Returns:
    
    the SF per instance
  - toCumulativeMarginDistributionString
```
public java.lang.String toCumulativeMarginDistributionString()
                                                      throws java.lang.Exception
```
    Output the cumulative margin distribution as a string suitable for input for gnuplot or similar package.
    
    Returns:
    
    the cumulative margin distribution
    
    Throws:
    
    java.lang.Exception - if the class attribute is nominal
  - toSummaryString
```
public java.lang.String toSummaryString()
```
    Calls toSummaryString() with no title and no complexity stats.
    
    Specified by:
    
    toSummaryString in interface Summarizable
    
    Returns:
    
    a summary description of the classifier evaluation
  - toSummaryString
```
public java.lang.String toSummaryString(boolean printComplexityStatistics)
```
    Calls toSummaryString() with a default title.
    
    Parameters:
    
    printComplexityStatistics - if true, complexity statistics are returned as well
    
    Returns:
    
    the summary string
  - toSummaryString
```
public java.lang.String toSummaryString(java.lang.String title,
                                        boolean printComplexityStatistics)
```
    Outputs the performance statistics in summary form. Lists number (and percentage) of instances classified correctly, incorrectly and unclassified. Outputs the total number of instances classified, and the number of instances (if any) that had no class value provided.
    
    Parameters:
    
    title - the title for the statistics
    
    printComplexityStatistics - if true, complexity statistics are returned as well
    
    Returns:
    
    the summary as a String
  - toMatrixString
```
public java.lang.String toMatrixString()
                                throws java.lang.Exception
```
    Calls toMatrixString() with a default title.
    
    Returns:
    
    the confusion matrix as a string
    
    Throws:
    
    java.lang.Exception - if the class is numeric
  - toMatrixString
```
public java.lang.String toMatrixString(java.lang.String title)
                                throws java.lang.Exception
```
    Outputs the performance statistics as a classification confusion matrix. For each class value, shows the distribution of predicted class values.
    
    Parameters:
    
    title - the title for the confusion matrix
    
    Returns:
    
    the confusion matrix as a String
    
    Throws:
    
    java.lang.Exception - if the class is numeric
  - toClassDetailsString
```
public java.lang.String toClassDetailsString()
                                      throws java.lang.Exception
```
    Generates a breakdown of the accuracy for each class (with default title), incorporating various information-retrieval statistics, such as true/false positive rate, precision/recall/F-Measure. Should be useful for ROC curves, recall/precision curves.
    
    Returns:
    
    the statistics presented as a string
    
    Throws:
    
    java.lang.Exception - if class is not nominal
  - toClassDetailsString
```
public java.lang.String toClassDetailsString(java.lang.String title)
                                      throws java.lang.Exception
```
    Generates a breakdown of the accuracy for each class, incorporating various information-retrieval statistics, such as true/false positive rate, precision/recall/F-Measure. Should be useful for ROC curves, recall/precision curves.
    
    Parameters:
    
    title - the title to prepend the stats string with
    
    Returns:
    
    the statistics presented as a string
    
    Throws:
    
    java.lang.Exception - if class is not nominal
  - numTruePositives
```
public double numTruePositives(int classIndex)
```
    Calculate the number of true positives with respect to a particular class. This is defined as
```
 correctly classified positives
 
```
    Parameters:
    
    classIndex - the index of the class to consider as "positive"
    
    Returns:
    
    the true positive rate
  - truePositiveRate
```
public double truePositiveRate(int classIndex)
```
    Calculate the true positive rate with respect to a particular class. This is defined as
```
 correctly classified positives
 ------------------------------
       total positives
 
```
    Parameters:
    
    classIndex - the index of the class to consider as "positive"
    
    Returns:
    
    the true positive rate
  - weightedTruePositiveRate
```
public double weightedTruePositiveRate()
```
    Calculates the weighted (by class size) true positive rate.
    
    Returns:
    
    the weighted true positive rate.
  - numTrueNegatives
```
public double numTrueNegatives(int classIndex)
```
    Calculate the number of true negatives with respect to a particular class. This is defined as
```
 correctly classified negatives
 
```
    Parameters:
    
    classIndex - the index of the class to consider as "positive"
    
    Returns:
    
    the true positive rate
  - trueNegativeRate
```
public double trueNegativeRate(int classIndex)
```
    Calculate the true negative rate with respect to a particular class. This is defined as
```
 correctly classified negatives
 ------------------------------
       total negatives
 
```
    Parameters:
    
    classIndex - the index of the class to consider as "positive"
    
    Returns:
    
    the true positive rate
  - weightedTrueNegativeRate
```
public double weightedTrueNegativeRate()
```
    Calculates the weighted (by class size) true negative rate.
    
    Returns:
    
    the weighted true negative rate.
  - numFalsePositives
```
public double numFalsePositives(int classIndex)
```
    Calculate number of false positives with respect to a particular class. This is defined as
```
 incorrectly classified negatives
 
```
    Parameters:
    
    classIndex - the index of the class to consider as "positive"
    
    Returns:
    
    the false positive rate
  - falsePositiveRate
```
public double falsePositiveRate(int classIndex)
```
    Calculate the false positive rate with respect to a particular class. This is defined as
```
 incorrectly classified negatives
 --------------------------------
        total negatives
 
```
    Parameters:
    
    classIndex - the index of the class to consider as "positive"
    
    Returns:
    
    the false positive rate
  - weightedFalsePositiveRate
```
public double weightedFalsePositiveRate()
```
    Calculates the weighted (by class size) false positive rate.
    
    Returns:
    
    the weighted false positive rate.
  - numFalseNegatives
```
public double numFalseNegatives(int classIndex)
```
    Calculate number of false negatives with respect to a particular class. This is defined as
```
 incorrectly classified positives
 
```
    Parameters:
    
    classIndex - the index of the class to consider as "positive"
    
    Returns:
    
    the false positive rate
  - falseNegativeRate
```
public double falseNegativeRate(int classIndex)
```
    Calculate the false negative rate with respect to a particular class. This is defined as
```
 incorrectly classified positives
 --------------------------------
        total positives
 
```
    Parameters:
    
    classIndex - the index of the class to consider as "positive"
    
    Returns:
    
    the false positive rate
  - weightedFalseNegativeRate
```
public double weightedFalseNegativeRate()
```
    Calculates the weighted (by class size) false negative rate.
    
    Returns:
    
    the weighted false negative rate.
  - matthewsCorrelationCoefficient
```
public double matthewsCorrelationCoefficient(int classIndex)
```
    Calculates the matthews correlation coefficient (sometimes called phi coefficient) for the supplied class
    
    Parameters:
    
    classIndex - the index of the class to compute the matthews correlation coefficient for
    
    Returns:
    
    the mathews correlation coefficient
  - weightedMatthewsCorrelation
```
public double weightedMatthewsCorrelation()
```
    Calculates the weighted (by class size) matthews correlation coefficient.
    
    Returns:
    
    the weighted matthews correlation coefficient.
  - recall
```
public double recall(int classIndex)
```
    Calculate the recall with respect to a particular class. This is defined as
```
 correctly classified positives
 ------------------------------
       total positives
 
```
    (Which is also the same as the truePositiveRate.)
    Parameters:
    
    classIndex - the index of the class to consider as "positive"
    
    Returns:
    
    the recall
  - weightedRecall
```
public double weightedRecall()
```
    Calculates the weighted (by class size) recall.
    
    Returns:
    
    the weighted recall.
  - precision
```
public double precision(int classIndex)
```
    Calculate the precision with respect to a particular class. This is defined as
```
 correctly classified positives
 ------------------------------
  total predicted as positive
 
```
    Parameters:
    
    classIndex - the index of the class to consider as "positive"
    
    Returns:
    
    the precision
  - weightedPrecision
```
public double weightedPrecision()
```
    Calculates the weighted (by class size) precision.
    
    Returns:
    
    the weighted precision.
  - fMeasure
```
public double fMeasure(int classIndex)
```
    Calculate the F-Measure with respect to a particular class. This is defined as
```
 2 * recall * precision
 ----------------------
   recall + precision
 
```
    Parameters:
    
    classIndex - the index of the class to consider as "positive"
    
    Returns:
    
    the F-Measure
  - weightedFMeasure
```
public double weightedFMeasure()
```
    Calculates the macro weighted (by class size) average F-Measure.
    
    Returns:
    
    the weighted F-Measure.
  - unweightedMacroFmeasure
```
public double unweightedMacroFmeasure()
```
    Unweighted macro-averaged F-measure. If some classes not present in the test set, they're just skipped (since recall is undefined there anyway) .
    
    Returns:
    
    unweighted macro-averaged F-measure.
  - unweightedMicroFmeasure
```
public double unweightedMicroFmeasure()
```
    Unweighted micro-averaged F-measure. If some classes not present in the test set, they have no effect. Note: if the test set is *single-label*, then this is the same as accuracy.
    
    Returns:
    
    unweighted micro-averaged F-measure.
  - setPriors
```
public void setPriors(Instances train)
               throws java.lang.Exception
```
    Sets the class prior probabilities.
    
    Parameters:
    
    train - the training instances used to determine the prior probabilities
    
    Throws:
    
    java.lang.Exception - if the class attribute of the instances is not set
  - getClassPriors
```
public double[] getClassPriors()
```
    Get the current weighted class counts.
    
    Returns:
    
    the weighted class counts
  - updatePriors
```
public void updatePriors(Instance instance)
                  throws java.lang.Exception
```
    Updates the class prior probabilities or the mean respectively (when incrementally training).
    
    Parameters:
    
    instance - the new training instance seen
    
    Throws:
    
    java.lang.Exception - if the class of the instance is not set
  - useNoPriors
```
public void useNoPriors()
```
    disables the use of priors, e.g., in case of de-serialized schemes that have no access to the original training set, but are evaluated on a set set.
  - equals
```
public boolean equals(java.lang.Object obj)
```
    Tests whether the current evaluation object is equal to another evaluation object.
    
    Overrides:
    
    equals in class java.lang.Object
    
    Parameters:
    
    obj - the object to compare against
    
    Returns:
    
    true if the two objects are equal
  - main
```
public static void main(java.lang.String[] args)
```
    A test method for this class. Just extracts the first command line argument as a classifier class name and calls evaluateModel.
    
    Parameters:
    
    args - an array of command line arguments, the first of which must be the class name of a classifier.

Class Evaluation

Field Summary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Field Detail

BUILT_IN_EVAL_METRICS

Constructor Detail

Evaluation

Evaluation

Method Detail

getAllEvaluationMetricNames

getHeader

getPluginMetrics

getPluginMetric

setMetricsToDisplay

getMetricsToDisplay

toggleEvalMetrics

setDiscardPredictions

getDiscardPredictions

areaUnderROC

weightedAreaUnderROC

areaUnderPRC

weightedAreaUnderPRC

confusionMatrix

crossValidateModel

crossValidateModel

crossValidateModel

evaluateModel

evaluateModel

evaluateModel

evaluationForSingleInstance

evaluateModelOnceAndRecordPrediction

evaluateModelOnce

evaluateModelOnce

evaluateModelOnceAndRecordPrediction

evaluateModelOnce

predictions

wekaStaticWrapper

numInstances

coverageOfTestCasesByPredictedRegions

sizeOfPredictedRegions

incorrect

pctIncorrect

totalCost

avgCost

correct

pctCorrect

unclassified

pctUnclassified

errorRate

kappa

getRevision

correlationCoefficient

meanAbsoluteError

meanPriorAbsoluteError

relativeAbsoluteError

rootMeanSquaredError

rootMeanPriorSquaredError

rootRelativeSquaredError

priorEntropy

KBInformation

KBMeanInformation

KBRelativeInformation

SFPriorEntropy

SFMeanPriorEntropy

SFSchemeEntropy

SFMeanSchemeEntropy

SFEntropyGain

SFMeanEntropyGain

toCumulativeMarginDistributionString

toSummaryString

toSummaryString

toSummaryString

toMatrixString

toMatrixString

toClassDetailsString

toClassDetailsString

numTruePositives

truePositiveRate