weka.classifiers.timeseries.eval

Class TSEvaluation

java.lang.Object

weka.classifiers.timeseries.eval.TSEvaluation

public class TSEvaluation
extends Object

Main evaluation routines for time series forecasting models. Also provides a command line interface for building and evaluating time series models.

Version:

$Revision: 52593 $

Author:

Mark Hall (mhall{[at]}pentaho{[dot]}com)

Constructor Summary

Constructors

Constructor and Description
TSEvaluation(weka.core.Instances trainingData, double testSplitSize) Constructor.
TSEvaluation(weka.core.Instances trainingData, weka.core.Instances testData) Constructor.

Method Summary

Methods

Modifier and Type	Method and Description
void	evaluateForecaster(TSForecaster forecaster, boolean buildModel, PrintStream... progress) Evaluate a forecaster on training and/or test data.
void	evaluateForecaster(TSForecaster forecaster, PrintStream... progress) Evaluate the supplied forecaster.
static void	evaluateForecaster(TSForecaster forecaster, String[] options) Evaluate the supplied forecaster using the supplied command-line options.
boolean	getEvaluateOnTestData() Get whether evaluation is to be performed on the test data.
boolean	getEvaluateOnTrainingData() Get whether evaluation is to be performed on the training data
List<TSEvalModule>	getEvaluationModules() Get the evaluation modules in use
boolean	getForecastFuture() Get whether future forecasts beyond the end of the training and/or test data will be generated.
ErrorModule	getPredictionsForTestData(int stepNumber) Get predictions for all targets for the specified step number on the test data
ErrorModule	getPredictionsForTrainingData(int stepNumber) Get predictions for all targets for the specified step number on the training data
boolean	getPrimeForTestDataWithTestData() Gets whether evaluation for the test data will begin by priming with the first x instances from the test data and then forecasting from step x + 1.
int	getPrimeWindowSize() Get the size of the priming window - i.e.
weka.core.Instances	getTestData() Get the test data (if any)
weka.core.Instances	getTrainingData() Get the training data (if any)
JPanel	graphFutureForecastOnTesting(GraphDriver driver, TSForecaster forecaster, List<String> targetNames) Graph historical and future forecasted values using a graph driver for the test.
JPanel	graphFutureForecastOnTraining(GraphDriver driver, TSForecaster forecaster, List<String> targetNames) Graph historical and future forecasted values using a graph driver for the training data.
JPanel	graphPredictionsForStepsOnTesting(GraphDriver driver, TSForecaster forecaster, String targetName, List<Integer> stepsToPlot, int instanceNumberOffset) Graph predicted values at the given step-ahead levels for a single target on the test data.
JPanel	graphPredictionsForStepsOnTraining(GraphDriver driver, TSForecaster forecaster, String targetName, List<Integer> stepsToPlot, int instanceNumberOffset) Graph predicted values at the given step-ahead levels for a single target on the training data.
JPanel	graphPredictionsForTargetsOnTesting(GraphDriver driver, TSForecaster forecaster, List<String> graphTargets, int graphStepNum, int primeWindowSize) Graph predicted values at the given step-ahead level for the supplied targets on the test data.
JPanel	graphPredictionsForTargetsOnTraining(GraphDriver driver, TSForecaster forecaster, List<String> graphTargets, int graphStepNum, int instanceNumberOffset) Graph predicted values at the given step-ahead level for the supplied targets on the training data.
String	printFutureTestForecast(TSForecaster forecaster) Print the target values from the test data followed by the future forecast from the end of the test data.
String	printFutureTrainingForecast(TSForecaster forecaster) Print the forecasted values (for all targets) beyond the end of the training data
String	printPredictionsForTestData(String title, String targetName, int stepAhead) Print the predictions for a given target at a given step-ahead level on the training data.
String	printPredictionsForTestData(String title, String targetName, int stepAhead, int instanceNumberOffset) Print the predictions for a given target at a given step-ahead level from a given offset on the training data.
String	printPredictionsForTrainingData(String title, String targetName, int stepAhead) Print the predictions for a given target at a given step-ahead level on the training data.
String	printPredictionsForTrainingData(String title, String targetName, int stepAhead, int instanceNumberOffset) Print the predictions for a given target at a given step-ahead level from a given offset on the training data.
void	setEvaluateOnTestData(boolean evalOnTest) Set whether to perform evaluation on the training data
void	setEvaluateOnTrainingData(boolean evalOnTraining) Set whether to perform evaluation on the training data
void	setEvaluationModules(String evalModNames) Set the evaluation modules to use/
void	setForecastFuture(boolean future) Set whether we should generate a future forecast beyond the end of the training and/or test data.
void	setHorizon(int horizon) Set the horizon - i.e.
void	setPrimeForTestDataWithTestData(boolean p) Set whether evaluation for test data should begin by priming with the first x test data instances and then forecasting from step x + 1.
void	setPrimeWindowSize(int primeSize) Set the size of the priming window - i.e.
void	setRebuildModelAfterEachTestForecastStep(boolean r) Set whether the forecasting model should be rebuilt after each forecasting step on the test data using both the training data and test data up to the current instance.
void	setTestData(weka.core.Instances testData) Set the test data to use
void	setTrainingData(weka.core.Instances train) Set the training data to use
String	toSummaryString() Generates a String containing the results of evaluating the forecaster.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TSEvaluation

public TSEvaluation(weka.core.Instances trainingData,
            double testSplitSize)
             throws Exception

Constructor.

Parameters:

trainingData - the training data

testSplitSize - the number or percentage of instances to hold out from the end of the training data to be test data.

Throws:

Exception - if a problem occurs.

TSEvaluation

public TSEvaluation(weka.core.Instances trainingData,
            weka.core.Instances testData)
             throws Exception

Constructor.

Parameters:

trainingData - the training data to use

testData - the test data to use

Throws:

Exception - if a problem occurs.

Method Detail

setTrainingData

public void setTrainingData(weka.core.Instances train)

Set the training data to use

Parameters:

train - the training data to use

setTestData

public void setTestData(weka.core.Instances testData)

Set the test data to use

Parameters:

testData - the test data to use

getTrainingData

public weka.core.Instances getTrainingData()

Get the training data (if any)

Returns:

the training data or null if none is in use

getTestData

public weka.core.Instances getTestData()

Get the test data (if any)

Returns:

the test data or null if none is in use.

setEvaluateOnTrainingData

public void setEvaluateOnTrainingData(boolean evalOnTraining)

Set whether to perform evaluation on the training data

Parameters:

evalOnTraining - true if evaluation is to be performed on the training data

getEvaluateOnTrainingData

public boolean getEvaluateOnTrainingData()

Get whether evaluation is to be performed on the training data

Returns:

true if evaluation is to be performed on the training data.

setEvaluateOnTestData

public void setEvaluateOnTestData(boolean evalOnTest)

Set whether to perform evaluation on the training data

Parameters:

evalOnTest - true if evalution is to be performed on the training data

getEvaluateOnTestData

public boolean getEvaluateOnTestData()

Get whether evaluation is to be performed on the test data.

Returns:

true if evaluation is to be performed on the test data.

setHorizon

public void setHorizon(int horizon)

Set the horizon - i.e. the number of steps to forecast into the future.

Parameters:

horizon - the number of steps to forecast into the future

setPrimeWindowSize

public void setPrimeWindowSize(int primeSize)

Set the size of the priming window - i.e. the number of historical instances to be presented to the forecaster before a forecast is requested

Parameters:

primeSize - the number of instances to prime with

getPrimeWindowSize

public int getPrimeWindowSize()

Get the size of the priming window - i.e. the number of historical instances that will be presented to the forecaster before a forecast is requested.

Returns:

the size of the priming window.

setPrimeForTestDataWithTestData

public void setPrimeForTestDataWithTestData(boolean p)

Set whether evaluation for test data should begin by priming with the first x test data instances and then forecasting from step x + 1. This is the only option if there is no training data and a model has been deserialized from disk. If we have training data, and it occurs immediately before the test data in time, then we can prime with the last x instances from the training data.

Parameters:

p - true if we should start evaluation of the test data by priming with the first instances from the test data.

getPrimeForTestDataWithTestData

public boolean getPrimeForTestDataWithTestData()

Gets whether evaluation for the test data will begin by priming with the first x instances from the test data and then forecasting from step x + 1. This is the only option if there is no training data and a model has been deserialized from disk. If we have training data, and it occurs immediately before the test data in time, then we can prime with the last x instances from the training data.

Returns:

true if evaluation of the test data will begin by priming with the first x instances from the test data.

setRebuildModelAfterEachTestForecastStep

public void setRebuildModelAfterEachTestForecastStep(boolean r)

Set whether the forecasting model should be rebuilt after each forecasting step on the test data using both the training data and test data up to the current instance.

Parameters:

r - true if the forecasting model should be rebuilt after each forecasting step on the test data to take into account all data up to the current point in time

setForecastFuture

public void setForecastFuture(boolean future)

Set whether we should generate a future forecast beyond the end of the training and/or test data.

Parameters:

future - true if future forecasts beyond the end of training/test data should be generated.

getForecastFuture

public boolean getForecastFuture()

Get whether future forecasts beyond the end of the training and/or test data will be generated.

Returns:

true if future forecasts will be generated.

setEvaluationModules

public void setEvaluationModules(String evalModNames)
                          throws Exception

Set the evaluation modules to use/

Parameters:

evalModNames - a comma-separated list of evaluation module names.

Throws:

Exception - if there are unknown evaluation modules requested.

getEvaluationModules

public List<TSEvalModule> getEvaluationModules()

Get the evaluation modules in use

Returns:

a list of the evaluation modules in use.

getPredictionsForTrainingData

public ErrorModule getPredictionsForTrainingData(int stepNumber)
                                          throws Exception

Get predictions for all targets for the specified step number on the training data

Parameters:

stepNumber - number of the step into the future to return predictions for

Returns:

the stepNumber step ahead predictions for all targets

Throws:

Exception - if there are no predictions available for the training data

getPredictionsForTestData

public ErrorModule getPredictionsForTestData(int stepNumber)
                                      throws Exception

Get predictions for all targets for the specified step number on the test data

Parameters:

stepNumber - number of the step into the future to return predictions for

Returns:

the stepNumber step ahead predictions for all targets

Throws:

Exception - if there are no predictions available for the test data

evaluateForecaster

public void evaluateForecaster(TSForecaster forecaster,
                      PrintStream... progress)
                        throws Exception

Evaluate the supplied forecaster. Trains the forecaster if a training set has been configured.

Parameters:

forecaster - the forecaster to evaluate

Throws:

Exception - if something goes wrong during evaluation

evaluateForecaster

public void evaluateForecaster(TSForecaster forecaster,
                      boolean buildModel,
                      PrintStream... progress)
                        throws Exception

Evaluate a forecaster on training and/or test data.

Parameters:

forecaster - the forecaster to evaluate

buildModel - true if the model is to be built (given that there is a training data set to build it with)

Throws:

Exception - if something goes wrong during evaluation

evaluateForecaster

public static void evaluateForecaster(TSForecaster forecaster,
                      String[] options)
                               throws Exception

Evaluate the supplied forecaster using the supplied command-line options.

Parameters:

forecaster - the forecaster to evaluate

options - an array of command-line options

Throws:

Exception - if a problem occurs during evaluation

printFutureTestForecast

public String printFutureTestForecast(TSForecaster forecaster)
                               throws Exception

Print the target values from the test data followed by the future forecast from the end of the test data.

Parameters:

forecaster - the forecaster in use

Returns:

a String containing the test target values and the future forecasted values beyond the end of the test data

Throws:

Exception - if something goes wrong.

printFutureTrainingForecast

public String printFutureTrainingForecast(TSForecaster forecaster)
                                   throws Exception

Print the forecasted values (for all targets) beyond the end of the training data

Parameters:

forecaster - the forecaster

Returns:

A string which contains the target values from the training data followed by predictions for all targets for 1 - horizon number of steps into the future.

Throws:

Exception

graphFutureForecastOnTraining

public JPanel graphFutureForecastOnTraining(GraphDriver driver,
                                   TSForecaster forecaster,
                                   List<String> targetNames)
                                     throws Exception

Graph historical and future forecasted values using a graph driver for the training data.

Parameters:

driver - the graph driver to graph with

forecaster - the forecaster in use

targetNames - the names of the targets to graph

Returns:

a JPanel containing the graph

Throws:

Exception - if a problem occurs during graphing

graphFutureForecastOnTesting

public JPanel graphFutureForecastOnTesting(GraphDriver driver,
                                  TSForecaster forecaster,
                                  List<String> targetNames)
                                    throws Exception

Graph historical and future forecasted values using a graph driver for the test.

Parameters:

driver - the graph driver to graph with

forecaster - the forecaster in use

targetNames - the names of the targets to graph

Returns:

a JPanel containing the graph

Throws:

Exception - if a problem occurs during graphing

graphPredictionsForStepsOnTraining

public JPanel graphPredictionsForStepsOnTraining(GraphDriver driver,
                                        TSForecaster forecaster,
                                        String targetName,
                                        List<Integer> stepsToPlot,
                                        int instanceNumberOffset)
                                          throws Exception

Graph predicted values at the given step-ahead levels for a single target on the training data.

Parameters:

driver - the graph driver to graph with

forecaster - the forecaster in use

targetName - the target to plot

stepsToPlot - a list of steps at which to plot the predicted target values - e.g. 3,4,5 would plot 3-step-ahead, 4-step-ahead and 5-step-ahead predictions

Returns:

a JPanel containing the graph

Throws:

Exception - if a problem occurs during graphing

graphPredictionsForStepsOnTesting

public JPanel graphPredictionsForStepsOnTesting(GraphDriver driver,
                                       TSForecaster forecaster,
                                       String targetName,
                                       List<Integer> stepsToPlot,
                                       int instanceNumberOffset)
                                         throws Exception

Graph predicted values at the given step-ahead levels for a single target on the test data.

Parameters:

driver - the graph driver to graph with

forecaster - the forecaster in use

targetName - the target to plot

stepsToPlot - a list of steps at which to plot the predicted target values - e.g. 3,4,5 would plot 3-step-ahead, 4-step-ahead and 5-step-ahead predictions

Returns:

a JPanel containing the graph

Throws:

Exception - if a problem occurs during graphing

graphPredictionsForTargetsOnTraining

public JPanel graphPredictionsForTargetsOnTraining(GraphDriver driver,
                                          TSForecaster forecaster,
                                          List<String> graphTargets,
                                          int graphStepNum,
                                          int instanceNumberOffset)
                                            throws Exception

Graph predicted values at the given step-ahead level for the supplied targets on the training data.

Parameters:

driver - the graph driver to graph with

forecaster - the forecaster in use

graphTargets - a list of targets to plot

graphStepNum - the step at which to plot the targets - e.g. 4 would result in a graph of the 4-step-ahead predictions for the targets

Returns:

a JPanel containing the graph

Throws:

Exception - if a problem occurs during graphing

graphPredictionsForTargetsOnTesting

public JPanel graphPredictionsForTargetsOnTesting(GraphDriver driver,
                                         TSForecaster forecaster,
                                         List<String> graphTargets,
                                         int graphStepNum,
                                         int primeWindowSize)
                                           throws Exception

Graph predicted values at the given step-ahead level for the supplied targets on the test data.

Parameters:

driver - the graph driver to graph with

forecaster - the forecaster in use

graphTargets - a list of targets to plot

graphStepNum - the step at which to plot the targets - e.g. 4 would result in a graph of the 4-step-ahead predictions for the targets

Returns:

a JPanel containing the graph

Throws:

Exception - if a problem occurs during graphing

printPredictionsForTrainingData

public String printPredictionsForTrainingData(String title,
                                     String targetName,
                                     int stepAhead)
                                       throws Exception

Print the predictions for a given target at a given step-ahead level on the training data.

Parameters:

title - a title for the output

targetName - the name of the target to print predictions for

stepAhead - the step-ahead level - e.g. 3 would print the 3-step-ahead predictions

Returns:

a String containing the predicted and actual values

Throws:

Exception - if the predictions can't be printed for some reason.

printPredictionsForTrainingData

public String printPredictionsForTrainingData(String title,
                                     String targetName,
                                     int stepAhead,
                                     int instanceNumberOffset)
                                       throws Exception

Print the predictions for a given target at a given step-ahead level from a given offset on the training data.

Parameters:

title - a title for the output

targetName - the name of the target to print predictions for

stepAhead - the step-ahead level - e.g. 3 would print the 3-step-ahead predictions

instanceNumberOffset - the offset from the start of the training data from which to print actual and predicted values

Returns:

a String containing the predicted and actual values

Throws:

Exception - if the predictions can't be printed for some reason.

printPredictionsForTestData

public String printPredictionsForTestData(String title,
                                 String targetName,
                                 int stepAhead)
                                   throws Exception

Print the predictions for a given target at a given step-ahead level on the training data.

Parameters:

title - a title for the output

targetName - the name of the target to print predictions for

stepAhead - the step-ahead level - e.g. 3 would print the 3-step-ahead predictions

Returns:

a String containing the predicted and actual values

Throws:

Exception - if the predictions can't be printed for some reason.

printPredictionsForTestData

public String printPredictionsForTestData(String title,
                                 String targetName,
                                 int stepAhead,
                                 int instanceNumberOffset)
                                   throws Exception

Print the predictions for a given target at a given step-ahead level from a given offset on the training data.

Parameters:

title - a title for the output

targetName - the name of the target to print predictions for

stepAhead - the step-ahead level - e.g. 3 would print the 3-step-ahead predictions

instanceNumberOffset - the offset from the start of the training data from which to print actual and predicted values

Returns:

a String containing the predicted and actual values

Throws:

Exception - if the predictions can't be printed for some reason.

toSummaryString

public String toSummaryString()
                       throws Exception

Generates a String containing the results of evaluating the forecaster.

Returns:

a String containing the results of evaluating the forecaster

Throws:

Exception - if a problem occurs