WekaClassifierEvaluationHadoopJob

java.lang.Object
- distributed.core.DistributedJob
- - weka.distributed.hadoop.HadoopJob
  - - weka.distributed.hadoop.WekaClassifierEvaluationHadoopJob

All Implemented Interfaces:

java.io.Serializable, CommandlineRunnable, EnvironmentHandler, OptionHandler, InstancesProducer, TextProducer
```
public class WekaClassifierEvaluationHadoopJob
extends HadoopJob
implements TextProducer, InstancesProducer, CommandlineRunnable
```
Hadoop job for running an evaluation of a classifier or regressor. Invokes up to four separate jobs (passes over the data). 1) Header creation, 2) optional randomly shuffled data chunk creation, 3) model construction and 4) model evaluation. Can perform evaluation on the training data, on a separate test set, or via cross-validation. In the case of the later, models for all folds are created and aggregated in one pass (job) and then evaluated on all folds in a second job.

Version:

$Revision: 11847 $

Author:

Mark Hall (mhall{[at]}pentaho{[dot]}com)

See Also:

Serialized Form

Nested Class Summary
- Nested classes/interfaces inherited from class distributed.core.DistributedJob
  distributed.core.DistributedJob.JobStatus

Field Summary
- Fields inherited from class weka.distributed.hadoop.HadoopJob
  COLT_JAR, DISTRIBUTED_WEKA_BASE_JAR, DISTRIBUTED_WEKA_HADOOP_JAR, JCOMMON_JAR, JFREECHART_JAR, LA4J_JAR, OPEN_CSV_JAR, TDIGEST_JAR
- Fields inherited from class distributed.core.DistributedJob
  WEKA_ADDITIONAL_PACKAGES_KEY

Constructor Summary

Constructors
Constructor and Description

WekaClassifierEvaluationHadoopJob()
Constructor

Constructors
Constructor and Description
`WekaClassifierEvaluationHadoopJob()` Constructor

Method Summary

All Methods Static Methods Instance Methods Concrete Methods
Modifier and Type	Method and Description
`Instances`	`getInstances()`
`java.lang.String[]`	`getJobOptionsOnly()` Get the options pertaining to this job only
`int`	`getNumNodesInCluster()` Get the number of nodes in the user's Hadoop cluster.
`java.lang.String[]`	`getOptions()`
`java.lang.String`	`getSampleFractionForAUC()` Get the percentage of predictions to retain (via uniform random sampling) for computing AUC and AUPRC.
`java.lang.String`	`getSeparateTestSetPath()` Get the path in HDFS to the separate test set to use.
`java.lang.String`	`getText()`
`java.lang.String`	`globalInfo()` Help info for this job
`java.util.Enumeration<Option>`	`listOptions()`
`static void`	`main(java.lang.String[] args)`
`java.lang.String`	`numNodesInClusterTipText()` Tip text for this property
`void`	`run(java.lang.Object toRun, java.lang.String[] args)`
`boolean`	`runJob()`
`java.lang.String`	`sampleFractionForAUCTipText()` Tip text for this property
`java.lang.String`	`separateTestSetPathTipText()` Tip text for this property
`void`	`setNumNodesInCluster(int n)` Set the number of nodes in the user's Hadoop cluster.
`void`	`setOptions(java.lang.String[] options)`
`void`	`setSampleFractionForAUC(java.lang.String f)` Set the percentage of predictions to retain (via uniform random sampling) for computing AUC and AUPRC.
`void`	`setSeparateTestSetPath(java.lang.String path)` Set the path in HDFS to the separate test set to use.

Methods inherited from class weka.distributed.hadoop.HadoopJob
additionalWekaPackagesTipText, cleanOutputDirectory, deubgTipText, getAdditionalWekaPackages, getBaseOptionsOnly, getDebug, getLoggingInterval, getMapNumber, getMapReduceJobConfig, getMapReduceNumber, getPathToWekaJar, getReduceNumber, loggingIntervalTipText, logMessage, pathToWekaJarTipText, setAdditionalWekaPackages, setDebug, setLoggingInterval, setMapReduceJobConfig, setPathToWekaJar

Methods inherited from class distributed.core.DistributedJob
environmentSubstitute, getAdditionalWekaPackageNames, getJobName, getJobStatus, getLog, logMessage, logMessage, makeOptionsStr, objectRowToInstance, parseInstance, postExecution, preExecution, setEnvironment, setJobDescription, setJobName, setJobStatus, setLog, setStatusMessagePrefix, stackTraceToString, statusMessage, stopJob

Methods inherited from class java.lang.Object
equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface weka.core.CommandlineRunnable
postExecution, preExecution

- Constructor Detail
  - WekaClassifierEvaluationHadoopJob
```
public WekaClassifierEvaluationHadoopJob()
```
    Constructor
- Method Detail
  - main
```
public static void main(java.lang.String[] args)
```
  - globalInfo
```
public java.lang.String globalInfo()
```
    Help info for this job
    
    Returns:
    
    help info for this job
  - listOptions
```
public java.util.Enumeration<Option> listOptions()
```
    Specified by:
    
    listOptions in interface OptionHandler
    
    Overrides:
    
    listOptions in class HadoopJob
  - getOptions
```
public java.lang.String[] getOptions()
```
    Specified by:
    
    getOptions in interface OptionHandler
    
    Overrides:
    
    getOptions in class HadoopJob
  - setOptions
```
public void setOptions(java.lang.String[] options)
                throws java.lang.Exception
```
    Specified by:
    
    setOptions in interface OptionHandler
    
    Overrides:
    
    setOptions in class HadoopJob
    
    Throws:
    
    java.lang.Exception
  - getJobOptionsOnly
```
public java.lang.String[] getJobOptionsOnly()
```
    Get the options pertaining to this job only
    
    Returns:
    
    the options for this job only
  - separateTestSetPathTipText
```
public java.lang.String separateTestSetPathTipText()
```
    Tip text for this property
    
    Returns:
    
    the tip text for this property
  - getSeparateTestSetPath
```
public java.lang.String getSeparateTestSetPath()
```
    Get the path in HDFS to the separate test set to use. Either this or the total number of folds should be specified (but not both).
    
    Returns:
    
    the path in HDFS to the separate test set to evaluate on
  - setSeparateTestSetPath
```
public void setSeparateTestSetPath(java.lang.String path)
```
    Set the path in HDFS to the separate test set to use. Either this or the total number of folds should be specified (but not both).
    
    Parameters:
    
    path - the path in HDFS to the separate test set to evaluate on
  - sampleFractionForAUCTipText
```
public java.lang.String sampleFractionForAUCTipText()
```
    Tip text for this property
    
    Returns:
    
    the tip text for this property
  - getSampleFractionForAUC
```
public java.lang.String getSampleFractionForAUC()
```
    Get the percentage of predictions to retain (via uniform random sampling) for computing AUC and AUPRC. If not specified, then no predictions are retained and these metrics are not computed.
    
    Returns:
    
    the fraction (between 0 and 1) of all predictions to retain for computing AUC/AUPRC.
  - setSampleFractionForAUC
```
public void setSampleFractionForAUC(java.lang.String f)
```
    Set the percentage of predictions to retain (via uniform random sampling) for computing AUC and AUPRC. If not specified, then no predictions are retained and these metrics are not computed.
    
    Parameters:
    
    f - the fraction (between 0 and 1) of all predictions to retain for computing AUC/AUPRC.
  - numNodesInClusterTipText
```
public java.lang.String numNodesInClusterTipText()
```
    Tip text for this property
    
    Returns:
    
    the tip text for this property
  - getNumNodesInCluster
```
public int getNumNodesInCluster()
```
    Get the number of nodes in the user's Hadoop cluster. Used for setting the number of reducers to use when performing a cross-validation.
    
    Returns:
    
    the number of nodes in the cluster
  - setNumNodesInCluster
```
public void setNumNodesInCluster(int n)
```
    Set the number of nodes in the user's Hadoop cluster. Used for setting the number of reducers to use when performing a cross-validation.
    
    Parameters:
    
    n - the number of nodes in the cluster
  - runJob
```
public boolean runJob()
               throws weka.distributed.DistributedWekaException
```
    Specified by:
    
    runJob in class distributed.core.DistributedJob
    
    Throws:
    
    weka.distributed.DistributedWekaException
  - getText
```
public java.lang.String getText()
```
    Specified by:
    
    getText in interface TextProducer
  - getInstances
```
public Instances getInstances()
```
    Specified by:
    
    getInstances in interface InstancesProducer
  - run
```
public void run(java.lang.Object toRun,
                java.lang.String[] args)
         throws java.lang.IllegalArgumentException
```
    Specified by:
    
    run in interface CommandlineRunnable
    
    Overrides:
    
    run in class distributed.core.DistributedJob
    
    Throws:
    
    java.lang.IllegalArgumentException

Class WekaClassifierEvaluationHadoopJob

Nested Class Summary

Nested classes/interfaces inherited from class distributed.core.DistributedJob

Field Summary

Fields inherited from class weka.distributed.hadoop.HadoopJob

Fields inherited from class distributed.core.DistributedJob

Constructor Summary

Method Summary

Methods inherited from class weka.distributed.hadoop.HadoopJob

Methods inherited from class distributed.core.DistributedJob

Methods inherited from class java.lang.Object

Methods inherited from interface weka.core.CommandlineRunnable

Constructor Detail

WekaClassifierEvaluationHadoopJob

Method Detail

main

globalInfo

listOptions

getOptions

setOptions

getJobOptionsOnly

separateTestSetPathTipText

getSeparateTestSetPath

setSeparateTestSetPath

sampleFractionForAUCTipText

getSampleFractionForAUC

setSampleFractionForAUC

numNodesInClusterTipText

getNumNodesInCluster

setNumNodesInCluster

runJob

getText

getInstances

run