weka.distributed.hadoop

Class HadoopJob

java.lang.Object

distributed.core.DistributedJob

weka.distributed.hadoop.HadoopJob

All Implemented Interfaces:

java.io.Serializable, CommandlineRunnable, EnvironmentHandler, OptionHandler

Direct Known Subclasses:

ArffHeaderHadoopJob, CorrelationMatrixHadoopJob, KMeansClustererHadoopJob, RandomizedDataChunkHadoopJob, WekaClassifierEvaluationHadoopJob, WekaClassifierHadoopJob, WekaScoringHadoopJob

public abstract class HadoopJob
extends distributed.core.DistributedJob
implements OptionHandler

Abstract base class for Hadoop jobs. Contains routines for installing Weka libraries in HDFS, running jobs and getting status information on running jobs.

Version:

$Revision: 13236 $

Author:

Mark Hall (mhall{[at]}pentaho{[dot]}com)

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class distributed.core.DistributedJob

distributed.core.DistributedJob.JobStatus

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	COLT_JAR The path to the colt.jar
static java.lang.String	DISTRIBUTED_WEKA_BASE_JAR The path to the distributedWekaBase.jar
static java.lang.String	DISTRIBUTED_WEKA_HADOOP_JAR The path to the distributedWekaHadoop.jar
static java.lang.String	JCOMMON_JAR The path to the jcommon jar
static java.lang.String	JFREECHART_JAR The path to the jfreechart jar
static java.lang.String	LA4J_JAR The path to the la4j.jar
static java.lang.String	OPEN_CSV_JAR The path to the opencsv.jar
static java.lang.String	TDIGEST_JAR The path to the t-digest.jar

Fields inherited from class distributed.core.DistributedJob

WEKA_ADDITIONAL_PACKAGES_KEY

Constructor Summary

Constructors

Constructor and Description
HadoopJob(java.lang.String jobName, java.lang.String jobDescription) Constructor for a HadoopJob

Method Summary

Modifier and Type	Method and Description
java.lang.String	additionalWekaPackagesTipText() Tip text for this property.
void	cleanOutputDirectory(org.apache.hadoop.mapreduce.Job job) Deletes the output directory for a job
java.lang.String	deubgTipText() Tip text for this property
java.lang.String	getAdditionalWekaPackages() Get a comma separated list of the names of additional weka packages to use with the job.
java.lang.String[]	getBaseOptionsOnly() Return the base options only (not the subclasses options or the options specific to the configuration)
boolean	getDebug() Get whether to output debug info.
java.lang.String	getLoggingInterval() Get the interval between output of logging information from running jobs.
static int	getMapNumber(java.lang.String taskID) Get the number of the map attempt from the supplied task ID string
MapReduceJobConfig	getMapReduceJobConfig() Get the main configuration to use with this job
static int	getMapReduceNumber(java.lang.String taskID, java.lang.String prefix) Extract the number of a map/reduce attempt from the supplied taskID string.
java.lang.String[]	getOptions()
java.lang.String	getPathToWekaJar() Get the path to the weka.jar file.
static int	getReduceNumber(java.lang.String taskID) Get the number of the reduce attempt from the supplied task ID string
java.util.Enumeration<Option>	listOptions()
java.lang.String	loggingIntervalTipText() Tip text for this property
void	logMessage(java.lang.String message) Log a message
java.lang.String	pathToWekaJarTipText() Tip text for this property
void	setAdditionalWekaPackages(java.lang.String packages) Set a comma separated list of the names of additional weka packages to use with the job.
void	setDebug(boolean debug) Set whether to output debug info.
void	setLoggingInterval(java.lang.String li) Set the interval between output of logging information from running jobs.
void	setMapReduceJobConfig(MapReduceJobConfig conf) Set the main configuration to use with this job
void	setOptions(java.lang.String[] options)
void	setPathToWekaJar(java.lang.String path) Set the path to the weka.jar file.

Methods inherited from class distributed.core.DistributedJob

environmentSubstitute, getAdditionalWekaPackageNames, getJobName, getJobStatus, getLog, logMessage, logMessage, makeOptionsStr, objectRowToInstance, parseInstance, postExecution, preExecution, run, runJob, setEnvironment, setJobDescription, setJobName, setJobStatus, setLog, setStatusMessagePrefix, stackTraceToString, statusMessage, stopJob

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DISTRIBUTED_WEKA_HADOOP_JAR

public static final java.lang.String DISTRIBUTED_WEKA_HADOOP_JAR

The path to the distributedWekaHadoop.jar

DISTRIBUTED_WEKA_BASE_JAR

public static final java.lang.String DISTRIBUTED_WEKA_BASE_JAR

The path to the distributedWekaBase.jar

OPEN_CSV_JAR

public static final java.lang.String OPEN_CSV_JAR

The path to the opencsv.jar

JFREECHART_JAR

public static final java.lang.String JFREECHART_JAR

The path to the jfreechart jar

JCOMMON_JAR

public static final java.lang.String JCOMMON_JAR

The path to the jcommon jar

COLT_JAR

public static final java.lang.String COLT_JAR

The path to the colt.jar

LA4J_JAR

public static final java.lang.String LA4J_JAR

The path to the la4j.jar

TDIGEST_JAR

public static final java.lang.String TDIGEST_JAR

The path to the t-digest.jar

Constructor Detail

HadoopJob

public HadoopJob(java.lang.String jobName,
                 java.lang.String jobDescription)

Constructor for a HadoopJob

Parameters:

jobName - the name of the job

jobDescription - a short description of the job

Method Detail

getMapReduceNumber

public static int getMapReduceNumber(java.lang.String taskID,
                                     java.lang.String prefix)

Extract the number of a map/reduce attempt from the supplied taskID string.

Parameters:

taskID - the taskID string

prefix - the prefix identifying the type of task (i.e. mapper or reducer)

Returns:

the task number

getMapNumber

public static int getMapNumber(java.lang.String taskID)

Get the number of the map attempt from the supplied task ID string

Parameters:

taskID - the task ID string

Returns:

the number of the map attempt

getReduceNumber

public static int getReduceNumber(java.lang.String taskID)

Get the number of the reduce attempt from the supplied task ID string

Parameters:

taskID - the task ID string

Returns:

the number of the reduce attempt

listOptions

public java.util.Enumeration<Option> listOptions()

Specified by:

listOptions in interface OptionHandler

getBaseOptionsOnly

public java.lang.String[] getBaseOptionsOnly()

Return the base options only (not the subclasses options or the options specific to the configuration)

Returns:

just the base options

getOptions

public java.lang.String[] getOptions()

Specified by:

getOptions in interface OptionHandler

setOptions

public void setOptions(java.lang.String[] options)
                throws java.lang.Exception

Specified by:

setOptions in interface OptionHandler

Throws:

java.lang.Exception

getMapReduceJobConfig

public MapReduceJobConfig getMapReduceJobConfig()

Get the main configuration to use with this job

Returns:

the main configuration to use with this job

setMapReduceJobConfig

public void setMapReduceJobConfig(MapReduceJobConfig conf)

Set the main configuration to use with this job

Parameters:

conf - the main configuration to use with this job

deubgTipText

public java.lang.String deubgTipText()

Tip text for this property

Returns:

the tip text for this property

getDebug

public boolean getDebug()

Get whether to output debug info. Some jobs may output more info to the log if this is turned on

Returns:

true if debug info is to be output

setDebug

public void setDebug(boolean debug)

Set whether to output debug info. Some jobs may output more info to the log if this is turned on

Parameters:

debug - true if debug info is to be output

pathToWekaJarTipText

public java.lang.String pathToWekaJarTipText()

Tip text for this property

Returns:

the tip text for this property

getPathToWekaJar

public java.lang.String getPathToWekaJar()

Get the path to the weka.jar file. Will be populated automatically if the classpath contains a weka.jar. The weka.jar is installed in HDFS and used in the classpath for map and reduce tasks.

Returns:

the path to the weka.jar.

setPathToWekaJar

public void setPathToWekaJar(java.lang.String path)

Set the path to the weka.jar file. Will be populated automatically if the classpath contains a weka.jar. The weka.jar is installed in HDFS and used in the classpath for map and reduce tasks.

Parameters:

path - the path to the weka.jar.

additionalWekaPackagesTipText

public java.lang.String additionalWekaPackagesTipText()

Tip text for this property.

Returns:

the tip text for this property.

getAdditionalWekaPackages

public java.lang.String getAdditionalWekaPackages()

Get a comma separated list of the names of additional weka packages to use with the job. Any jar files in the main package directory and the lib directory of the package will get installed in HDFS and placed in the classpath of map and reduce tasks

Returns:

a comma separated list of weka packages to use with the job

setAdditionalWekaPackages

public void setAdditionalWekaPackages(java.lang.String packages)

Set a comma separated list of the names of additional weka packages to use with the job. Any jar files in the main package directory and the lib directory of the package will get installed in HDFS and placed in the classpath of map and reduce tasks

Parameters:

packages - a comma separated list of weka packages to use with the job

loggingIntervalTipText

public java.lang.String loggingIntervalTipText()

Tip text for this property

Returns:

tip text for this property

getLoggingInterval

public java.lang.String getLoggingInterval()

Get the interval between output of logging information from running jobs.

Returns:

the interval (in seconds) between output of logging information

setLoggingInterval

public void setLoggingInterval(java.lang.String li)

Set the interval between output of logging information from running jobs.

Parameters:

li - the interval (in seconds) between output of logging information

cleanOutputDirectory

public void cleanOutputDirectory(org.apache.hadoop.mapreduce.Job job)
                          throws java.io.IOException

Deletes the output directory for a job

Parameters:

job - the Job object to delete the output directory for

Throws:

java.io.IOException - if a problem occurs

logMessage

public void logMessage(java.lang.String message)

Log a message

Overrides:

logMessage in class distributed.core.DistributedJob

Parameters:

message - the message to log