distributed.hadoop

Class MapReduceJobConfig

java.lang.Object

distributed.core.DistributedJobConfig

distributed.hadoop.AbstractHadoopJobConfig

distributed.hadoop.MapReduceJobConfig

All Implemented Interfaces:

java.io.Serializable, OptionHandler

public class MapReduceJobConfig
extends AbstractHadoopJobConfig
implements OptionHandler

The main job configuration used by Weka Hadoop jobs

Version:

$Revision: 12439 $

Author:

Mark Hall (mhall{[at]}pentaho{[dot]}com)

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	COMBINER_CLASS Internal key for the name of the combiner class
static java.lang.String	HADOOP_JOB_TRACKER_HOST Internal key for the Hadoop property for the job tracker host
static java.lang.String	HADOOP_MAPRED_MAX_SPLIT_SIZE Internal key for the Hadoop 1 property for the maximum block size
static java.lang.String	HADOOP_TASKTRACKER_REDUCE_TASKS_MAXIMUM Internal key for the Hadoop 1 property for the maximum number of number of reducers to run per node
static java.lang.String	HADOOP2_MAPRED_MAX_SPLIT_SIZE Internal key for the Haddop 2 property for the maximum block size
static java.lang.String	HADOOP2_TASKTRACKER_REDUCE_TASKS_MAXIMUM Internal key for the Hadoop 2 property for the maximum number of number of reducers to run per node
static java.lang.String	INPUT_FORMAT_CLASS Internal key for the name of the input format class to use
static java.lang.String	INPUT_PATHS Internal key for the input path(s) to use for the job
static java.lang.String	MAP_OUTPUT_KEY_CLASS Internal key for the name of the map output key class to use
static java.lang.String	MAP_OUTPUT_VALUE_CLASS Internal key for the name of the map output value class to use
static java.lang.String	MAPPER_CLASS Internal key for the name of the mapper class
static java.lang.String	MAPRED_MAX_SPLIT_SIZE Internal key for the maximum block size (for splitting data) to use
static java.lang.String	NUM_MAPPERS Internal key for the number of mappers to use
static java.lang.String	NUM_REDUCERS Internal key for the number of reducers to use
static java.lang.String	OUTPUT_FORMAT_CLASS Internal key for the name of the output format class to use
static java.lang.String	OUTPUT_KEY_CLASS Internal key for the name of the (job/reducer) output key to use
static java.lang.String	OUTPUT_PATH Internal key for the output path to use for the job
static java.lang.String	OUTPUT_VALUE_CLASS Internal key for the name of the (job/reducer) output value to use
static java.lang.String	REDUCER_CLASS Internal key for the name of the reducer class
static java.lang.String	TASK_TRACKER_MAP_MAXIMUM Internal key for the maximum number of mappers that will run on a node concurrently
static java.lang.String	YARN_RESOURCE_MANAGER_ADDRESS Internal key for the Hadoop property for the yarn resource manager address
static java.lang.String	YARN_RESOURCE_MANAGER_SCHEDULER_ADDRESS Internal key for the Hadoop property for the yarn resource manager scheduler address.

Fields inherited from class distributed.hadoop.AbstractHadoopJobConfig

DEFAULT_HOST, DEFAULT_PORT, DEFAULT_PORT_YARN, JOBTRACKER_HOST, JOBTRACKER_PORT

Constructor Summary

Constructors

Constructor and Description
MapReduceJobConfig() Constructor - sets defaults

Method Summary

Modifier and Type	Method and Description
org.apache.hadoop.mapreduce.Job	configureForHadoop(java.lang.String jobName, org.apache.hadoop.conf.Configuration conf, Environment env) Apply the settings encapsulated in this config and return a Job object ready for execution.
void	deleteOutputDirectory(org.apache.hadoop.mapreduce.Job job, Environment env) Clean the output directory specified for the supplied job
java.lang.String	getCombinerClass() Get the name of the reducer class (if any) to use.
HDFSConfig	getHDFSConfig() Get the HDFSConfig to use
java.lang.String	getHDFSHost() Get the HDFS host (name node)
java.lang.String	getHDFSPort() Get the HDFS port
java.lang.String	getInputFormatClass() Get the name of the input format class to use.
java.lang.String	getInputPaths() Get the input path(s) to use
java.lang.String	getMapOutputKeyClass() Get the name of the map output key class to use.
java.lang.String	getMapOutputValueClass() Get the name of the map output value class to use.
java.lang.String	getMapperClass() Get the mapper class name to use.
java.lang.String	getMapredMaxSplitSize() Get the maximum split size (in bytes).
java.lang.String	getNumberOfMaps() Get the number of maps to use.
java.lang.String	getNumberOfReducers() Get the number of reducers to use.
java.lang.String[]	getOptions()
java.lang.String	getOutputFormatClass() Get the name of the output format class to use.
java.lang.String	getOutputKeyClass() Get the name of the (reducer) output key class to use
java.lang.String	getOutputPath() Get the output path to use
java.lang.String	getOutputValueClass() Get the name of the (reducer) output value class to use
java.lang.String	getReducerClass() Get the name of the reducer class to use.
java.lang.String	getTaskTrackerMapTasksMaximum() Get the maximum number of map tasks to run concurrently by a task tracker (node).
java.lang.String	HDFSHostTipText() Get the tool tip text for this property
java.lang.String	HDFSPortTipText() Get the tool tip text for this property
java.lang.String	inputPathsTipText() Get the tip text for this property
java.util.Enumeration<Option>	listOptions()
java.lang.String	numberOfMapsTipText() Get the tool tip text for this property
java.lang.String	numberOfReducersTipText() Get the tool tip text for this property
java.lang.String	outputPathTipText() Get the tip text for this property
void	setCombinerClass(java.lang.String combinerClass) Set the name of the reducer class (if any) to use.
void	setHDFSConfig(HDFSConfig config) Set the HDFSConfig to use
void	setHDFSHost(java.lang.String host) Set the HDFSHost (name node)
void	setHDFSPort(java.lang.String port) Set the HDFS port
void	setInputFormatClass(java.lang.String inputFormatClass) Set the name of the input format class to use.
void	setInputPaths(java.lang.String inputPaths) Set the input path(s) to use
void	setMapOutputKeyClass(java.lang.String mapOutputKeyClass) Set the name of the map output key class to use.
void	setMapOutputValueClass(java.lang.String mapOutputValueClass) Set the name of the map output value class to use.
void	setMapperClass(java.lang.String mapperClass) Set the mapper class name to use.
void	setMapredMaxSplitSize(java.lang.String maxSize) Set the maximum split size (in bytes).
void	setNumberOfMaps(java.lang.String nM) Set the number of maps to use.
void	setNumberOfReducers(java.lang.String nR) Set the number of reducers to use.
void	setOptions(java.lang.String[] options)
void	setOutputFormatClass(java.lang.String outputFormatClass) Set the name of the output format class to use.
void	setOutputKeyClass(java.lang.String outputKeyClass) Set the name of the (reducer) output key class to use
void	setOutputPath(java.lang.String outputPath) Set the output path to use
void	setOutputValueClass(java.lang.String outputValueClass) Set the name of the (reducer) output value class to use
void	setReducerClass(java.lang.String reducerClass) Set the name of the reducer class to use.
void	setTaskTrackerMapTasksMaximum(java.lang.String mmt) Set the maximum number of map tasks to run concurrently by a task tracker (node).
java.lang.String	taskTrackerMapTasksMaximumTipText() Get the tool tip text for this property

Methods inherited from class distributed.hadoop.AbstractHadoopJobConfig

getJobTrackerHost, getJobTrackerPort, isHadoop2, jobTrackerHostTipText, jobTrackerPortTipText, setJobTrackerHost, setJobTrackerPort

Methods inherited from class distributed.core.DistributedJobConfig

clearUserSuppliedProperties, getProperty, getPropertyNames, getUserSuppliedProperties, getUserSuppliedProperty, getUserSuppliedPropertyNames, isEmpty, setProperty, setUserSuppliedProperty

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

NUM_MAPPERS

public static final java.lang.String NUM_MAPPERS

Internal key for the number of mappers to use

See Also:

Constant Field Values

NUM_REDUCERS

public static final java.lang.String NUM_REDUCERS

Internal key for the number of reducers to use

See Also:

Constant Field Values

TASK_TRACKER_MAP_MAXIMUM

public static final java.lang.String TASK_TRACKER_MAP_MAXIMUM

Internal key for the maximum number of mappers that will run on a node concurrently

See Also:

Constant Field Values

MAPPER_CLASS

public static final java.lang.String MAPPER_CLASS

Internal key for the name of the mapper class

See Also:

Constant Field Values

REDUCER_CLASS

public static final java.lang.String REDUCER_CLASS

Internal key for the name of the reducer class

See Also:

Constant Field Values

COMBINER_CLASS

public static final java.lang.String COMBINER_CLASS

Internal key for the name of the combiner class

See Also:

Constant Field Values

INPUT_FORMAT_CLASS

public static final java.lang.String INPUT_FORMAT_CLASS

Internal key for the name of the input format class to use

See Also:

Constant Field Values

OUTPUT_FORMAT_CLASS

public static final java.lang.String OUTPUT_FORMAT_CLASS

Internal key for the name of the output format class to use

See Also:

Constant Field Values

MAP_OUTPUT_KEY_CLASS

public static final java.lang.String MAP_OUTPUT_KEY_CLASS

Internal key for the name of the map output key class to use

See Also:

Constant Field Values

MAP_OUTPUT_VALUE_CLASS

public static final java.lang.String MAP_OUTPUT_VALUE_CLASS

Internal key for the name of the map output value class to use

See Also:

Constant Field Values

OUTPUT_KEY_CLASS

public static final java.lang.String OUTPUT_KEY_CLASS

Internal key for the name of the (job/reducer) output key to use

See Also:

Constant Field Values

OUTPUT_VALUE_CLASS

public static final java.lang.String OUTPUT_VALUE_CLASS

Internal key for the name of the (job/reducer) output value to use

See Also:

Constant Field Values

INPUT_PATHS

public static final java.lang.String INPUT_PATHS

Internal key for the input path(s) to use for the job

See Also:

Constant Field Values

OUTPUT_PATH

public static final java.lang.String OUTPUT_PATH

Internal key for the output path to use for the job

See Also:

Constant Field Values

MAPRED_MAX_SPLIT_SIZE

public static final java.lang.String MAPRED_MAX_SPLIT_SIZE

Internal key for the maximum block size (for splitting data) to use

See Also:

Constant Field Values

HADOOP_JOB_TRACKER_HOST

public static final java.lang.String HADOOP_JOB_TRACKER_HOST

Internal key for the Hadoop property for the job tracker host

See Also:

Constant Field Values

YARN_RESOURCE_MANAGER_ADDRESS

public static final java.lang.String YARN_RESOURCE_MANAGER_ADDRESS

Internal key for the Hadoop property for the yarn resource manager address

See Also:

Constant Field Values

YARN_RESOURCE_MANAGER_SCHEDULER_ADDRESS

public static final java.lang.String YARN_RESOURCE_MANAGER_SCHEDULER_ADDRESS

Internal key for the Hadoop property for the yarn resource manager scheduler address. Weka will use yarn.resource.manager.address:8030 to set this. If this is not appropriate for a particular cluster it can be overridden using -user-prop arguments to jobs, or by placing the cluster configuration directory in the classpath when running jobs.

See Also:

Constant Field Values

HADOOP_MAPRED_MAX_SPLIT_SIZE

public static final java.lang.String HADOOP_MAPRED_MAX_SPLIT_SIZE

Internal key for the Hadoop 1 property for the maximum block size

See Also:

Constant Field Values

HADOOP2_MAPRED_MAX_SPLIT_SIZE

public static final java.lang.String HADOOP2_MAPRED_MAX_SPLIT_SIZE

Internal key for the Haddop 2 property for the maximum block size

See Also:

Constant Field Values

HADOOP_TASKTRACKER_REDUCE_TASKS_MAXIMUM

public static final java.lang.String HADOOP_TASKTRACKER_REDUCE_TASKS_MAXIMUM

Internal key for the Hadoop 1 property for the maximum number of number of reducers to run per node

See Also:

Constant Field Values

HADOOP2_TASKTRACKER_REDUCE_TASKS_MAXIMUM

public static final java.lang.String HADOOP2_TASKTRACKER_REDUCE_TASKS_MAXIMUM

Internal key for the Hadoop 2 property for the maximum number of number of reducers to run per node

See Also:

Constant Field Values

Constructor Detail

MapReduceJobConfig

public MapReduceJobConfig()

Constructor - sets defaults

Method Detail

listOptions

public java.util.Enumeration<Option> listOptions()

Specified by:

listOptions in interface OptionHandler

Overrides:

listOptions in class distributed.core.DistributedJobConfig

setOptions

public void setOptions(java.lang.String[] options)
                throws java.lang.Exception

Specified by:

setOptions in interface OptionHandler

Overrides:

setOptions in class distributed.core.DistributedJobConfig

Throws:

java.lang.Exception

getOptions

public java.lang.String[] getOptions()

Specified by:

getOptions in interface OptionHandler

Overrides:

getOptions in class distributed.core.DistributedJobConfig

setHDFSConfig

public void setHDFSConfig(HDFSConfig config)

Set the HDFSConfig to use

Parameters:

config - the HDFSConfig to use

getHDFSConfig

public HDFSConfig getHDFSConfig()

Get the HDFSConfig to use

Returns:

the HDFSConfig to use

HDFSHostTipText

public java.lang.String HDFSHostTipText()

Get the tool tip text for this property

Returns:

the tool tip text for this property

setHDFSHost

public void setHDFSHost(java.lang.String host)

Set the HDFSHost (name node)

Parameters:

host - the HDFS host

getHDFSHost

public java.lang.String getHDFSHost()

Get the HDFS host (name node)

Returns:

the HDFS host

HDFSPortTipText

public java.lang.String HDFSPortTipText()

Get the tool tip text for this property

Returns:

the tool tip text for this property

setHDFSPort

public void setHDFSPort(java.lang.String port)

Set the HDFS port

Parameters:

port - the HDFS port

getHDFSPort

public java.lang.String getHDFSPort()

Get the HDFS port

Returns:

the HDFS port

numberOfMapsTipText

public java.lang.String numberOfMapsTipText()

Get the tool tip text for this property

Returns:

the tool tip text for this property

setNumberOfMaps

public void setNumberOfMaps(java.lang.String nM)

Set the number of maps to use. This is just a hint to the underlying Hadoop framework for how many maps to use. Using setMapredMaxSplitSize(), which sets the Hadoop property mapred.max.split.size, gives greater control over how many maps will be run (and thus how much data each map processes).

Parameters:

nM - the number of maps to use

getNumberOfMaps

public java.lang.String getNumberOfMaps()

Get the number of maps to use. This is just a hint to the underlying Hadoop framework for how many maps to use. Using setMapredMaxSplitSize(), which sets the Hadoop property mapred.max.split.size, gives greater control over how many maps will be run (and thus how much data each map processes).

Returns:

the number of maps to use

taskTrackerMapTasksMaximumTipText

public java.lang.String taskTrackerMapTasksMaximumTipText()

Get the tool tip text for this property

Returns:

the tool tip text for this property

setTaskTrackerMapTasksMaximum

public void setTaskTrackerMapTasksMaximum(java.lang.String mmt)

Set the maximum number of map tasks to run concurrently by a task tracker (node). The cluster setting for this will be used if not specified here

Parameters:

mmt - the maximum number of map tasks to run concurrently by a task tracker

getTaskTrackerMapTasksMaximum

public java.lang.String getTaskTrackerMapTasksMaximum()

Get the maximum number of map tasks to run concurrently by a task tracker (node). The cluster setting for this will be used if not specified here

Returns:

the maximum number of map tasks to run concurrently by a task tracker

numberOfReducersTipText

public java.lang.String numberOfReducersTipText()

Get the tool tip text for this property

Returns:

the tool tip text for this property

setNumberOfReducers

public void setNumberOfReducers(java.lang.String nR)

Set the number of reducers to use. Weka jobs set this property automatically

Parameters:

nR - the number of reducers to use.

getNumberOfReducers

public java.lang.String getNumberOfReducers()

Get the number of reducers to use. Weka jobs set this property automatically

Returns:

the number of reducers to use.

setMapperClass

public void setMapperClass(java.lang.String mapperClass)

Set the mapper class name to use. Weka jobs configure this automatically.

Parameters:

mapperClass - the mapper class name

getMapperClass

public java.lang.String getMapperClass()

Get the mapper class name to use. Weka jobs configure this automatically.

Returns:

the mapper class name

setReducerClass

public void setReducerClass(java.lang.String reducerClass)

Set the name of the reducer class to use. Weka jobs set this automatically.

Parameters:

reducerClass - the name of the reducer class

getReducerClass

public java.lang.String getReducerClass()

Get the name of the reducer class to use. Weka jobs set this automatically.

Returns:

the name of the reducer class

setCombinerClass

public void setCombinerClass(java.lang.String combinerClass)

Set the name of the reducer class (if any) to use. Weka jobs may set this automatically

Parameters:

combinerClass - the name of the combiner class to use

getCombinerClass

public java.lang.String getCombinerClass()

Get the name of the reducer class (if any) to use. Weka jobs may set this automatically

Returns:

the name of the combiner class to use

setInputFormatClass

public void setInputFormatClass(java.lang.String inputFormatClass)

Set the name of the input format class to use. Weka jobs set this automatically.

Parameters:

inputFormatClass - the name of the input format class to use

getInputFormatClass

public java.lang.String getInputFormatClass()

Get the name of the input format class to use. Weka jobs set this automatically.

Returns:

the name of the input format class to use

setOutputFormatClass

public void setOutputFormatClass(java.lang.String outputFormatClass)

Set the name of the output format class to use. Weka jobs set this automatically.

Parameters:

outputFormatClass - the name of the output format class to use.

getOutputFormatClass

public java.lang.String getOutputFormatClass()

Get the name of the output format class to use. Weka jobs set this automatically.

Returns:

the name of the output format class to use.

setMapOutputKeyClass

public void setMapOutputKeyClass(java.lang.String mapOutputKeyClass)

Set the name of the map output key class to use. Weka jobs set this automatically.

Parameters:

mapOutputKeyClass - the name of the map output key class

getMapOutputKeyClass

public java.lang.String getMapOutputKeyClass()

Get the name of the map output key class to use. Weka jobs set this automatically.

Returns:

the name of the map output key class

setMapOutputValueClass

public void setMapOutputValueClass(java.lang.String mapOutputValueClass)

Set the name of the map output value class to use. Weka jobs set this automatically.

Parameters:

mapOutputValueClass - the name of the map output value class

getMapOutputValueClass

public java.lang.String getMapOutputValueClass()

Get the name of the map output value class to use. Weka jobs set this automatically.

Returns:

the name of the map output value class

setOutputKeyClass

public void setOutputKeyClass(java.lang.String outputKeyClass)

Set the name of the (reducer) output key class to use

Parameters:

outputKeyClass - the name of the output key class to use

getOutputKeyClass

public java.lang.String getOutputKeyClass()

Get the name of the (reducer) output key class to use

Returns:

the name of the output key class to use

setOutputValueClass

public void setOutputValueClass(java.lang.String outputValueClass)

Set the name of the (reducer) output value class to use

Parameters:

outputValueClass - the name of the output value class to use

getOutputValueClass

public java.lang.String getOutputValueClass()

Get the name of the (reducer) output value class to use

Returns:

the name of the output value class to use

inputPathsTipText

public java.lang.String inputPathsTipText()

Get the tip text for this property

Returns:

the tip text for this property

setInputPaths

public void setInputPaths(java.lang.String inputPaths)

Set the input path(s) to use

Parameters:

inputPaths - the input paths to use

getInputPaths

public java.lang.String getInputPaths()

Get the input path(s) to use

Returns:

the input paths to use

outputPathTipText

public java.lang.String outputPathTipText()

Get the tip text for this property

Returns:

the tip text for this property

setOutputPath

public void setOutputPath(java.lang.String outputPath)

Set the output path to use

Parameters:

outputPath - the output path to use

getOutputPath

public java.lang.String getOutputPath()

Get the output path to use

Returns:

the output path to use

setMapredMaxSplitSize

public void setMapredMaxSplitSize(java.lang.String maxSize)

Set the maximum split size (in bytes). This can be used to control the number of maps that run. The default block size of 64Mb may be too large for some batch learning tasks depending on data characteristics, choice of learning algorithm and available RAM on the node.

Parameters:

maxSize - the maximum split size (in bytes)

getMapredMaxSplitSize

public java.lang.String getMapredMaxSplitSize()

Get the maximum split size (in bytes). This can be used to control the number of maps that run. The default block size of 64Mb may be too large for some batch learning tasks depending on data characteristics, choice of learning algorithm and available RAM on the node.

Returns:

the maximum split size (in bytes)

configureForHadoop

public org.apache.hadoop.mapreduce.Job configureForHadoop(java.lang.String jobName,
                                                          org.apache.hadoop.conf.Configuration conf,
                                                          Environment env)
                                                   throws java.io.IOException,
                                                          java.lang.ClassNotFoundException

Apply the settings encapsulated in this config and return a Job object ready for execution.

Parameters:

jobName - the name of the job

conf - the Configuration object that will be wrapped in the Job

env - environment variables

Returns:

a configured Job object

Throws:

java.io.IOException - if a problem occurs

java.lang.ClassNotFoundException - if various classes are not found

deleteOutputDirectory

public void deleteOutputDirectory(org.apache.hadoop.mapreduce.Job job,
                                  Environment env)
                           throws java.io.IOException

Clean the output directory specified for the supplied job

Parameters:

job - the job to clean the output directory for

env - environment variables

Throws:

java.io.IOException - if a problem occurs