distributed.spark

Class SparkJobConfig

java.lang.Object

distributed.core.DistributedJobConfig

distributed.spark.BaseSparkJobConfig

distributed.spark.SparkJobConfig

All Implemented Interfaces:

java.io.Serializable, OptionHandler

public class SparkJobConfig
extends BaseSparkJobConfig

Basic connection options and settings for a batch Spark job.

Author:

Mark Hall (mhall{[at]}pentaho{[dot]}com)

See Also:

Serialized Form

Field Summary

Fields inherited from class distributed.spark.BaseSparkJobConfig

COLT_JAR, DEFAULT_HDFS_PORT, DEFAULT_MESOS_MASTER_PORT, DEFAULT_SPARK_MASTER_PORT, DISTRIBUTED_WEKA_BASE_JAR, DISTRIBUTED_WEKA_SPARK_JAR, HADOOP_FS_DEFAULT_NAME, JCOMMON_JAR, JFREECHART_JAR, LA4J_JAR, MASTER_HOST, MASTER_PORT, OPEN_CSV_JAR, SPARK_HOME_DIR, TDIGEST_JAR

Constructor Summary

Constructors

Constructor and Description
SparkJobConfig()

Method Summary

Modifier and Type	Method and Description
java.lang.String	getInputFile() Get the path to the file to process by this job.
java.lang.String	getMaxInputSlices() Get the maximum number of input splits (partitions) to create from the input file.
java.lang.String	getMinInputSlices() Get the minimum number of input splits to create from the input file.
java.lang.String[]	getOptions()
java.lang.String	getOutputDir() Get the output directory for this job
java.lang.String	inputFileTipText() Tip text for this property
java.util.Enumeration<Option>	listOptions()
java.lang.String	maxInputSlicesTipText()
java.lang.String	minInputSlicesTipText() Tip text for this property
java.lang.String	outputDirTipText() Tip text for this property
void	setInputFile(java.lang.String filePath) Set the path to the file to process by this job.
void	setMaxInputSlices(java.lang.String max) Set the maximum number of input splits (partitions) to create from the input file.
void	setMinInputSlices(java.lang.String slices) Set the minimum number of input splits (partitions) to create from the input file.
void	setOptions(java.lang.String[] options)
void	setOutputDir(java.lang.String dir) Set the output directory for this job

Methods inherited from class distributed.spark.BaseSparkJobConfig

addWekaLibrariesToSparkContext, availableClusterMemoryTipText, getAvailableClusterMemory, getBaseSparkContext, getDefaultPortForMaster, getInMemoryDataOverheadFactor, getMasterHost, getMasterPort, getMemoryFraction, getPathToWekaJar, getSparkHomeDirectory, getWekaPackages, inMemoryDataOverheadFactorTipText, masterHostTipText, masterPortTipText, memoryFractionTipText, pathToWekaJarTipText, setAvailableClusterMemory, setInMemoryDataOverheadFactor, setMasterHost, setMasterPort, setMemoryFraction, setPathToWekaJar, setSparkHomeDirectory, setWekaPackages, sparkHomeDirectoryTipText, wekaPackagesTipText

Methods inherited from class distributed.core.DistributedJobConfig

clearUserSuppliedProperties, getProperty, getPropertyNames, getUserSuppliedProperties, getUserSuppliedProperty, getUserSuppliedPropertyNames, isEmpty, setProperty, setUserSuppliedProperty

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SparkJobConfig

public SparkJobConfig()

Method Detail

listOptions

public java.util.Enumeration<Option> listOptions()

Specified by:

listOptions in interface OptionHandler

Overrides:

listOptions in class BaseSparkJobConfig

getOptions

public java.lang.String[] getOptions()

Specified by:

getOptions in interface OptionHandler

Overrides:

getOptions in class BaseSparkJobConfig

setOptions

public void setOptions(java.lang.String[] options)
                throws java.lang.Exception

Specified by:

setOptions in interface OptionHandler

Overrides:

setOptions in class BaseSparkJobConfig

Throws:

java.lang.Exception

inputFileTipText

public java.lang.String inputFileTipText()

Tip text for this property

Returns:

the tip text for this property

getInputFile

public java.lang.String getInputFile()

Get the path to the file to process by this job. This can be on the local file system or in HDFS. Use hdfs:// to specify a file in HDFS.

Returns:

the path to the file to process by the job

setInputFile

public void setInputFile(java.lang.String filePath)

Set the path to the file to process by this job. This can be on the local file system or in HDFS. Use hdfs:// to specify a file in HDFS.

Parameters:

filePath - the path to the file to process by the job

minInputSlicesTipText

public java.lang.String minInputSlicesTipText()

Tip text for this property

Returns:

the tip text for this property

getMinInputSlices

public java.lang.String getMinInputSlices()

Get the minimum number of input splits to create from the input file. By default there is 1 split per 64Mb block (HDFS block size), so there can't be fewer splits than there are blocks in the file. However, there can be more splits.

Returns:

the number of splits to create

setMinInputSlices

public void setMinInputSlices(java.lang.String slices)

Set the minimum number of input splits (partitions) to create from the input file. By default there is 1 split per 64Mb block (HDFS block size), so there can't be fewer splits than there are blocks in the file. However, there can be more splits.

Parameters:

slices - the number of splits to create

maxInputSlicesTipText

public java.lang.String maxInputSlicesTipText()

Returns:

the tip text for this property

getMaxInputSlices

public java.lang.String getMaxInputSlices()

Get the maximum number of input splits (partitions) to create from the input file. If this is fewer than the natural number of partitions (i.e. as determined by the HDFS block size) then this will trigger a coalesce.

Returns:

the maximum number of splits/partitions to create from an input file.

setMaxInputSlices

public void setMaxInputSlices(java.lang.String max)

Set the maximum number of input splits (partitions) to create from the input file. If this is fewer than the natural number of partitions (i.e. as determined by the HDFS block size) then this will trigger a coalesce.

Parameters:

max - the maximum number of splits/partitions to create from an input file.

outputDirTipText

public java.lang.String outputDirTipText()

Tip text for this property

Returns:

the tip text for this property

getOutputDir

public java.lang.String getOutputDir()

Get the output directory for this job

Returns:

the output directory for this job

setOutputDir

public void setOutputDir(java.lang.String dir)

Set the output directory for this job

Parameters:

dir - the output directory for this job