weka.distributed.spark

Class SparkJob.NoKeyTextOutputFormat<K,V>

java.lang.Object

org.apache.hadoop.mapreduce.OutputFormat<K,V>

org.apache.hadoop.mapreduce.lib.output.FileOutputFormat<K,V>

org.apache.hadoop.mapreduce.lib.output.TextOutputFormat<K,V>

weka.distributed.spark.SparkJob.NoKeyTextOutputFormat<K,V>

Type Parameters:

K - Key type

V - Value type

Enclosing class:

SparkJob

public static class SparkJob.NoKeyTextOutputFormat<K,V>
extends org.apache.hadoop.mapreduce.lib.output.TextOutputFormat<K,V>

Subclass of TextOutputFormat that does not write the key. We need this as PairRDDFunctions.saveAsNewAPIHadoopFile() does not wrap its key/values in org.apache.hadoop.io.* classes, so specifying a NullWritable for the key does not suppress the key.

Author:

Mark Hall (mhall{[at]}pentaho{[dot]}com)

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.mapreduce.lib.output.FileOutputFormat

org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.Counter

Constructor Summary

Constructors

Constructor and Description
NoKeyTextOutputFormat()

Method Summary

Modifier and Type	Method and Description
org.apache.hadoop.mapreduce.RecordWriter<K,V>	getRecordWriter(org.apache.hadoop.mapreduce.TaskAttemptContext job)

Methods inherited from class org.apache.hadoop.mapreduce.lib.output.FileOutputFormat

checkOutputSpecs, getCompressOutput, getDefaultWorkFile, getOutputCommitter, getOutputCompressorClass, getOutputPath, getPathForWorkFile, getUniqueFile, getWorkOutputPath, setCompressOutput, setOutputCompressorClass, setOutputPath

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

NoKeyTextOutputFormat

public NoKeyTextOutputFormat()

Method Detail

getRecordWriter

public org.apache.hadoop.mapreduce.RecordWriter<K,V> getRecordWriter(org.apache.hadoop.mapreduce.TaskAttemptContext job)
                                                              throws java.io.IOException,
                                                                     java.lang.InterruptedException

Overrides:

getRecordWriter in class org.apache.hadoop.mapreduce.lib.output.TextOutputFormat<K,V>

Throws:

java.io.IOException

java.lang.InterruptedException