weka.distributed.spark

Class SparkUtils

java.lang.Object

weka.distributed.spark.SparkUtils

public class SparkUtils
extends java.lang.Object

Some utility routines pertaining to files

Version:

$Revision: $

Author:

Mark Hall (mhall{[at]}pentaho{[dot]}com)

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	SparkUtils.WekaLoggingPrintWriter Used for hooking into Spark's log4j logging via a WriterAppender

Constructor Summary

Constructors

Constructor and Description
SparkUtils()

Method Summary

Modifier and Type	Method and Description
static java.lang.String	addSubdirToPath(java.lang.String parent, java.lang.String subdirName) Adds a subdirectory to a parent path.
static boolean	checkFileExists(java.lang.String file) Check that the named file exists on either the local file system or HDFS.
static void	deleteDirectory(java.lang.String path) Delete a directory (and all contents).
static org.apache.hadoop.conf.Configuration	getFSConfigurationForPath(java.lang.String path, java.lang.String[] pathOnly) Returns a Configuration object configured with the name node and port present in the supplied path (hdfs://host:port/path).
static long	getSizeInBytesOfPath(java.lang.String path) Get the size in bytes of a file/directory
static org.apache.log4j.WriterAppender	initSparkLogAppender(distributed.core.DistributedJob job) Initialize and return an appender for hooking into Spark's log4j logging and directing it to Weka's log
static java.io.InputStream	openFileForRead(java.lang.String file) Opens the named file for reading on either the local file system or HDFS.
static java.io.OutputStream	openFileForWrite(java.lang.String file) Open the named file for writing to on either the local file system or HDFS.
static java.io.PrintWriter	openTextFileForWrite(java.lang.String file) Open the named file as a text file for writing to on either the local file system or any other protocol specific file system supported by Hadoop.
static void	removeSparkLogAppender(org.apache.log4j.WriterAppender appender) Remove the supplied appender from Spark's logging.
static java.lang.String	resolveLocalOrOtherFileSystemPath(java.lang.String original) Takes an input path and returns a fully qualified absolute one.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SparkUtils

public SparkUtils()

Method Detail

addSubdirToPath

public static java.lang.String addSubdirToPath(java.lang.String parent,
                                               java.lang.String subdirName)

Adds a subdirectory to a parent path. Handles local and hdfs:// files

Parameters:

parent - the parent (may include the hdfs://host:port part

subdirName - the name of the subdirectory to add

Returns:

the parent path with the sudirectory appended.

getFSConfigurationForPath

public static org.apache.hadoop.conf.Configuration getFSConfigurationForPath(java.lang.String path,
                                                                             java.lang.String[] pathOnly)

Returns a Configuration object configured with the name node and port present in the supplied path (hdfs://host:port/path). Also returns the path-only part of the URI. Note that absolute paths will require an extra /. E.g. hdfs://host:port//users/fred/input. Also handles local files system paths if no protocol is supplied - e.g. bob/george for a relative path (relative to the current working directory) or /bob/george for an absolute path.

Parameters:

path - the URI or local path from which to configure

pathOnly - will hold the path-only part of the URI

Returns:

a Configuration object

resolveLocalOrOtherFileSystemPath

public static java.lang.String resolveLocalOrOtherFileSystemPath(java.lang.String original)
                                                          throws java.io.IOException

Takes an input path and returns a fully qualified absolute one. Handles local and non-local file systems. Original path can be a relative one. In the case of a local file system, the relative path (relative to the working directory) is made into an absolute one. In the case of a file system such as HDFS, an absolute path will require an additional '/' - E.g. hdfs://host:port//users/fred/input - otherwise it will be treated as relative (to the user's home directory in HDFS). In either case, the returned path will be an absolute one.

Parameters:

original - original path (either relative or absolute) on a file system

Returns:

absolute path

Throws:

java.io.IOException - if a problem occurs

deleteDirectory

public static void deleteDirectory(java.lang.String path)
                            throws java.io.IOException

Delete a directory (and all contents).

Parameters:

path - the path to the directory to delete

Throws:

java.io.IOException - if the path is not a directory or a problem occurs

openFileForRead

public static java.io.InputStream openFileForRead(java.lang.String file)
                                           throws java.io.IOException

Opens the named file for reading on either the local file system or HDFS. HDFS files should use the form "hdfs://host:port/<path>"

Parameters:

file - the file to open for reading on either the local or HDFS file system

Returns:

an InputStream for the file

Throws:

java.io.IOException - if a problem occurs

openFileForWrite

public static java.io.OutputStream openFileForWrite(java.lang.String file)
                                             throws java.io.IOException

Open the named file for writing to on either the local file system or HDFS. HDFS files should use the form "hdfs://host:port/<path>". Note that, on the local file system, the directory path must exist. Under HDFS, the path is created automatically.

Parameters:

file - the file to write to

Returns:

an OutputStream for writing to the file

Throws:

java.io.IOException - if a problem occurs

openTextFileForWrite

public static java.io.PrintWriter openTextFileForWrite(java.lang.String file)
                                                throws java.io.IOException

Open the named file as a text file for writing to on either the local file system or any other protocol specific file system supported by Hadoop. Protocol files should use the form "protocol://host:port/<path>." Note that, on the local file system, the directory path must exist.

Parameters:

file - the file to write to

Returns:

an PrintWriter for writing to the file

Throws:

java.io.IOException - if a problem occurs

checkFileExists

public static boolean checkFileExists(java.lang.String file)
                               throws java.io.IOException

Check that the named file exists on either the local file system or HDFS.

Parameters:

file - the file to check

Returns:

true if the file exists on the local file system or in HDFS

Throws:

java.io.IOException - if a problem occurs

getSizeInBytesOfPath

public static long getSizeInBytesOfPath(java.lang.String path)
                                 throws java.io.IOException

Get the size in bytes of a file/directory

Parameters:

path - the path to the file/directory

Returns:

the size in bytes

Throws:

java.io.IOException - if a problem occurs

initSparkLogAppender

public static org.apache.log4j.WriterAppender initSparkLogAppender(distributed.core.DistributedJob job)

Initialize and return an appender for hooking into Spark's log4j logging and directing it to Weka's log

Parameters:

job - the job to initialize the appender for

Returns:

an appender

removeSparkLogAppender

public static void removeSparkLogAppender(org.apache.log4j.WriterAppender appender)

Remove the supplied appender from Spark's logging. This should be called at the end of a job

Parameters:

appender - the appender to remove