distributed.hadoop

Class HDFSUtils

java.lang.Object

distributed.hadoop.HDFSUtils

public class HDFSUtils
extends java.lang.Object

Bunch of utility routines for HDFS. Includes stuff for moving/copying files in and out of HDFS, installing Weka libraries in HDFS, adding stuff to the distributed cache for a job and populating the classpath for map/reduce tasks.

Version:

$Revision: 11837 $

Author:

Mark Hall (mhall{[at]}pentaho{[dot]}com)

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	WEKA_LIBRARIES_LOCATION The default location in HDFS to place weka.jar and other libraries for inclusion in the classpath of the hadoop nodes
static java.lang.String	WEKA_TEMP_DISTRIBUTED_CACHE_FILES Staging location for non library files to be distributed to the nodes by the distributed cache
static java.lang.String	WINDOWS_ACCESSING_HADOOP_ON_LINUX_SYS_PROP Users need to set HADOOP_ON_LINUX environment variable to "true" if accessing a *nix Hadoop cluster from Windows so that we can post-process the job classpath in the Configuration file to use ':' rather than ';' as separators.

Constructor Summary

Constructors

Constructor and Description
HDFSUtils()

Method Summary

Modifier and Type	Method and Description
static void	addFilesClasspath(HDFSConfig hdfsConfig, org.apache.hadoop.conf.Configuration conf, java.util.List<java.lang.String> paths, Environment env) Adds a set of files in HDFS to the classpath for hadoop nodes (via the DistributedCache)
static void	addFilesToDistributedCache(HDFSConfig hdfsConfig, org.apache.hadoop.conf.Configuration conf, java.util.List<java.lang.String> paths, Environment env) Adds a set of files to the distributed cache for the supplied Configuration
static void	addFileToClasspath(HDFSConfig hdfsConfig, org.apache.hadoop.conf.Configuration conf, java.lang.String path, Environment env) Adds a file in HDFS to the classpath for hadoop nodes (via the DistributedCache)
static java.lang.String	addFileToDistributedCache(HDFSConfig hdfsConfig, org.apache.hadoop.conf.Configuration conf, java.lang.String path, Environment env) Adds a file to the distributed cache for the supplied Configuration
static void	addWekaInstalledFilesToClasspath(HDFSConfig hdfsConfig, org.apache.hadoop.conf.Configuration conf, java.util.List<java.lang.String> paths, Environment env) Add a list of files, relative to the root of the Weka installation directory in HDFS (i.e.
static void	checkForWindowsAccessingHadoopOnLinux(org.apache.hadoop.conf.Configuration conf) If accessing Hadoop running on a *nix system from Windows then we have to post-process the classpath setup for the job because it will contain ';' rather than ':' as the separator.
static void	copyFilesToWekaHDFSInstallationDirectory(java.util.List<java.lang.String> localFiles, HDFSConfig config, Environment env, boolean overwrite) Copy a set of local files into the Weka installation directory in HDFS
static void	copyToHDFS(java.lang.String localFile, java.lang.String hdfsPath, HDFSConfig config, Environment env, boolean overwrite) Copy a local file into HDFS
static void	deleteDirectory(HDFSConfig hdfsConfig, org.apache.hadoop.conf.Configuration conf, java.lang.String path, Environment env) Delete a directory in HDFS
static void	deleteFile(HDFSConfig hdfsConfig, org.apache.hadoop.conf.Configuration conf, java.lang.String path, Environment env) Delete a file in HDFS
static void	main(java.lang.String[] args)
static void	moveInHDFS(java.lang.String source, java.lang.String target, HDFSConfig config, Environment env) Move a file from one location to another in HDFS
static java.lang.String	resolvePath(java.lang.String path, Environment env) Utility method to resolve all environment variables in a given path
static void	serializeObjectToDistributedCache(java.lang.Object toSerialize, HDFSConfig hdfsConfig, org.apache.hadoop.conf.Configuration conf, java.lang.String fileNameInCache, Environment env) Serializes the given object into a file in the staging area in HDFS and then adds that file to the distributed cache for the configuration

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

WEKA_LIBRARIES_LOCATION

public static final java.lang.String WEKA_LIBRARIES_LOCATION

The default location in HDFS to place weka.jar and other libraries for inclusion in the classpath of the hadoop nodes

See Also:

Constant Field Values

WINDOWS_ACCESSING_HADOOP_ON_LINUX_SYS_PROP

public static final java.lang.String WINDOWS_ACCESSING_HADOOP_ON_LINUX_SYS_PROP

Users need to set HADOOP_ON_LINUX environment variable to "true" if accessing a *nix Hadoop cluster from Windows so that we can post-process the job classpath in the Configuration file to use ':' rather than ';' as separators.

See Also:

Constant Field Values

WEKA_TEMP_DISTRIBUTED_CACHE_FILES

public static final java.lang.String WEKA_TEMP_DISTRIBUTED_CACHE_FILES

Staging location for non library files to be distributed to the nodes by the distributed cache

See Also:

Constant Field Values

Constructor Detail

HDFSUtils

public HDFSUtils()

Method Detail

resolvePath

public static java.lang.String resolvePath(java.lang.String path,
                                           Environment env)

Utility method to resolve all environment variables in a given path

Parameters:

path - the path in HDFS

env - environment variables to use

Returns:

the path with all environment variables resolved

moveInHDFS

public static void moveInHDFS(java.lang.String source,
                              java.lang.String target,
                              HDFSConfig config,
                              Environment env)
                       throws java.io.IOException

Move a file from one location to another in HDFS

Parameters:

source - the source path in HDFS

target - the target path in HDFS

config - the HDFSConfig with connection details

env - environment variables

Throws:

java.io.IOException - if a problem occurs

copyToHDFS

public static void copyToHDFS(java.lang.String localFile,
                              java.lang.String hdfsPath,
                              HDFSConfig config,
                              Environment env,
                              boolean overwrite)
                       throws java.io.IOException

Copy a local file into HDFS

Parameters:

localFile - the path to the local file

hdfsPath - the destination path in HDFS

config - the HDFSConfig containing connection details

env - environment variables

overwrite - true if the destination should be overwritten (if it already exists)

Throws:

java.io.IOException - if a problem occurs

copyFilesToWekaHDFSInstallationDirectory

public static void copyFilesToWekaHDFSInstallationDirectory(java.util.List<java.lang.String> localFiles,
                                                            HDFSConfig config,
                                                            Environment env,
                                                            boolean overwrite)
                                                     throws java.io.IOException

Copy a set of local files into the Weka installation directory in HDFS

Parameters:

localFiles - a list of local files to copy

config - the HDFSConfig containing connection details

env - environment variables

overwrite - true if the destination file should be overwritten (if it exists already)

Throws:

java.io.IOException - if a problem occurs

checkForWindowsAccessingHadoopOnLinux

public static void checkForWindowsAccessingHadoopOnLinux(org.apache.hadoop.conf.Configuration conf)

If accessing Hadoop running on a *nix system from Windows then we have to post-process the classpath setup for the job because it will contain ';' rather than ':' as the separator.

Parameters:

conf - the Configuration to fix up.

addFileToClasspath

public static void addFileToClasspath(HDFSConfig hdfsConfig,
                                      org.apache.hadoop.conf.Configuration conf,
                                      java.lang.String path,
                                      Environment env)
                               throws java.io.IOException

Adds a file in HDFS to the classpath for hadoop nodes (via the DistributedCache)

Parameters:

hdfsConfig - the HDFSConfig object with host and port set

conf - the Configuration object that will be changed by this operation

path - the path to the file (in HDFS) to be added to the classpath for hadopp nodes

env - any environment variables

Throws:

java.io.IOException - if a problem occurs

addFilesClasspath

public static void addFilesClasspath(HDFSConfig hdfsConfig,
                                     org.apache.hadoop.conf.Configuration conf,
                                     java.util.List<java.lang.String> paths,
                                     Environment env)
                              throws java.io.IOException

Adds a set of files in HDFS to the classpath for hadoop nodes (via the DistributedCache)

Parameters:

hdfsConfig - the HDFSConfig object with host and port set

conf - the Configuration object that will be changed by this operation

paths - a list of paths (in HDFS) to be added to the classpath for hadopp nodes

env - any environment variables

Throws:

java.io.IOException - if a problem occurs

addWekaInstalledFilesToClasspath

public static void addWekaInstalledFilesToClasspath(HDFSConfig hdfsConfig,
                                                    org.apache.hadoop.conf.Configuration conf,
                                                    java.util.List<java.lang.String> paths,
                                                    Environment env)
                                             throws java.io.IOException

Add a list of files, relative to the root of the Weka installation directory in HDFS (i.e. WEKA_LIBRARIES_LOCATION), to the classpath for the mappers and reducers.

Parameters:

hdfsConfig -

conf -

paths - a list of paths (relative to the Weka installation root in HDFS) to add to the classpath for mappers and reducers

env -

Throws:

java.io.IOException

deleteDirectory

public static void deleteDirectory(HDFSConfig hdfsConfig,
                                   org.apache.hadoop.conf.Configuration conf,
                                   java.lang.String path,
                                   Environment env)
                            throws java.io.IOException

Delete a directory in HDFS

Parameters:

hdfsConfig - the HDFSConfig to use with connection details set

conf - the Configuration object

path - the path to delete

env - environment variables

Throws:

java.io.IOException - if a problem occurs

deleteFile

public static void deleteFile(HDFSConfig hdfsConfig,
                              org.apache.hadoop.conf.Configuration conf,
                              java.lang.String path,
                              Environment env)
                       throws java.io.IOException

Delete a file in HDFS

Parameters:

hdfsConfig - the HDFSConfig to use with connection details set

conf - the Configuration object

path - the path to delete

env - environment variables

Throws:

java.io.IOException - if a problem occurs

serializeObjectToDistributedCache

public static void serializeObjectToDistributedCache(java.lang.Object toSerialize,
                                                     HDFSConfig hdfsConfig,
                                                     org.apache.hadoop.conf.Configuration conf,
                                                     java.lang.String fileNameInCache,
                                                     Environment env)
                                              throws java.io.IOException

Serializes the given object into a file in the staging area in HDFS and then adds that file to the distributed cache for the configuration

Parameters:

toSerialize - the object to serialize

hdfsConfig - the hdfs configuration to use

conf - the job configuration to configure

fileNameInCache - the file name only for the serialized object in the cache

env - environment variables

Throws:

java.io.IOException - if a problem occurs

addFileToDistributedCache

public static java.lang.String addFileToDistributedCache(HDFSConfig hdfsConfig,
                                                         org.apache.hadoop.conf.Configuration conf,
                                                         java.lang.String path,
                                                         Environment env)
                                                  throws java.io.IOException

Adds a file to the distributed cache for the supplied Configuration

Parameters:

hdfsConfig - the hdfs configuration to use

conf - the job configuration to configure

path - the path to the file to add. This can be a local file, in which case it is first staged in HDFS, or a file in HDFS.

env - environment variables

Returns:

the filename only part of the path (this is what will be accessible from the distributed cache to a client via standard Java file IO)

Throws:

java.io.IOException - if a problem occurs

addFilesToDistributedCache

public static void addFilesToDistributedCache(HDFSConfig hdfsConfig,
                                              org.apache.hadoop.conf.Configuration conf,
                                              java.util.List<java.lang.String> paths,
                                              Environment env)
                                       throws java.lang.Exception

Adds a set of files to the distributed cache for the supplied Configuration

Parameters:

hdfsConfig - the hdfs configuration to use

conf - the job configuration to configure

paths - a list of paths to to add to the distributed cache. These can be a local files, in which case they are first staged in HDFS, or a files in HDFS, or a mixture of both.

env - environment variables from the distributed cache to a client via standard Java file IO)

Throws:

java.io.IOException - if a problem occurs

java.lang.Exception

main

public static void main(java.lang.String[] args)