weka.distributed

Class KMeansReduceTask

java.lang.Object

weka.distributed.KMeansReduceTask

All Implemented Interfaces:

java.io.Serializable

public class KMeansReduceTask
extends java.lang.Object
implements java.io.Serializable

Reduce task for k-means clustering. Processes partial cluster summary metadata for a particular run in order to produce a set of Instances that contains new cluster centroids.

Version:

$Revision: $

Author:

Mark Hall (mhall{[at]}pentaho{[dot]}com)

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
KMeansReduceTask()

Method Summary

Modifier and Type	Method and Description
static weka.core.Instances	computeDistancePrimingDataFromDistanceFunctions(java.util.List<weka.core.NormalizableDistance> distanceFuncs, weka.core.Instances headerNoSummary) Utility function to examine the attribute ranges in a bunch of distance functions and return a two instance dataset with the global mins/maxes of numeric attributes set.
java.util.List<weka.core.Instances>	getAggregatedCentroidSummaries() Get the aggregated summary data for each individual centroid.
weka.core.Instances	getCentroidsForRun() Return the centroids for the run
weka.core.Instances	getGlobalDistanceFunctionPrimingData() Get the global distance function priming data.
int	getIterationNumber() Get the current iteration number
int	getRunNumber() Get the run number
double	getTotalWithinClustersError() Get the total within cluster error for this run
KMeansReduceTask	reduceClusters(int runNumber, int iterationNumber, weka.core.Instances headerNoSummary, java.util.List<java.util.List<weka.core.Instances>> clusterSummaries) Reduce the cluster centroid summary metadata instances for a particular run in order to produce a new set of Instances that contains the new cluster centroids for the run.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

KMeansReduceTask

public KMeansReduceTask()

Method Detail

reduceClusters

public KMeansReduceTask reduceClusters(int runNumber,
                                       int iterationNumber,
                                       weka.core.Instances headerNoSummary,
                                       java.util.List<java.util.List<weka.core.Instances>> clusterSummaries)
                                throws DistributedWekaException

Reduce the cluster centroid summary metadata instances for a particular run in order to produce a new set of Instances that contains the new cluster centroids for the run. Adds the total within cluster error to the relation name of the instances. If the iteration number is 0 then also generates a two instance data set that can be used for initializing a distance function. The two instances contain global minimum and maximum values for numeric attributes respectively, which is used in the distance function for normalization. This particular dataset is useful when filters (beyond missing values replacement) have been specified for k-means and it is not possible to use the summary stats in the global ARFF header file for initializing the distance function

Parameters:

runNumber - the current run number

iterationNumber - the current iteration number of k-means

headerNoSummary - the global ARFF header (as computed by the ArffHeader job on the entire dataset, and having passed through any preprocessing filters). We need this so that the correct index for nominal attribute values can be set in the new centroids (map tasks accumulating summary stats when clustering partitions of the data may see nominal values in different orders, or not see some values at all, compared to the global header)

clusterSummaries - a list of cluster summary information. Each inner list of Instances will have been generated by a map task on a subset of the data. Each instances object in the list contains the summary stats for one cluster centroid. Inner lists are in order of centroid number. A particular Instances entry in a list may be null - this indicates that the cluster was empty within that particular map task (i.e. no training instances were assigned to it)

Returns:

an instance of KMeansReduceTask with new centroids and supporting data computed.

Throws:

DistributedWekaException - if a problem occurs

getCentroidsForRun

public weka.core.Instances getCentroidsForRun()

Return the centroids for the run

Returns:

the centroids as a set of instances

getAggregatedCentroidSummaries

public java.util.List<weka.core.Instances> getAggregatedCentroidSummaries()

Get the aggregated summary data for each individual centroid. This is represented as a Instances header with summary meta attributes

Returns:

a list of summary meta data

getGlobalDistanceFunctionPrimingData

public weka.core.Instances getGlobalDistanceFunctionPrimingData()

Get the global distance function priming data. This contains global mins and maxes for attributes in the transformed (by any filters) space

Returns:

the global distance function priming data

getRunNumber

public int getRunNumber()

Get the run number

Returns:

the run number

getIterationNumber

public int getIterationNumber()

Get the current iteration number

Returns:

the current iteration number

getTotalWithinClustersError

public double getTotalWithinClustersError()

Get the total within cluster error for this run

Returns:

the total within cluster error for this run

computeDistancePrimingDataFromDistanceFunctions

public static weka.core.Instances computeDistancePrimingDataFromDistanceFunctions(java.util.List<weka.core.NormalizableDistance> distanceFuncs,
                                                                                  weka.core.Instances headerNoSummary)
                                                                           throws DistributedWekaException

Utility function to examine the attribute ranges in a bunch of distance functions and return a two instance dataset with the global mins/maxes of numeric attributes set. This can be used to "prime" a distance function.

Parameters:

distanceFuncs - a list of distance functions (where each potentially has only seen part of the overall dataset

headerNoSummary - the header of the data that the distance functions have been seeing

Returns:

a priming data set with global min and max values for numeric attributes

Throws:

DistributedWekaException - if a problem occurs