weka.core.tokenizers

Class Tokenizer

java.lang.Object

weka.core.tokenizers.Tokenizer

All Implemented Interfaces:

java.io.Serializable, java.util.Enumeration<java.lang.String>, OptionHandler, RevisionHandler

Direct Known Subclasses:

AlphabeticTokenizer, CharacterDelimitedTokenizer, CharacterNGramTokenizer

public abstract class Tokenizer
extends java.lang.Object
implements java.util.Enumeration<java.lang.String>, OptionHandler, java.io.Serializable, RevisionHandler

A superclass for all tokenizer algorithms.

Version:

$Revision: 10203 $

Author:

FracPete (fracpete at waikato dot ac dot nz)

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
Tokenizer()

Method Summary

Modifier and Type	Method and Description
java.lang.String[]	getOptions() Gets the current option settings for the OptionHandler.
abstract java.lang.String	globalInfo() Returns a string describing the stemmer
abstract boolean	hasMoreElements() Tests if this enumeration contains more elements.
java.util.Enumeration<Option>	listOptions() Returns an enumeration of all the available options..
abstract java.lang.String	nextElement() Returns the next element of this enumeration if this enumeration object has at least one more element to provide.
static void	runTokenizer(Tokenizer tokenizer, java.lang.String[] options) initializes the given tokenizer with the given options and runs the tokenizer over all the remaining strings in the options array.
void	setOptions(java.lang.String[] options) Sets the OptionHandler's options using the given list.
abstract void	tokenize(java.lang.String s) Sets the string to tokenize.
static java.lang.String[]	tokenize(Tokenizer tokenizer, java.lang.String[] options) initializes the given tokenizer with the given options and runs the tokenizer over all the remaining strings in the options array.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface weka.core.OptionHandler

makeCopy

Methods inherited from interface weka.core.RevisionHandler

getRevision

Constructor Detail

Tokenizer

public Tokenizer()

Method Detail

globalInfo

public abstract java.lang.String globalInfo()

Returns a string describing the stemmer

Returns:

a description suitable for displaying in the explorer/experimenter gui

listOptions

public java.util.Enumeration<Option> listOptions()

Returns an enumeration of all the available options..

Specified by:

listOptions in interface OptionHandler

Returns:

an enumeration of all available options.

getOptions

public java.lang.String[] getOptions()

Gets the current option settings for the OptionHandler.

Specified by:

getOptions in interface OptionHandler

Returns:

the list of current option settings as an array of strings

setOptions

public void setOptions(java.lang.String[] options)
                throws java.lang.Exception

Sets the OptionHandler's options using the given list. All options will be set (or reset) during this call (i.e. incremental setting of options is not possible).

Specified by:

setOptions in interface OptionHandler

Parameters:

options - the list of options as an array of strings

Throws:

java.lang.Exception - if an option is not supported

hasMoreElements

public abstract boolean hasMoreElements()

Tests if this enumeration contains more elements.

Specified by:

hasMoreElements in interface java.util.Enumeration<java.lang.String>

Returns:

true if and only if this enumeration object contains at least one more element to provide; false otherwise.

nextElement

public abstract java.lang.String nextElement()

Returns the next element of this enumeration if this enumeration object has at least one more element to provide.

Specified by:

nextElement in interface java.util.Enumeration<java.lang.String>

Returns:

the next element of this enumeration.

tokenize

public abstract void tokenize(java.lang.String s)

Sets the string to tokenize. Tokenization happens immediately.

Parameters:

s - the string to tokenize

tokenize

public static java.lang.String[] tokenize(Tokenizer tokenizer,
                                          java.lang.String[] options)
                                   throws java.lang.Exception

initializes the given tokenizer with the given options and runs the tokenizer over all the remaining strings in the options array. If no strings remained in the option string then data is read from stdin, line by line.

Parameters:

tokenizer - the tokenizer to use

options - the options for the tokenizer

Returns:

the tokenized strings

Throws:

java.lang.Exception - if setting of options or tokenization fails

runTokenizer

public static void runTokenizer(Tokenizer tokenizer,
                                java.lang.String[] options)

initializes the given tokenizer with the given options and runs the tokenizer over all the remaining strings in the options array. The generated tokens are then printed to stdout. If no strings remained in the option string then data is read from stdin, line by line.

Parameters:

tokenizer - the tokenizer to use

options - the options for the tokenizer