org.apache.lucene.analysis
Class LowerCaseTokenizer
public final
class
LowerCaseTokenizer
extends LetterTokenizer
LowerCaseTokenizer performs the function of LetterTokenizer
and LowerCaseFilter together. It divides text at non-letters and converts
them to lower case. While it is functionally equivalent to the combination
of LetterTokenizer and LowerCaseFilter, there is a performance advantage
to doing the two tasks at once, hence this (redundant) implementation.
Note: this does a decent job for most European languages, but does a terrible
job for some Asian languages, where words are not separated by spaces.
Method Summary |
protected char | normalize(char c) Collects only characters which satisfy
{@link Character#isLetter(char)}. |
public LowerCaseTokenizer(Reader in)
Construct a new LowerCaseTokenizer.
protected char normalize(char c)
Collects only characters which satisfy
{@link Character#isLetter(char)}.
Copyright © 2000-2007 Apache Software Foundation. All Rights Reserved.