Searched defs:TokenStream (Results 1 - 1 of 1) sorted by relevance

/lucene-3.6.0/lucene/core/src/java/org/apache/lucene/analysis/
H A DTokenStream.java32 * A <code>TokenStream</code> enumerates the sequence of tokens, either from
37 * <li>{@link Tokenizer}, a <code>TokenStream</code> whose input is a Reader; and
38 * <li>{@link TokenFilter}, a <code>TokenStream</code> whose input is another
39 * <code>TokenStream</code>.
41 * A new <code>TokenStream</code> API has been introduced with Lucene 2.9. This API
46 * <code>TokenStream</code> now extends {@link AttributeSource}, which provides
47 * access to all of the token {@link Attribute}s for the <code>TokenStream</code>.
53 * <b>The workflow of the new <code>TokenStream</code> API is as follows:</b>
55 * <li>Instantiation of <code>TokenStream</code>/{@link TokenFilter}s which add/get
57 * <li>The consumer calls {@link TokenStream#rese
85 public abstract class TokenStream extends AttributeSource implements Closeable { class in inherits:AttributeSource,Closeable
90 protected TokenStream() { method in class:TokenStream
98 protected TokenStream(AttributeSource input) { method in class:TokenStream
106 protected TokenStream(AttributeFactory factory) { method in class:TokenStream
[all...]

Completed in 32 milliseconds