Lines Matching defs:tokens
55 # These are the types of tokens that the lexer can produce.
56 tokens = ("FTERM", "TERM", "LPAREN", "RPAREN", "AND", "OR", "QUOTE1",
66 # This rule causes spaces to break tokens, but the space itself is not
75 # tokens.
181 shows the user exactly which tokens are produced from the input
193 """This class defines the parser which converts a stream of tokens into
197 # Use the same set of tokens as the lexer.
198 tokens = QueryLexer.tokens
200 # Define the precendence and associativity of certain tokens to
446 The "text" parameter is the tokens and syntax of the query.
1325 # Find the possible tokens for this action.
1334 # If this search term matches any of the tokens the
1473 # tokens, return results for every known token.