JFlexTokenizerTest.java revision 1006
1004N/A * The contents of this file are subject to the terms of the 1004N/A * Common Development and Distribution License (the "License"). 1004N/A * You may not use this file except in compliance with the License. 1004N/A * language governing permissions and limitations under the License. 1004N/A * When distributing Covered Code, include this CDDL HEADER in each 1004N/A * If applicable, add the following below this CDDL HEADER, with the 1004N/A * fields enclosed by brackets "[]" replaced with your own identifying 1004N/A * information: Portions Copyright [yyyy] [name of copyright owner] 1004N/A * Copyright 2010 Sun Micosystems. All rights reserved. 1004N/A * Use is subject to license terms. 1004N/A * Unit tests for JFlexTokenizer. 1004N/A * Test that the various sub-classes of JFlexTokenizerTest return the 1004N/A * correct offsets for the tokens. They used to give wrong values for 1004N/A * the last token. Bug #15858. 1004N/A // The Fortran tokenizer doesn't accept the default input text, so 1004N/A // create a text fragment that it understands 1004N/A "1 token1 = token2 + token3",
1004N/A * Helper method for {@link #testOffsetAttribute()} that runs the test 1004N/A * on one single implementation class. 1004N/A * Helper method for {@link #testOffsetAttribute()} that runs the test 1004N/A * on one single implementation class with the specified input text and 1006N/A * The fix for bug #15858 caused a regression in ShSymbolTokenizer where 1006N/A * variables on the form {@code ${VARIABLE}} were not correctly indexed 1006N/A * if they were inside a quoted string. The closing brace would be part of 1006N/A * the indexed term in that case. 1006N/A // Shell command to tokenize 1006N/A // "echo" is an ignored token in ShSymbolTokenizer, "xyz" is a string 1006N/A // and not a symbol. Therefore, expect just the two tokens that name