Class Tokenizer


  • public class Tokenizer
    extends java.lang.Object
    Taken from the Loop programming language compiler pipeline.
    • Field Detail

      • input

        private final java.lang.String input
      • DELIMITERS

        private static final int[] DELIMITERS
      • STRING_TERMINATORS

        private static final boolean[] STRING_TERMINATORS
    • Constructor Detail

      • Tokenizer

        public Tokenizer​(java.lang.String input)
    • Method Detail

      • tokenize

        public java.util.List<Token> tokenize()
      • isWhitespace

        private static boolean isWhitespace​(char c)
      • isSingleTokenDelimiter

        static boolean isSingleTokenDelimiter​(char c)
      • detokenize

        public static java.lang.String detokenize​(java.util.List<Token> tokens)
      • isDelimiter

        private static boolean isDelimiter​(char c)
      • bakeToken

        private static void bakeToken​(java.util.List<Token> tokens,
                                      char[] input,
                                      int i,
                                      int start)