work in progress...
Possible uses:
- Syntax Highlighting
- Semantic Highlighting
- Unused variables
- Unresolved variable references
- Code block matching
- Linting
Future intentions:
- Integrate with a similarly designed XSLT 3.0 Tokenizer
Design Goals:
- Single pass of text, character by character
- Disambiguate tokens in the same pass
- Use single character lookahead
- No regex
- Manage scope of tokens in an object tree
- Not a parser, so theres no AST