Commit a9b8bba
committed
feat(ie-css): implement CSS tokenizer per CSS Syntax Level 3 #67
CSS tokenizer with all token types per the spec:
- CssToken enum: Ident, Function, AtKeyword, Hash(Id/Unrestricted),
String, BadString, Url, BadUrl, Delim, Number(Integer/Number),
Percentage, Dimension, Whitespace, Cdo, Cdc, punctuation, Eof
- Input preprocessing: CR/LF normalization, null replacement
- Escape handling: hex escapes (up to 6 digits), literal escapes
- Number consumption: integer, float, scientific notation
- String tokens: single/double quoted with escape handling
- URL tokens: unquoted URL consumption
- Ident-like: identifier, function, url() detection
- Comments consumed and discarded
- Iterator-based: impl Iterator<Item = CssToken>
- 26 unit tests covering all token types and edge cases1 parent 447aedc commit a9b8bba
2 files changed
Lines changed: 775 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
6 | 6 | | |
7 | 7 | | |
8 | 8 | | |
| 9 | + | |
9 | 10 | | |
10 | 11 | | |
11 | 12 | | |
| 13 | + | |
0 commit comments