Skip to content

Commit a9b8bba

Browse files
committed
feat(ie-css): implement CSS tokenizer per CSS Syntax Level 3 #67
CSS tokenizer with all token types per the spec: - CssToken enum: Ident, Function, AtKeyword, Hash(Id/Unrestricted), String, BadString, Url, BadUrl, Delim, Number(Integer/Number), Percentage, Dimension, Whitespace, Cdo, Cdc, punctuation, Eof - Input preprocessing: CR/LF normalization, null replacement - Escape handling: hex escapes (up to 6 digits), literal escapes - Number consumption: integer, float, scientific notation - String tokens: single/double quoted with escape handling - URL tokens: unquoted URL consumption - Ident-like: identifier, function, url() detection - Comments consumed and discarded - Iterator-based: impl Iterator<Item = CssToken> - 26 unit tests covering all token types and edge cases
1 parent 447aedc commit a9b8bba

2 files changed

Lines changed: 775 additions & 0 deletions

File tree

crates/ie-css/src/lib.rs

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,8 @@
66
pub mod parser;
77
pub mod selector;
88
pub mod style;
9+
pub mod tokenizer;
910

1011
pub use parser::parse_stylesheet;
1112
pub use style::ComputedStyle;
13+
pub use tokenizer::{CssToken, CssTokenizer};

0 commit comments

Comments
 (0)