Lexer.md

Jump to bottom

Dark Meta edited this page Aug 18, 2025 · 1 revision

Lexer

Hardened, Unicode‑NFC normalized lexer with explicit marker glyphs:

AI blocks: ⚡ ... ⛓ → yields opaque AIOnlyBlock token (requires authorization)
Comments: line ⍝ and nested block ⦅ ... ⦆
Numeric glyphs: 𝍠(0) … 𝍩(9); optional mixed numerals (feature‑flag)
Identifiers: Unicode XID_Start/XID_Continue
Safety: disallows zero‑width control characters; strict errors on invalid glyphs

See repo src/core/lexer.rs for the full implementation and tests.