Skip to content

Lexer.md

Dark Meta edited this page Aug 18, 2025 · 1 revision

Lexer

Hardened, Unicode‑NFC normalized lexer with explicit marker glyphs:

  • AI blocks: ⚡ ... ⛓ → yields opaque AIOnlyBlock token (requires authorization)
  • Comments: line and nested block ⦅ ... ⦆
  • Numeric glyphs: 𝍠(0) … 𝍩(9); optional mixed numerals (feature‑flag)
  • Identifiers: Unicode XID_Start/XID_Continue
  • Safety: disallows zero‑width control characters; strict errors on invalid glyphs

See repo src/core/lexer.rs for the full implementation and tests.

Clone this wiki locally