You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
docs: update roadmap, README, and package metadata for Phase 7 (#14)
- Rewrite README with install instructions, full feature list,
known limitations, and link to live demo
- Update CLAUDE.md roadmap: mark Phases 1-7 complete, consolidate
deferred items under "Open issues" with GitHub issue links
- Add keywords and files fields to package.json for discoverability
- Heredocs require remembering an arbitrary delimiter word
131
-
- Regex vs division requires knowing what token came before `/`
132
-
133
-
An external tokenizer is a TypeScript function that the parser calls when it needs a token the grammar can't express. It reads characters from the input stream, decides what token to emit, and hands control back to the parser. Declared in the grammar with `@external tokens tokenizer from "./tokens" { Token1, Token2 }` and implemented in `src/tokens.ts` as `ExternalTokenizer` instances. See `@lezer/javascript` and `@lezer/python` for working examples.
This is what makes the package actually *usable* in an editor, not just a parser.
164
-
165
-
-[ ]**Indentation**: proper `indentNodeProp` for every block construct; use `delimitedIndent` for `[]`/`{}`; deindent on `end`/`else`/`elsif`/`when`/`rescue`/`ensure`
-[x] Pin operator: `in ^variable` (scoped to InClause)
145
+
- Deferred:
146
+
- Guard clauses (`in x if x > 0`) — conflicts with IfStatement in LR parser
147
+
148
+
### Phase 6: Editor integration ✅
149
+
-[x] Indentation: `indentNodeProp` for all block constructs, `delimitedIndent` for `[]`/`{}`/`()`
150
+
-[x] Folding: method/class/module bodies, blocks, control flow, strings
151
+
-[x] Autocompletion: 31 Ruby keywords via `completeFromList()`
152
+
-[x] Bracket closing, comment toggling
153
+
-[x] 67 test cases
154
+
155
+
### Phase 7: Production readiness ✅
156
+
-[x] Demo page with GitHub Pages deployment
157
+
-[x] README with installation and usage docs
158
+
-[x] LICENSE (MIT)
159
+
160
+
### Open issues (external tokenizer required)
161
+
These are the remaining hard problems that need `src/tokens.ts`:
162
+
-[#7](https://github.com/jeanpaulsio/codemirror-lang-ruby/issues/7) — Regex literals (`/pattern/` vs division)
163
+
-[#8](https://github.com/jeanpaulsio/codemirror-lang-ruby/issues/8) — Block attachment to method calls
164
+
-[#9](https://github.com/jeanpaulsio/codemirror-lang-ruby/issues/9) — Bare method calls without parens
165
+
-[#10](https://github.com/jeanpaulsio/codemirror-lang-ruby/issues/10) — Heredoc support
166
+
-[#11](https://github.com/jeanpaulsio/codemirror-lang-ruby/issues/11) — `%`-literal interpolation and non-bracket delimiters
176
167
177
168
## Coding Standards
178
169
@@ -185,15 +176,17 @@ This is what makes the package actually *usable* in an editor, not just a parser
185
176
186
177
## Known Hard Problems
187
178
188
-
These are documented so you can plan around them:
179
+
These require an external tokenizer (`src/tokens.ts`) and are tracked as GitHub issues:
189
180
190
-
1.**`/` ambiguity**: `/regex/` vs `a / b`. Requires an external tokenizer that tracks whether a `/` is in a "regex-allowed" position (after operators, keywords, `(`, `[`, etc.) vs "division" position (after identifiers, numbers, `)`, `]`). See `@lezer/javascript` for exactly this pattern.
181
+
1.**`/` ambiguity** (#7): `/regex/` vs `a / b`. Requires an external tokenizer that checks preceding context. See `@lezer/javascript` for this pattern.
191
182
192
-
2.**String interpolation**: `"hello #{1 + 2}"` nests arbitrary Ruby expressions inside strings. Requires an external tokenizer that tracks brace depth. See `@lezer/javascript` template literal handling.
183
+
2.**Block attachment** (#8): `items.each { |x| x }`. The `MethodCall` expression reduces before `{` can attach. Needs grammar restructure or external tokenizer.
193
184
194
-
3.**Heredocs**: `<<~RUBY\n code\nRUBY` -- the delimiter is arbitrary, can be indented, and can stack (`<<A; <<B`). This is one of the hardest parts. tree-sitter-ruby uses an external scanner for this.
185
+
3.**Optional parentheses** (#9): `puts "hello"` vs `puts("hello")`. The biggest source of parse conflicts in Ruby. tree-sitter-ruby uses an external scanner.
195
186
196
-
4.**Optional parentheses**: `puts "hello"` and `puts("hello")` are both valid. This creates ambiguity in the grammar. Lezer's GLR mode (`@ambiguity`) may help, but method calls without parens are the biggest source of parse conflicts in Ruby.
187
+
4.**Heredocs** (#10): `<<~RUBY\n code\nRUBY`. The delimiter is arbitrary, can be indented, and can stack. Needs a `ContextTracker`.
188
+
189
+
5.**String interpolation** is SOLVED -- uses `@local tokens` pattern from `@lezer/javascript`.
**Work in progress.** The grammar covers core Ruby constructs (classes, modules, methods, control flow, blocks, lambdas, literals). See CLAUDE.md for the full roadmap of what still needs to be built.
0 commit comments