You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
docs: update README with parse accuracy stats and final roadmap (#20)
- Add real-world parse accuracy table (99.7% Rails, 98.0% Devise)
- Update feature list to reflect all solved hard problems
- Update known limitations to current state
- Mark all roadmap phases complete in CLAUDE.md
- Close all 5 deferred issues (resolved in PRs #15-#19)
- Add "Built with Claude Code" attribution
-[x] Pin operator: `in ^variable` (scoped to InClause)
145
-
- Deferred:
146
-
- Guard clauses (`in x if x > 0`) — conflicts with IfStatement in LR parser
143
+
- Guard clauses (`in x if x > 0`) deferred — conflicts with IfStatement
147
144
148
145
### Phase 6: Editor integration ✅
149
-
-[x] Indentation: `indentNodeProp` for all block constructs, `delimitedIndent` for `[]`/`{}`/`()`
150
-
-[x] Folding: method/class/module bodies, blocks, control flow, strings
151
-
-[x] Autocompletion: 31 Ruby keywords via `completeFromList()`
146
+
-[x] Indentation for all block constructs, `delimitedIndent` for `[]`/`{}`/`()`
147
+
-[x] Folding for method/class/module bodies, blocks, control flow, strings
148
+
-[x] Autocompletion: 31 Ruby keywords
152
149
-[x] Bracket closing, comment toggling
153
-
-[x]67 test cases
150
+
-[x]83 test cases
154
151
155
152
### Phase 7: Production readiness ✅
156
153
-[x] Demo page with GitHub Pages deployment
157
-
-[x] README with installation and usage docs
154
+
-[x] README with real-world parse accuracy stats
155
+
-[x] All 5 deferred external tokenizer issues resolved (#7-#11)
158
156
-[x] LICENSE (MIT)
159
157
160
-
### Open issues (external tokenizer required)
161
-
These are the remaining hard problems that need `src/tokens.ts`:
162
-
-[#7](https://github.com/jeanpaulsio/codemirror-lang-ruby/issues/7) — Regex literals (`/pattern/` vs division)
163
-
-[#8](https://github.com/jeanpaulsio/codemirror-lang-ruby/issues/8) — Block attachment to method calls
164
-
-[#9](https://github.com/jeanpaulsio/codemirror-lang-ruby/issues/9) — Bare method calls without parens
165
-
-[#10](https://github.com/jeanpaulsio/codemirror-lang-ruby/issues/10) — Heredoc support
166
-
-[#11](https://github.com/jeanpaulsio/codemirror-lang-ruby/issues/11) — `%`-literal interpolation and non-bracket delimiters
167
-
168
158
## Coding Standards
169
159
170
160
-**Test-driven**: Write the test case first, then fix the grammar to make it pass
@@ -174,19 +164,16 @@ These are the remaining hard problems that need `src/tokens.ts`:
174
164
-**Reference tree-sitter-ruby**: When in doubt about how Ruby parses something, check tree-sitter-ruby's grammar and test corpus
175
165
-**External tokenizers**: When the grammar notation can't express something (string interpolation, heredocs, regex/division ambiguity), use external tokenizers in a separate `tokens.ts` file. See `@lezer/python` and `@lezer/javascript` for examples.
176
166
177
-
## Known Hard Problems
178
-
179
-
These require an external tokenizer (`src/tokens.ts`) and are tracked as GitHub issues:
180
-
181
-
1.**`/` ambiguity** (#7): `/regex/` vs `a / b`. Requires an external tokenizer that checks preceding context. See `@lezer/javascript` for this pattern.
182
-
183
-
2.**Block attachment** (#8): `items.each { |x| x }`. The `MethodCall` expression reduces before `{` can attach. Needs grammar restructure or external tokenizer.
184
-
185
-
3.**Optional parentheses** (#9): `puts "hello"` vs `puts("hello")`. The biggest source of parse conflicts in Ruby. tree-sitter-ruby uses an external scanner.
167
+
## Solved Hard Problems
186
168
187
-
4.**Heredocs** (#10): `<<~RUBY\n code\nRUBY`. The delimiter is arbitrary, can be indented, and can stack. Needs a `ContextTracker`.
169
+
All five originally-deferred problems have been resolved using external tokenizers in `src/tokens.ts`:
188
170
189
-
5.**String interpolation** is SOLVED -- uses `@local tokens` pattern from `@lezer/javascript`.
171
+
1.**`/` ambiguity** — `regexTokenizer` uses `stack.canShift()` to emit Regex or divideOp
172
+
2.**Block attachment** — `BlockCall` expression with `!blockCall` precedence and `~blockOrHash` GLR
173
+
3.**Bare method calls** — `@extend` with curated keyword list for 25 common Ruby methods
174
+
4.**Heredocs** — `lessThanTokenizer` scans entire heredoc as one token, also handles `<`/`<=`
175
+
5.**`%`-literals** — `percentLiteralTokenizer` handles all delimiters, also handles `%` modulo
176
+
6.**String interpolation** — `@local tokens` pattern from `@lezer/javascript`
These are tracked as [open issues](https://github.com/jeanpaulsio/codemirror-lang-ruby/issues) and require external tokenizers:
42
-
43
-
- No regex literals (`/pattern/` conflicts with division)
44
-
- No block attachment to method calls (`items.each { |x| x }`)
45
-
- No bare method calls without parens (`puts "hello"`)
46
-
- No heredocs (`<<~RUBY`)
47
-
- No `%`-literal interpolation or non-bracket delimiters
57
+
- Heredoc and `%`-literal bodies are opaque tokens (no interpolation highlighting inside)
58
+
- Hash symbol-key shorthand (`{ name: "Alice" }`) not supported (`:` conflicts with Symbol token when blocks are expressions)
59
+
- Setter method definitions (`def name=(val)`) and subscript access (`arr[0]`) not yet supported
60
+
-`<<` left-shift operator not distinguished from heredoc start
48
61
49
62
## Development
50
63
51
64
```bash
52
65
npm install # Install dependencies
53
66
npm run build # Build grammar + bundle to dist/
54
-
npm test# Run all 67 grammar tests
67
+
npm test# Run all 83 grammar tests
55
68
npm run lint # TypeScript type check
56
69
npm run demo:build # Build the demo page
57
70
```
58
71
72
+
## Built with Claude Code
73
+
74
+
This entire project — grammar, external tokenizers, tests, editor integration, and demo — was written by [Claude Code](https://claude.ai/code), guided by [@jeanpaulsio](https://github.com/jeanpaulsio).
0 commit comments