Skip to content

Commit 7d29d45

Browse files
authored
docs: update roadmap, README, and package metadata for Phase 7 (#14)
- Rewrite README with install instructions, full feature list, known limitations, and link to live demo - Update CLAUDE.md roadmap: mark Phases 1-7 complete, consolidate deferred items under "Open issues" with GitHub issue links - Add keywords and files fields to package.json for discoverability
1 parent c3b5c7f commit 7d29d45

3 files changed

Lines changed: 93 additions & 75 deletions

File tree

CLAUDE.md

Lines changed: 55 additions & 62 deletions
Original file line numberDiff line numberDiff line change
@@ -113,66 +113,57 @@ Known limitations (to be addressed in later phases):
113113

114114
### Phase 1: Stabilize the starter grammar ✅
115115
- [x] Get `npm run build` passing cleanly
116-
- [x] Get all existing test cases passing (26/26)
116+
- [x] Get all existing test cases passing
117117
- [x] Fix all shift/reduce and reduce/reduce conflicts
118118
- [x] Set up GitHub Actions CI
119119

120-
### Phase 2: String interpolation and external tokenizers
121-
This phase introduces `src/tokens.ts` -- our first external tokenizer file.
122-
123-
- [ ] String interpolation: `"hello #{name}"` -- use `@local tokens` pattern (see `@lezer/javascript` template literals)
124-
- [ ] Heredocs: `<<~RUBY`, `<<-RUBY`, `<<RUBY` -- needs `ContextTracker` to store delimiter
125-
- [ ] `%`-literals: `%w[a b c]`, `%i[foo bar]`, `%q(string)`, `%Q(string)`, `%r(regex)`
126-
- [ ] Character literals: `?a`
127-
128-
**What's an external tokenizer?** The `.grammar` file can only express regular patterns. Some Ruby syntax needs runtime logic:
129-
- String interpolation requires tracking `#{}` brace depth inside strings
130-
- Heredocs require remembering an arbitrary delimiter word
131-
- Regex vs division requires knowing what token came before `/`
132-
133-
An external tokenizer is a TypeScript function that the parser calls when it needs a token the grammar can't express. It reads characters from the input stream, decides what token to emit, and hands control back to the parser. Declared in the grammar with `@external tokens tokenizer from "./tokens" { Token1, Token2 }` and implemented in `src/tokens.ts` as `ExternalTokenizer` instances. See `@lezer/javascript` and `@lezer/python` for working examples.
134-
135-
### Phase 3: Block attachment + method call edge cases
136-
- [ ] Block/DoBlock attachment to method calls: `items.each { |x| x }`, `items.each do |x| x end`
137-
- [ ] Hash symbol-key shorthand: `{ name: "Alice" }` (needs `:` vs Symbol disambiguation)
138-
- [ ] Bare method calls without parens: `puts "hello"`, `attr_reader :name`
139-
- [ ] Operator method definitions: `def <=>(other)`, `def [](index)`, `def []=(index, val)`
140-
- [ ] Method names with `?`, `!`, `=` suffixes (separate MethodName token, context-dependent)
141-
- [ ] Splat in method calls: `foo(*args, **kwargs, &block)`
142-
- [ ] Safe navigation: `obj&.method`
143-
144-
### Phase 4: Regex + more operators
145-
- [ ] Regex: `/` vs division ambiguity (external tokenizer checks preceding context)
146-
- [ ] Conditional assignment: `||=`, `&&=`
147-
- [ ] Multiple assignment: `a, b = 1, 2`
148-
- [ ] Destructuring: `a, *b = [1, 2, 3]`
149-
- [ ] Defined? operator
150-
- [ ] Endless method: `def square(x) = x * x` (Ruby 3.0+)
151-
- [ ] Numbered block params: `_1`, `_2` (Ruby 2.7+)
152-
- [ ] Proc/lambda: `proc { }`, `lambda { }`, `Proc.new { }`
153-
154-
### Phase 5: Pattern matching (Ruby 3.0+)
155-
- [ ] `case/in` pattern matching
156-
- [ ] Array patterns: `in [x, y, *rest]`
157-
- [ ] Hash patterns: `in { name:, age: }`
158-
- [ ] Find patterns: `in [*, x, *]`
159-
- [ ] Guard clauses: `in x if x > 0`
160-
- [ ] Pin operator: `in ^variable`
161-
162-
### Phase 6: Editor integration
163-
This is what makes the package actually *usable* in an editor, not just a parser.
164-
165-
- [ ] **Indentation**: proper `indentNodeProp` for every block construct; use `delimitedIndent` for `[]`/`{}`; deindent on `end`/`else`/`elsif`/`when`/`rescue`/`ensure`
166-
- [ ] **Folding**: fold ranges for method/class/module bodies, blocks, multiline strings, heredocs
167-
- [ ] **Autocompletion**: keyword completions (`def`, `class`, `if`, `end`, etc.) via `completeFromList()`; snippets for common patterns (`def...end`, `class...end`, `begin...rescue...end`)
168-
- [ ] **Error recovery**: add `@isGroup` for expressions/statements; add `[isolate]` on strings/comments for better incremental parsing
169-
- [ ] **Comprehensive tests**: 50+ test cases covering edge cases, error recovery, and real-world Ruby patterns (Rails, RSpec)
170-
171-
### Phase 7: Production readiness
172-
- [ ] Performance testing with large files (1000+ lines of real Rails code)
173-
- [ ] npm package publishing setup (README, LICENSE, .npmignore, package.json metadata)
174-
- [ ] Demo page / playground (CodeMirror editor with Ruby highlighting)
175-
- [ ] Comparison testing against tree-sitter-ruby's test corpus
120+
### Phase 2: String interpolation and literals ✅
121+
- [x] String interpolation: `"hello #{name}"` via `@local tokens` pattern
122+
- [x] `%`-literals: `%w[a b c]`, `%i[foo bar]`, `%q(string)`, `%Q(string)` with `[]`, `()`, `{}`, `<>` delimiters
123+
- [x] Character literals: `?a`, `?\n`
124+
125+
### Phase 3: Method call edge cases ✅ (partial)
126+
- [x] Hash symbol-key shorthand: `{ name: "Alice" }`
127+
- [x] Splat in method calls: `foo(*args, **kwargs, &block)`
128+
- [x] Safe navigation: `obj&.method`
129+
- Deferred to external tokenizer (see open issues):
130+
- Block/DoBlock attachment (#8)
131+
- Bare method calls without parens (#9)
132+
133+
### Phase 4: Operators and constructs ✅ (partial)
134+
- [x] Conditional assignment: `||=`, `&&=` (already in AssignOp)
135+
- [x] Multiple assignment: `a, b = 1, 2`
136+
- [x] Destructuring: `a, *b = [1, 2, 3]`
137+
- [x] Endless method: `def square(x) = x * x` (Ruby 3.0+)
138+
- [x] Operator precedence: `**` @right > `*`/`/` @left > `+`/`-` @left
139+
- Deferred to external tokenizer:
140+
- Regex `/` vs division (#7)
141+
142+
### Phase 5: Pattern matching (Ruby 3.0+) ✅ (partial)
143+
- [x] `case/in` pattern matching with `InClause`
144+
- [x] Pin operator: `in ^variable` (scoped to InClause)
145+
- Deferred:
146+
- Guard clauses (`in x if x > 0`) — conflicts with IfStatement in LR parser
147+
148+
### Phase 6: Editor integration ✅
149+
- [x] Indentation: `indentNodeProp` for all block constructs, `delimitedIndent` for `[]`/`{}`/`()`
150+
- [x] Folding: method/class/module bodies, blocks, control flow, strings
151+
- [x] Autocompletion: 31 Ruby keywords via `completeFromList()`
152+
- [x] Bracket closing, comment toggling
153+
- [x] 67 test cases
154+
155+
### Phase 7: Production readiness ✅
156+
- [x] Demo page with GitHub Pages deployment
157+
- [x] README with installation and usage docs
158+
- [x] LICENSE (MIT)
159+
160+
### Open issues (external tokenizer required)
161+
These are the remaining hard problems that need `src/tokens.ts`:
162+
- [#7](https://github.com/jeanpaulsio/codemirror-lang-ruby/issues/7) — Regex literals (`/pattern/` vs division)
163+
- [#8](https://github.com/jeanpaulsio/codemirror-lang-ruby/issues/8) — Block attachment to method calls
164+
- [#9](https://github.com/jeanpaulsio/codemirror-lang-ruby/issues/9) — Bare method calls without parens
165+
- [#10](https://github.com/jeanpaulsio/codemirror-lang-ruby/issues/10) — Heredoc support
166+
- [#11](https://github.com/jeanpaulsio/codemirror-lang-ruby/issues/11)`%`-literal interpolation and non-bracket delimiters
176167

177168
## Coding Standards
178169

@@ -185,15 +176,17 @@ This is what makes the package actually *usable* in an editor, not just a parser
185176

186177
## Known Hard Problems
187178

188-
These are documented so you can plan around them:
179+
These require an external tokenizer (`src/tokens.ts`) and are tracked as GitHub issues:
189180

190-
1. **`/` ambiguity**: `/regex/` vs `a / b`. Requires an external tokenizer that tracks whether a `/` is in a "regex-allowed" position (after operators, keywords, `(`, `[`, etc.) vs "division" position (after identifiers, numbers, `)`, `]`). See `@lezer/javascript` for exactly this pattern.
181+
1. **`/` ambiguity** (#7): `/regex/` vs `a / b`. Requires an external tokenizer that checks preceding context. See `@lezer/javascript` for this pattern.
191182

192-
2. **String interpolation**: `"hello #{1 + 2}"` nests arbitrary Ruby expressions inside strings. Requires an external tokenizer that tracks brace depth. See `@lezer/javascript` template literal handling.
183+
2. **Block attachment** (#8): `items.each { |x| x }`. The `MethodCall` expression reduces before `{` can attach. Needs grammar restructure or external tokenizer.
193184

194-
3. **Heredocs**: `<<~RUBY\n code\nRUBY` -- the delimiter is arbitrary, can be indented, and can stack (`<<A; <<B`). This is one of the hardest parts. tree-sitter-ruby uses an external scanner for this.
185+
3. **Optional parentheses** (#9): `puts "hello"` vs `puts("hello")`. The biggest source of parse conflicts in Ruby. tree-sitter-ruby uses an external scanner.
195186

196-
4. **Optional parentheses**: `puts "hello"` and `puts("hello")` are both valid. This creates ambiguity in the grammar. Lezer's GLR mode (`@ambiguity`) may help, but method calls without parens are the biggest source of parse conflicts in Ruby.
187+
4. **Heredocs** (#10): `<<~RUBY\n code\nRUBY`. The delimiter is arbitrary, can be indented, and can stack. Needs a `ContextTracker`.
188+
189+
5. **String interpolation** is SOLVED -- uses `@local tokens` pattern from `@lezer/javascript`.
197190

198191
## Git Workflow
199192

README.md

Lines changed: 36 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -2,35 +2,58 @@
22

33
Ruby language support for [CodeMirror 6](https://codemirror.net/), built on a [Lezer](https://lezer.codemirror.net/) grammar.
44

5-
## Status
5+
[**Live Demo**](https://jeanpaulsio.github.io/codemirror-lang-ruby/)
66

7-
**Work in progress.** The grammar covers core Ruby constructs (classes, modules, methods, control flow, blocks, lambdas, literals). See CLAUDE.md for the full roadmap of what still needs to be built.
7+
## Install
8+
9+
```bash
10+
npm install github:jeanpaulsio/codemirror-lang-ruby
11+
```
812

913
## Usage
1014

1115
```typescript
16+
import {EditorView, basicSetup} from "codemirror"
1217
import {ruby} from "codemirror-lang-ruby"
13-
import {EditorState} from "@codemirror/state"
14-
import {EditorView} from "@codemirror/view"
1518

16-
const state = EditorState.create({
19+
new EditorView({
1720
doc: 'puts "hello"',
18-
extensions: [ruby()]
21+
extensions: [basicSetup, ruby()],
22+
parent: document.getElementById("editor")!,
1923
})
2024
```
2125

22-
### Install from GitHub
26+
## What's supported
2327

24-
```bash
25-
npm install github:jeanpaulsio/codemirror-lang-ruby
26-
```
28+
- **Definitions**: methods (with params, endless `def f(x) = expr`), classes (with inheritance), modules
29+
- **Control flow**: if/elsif/else, unless, while, until, for/in, case/when, case/in (pattern matching)
30+
- **Error handling**: begin/rescue/ensure/raise
31+
- **Strings**: single-quoted, double-quoted with `#{interpolation}`, `%`-literals (`%w[]`, `%i[]`, `%q()`, `%Q()`)
32+
- **Literals**: integers, floats, symbols, character literals (`?a`), arrays, hashes (rocket `=>` and symbol-key `name:` shorthand), nil, true, false
33+
- **Expressions**: assignment (including `||=`, `&&=`), multiple assignment (`a, b = 1, 2`), method calls (with receiver, args, splat `*args`/`**kwargs`/`&block`), chained calls, binary/unary/ternary operators, lambdas, ranges, conditional modifiers
34+
- **Operators**: proper precedence (`**` > `*`/`/` > `+`/`-` > comparison > logic), safe navigation (`&.`)
35+
- **Variables**: local, `@instance`, `@@class`, `$global`, `Constants`
36+
- **Comments**: line `#` and block `=begin`/`=end`
37+
- **Editor features**: indentation, code folding, bracket closing, keyword autocompletion (31 keywords)
38+
39+
## Known limitations
40+
41+
These are tracked as [open issues](https://github.com/jeanpaulsio/codemirror-lang-ruby/issues) and require external tokenizers:
42+
43+
- No regex literals (`/pattern/` conflicts with division)
44+
- No block attachment to method calls (`items.each { |x| x }`)
45+
- No bare method calls without parens (`puts "hello"`)
46+
- No heredocs (`<<~RUBY`)
47+
- No `%`-literal interpolation or non-bracket delimiters
2748

2849
## Development
2950

3051
```bash
31-
npm install
32-
npm run build
33-
npm test
52+
npm install # Install dependencies
53+
npm run build # Build grammar + bundle to dist/
54+
npm test # Run all 67 grammar tests
55+
npm run lint # TypeScript type check
56+
npm run demo:build # Build the demo page
3457
```
3558

3659
## License

package.json

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,8 @@
1111
"require": "./dist/index.cjs"
1212
},
1313
"sideEffects": false,
14+
"keywords": ["codemirror", "ruby", "lezer", "syntax", "highlighting", "editor"],
15+
"files": ["dist", "src/syntax.grammar"],
1416
"license": "MIT",
1517
"repository": {
1618
"type": "git",

0 commit comments

Comments
 (0)