Skip to content

Commit 3227543

Browse files
jeanpaulsioclaude
andauthored
Fix 35 grammar bugs and add comprehensive test coverage (575 tests) (#22)
* feat: fix 35 grammar bugs across 8 root causes Resolve all bugs from grammar-bug-list.md traced to 8 root causes: - Root Cause B: whitespace/semicolons — newlines and semicolons as skip tokens - Root Cause A: hash symbol-key shorthand — KeywordParam/KeywordArg with colonOp - Root Cause C: bare method calls — Rails DSL methods, raise as bareMethod - Root Cause G: rescue clause — bare rescue (=> e), multiple exception classes - Root Cause D: operators — bitwise/shift via external tokenizers, token precedence DFA fix (AssignOp above ArithOp), greaterThanTokenizer for >/>=/>>/>>= family - Root Cause H: block pass — BlockPassExpression in ParenExpression, BlockParam GLR - Root Cause E: scoped class names — className with :: separators, GLR on Constant - Root Cause F: statements as expressions — if/unless/case/begin in expression via GLR Additional fixes: - BUG-009: statements as expressions (x = if/case/begin) - BUG-012: Rails DSL bare methods (validates, has_many, before_action, etc.) - BUG-017: operator method definitions (def <=>, def [], def []=) - BUG-018: while/until/for do — @extend<Identifier,"do"> with !loopBody precedence - BUG-022: setter method definitions (def name=) - BUG-024: match operators (=~, !~, ===) - BUG-031: block pass (&block) in method calls - BUG-034: expanded symbol tokenizer (:foo?, :foo!, :"quoted", operator symbols) 91 fileTests pass, lint clean. 87/89 comprehensive bug reproductions pass. * fix: bare rescue in method body and lambda without params - MethodBody now supports implicit rescue/ensure (no begin wrapper needed) - Lambda ParamList is now optional, fixing `-> { expr }` syntax - Fixes BUG-010 (bare rescue) and BUG-012d (scope with lambda) - 94 tests passing * feat: grammar fixes, folding, and comprehensive test suites Fix Constant(args) as method call (e.g. URI(url)), self.method= setter assignment, folding for def/class/module blocks, trailing dot method chain continuation indent, and super keyword highlighting. Add 6 test suites: grammar (105), integration (10), highlight (31), indent (51), fold (16), completion (10) — 223 total tests. Update README with expanded known limitations. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat: comprehensive test coverage (575 passing) and indent fixes - Add 247 indent tests covering all 32 sections of ruby-indent-cases.md - Add 139 highlight tests covering all Part 1 spec sections (1.1-1.11) - Add 39 grammar accuracy + performance tests (Parts 7 and 9) - Expand fold tests to 25 cases covering Part 2 (2.1-2.11) - Expand completion tests with Part 6 context cases - Fix INDENT_AFTER to recognize `private def`, `protected def`, etc. - Fix deindent scanner to track {}/do-end nesting separately Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
1 parent c1efd3b commit 3227543

16 files changed

Lines changed: 5248 additions & 100 deletions

README.md

Lines changed: 9 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -50,22 +50,24 @@ Tested against popular open source Ruby projects (large, representative files):
5050
- **Bare method calls**: `puts "hello"`, `require "json"`, `attr_reader :name`, `include Comparable` (25 common Ruby methods)
5151
- **Variables**: local, `@instance`, `@@class`, `$global`, `Constants`
5252
- **Comments**: line `#` and block `=begin`/`=end`
53-
- **Editor features**: smart indentation (119 test cases), code folding, bracket closing, keyword autocompletion (31 keywords)
53+
- **Editor features**: smart indentation, code folding, bracket closing, keyword autocompletion (31 keywords)
5454

5555
## Known limitations
5656

57-
- Blank lines inside class/method bodies can produce spurious error nodes (~38% of parse errors)
58-
- Multi-arg bare method calls (`raise ArgumentError, "msg"`) not fully supported
59-
- Compound assignment operators (`+=`, `-=`, `|=`) in some contexts
60-
- Heredoc and `%`-literal bodies are opaque tokens (no interpolation highlighting inside)
61-
- `<<` left-shift operator not distinguished from heredoc start
57+
- **Heredocs as arguments or in chains**`foo(<<~SQL)` and `<<~HEREDOC.strip` don't parse correctly. Heredocs assigned to variables (`x = <<~SQL`) work fine. The limitation is architectural: the heredoc body starts on the next line but the closing `)` or `.method` needs to be parsed on the current line, which requires a split-token approach not yet implemented.
58+
- **Leading `::` scope resolution**`::TopLevel` as a standalone expression doesn't parse. Scoped references with a receiver (`Foo::Bar`, `Foo::Bar::Baz`) work fine.
59+
- **Advanced pattern matching** — Basic `case/in` with array patterns and pin operators work. Hash patterns (`in {name: String => n}`), guard clauses (`in x if x > 0`), and find patterns (`in [*, 2, *]`) are not yet supported.
60+
- **Heredoc and `%`-literal bodies** are opaque tokens (no interpolation highlighting inside).
61+
- **Inline rescue as a standalone expression**`value = foo rescue nil` works (rescue in assignments), but `foo rescue bar` as a standalone expression outside of assignment context is not supported.
62+
- **Newline as statement separator** — Ruby uses newlines to separate statements, but the grammar is whitespace-insensitive. An expression followed by `if` on the next line may be parsed as a conditional modifier (e.g., `x = 1\nif cond` parses as `x = (1 if cond)`).
63+
- **Symbol-key shorthand in method calls**`foo(name: "value")` with symbol-key syntax is not yet supported. Use rocket syntax `foo(:name => "value")` as a workaround.
6264

6365
## Development
6466

6567
```bash
6668
npm install # Install dependencies
6769
npm run build # Build grammar + bundle to dist/
68-
npm test # Run all 89 grammar tests
70+
npm test # Run all 105 grammar tests
6971
npm run lint # TypeScript type check
7072
npm run demo:build # Build the demo page
7173
```

src/highlight.ts

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,8 @@
11
import {styleTags, tags as t} from "@lezer/highlight"
22

33
export const rubyHighlighting = styleTags({
4-
"def class module end do begin rescue ensure raise return yield": t.keyword,
4+
"def class module end do begin rescue ensure return yield super": t.keyword,
5+
"BareMethodCall/raise": t.keyword,
56
"if elsif else unless case when then while until for in break next": t.keyword,
67
"and or not": t.keyword,
78
"nil true false self": t.atom,
@@ -39,6 +40,7 @@ export const rubyHighlighting = styleTags({
3940
". &.": t.derefOperator,
4041
", ;": t.separator,
4142
'ArithOp "-" "*" "**"': t.arithmeticOperator,
43+
'"~" "|" "&" "^" shiftLeftOp shiftRightOp': t.bitwiseOperator,
4244
CompareOp: t.compareOperator,
4345
"AssignOp": t.updateOperator,
4446
LogicOp: t.logicOperator,

src/index.ts

Lines changed: 31 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -43,7 +43,7 @@ const rubyCompletion = completeFromList([
4343

4444
// Block-opening keyword at line start (or after `=` for assignment forms)
4545
const INDENT_KEYWORD = /\b(def|class|module|if|unless|while|until|for|case|begin)\b/
46-
const INDENT_AFTER = /^\s*(def|class|module|if|unless|while|until|for|case|begin)\b/
46+
const INDENT_AFTER = /^\s*(?:(?:private|protected|public|private_class_method|public_class_method)\s+)?(def|class|module|if|unless|while|until|for|case|begin)\b/
4747
// Also matches `x = if condition`, `x = begin`, `@foo ||= begin`, etc.
4848
const INDENT_ASSIGN = /[=]\s*(if|unless|case|begin)\b/
4949
const INDENT_END = /\b(do)\s*(\|[^|]*\|)?\s*(#.*)?$|\{\s*(\|[^|]*\|)?\s*(#.*)?$/
@@ -64,7 +64,7 @@ const DEINDENT_CLOSE = /^\s*[\}\]\)]/
6464
const INTERMEDIATE = /^\s*(else|elsif|when|in|rescue|ensure)\b/
6565

6666
// Line ends with a continuation indicator (trailing operator, comma, backslash)
67-
const CONTINUATION = /(\+|-|\*|&&|\|\||\\|,)\s*(#.*)?$/
67+
const CONTINUATION = /(\+|-|\*|&&|\|\||\\|,|\.)\s*(#.*)?$/
6868

6969
// Line starts with a dot (method chaining continuation)
7070
const LEADING_DOT = /^\s*\./
@@ -81,14 +81,35 @@ function rubyIndentService(cx: IndentContext, pos: number): number | undefined {
8181

8282
// Current line starts with a deindent keyword → find the right level
8383
if (DEINDENT_ON.test(text)) {
84+
const isEnd = /^\s*end\b/.test(text)
8485
// Scan backwards to find the matching opening keyword at the right nesting level
8586
let depth = 0
87+
let braceDepth = 0
8688
for (let i = lineNum - 1; i >= 1; i--) {
8789
const prev = doc.line(i).text
90+
// Track } closers for brace-style blocks
91+
if (/^\s*\}/.test(prev)) braceDepth++
8892
if (/^\s*end\b/.test(prev)) depth++
89-
else if (INDENT_AFTER.test(prev) || INDENT_ASSIGN.test(prev) || INDENT_END.test(prev)) {
93+
else if (INDENT_AFTER.test(prev) || INDENT_ASSIGN.test(prev)) {
9094
if (depth === 0) return cx.lineIndent(doc.line(i).from)
9195
depth--
96+
} else if (INDENT_END.test(prev)) {
97+
// Distinguish brace-style ({) from do-style openers
98+
if (/\{\s*(\|[^|]*\|)?\s*(#.*)?$/.test(prev)) {
99+
// Brace-style opener — pair with } closers
100+
if (braceDepth > 0) {
101+
braceDepth--
102+
} else if (!isEnd) {
103+
// Non-end keywords (else, rescue, etc.) can match brace openers
104+
if (depth === 0) return cx.lineIndent(doc.line(i).from)
105+
depth--
106+
}
107+
// `end` never closes a `{`, so skip this opener
108+
} else {
109+
// do-style opener
110+
if (depth === 0) return cx.lineIndent(doc.line(i).from)
111+
depth--
112+
}
92113
}
93114
}
94115
return 0
@@ -230,6 +251,13 @@ export const rubyLanguage = LRLanguage.define({
230251
}),
231252
foldNodeProp.add({
232253
"ClassBody ModuleBody MethodBody Block DoBlock BeginBlock": foldInside,
254+
"MethodDef ClassDef ModuleDef"(tree, state) {
255+
// Fold from end of first line to before `end`
256+
const firstLine = state.doc.lineAt(tree.from)
257+
const lastLine = state.doc.lineAt(tree.to)
258+
if (firstLine.number >= lastLine.number) return null
259+
return {from: firstLine.to, to: lastLine.from}
260+
},
233261
"IfStatement UnlessStatement WhileStatement UntilStatement ForStatement CaseStatement"(tree) {
234262
return {from: tree.from, to: tree.to}
235263
},

0 commit comments

Comments
 (0)