LSP support for HEEX ~H sigil contents & tree-sitter-heex

[superhawk610]

Resolves #74.

This PR extends the existing Elixir tokenizer to parse the contents of ~H sigils and resolve any function components and expressions contained within so that textDocument/definition works. This allows all common LSP functionality within ~H sigils, commonly found in Phoenix LiveView render functions and function components. It also extends the existing tree-sitter-elixir parser with tree-sitter-heex in nested sub-trees within sigils.

HEEX Parsing

TokenizeHEEX will only output a minimal subset of HEEX contents:

• TokModule / TokIdent for function components
• any Elixir tokens within <% .. %> / { .. } interpolations
• TokHEEXOpenTag / TokHEEXCloseTag for HTML < and </ tags
• TokEOL for end-of-line

All other HEEX contents are ignored during tokenization.

The parser has also been extended to read .foo as a function call to foo() when it occurs immediately after a TokHEEXOpenTag / TokHEEXCloseTag. Function components with a module prefix, e.g. <Foo.bar />, are already handled by the existing parser.

Tree-sitter Integration

The existing tree-sitter tree stored by the document cache is a single tree with a single language. It's been extended with new Tree / TreeNode types that support nested trees with different languages. A typical Elixir document will now be modeled with a 3-level Elixir->HEEX->Elixir tree:

t1 := &Tree{   /* def render(assigns) do\n~H"<div class={foo()} />"\nend */
  Root: nil,
  Trunk: &rootElixirTree,
  Language: LangElixir,
  Branches: {
    NodeId: t2 := &Tree{   /* <div class={foo()} /> */
      Root: t1.TrunkNode(),
      Trunk: &nestedHeexTree,
      Language: LangHeex,
      Branches: {
        NodeId: t3 := &Tree{   /* foo() */
          Root: t2.TrunkNode(),
          Trunk: &nestedElixirTree,
          Language: LangElixir,
          Branches: nil,
        },
      },
    },
  },
}

The TreeNode type is a light wrapper around *tree_sitter.Node that also tracks which Tree the node is a member of, facilitates traversal between nested trees, and wraps common methods like StartByte(), EndByte(), StartPosition(), EndPosition(), and Utf8Text() to correctly handle byte offsets from the root tree.

---

Note

Medium Risk
Touches core document parsing, caching, and all tree-sitter-driven LSP paths; scope is large but covered by new HEEX and tree tests and an index version bump.

Overview
Adds Phoenix HEEX support inside ~H sigils so LSP navigation (go-to-definition, references, hover, completion, rename, highlights) works on LiveView templates, not only plain Elixir.

The tokenizer now lexes ~H bodies via TokenizeHeex (components like <.foo />, <Foo.bar />, {...} / <% %> interpolations) and maps <.name after HEEX open tags to function calls in the reference indexer. ExpressionAtCursor gains HEEX-aware cursor context tests.

Tree-sitter is refactored from a single Elixir parse to a nested treesitter.Tree: Elixir trunk with HEEX branches on ~H quoted_content and Elixir branches on HEEX expression_value, using tree-sitter-heex. DocumentStore keeps per-language parsers and caches this composite tree; LSP handlers call methods on *treesitter.Tree (e.g. FindVariableOccurrences) so positions resolve across nested languages.

Index version is bumped to 13 for the parser/index changes.

^{Reviewed by Cursor Bugbot for commit 713f6f3. Bugbot is set up for automated code reviews on this repo. Configure here.}

[JesseHerrick]

Hey @superhawk610, nice work on this! However, I would suggest that we take a slightly different approach. I think that we should index HEEX (both inside sigils and heex files) as well using the tokenizer and parser. My reasoning:

• We plan on getting rid of tree-sitter at some point. It adds quite a bit of complexity.
• In this current approach, go to definition works but go to references wouldn't. We also gain other features by indexing these files.

I realize that this is more complex to build up front, but I think the end result would be well worth it.

[superhawk610]

I'm working on indexing HEEX with the tokenizer/parser so it's incorporated with the existing process. I'm not confident I can write a full tokenizer for HEEX's grammar, so I'm going with a hybrid approach for now that still shells out to tree-sitter-heex. I'm hoping that once this first step is done, it will provide most of the plumbing and we'll just need a tokenizer for HEEX.

[superhawk610]

At this point go-to definition works, but find all references doesn't since the tree-sitter parsing in variables.go treats HEEX sigil contents as opaque. I've taken a pass at a possible approach through this using nested HEEX tree-sitter trees, but this has grown quite a bit in scope. I think a good next step would be to think this through a bit more thoroughly at a high level and plan the implementation more explicitly. What do you think?

[JesseHerrick]

I'm working on indexing HEEX with the tokenizer/parser so it's incorporated with the existing process. I'm not confident I can write a full tokenizer for HEEX's grammar, so I'm going with a hybrid approach for now that still shells out to tree-sitter-heex. I'm hoping that once this first step is done, it will provide most of the plumbing and we'll just need a tokenizer for HEEX.

I can take a stab at this part if you'd like - I can't promise a specific timeline though. Unfortunately, we can't ship something that shells out to tree-sitter during indexing. It's extremely important that we don't have performance regressions during indexing. There's too much overhead in calling out to tree-sitter. Remote's codebase has >57k files to parse, so we need indexing to be as fast as possible.

[superhawk610]

No expectations at all on timeline! I've gotten this to handle what I need (go-to definition within HEEX sigils), and I don't mind just building from my fork if this ends up being too much work or too niche. Totally agree on avoiding performance regressions, Dexter's speed is its best selling point amongst my peers! 😎

Please feel free to contribute whatever you'd like to this branch, take over or use some/any of this PR, or close it out if it's not in the cards.

[JesseHerrick]

I'll see if I can take a stab at it on this PR sometime this week and we can ship it together. In the meantime, feel free to keep shipping things to this branch and I'll add stuff when I can.

[superhawk610]

I started on a barebones HEEX tokenizer. The line between tokenizer and parser is a bit blurry and this may lean a bit far into parsing. My aim is to get a simple replacement for tree-sitter-heex that can tokenize at minimum: TokModule, TokIdent, TokDot, and recursively tokenize interpolated expressions. If this seems to be moving in the right direction, I may have some more time later in the week to continue.

[superhawk610]

OK, HEEX tokenizer is now on par with the tree-sitter-heex approach, though lacking some edge-case coverage.

• scanInterpolation / scanUntil don't handle early occurrences of the terminator, e.g. <div class={"{}"}> will terminate at the first } rather than the second
• HEEX special forms e.g. <%= for, <%= case, etc. aren't parsed correctly
• malformed HTML can probably get the tokenizer stuck in an infinite loop added FuzzTokenizeHeex and caught a couple degenerate cases, ran for 10 minutes afterward without catching anything else
• find all references still doesn't work, as the treesitter module needs to be updated to traverse the new heex nested sub-tree

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{Reviewed by Cursor Bugbot for commit 429f6bd. Configure here.}

[superhawk610]

Alright @JesseHerrick updated the approach per your request. HEEX files and sigil contents are now fully indexed and the existing tree-sitter-elixir tree recursively parses nested HEEX/Elixir sub-trees. TokenizeHeex aims to parse as little of the HEEX contents as is necessary to find relevant Elixir function calls and interpolations.

Let me know what you think!

[JesseHerrick]

Excellent @superhawk610! I will take a look as soon as I can.

[JesseHerrick]

Blocking: This stops at the first } without tracking nested braces. For example, {choose(%{}, SharedLib.Worker.run())} stops after %{}, so the worker call is never indexed. Please track brace depth when the terminator is } and add this as a regression test.

[JesseHerrick]

Performance: Each interpolation passes the entire remaining template suffix into tokenizeUntil, so preallocating len(source)/8 reserves space for far more tokens than the nested expression can contain. Repeating this for every interpolation produces quadratic allocation growth. With 800 expressions I measured roughly 78.4 MB/op and 4.49 ms/op.

Something like:

tokenCapacity := len(source) / 8
lineCapacity := 64
if terminator != nil {
    tokenCapacity = 16
    lineCapacity = 8
}

tokens := make([]Token, 0, tokenCapacity)
lineStarts := make([]int, 1, lineCapacity)

These are only initial capacities; normal append growth still handles larger expressions. This reduced the benchmark to about 3.86 MB/op and 0.42 ms/op without imposing a hard limit.

[JesseHerrick]

scanInterpolation merges the nested tokens but not result.LineStarts. A multiline {...} expression therefore produces an incomplete parent line-start table and incorrect columns after the expression.

The nested tokenizer already creates this table, so the fix only needs to copy one integer per newline:

for _, lineStart := range result.LineStarts[1:] {
    lineStarts = append(lineStarts, start+lineStart)
}

The [1:] skips the nested table’s initial zero and avoids duplicating the parent line start. This remains O(number of lines), with amortized append growth.

[JesseHerrick]

Blocking for editor buffers: If the closing heredoc delimiter is missing, the scanner reaches EOF and this still subtracts three bytes from contentsEnd. The same issue occurs for single-character delimiters at line 797. Incomplete ~H sigils are common while typing and currently lose their final 1–3 bytes.

Please track whether the delimiter was actually closed. Only trim the delimiter and consume modifier letters after a confirmed close; otherwise preserve the remaining content through EOF. Regression cases should include incomplete ~H""", ~H", and ~H[ inputs.

[JesseHerrick]

Dynamic attributes such as <div {dynamic_attrs()}> fall through this default branch byte-by-byte, so the expression is never tokenized or indexed. Please add a tag-scanner branch for a top-level { after the tag name and reuse scanInterpolation, just as named attribute expressions do. A regression should verify that dynamic_attrs/0 resolves from both normal and self-closing tags.

[JesseHerrick]

HEEX does not evaluate {...} inside <script>, <style>, or descendants of phx-no-curly-interpolation, although <%= ... %> remains active. Currently JavaScript/CSS braces are tokenized as Elixir, producing false definitions and references.

Please track whether the current tag stack disables curly interpolation, while continuing to recognize EEx. The stack should account for nested tags, closing tags, self-closing/void tags, and ordinary < operators inside script bodies. Tests should also verify that the disabling text inside an attribute value does not accidentally disable interpolation.

[JesseHerrick]

Blocking: Local function components are gated by len(injectors) > 0, but their resolution is unrelated to imported or use-injected modules. A module containing <.card /> and defp card/1 with no unrelated injector records no reference.

Please detect local HEEX components whenever there is a current module, before entering the injector-specific bare-call logic. Add coverage for modules both with and without injectors, ensuring the injector case does not produce duplicate references.

[JesseHerrick]

This detection is only reached through the main TokIdent path. Definition and attribute lines jump to extractRefsForLine and then advance past the complete line, so an inline template such as def render(assigns), do: ~H"<.card />" never visits this component token.

Before advancing past those lines, please scan their token range for the same local-component pattern, ideally through a shared helper so inline and block forms cannot diverge. Add equivalent inline and multiline regression cases.

[JesseHerrick]

Performance: For every nested node, this first converts the entire parent HEEX source to a string and only then slices it. Repeated traversal therefore becomes quadratic in copied bytes; an 800-expression variable-occurrence benchmark allocated about 45 MB/op.

TreeNode.StartByte() and EndByte() already return absolute offsets, so this can directly slice the top-level source:

return string(src[tn.StartByte():tn.EndByte()])

Please add a nested-tree offset regression and a variable-occurrence allocation benchmark.

[JesseHerrick]

This branch selection does not fully match the HEEX grammar or runtime semantics:

• EEx control-flow directives use partial_expression_value and ending_expression_value, so conditions and bindings inside <%= ... %> blocks are currently not parsed as nested Elixir.
• Ordinary expression_value nodes inside <script>, <style>, and phx-no-curly-interpolation must not be parsed as Elixir, while EEx there must remain active.

Please support the partial/ending node kinds and filter ordinary expression branches using exact disabled-curly tag/attribute ranges. Tests should cover both missing EEx definitions and false JavaScript/CSS definitions.

[JesseHerrick]

Now that ChildAtPosition descends into HEEX branches, a template assign such as @title reaches the module-attribute path below at line 153. Inside HEEX this is a runtime assign, not an Elixir module attribute. The Definition handler also checks ModuleAttributeAtCursor first, so {@title} can incorrectly jump to an unrelated module-level @title.

Please detect whether the cursor node/position is inside a HEEX tree and exclude it from module-attribute resolution and occurrence collection. Definition should likewise skip its module-attribute fast path for HEEX positions. Add a regression containing both forms in the same module.

[JesseHerrick]

Blocking for rename correctness: Moving occurrence lookup onto the nested Tree allows traversal into HEEX branches, but the existing scope model still assumes one continuous Elixir AST. It does not encode bindings that span separate HEEX directives.

As a result, an item bound by <%= for item <- items do %> is conflated with item after <% end %>; same-named case/fn clauses can cross-contaminate; and :for/:let bindings can escape their containing element.

Please model these as source ranges spanning the relevant HEEX nodes, assign each occurrence to its smallest active range, and filter rename/collision results to the cursor’s range. Regression coverage should include EEx loops, shadowed outer variables, adjacent case clauses, and :for/:let.

[JesseHerrick]

Now that HEEX sigil subtrees are traversed, collectTokenOccurrences also needs to recognize the node kinds produced by tree-sitter-heex. Local component names are emitted as function, and remote component modules as module; the current checks only accept Elixir identifier and alias nodes.

Please treat function like identifier and module like alias, including dotted-segment matching. Add highlight/reference tests for local component open/close tags and both portions of a remote component.