[RFC] Semantic Highlighting Support for Twirl Templates (for syntax highlighting)

[AsadHumayun]

Hi Play community,

Intro

I've recently been working on trying to bring semantic highlighting support for Twirl templates. This is in an effort to help better the tooling ecosystem surrounding Play's Twirl templating engine. I did write an initial TextMate grammar (which is essentially a set of cleverly-applied regexes) to help bring some syntax highlighting (now merged into the Metals vscode extension); however, due to the nature of regex, it was not possible to give completely accurate colouring and styling due to how interleaved Twirl allows Scala and HTML to become. Semantic highlighting would resolve this, and is designed to enrich the colouring applied by regex by getting information from the compiler. You can find more information here. You can find more information on semantic tokens on this part of the LSP spec.

Progress

Approach 1: Traverse Twirl AST

This version worked in terms of traversing the Twirl AST and emitting its tokens; however, the issue with this method was that Scala's PresentationCompiler lacked the Scala context required to produce meaningful semantic tokens. We needed something that could preserve the semantic meaning of Scala fragments.

Approach 2: Locate Compiled Twirl Files, Extract Tokens & Map them Back to the Twirl Source File

I have been able to make good progress, but I am stuck on what seems to be the last step:

• getting the correct lengths of each semantic token. Since the TwirlCompiler generates boilerplate, and then Twirl template does not include this boilerplate, it can cause the lengths of certain tokens to become mismatched.

The current flow is:

1. get compiled Twirl templates (scala files)
2. language server will search for and extract compiled files
3. get semantic tokens for the compiled scala file (not the twirl file)
4. map through each semantic token, check to see whether its position falls into a region of a MATRIX entry
   the logic behind this was that MATRIX entries contain "symbols of interest" - things that are also in the Twirl file
5. if NO, the token can be ignored and filtered out
6. if YES, construct a new semantic token with the new Twirl positions (we receive the positions from the Scala file), retaining the token type (thereby preserving colouring/styling)
   this is the part I am struggling on.
7. send back to editor to apply styling

I am struggling with 6. The issue is that, because when Twirl files are compiled, TwirlCompiler adds various boilerplate methods, structuring, inline source mapping comments, etc, that cause the length property that we receive from the semantic tokens obtained in step 3 inaccurate. This can lead to inaccurate or incomplete colouring in the editor.

Also note that in order to enable the condition check in step 4, I had to write a small patch to TwirlCompiler to enrich the source mapping MATRIX data.
You can find this here. No tests exist as of now, just wanted to confirm that I can get semantic highlighting working this way first. Also wanted to gauge sentiment around potentially making changes to TiwrlCompiler - is this something possible/desired?

Approach 3: Virtual Files + Twirl AST

After taking a bit of a break from the problem, I have come up with potentially another way of getting things working:

• Use TwirlParser to traverse AST
  • while doing this, also: 1) build MATRIX source mappings, and 2) build the synthetic virtual file that will be sent to PresentationCompiler for nodes. 1 pass for both.
• Extract semantic tokens from this file
• Delta-decode them to get absolute positions
• Check if they fall within the range of a MATRIX entry
• Make a new semantic token, updating the positions
• Emit these tokens to apply colouring.

// imports from build.sbt TwirlImports key
import x.y.Y;
import x.z.Z;

class TwirlView(<template params go here>) {
   def compute = {
        Y()
    }
}

Also, around the build.sbt key "TwirlKeys.templateImports", this will need work on BSP to be properly linked up and exposed on the Metals side. This would involve getting a new BSP spec in, community approvals on their side, and then time for the rest of the tooling to uptake new package releases, as well as the time for me to learn their codebase and write everything up. We will still need to extract this somehow for the time being - maybe just read the compiled Scala file's imports.

This approach takes kind of a hybrid approach, sort of combining my first 2 attempts/ideas at this. This will still be roughly another 2-4 weeks worth of work (possibly much more for getting solutions merged into BSP), and I have already been at this since the end of March.

Note that this approach avoids adding Twirl's boilerplate into the generated Scala, as well as inline source mapping comments, which should avoid the lengths issue of that killed Approach #2.
We are also maintaining our own MATRIX mapping, which will be different from that of TwirlCompiler's as in it will be simpler. I might rename this to something else later to prevent confusion. None of Approach 3 has been written up yet.

Any and all feedback/ideas/suggestions is welcome.
I would love to contribute towards tooling for colouring for Twirl templates and otherwise general ecosystem support. I am also looking into getting LSP diagnostics support for Twirl written up and merged into Metals, which should hopefully be much simpler... hopefully...