Add semantic highlighting data structures, encoding and default capabilities

[martin-azpillaga]

To facilitate the implementation of semantic highlighting in language servers, I suggest adding the following reusable parts to this plugin:

1. Data structures to hold the default list of semantic token categories, semantic token modifiers and semantic tokens:

data class SemanticToken(val position: Position, val type: SemanticTokenType, val modifiers: List<SemanticTokenModifier>)

enum class SemanticTokenType (val legendName: String) {
    NAMESPACE("namespace"),
    CLASS("class"),
    ENUM("enum"),
    INTERFACE("interface"),
    STRUCT("struct"),
    TYPE_PARAMETER("typeParameter"),
    TYPE("type"),
    PARAMETER("parameter"),
    VARIABLE("variable"),
    PROPERTY("property"),
    ENUM_MEMBER("enumMember"),
    DECORATOR("decorator"),
    EVENT("event"),
    FUNCTION("function"),
    METHOD("method"),
    MACRO("macro"),
    LABEL("label"),
    COMMENT("comment"),
    STRING("string"),
    KEYWORD("keyword"),
    NUMBER("number"),
    REGULAR_EXPRESSION("regexp"),
    OPERATOR("operator")
}

enum class SemanticTokenModifier(val legendName: String, val bit: Int) {
    DECLARATION("declaration", 1),
    DEFINITION("definition", 2),
    READ_ONLY("readonly", 4),
    STATIC("static", 8),
    DEPRECATED("deprecated", 16),
    ABSTRACT("abstract", 32),
    ASYNCHRONOUS("async", 64),
    MODIFICATION("modification", 128),
    DOCUMENTATION("documentation", 256),
    DEFAULT_LIBRARY("defaultLibrary", 512)
}

2. A function to encode a list of semantic tokens to the numerical encoding the LSP uses:

fun encode(tokens: List<SemanticToken>): SemanticTokens {
    var lastLine = 1 // ? To offset that Kolasu lines start at 1
    var lastColumn = 0
    val data = mutableListOf<Int>()
    for (token in tokens) {
        data.addAll(listOf(
            token.position.start.line - lastLine,
            if (token.position.start.line == lastLine) token.position.start.column - lastColumn else token.position.start.column,
            token.position.end.column - token.position.start.column, // ! assumes tokens are in a single line
            token.type.ordinal,
            token.modifiers.sumOf { it.bit }
        ))
        lastLine = token.position.start.line
        lastColumn = token.position.start.column
    }
    return SemanticTokens(data)
}

3. Configuration to enable semantic highlighting capabilities in the server initialization request:

capabilities.semanticTokensProvider = SemanticTokensWithRegistrationOptions().apply {
    legend = SemanticTokensLegend(SemanticTokenType.values().map { it.legendName }, SemanticTokenModifier.values().map { it.legendName });
    full = Either.forLeft(true)
}

[loradd]

🆙

I am fine with this suggestion, perhaps we could merge it before the article on syntax/semantic highlighting comes out so that we do not need to say that the feature is still not public.

What do you think?

[martin-azpillaga]

I think we could also extract this bit of logic to make the APIs a bit friendlier:

override fun semanticTokensFull(params: SemanticTokensParams): CompletableFuture<SemanticTokens> {
    val ast = files[params.textDocument.uri]?.root ?: return CompletableFuture.completedFuture(SemanticTokens())
    val tokens = semanticTokens(ast)
    return CompletableFuture.completedFuture(encode(tokens))
}

open fun semanticTokens(ast: T): List<SemanticToken> = listOf()

Like this, developers just need to populate the list of semantic tokens starting from an ast, which I find more intuitive than dealing with LSP specific constructs.

Let's discuss whether to publish these changes before the article quickly in the next technical meeting.