Add Language detection and dispatch logic#210
Merged
Conversation
Imported racket-lexer does not return a good meaningful type for some tokens, so add a extra pass to normalize them.
Add doc-lang module for parseing #lang, #reader, and raw module forms, extract language info, and match against a predefined known language list. Fallback to URI suffix based recognization if not found language header.
6cdh
marked this pull request as draft
April 25, 2026 14:17
dannypsnl
reviewed
Apr 25, 2026
- Stop pass indenter to services, it's not used by them - Improve dignostics module - Skip formatting for not-Sexp languages - Move indenter lookup into doc-lang.rkt
And make test not require rhombus installed.
Also removed get-indenter test because it returns false now for all builtin languages.
dannypsnl
reviewed
Apr 26, 2026
Extract snapshot.rkt and state.rkt from monolithic lexer.rkt, update callers and tests.
Fix semantic-token requests that miss current lexer-derived `#;` comment tokens after expansion has already failed. Add Check-Syntax-Status struct and gate semantic-token wait on running status: - Rename new-trace to check-syntax-finished scheduler signal. Add with-read-safedoc / with-write-safedoc eliminators.
# Conflicts: # doclib/check-syntax.rkt
Handle `.rktd` file policy logic in doc-lang module
Collaborator
Author
|
I think this PR is ready to merge. |
6cdh
marked this pull request as ready for review
June 6, 2026 15:14
dannypsnl
reviewed
Jun 7, 2026
dannypsnl
reviewed
Jun 7, 2026
dannypsnl
approved these changes
Jun 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The current LSP server's behavior for non-Racket or non-Lisp languages is unclear. For example, we don't have clear idea what features support what languages. Almost all features target to Racket, but services that operate on expanded code seems more general purpose.
This PR adds language detection and dispatch logic depends on language. And add docs for behavior for various languages.
TODO:
#lang ...(including#lang reader ...),#reader ...,(module id mod-path ...), if no languag header found, fallback to guess language from special file suffix, like.rhm,.scrbl#;The language header detection is based on the Sexp token tree parser. I didn't use regex because comments can appear before language line, especially Sexp comment. So it builds a small Sexp token tree before the language header. Then recognize the language, if it's in the predefined list, and is a Sexp language, the token tree parser will continue to parse the rest code (continue parsing logic not yet implemented). For non-Sexp language, its Sexp token tree only contains the code before and including the language header. If the language header is missing, the Sexp token tree parser can potentially read the whole document.
The token tree parser can also support non Sexp, but not yet. Because each non Sexp language can have its special syntax, and make the implementation more complex.