`xdoc_find_token` resolves to adjacent punctuation instead of the string literal when the caret sits right before an unspaced quote

[Hanatarou]

Summary

Several features that rely on xdoc_find_token() (in R/utils.R) — at least textDocument/prepareTypeHierarchy, textDocument/hover, textDocument/definition, textDocument/selectionRange and textDocument/documentHighlight — fail to resolve the intended token when the caret sits exactly on the boundary between a punctuation character and a string literal that has no whitespace between them (e.g. ("ClassName" or ,"ClassName"). In that situation the function returns the punctuation token instead of the STR_CONST, and callers that only accept a whitelist of token types (e.g. detect_type_definition() in R/type_hierarchy.R, which only accepts SYMBOL, SYMBOL_FUNCTION_CALL, STR_CONST) then report "not supported" even though the caret is visually right next to a valid symbol.

Reproduction

setClass("BaseEntity",
  representation(id = "character")
)

1. Place the caret immediately before the opening quote of "BaseEntity" (i.e. right after the ().
2. Trigger textDocument/prepareTypeHierarchy at that position.
3. Expected: resolves the STR_CONST "BaseEntity" and returns the type definition (same as if the caret were one column to the right, inside the string).
4. Actual: prepare_type_hierarchy_reply returns NULL ("this symbol isn't supported").

Now compare with a case where there IS whitespace before the quote:

setClass("AdminUser",
  contains = "User",
  representation(department = "character")
)

Place the caret immediately before the opening quote of "User" (right after = ). This resolves correctly, because of the space character described below.

Root cause

xdoc_find_token <- function(x, line, col) {
    xpath <- glue("//*[not(*)][(@line1 < {line} or (@line1 = {line} and @col1 <= {col})) and
                    (@line2 > {line} or (@line2 = {line} and @col2 >= {col}-1))]",
        line = line, col = col)
    xml_find_first(x, xpath)
}

This selects every leaf node (terminal token) whose [col1, col2] span touches the caret's
gap position, then takes the first match in document order via xml_find_first. When two
tokens are directly adjacent with no whitespace between them (e.g. ( immediately followed
by "), the gap position between them satisfies the XPath condition for both tokens
simultaneously — col2 of the left token equals col1 - 1 of the right token, so both
col2 >= col-1 (left token) and col1 <= col (right token) hold at that exact gap. Since the
left token (the punctuation) always appears earlier in document order, it always wins the tie,
regardless of which one is semantically relevant at the caret position.

When there IS a whitespace character separating the two tokens (whitespace is not itself a
parse-tree node), no such tie exists — only the string literal's STR_CONST node satisfies the
condition at that gap, so resolution is unambiguous and correct. This is why the exact same
"caret before the opening quote" scenario behaves differently depending purely on whether the
preceding argument character is punctuation glued to the string or is followed by a space.

Confirmed with getParseData():

code <- 'setClass("BaseEntity", representation(id = "character"))'
pd <- getParseData(parse(text = code, keep.source = TRUE))
print(pd[pd$token %in% c("'('","STR_CONST"), c("token","text","col1","col2")])

   token           text col1 col2
     '('              (    9    9
STR_CONST "BaseEntity"   10   21

( ends at column 9, STR_CONST starts at column 10 — directly adjacent, no gap between them,
so the caret position "column 10" (right before the quote) satisfies both nodes' XPath
conditions, and ( wins by document order.

Suggested fix

When xml_find_first returns more than one candidate at a boundary gap (or, equivalently, when
querying all matches instead of just the first), prefer a non-punctuation / content-bearing
token (STR_CONST, SYMBOL, SYMBOL_FUNCTION_CALL, SYMBOL_SUB, etc.) over a bare operator or
bracket token when both touch the same gap. Concretely, this could be done by ordering the
XPath result set by a priority computed from local-name(), or by filtering out a short
denylist of punctuation-only token types ('(', ')', ',', '[', ']', '{', '}',
OP-LEFT-PAREN, OP-RIGHT-PAREN, OP-COMMA, etc.) before taking the first result, falling back
to the punctuation token only if nothing else matches.

Environment

• languageserver version: (please confirm against packageVersion("languageserver");
  xdoc_find_token on the master branch as of this writing has the same tie-breaking logic)
• R version: 4.3.3
• OS: Windows (reproduced), pattern is OS-independent (pure parse-tree issue, not
  platform-specific)

Related

This is independent from #738 (the contains= type-hierarchy resolution bug)
— that one is in R/type_hierarchy.R's find_s4_supertypes/find_s4_subtypes/
find_refclass_supertypes/find_refclass_subtypes; this one is in the shared
xdoc_find_token helper in R/utils.R and affects every feature that calls it, not just type
hierarchy.