Content assist does not work for common prefix rules

[cdietrich]

given the grammar

entry Model:
    (persons+=Person | greetings+=Greeting)*;

Person:
    'person' name=ID;

Greeting:
    GreetingByName | GreetingByAlias;


GreetingByName: 
    'hello' person=[Person:ID];

GreetingByAlias:
    'hello' alias=ID (':' data=PersonRef ('{'
        
    '}')?);

PersonRef:
    person=[Person:ID];

content assist does not work at

person Aaaaa

hello a : |

not at

person Aaaaa

hello a : A|

[cdietrich]

this one still seems to be present with @msujew latest fix

[cdietrich]

test('xxxxxx', async () => {
        const grammar = `
            grammar Test

            entry Model:
    (persons+=Person | greetings+=Greeting)*;

Person:
    'person' name=ID;

Greeting:
    GreetingByName | GreetingByAlias;


GreetingByName: 
    'hello' person=[Person:ID];

GreetingByAlias:
    'hello' alias=ID (':' data=PersonRef ('{'
        
    '}')?);

PersonRef:
    person=[Person:ID];

            hidden terminal WS: /\\s+/;
            terminal ID returns string: /[_a-zA-Z][\\w_]*/;
        `;
        const services = await createServicesForGrammar({ grammar,
            module: {
                lsp: {
                    // CompletionProvider that shows ALL completions for testing purposes
                    CompletionProvider: (services: LangiumServices) =>
                        new (class extends DefaultCompletionProvider {
                            protected override filterKeyword(_context: CompletionContext, _keyword: GrammarAST.Keyword): boolean {
                                return true;
                            }

                            protected override continueCompletion(_items: CompletionItem[]): boolean {
                                return true;
                            }

                            protected override performNextTokenCompletion(_document: LangiumDocument, _text: string, _offset: number, _end: number): boolean {
                                return true;
                            }
                        })(services),
                },
                references: {
                    Linker: (services: LangiumServices) => new DefaultLinker(services),
                    NameProvider: () => new DefaultNameProvider(),
                    ScopeProvider: (services: LangiumServices) => new DefaultScopeProvider(services),
                    ScopeComputation: (services: LangiumServices) => new DefaultScopeComputation(services),
                    References: (services: LangiumServices) => new DefaultReferences(services)
                }

            }});
        const completion = expectCompletion(services);
        const text = `person Aaaaa

hello a : A<|> {}`;
        await completion({
            text,
            index: 0,
            expectedItems: ['{', 'Aaaaa']
        });
    });

(if i did not mess sth up)

[msujew]

Yep, I've noticed that as well. Though it has nothing to do with the follow element computation (at least not directly). For some reason, our parser seems to think that hello a : is a GreetingByName with an invalid : suffix. I'll take a look at this as well soon.

[Kheirlb]

Looking at @dhrubomoy's comment #2103 (comment), I agree that there is something fishy with the token index.

For me, completion works as long as the this.nextTokenIndex in LangiumCompletionParser evalatues to tokens.length - 2 which points at the ID token BEFORE the : token, and all of the "NextFeature" matching works as expected (I think). Completion fails whenever this.nextTokenIndex points to tokens.length - 1, which ends up being the : token, which fails at the Regex here

langium/packages/langium/src/lsp/completion/follow-element-computation.ts

Line 337 in 378d475

return terminalRegex(rule).test(token.image);

since : is not a terminal.

Not sure if this is any help.

[msujew]

Like I said, it's an issue in the parser. I just had some time to take a closer look at it. I believe I was able to pinpoint the issue to the lookahead behavior in the presence of "incomplete" input. Adopters updating their chevrotain-allstar version will automatically pull the fix. #2146 Is just there to pull up the version for all adopters and to add a regression test for this.