GenericLexerCallbacks

Generic Lexer Callbacks

Sometimes it could be usefull to discriminate tokens after the lexer scan. For example Prolog discriminates variable and atom identifier base on the first letter (upper case for variables, and lower for atoms) GenericLexer does not allow many pattern for identifier, so we can hook the lexer result before it get send to the parser and subtype tokens as wanted.

public enum Extensions {
	[Lexeme(GenericToken.Extension)] 
	DATE,

	[Lexeme(GenericToken.Double)] 
	DOUBLE,
}

Defining callbacks

All callbacks are static method of a single class. Callbacks are methods taking a Token<IN> as parameter and returning a Token<IN>. Every callback method is tagged with the TokenCallback attribute. this attributes has 1 parameter which is the int value of the tokens enum.

public class TestCallbacks
    {

		public enum CallbackTokens
		{

			 [Lexeme(GenericToken.Identifier)] 
			 IDENTIFIER = 1,

			 VARIABLE = 2,

			 ATOM = 3

		}

        [TokenCallback((int)CallbackTokens.IDENTIFIER)]
        public static Token<CallbackTokens> TranslateIdentifier(Token<CallbackTokens> token)
        {
            if (char.IsUpper(token.Value[0]))
            {
                token.TokenID = CallbackTokens.VARIABLE;
            }
			else {
				token.TokenID = CallbackTokens.ATOM;
			}
            
            return token;
        } 
        
    }

linking lexer to callbacks

public enum Extensions {
	[Lexeme(GenericToken.Extension)] 
	DATE,

	[Lexeme(GenericToken.Double)] 
	DOUBLE,
}