Skip to content

lexer_macros.org

Latest commit

 

History

History
136 lines (97 loc) · 5.18 KB

lexer_macros.org

File metadata and controls

136 lines (97 loc) · 5.18 KB

Glint’s Token Macros

Theory

First, let’s look at the first couple “steps” of compilation of a Glint program.

The Glint source code is read, and separated into logical units known as tokens. The tokens are used by the parser (or, more tersely, the syntactic analyser) to form a tree structure that represents the meaning of the Glint program, or what it is meant to be doing.

SOURCE CODE
     |
     V
LEXICAL ANALYSIS
     |
     V
SYNTACTIC ANALYSIS
     |
     V
SEMANTIC ANALYSYS
     |
     V
...

Okay, cool, why did we have to learn all that just to learn about lexer macros? Well, lexer macros are a way to “reach into” the Glint compiler from the source code.

SOURCE CODE<-----.
     |           V
     |           LEXER MACROS
     V           ^
LEXICAL ANALYSIS-°
     |
     V
...

And, truthfully, once a macro has been lexed, it’s application (or expansion) is more like this (where the lexer is operating on itself).

SOURCE CODE
     |
     |     ,-----LEXER MACROS
     V     V     ^
LEXICAL ANALYSIS-°
     |
     V
...

So, why would we want to reach into the inner workings of the language? Most of the time, to do weird or stupid stuff, or to make life easier (and sometimes both!). Also, why not.

Practice

To begin a macro, we use the macro keyword. To end a macro, we use the endmacro keyword.

The following is lexer macros in their simplest form.

macro <name> emits <output> endmacro

Note that lexer macros do not require expression separators, as expressions have not yet been formed at the time of lexical analysis. There are only tokens. So, it could be said that the macro is “eaten” by the lexer (more accurately, the tokens that make up the macro’s definition).

macro empty_macro emits endmacro
macro simple_macro emits 69 endmacro
;; macro emits endmacro; ;; invalid! no name :(

Writing simple_macro anywhere in the program following it’s definition above will macro-expand into the number literal 69.

Macro Parameters

A macro parameter is a token that is discarded upon expansion of the macro, but also enforced that it is there.

;; empty macro with '!' macro parameter
macro foo ! emits endmacro;

foo ! ;; expands to nothing
foo ;; ¡ERROR! Ill-formed macro invocation: got '', expected '!'

This doesn’t appear that useful in our little example, but it can be very powerful to enforce a syntax for something that is not supported in the language (i.e. braces wrapped around something means it is dereferenced, or something). It can also be useful when used in conjunction with macro arguments.

Macro Arguments

A macro may be given named parameters such that they may be duplicated in it’s output.

macro foo $x emits $x $x endmacro;

foo 20 ;; expands to "20 20"

The idea is that, sometimes, you want to be able to take input into your macro to expand into different code based on what the user passes to it, not just a hard-coded sequence of tokens. This does that.

macro foo + $x emits $x endmacro;

foo + 20 ;; expands to "20"
foo 20 ;; ¡ERROR! Ill-formed macro invocation: got '20', expected '+'

Macro Argument Selectors

Macro arguments may be given a single selector following the name identifier.

$<name><selector>
:token
Captures a token. (default)
:expr
Captures a parsed expression rather than a lexed token.
:expr_once
Captures a parsed expression rather than a lexed token, and ensures that the expression is only ever evaluated once, no matter how many times it appears in the macro’s output during expansion.

This becomes very powerful, as macros may operate on parsed expressions rather than lexed tokens. This reaches another layer further into the inner workings of the language, interacting with syntactic analysis.

Hygienic Expansion and Generating Symbols

macro <name> defines <identifiers> emits <output> endmacro

defines allows the macro author to declare that the macro defines a variable. The compiler will give (or generate) that variable a unique name (or symbol) upon each invocation of the macro, such that weird shadowing errors do not occur. For example, if the macro user defines a variable named the same thing that the macro author uses, then the macro expansion would cause a redefinition error. Since nobody wants programs with errors, Glint provides the defines list so that any use of that defined identifier in the macro expansion will be given a unique name within that expansion.

The TL;DR is that defines allows you to create a definitely-unused name within a macro’s output to avoid redefinition errors, and things like that.

macro foo defines x emits x endmacro

foo

This would emit an error: something like Unknown symbol '__L0'. The compiler generates a unique name, __L0 in this case, to replace x with for each invocation. If we called foo again, we’d probably get __L1 for that invocation, and so on and so forth.