Skip to content

lexical.md

Latest commit

 

History

History
484 lines (382 loc) · 8.77 KB

lexical.md

File metadata and controls

484 lines (382 loc) · 8.77 KB
Raw

Lexical Structure

This document describes the lexical structure of AffineScript - how source code is broken into tokens.

Table of Contents

  1. Character Set
  2. Whitespace
  3. Comments
  4. Identifiers
  5. Keywords
  6. Literals
  7. Operators
  8. Punctuation
  9. Special Tokens

Character Set

AffineScript source files are encoded in UTF-8. The lexer supports full Unicode for:

  • Identifiers (letters from any script)
  • String literals
  • Comments
// Greek identifiers
let alpha = 1
let beta = 2

// Emoji in strings (but not identifiers)
let greeting = "Hello! :)"

// Unicode operators
let result = x `add` y

Whitespace

Whitespace characters are used to separate tokens and are otherwise ignored:

Character Name Unicode
Space U+0020
\t Tab U+0009
\n Newline U+000A
\r Carriage Return U+000D 
// All equivalent:
let x = 1
let  x  =  1
let	x	=	1

Comments

Line Comments

Start with // and continue to end of line:

// This is a line comment
let x = 42  // Inline comment

Block Comments

Enclosed in /* */ and can be nested:

/* This is a block comment */

/*
  Multi-line
  block comment
*/

/* Nested /* comments */ are supported */

Documentation Comments (Planned)

/// Documentation for the following item
/// Supports **markdown** formatting
fn documented_function() -> Unit { }

//! Module-level documentation
//! Describes the current module

Identifiers

Rules

  • Start with a letter (any Unicode letter) or underscore
  • Followed by letters, digits, or underscores
  • Case-sensitive
identifier     = (letter | '_') (letter | digit | '_')*
letter         = <Unicode Letter category>
digit          = '0'..'9'

Examples

// Valid identifiers
x
_private
camelCase
snake_case
PascalCase
SCREAMING_CASE
alpha1
_0

// Invalid
0start      // Cannot start with digit
kebab-case  // Hyphen not allowed

Reserved Patterns

  • _ alone is a wildcard pattern, not a valid identifier
  • Identifiers starting with _ suppress unused warnings
let _ = unused_value()   // Wildcard, value discarded
let _temp = compute()    // Valid identifier, unused warning suppressed

Keywords

AffineScript reserves the following keywords:

Value Keywords

Keyword Description
true Boolean true
false Boolean false

Declaration Keywords

Keyword Description
fn Function declaration
let Variable binding
type Type alias
struct Struct definition
enum Enum definition
trait Trait declaration
impl Implementation block
effect Effect declaration
mod Module declaration

Modifier Keywords

Keyword Description
pub Public visibility
mut Mutable reference
own Owned value
ref Borrowed reference
total Total function marker
unsafe Unsafe block

Control Flow Keywords

Keyword Description
if Conditional
else Else branch
match Pattern matching
while While loop
for For loop
in Iterator binding
loop Infinite loop
return Early return
break Loop break
continue Loop continue

Type Keywords

Keyword Description
where Type constraints
as Type coercion
Self Self type

Effect Keywords

Keyword Description
handle Effect handler
resume Resume continuation
perform Perform effect

Module Keywords

Keyword Description
use Import
from Import source

Reserved for Future

Keyword Description
async Async functions
await Await expressions
yield Generator yield
macro Macro definitions
do Do notation
forall Universal quantification
exists Existential quantification

Literals

Integer Literals

// Decimal
42
1_000_000  // Underscores for readability

// Hexadecimal
0xFF
0xDEAD_BEEF

// Binary
0b1010
0b1111_0000

// Octal
0o755
0o777

Floating-Point Literals

3.14
2.0
1e10
1.5e-3
2.5E+10

Character Literals

'a'
'\n'     // Newline
'\t'     // Tab
'\\'     // Backslash
'\''     // Single quote
'\0'     // Null
'\x7F'   // Hex escape (ASCII)
'\u{1F600}'  // Unicode escape

String Literals

"Hello, World!"
"Line 1\nLine 2"
"Tab\there"
"Quote: \"Hello\""
"Unicode: \u{1F600}"

// Multi-line strings
"This is a
multi-line
string"

Raw String Literals (Planned)

r"No \n escapes here"
r#"Can include "quotes" freely"#
r##"Even includes #"# literally"##

String Interpolation (Planned)

let name = "Alice"
let greeting = "Hello, ${name}!"
let math = "1 + 1 = ${1 + 1}"

Operators

Arithmetic Operators

Operator Description
+ Addition
- Subtraction
* Multiplication
/ Division
% Modulo

Comparison Operators

Operator Description
== Equality
!= Inequality
< Less than
> Greater than
<= Less or equal
>= Greater or equal

Logical Operators

Operator Description
&& Logical AND
|| Logical OR
! Logical NOT

Bitwise Operators

Operator Description
& Bitwise AND
| Bitwise OR
^ Bitwise XOR
~ Bitwise NOT
<< Left shift
>> Right shift

Assignment Operators

Operator Description
= Assignment
+= Add-assign
-= Subtract-assign
*= Multiply-assign
/= Divide-assign
%= Modulo-assign
&= AND-assign
|= OR-assign
^= XOR-assign
<<= Left shift-assign
>>= Right shift-assign

Other Operators

Operator Description
-> Function arrow
-{E}-> Effectful arrow
=> Match arm / lambda
.. Range (exclusive)
..= Range (inclusive)
++ String/list concatenation
|> Pipe operator
? Error propagation
@ Pattern binding

Punctuation

Symbol Description
( ) Grouping, tuples, function calls
[ ] Array indexing, type parameters
{ } Blocks, records
, Separator
; Statement terminator
: Type annotation
:: Path separator
. Field access
.. Row variable prefix

Special Tokens

Quantity Annotations

For linear/affine type annotations:

Token Meaning
0 Erased (compile-time only)
1 Linear (exactly once)
omega or w Unrestricted 
fn linear_use(x: 1 Resource) -> Unit { ... }
fn erased_type[0 T]() -> Unit { ... }

Row Variables

// ..name introduces a row variable
fn extend[r](rec: {..r}) -> {x: Int, ..r} {
  {x: 42, ..rec}
}

Wildcards

let _ = ignored_value
let (x, _) = pair
match value {
  Some(_) -> "has value",
  None -> "empty"
}

Operator Precedence

From highest to lowest precedence:

Level Operators Associativity
1 . :: [] () Left
2 ! ~ - (unary) Right
3 * / % Left
4 + - Left
5 << >> Left
6 & Left
7 ^ Left
8 | Left
9 ++ Right
10 == != < > <= >= Left
11 && Left
12 || Left
13 |> Left
14 .. ..= Non-assoc
15 = += etc. Right

Lexer Error Messages

Common lexer errors:

error[E0001]: unexpected character
  --> src/main.affine:1:5
  |
1 | let @x = 5
  |     ^ unexpected '@'
  |
  = help: identifiers cannot contain '@'

error[E0002]: unterminated string literal
  --> src/main.affine:1:9
  |
1 | let s = "hello
  |         ^^^^^^ string literal not closed
  |
  = help: add closing `"` at end of string

error[E0003]: invalid escape sequence
  --> src/main.affine:1:13
  |
1 | let s = "hello\q"
  |               ^^ unknown escape sequence
  |
  = help: valid escapes are: \n, \t, \r, \\, \', \", \0, \xNN, \u{NNNN}

See Also