|
| 1 | +# Self-Hosting Compiler Implementation Notes |
| 2 | + |
| 3 | +## Missing Language Features for lexer.eph |
| 4 | + |
| 5 | +The lexer implementation reveals several features that Ephapax needs before it can be self-hosting: |
| 6 | + |
| 7 | +### 1. **String Operations** (CRITICAL) |
| 8 | +```ephapax |
| 9 | +// Need these as built-in functions or stdlib: |
| 10 | +fn string_len(s: String): I32 |
| 11 | +fn string_char_code(s: String, idx: I32): I32 // Get char as ASCII |
| 12 | +fn string_substr(s: String, start: I32, len: I32): String |
| 13 | +fn string_from_char(c: I32): String // ASCII to string |
| 14 | +fn string_concat(s1: String, s2: String): String |
| 15 | +``` |
| 16 | + |
| 17 | +**Status**: Currently missing from Ephapax |
| 18 | +**Priority**: HIGH - Required for lexer/parser |
| 19 | + |
| 20 | +### 2. **List/Array Operations** |
| 21 | +```ephapax |
| 22 | +// Need list/vector operations: |
| 23 | +fn append<T>(list: [T], item: T): [T] |
| 24 | +fn list_len<T>(list: [T]): I32 |
| 25 | +fn list_get<T>(list: [T], idx: I32): T |
| 26 | +fn list_map<T, U>(list: [T], f: T -> U): [U] |
| 27 | +fn list_fold<T, Acc>(list: [T], init: Acc, f: (Acc, T) -> Acc): Acc |
| 28 | +``` |
| 29 | + |
| 30 | +**Status**: Need to implement growable arrays |
| 31 | +**Priority**: HIGH - Required for token lists, AST nodes |
| 32 | + |
| 33 | +### 3. **Record Syntax Clarification** |
| 34 | +```ephapax |
| 35 | +// Are these equivalent? |
| 36 | +type Token = { kind: TokenKind, lexeme: String, ... } |
| 37 | +let tok = { kind: TokInt, lexeme: "42", line: 1, col: 1 } |
| 38 | +
|
| 39 | +// Or do we need explicit constructor? |
| 40 | +type Token = Token(TokenKind, String, I32, I32) |
| 41 | +let tok = Token(TokInt, "42", 1, 1) |
| 42 | +``` |
| 43 | + |
| 44 | +**Status**: Need to clarify syntax |
| 45 | +**Priority**: MEDIUM - Affects ergonomics |
| 46 | + |
| 47 | +### 4. **Equality for Strings** |
| 48 | +```ephapax |
| 49 | +if word == "fn" then TokKeyword else ... |
| 50 | +``` |
| 51 | + |
| 52 | +**Status**: Need `==` operator for strings |
| 53 | +**Priority**: HIGH - Required for keyword matching |
| 54 | + |
| 55 | +### 5. **Tuples/Pair Returns** |
| 56 | +```ephapax |
| 57 | +fn lex_token(lexer: Lexer): (Token, Lexer) = ... |
| 58 | +let (tok, lexer2) = lex_token(lexer) in ... |
| 59 | +``` |
| 60 | + |
| 61 | +**Status**: Need tuple support |
| 62 | +**Priority**: HIGH - State threading pattern |
| 63 | + |
| 64 | +### 6. **Pattern Matching on Records** |
| 65 | +```ephapax |
| 66 | +case tok of |
| 67 | +| { kind: TokEof, ... } -> ... |
| 68 | +| { kind: TokIdent, lexeme: name, ... } -> ... |
| 69 | +``` |
| 70 | + |
| 71 | +**Status**: Need pattern destructuring |
| 72 | +**Priority**: MEDIUM - Can workaround with field access |
| 73 | + |
| 74 | +### 7. **Recursive Types** |
| 75 | +```ephapax |
| 76 | +type Expr = |
| 77 | + | ELambda(String, Expr) // Expr references itself |
| 78 | + | EApp(Expr, Expr) |
| 79 | + | ELet(String, Expr, Expr) |
| 80 | +``` |
| 81 | + |
| 82 | +**Status**: Need support for recursive ADTs |
| 83 | +**Priority**: HIGH - Required for AST |
| 84 | + |
| 85 | +## Implementation Order |
| 86 | + |
| 87 | +### Phase A: Core Language Extensions (1-2 weeks) |
| 88 | + |
| 89 | +**Before we can continue self-hosting:** |
| 90 | + |
| 91 | +1. **String library** (runtime.eph or WASM imports) |
| 92 | + - Implement basic string operations |
| 93 | + - Add to stdlib or as built-ins |
| 94 | + |
| 95 | +2. **List/Vector type** (stdlib) |
| 96 | + - Growable array implementation |
| 97 | + - Linear ownership for elements |
| 98 | + - Basic operations (append, get, len) |
| 99 | + |
| 100 | +3. **Tuple syntax** |
| 101 | + - Parser support for `(T, U)` |
| 102 | + - Destructuring in `let` bindings |
| 103 | + - Type checker support |
| 104 | + |
| 105 | +4. **Record pattern matching** |
| 106 | + - Destructure records in case expressions |
| 107 | + - Wildcard patterns (`...`) |
| 108 | + |
| 109 | +### Phase B: Continue Self-Hosting (2-3 weeks) |
| 110 | + |
| 111 | +5. **Complete parser.eph** (depends on Phase A) |
| 112 | +6. **Complete typechecker.eph** |
| 113 | +7. **Complete wasm_codegen.eph** |
| 114 | +8. **Complete main.eph** |
| 115 | + |
| 116 | +### Phase C: Bootstrap (1 week) |
| 117 | + |
| 118 | +9. **Use Rust compiler to compile Ephapax compiler** |
| 119 | +10. **Use Ephapax compiler to compile itself** |
| 120 | +11. **Verify bit-identical output** |
| 121 | + |
| 122 | +## Alternative Approach: Simplified Lexer |
| 123 | + |
| 124 | +We could simplify the lexer to avoid missing features: |
| 125 | + |
| 126 | +```ephapax |
| 127 | +// Simplified lexer using only available features |
| 128 | +fn lex_char(source: String, pos: I32): TokenKind = |
| 129 | + // Character-by-character processing |
| 130 | + // No string operations needed (just char codes) |
| 131 | + ... |
| 132 | +
|
| 133 | +fn lex_all(source: String): I32 = |
| 134 | + // Process entire source in one pass |
| 135 | + // No token list - print as we go |
| 136 | + ... |
| 137 | +``` |
| 138 | + |
| 139 | +**Tradeoff**: Less clean, but works with current language |
| 140 | + |
| 141 | +## Decision Point |
| 142 | + |
| 143 | +**Option 1**: Pause self-hosting, implement missing features in Rust compiler first |
| 144 | +- Cleaner self-hosting code |
| 145 | +- More complete language |
| 146 | +- Longer path to self-hosting |
| 147 | + |
| 148 | +**Option 2**: Continue with simplified approach using only available features |
| 149 | +- Messier code |
| 150 | +- Get to self-hosting faster |
| 151 | +- Refactor later |
| 152 | + |
| 153 | +**Option 3**: Implement minimal runtime in .eph first (runtime.eph) |
| 154 | +- String ops as Ephapax functions |
| 155 | +- Lists as linked lists (inefficient but works) |
| 156 | +- Pure Ephapax (no WASM imports) |
| 157 | + |
| 158 | +## Recommended: Option 1 |
| 159 | + |
| 160 | +**Implement missing features in Rust compiler first**, then complete self-hosting with clean code. |
| 161 | + |
| 162 | +### Priority Features to Add: |
| 163 | + |
| 164 | +1. **String operations** (1 day) |
| 165 | + - Add to `ephapax-stdlib` crate |
| 166 | + - WASM runtime helpers |
| 167 | + |
| 168 | +2. **Growable list type** (2 days) |
| 169 | + - Implement in `ephapax-stdlib` |
| 170 | + - Linear ownership tracked |
| 171 | + |
| 172 | +3. **Tuple syntax** (2 days) |
| 173 | + - Parser + type checker support |
| 174 | + - Codegen |
| 175 | + |
| 176 | +4. **Pattern matching enhancements** (1 day) |
| 177 | + - Record destructuring |
| 178 | + - Wildcards |
| 179 | + |
| 180 | +**Total**: ~1 week to enable self-hosting |
| 181 | + |
| 182 | +## Next Steps |
| 183 | + |
| 184 | +1. **Pause lexer.eph completion** |
| 185 | +2. **Add string/list support to Rust compiler** |
| 186 | +3. **Add tuple syntax** |
| 187 | +4. **Resume self-hosting with complete features** |
| 188 | + |
| 189 | +OR |
| 190 | + |
| 191 | +1. **Continue with simplified lexer** |
| 192 | +2. **Accept messier code initially** |
| 193 | +3. **Refactor after bootstrap** |
| 194 | + |
| 195 | +--- |
| 196 | + |
| 197 | +**Current Status**: lexer.eph written but requires language extensions |
| 198 | +**Blocker**: String operations, lists, tuples |
| 199 | +**Resolution**: TBD by user preference |
0 commit comments