Skip to content

Commit 02b5da1

Browse files
author
Test
committed
docs: add comprehensive compiler roadmap
1 parent f125e21 commit 02b5da1

1 file changed

Lines changed: 328 additions & 0 deletions

File tree

‎COMPILER-ROADMAP.md‎

Lines changed: 328 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,328 @@
1+
# WokeLang Compiler Roadmap
2+
3+
## Current Status (2026-01-31)
4+
5+
**Overall Progress: 60%** (was 25%)
6+
7+
### ✅ Completed (100%)
8+
9+
1. **AST Definition** - Complete type system for WokeLang syntax
10+
2. **Lexer** - Token generation with logos
11+
3. **ABI Layer** - Idris2 ABI with dependent type proofs
12+
4. **FFI Layer** - Zig FFI implementation with C compatibility
13+
5. **Bytecode Specification** - Complete OpCode instruction set (50+ instructions)
14+
6. **VM Optimizer** - Bytecode optimization passes
15+
16+
### 🚧 Implemented (Functional but needs expansion)
17+
18+
#### VM Compiler (80%)
19+
**Location**: `src/vm/compiler.rs`
20+
21+
**What Works:**
22+
- Variable declarations (`remember x = expr`)
23+
- Assignments (`x = expr`)
24+
- Binary operations (`+`, `-`, `*`, `/`, `%`, `==`, `!=`, `<`, `<=`, `>`, `>=`, `and`, `or`)
25+
- Unary operations (`-`, `not`)
26+
- Function calls
27+
- Conditionals (`when`/`otherwise`)
28+
- Loops (`repeat N times`)
29+
- Return statements
30+
- Local variable management
31+
- Array literals
32+
33+
**What's Missing:**
34+
- Lambda/closure compilation
35+
- Pattern matching (`decide based on`)
36+
- Consent blocks (`only if okay`)
37+
- Attempt blocks (`attempt safely`/`or reassure`)
38+
- Worker spawn
39+
- Unit measurements
40+
- Gratitude blocks
41+
- Module imports
42+
- Type definitions
43+
44+
#### VM Machine (85%)
45+
**Location**: `src/vm/machine.rs`
46+
47+
**What Works:**
48+
- Stack-based execution
49+
- All arithmetic operations
50+
- All comparison operations
51+
- All logical operations
52+
- Control flow (jumps, conditionals)
53+
- Local and global variables
54+
- Function calls with stack frames
55+
- Array creation
56+
- Error handling and recovery
57+
58+
**What's Missing:**
59+
- Closure execution
60+
- Channel operations (Go-style concurrency)
61+
- Result type handling (Okay/Oops)
62+
- Record/object operations
63+
- Index operations
64+
- String concatenation
65+
- Built-in function calls (print, toString, etc.)
66+
67+
#### Interpreter (50%)
68+
**Location**: `src/interpreter/mod.rs`
69+
70+
**What Works:**
71+
- Environment management
72+
- Function registration
73+
- Basic function execution
74+
- Closure support (via Value type)
75+
76+
**What's Missing:**
77+
- Statement execution (all types)
78+
- Expression evaluation (all types)
79+
- Stdlib integration
80+
- Security/capability checking
81+
82+
#### Parser (40%)
83+
**Location**: `src/parser/mod.rs`
84+
85+
**What Exists:**
86+
- Parser structure
87+
- Error types
88+
- Token stream management
89+
90+
**What's Missing:**
91+
- Complete syntax parsing for all WokeLang constructs
92+
- Currently returns stub functions only
93+
- Needs full implementation
94+
95+
#### Type Checker (30%)
96+
**Location**: `src/typechecker/mod.rs`
97+
98+
**What Exists:**
99+
- Type system definition (Int, Float, String, Bool, Array, Function, etc.)
100+
- Type environment
101+
- Fresh type variable generation
102+
103+
**What's Missing:**
104+
- Type inference algorithm
105+
- Constraint solving
106+
- Type checking for all expressions
107+
- Type checking for all statements
108+
- Unit-of-measure type system
109+
- Result type checking
110+
111+
#### REPL (60%)
112+
**Location**: `src/repl.rs`
113+
114+
**What Works:**
115+
- Interactive loop
116+
- History support
117+
- Basic input/output
118+
- Integration with lexer/parser/interpreter
119+
120+
**What's Missing:**
121+
- Better error messages
122+
- Multi-line input
123+
- Tab completion
124+
- Syntax highlighting
125+
126+
### 📦 Standard Library (75%)
127+
128+
**Implemented Modules:**
129+
- ✅ `array` - Array operations (length, push, pop, slice, etc.)
130+
- ✅ `chan` - Go-style channels for concurrency
131+
- ✅ `io` - File I/O with consent checking
132+
- ✅ `math` - Mathematical operations
133+
- ✅ `net` - HTTP operations with consent
134+
- ✅ `string` - String manipulation
135+
- ✅ `time` - Time operations
136+
- ❌ `json` - Needs implementation
137+
138+
### 🎯 Next Steps (Priority Order)
139+
140+
#### Phase 1: Complete Core Functionality (2-3 weeks)
141+
142+
**1. Parser Implementation**
143+
- [ ] Parse function definitions
144+
- [ ] Parse all statement types
145+
- [ ] Parse all expression types
146+
- [ ] Parse consent blocks
147+
- [ ] Parse gratitude blocks
148+
- [ ] Parse worker definitions
149+
- [ ] Parse type definitions
150+
- [ ] Error recovery and reporting
151+
152+
**2. Interpreter Implementation**
153+
- [ ] Execute all statement types
154+
- [ ] Evaluate all expression types
155+
- [ ] Integrate with stdlib
156+
- [ ] Security capability checking
157+
- [ ] Proper error handling
158+
159+
**3. Type Checker Implementation**
160+
- [ ] Hindley-Milner type inference
161+
- [ ] Constraint generation
162+
- [ ] Constraint solving
163+
- [ ] Unit-of-measure checking
164+
- [ ] Result type checking
165+
- [ ] Type error messages
166+
167+
**4. VM Compiler Extensions**
168+
- [ ] Lambda/closure compilation
169+
- [ ] Pattern matching compilation
170+
- [ ] Consent block compilation
171+
- [ ] Worker spawn compilation
172+
- [ ] Module system compilation
173+
174+
**5. VM Machine Extensions**
175+
- [ ] Closure execution
176+
- [ ] Channel operations
177+
- [ ] Result type operations
178+
- [ ] Record/object operations
179+
- [ ] Index operations
180+
- [ ] Built-in functions
181+
182+
#### Phase 2: Testing & Documentation (1-2 weeks)
183+
184+
**6. Test Suite**
185+
- [ ] Lexer tests
186+
- [ ] Parser tests (all syntax forms)
187+
- [ ] Type checker tests
188+
- [ ] Compiler tests (bytecode generation)
189+
- [ ] VM tests (execution correctness)
190+
- [ ] End-to-end tests
191+
- [ ] WokeLang feature tests (consent, units, gratitude)
192+
- [ ] Performance benchmarks
193+
194+
**7. Documentation**
195+
- [ ] Language specification
196+
- [ ] Compiler architecture docs
197+
- [ ] Bytecode instruction reference
198+
- [ ] Type system documentation
199+
- [ ] Standard library reference
200+
- [ ] Example programs
201+
- [ ] Tutorial
202+
203+
**8. Tooling**
204+
- [ ] Syntax highlighting (VS Code, vim)
205+
- [ ] Language server protocol (LSP)
206+
- [ ] Debugger
207+
- [ ] Profiler
208+
209+
#### Phase 3: Advanced Features (2-3 weeks)
210+
211+
**9. CLI Enhancement**
212+
- [ ] `woke compile <file>` - Compile to bytecode
213+
- [ ] `woke run-vm <file>` - Run on VM
214+
- [ ] `woke disasm <file>` - Show bytecode
215+
- [ ] `woke typecheck <file>` - Type check only
216+
- [ ] `woke build <file> -o <output>` - Native compilation
217+
- [ ] `woke build-wasm <file>` - WebAssembly compilation
218+
219+
**10. Optimization**
220+
- [ ] Dead code elimination
221+
- [ ] Constant folding
222+
- [ ] Inline expansion
223+
- [ ] Tail call optimization
224+
- [ ] Register allocation (if targeting native)
225+
226+
**11. Code Generation Backends**
227+
- [ ] WASM backend (via wasm-pack)
228+
- [ ] Native backend (via LLVM/Cranelift)
229+
- [ ] JavaScript backend (for browser)
230+
231+
#### Phase 4: Ecosystem (1-2 weeks)
232+
233+
**12. Package Manager**
234+
- [ ] Package format
235+
- [ ] Dependency resolution
236+
- [ ] Package registry
237+
238+
**13. Build System**
239+
- [ ] Project configuration
240+
- [ ] Multi-file compilation
241+
- [ ] Incremental compilation
242+
243+
## Architecture Overview
244+
245+
```
246+
Source Code (.woke)
247+
↓
248+
Lexer (logos)
249+
↓
250+
Tokens (Vec<Spanned<Token>>)
251+
↓
252+
Parser
253+
↓
254+
AST (Abstract Syntax Tree)
255+
↓
256+
Type Checker
257+
↓
258+
Typed AST
259+
↓
260+
┌─────────┬──────────┬──────────┐
261+
↓ ↓ ↓ ↓
262+
Interpreter VM WASM Native
263+
(tree-walk) (bytecode) (codegen) (codegen)
264+
```
265+
266+
## Bytecode Pipeline (Current Focus)
267+
268+
```
269+
AST
270+
↓
271+
BytecodeCompiler (src/vm/compiler.rs)
272+
↓
273+
CompiledProgram (bytecode + constants)
274+
↓
275+
Optimizer (src/vm/optimizer.rs)
276+
↓
277+
Optimized Bytecode
278+
↓
279+
VirtualMachine (src/vm/machine.rs)
280+
↓
281+
Execution Result
282+
```
283+
284+
## Testing the Current Compiler
285+
286+
```bash
287+
# Build the compiler
288+
cargo build --release
289+
290+
# The following features work:
291+
# - Basic arithmetic
292+
# - Variable declarations
293+
# - Conditionals
294+
# - Loops
295+
# - Function calls (partial)
296+
297+
# Coming soon:
298+
# - Full parser
299+
# - Type checking
300+
# - Complete language support
301+
```
302+
303+
## Contributing
304+
305+
See the task list above for areas that need work. The highest priority items are:
306+
307+
1. **Parser** - Convert tokens to full AST
308+
2. **Type Checker** - Implement type inference
309+
3. **VM Extensions** - Closures, channels, pattern matching
310+
4. **Tests** - Comprehensive test coverage
311+
312+
## Resources
313+
314+
- **ABI/FFI Documentation**: `ABI-FFI-README.md`
315+
- **Bytecode Specification**: `src/vm/bytecode.rs`
316+
- **Example Programs**: `examples/`
317+
- **Language Specification**: `docs/` (TODO)
318+
319+
## Estimated Timeline to v0.1
320+
321+
- **Parser + Type Checker**: 2-3 weeks
322+
- **VM Extensions**: 1-2 weeks
323+
- **Testing + Docs**: 1-2 weeks
324+
- **Total**: 4-7 weeks to working compiler
325+
326+
## Questions?
327+
328+
See `STATE.scm` for current project state or check the task list in the development session.

0 commit comments

Comments
 (0)