essays

Author: acharneski

essays/NAM/README.md

@@ -0,0 +1,219 @@
+## Synthesis
+
+# Unified Synthesis: Numbers as Machines — A Generator-Based Numerics Library
+
+## Preamble
+
+Four perspectives — Technical/Implementation, Mathematical/Theoretical, Software Engineering/API, and Performance/Hardware — have independently analyzed this proposal. This synthesis identifies convergent conclusions, maps genuine tensions, and produces actionable recommendations. The overall consensus level is assessed at **0.74**, reflecting strong agreement on the theoretical foundation and significant divergence on implementation readiness and specific design choices.
+
+---
+
+## I. Points of Strong Convergence (High Consensus)
+
+### 1. The Core Abstraction Is Sound
+
+All four perspectives affirm that the fundamental primitive — `step : State → (digit, State)` — is mathematically well-founded and computationally elegant. The mathematical analysis identifies this as a **final coalgebra** for the digit-stream functor, which is the correct categorical framing. The software engineering perspective calls it "the ideal foundation for composability." The technical perspective confirms LLVM can optimize it well for the finite-automaton case. The performance perspective notes it fits naturally in registers for small state.
+
+**Consensus conclusion:** The coinductive digit-generator model is a genuine and sound contribution. It is not merely a metaphor — it corresponds to established mathematics (computable analysis, stream transducers, coalgebra theory).
+
+### 2. The Two-Tier Complexity Split Is Real and Must Be Reflected in the Design
+
+Every perspective independently arrives at a fundamental bifurcation:
+
+| Tier | Examples | State Properties | Fork Cost |
+|---|---|---|---|
+| **Automaton class** | Rationals, algebraic irrationals | Fixed-size, inline | O(1) — true struct copy |
+| **Series class** | Transcendentals (π, e, ζ(3)) | Grows with computation depth | O(log n) — must deep-copy accumulators |
+
+The technical perspective proposes explicit `FiniteNumVM` and `SeriesNumVM` structs. The mathematical perspective notes that p-adic rationals are strictly easier than real transcendentals (digit commitment is local in p-adics, non-local in reals). The performance perspective shows that the `void *payload` pointer breaks value semantics for the series class. The software engineering perspective identifies this as the source of the memoization/forking interaction problem.
+
+**Consensus conclusion:** The single `NumVMState` struct is architecturally insufficient. A two-tier ABI is required, with explicit and honest fork semantics for each tier.
+
+### 3. Carry Propagation Is the Central Unsolved Problem
+
+Three of four perspectives independently flag carry propagation as the most serious practical obstacle. The technical perspective calls it "fundamentally non-local." The software engineering perspective notes it "glosses over the most difficult composability problem." The mathematical perspective cites the digit boundary problem (numbers near `0.999... = 1.000...`) as creating potential non-termination.
+
+All three converge on the same solution: **signed-digit (redundant) representation**, where digits lie in `{-(b-1), ..., b-1}`, making addition carry-free digit-by-digit. This is the standard solution in iRRAM and related exact real arithmetic systems.
+
+**Consensus conclusion:** The proposal must adopt signed-digit representation for the real-number arithmetic layer. This is not optional — without it, the addition combinator is not a well-defined local operation.
+
+### 4. The BBP/Skip-Ahead Primitive Is Valuable but Overstated
+
+The mathematical and technical perspectives both affirm that the BBP formula explanation is a valid *restatement* of known results, not a new mechanistic explanation. The original mechanism has been understood since Bailey-Borwein-Plouffe (1997). The performance perspective identifies `skip(n, state)` as the most JIT-friendly operation in the model. The software engineering perspective calls it "novel and useful" as a library primitive.
+
+**Consensus conclusion:** The `skip` primitive is a genuine contribution to the library API. The theoretical framing as "automaton-codec resonance" is insightful but should be presented as a reformulation, not a discovery. The skip function cannot be derived automatically — it requires per-constant manual implementation.
+
+### 5. The Proposal Understates Existing Prior Art
+
+The mathematical perspective explicitly notes that exact real arithmetic (iRRAM, MPFR, Haskell's `Data.Number.CReal`), computable analysis (Weihrauch complexity), and automatic sequences (Allouche-Shallit) all provide prior foundations. The software engineering perspective recommends studying these systems before finalizing the design. The technical perspective notes that the "Interval Refinement Engine" is essentially interval arithmetic combined with ERA, studied since the 1980s.
+
+**Consensus conclusion:** The proposal should be repositioned from "novel framework" to "novel synthesis and implementation strategy" that builds on established mathematical foundations. The genuine novelty lies in the unified ABI, the codec/base separation, and the skip primitive — not in the underlying mathematics.
+
+---
+
+## II. Significant Tensions and Conflicts
+
+### Tension 1: "Forking Is a Struct Copy" vs. Unbounded Precision
+
+This is the central architectural contradiction, identified by three perspectives:
+
+- **Technical:** "The document cannot simultaneously claim 'forking is a struct copy' and have payload point to mutable state."
+- **Performance:** "`void *payload` immediately breaks the pure value copy forking claim."
+- **Software Engineering:** "Forking semantics interact badly with memoization."
+
+**Resolution:** Accept that fork cost is tier-dependent. For the automaton class, fork is O(1) and is a true struct copy. For the series class, fork is O(log n) in the computation depth and requires explicit deep copy of accumulator state. Document this honestly. The "forking is a struct copy" claim should be scoped to the automaton tier only.
+
+### Tension 2: Lazy Evaluation vs. LLVM Optimization
+
+The software engineering and performance perspectives are in partial conflict about LLVM's role:
+
+- **Performance:** "LLVM excels at optimizing eager, statically-shaped computations. Lazy generator graphs with dynamic demand patterns are harder."
+- **Technical:** "LLVM's claims are largely correct for the finite-automaton case" but "overstated" for series class.
+
+**Resolution:** Both are correct for different regimes. LLVM optimization applies to *statically-known, compile-time-fixed* expression trees. For *runtime-constructed* expression trees (parsing, dynamic matrix construction), a JIT compilation step (expression-tree compilation to a single LLVM function) is required. The proposal should specify which regime it targets and provide the JIT path for the dynamic case.
+
+### Tension 3: Memoization Policy
+
+The software engineering perspective identifies three incompatible memoization strategies (full, none, partial) without a clear recommendation. The technical perspective proposes an explicit `MemoVM` wrapper. The performance perspective recommends a bounded LRU cache sized to L2/L3 capacity.
+
+**Resolution:** These are compatible at different layers. The correct architecture is:
+1. **No implicit global memoization** (avoids hidden state)
+2. **Explicit `MemoVM` wrapper** with configurable cache size (technical perspective)
+3. **Bounded LRU implementation** sized to hardware cache (performance perspective)
+4. **User-facing API** that exposes `.streaming()` vs. `.cached(max_digits=N)` modes (software engineering perspective)
+
+### Tension 4: Mathematical Rigor vs. Implementation Pragmatism
+
+The mathematical perspective rates several claims as "incorrect" (digit extraction always terminates, "first mechanistic explanation" of BBP) while the technical perspective rates the overall framework at 0.72 confidence and calls it "worth building." The software engineering perspective rates ergonomics as "essentially absent" while still identifying genuine innovations.
+
+**Resolution:** These are not in conflict — they address different questions. The mathematical critique targets *claims*, not the *framework*. The technical and engineering critiques target *implementation gaps*, not the *concept*. The synthesis is: the concept is sound, several specific claims are wrong or overstated, and the implementation gaps are real but addressable.
+
+---
+
+## III. Critical Gaps Requiring Resolution Before Implementation
+
+Ranked by severity (all four perspectives contributing):
+
+### Gap 1 (Critical): Digit Commitment and Non-Termination
+The interval refinement engine will fail to terminate for inputs near digit boundaries. This is not an edge case — it affects any computation whose result is near a representable boundary. **Required:** Formal treatment of when digit commitment is guaranteed, adoption of signed-digit representation to make addition local, and explicit documentation of the non-termination cases.
+
+### Gap 2 (Critical): ABI Aliasing from `void *payload`
+The current `NumVMState` struct creates aliased mutable state on fork. **Required:** Two-tier ABI with inline state for automaton class and explicit deep-copy semantics for series class.
+
+### Gap 3 (High): Comparison Semantics
+Equality of real numbers is undecidable. The proposal does not address this. **Required:** Interval-based predicate API (`definitely_less_than`, `agrees_with(digits=N)`) rather than exact equality.
+
+### Gap 4 (High): Tail Bound Oracle Implementation
+The convergence bound requirement places mathematical sophistication demands on library users. **Required:** Built-in tail bound oracles for all standard constants and functions, with a documented (not hidden) interface for user-defined constants.
+
+### Gap 5 (Medium): Carry Propagation Locality
+Without signed-digit representation, addition is not a well-defined local combinator. **Required:** Adopt Avizienis signed-digit representation as the internal arithmetic layer.
+
+### Gap 6 (Medium): User-Facing API
+The proposal describes an execution substrate, not a usable library. **Required:** Stratified API design (primitive / combinator / user layers) with operator overloading, precision contexts, and explicit memoization policy.
+
+---
+
+## IV. What Is Genuinely Novel and Worth Preserving
+
+Across all four perspectives, the following are identified as genuine contributions not reducible to prior work:
+
+1. **The unified ABI for digit generators** — a single protocol spanning rationals through transcendentals, enabling composition without impedance mismatch
+2. **The codec/base separation** — cleanly separating number identity from representation
+3. **The `skip(n, state)` primitive as a first-class ABI element** — making BBP-style fast-forward a library primitive rather than a one-off optimization
+4. **Memory complexity as generator state dimension** — a novel complexity metric for numerical computation
+5. **P-adic numbers as periodic automata in the same framework** — a natural fit that existing libraries do not exploit
+6. **The MUX tree / coalgebraic foundation** — the correct mathematical framing for coinductive digit streams
+
+---
+
+## V. Unified Implementation Roadmap
+
+### Phase 1: Automaton Tier (High Confidence, Build Now)
+
+```c
+typedef struct {
+    uint32_t base;
+    uint32_t phase;
+    uint64_t state[4];  // sufficient for degree-4 algebraic
+} AutomatonVM;
+```
+
+- Rationals (periodic automata, true value semantics)
+- Quadratic irrationals (second-order recurrences)
+- P-adic numbers (periodic in p-adic base)
+- Base conversion codecs
+- Skip-ahead via matrix exponentiation
+- Full LLVM optimization applies; fork = struct copy is correct here
+
+### Phase 2: Series Tier (Medium Confidence, Requires Research)
+
+```c
+typedef struct {
+    uint32_t base;
+    uint32_t index;
+    const SeriesSpec *spec;     // immutable, safe to alias
+    ArbitraryInt *accum;        // mutable, deep-copy on fork
+    ArbitraryInt *error_bound;  // mutable, deep-copy on fork
+} SeriesVM;
+```
+
+- Classical transcendentals (π, e, log 2) with built-in tail bound oracles
+- Signed-digit internal representation for carry-free addition
+- Explicit fork cost documentation: O(log n) in computation depth
+- Bounded LRU memoization cache
+
+### Phase 3: User-Facing API (Required for Adoption)
+
+```python
+# Precision context
+with precision_context(digits=50):
+    result = sin(pi/4) + sqrt(2)
+
+# Explicit comparison
+x.agrees_with(y, digits=20)
+x.definitely_less_than(y)  # Returns True/False/Unknown
+
+# Explicit memoization policy
+x = sqrt(2).cached(max_digits=1000)
+x = pi.streaming()  # O(1) space, sequential access only
+
+# Forking with honest cost
+x, y = pi.fork()  # Documents O(log n) cost
+```
+
+### Phase 4: JIT Compilation (Performance-Critical Applications)
+
+- Expression-tree compilation: `compile(expr_tree) → NumVMFn`
+- Eliminates function pointer dispatch for runtime-constructed expressions
+- Struct-of-arrays layout for vector/matrix operations
+- Batched digit computation API for SIMD vectorization
+
+---
+
+## VI. Claims Requiring Correction
+
+| Original Claim | Corrected Status |
+|---|---|
+| "Forking is a struct copy" | True for automaton tier only; O(log n) for series tier |
+| "LLVM handles inlining across the generator graph" | True for static, compile-time-known trees; requires JIT for dynamic trees |
+| "First mechanistic explanation of BBP" | False — prior art in Bailey-Borwein-Plouffe (1997) and subsequent work |
+| "Digit extraction always terminates" | False for reals near digit boundaries — a fundamental non-termination case |
+| "Randomness is encrypted determinism" | Philosophical position, not mathematical theorem; conflates distinct concepts |
+| Complexity hierarchy table as established fact | Should be presented as conjectures; quantitative claims (e.g., "3-4 fields" for π) lack formal proof |
+| "Tail bound oracle" as an engineering concern | It is a mathematical barrier requiring per-constant convergence proofs |
+
+---
+
+## VII. Overall Assessment
+
+**Consensus Level: 0.74**
+
+The proposal describes a theoretically sound and genuinely useful framework. The automaton tier is ready to build with high confidence. The series tier has fundamental tensions that require resolution — particularly carry propagation, fork semantics, and digit commitment — before it can be called a production system. The user-facing API is essentially absent and must be designed before the library can achieve adoption.
+
+The framework's most important contribution is the *unified protocol* that allows rationals, algebraic numbers, p-adic numbers, and transcendentals to compose through the same interface. This is a real advance over existing systems (MPFR, iRRAM, mpmath) which treat these as separate domains.
+
+The path from compelling research prototype to production numerics library requires: (1) honest two-tier ABI, (2) signed-digit arithmetic for carry locality, (3) interval-based comparison semantics, (4) stratified user API, and (5) engagement with the computable analysis literature that provides the mathematical foundations already developed for exactly this problem domain.
+
+**Build it. Fix the ABI. Engage the prior art. Scope the claims accurately.**
+