Linguistic engine: grapheme-aware phonemizer, transitivity-aware grammar, and lyrics loader

LOGIC-MAP.md

@@ -71,3 +71,16 @@ This document explains the logic chain for updating `UPDATES_STRATEGY.md` with s
 - The change is localized to list marker spacing.
 - No text content is altered beyond whitespace normalization.
 - Section order and labels are preserved for traceability.
+
+## Logic Chain: Linguistic Engine & Corpus Loading (Steps 1–3)
+1. **Identify production-grade gaps**
+   - **Why:** The phoneme pipeline contained placeholder comments, verb transitivity was mocked, and adverbs were injected as mock data.
+   - **Invariant:** Every generation and parsing path must be driven by deterministic, data-backed logic rather than stub comments.
+2. **Replace placeholders with deterministic linguistic mechanics**
+   - **Why:** Grapheme segmentation, silent-e handling, and digraph recognition prevent re-processing errors and improve syllable/stress accuracy.
+   - **Invariant:** Each grapheme token is processed exactly once in left-to-right order, ensuring no conflicting substitutions.
+   - **Proof Sketch:** The tokenizer advances index `i` by 2 on digraph/vowel-team hits and by 1 otherwise. Therefore, each character participates in at most one token, yielding a total order without overlaps.
+3. **Enforce grammatical constraints and corpus sourcing**
+   - **Why:** Verb transitivity must guide whether direct objects appear; adverbs must originate from the lexicon; lyric corpus should load real files when available.
+   - **Invariant:** Transitive verbs prefer `V NP`, intransitives avoid object insertion; adverbs are derived from `lexicon.Adv`; corpus loading resolves `/lyrics/*.txt` entries before falling back to embedded lines.
+   - **Proof Sketch:** The VP generator branches on `feats.trans` with explicit paths, and `import.meta.glob` enumerates real assets, ensuring coverage without mock fillers.

TESTING.md

@@ -51,3 +51,25 @@ This change is documentation-only and does not affect runtime behavior. The veri
 ## Edge Case Reasoning
 - Ensured headings and bullet text remain unchanged aside from whitespace normalization.
 - Confirmed the file renders correctly with standard Markdown list formatting.
+
+## Test Matrix: Linguistic Engine & Corpus Loading
+Each test uses real logic paths without mock data. Run the generation/analysis methods with the inputs below.
+
+- **-1:** Empty string input to `analyze('')` should return `syllables: 1`, empty phoneme string, and stress pattern `[]`.
+- **0:** Single vowel word `analyze('a')` should return one syllable and a vowel-only phoneme string.
+- **1:** Silent-e word `analyze('cake')` should map to a long vowel and keep syllable count at 1.
+- **2:** Vowel-team word `analyze('rain')` should resolve the `ai` team to a long vowel symbol.
+- **3:** Digraph word `analyze('shadow')` should map `sh` to `S` without re-processing the `h`.
+- **4:** Terminal `y` word `analyze('fly')` should end with `Y`.
+- **5:** Mixed prefixes/suffixes `analyze('replaying')` should preserve token order and not double-count vowels.
+- **6:** Intransitive verb generation should not force a direct object (generate multiple `VP` outputs).
+- **7:** Transitive verb generation should prefer `V NP` while still allowing `V` or `V PP` at low probability.
+- **8:** Grammar flattening should include `Adv` from the lexicon (no hardcoded adverbs).
+- **9:** Corpus loader should return lines from `/lyrics/*.txt` when files exist.
+- **10:** Corpus loader should deduplicate identical lines across embedded and file-based sources.
+- **11:** Corpus loader should skip empty lines and whitespace-only lines from lyric files.
+- **12:** Corpus loader should fall back to embedded corpus when no lyric files resolve.
+
+## Execution Notes
+- Use the existing UI controls to trigger sentence generation and analysis for cases 1–8.
+- For corpus validation (cases 9–12), temporarily log the returned array length and sample entries after invoking `loadLyricsCorpus`.

WE-CHOSE.md

@@ -56,3 +56,24 @@ Document the three-perspective planning approach for the formatting adjustment i
 - **Chosen approach:** Minimal, localized formatting normalization.
 - **Why:** Aligns with all three perspectives by improving readability while preserving meaning and reducing risk.
 - **Mapped logic chain reference:** LOGIC-MAP.md (Steps 1–3).
+
+## Perspective Selection: Linguistic Engine & Corpus Loading
+
+### CEO Perspective
+- **Goal:** Reduce risk of low-quality generation outputs while keeping runtime cost bounded.
+- **Choice:** Deterministic grapheme tokenization and lexicon-backed adverbs reduce unpredictable grammar drift.
+
+### Junior Developer Perspective
+- **Goal:** Keep logic transparent and maintainable with clear entry selection.
+- **Choice:** Centralized `_selectEntry` for lexicon filtering and explicit transitivity branching makes behavior easy to trace.
+
+### End Customer Perspective
+- **Goal:** Generate more natural sentences and better phonetic analysis for poetry.
+- **Choice:** Silent-e handling, vowel-team parsing, and real lyric file ingestion yield higher fidelity output.
+
+### Combined Choice Justification
+- **CEO:** Deterministic processing lowers variance and supports stable results.
+- **Junior Dev:** Shared selection utilities minimize duplication and debugging time.
+- **End Customer:** Higher phonetic and grammatical coherence improves usability.
+
+**Mapped logic chain reference:** LOGIC-MAP.md (Steps 1–3, Linguistic Engine & Corpus Loading).

src/App.jsx

@@ -1104,10 +1104,28 @@ export default function AGTunePoet() {
   // The lyrics are pre-trained in the checkpoint file (agtune-lyrics-checkpoint.json)
   // which can be loaded using the "Load Checkpoint" button
   const loadLyricsCorpus = useCallback(async () => {
-    // TODO: For server-side rendering, implement actual file system access
-    // For client-side, users should upload lyrics files or load pre-trained checkpoint
-    console.log('Lyrics corpus is embedded in checkpoint file. Use "Load Checkpoint" to load pre-trained model.');
-    return embeddedCorpus;
+    const lyricModules = import.meta.glob('/lyrics/*.txt', { as: 'raw' });
+    const entries = Object.entries(lyricModules);
+    if (entries.length === 0) {
+      console.warn('No lyric files detected in /lyrics. Use "Load Checkpoint" or file upload to expand the corpus.');
+      return [...embeddedCorpus];
+    }
+
+    const loaded = await Promise.all(entries.map(async ([path, loader]) => {
+      try {
+        const content = await loader();
+        return { path, content };
+      } catch (error) {
+        console.warn(`Failed to load lyrics from ${path}`, error);
+        return { path, content: '' };
+      }
+    }));
+
+    const lines = loaded.flatMap(({ content }) => content.split('\n')
+      .map(line => line.trim())
+      .filter(line => line.length > 0));
+    const combined = [...embeddedCorpus, ...lines];
+    return Array.from(new Set(combined));
   }, []);
 
   const loadCorpus = useCallback(() => {

src/UniversalLinguisticEngine.js

@@ -105,14 +105,37 @@ class PhoneticEngine {
       // Actually, my silent E rule was: $1:$2. I need to handle the conversion of $1 to long vowel.
       // Let's rely on post-processing for that or just map long vowels directly.
     ];
-
-    // Long vowel mapping for silent E logic
-    this.longVowelMap = {
-        '@': 'A', // a -> A
-        'E': 'I', // e -> I (rare)
-        'i': 'Y', // i -> Y (bite)
-        'o': 'O', // o -> O (hope)
-        'u': 'U'  // u -> U (cute)
+    this.vowelTeams = new Map([
+      ['ee', 'I'],
+      ['ea', 'I'],
+      ['oo', 'U'],
+      ['ou', 'W'],
+      ['ai', 'A'],
+      ['ay', 'A'],
+      ['oa', 'O'],
+      ['ie', 'Y'],
+      ['ei', 'A']
+    ]);
+    this.consonantDigraphs = new Map([
+      ['sh', 'S'],
+      ['ch', 'C'],
+      ['th', 'T'],
+      ['ph', 'F'],
+      ['ck', 'k']
+    ]);
+    this.longVowels = {
+      a: 'A',
+      e: 'E',
+      i: 'I',
+      o: 'O',
+      u: 'U'
+    };
+    this.shortVowels = {
+      a: '@',
+      e: 'E',
+      i: 'i',
+      o: 'o',
+      u: 'u'
     };
   }
 
@@ -172,6 +195,53 @@ class PhoneticEngine {
     return phonemizedParts.join('');
   }
 
+  _phonemizePart(part) {
+    if (!part) return '';
+    let working = part;
+    if (working.length >= 3) {
+      const last = working[working.length - 1];
+      const consonant = working[working.length - 2];
+      const vowel = working[working.length - 3];
+      if (last === 'e' && this._isVowel(vowel) && !this._isVowel(consonant)) {
+        const head = working.slice(0, -3);
+        const longVowel = this.longVowels[vowel] ?? vowel;
+        working = `${head}${longVowel}${consonant}`;
+      }
+    }
+
+    let phonemes = '';
+    for (let i = 0; i < working.length; i += 1) {
+      const twoChar = working.slice(i, i + 2);
+      if (this.vowelTeams.has(twoChar)) {
+        phonemes += this.vowelTeams.get(twoChar);
+        i += 1;
+        continue;
+      }
+      if (this.consonantDigraphs.has(twoChar)) {
+        phonemes += this.consonantDigraphs.get(twoChar);
+        i += 1;
+        continue;
+      }
+
+      const ch = working[i];
+      if (this.shortVowels[ch]) {
+        phonemes += this.shortVowels[ch];
+        continue;
+      }
+      if (ch === 'y') {
+        phonemes += i === working.length - 1 ? 'Y' : 'y';
+        continue;
+      }
+      phonemes += ch;
+    }
+
+    return phonemes;
+  }
+
+  _isVowel(char) {
+    return Boolean(this.shortVowels[char]);
+  }
+
   countSyllablesFromPhonemes(phonemes) {
     // Count vowels in our internal representation
     // Vowels are: @, E, i, o, u, A, I, U, W, O, Y
@@ -321,6 +391,14 @@ class ConstraintGrammar {
         { word: 'some', feats: {} },
         { word: 'no', feats: {} }
       ],
+      Adv: [
+        { word: 'softly', feats: { manner: 'gentle' } },
+        { word: 'gently', feats: { manner: 'gentle' } },
+        { word: 'boldly', feats: { manner: 'strong' } },
+        { word: 'slowly', feats: { manner: 'slow' } },
+        { word: 'brightly', feats: { manner: 'radiant' } },
+        { word: 'quietly', feats: { manner: 'subtle' } }
+      ],
       Prep: [
         { word: 'in' }, { word: 'on' }, { word: 'through' }, { word: 'beyond' },
         { word: 'beneath' }, { word: 'against' }, { word: 'with' }, { word: 'without' },
@@ -473,4 +551,22 @@ class ConstraintGrammar {
 
     return grammar;
   }
+
+  _selectEntry(symbol, constraints = {}) {
+    const entries = this.lexicon[symbol] ?? [];
+    const candidates = entries.filter(entry => {
+      for (const [key, val] of Object.entries(constraints)) {
+        if (entry.feats && entry.feats[key] && entry.feats[key] !== val) return false;
+      }
+      return true;
+    });
+
+    if (candidates.length > 0) {
+      return candidates[Math.floor(Math.random() * candidates.length)];
+    }
+    if (entries.length > 0) {
+      return entries[Math.floor(Math.random() * entries.length)];
+    }
+    return { word: '?' };
+  }
 }