Skip to content

SCRIPTS.md

Latest commit

 

History

History
299 lines (224 loc) · 6.88 KB

SCRIPTS.md

File metadata and controls

299 lines (224 loc) · 6.88 KB
Raw

AG-TUNE Training Scripts Reference

Available Scripts

npm run train

Purpose: Train the model on all lyrics from the lyrics/ folder

What it does:

  • Loads 15 lyrics files (847+ lines)
  • Builds vocabulary of 1,675+ unique words
  • Computes co-occurrence embeddings
  • Trains Kernel PCA for 8D emotional space
  • Runs 100 epochs of TD-lambda learning
  • Saves checkpoint to agtune-lyrics-checkpoint.json

Output:

Vocabulary size: 1675 words
Training Kernel PCA...
Training TD value estimator for 100 epochs...
Average Reward: 4.8368
✓ Checkpoint saved

Duration: ~20-30 seconds

npm run verify

Purpose: Verify the model retains information indefinitely

What it does:

  • Loads the trained checkpoint
  • Runs 3 save/load cycles
  • Verifies vectors are preserved exactly
  • Tests vocabulary completeness (99%+)
  • Validates data integrity checksums

Output:

✓ Cycle #1 passed - all data preserved
✓ Cycle #2 passed - all data preserved
✓ Cycle #3 passed - all data preserved
✓ Vocabulary coverage excellent (99.41%)
✓ Vocabulary integrity preserved

Duration: ~5-10 seconds

npm run continuous

Purpose: Demonstrate continuous training across multiple sessions

What it does:

  • Session 1: Trains on 3 lyrics files
  • Saves checkpoint
  • Session 2: Loads checkpoint, adds 3 more files
  • Saves updated checkpoint
  • Session 3: Loads checkpoint, adds 3 more files
  • Verifies all original words retained

Output:

Session 1: 135 lines, 523 words
Session 2: 334 lines total, 1009 words
Session 3: 563 lines total, 1346 words
✓ "ghost" retained across 3 sessions
✓ "shadow" retained across 3 sessions
✓ "death" retained across 3 sessions

Duration: ~30-40 seconds

npm run test-engine

Purpose: Test core algorithm components

What it does:

  • Tests Kernel PCA implementation
  • Tests Floyd cycle detector
  • Tests FFT meter analyzer
  • Validates all algorithms work correctly

Output:

✓ Kernel PCA working
✓ Cycle detector working
✓ Meter analyzer working
All core algorithms functional! ✓

Duration: < 1 second

npm run dev

Purpose: Start the web UI development server

What it does:

  • Starts Vite dev server
  • Opens web interface at http://localhost:5173
  • Enables training and poetry generation in browser

Features:

  • Upload custom training corpus
  • Train model with configurable epochs
  • Generate poetry with trained model
  • Visualize emotional space and weights
  • Save/load checkpoints

npm run build

Purpose: Build for production deployment

What it does:

  • Bundles React app with Vite
  • Optimizes for production
  • Outputs to dist/ folder

Training Workflow Examples

Basic Training

# 1. Train on all lyrics
npm run train

# 2. Verify retention
npm run verify

# 3. Start web UI
npm run dev

Continuous Learning

# 1. Initial training
npm run train

# 2. Test continuous training
npm run continuous

# 3. Verify everything works
npm run verify

Development Testing

# 1. Test algorithms
npm run test-engine

# 2. Train model
npm run train

# 3. Start UI
npm run dev

File Outputs

agtune-lyrics-checkpoint.json

  • Size: ~1.7 MB
  • Contents: Vocabulary, embeddings, emotional space, PCA params, TD weights
  • Use: Load in web UI or scripts to use trained model
  • Persistence: Retains information indefinitely

Temporary Files

  • temp-checkpoint-*.json: Used by verification script (auto-deleted)
  • continuous-checkpoint.json: Used by continuous demo (auto-deleted)

Common Issues

"No checkpoint found"

Problem: Running verify before training
Solution: Run npm run train first

"Invalid checkpoint file"

Problem: Corrupted or incorrect JSON
Solution: Delete checkpoint and run npm run train again

Low vocabulary coverage

Problem: Some lyrics files missing or empty
Solution: Check lyrics/ folder has all .txt files

Training takes too long

Problem: Large corpus or slow hardware
Solution: Reduce epochs in train-lyrics.js or use fewer files

Performance Benchmarks

Script Duration Memory CPU
train 20-30s ~100MB High
verify 5-10s ~50MB Medium
continuous 30-40s ~100MB High
test-engine <1s ~20MB Low

Advanced Usage

Custom Training Parameters

Edit train-lyrics.js to modify:

const epochs = 100;  // More epochs = better learning
const alpha = 0.01;  // Learning rate
const gamma = 0.95;  // Discount factor
const lambda = 0.8;  // Eligibility trace decay

Training Subset of Lyrics

Create a custom script:

import { AGTuneEngine } from './train-lyrics.js';

const engine = new AGTuneEngine();
const customLyrics = ['verse 1', 'verse 2', ...];
engine.train(customLyrics, 50);
engine.saveCheckpoint('custom-checkpoint.json');

Monitoring Training Progress

Modify the training loop to log more frequently:

for (let epoch = 0; epoch < epochs; epoch++) {
  const reward = engine.train(corpus, 1);
  console.log(`Epoch ${epoch}: ${reward.toFixed(4)}`);
}

Integration with Web UI

Load Checkpoint in Browser

  1. Run npm run train to create checkpoint
  2. Start npm run dev
  3. Click "Load Checkpoint" button
  4. Upload agtune-lyrics-checkpoint.json
  5. Model is now trained and ready to generate

Upload Custom Lyrics

  1. In web UI, click "Upload Training Corpus"
  2. Select a .txt file with lyrics (one line per verse)
  3. Click "Train Model"
  4. Wait for training to complete
  5. Generate poetry with your custom style

Troubleshooting

Script Errors

Module not found errors:

  • Ensure you're in the project root directory
  • Run npm install first

Permission denied errors:

  • Make scripts executable: chmod +x *.js
  • Or run with node explicitly: node train-lyrics.js

Out of memory errors:

  • Reduce corpus size
  • Lower number of epochs
  • Close other applications

Data Issues

Missing words in vocabulary:

  • Some words filtered as too short (<10 chars per line)
  • Title lines ("icryafterikill") are intentionally filtered
  • This is normal and expected

Checkpoint file corrupted:

  • Delete and re-run training
  • Check disk space
  • Ensure write permissions

Best Practices

  1. Always verify after training: Run npm run verify to ensure quality
  2. Save checkpoints frequently: During long training sessions
  3. Test incrementally: Use npm run test-engine before full training
  4. Monitor resources: Close unnecessary apps during training
  5. Backup checkpoints: Copy important checkpoints to safe location

See Also

All scripts are production-ready and tested for correctness. ✓