Skip to content

ARCHITECTURE.md

Latest commit

 

History

History
128 lines (109 loc) · 6.58 KB

ARCHITECTURE.md

File metadata and controls

128 lines (109 loc) · 6.58 KB

Architecture

Overview

Mofusand Synth is a Tauri 2 + SvelteKit desktop app that converts YouTube audio into 8-bit chiptune. The Rust backend downloads audio via yt-dlp and handles file I/O through three IPC commands. The Svelte frontend owns all audio processing and offers two conversion modes:

  • DSP Crush — degrades the original recording with a bitcrusher / lowpass / sample-rate-reduction chain (real-time, tweakable). Optional vocal removal.
  • True Chiptune — transcribes the song to notes with Spotify's basic-pitch ML model, then re-synthesizes those notes on square/triangle/pulse/sawtooth oscillators.

Both modes ultimately produce an AudioBuffer that flows through one shared playback engine (play/pause/seek) and one shared WAV export path.

System Layers

┌──────────────────────────────────────────────────────────────┐
│                  SvelteKit Frontend (Webview)                  │
│                                                                │
│  +page.svelte ── UrlInput ── Player ── ChiptuneControls        │
│                                                                │
│  DSP mode:      source → preGain → WaveShaper → lowpass         │
│                 → downsampler(AudioWorklet) → destination       │
│                                                                │
│  Chiptune mode: original → transcribe(basic-pitch)             │
│                 → notes → renderChiptune() → AudioBuffer        │
│                 → source → cleanGain → destination              │
└───────────────────────────┬────────────────────────────────────┘
                            │ Tauri IPC (invoke)
┌───────────────────────────▼────────────────────────────────────┐
│                    Rust / Tauri Commands                        │
│  download_audio(url)            → { path, title }               │
│  read_audio_file(path)          → Vec<u8>                       │
│  save_audio_file(bytes, name)   → ()  (native save dialog)      │
└───────────────────────────┬────────────────────────────────────┘
                            │ std::process::Command
┌───────────────────────────▼────────────────────────────────────┐
│                      yt-dlp (external binary)                   │
│  Downloads YouTube audio (mp3) to the OS temp directory         │
└──────────────────────────────────────────────────────────────┘

Directory Structure

mofusand-synth/
├── src-tauri/                       # Rust / Tauri backend
│   ├── src/
│   │   ├── main.rs                  # thin entry → lib::run()
│   │   ├── lib.rs                   # Tauri builder, registers commands
│   │   └── commands/
│   │       ├── mod.rs
│   │       ├── download.rs          # download_audio (yt-dlp)
│   │       └── file.rs              # read_audio_file, save_audio_file
│   ├── capabilities/default.json    # dialog:allow-save permission
│   ├── Cargo.toml
│   └── tauri.conf.json              # 620×800 window, csp: null
├── src/                             # SvelteKit frontend
│   ├── app.css                      # global styles + Mofusand theme
│   ├── app.html
│   ├── routes/
│   │   ├── +layout.js               # ssr = false (SPA mode)
│   │   ├── +layout.svelte           # imports app.css
│   │   └── +page.svelte             # root: state + Tauri invocations
│   └── lib/
│       ├── audio.js                 # makeBitCrushCurve, encodeWav
│       ├── transcribe.js            # basic-pitch wrapper (audio → notes)
│       ├── chiptune.js              # notes → rendered chiptune AudioBuffer
│       ├── worklet/downsampler.js   # AudioWorklet sample-rate reducer
│       └── components/
│           ├── UrlInput.svelte
│           ├── Player.svelte        # both modes, playback, download
│           └── ChiptuneControls.svelte  # DSP sliders
├── static/
│   └── model/                       # basic-pitch model (served at /model/)
│       ├── model.json
│       └── group1-shard1of1.bin
└── docs/

Data Flow

Download (both modes)

paste URL → +page.svelte invoke("download_audio", {url})
  → Rust yt-dlp → { path, title }
  → invoke("read_audio_file", {path}) → bytes
  → Player decodes → originalBuffer

DSP Crush mode

originalBuffer → BufferSource → [vocal removal?] → preGain
  → WaveShaper(bitcrush) → BiquadFilter(lowpass)
  → AudioWorklet(downsampler) → destination

Sliders (Bit Depth / Sample Rate / Wave Crush) update nodes live.

True Chiptune mode

originalBuffer → resample mono 22050Hz → basic-pitch.evaluateModel()
  → note events (pitchMidi, startTimeSeconds, durationSeconds, amplitude)
  → renderChiptune(notes) [OfflineAudioContext: oscillators + envelopes]
  → chiptuneBuffer → BufferSource → cleanGain → destination

Notes are cached; changing the waveform only re-runs renderChiptune.

Download

DSP mode:      OfflineAudioContext re-renders effects chain → encodeWav
Chiptune mode: chiptuneBuffer already rendered → encodeWav
  → invoke("save_audio_file", {bytes, filename}) → native dialog

Key Constraints

  • yt-dlp must be on PATH — surfaced as an inline error if missing.
  • Tauri capabilities must allow dialog:allow-save.
  • CSP is disabled (csp: null) so tfjs and the local model load freely.
  • Vocal removal uses L−R channel cancellation; requires a stereo source.
  • basic-pitch transcription is best on melodic content; very dense mixes get noisy.
  • Audio is held in memory (Vec<u8> / AudioBuffer) — fine for typical song lengths.