MusicGenText

MusicGenText — end-to-end TEXT-CONDITIONED MusicGen generation

The successor to MusicGenSmoke. The smoke test stubbed the text encoder (it fed a hand-built ramp tensor in place of the T5 output); this example wires the REAL T5 text encoder, so a free-text prompt genuinely steers the generated music.

prompt text
  -> T5 SentencePiece tokenizer (real) OR fixed ids (pico demo)
  -> T5 text ENCODER (BuildT5FromSafeTensors)
  -> the encoder's final hidden states (EncSeq x text_d_model)
  -> MusicGen enc_to_dec_proj -> cross-attention conditioning
  -> MusicGen DECODER greedy/sampled delay-pattern decode -> [K][frames]
  -> EnCodec audio DECODER (BuildEnCodecFromSafeTensors) -> mono waveform
  -> SaveVolumeToWav16 -> a .wav clip

No new layer types are added — this is pure example wiring composing three fully-landed importers (BuildT5FromSafeTensors, BuildMusicGenFromSafeTensors, BuildEnCodecFromSafeTensors) plus the TNeuralHFTokenizer and the native neuralhfhub Hub downloader.

Running

Pico demo (default, no network)

With no arguments it runs a self-contained pico demo on the committed random fixtures (pure CPU, a fraction of a second):

cd examples/MusicGenText
./MusicGenText            # writes musicgen_text_demo.wav

The weights are untrained random, so the clip is noise, not music — this only exercises the full text→audio wiring end to end.

Real music (--download)

--download fetches three standard public HF repos through the native Pascal Hub helper (neuralhfhub — no Python) and imports each directly from its own snapshot, with no split/convert step:

repo	role	note
facebook/musicgen-small	LM decoder	the importer reads only the decoder.* / enc_to_dec_proj.* keys and ignores the bundled text/audio encoders
t5-base	text encoder + SentencePiece tokenizer	MusicGen's frozen text encoder is t5-base
facebook/encodec_32khz	32 kHz audio codec	normalize=false, imports cleanly

cd examples/MusicGenText
./MusicGenText --download --prompt "warm lo-fi hip hop beat with mellow piano"
./MusicGenText --download --prompt "..." --seconds 8 --guidance 3.0
./MusicGenText --download --prompt "..." --topk 250 --temperature 1.0

Downloads (~3.5 GB total) are cached under ~/.cache/neural-api/hub, so re-runs are offline. Repo overrides: --musicgen-repo / --t5-repo / --encodec-repo. For gated repos set HF_TOKEN in the environment. --seconds N sets the clip length (converted to EnCodec frames via the codec frame rate); --frames N overrides the frame count directly; --guidance defaults to MusicGen's 3.0 in --download mode.

Performance: GPU + AVX

This example offloads its hot matmuls to the GPU and relies on AVX for the EnCodec codec. Both are build-time decisions, so the program prints a build-flags banner at startup (e.g. [build] flags: AVX2 CPU64 OpenCL Release) — if it shows no AVX* or no OpenCL, you are running the slow scalar/CPU path.

GPU (OpenCL)

Built with -dOpenCL (the default .lpi config), the T5 text encoder and the MusicGen decoder offload their conv/linear matmuls to the GPU by default — including the width-1 step twins the KV-cache decode loop runs on (TMusicGenModel.EnableOpenCL). The EnCodec codec is a hand-rolled conv path (not a TNNet) and always synthesizes on CPU.

flag	effect
(none)	GPU on by default when built with -dOpenCL
--no-gpu	force CPU
--gpu-platform N	pick the OpenCL platform (default 0)
--gpu-device N	pick the OpenCL device (default 0)

Any device-selection failure falls back to CPU with a message. The pico demo's tiny layers fall below the per-layer GPU size threshold, so GPU mainly matters with --download.

AVX

The EnCodec LSTM / matmul speed depends entirely on whether an AVX define reached the compiler. -dAVX2 / -dAVX / -dAVX512 (with CPU64) select the hand-written asm dot-product; without them the scalar fallback compiles and DotProduct runs no faster than a plain loop. If the banner prints no AVX*, rebuild with -dAVX2.

Instrumentation

Every run prints always-on [time] ... lines for each load and pipeline step (tokenizer, T5 load + forward, MusicGen load + decode, EnCodec load + synthesis) and a per-codebook code-stack health summary (id range, distinct ids, and the most-repeated id) that flags a COLLAPSED? decode. --print-model (alias --summary) additionally dumps the per-layer weight tables.

Fixtures

Generated by the extended tools/musicgen_tiny_fixture.py, matched to the MusicGen decoder fixture's cross-attention dim so the shapes line up:

• tests/fixtures/tiny_musicgen_t5enc.{safetensors,_config.json} — a tiny standard T5 checkpoint at the matched d_model = text_d_model (the example runs only its encoder).
• tests/fixtures/tiny_musicgen_encodec.{safetensors,_config.json} — a tiny EnCodec decoder at codebook_size = MusicGen vocab so the generated code ids are valid RVQ indices, with >= num_codebooks quantizers.

Test

TestMusicGenTextWiring (in tests/TestNeuralPretrained.pas) asserts the pipeline runs, is deterministic (same prompt → identical codes + waveform), and that the text conditioning is live (a different prompt steers the codes).

Landed

• Classifier-free guidance (--guidance), top-k / temperature sampling (--topk / --temperature), and KV-cache incremental decode (greedy default; --no-cache forces the full re-encode loop). Under guidance the two passes per step each keep their own KV-cache (dual-twin path).
• Real tokenizer + real downloaded checkpoint via --download (above).
• GPU (OpenCL) offload of the T5 encoder + MusicGen decoder, on by default with -dOpenCL (--no-gpu / --gpu-platform / --gpu-device).
• AVX-vectorized EnCodec path plus a startup build-flags banner, always-on timing + code-stack health instrumentation, and a --print-model weight-table dump. See Performance: GPU + AVX above.

Follow-ups (deferred)

• Stereo — the audio_channels = 2 (2K-codebook) layout (the importer currently rejects audio_channels=2).