Skip to content

Add PEFT audio LoRA sidecar converter#2384

Open
psymon-ai wants to merge 1 commit into
google-ai-edge:mainfrom
psymon-ai:gilbeot-audio-lora-converter
Open

Add PEFT audio LoRA sidecar converter#2384
psymon-ai wants to merge 1 commit into
google-ai-edge:mainfrom
psymon-ai:gilbeot-audio-lora-converter

Conversation

@psymon-ai
Copy link
Copy Markdown

Summary

Adds a deterministic PEFT Audio LoRA -> LiteRT-LM LoRA sidecar converter.

This is intended as a small first step toward a public Gemma audio LoRA
deployment path. It does not claim end-to-end Gemma Audio LoRA support yet; it
only adds the sidecar compiler and validation gates needed by a future
LoRA-ready audio graph/runtime path.

What changed

  • Added audio_lora_converter.py to compile PEFT safetensors adapters into the
    LiteRT-LM LoRA sidecar TFLite format.
  • Added audio_lora_converter_cli.py as a Bazel CLI entry point.
  • Added mapping from common audio_tower PEFT keys to LiteRT-LM audio LoRA
    tensor names, for example:
lora_audio_attn_q_a_weight_0
lora_audio_ff1_l1_b_weight_3
lora_audio_lconv_end_a_weight_10
lora_audio_output_proj_b_weight_0
  • Folded PEFT alpha / rank into each lora_B tensor because the current
    sidecar format stores rank metadata but not a separate alpha value.
  • Added strict conversion defaults for:
    • missing target inputs,
    • unmatched PEFT keys,
    • unused PEFT keys,
    • unconsumed target inputs.
  • Kept the converter audio-scoped: language-model LoRA keys are reported as
    unmatched unless callers intentionally filter to audio tensors with
    --peft_key_regex.
  • Added a NumPy-free safetensors fallback for Bazel Python environments.
  • Avoided adding new PyPI runtime dependencies: the converter uses the existing
    @flatbuffers//:runtime_py target and includes a small schema-specific
    TFLite reader/writer for the fields it needs.
  • Updated runtime/util/lora_util so audio LoRA input names are recognized by
    the existing LoRA-name utility.
  • Added a C++ LoraData test that reads a sidecar generated by this converter.
  • Added unit tests and converter documentation.

Tests

bazel test --nocache_test_results \
  //python/litert_lm_builder:audio_lora_converter_test \
  //runtime/util:lora_util_test \
  //runtime/util:lora_data_test

All three tests pass locally.

I also checked that the Bazel CLI entry point imports cleanly without any
manual import fallback:

bazel run //python/litert_lm_builder:audio_lora_converter_cli -- --help

Verification

Local verification environment:

Host: Windows 11
Build environment: WSL
Bazel entry point: bazelisk
PR branch: gilbeot-audio-lora-converter
Commit: 2f9ae5e4b9b8a88e7daaf164109784fb3b41dd5d

Observed Bazel result:

Executed 3 out of 3 tests: 3 tests pass.
//python/litert_lm_builder:audio_lora_converter_test  PASSED in 0.2s
//runtime/util:lora_util_test                         PASSED in 0.0s
//runtime/util:lora_data_test                         PASSED in 0.1s

I also validated the converter against a real Gemma audio PEFT adapter and a
LoRA-ready audio graph:

tensors_written: 240
target_inputs: 240
missing_target_inputs: 0
unmatched_peft_keys: 0
unused_peft_keys: 0
unconsumed_target_inputs: 0

The generated sidecar matched the existing reference warm sidecar byte-for-byte.

Out of scope

  • Rewriting Gemma audio graphs to expose LoRA inputs.
  • Runtime audio-executor sidecar binding.
  • Claiming end-to-end Gemma Audio LoRA quality or device CER improvement.
  • Supporting arbitrary multi-LoRA composition.

The converter expects a target audio encoder graph that already exposes
LiteRT-LM audio LoRA input names. Runtime binding and LoRA-ready Gemma audio
graph export should be handled in follow-up PRs.

@google-cla
Copy link
Copy Markdown

google-cla Bot commented May 27, 2026

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant