QVAC-21937 fix: harden embed-llamacpp model cache integrity by tobi-legan · Pull Request #3196 · tetherto/qvac

tobi-legan · 2026-07-09T20:57:08Z

🎯 What problem does this PR solve?

Makes the embed-llamacpp integration-model cache deterministic and integrity-checked.
Ensures desktop downloads, restored caches, Android pre-staging, and generated pins cannot silently trust stale or unauthenticated model bytes.

Part of QVAC-21937.

📝 How does it solve it?

Uses one manifest as the source of truth for immutable revisions, SHA-256, and byte size for both integration models.
Verifies cached and Android-prestaged files before use and fails closed on missing or malformed integrity data.
Requires canonical Hugging Face LFS SHA/size metadata for .gguf artifacts even when Hugging Face responds directly with HTTP 200.
Keeps direct non-model responses such as JSON configuration files valid without LFS metadata.
Generates mobile pre-staging from the manifest while preserving the literal require needed by mobile bundling.
Hardens redirect cleanup and fallback handling, restricts Hugging Face credentials to the exact trusted HTTPS host, and redacts signed URL credentials.

🧪 How was it tested?

Final head: 7ed2d24a88c123f24deeba6785dbea1f7d88cddc

Review finding fixed by 7ed2d24a8:
- direct HTTP 200 responses for Hugging Face .gguf URLs can no longer establish trusted pins without canonical x-linked-etag and x-linked-size;
- a regression proves the response is rejected with CanonicalMetadataError;
- direct non-model responses such as config.json remain supported.
Current-head review-fix proof:
- type declarations, lint, and unit tests passed;
- Bare suite: 28/28 tests, 71/71 assertions;
- manifest/prestage suite: 17/17 tests;
- the direct-200 .gguf regression passed.
Seven-platform integration proof:
- 7/7 platform jobs passed;
- both models restored from the manifest-addressed cache and passed SHA-256 and byte-size verification;
- warm result: 0 downloaded, 2 already cached;
- an initial Linux x64 native exit 139 passed on retry and did not reproduce.
Local final-head validation passed the focused generator suite, full npm run test:unit, npm run lint, and git diff --check.

The current-head proof workflow was force-cancelled only after its targeted ts-checks job passed, avoiding unrelated duplicate native builds. The final commit changes manifest-pin generation and its regression test, not runtime model loading, so the prior seven-platform integration proof remains applicable.

🚦 Review and merge notes

No package version bump: this changes test/CI integrity tooling and development-only dependencies, not the published API.
The red Android co-load status is a broad 106-test SDK/plugin smoke run that timed out with 67 suite-wide failures; it is not the Embed model-cache integration run above.
No unresolved review thread remains; the changes-requesting reviewer is assigned for re-review.
This PR is independent and should merge first, before QVAC-21937 fix: harden llm-llamacpp model cache integrity #3195 and QVAC-21937 fix: pin diffusion integration model integrity #3234.

github-actions · 2026-07-09T20:57:20Z

Review Status

Current Status: ✅ APPROVED
Approvals so far: Team Lead: 1, Member: 1

- add test/integration/models.manifest.json as the single source of truth for model URLs + integrity; actions/cache-models folds it into the cache key so unrelated test-logic edits no longer churn the key or force a re-save - wire ensureModel to resolve URL/sha256/bytes from the manifest, keeping the MODEL_CONFIGS download url as a fallback; lazy-load the native addon so the download/integrity helpers are usable without a compiled binding - add offline unit tests (test/unit/ensure-model-manifest.test.js) covering manifest resolution, integrity re-download, fallback, and manifest shape - add scripts/pin-model-manifest.mjs to pin sha256/bytes later - run the darwin-arm64 integration leg on qvac-macos26-arm64-gpu Ref: QVAC-21937

Make the integration-test model tooling uniform with diffusion-cpp: - ensureModel now takes an object ({ modelName, modelDir, manifest, download }), matching diffusion-cpp/llm-llamacpp; update all callsites - log the "no sha256/bytes pinned — integrity check SKIPPED" hint pointing at scripts/generate-model-manifest.mjs, same as diffusion - rename the pinning script to scripts/generate-model-manifest.mjs (verbatim diffusion copy) and the unit suite to test/unit/ensure-model-integrity.test.js - manifest integrity stays null until pinned in CI (HF_TOKEN + network); the suite tolerates null now and enforces shape once pinned Ref: QVAC-21937 Co-authored-by: Cursor <cursoragent@cursor.com>

github-actions · 2026-07-13T11:19:08Z

Mobile integration tests — @qvac/translation-nmtcpp (Android)

Result: passed

metric	value
Devices passed	2
Devices failed	0
Test cases total	6
Test cases passed	6
Test cases failed	0
Test cases skipped	0

View workflow run

github-actions · 2026-07-13T11:23:59Z

Mobile integration tests — @qvac/transcription-parakeet (iOS)

Result: passed

metric	value
Devices passed	2
Devices failed	0
Test cases total	6
Test cases passed	6
Test cases failed	0
Test cases skipped	0

View workflow run

github-actions · 2026-07-13T11:25:36Z

Mobile integration tests — @qvac/embed-llamacpp (Android)

Result: passed

metric	value
Devices passed	6
Devices failed	0
Test cases total	18
Test cases passed	18
Test cases failed	0
Test cases skipped	0

View workflow run

github-actions · 2026-07-13T11:27:19Z

Mobile integration tests — @qvac/diffusion-cpp (Android)

Result: passed

metric	value
Devices passed	3
Devices failed	0
Test cases total	9
Test cases passed	9
Test cases failed	0
Test cases skipped	0

View workflow run

github-actions · 2026-07-13T11:27:28Z

Mobile integration tests — @qvac/tts-ggml (iOS)

Result: passed

metric	value
Devices passed	2
Devices failed	0
Test cases total	6
Test cases passed	6
Test cases failed	0
Test cases skipped	0

View workflow run

tobi-legan · 2026-07-13T11:30:30Z

Manual workflow-dispatch validation passed on the final pinned head: https://github.com/tetherto/qvac/actions/runs/29245647708

All 7 platform jobs passed.
Self-hosted cache restored successfully.
Both models passed SHA-256/byte verification in the warm step.
Warm result: 0 downloaded, 2 already cached.
Integration tests consumed verified cached models 72 times with no model download attempts.

github-actions · 2026-07-13T11:35:44Z

Mobile integration tests — @qvac/llm-llamacpp (iOS)

Result: passed

metric	value
Devices passed	24
Devices failed	0
Test cases total	72
Test cases passed	72
Test cases failed	0
Test cases skipped	0

View workflow run

github-actions · 2026-07-13T11:41:25Z

Mobile integration tests — @qvac/vla-ggml (Android)

Result: passed

metric	value
Devices passed	6
Devices failed	0
Test cases total	18
Test cases passed	18
Test cases failed	0
Test cases skipped	0

View workflow run

github-actions · 2026-07-13T11:47:19Z

Mobile integration tests — @qvac/ocr-onnx (Android)

Result: passed

metric	value
Devices passed	6
Devices failed	0
Test cases total	18
Test cases passed	18
Test cases failed	0
Test cases skipped	0

View workflow run

github-actions · 2026-07-13T11:48:36Z

Mobile integration tests — @qvac/llm-llamacpp (Android)

Result: passed

metric	value
Devices passed	18
Devices failed	0
Test cases total	54
Test cases passed	54
Test cases failed	0
Test cases skipped	0

View workflow run

github-actions · 2026-07-13T11:54:21Z

Mobile integration tests — @qvac/ocr-onnx (iOS)

Result: passed

metric	value
Devices passed	4
Devices failed	0
Test cases total	12
Test cases passed	12
Test cases failed	0
Test cases skipped	0

View workflow run

github-actions · 2026-07-13T11:59:59Z

Mobile integration tests — @qvac/tts-ggml (Android)

Result: failed

metric	value
Devices passed	1
Devices failed	1
Test cases total	6
Test cases passed	5
Test cases failed	1
Test cases skipped	0

View workflow run

gianni-cor · 2026-07-13T17:07:41Z

bare-crypto appears to be test-only here. It is imported by test/integration/utils.js, while the package files list excludes the test tree, so published runtime code does not require it. The integration and mobile test setup uses a normal npm install, which installs dev dependencies.

Could we move bare-crypto to devDependencies? Keeping it in production dependencies makes every consumer install an unused package.

gianni-cor · 2026-07-13T17:07:41Z

There is still a fail-open path when the model manifest cannot be loaded or does not contain the requested model.

loadManifest() catches missing or malformed JSON and returns null. ensureModel() then falls back to the mutable URL in MODEL_CONFIGS, sets hasIntegrity to false, and accepts any existing non-empty cached file without checking SHA-256 or expected size. Android pre-staging also receives entry === null, reducing verification to a non-empty-file check.

This means a packaging mistake, malformed manifest, or missing entry silently disables the integrity guarantees introduced by this PR. Could the default manifest and every configured integration model entry be required, with a clear error instead of falling back? Any fallback needed by tests could remain explicit and test-only.

Fail closed when manifest data is unavailable, keep test-only crypto out of production dependencies, and run immutable pre-stage coverage in the unit command. Co-authored-by: Cursor <cursoragent@cursor.com>

…bed-cache-keys

tobi-legan · 2026-07-13T18:02:52Z

Addressed both follow-up findings on the latest head.

The default manifest and requested model entry are now mandatory and must include valid SHA-256 and byte pins. MODEL_CONFIGS can no longer bypass a missing manifest entry.
Cached and Android pre-staged files therefore cannot fall back to non-empty-only acceptance.
bare-crypto moved to devDependencies because it is only used by the unpublished test tree.
Repeated verification of an unchanged file is deduplicated within the integration process.
The pre-stage generator now rejects mutable main/master URLs and its suite runs from npm run test:unit. The full unit command passes: 28 Brittle tests and 4 Node pre-stage tests.
Latest main is merged.

Final workflow-dispatch validation is running here: https://github.com/tetherto/qvac/actions/runs/29272850432

Co-authored-by: Cursor <cursoragent@cursor.com>

github-actions · 2026-07-16T08:09:14Z

QVAC E2E — `android` — ✅ all tests passed (94/106, 2643s)

Config: suite=smoke · filter=(none) · exclude=(none)
View run · Artifacts: reports · Device Farm logs

tobi-legan requested review from a team as code owners July 9, 2026 20:57

tobi-legan and others added 2 commits July 9, 2026 22:24

tobi-legan force-pushed the feature-qvac-21937-embed-cache-keys branch from 3fd25b2 to 7941d0f Compare July 9, 2026 21:26

tobi-legan temporarily deployed to release July 9, 2026 21:26 — with GitHub Actions Inactive

covert-oddity previously approved these changes Jul 13, 2026

View reviewed changes

GSServita previously approved these changes Jul 13, 2026

View reviewed changes

tobi-legan and others added 2 commits July 13, 2026 19:01

fix[notask]: require pinned embed model manifests

c07ca33

Fail closed when manifest data is unavailable, keep test-only crypto out of production dependencies, and run immutable pre-stage coverage in the unit command. Co-authored-by: Cursor <cursoragent@cursor.com>

Merge remote-tracking branch 'origin/main' into feature-qvac-21937-em…

40b9166

…bed-cache-keys

This comment has been minimized.

Sign in to view

fix[notask]: harden embed manifest pin generation

6f5f5fc

Co-authored-by: Cursor <cursoragent@cursor.com>

This comment has been minimized.

Sign in to view

This was referenced Jul 14, 2026

QVAC-21937 fix: harden llm-llamacpp model cache integrity #3195

Merged

QVAC-21937 fix: pin diffusion integration model integrity #3234

Merged

fix[notask]: require embed model pin provenance

7ed2d24

Co-authored-by: Cursor <cursoragent@cursor.com>

This comment has been minimized.

Sign in to view

gianni-cor approved these changes Jul 15, 2026

View reviewed changes

covert-oddity approved these changes Jul 16, 2026

View reviewed changes

Merge branch 'main' into feature-qvac-21937-embed-cache-keys

36c650c

Uh oh!

Conversation

tobi-legan commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🎯 What problem does this PR solve?

📝 How does it solve it?

🧪 How was it tested?

🚦 Review and merge notes

Uh oh!

github-actions Bot commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review Status

Uh oh!

github-actions Bot commented Jul 13, 2026

Mobile integration tests — @qvac/translation-nmtcpp (Android)

Uh oh!

github-actions Bot commented Jul 13, 2026

Mobile integration tests — @qvac/transcription-parakeet (iOS)

Uh oh!

github-actions Bot commented Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Mobile integration tests — @qvac/embed-llamacpp (Android)

Uh oh!

github-actions Bot commented Jul 13, 2026

Mobile integration tests — @qvac/diffusion-cpp (Android)

Uh oh!

github-actions Bot commented Jul 13, 2026

Mobile integration tests — @qvac/tts-ggml (iOS)

Uh oh!

tobi-legan commented Jul 13, 2026

Uh oh!

github-actions Bot commented Jul 13, 2026

Mobile integration tests — @qvac/llm-llamacpp (iOS)

Uh oh!

github-actions Bot commented Jul 13, 2026

Mobile integration tests — @qvac/vla-ggml (Android)

Uh oh!

github-actions Bot commented Jul 13, 2026

Mobile integration tests — @qvac/ocr-onnx (Android)

Uh oh!

github-actions Bot commented Jul 13, 2026

Mobile integration tests — @qvac/llm-llamacpp (Android)

Uh oh!

github-actions Bot commented Jul 13, 2026

Mobile integration tests — @qvac/ocr-onnx (iOS)

Uh oh!

github-actions Bot commented Jul 13, 2026

Mobile integration tests — @qvac/tts-ggml (Android)

Uh oh!

gianni-cor commented Jul 13, 2026

Uh oh!

gianni-cor commented Jul 13, 2026

Uh oh!

This comment has been minimized.

tobi-legan commented Jul 13, 2026

Uh oh!

This comment has been minimized.

This comment has been minimized.

github-actions Bot commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

QVAC E2E — android — ✅ all tests passed (94/106, 2643s)

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

tobi-legan commented Jul 9, 2026 •

edited

Loading

github-actions Bot commented Jul 9, 2026 •

edited

Loading

github-actions Bot commented Jul 13, 2026 •

edited

Loading

github-actions Bot commented Jul 16, 2026 •

edited

Loading

QVAC E2E — `android` — ✅ all tests passed (94/106, 2643s)