fix(recipes): bump Hugging Face Hub helper-job pins to 1.16.4

[MatejKosec]

Summary

Updates the Hugging Face Hub CLI pin used by Dynamo model-cache and vLLM LoRA helper jobs from 1.11.0 to 1.16.4.

Issue #10857 reports that the Qwen3-32B model-cache job fails after installing huggingface_hub==1.11.0 because the hf CLI import path is missing click. The issue reporter confirmed that huggingface_hub==1.16.4 installs the needed CLI dependencies and allows the same hf download command to run.

Scope

This change is limited to helper YAML files that install Hugging Face Hub immediately before an hf download command. It preserves the existing package spelling in each file:

• huggingface_hub==1.11.0 → huggingface_hub==1.16.4
• huggingface-hub==1.11.0 → huggingface-hub==1.16.4

No model IDs, revisions, cache paths, PVCs, image names, resource requests, awscli pins, HF token wiring, Dockerfiles, runtime code, operator code, Rust code, or Python code are changed.

Files changed

The diff contains 18 one-line YAML replacements under:

• recipes/*/model-cache/model-download*.yaml
• examples/backends/vllm/deploy/lora/**/sync-lora-job.yaml

Validation

The factory run validated the generated branch before publication:

• pre-commit run --files <18 changed YAML files> --hook-stage manual passed.
• Static inspection confirmed no huggingface[_-]hub==1.11.0 pins remain in the planned recipe/example scope.
• Static inspection confirmed every changed file retains an hf download command and that the diff consists only of exact 1.11.0 → 1.16.4 pin replacements.
• A clean temporary Python environment installed huggingface_hub==1.16.4 and imported both click and huggingface_hub.cli.hf.main successfully.

A full Kubernetes model-cache job was not run because it would require cluster/PVC/HF-token setup and large model downloads. This PR addresses the CLI dependency pin that caused the import failure.

Summary by CodeRabbit

• Chores
  • Updated the Hugging Face client dependency used by several model download and sync jobs.
  • These jobs now use a newer, consistent version across supported recipes and backends.
  • No changes were made to download behavior, job structure, or other runtime settings.

Closes DYN-3335

[github-actions]

👋 Hi MatejKosec! Thank you for contributing to ai-dynamo/dynamo.

Just a reminder: The NVIDIA Test Github Validation CI runs an essential subset of the testing framework to quickly catch errors.Your PR reviewers may elect to test the changes comprehensively before approving your changes.

🚀

[coderabbitai]

Walkthrough

The PR updates Kubernetes job startup scripts across LoRA sync and recipe model-download workflows to install huggingface-hub/huggingface_hub 1.16.4 instead of 1.11.0. The existing download commands and other job settings are unchanged.

Changes

Hugging Face client pin update

Layer / File(s)	Summary
LoRA sync jobs examples/backends/vllm/deploy/lora/multimodal/sync-lora-job.yaml, examples/backends/vllm/deploy/lora/sync-lora-job.yaml	The LoRA sync job scripts replace the pinned huggingface-hub install version with 1.16.4; awscli and the sync flow stay the same.
Model download jobs recipes/deepseek-r1/model-cache/model-download*.yaml, recipes/deepseek-v32-fp4/model-cache/model-download.yaml, recipes/deepseek-v4/deepseek-v4-*/model-cache/model-download.yaml, recipes/glm-5-nvfp4/model-cache/model-download.yaml, recipes/gpt-oss-120b/model-cache/model-download.yaml, recipes/kimi-k2.5/model-cache/model-download.yaml, recipes/llama-3-70b/model-cache/model-download.yaml, recipes/nemotron-3-nano-omni/model-cache/model-download.yaml, recipes/nemotron-3-super-fp8/model-cache/model-download.yaml, recipes/qwen3-235b-a22b-fp8/model-cache/model-download.yaml, recipes/qwen3-32b-fp8/model-cache/model-download.yaml, recipes/qwen3-32b/model-cache/model-download.yaml, recipes/qwen3-vl-30b/model-cache/model-download.yaml, recipes/qwen3.6-35b/model-cache/model-download.yaml	The recipe job scripts replace the pinned huggingface_hub install version with 1.16.4 before the existing hf download commands.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly states the main change: bumping the Hugging Face Hub helper-job pins to 1.16.4.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description check	✅ Passed	The PR description covers summary, scope, files, validation, and issue linkage, with only minor template-heading differences.

---

_{Comment @coderabbitai help to get the list of available commands.}

[devin-ai-integration]

Devin Review found 0 potential issues.

[devin-ai-integration]

Devin Review found 0 new potential issues.

[MatejKosec]

deploy-operator failure — upstream CI image lag, not this PR's diff

The only failing required check is deploy-status-check, red because deploy-operator crashloops crd-apply:

flag provided but not defined: -conversion-webhook-service-name
Usage of /crd-apply:
Exit Code: 2  → CrashLoopBackOff

Why this PR hits it. This PR's diff is 18 YAML pin bumps; it touches no operator files, so the operator build job is skipped. deploy-operator then falls back to the floating main-operator image tag (.github/workflows/pr.yaml → operator_tag: ... || 'main-operator'):

OPERATOR_TAG: main-operator
Using operator image: ${OPERATOR_REPO}:main-operator

That floating image predates commit d8cdf133 (2026-06-24, "fail closed conversion admission in manual cert mode"), which added -conversion-webhook-service-name to both the chart (deploy/helm/.../deployment.yaml) and crd-apply/main.go atomically. The chart on current main passes the flag; the stale main-operator binary rejects it.

This is systemic for any flag-bearing PR that skips the operator build, not specific to #10986:

PR	base behind main	chart carries flag?	deploy-operator
#10986 (this)	current	yes	fail
#11034	12 behind	yes	fail
#11039	169 behind	no (predates d8cdf13)	pass
#11037	169 behind	no (predates d8cdf13)	pass
main post-merge	—	yes, but builds fresh <sha>-operator	pass

Main's own post-merge deploy-operator passes because its Operator build job runs and deploys a fresh <sha>-operator image — main never exercises the stale floating tag, which is why this doesn't show up on main.

Fix is upstream: republish the main-operator floating tag from current main (so it carries the flag), or stop the PR deploy path from falling back to a stale floating tag when the operator build is skipped. I've re-run deploy-operator in case the tag gets republished; if it fails identically, no change to this PR will make it green. Posting this for visibility so the upstream CI/operator owners can republish the tag. The PR diff itself is review-clean (2× APPROVED, Devin 0 blocking).

[MatejKosec]

Update (2026-06-29): re-ran deploy-operator three times; the latest attempts fail earlier, on a second, independent CI-infra issue — vCluster never provisions:

### VCLUSTER OPERATOR DEPLOYMENT FAILED
No resources found in gh-id-...-pr-10986-dt namespace.
vCluster kubeconfig not found — vCluster never connected successfully

The ephemeral namespace comes up empty on prod-deploy-tester-v1, so the job dies in connect-vcluster before reaching the operator/chart step. PR #11034's most recent deploy-operator (job 83973793038, 04:27 today) shows the same vCluster flake; its earlier attempt (32m) reached crd-apply and hit the -conversion-webhook-service-name flag error. So deploy-operator is currently blocked by two distinct CI-infra issues:

1. Floating main-operator tag stale vs the flag from d8cdf133 (2026-06-24) — surfaces whenever vCluster does connect and the chart passes the flag (see matrix in parent comment).
2. vCluster provisioning flake on prod-deploy-tester-v1 — namespace never connects, job fails in ~2 min, surfaces now.

Neither is caused by this PR's diff (18 YAML pin bumps) or fixable from it. Holding for the CI/operator owners to republish main-operator and stabilize the deploy-test runner. All other checks pass; reviews are clean.

[devin-ai-integration]

Devin Review found 0 new potential issues.

[devin-ai-integration]

Devin Review found 1 new potential issue.

⚠️ 1 issue in files not directly in the diff

⚠️ Version bump missed one recipe file, leaving it on the old dependency version (recipes/qwen3-vl-32b-fp8/model-cache/model-download.yaml:36)

One model-download recipe still pins the old library version (huggingface_hub==1.11.0 at recipes/qwen3-vl-32b-fp8/model-cache/model-download.yaml:36) while every other file was updated to 1.16.4, so this recipe will install a different, outdated version.

Impact: The Qwen3-VL-32B-FP8 model download job will use an older library version than all other recipes, potentially missing bug fixes or behaving inconsistently.

Incomplete mechanical transformation across YAML files

The PR updates huggingface_hub from 1.11.0 to 1.16.4 in 17 YAML files. However, recipes/qwen3-vl-32b-fp8/model-cache/model-download.yaml:36 was not included in the diff and still contains pip install --no-cache-dir huggingface_hub==1.11.0. Every other pinned occurrence in the repo has been updated to 1.16.4.

[devin-ai-integration]

Devin Review found 0 new potential issues.

[dagil-nvidia]

LGTM