Add NVIDIA Cosmos 3 Edge: reasoner (VLM surface) + generative world model

[probicheaux]

Adds NVIDIA Cosmos 3 Edge (HF model, blog) as a core model family, in two stacked pieces:
(corresponding sdk registry PR: https://github.com/roboflow/model-registry-sdk/pull/8)

Reasoner (cosmos-3-edge, task vlm, backend hugging-face)

• Cosmos3EdgeReasoner in inference_models following the standard VLM contract (prompt(), staged pre/generate/post), plus prompt_video() for multi-frame reasoning.
• Serving bridge adapter → existing /infer/lmm route (no new HTTP surface), gated by COSMOS3_ENABLED (default on; deployments that shouldn't serve it disable via env).
• Workflow block roboflow_core/cosmos3_edge@v1 emitting language_model_output (chains into vlm_as_detector etc.), GPU-gated via get_restrictions().

Generator (cosmos-3-edge-world, new task world-model, backend custom)

• Cosmos3EdgeWorldModel: generate_video() (image-to-video), start_rollout()/forward_dynamics() with explicit opaque state_dict threading (raises on missing session), inverse_dynamics() returning a Cosmos3ActionTrajectory (action_space selects the embodiment domain: umi, av, droid_lerobot, …).
• Runtime seam: the model package ships a self-contained cosmos3_generator_runtime.py imported via import_class_from_file (moondream2 precedent) — no new core inference_models dependencies. Reference runtime (diffusers Cosmos3OmniPipeline, all modes wired) + weight-pull tooling in inference_models/development/cosmos3/.

Verified with real weights (NVIDIA L4, 24 GB)

Output-quality checks against NVIDIA's official example assets:

• Reasoner prompt() produces a coherent multi-step robot task plan for the official reasoning example; generate_video() produces a coherent temporal continuation of the official i2v conditioning image.
• Forward dynamics threads the official UMI 2×16-action chunks across a session (chunk 1 conditioned on chunk 0's final frame).
• Inverse dynamics on the official AV example reproduces the published Edge output at 0.945 correlation (MAE 0.10, 15 denoise steps).
• Test suites: 14 model + 8 block unit tests green; 3 real-weights integration tests green (env-gated -m slow).

Timings / VRAM per mode (bf16, defaults = 35 denoise steps unless noted):

Mode (settings)	Time	Peak VRAM
Reasoner — load (weights resident)	3.5 s	4.5 GiB
Reasoner — prompt(), single image, 256 new tokens	8.2 s	4.6 GiB
Reasoner — prompt_video(), 16 frames, 256 new tokens	7.6 s	4.7 GiB
World — load (weights resident)	6.3 s	7.6 GiB
World — generate_video(), 25 frames @ 480p	26.4 s	12.2 GiB
World — generate_video(), 13 frames @ 240p	5.9 s	8.7 GiB
World — forward_dynamics(), 16-action UMI chunk @ 256 tier	5.9 s	8.4 GiB
World — inverse_dynamics(), 61-frame AV video @ 480 tier	10.5 s	8.3 GiB

Both towers share the MoT weight shards — development/cosmos3/pull_weights.py downloads one snapshot, materializes both package layouts via hardlinks, and mirrors them to GCS (--gcs-dest gs://…; --dry-run prints the exact file manifest, including the runtime module the world package must contain).

Remaining before merge

• Publish weights: run pull_weights.py --towers reasoner,world --runtime-module reference_generator_runtime.py --gcs-dest gs://<bucket>/<prefix> and register both packages in the model registry (staging → production). The integration tests then switch from env-var local dirs to downloaded-package fixtures like the other suites.
• Dependency pins → releases: requirements/requirements.cosmos.txt pins transformers/diffusers to validated git SHAs because cosmos3_edge (transformers ≥5.15) and the Cosmos3OmniTransformer checkpoint format (diffusers ≥0.40) are unreleased; the repo also caps transformers<=5.9.0. When the releases land: swap to version pins, raise the cap, and wire an extras_require entry (git URLs can't ship in pypi extras).
• Reviewer confirmation of names (part of this review, not follow-up work): arch strings cosmos-3-edge/cosmos-3-edge-world, WORLD_MODEL_TASK, COSMOS3_ENABLED, BackendType.CUSTOM for the world package.

Explicitly out of scope (follow-up PRs): step-wise policy mode (DROID), and an HTTP/workflow-block surface for generative outputs (LMMInferenceResponse is text-only; frames/actions need new entities and kinds).

🤖 Generated with Claude Code

[github-actions]

👋 Thanks for the pull request! Here is how automated Claude review works here, so you spend credits (and reviewer time) wisely.

🚦 This PR is marked Ready for review, so automated Claude review will run — and every pass spends real credits.

Warning

💸 The Claude reviewer bills in credits, not vibes

Automated review spins up a real agent that reads real code and spends real credits on every pass. It is glad to help — but it is not a rubber duck, a linter you poke in a loop, or a substitute for reading the contributing guide. Treat it like an expensive senior reviewer whose time you booked, and show up prepared.

Draft when unsure, Ready when you mean it:

• 🌱 Not sure the PR is in good shape yet? Keep it (or set it back) as a draft — drafts pause review, so you can push and iterate without burning credits on a moving target.
• 💪 Feel strong about the contents? Mark it Ready for review and the reviewer will take a look.

However you get there, arrive prepared:

• 🧱 Bring a SOLID, thorough PR. Point your local agent at our skills/ to tune it to our guidelines first — or, if you are one of those fabled carbon-based contributors, read them yourself. A half-baked diff costs exactly the same to review as a finished one.
• ✅ Resolve every comment before you re-request review. Re-requesting with threads still open means paying twice for the same conversation.
• 🔁 Do not use CI review as an inner loop for a local agent. The reviewer is not a step-by-step debugger — do the unfolding locally and arrive with the answer, not the search.
• 🙋 If something looks off, ask a human. One question to a maintainer is cheaper and faster than three rounds of agent re-review chasing a misread.

Reviews are not free. A draft costs nothing to review; a Ready PR is a promise that it is worth reviewing.

• Prefer to skip automated review entirely? Add the skip-claude-review label.

[probicheaux]

staging registry and workflow works

[github-actions]

🤖 Claude review started at commit 3d0283849b59d9d51613173241223503a1bb48dc.

New commits are not auto-reviewed. Add the claude-review label (remove & re-add it to trigger again) when you want another review.

[github-actions]

🤖 Claude review started at commit 3d0283849b59d9d51613173241223503a1bb48dc.

New commits are not auto-reviewed. Add the claude-review label (remove & re-add it to trigger again) when you want another review.

[github-actions]

🤖 Claude review started at commit 3d0283849b59d9d51613173241223503a1bb48dc.

New commits are not auto-reviewed. Add the claude-review label (remove & re-add it to trigger again) when you want another review.

[github-actions]

🤖 Claude review started at commit 3d0283849b59d9d51613173241223503a1bb48dc.

New commits are not auto-reviewed. Add the claude-review label (remove & re-add it to trigger again) when you want another review.

[github-actions]

COSMOS3_ENABLED defaults to True, but the reasoner cannot actually load on a standard install.

With the default on (and USE_INFERENCE_MODELS), inference/models/utils.py registers ("vlm","cosmos-3-edge") / ("text-image-pairs","cosmos-3-edge") into ROBOFLOW_MODEL_TYPES. The module import chain only needs base transformers symbols (AutoModelForImageTextToText, AutoProcessor), so registration succeeds and the model/block appears available.

But loading the weights needs transformers>=5.15 and diffusers>=0.40 (per requirements/requirements.cosmos.txt and the PR notes), whereas requirements/requirements.transformers.txt caps transformers>=4.57.3,<=5.9.0, and requirements.cosmos.txt is wired into no pip extra / setup.py. So on any standard pip install inference[...], a self-hosted GPU user who runs the block in LOCAL mode will hit a runtime failure at AutoModelForImageTextToText.from_pretrained (config model_type: cosmos3_edge unknown to transformers ≤5.9), even though get_restrictions() only HARD-blocks CPU — implying local GPU is supported.

Question (IMPORTANT): is local/self-hosted execution meant to be supported at merge? If only the hosted serving image carries the right deps, consider defaulting COSMOS3_ENABLED off until the dependency releases land + an extra is wired, or otherwise surfacing a clear "dependencies missing" error rather than a bare transformers load failure. See the action-item comment.

[github-actions]

This replaces the previous whole-directory ignore opsx/ with only opsx/.venv-cosmos3/, so the rest of opsx/ is no longer git-ignored. If that narrowing is unintentional, files under opsx/ could now be accidentally committed. Was dropping the blanket opsx/ ignore intended? (optional)

[github-actions]

@probicheaux

⏳ This PR is ON HOLD pending your answers — the review will not advance to sign-off and the PR should not proceed to merge until the IMPORTANT questions below are resolved. Your own PR description also lists unchecked "Remaining before merge" items (weight publication + registry registration, dependency-pin → release wiring, reviewer name confirmation), which keep this open.

I reviewed the shipping surfaces (workflow block roboflow_core/cosmos3_edge@v1, the InferenceModelsCosmos3ReasonerAdapter, and the inference_models reasoner). The code faithfully mirrors established patterns (Qwen VLM block, the qwen25vl inference_models adapter, moondream2's import_class_from_file runtime loading), so I found no hard correctness blocker in those paths. The open items are contract / rollout / dependency questions I cannot resolve from code alone.

IMPORTANT — please answer

1. Local execution vs. dependency availability. COSMOS3_ENABLED defaults to True, so the block/model registers and appears runnable, and get_restrictions() only HARD-blocks SELF_HOSTED_CPU (implying local GPU is supported). But loading needs transformers>=5.15 / diffusers>=0.40, while requirements/requirements.transformers.txt caps transformers>=4.57.3,<=5.9.0 and requirements/requirements.cosmos.txt is wired into no pip extra. On a standard pip install, a self-hosted GPU user running LOCAL mode will hit a runtime failure at AutoModelForImageTextToText.from_pretrained (unknown model_type: cosmos3_edge). Is local/self-hosted execution intended to be supported at merge, or only the hosted serving image? If the latter, should the default be off (or a clearer "dependencies missing" error be raised) until the dep releases land and an extra is wired? (see inline comment on inference/core/env.py:262)

2. Naming / contract lock-in (also flagged in your description as "part of this review"). Once weights are published in the model registry under cosmos-3-edge / cosmos-3-edge-world, these identifiers — plus the new WORLD_MODEL_TASK = "world-model" task string, BackendType.CUSTOM for the world package, and the COSMOS3_ENABLED env name — become a backward-compat contract (registry keys, ROBOFLOW_MODEL_TYPES / models_registry keys, block type roboflow_core/cosmos3_edge@v1). Renaming after publication breaks users. Can you confirm these names are final and have maintainer sign-off before weights are published? (escalated below.)

3. Merge sequencing vs. weight publication. Your checklist lists "Publish weights + register both packages (staging → production)" as remaining. With the block enabled by default, merging before production registration means the block is visible in the FE but fails for users hitting production. Is the plan to merge only after production registration, or to merge dormant and rely on the runtime error? Please confirm the intended sequence.

Optional

4. .gitignore: the change replaces the blanket opsx/ ignore with only opsx/.venv-cosmos3/, un-ignoring the rest of opsx/. Intentional? (inline comment on .gitignore:220)

Tests to add (non-blocking)

• A workflow integration test for the new block under tests/workflows/integration_tests/execution/ (the workflows-blocks convention expects one for a new/output-changing block). Assert the block runs in a compiled workflow and emits an output of kind language_model_output that chains into a downstream consumer. The current unit test only exercises the manifest + _combine_prompt, not run() / the LOCAL↔REMOTE dispatch.

---

Escalation to code-owners

@PawelPeczek-Roboflow @grzegorz-roboflow @dkosowski87 — decision needed on Q2 (naming/contract): this PR introduces registry- and API-visible identifiers (cosmos-3-edge, cosmos-3-edge-world, task world-model, BackendType.CUSTOM, env COSMOS3_ENABLED) that become a hard backward-compat contract the moment weights are published to the production registry. The author has explicitly asked reviewers to confirm these names. Please confirm/authorize the names (and the merge-before/after-publication sequence in Q3) before this ships.

---

Re-review is not automatic: new commits are not auto-reviewed. After answering and/or pushing changes, add the claude-review label (remove & re-add to re-trigger) to request a fresh review. Unanswered IMPORTANT questions may keep this PR out of a release.

Reviewed at HEAD: 3d02838

[github-actions]

📦 Release coordination (maintainers)

inference-models requires a version bump for release. This PR adds two new model families to the inference_models package (cosmos-3-edge reasoner and cosmos-3-edge-world generator), new registry entries, a new task (world-model), and new configuration knobs.

• ✅ The contributor has already added a user-facing entry under ## Unreleased in inference_models/docs/changelog.md — no changelog action needed from the author.
• At release time, maintainers own the version change: bump the inference_models version, move the Unreleased entries into the final release section, and leave a fresh Unreleased heading.

No Execution Engine version bump is required: this PR adds a new Workflow block (additive type roboflow_core/cosmos3_edge@v1) but does not change EE behavior, so no docs/workflows/execution_engine_changelog.md entry or EXECUTION_ENGINE_V1_VERSION change is needed.

Note the still-open dependency item (requirements/requirements.cosmos.txt pins git SHAs and is not wired into any pypi extra) — tracked in the action-item comment and the author's own "Remaining before merge" checklist.

Reviewed at HEAD: 3d02838