Merge remote-tracking branch 'upstream/main' into sefi-image-diffusers

Author: jmliu206

.ai/AGENTS.md

@@ -30,14 +30,17 @@ Strive to write code as simple and explicit as possible.
 - **Models** — see [models.md](models.md) for model conventions, attention pattern, implementation rules, dependencies, and gotchas. For adding or converting a model, use the [model-integration](./skills/model-integration/SKILL.md) skill.
 - **Pipelines** — see [pipelines.md](pipelines.md) for pipeline conventions, patterns, and gotchas.
 - **Modular pipelines** — see [modular.md](modular.md) for modular pipeline conventions, patterns, and gotchas.
+- **Tests** — see [testing.md](testing.md) for test conventions: required test layers, tester mixins, and dummy-component rules.
 
 ## Skills
 
 Task-specific guides live in `.ai/skills/` and are loaded on demand by AI agents. Available skills include:
 
 - [model-integration](./skills/model-integration/SKILL.md) (adding/converting pipelines)
+- [custom-blocks](./skills/custom-blocks/SKILL.md) (packaging a `ModularPipelineBlocks` subclass for the Hub)
+- [diffusers-cli](./skills/diffusers-cli/SKILL.md) (running pipelines, inspecting schemas, and using the Diffusers CLI)
 - [self-review](./skills/self-review/SKILL.md) (pre-PR self-review against the project rules)
 
 ## Self-review before a PR
 
-Before opening a PR, run self-review against [review-rules.md](review-rules.md). The [self-review skill](skills/self-review/SKILL.md) runs this as the same pass the `@claude` CI reviewer uses.
+Before opening a PR, run self-review against [review-rules.md](review-rules.md). The [self-review skill](skills/self-review/SKILL.md) runs this as the same pass the `@claude` CI reviewer uses. Share the final report on the PR (description or comment) — see the skill for details.

.ai/models.md

@@ -6,6 +6,7 @@ Linked from `AGENTS.md`, `skills/model-integration/SKILL.md`, and `review-rules.
 ## Coding style
 
 - All layer calls should be visible directly in `forward` — avoid helper functions that hide `nn.Module` calls.
+- Prefer descriptive variable names over short ones. For example, prefer spelling out `query` over `q`.
 - Avoid graph breaks for `torch.compile` compatibility — do not insert NumPy operations in forward implementations and any other patterns that can break `torch.compile` compatibility with `fullgraph=True`.
 - No new mandatory dependency without discussion (e.g. `einops`). Optional deps guarded with `is_X_available()` and a dummy in `utils/dummy_*.py`.
 
@@ -182,4 +183,8 @@ Boolean gate. If `False` (default), calling that method raises `ValueError`. All
       ```
       See `transformer_flux.py`, `transformer_flux2.py`, `transformer_wan.py`, `unet_2d_condition.py`, and `pipeline_pixart_alpha.py` for reference usages. Never leave an unconditional `torch.float64` in the model.
 
-6. **Using `torch.empty`.** - Do not use `torch.empty` to initialize parameters. Use `torch.zeros` or `torch.ones`, instead.
+6. **Using `torch.empty`.** - Do not use `torch.empty` to initialize parameters. Use `torch.zeros` or `torch.ones`, instead.
+
+7. **Tensor contiguity.** - Non-contiguous tensors can degrade performance. Therefore, try to maintain contiguity
+of the tensors whenever possible. A non-contiguous tensor is usually produced because of the operations. A common
+example is a `flatten()` followed by a `transpose()`. This sequence is known to produce non-contiguous layouts. So, prefer calling `contiguous()` on the output tensor to maintain performance.

.ai/modular.md

@@ -80,3 +80,11 @@ src/diffusers/pipelines/<model>/
 7. **Don't modify the state of a registered component on the fly.** From inside `__call__` or other helper methods, don't change the state of `self.text_encoder` / `self.transformer` / `self.vae` — no in-place `.to(dtype/device)`, no setting attributes/buffers or swapping submodules. Components are shared and routinely reused across pipelines, so a per-call mutation may silently change another pipeline's outputs. You should pass a component that's already in the right state, and document that expectation explicitly. Only when that's genuinely inconvenient and you must change state for the duration of a call — e.g. swapping in an attention processor — save the original first and restore it before returning, so the component is left exactly as you found it. The PAG pipelines are the reference for this: `pipeline_pag_sd.py` snapshots `original_attn_proc = self.unet.attn_processors`, installs the PAG processors for the denoising loop, then calls `self.unet.set_attn_processor(original_attn_proc)` at the end of `__call__`.
 
 8. **Don't reimplement `DiffusionPipeline`.** A pipeline subclass adds only *pipeline-specific* steps (`__call__`, `check_inputs`, `encode_prompt`, `prepare_latents`, …). Device placement, offloading, and component loading/registration already live on the base class — don't add your own; use what's there.
+
+9. **Build `callback_kwargs` with a loop, never a dict comprehension.** `{k: locals()[k] for k in callback_on_step_end_tensor_inputs}` always raises `KeyError`: inside a comprehension, `locals()` is the comprehension's own scope, not `__call__`'s. Use the standard form (see `pipeline_stable_diffusion.py`):
+   ```python
+   callback_kwargs = {}
+   for k in callback_on_step_end_tensor_inputs:
+       callback_kwargs[k] = locals()[k]
+   ```
+   The bug is invisible until someone actually passes `callback_on_step_end` — the `PipelineTesterMixin` callback tests are what catch it.

.ai/pipelines.md

@@ -7,6 +7,7 @@ Before reviewing, read and apply the guidelines in:
 - [models.md](models.md) — model conventions, attention pattern, implementation rules, dependencies, gotchas
 - [pipelines.md](pipelines.md) — pipeline conventions, coding style, gotchas
 - [modular.md](modular.md) — modular pipeline conventions, patterns, common mistakes
+- [testing.md](testing.md) — test conventions: required test layers, tester mixins, dummy-component rules. When a PR adds or changes tests, check them against this guide.
 - [skills/model-integration/pitfalls.md](skills/model-integration/pitfalls.md) — known pitfalls causing numerical discrepancies between the reference implementation and the diffusers port (dtype mismatches, config assumptions, etc.)
 
 ## Common mistakes
@@ -20,7 +21,7 @@ Common mistakes are covered in the common-mistakes / gotcha sections in [AGENTS.
 A PR can leave existing docs stale or surface a pattern worth recording. Scan the docs related to what the PR touches and flag updates as a **suggestions / additional info** section (not blocking):
 
 - **Usage docs.** New or changed public behavior — a new pipeline/model, a new argument, changed defaults, a renamed API — should have matching updates in `docs/`, docstrings, and examples. Flag any that now describe outdated behavior or that are missing for the new surface.
-- **Agent docs.** If the review turns up a rule, pattern, or common gotcha that isn't written down yet — especially one the author got wrong or that you had to reason out — propose adding it to the relevant agent guide ([AGENTS.md](AGENTS.md), [models.md](models.md), [pipelines.md](pipelines.md), [modular.md](modular.md), a skill, or this file) so the next contributor/agent gets it for free instead of repeating the mistake.
+- **Agent docs.** If the review turns up a rule, pattern, or common gotcha that isn't written down yet — especially one the author got wrong or that you had to reason out — propose adding it to the relevant agent guide ([AGENTS.md](AGENTS.md), [models.md](models.md), [pipelines.md](pipelines.md), [modular.md](modular.md), a skill, or this file) so the next contributor/agent gets it for free instead of repeating the mistake. Human review comments on the PR are a good source for these: if a human reviewer pointed something out and your review missed it, that usually indicates a doc gap — figure out what's missing and propose the addition.
 
 ## Dead code analysis (new models)
 

.ai/review-rules.md

@@ -0,0 +1,157 @@
+---
+name: custom-blocks
+description: >
+  Use when the user has written (or wants to write) a `ModularPipelineBlocks`
+  subclass in a local Python file and needs to package it into a Hub-uploadable
+  directory. Covers the workflow from a single `block.py` file to a published
+  custom-block repo that consumers can load via
+  `ModularPipeline.from_pretrained(<repo>, trust_remote_code=True)`.
+---
+
+## What this skill is for
+
+A `ModularPipelineBlocks` subclass is a unit of pipeline logic — input/output spec plus a `__call__` — that
+slots into diffusers' modular pipeline composition. Once you have one defined locally, you almost always want to
+publish it as a small Hub repo so others can `from_pretrained` it. `diffusers-cli custom_blocks` automates the
+packaging step: it parses your Python file, instantiates the chosen block class, and writes a
+`save_pretrained`-style directory in your cwd that's ready to push to the Hub.
+
+Use this skill when:
+
+- The user is writing a custom modular block and asks "how do I publish this?" or "package this for the Hub".
+- The user has a `block.py` (or similar) file with one or more `ModularPipelineBlocks` subclasses.
+- You're scaffolding a new modular pipeline repo and need the on-disk layout that `ModularPipelineBlocks.from_pretrained`
+  expects.
+
+Don't use this skill for: running an existing modular pipeline (`diffusers-cli run`), introspecting one
+(`diffusers-cli schema`), or writing the block class itself — this skill packages an *already-written* block.
+
+## The end-to-end workflow
+
+```
+[you: write block.py] → diffusers-cli custom_blocks → [packaged dir in cwd]
+                                                            ↓
+                                            hf upload <repo> .
+                                                            ↓
+                                consumers: ModularPipeline.from_pretrained(<repo>, trust_remote_code=True)
+                                           diffusers-cli schema --model <repo> --trust-remote-code
+                                           diffusers-cli run --model <repo> --trust-remote-code ...
+```
+
+The skill covers the middle box. The bookends (writing the block and uploading) are out of scope.
+
+## Command surface
+
+```bash
+diffusers-cli custom_blocks [--block_module_name <file.py>] [--block_class_name <ClassName>]
+```
+
+### Flags
+
+- `--block_module_name <file>` — Python file containing the block class. Defaults to `block.py` in the cwd.
+- `--block_class_name <name>` — Which class in the file to package. Optional: if omitted, the CLI parses the
+  file with `ast`, finds every class that inherits from `ModularPipelineBlocks`, and uses the first one (with
+  an info log naming the others). Specify explicitly when the file defines more than one block and you want a
+  specific one.
+
+### What it does
+
+1. **AST scan**: parses `<file>` without executing it, walks top-level `ClassDef` nodes, and collects every
+   class whose `bases` include `ModularPipelineBlocks`.
+2. **Pick a class**: uses `--block_class_name` if given, else the first found. Errors with the list of available
+   classes if your name doesn't match.
+3. **Load and save**: imports the file via `importlib.util.spec_from_file_location` (this does execute the
+   module — make sure your block.py is something you trust to run), instantiates the chosen class with no
+   constructor args, and calls `.save_pretrained(os.getcwd())`.
+
+The result is a Hub-uploadable directory laid out the way `ModularPipelineBlocks.from_pretrained` expects:
+your block source, an `auto_map` in the config so consumers know to load it with `trust_remote_code=True`,
+and any artifacts `save_pretrained` writes for that block class.
+
+## End-to-end example
+
+Given a `block.py` like:
+
+```python
+from diffusers.modular_pipelines import ModularPipelineBlocks, InputParam, OutputParam
+
+class MyDenoiseBlock(ModularPipelineBlocks):
+    model_name = "my-denoise"
+
+    @property
+    def inputs(self):
+        return [
+            InputParam("latents", type_hint="torch.Tensor", required=True, description="Noisy latents."),
+            InputParam("guidance_scale", type_hint="float", default=7.5),
+        ]
+
+    @property
+    def intermediate_outputs(self):
+        return [OutputParam("latents", type_hint="torch.Tensor")]
+
+    def __call__(self, components, state):
+        # ... denoising logic ...
+        return components, state
+```
+
+Package it:
+
+```bash
+diffusers-cli custom_blocks --block_module_name block.py
+```
+
+Output in cwd:
+
+```
+./
+├── block.py
+├── modular_config.json  # contains auto_map → MyDenoiseBlock
+└── (any state files MyDenoiseBlock.save_pretrained writes)
+```
+
+Upload to the Hub:
+
+```bash
+hf upload my-user/my-denoise-block .
+```
+
+Consumers can now use it:
+
+```python
+from diffusers import ModularPipeline
+pipe = ModularPipeline.from_pretrained("my-user/my-denoise-block", trust_remote_code=True)
+```
+
+Or via CLI:
+
+```bash
+diffusers-cli schema --model my-user/my-denoise-block --trust-remote-code
+diffusers-cli run --model my-user/my-denoise-block --trust-remote-code \
+    --pipeline-kwargs '{"latents": "...", "guidance_scale": 7.5}'
+```
+
+## Common errors
+
+- **`Could not parse '<file>': SyntaxError`** — the file isn't valid Python. Fix the syntax; the AST step runs
+  before any execution.
+- **`block_class_name could not be retrieved. Available classes from <file>: [ClassA, ClassB]`** — your
+  `--block_class_name` doesn't match any `ModularPipelineBlocks` subclass found. Pick from the list shown.
+- **No classes found**: silent — the command will try to use the first entry in an empty list and raise
+  `IndexError`. If you hit that, double-check your class actually inherits from `ModularPipelineBlocks`
+  (the AST scan looks for that literal base-class name; aliased imports like `from diffusers import ...
+  as MPB` won't be picked up).
+- **Block requires constructor args**: the command calls `<ClassName>()` with no args. If your block needs
+  `__init__` parameters, refactor to take them from `state`/`components` at `__call__` time instead, or
+  hardcode defaults in `__init__`.
+
+## Verifying the install
+
+If `diffusers-cli` isn't on PATH, see the install verification section of
+[`../diffusers-cli/SKILL.md`](../diffusers-cli/SKILL.md#verifying-the-cli-is-installed).
+
+## Related
+
+- [`diffusers-cli` skill](../diffusers-cli/SKILL.md) — once your block is uploaded, `schema`/`run`
+  let you call it from the terminal without writing Python.
+- diffusers' [modular pipelines docs](../../../docs/source/en/modular_diffusers) — for writing the block
+  class itself.

.ai/skills/custom-blocks/SKILL.md

@@ -0,0 +1,68 @@
+---
+name: diffusers-cli
+description: >
+  Use when the user wants to run a diffusers pipeline from a terminal (one-off
+  generation, batch jobs, smoke-testing a new model), run on HF Sandbox
+  hardware via `--remote`, introspect a pipeline's input schema before
+  calling it, or attach a LoRA at inference time. Prefer this over writing
+  ad-hoc Python scripts for generation tasks.
+---
+
+## Overview
+
+`diffusers-cli` is the shipped CLI in `src/diffusers/commands/`. Subcommands relevant to agentic use:
+
+| Command         | Purpose                                                                                                                                                                                       |
+| --------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `run`      | Run any `DiffusionPipeline` or `ModularPipeline`. Forwards `--pipeline-kwargs` verbatim, saves output by detecting its runtime type, optionally runs on HF Jobs via `--remote`.                |
+| `schema`      | Print the input schema for a pipeline repo (kwarg names, types, defaults, descriptions). **No weights downloaded** — only the small index file.                                               |
+| `custom_blocks` | Package a local `ModularPipelineBlocks` subclass for the Hub.                                                                                                                                 |
+| `env`           | Print versions of diffusers + torch + transformers + accelerate + safetensors + CUDA + GPU info. Use when investigating environment issues, dtype/precision support, or building bug reports. |
+
+## When to read which file
+
+Most agentic work goes through `run`. Read the matching reference file before constructing a command:
+
+- **[`run.md`](run.md)** — full reference for `diffusers-cli run`. Covers `--pipeline-kwargs`
+  semantics and the shell-quoting gotcha, LoRA via `--lora`, optimization flags (`--dtype`, `--cpu-offload`,
+  `--attention-backend`, `--vae-tiling/slicing`), output handling and `--push-to` bucket uploads, the full
+  `--remote` HF Jobs flow (image, container command, log streaming, timing payload, artifact download), and
+  context parallel (`--context-parallel`) for both local-torchrun and `--remote` paths.
+
+The other commands are small enough that `diffusers-cli <command> --help` is the canonical reference:
+
+```bash
+diffusers-cli schema --help
+diffusers-cli custom_blocks --help
+diffusers-cli env --help
+```
+
+## When NOT to use this skill
+
+- Multi-stage workflows where you need intermediate tensor manipulation between pipelines → write Python.
+- Training or fine-tuning → CLI only covers inference.
+- Anything requiring `quantization_config` or other low-level loader knobs not exposed by the CLI flags → write
+  Python. (`device_map` is exposed as `--device-map`; see [run.md](run.md#optimization-flags).)
+
+## Verifying the CLI is installed
+
+The console entry point is registered in `pyproject.toml` (`diffusers-cli =
+"diffusers.commands.diffusers_cli:main"`). If `diffusers-cli` is not on PATH after `pip install -e .`, reinstall
+with `pip install -e . --force-reinstall --no-deps` and check `which diffusers-cli`. If the installed binary is
+missing recent features (e.g. you see `unrecognized arguments: --lora`), reinstall.
+
+## Output formats
+
+`--format {auto, human, agent, json}` (top-level flag, must appear before the subcommand):
+
+- **`human`** — plain-text indented output for terminals (default when not running under an agent harness). No ANSI color.
+- **`agent`** — TSV tables and `key=value` lines. Auto-selected when an agent env var is present
+  (`CLAUDECODE`, `CLAUDE_CODE`, `CODEX_SANDBOX`, `CURSOR_AI`, `AIDER_AI_CONTEXT`, `GH_COPILOT_AGENT`,
+  `AI_AGENT`). Token-cheap for LLM agents to read.
+- **`json`** — compact JSON. Use for programmatic parsing (scripts, services) where type fidelity and nested
+  structures matter.
+
+`stdout` carries data; `stderr` carries hints/warnings/progress — parseable output is never polluted.
+
+Rule of thumb: `--format json` for scripts that will `json.loads()` the output, otherwise leave it on
+auto-detect (`agent` for LLMs, `human` for terminals).