Skip to content

Commit e3a5c69

Browse files
[misc]: import add-model skill stack to .agents/skills/ (hao-ai-lab#1308)
Co-authored-by: Raghav <ragg04@gmail.com>
1 parent f633e30 commit e3a5c69

38 files changed

Lines changed: 4658 additions & 4 deletions

File tree

Lines changed: 173 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,173 @@
1+
---
2+
name: add-model-01-prep
3+
description: Use at the start of a FastVideo model port to gather required inputs, inspect/download HF weights, clone and install the official reference repo in the current environment, create a local_tests README skeleton, and produce a handoff before conversion or implementation.
4+
---
5+
6+
# Add Model Prep
7+
8+
## Goal
9+
10+
Prepare external assets and the shared parity-test environment for a FastVideo
11+
model port. Stop before writing conversion scripts, model components, pipeline
12+
code, registry entries, or executable parity tests.
13+
14+
## Ask First
15+
16+
Ask once, then proceed if the HF token is already exported:
17+
18+
```text
19+
Before prep: (1) official reference repo or Diffusers pipeline URL, (2) HF repo
20+
id or local weights path and whether it has a root model_index.json, (3) target
21+
model_family, (4) workload types, (5) which token env var is exported:
22+
HF_TOKEN, HUGGINGFACE_HUB_TOKEN, or HF_API_KEY, (6) may I stage clone and
23+
weights under the FastVideo repo root, and (7) may I install official reference
24+
dependencies into the current FastVideo conda/env for parity tests?
25+
```
26+
27+
Useful optional inputs: `pipeline_class`, `reference_dir`, `hf_revision`,
28+
`official_revision`, `reuse_hints`, `download_scope`.
29+
30+
## Rules
31+
32+
- Follow `../add-model/shared/common_rules.md` for token/auth safety, state files,
33+
escape hatches, and skip/pass semantics.
34+
- Run from the FastVideo repo root.
35+
- Use repo-relative defaults: `<ReferenceDir>/`,
36+
`official_weights/<model_family>/`, `converted_weights/<model_family>/`.
37+
- Install official reference deps into the current FastVideo environment, not a
38+
new venv/conda env, so parity tests run both implementations with one shared
39+
numeric stack.
40+
- If the reference is a Diffusers class/package instead of a cloneable repo,
41+
record import path and version instead of cloning.
42+
- Prep may create only the local-test README and `PORT_STATUS.md` skeletons;
43+
executable `.py` parity tests belong to `../add-model-02-parity/SKILL.md`.
44+
45+
## Escape Hatches
46+
47+
Follow `../add-model/shared/common_rules.md`. Prep-specific ask cases include
48+
overwriting an existing clone or weight directory, installing untrusted/private
49+
deps, choosing between incompatible official references, large downloads outside
50+
the agreed scope, or missing gated-repo auth setup by env var name.
51+
52+
## Workflow
53+
54+
1. Verify the repo:
55+
56+
```bash
57+
git rev-parse --show-toplevel
58+
```
59+
60+
Expected markers: `fastvideo/`, `scripts/checkpoint_conversion/`,
61+
`scripts/huggingface/download_hf.py`, `fastvideo/registry.py`.
62+
63+
2. Inspect HF or local weight layout:
64+
65+
```bash
66+
python ".agents/skills/add-model-01-prep/scripts/inspect_hf_layout.py" \
67+
"Org/Model" \
68+
--revision "<revision>" \
69+
--json
70+
```
71+
72+
For a local path, replace `Org/Model` with `/path/to/weights`. Record
73+
`source_layout`, `needs_conversion`, `model_index_class`, and
74+
`components_seen`.
75+
76+
3. Download HF weights if needed:
77+
78+
```bash
79+
python ".agents/skills/add-model-01-prep/scripts/download_hf_weights.py" \
80+
"Org/Model" \
81+
"official_weights/<model_family>" \
82+
--revision "<revision>"
83+
```
84+
85+
For selected files, repeat `--file-name`. For partial snapshots, repeat
86+
`--allow-pattern` or `--ignore-pattern`. If the user provided a local path,
87+
record it instead of copying large weights by default.
88+
89+
4. Clone the official reference repo if applicable:
90+
91+
```bash
92+
python ".agents/skills/add-model-01-prep/scripts/clone_reference_repo.py" \
93+
"<official_repo_url>" \
94+
"<ReferenceDir>" \
95+
--branch "<tag-or-branch>" \
96+
--commit "<commit-sha>" \
97+
--update-gitignore
98+
```
99+
100+
Omit `--branch`, `--commit`, or `--update-gitignore` when not needed. The
101+
helper refuses to overwrite existing paths and prints remote/HEAD instead.
102+
103+
5. Keep prep assets ignored. Ensure `.gitignore` includes relevant entries:
104+
105+
```gitignore
106+
/<ReferenceDir>/
107+
/official_weights/
108+
/converted_weights/
109+
```
110+
111+
6. Follow the official repo's setup instructions in the current environment.
112+
Inspect dependency files and README install docs before installing anything:
113+
114+
- `README*`, install docs, or model-card instructions.
115+
- `requirements*.txt`, `pyproject.toml`, `setup.py`, `environment.yml`.
116+
117+
Use the current FastVideo conda/env. Do not create a new env even if upstream
118+
docs recommend one; translate the needed install commands into the active env.
119+
Prefer editable/no-deps first so the official source is importable without
120+
changing shared pins:
121+
122+
```bash
123+
uv pip install --no-deps -e ./<ReferenceDir>
124+
```
125+
126+
Then install only missing official deps needed for parity imports. Stop before
127+
installing requirements that would change FastVideo's core stack. If upstream
128+
requires private/non-PyPI deps, record that parity needs a local stub helper
129+
rather than pretending setup is complete.
130+
131+
7. Create the model-family local test skeleton and top-level port state file:
132+
133+
```bash
134+
mkdir -p tests/local_tests/<model_family>
135+
cp ".agents/skills/add-model-01-prep/templates/local_tests_readme.md" \
136+
tests/local_tests/<model_family>/README.md
137+
cp ".agents/skills/add-model-01-prep/templates/port_status.md" \
138+
tests/local_tests/<model_family>/PORT_STATUS.md
139+
```
140+
141+
Edit every placeholder in the README and `PORT_STATUS.md`. The README gives
142+
later review agents enough information to reproduce the shared environment and
143+
run/review parity work:
144+
145+
- official code URL or import path, local clone path, and commit/version;
146+
- HF URL or local weight path, revision, access notes, and token env var name
147+
only;
148+
- commands already run and any blocked official dependency installs;
149+
- shared-env install commands to re-run without changing core pins;
150+
- expected local parity test paths and pytest commands;
151+
- private-dependency stubs or known setup gaps;
152+
- PR/review notes explaining which parity tests are required before handoff.
153+
154+
Do not include raw tokens, absolute cache paths that are not repo-reproducible,
155+
or large generated outputs. If prep is blocked before imports work, still create
156+
the README with `official_env_status=blocked` and the exact blocker.
157+
158+
`PORT_STATUS.md` must follow `../add-model/contracts/port_state.md`. Record open
159+
questions and prep issues immediately, using stable IDs such as `Q001` and
160+
`I001`. Keep resolved questions/issues in the table with a resolution instead of
161+
deleting them.
162+
163+
## Handoff
164+
165+
End with the canonical prep handoff contract from
166+
`../add-model/contracts/prep_handoff.md` and update the shared state files before
167+
handoff.
168+
169+
## Helper Scripts
170+
171+
- `scripts/inspect_hf_layout.py`: classify HF/local layout.
172+
- `scripts/download_hf_weights.py`: download HF snapshot or selected files.
173+
- `scripts/clone_reference_repo.py`: clone reference repo safely.
Lines changed: 123 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,123 @@
1+
#!/usr/bin/env python3
2+
"""Clone an official reference repo without overwriting existing paths."""
3+
4+
from __future__ import annotations
5+
6+
import argparse
7+
import subprocess
8+
import sys
9+
from pathlib import Path
10+
11+
12+
def parse_args() -> argparse.Namespace:
13+
parser = argparse.ArgumentParser(
14+
description="Clone a reference repo for FastVideo parity tests."
15+
)
16+
parser.add_argument("repo_url", help="Official reference repository URL")
17+
parser.add_argument("target_dir", help="Directory to clone into")
18+
parser.add_argument("--branch", help="Branch or tag to clone")
19+
parser.add_argument("--commit", help="Commit SHA to check out after clone")
20+
parser.add_argument(
21+
"--update-gitignore",
22+
action="store_true",
23+
help="Add the target directory to .gitignore if missing",
24+
)
25+
parser.add_argument(
26+
"--gitignore",
27+
default=".gitignore",
28+
help="Path to gitignore file when --update-gitignore is used",
29+
)
30+
return parser.parse_args()
31+
32+
33+
def run(command: list[str], check: bool = True) -> subprocess.CompletedProcess[str]:
34+
return subprocess.run(
35+
command,
36+
check=check,
37+
text=True,
38+
capture_output=True,
39+
)
40+
41+
42+
def print_existing_repo_info(target: Path) -> int:
43+
print(f"target_exists: {target}")
44+
if not (target / ".git").exists():
45+
print("error: target exists but is not a git repo", file=sys.stderr)
46+
return 1
47+
48+
remote = run(["git", "-C", str(target), "remote", "-v"], check=False)
49+
head = run(["git", "-C", str(target), "rev-parse", "HEAD"], check=False)
50+
if remote.stdout:
51+
print("remote_v:")
52+
print(remote.stdout.rstrip())
53+
if head.stdout:
54+
print(f"head: {head.stdout.strip()}")
55+
print("not_overwritten: true")
56+
return 0
57+
58+
59+
def gitignore_entry_for(target: Path) -> str:
60+
root = Path.cwd().resolve()
61+
resolved = target.resolve()
62+
try:
63+
relative = resolved.relative_to(root)
64+
except ValueError as exc:
65+
raise ValueError(
66+
"--update-gitignore requires target_dir to be under the current directory"
67+
) from exc
68+
69+
text = relative.as_posix().rstrip("/")
70+
return "/" + text + "/"
71+
72+
73+
def update_gitignore(path: Path, target: Path) -> bool:
74+
entry = gitignore_entry_for(target)
75+
existing = path.read_text().splitlines() if path.exists() else []
76+
if entry in existing:
77+
return False
78+
79+
new_text = "\n".join(existing).rstrip("\n")
80+
if new_text:
81+
new_text += "\n"
82+
new_text += entry + "\n"
83+
path.write_text(new_text)
84+
return True
85+
86+
87+
def main() -> int:
88+
args = parse_args()
89+
target = Path(args.target_dir)
90+
91+
if target.exists():
92+
return print_existing_repo_info(target)
93+
94+
command = ["git", "clone", "--depth", "1"]
95+
if args.branch:
96+
command.extend(["--branch", args.branch])
97+
command.extend([args.repo_url, str(target)])
98+
99+
try:
100+
run(command)
101+
if args.commit:
102+
run(["git", "-C", str(target), "fetch", "--depth", "1", "origin", args.commit])
103+
run(["git", "-C", str(target), "checkout", args.commit])
104+
except subprocess.CalledProcessError as exc:
105+
if exc.stdout:
106+
print(exc.stdout, end="")
107+
if exc.stderr:
108+
print(exc.stderr, end="", file=sys.stderr)
109+
return exc.returncode
110+
111+
head = run(["git", "-C", str(target), "rev-parse", "HEAD"])
112+
print(f"cloned: {target}")
113+
print(f"head: {head.stdout.strip()}")
114+
115+
if args.update_gitignore:
116+
changed = update_gitignore(Path(args.gitignore), target)
117+
print(f"gitignore_updated: {str(changed).lower()}")
118+
119+
return 0
120+
121+
122+
if __name__ == "__main__":
123+
raise SystemExit(main())

0 commit comments

Comments
 (0)