Add failed-ingest recovery and pre-reuse changelog validation [skip-sweep] by Oseltamivir · Pull Request #1821 · SemiAnalysisAI/InferenceX

Oseltamivir · 2026-06-18T02:12:50Z

Summary

add /recover-failed-ingest <failed-run-or-job-url> [source-run-id] as a thin command over tested recovery helpers
run check-changelog before reuse authorization or setup; validate sweep-label exclusivity, duplicate YAML keys, full schema, byte-exact append-only history, PR links, separators, and generated sweep configuration
allow XXX only on PR additions; canonicalize it in the supported merge helper and reject it on push to main
stop link-only correction PRs after the CPU gate via has-additions=false
preserve current main changelog bytes during conflict resolution, always push a fresh synchronization SHA, wait for check-changelog on that SHA, and verify the PR head again before admin merge
validate reusable fixed-sequence, agentic point/raw/aggregate, and eval raw/aggregate identities exactly, including duplicate and unexpected rows and expired artifacts
harden recovery workflow validation for exact confirmation gating and workflow/job-level read-only permissions
correct the two stale XXX links and the incorrect repository link in perf-changelog.yaml
make no repository-rules changes

Validation

73 focused validator/reuse/recovery tests passed
156 matrix-generation/schema tests passed
full utils/ suite: 290 passed; one pre-existing isolated failure remains in test_processor_loads_traces_jsonl_for_theoretical_cache, outside this diff
ruff, actionlint, shellcheck, bash -n, py_compile, and git diff --check passed
both historical recovery workflows pass the CPU-only workflow validator
live PR [NV]Add GLM-5 NVFP4 GB300 disagg-non-mtp TRT-LLM benchmarks via Dynamo #1798 artifacts validate as 20 fixed-sequence rows and 2 eval jobs
live agentic run 27718128616 validates as 4 exact point/raw/aggregate identities
failed push run 27735927631 proved fail-closed ordering; restored runs passed
final push test: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/27737488235
final production PR gate: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/27737489942

Note

Medium Risk
Changes merge-time ingest gating and artifact validation for official benchmark database publishing; mistakes could block merges or allow incorrect reuse, though extensive tests and fail-closed checks mitigate that.

Overview
Adds a CPU-only check-changelog job ahead of reuse authorization and sweep setup in run-sweep.yml, replacing the old trailing-newline check and moving sweep-label conflict detection into that job. Validation enforces append-only history, schema/duplicate-key rules, PR link policy (XXX only on PR additions), generated sweep config, and gates downstream work with has-additions so link-only fixes stop before GPU setup.

Introduces validate_perf_changelog.py, prepare_perf_changelog_merge.py, and recover_failed_ingest.py, plus a /recover-failed-ingest command and test-changelog-gate workflow with broad pytest coverage (including parsing real run-sweep.yml if conditions). merge_with_reuse.sh now uses the merge helpers, always pushes a sync SHA, waits for check-changelog, and re-checks PR head before admin merge.

Reuse validation tightens: find_reusable_sweep_run.py accepts agentic point artifacts and ignores expired artifacts; validate_reusable_sweep_artifacts.py requires exact fixed-sequence, agentic (point/raw/aggregate), and eval identities and rejects duplicates/unexpected rows. Merge-time artifact filtering keeps agentic_* and drops agentic_aggregated from the allowlist change shown in diff (actually it changed from agentic_aggregated to agentic_*).

perf-changelog.yaml gets three canonical PR link corrections. Docs (README, KLAUD_DEBUG.md) describe the new gate and merge path.

^{Reviewed by Cursor Bugbot for commit 5b9ce1e. Bugbot is set up for automated code reviews on this repo. Configure here.}

Oseltamivir · 2026-06-18T02:18:17Z

perf-changelog.yaml audit results:

Structural checks pass: 495 entries parse as YAML and validate as ChangelogEntry; the file has a final newline and no CR characters, tabs, or NUL bytes.
There are 16 trailing-whitespace lines. Thirteen are whitespace-only separators; content is affected at L2036, L2483, and the multiline pr-link at L3596-L3597.
Two stale placeholders are resolvable: L2531 should reference PR Update minimaxm2.5-fp8-mi355x-vllm vLLM ROCm image to v0.21.0 #1410, and L2585 should reference PR Update dsr1-fp4-b300-sglang SGLang image to v0.5.12-cu130 #1418.
L3702 points to nonexistent NVIDIA/InferenceMAX#1722; the merged source is SemiAnalysisAI/InferenceX#1722.
Five exact duplicate pairs exist. Each traces to an explicit reappend or sweep-retrigger commit, so they are intentional history and were not deduplicated.

No historical changelog lines were edited: the file is append-only, and non-whitespace deletions are rejected by process_changelog.py. The command now reports these conditions and applies strict formatting/link checks to entries introduced by the target PR.

Oseltamivir · 2026-06-18T03:12:56Z

Fixed the three audited changelog links in commit ae01bdb3:

minimaxm2.5-fp8-mi355x-vllm: XXX -> PR Update minimaxm2.5-fp8-mi355x-vllm vLLM ROCm image to v0.21.0 #1410
dsr1-fp4-b300-sglang: XXX -> PR Update dsr1-fp4-b300-sglang SGLang image to v0.5.12-cu130 #1418
minimaxm2.5-fp4-b200-trt: corrected NVIDIA/InferenceMAX#1722 -> SemiAnalysisAI/InferenceX#1722

All three PRs are merged. The full file still parses and validates as 495 ChangelogEntry records, every pr-link is now a canonical InferenceX PR URL, and no surrounding whitespace or entry ordering changed. The PR title and commit retain [skip-sweep] because these are historical metadata replacements.

Oseltamivir · 2026-06-18T03:47:16Z

Added commit 3d432bb7 to close the reuse validation gap.

The new order is check-changelog -> reuse-sweep-gate -> setup/GPU jobs. check-changelog always evaluates the final PR changelog diff first. It validates the complete YAML/schema, append-only ordering, changed-line whitespace, PR-link rules, and runs the same process_changelog.py generation used by setup for new entries.

utils/merge_with_reuse.sh now waits up to 15 minutes for check-changelog to pass on the exact pushed conflict-resolution SHA before invoking the admin squash merge. A failed, cancelled, or missing check stops the merge.

This reproduces and rejects both prior failure classes: PR #1767 malformed YAML and PR #1798 historical deletion/invalid base state.

Oseltamivir · 2026-06-18T03:59:22Z

Added push-triggered changelog gate coverage in commit a77ab9dc.

Push test run 27735671982 passed 33 tests, accepted a known-good push diff, rejected the PR [NVIDIA][GB300] update DSR1 FP8 GB300 TRTLLM image to latest #1767/[NV]Add GLM-5 NVFP4 GB300 disagg-non-mtp TRT-LLM benchmarks via Dynamo #1798 bad merge states, validated this PR, then confirmed validation -> reuse gate -> setup skipped.
Production run 27735673180 independently ran check-changelog on the same SHA and completed it before gate/setup resolution.
No GPU jobs were launched.

Oseltamivir · 2026-06-18T04:06:33Z

Fail-closed behavior was tested with a temporary intentional failure.

Failed push run 27735927631: Test changelog validation rejected the known-invalid PR [NVIDIA][GB300] update DSR1 FP8 GB300 TRTLLM image to latest #1767 changelog state; Simulate authorized reuse and Setup must be skipped were skipped; the independent result assertion passed.
Restored push run 27735961178: the normal validation/reuse path returned green after removing the temporary failure injection.
Restored production run 27735962638: check-changelog passed on the final head.

The final workflow content is identical to commit a77ab9dc; only the intentional failure and restoration commits remain in history.

…ingest-command

Oseltamivir · 2026-06-18T04:55:15Z

Final verification for e0a422b8da5493cc54c054cd2d3d619dff0fa909:

All 9 review threads have been addressed and resolved.
Final push test passed: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/27737488235
Final production PR gate passed: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/27737489942
The intentional fail-closed test failed as expected: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/27735927631
Focused tests: 73 passed. Matrix tests: 156 passed. Full utils suite: 290 passed, with one unrelated pre-existing failure in test_processor_loads_traces_jsonl_for_theoretical_cache.
Live artifact validation passed for run 27622347964 (20 fixed-sequence rows, 2 eval jobs) and run 27718128616 (4 agentic rows).
ruff, actionlint, shellcheck, bash -n, py_compile, and git diff --check passed.
No repository rules or branch-protection settings were changed.

…s [skip-sweep] Add utils/test_run_sweep_gating.py: parse the real check-changelog -> reuse-sweep-gate -> setup `if` conditions from run-sweep.yml and evaluate them with a minimal GitHub Actions expression engine, so coverage cannot drift from production. Covers every gating case: a curated 18-case truth table plus a 3,464-case exhaustive cross-product (action x draft x label-config x label-name x validator outcome x has-additions x reuse x skip-sweep) cross-checked against an independent reference spec. The engine is grounded against the real outcome of run-sweep run 27737489942. Wire the test into test-changelog-gate.yml (the prior simulation jobs are kept as a real-Actions smoke test).

The gating DAG (check-changelog -> reuse-sweep-gate -> setup) is now covered exhaustively by utils/test_run_sweep_gating.py against the real run-sweep.yml conditions, so the hand-copied simulation jobs (test-reuse-sweep-gate, test-setup, test-metadata-only, test-metadata-setup, verify-ordering) and the stub has-additions output are redundant. Reduce test-changelog-gate.yml to a lean pytest workflow matching test-matrix-logic.yml / test-process-result.yml, keeping the unit tests, historical push-validation pins, and live-PR check.

Opening or reopening a PR that already carries a sweep label triggered a full benchmark sweep via run-sweep.yml. Restrict pull_request triggers to ready_for_review, synchronize, labeled, and unlabeled so a sweep starts only on an explicit push or (re)label. Update the gating tests to assert opened/reopened are excluded and drop those cases from the exhaustive cross-product.

…_tests [skip-sweep] Relocate the four test files added by this PR (test_validate_perf_changelog, test_prepare_perf_changelog_merge, test_recover_failed_ingest, test_run_sweep_gating) into utils/changelog_gate_tests/. A conftest.py prepends utils/ to sys.path so the top-level module imports still resolve under pytest's prepend import mode, and the repo-root lookups move from parents[1] to parents[2]. Source modules stay in utils/, so no production invocation paths change; only test-changelog-gate.yml's pytest targets and path filter are updated. The two pre-existing reuse tests remain colocated in utils/.

cursor

Cursor Bugbot has reviewed your changes and found 1 potential issue.

There are 2 total unresolved issues (including 1 from previous review).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 0452e5e. Configure here.}

Add a regression test for resolve_conflict_bytes with more than one appended PR contribution, asserting entries are separated by exactly one blank line with a single trailing newline. This was previously untested; the behavior is correct (each block ends in a newline, so the single-newline join yields one blank line, and resolve_conflict_bytes self-validates via validate_raw_change).

Oseltamivir requested a review from a team June 18, 2026 02:12

github-project-automation Bot added this to InferenceMAX Board Jun 18, 2026

cursor Bot reviewed Jun 18, 2026

View reviewed changes

Comment thread .claude/commands/recover-failed-ingest.md Outdated

Comment thread .claude/commands/recover-failed-ingest.md Outdated

Comment thread .claude/commands/recover-failed-ingest.md Outdated

Oseltamivir force-pushed the workflow/add-recover-ingest-command branch 2 times, most recently from d736775 to 2a17676 Compare June 18, 2026 02:18

cursor Bot reviewed Jun 18, 2026

View reviewed changes

Comment thread .claude/commands/recover-failed-ingest.md Outdated

docs: add failed-ingest recovery command [skip-sweep]

ae01bdb

Oseltamivir force-pushed the workflow/add-recover-ingest-command branch from 2a17676 to ae01bdb Compare June 18, 2026 03:12

Oseltamivir changed the title ~~Add failed-ingest recovery command~~ Add failed-ingest recovery command [skip-sweep] Jun 18, 2026

cursor Bot reviewed Jun 18, 2026

View reviewed changes

Comment thread .claude/commands/recover-failed-ingest.md

Comment thread .claude/commands/recover-failed-ingest.md Outdated

fix(ci): validate changelog before sweep reuse [skip-sweep]

3d432bb

Oseltamivir changed the title ~~Add failed-ingest recovery command [skip-sweep]~~ Add failed-ingest recovery and pre-reuse changelog validation [skip-sweep] Jun 18, 2026

cursor Bot reviewed Jun 18, 2026

View reviewed changes

Comment thread utils/merge_with_reuse.sh

Comment thread utils/validate_perf_changelog.py Outdated

test(ci): add push coverage for changelog gate [skip-sweep]

a77ab9d

cursor Bot reviewed Jun 18, 2026

View reviewed changes

Comment thread utils/validate_perf_changelog.py Outdated

Oseltamivir added 2 commits June 18, 2026 12:04

test(ci): exercise changelog gate failure [skip-sweep]

0aa145a

test(ci): restore changelog gate success path [skip-sweep]

7a44b36

Oseltamivir added 3 commits June 18, 2026 12:37

fix(ci): harden changelog reuse and recovery [skip-sweep]

19d98b1

Merge remote-tracking branch 'origin/main' into workflow/add-recover-…

32d1bac

…ingest-command

fix(ci): close reuse validation edge cases [skip-sweep]

e0a422b

cursor Bot reviewed Jun 18, 2026

View reviewed changes

Comment thread utils/validate_reusable_sweep_artifacts.py

Oseltamivir added 2 commits June 18, 2026 14:55

cursor Bot reviewed Jun 18, 2026

View reviewed changes

Comment thread utils/prepare_perf_changelog_merge.py

Oseltamivir merged commit 862fcad into main Jun 18, 2026
31 checks passed

Oseltamivir deleted the workflow/add-recover-ingest-command branch June 18, 2026 08:10

github-project-automation Bot moved this to Done in InferenceMAX Board Jun 18, 2026

Oseltamivir mentioned this pull request Jun 19, 2026

[codex] Validate changelog matrix before sweep reuse #1853

Merged

Conversation

Oseltamivir commented Jun 18, 2026 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Validation

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Oseltamivir commented Jun 18, 2026

Uh oh!

Uh oh!

Oseltamivir commented Jun 18, 2026

Uh oh!

Uh oh!

Uh oh!

Oseltamivir commented Jun 18, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Oseltamivir commented Jun 18, 2026

Uh oh!

Oseltamivir commented Jun 18, 2026

Uh oh!

Oseltamivir commented Jun 18, 2026

Uh oh!

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Oseltamivir commented Jun 18, 2026 •

edited by cursor Bot

Loading