Apply ingestion state during archive discovery

[tomvothecoder]

Description

Apply ingestion state during archive discovery so known execution IDs skip earlier and archive scan counters stay consistent.

• Skip previously processed execution IDs before validation during discovery
• Keep case-level logging and summary counters consistent with early skips
• Preserve merged processed execution state after successful ingestion
• Closes [DevOps]: Apply ingestion state during archive discovery #238

Checklist

• Code follows project style guidelines
• Self-reviewed code
• No new warnings
• Tests added or updated (if needed)
• All tests pass (locally and CI/CD)
• Documentation/comments updated (if needed)
• Breaking change noted (if applicable)

Deployment Notes (if any)

None.

[Copilot]

Pull request overview

This PR updates the NERSC archive ingestor’s discovery phase to apply persisted ingestion state earlier (during directory discovery) so previously processed execution IDs are skipped before validation, while keeping discovery/logging counters consistent and preserving merged processed state after ingestion.

Changes:

• Apply “already processed” filtering during archive execution discovery via a precomputed processed_ids_by_key mapping.
• Add and propagate a new discovery/logging counter: rejected_existing_execution_ids.
• Merge previously processed execution IDs into persisted state after successful ingestion (instead of overwriting).

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File	Description
backend/app/scripts/ingestion/nersc_archive_ingestor.py	Skips known execution IDs earlier during discovery, logs new counter fields, and merges processed execution IDs when recording successful ingestions.
backend/tests/features/ingestion/test_nersc_archive_ingestor.py	Extends assertions for the new counter and adds coverage for early skipping of previously processed archive execution IDs.

[tomvothecoder]

Summary: Looks correct for an early-state dry run. Also cut down time from ~309s to ~223s (original run without early state check).

• SCAN_MODE="staging"
• Summary:
  • Completed successfully on 2026-07-09: exit_code=0, ~223s
  • Ran in mode=dry-run
  • Scanned 2837 cases / 4875 execution dirs total
  • 0 submission-qualified cases
  • Rejections (executions):
    • 3503 already processed
    • 1372 incomplete
  • 0 accepted, 0 deferred, 0 invalid
  • The dominant incomplete pattern was missing:
    • CaseStatus
    • GIT_DESCRIBE
    • README.case
    • e3sm_timing
    • env_build.xml
    • env_case.xml
  • Small exceptions existed in 9 incomplete runs:
    • 5 missed only README.case, env_build.xml, env_case.xml
    • 2 also missed env_run.xml
    • 2 missed the 5 metadata/status files without e3sm_timing
  • Note: the log’s final execution_dirs_scanned=1372 corresponds to the newly evaluated dirs in this pass; 4875 is the total execution count across all case summaries
• Full log: nersc.sh.staging.early-state-check.log

[tomvothecoder]

@TonyB9000 I used Codex to address #238, which it did relatively quickly. I performed a test run above and all seems well.

Next would be addressing #205 and #261.