Testing and documentation improvements: unit tests, docstrings, CI docs gate, entry-point fixes

[ggmarshall]

Summary

Documentation, testing, and bug-fix pass over the workflow-plumbing layer. Unit suite grows from 13 to 62 tests, and writing them surfaced six latent bugs, all fixed here.

Bugs fixed

• par-geds-pht-lq console script was never registered in [project.scripts], although pht_pars_geds.smk invokes it — the rule failed with command not found.
• create-chankeylist entry point pointed at a deleted module, producing a broken console script on install; removed (nothing references it).
• mutliprocess_mode config key was misspelled — common.smk reads the correct spelling, so the setting was silently ignored. Renamed in the config (value equals the default, so behavior is unchanged).
• get_analysis_runs checked Path(ignore_keys_file).suffix where it meant analysis_runs_file, crashing on Path(None) whenever a runlist was passed without an ignore file.
• build_filelist assumed dict-shaped ignore_keys and crashed on the flat lists produced from .keylist files.
• find_gen_runs split the full path on - for concat tiers, mis-parsing run specs whenever the production path contains a hyphen (e.g. prod-blind); it now splits the filename only.

Tests (13 → 62)

• methods/paths.py and patterns.py: resolvers, error cases, link_external_paths symlinking, out-of-cycle /tmp fallback
• scripts/flow/: build_filelist (run expansion, runlist/keylist selection, multi-selectors, concat handling, end-to-end on touched files), merge-channels and merge-in-channels (yaml/pickle/shelve/lh5 with real lgdo objects, db path rewriting), build_chanlist, write_filelist, complete_run
• complete_run.py / write_filelist.py were restructured so the Snakemake-injected object import is guarded and the logic is importable; the glue itself is tested by injecting a fake object into snakemake.script and executing via runpy

Documentation

• AGENTS.md condensed (268 → 97 lines) to non-discoverable content: two-package split with legend-dataflow-scripts, rule→script invocation styles, PRODENV/$_ expansion, commands, CI data-scale constraints
• New CONTRIBUTING.md, linked from README
• Inline comments in dataflow-config.yaml (values unchanged)
• Module docstrings for all 23 undocumented scripts/ modules and method docstrings for FileKey/CalGrouping, so the apidoc-generated API reference is no longer empty stubs
• User manual: document multiprocess_mode/max_processes, correct the $_ placeholder description (expands to the config directory, not $PRODENV)

CI

• New build-docs job gating the strict sphinx-build -W build (previously RTD-only); fixed the five pre-existing RST warnings that were already failing it

Test plan

• python -m pytest — 62 tests pass
• pre-commit run --all-files clean on all touched files
• make -C docs builds with warnings-as-errors
• Config edits verified comment-only by comparing parsed YAML before/after

🤖 Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 80.08%. Comparing base (845a095) to head (af9378c).
⚠️ Report is 6 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff             @@
##             main     #161       +/-   ##
===========================================
+ Coverage   62.23%   80.08%   +17.84%     
===========================================
  Files           7       12        +5     
  Lines         662      974      +312     
===========================================
+ Hits          412      780      +368     
+ Misses        250      194       -56     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[ggmarshall]

Systematic review of workflow/src/legenddataflow/ (scripts + methods): crash-bug fixes, error-message improvements, and an error-handling sweep so foreseeable failures raise named errors instead of cryptic tracebacks. Companion to legend-exp/legend-dataflow-scripts#39.

Important

Depends on legend-exp/legend-dataflow-scripts#39. The scripts now import shared helpers (expand_filelist, check_pulser_mask, require_config_keys, get_rule_config, get_channel_config) from legenddataflowscripts.utils, replacing the short-lived local duplicates. CI will fail until that PR is merged, released, and the legend-dataflow-scripts==0.3.3 pin here is bumped — hence draft.

Crash bugs

• par-geds-pht-{aoe,lq,ecal-part} were unrunnable: they read args.files while the option is --input-files; aoe additionally called run_aoe_calibration with a missing plot_dicts positional and an unexpected log= kwarg.
• build-tier-raw-blind was broken three ways: stale from legendmeta import TextDB import, an AttributeError in the blind-curve channel lookup, and the nonexistent "blsns" system name (real channel maps use bsln). It also now refuses to write an unblinded output if no blindable channel exists.
• Merge scripts: NameError for --out-db without --in-db, unknown output extensions silently wrote nothing, and the lh5 merge truncated multi-underscore identifiers (svm_train → svm) — now derived via fkey.identifier with named errors for missing identifiers/fields and mixed-channel inputs.
• Branch-scope crashes: plot_dicts undefined when run_lq: false, popped common plot dict reused after pop, unconditional plot save in fast.py, psp/average input-matching NameErrors, evt.py len(None) on omitted --xtc-file/--ann-file.

Wrong results

• build_filelist.expand_runs skipped entries when a run list contained more than one rNNN..rNNN range (pop-while-iterating).
• skm.py mutated keep_fields while iterating, silently dropping the field after any expanded sub-table.
• tcm.py per-fcid configs were keyed by string but looked up with an int fcid, silently splatting the whole mapping into build_tcm.
• qc_phy.py re-read all previously accumulated files per filelist, misaligning the baseline mask against the data.

Crashed jobs are now visible in run summaries

check_log_files only grepped for the literal ERROR token, but uncaught tracebacks written by build_log's excepthook carry no such token — a crashed job produced a summary saying "no errors". The scan now detects traceback blocks and reports each exception line (e.g. crashed.log : KeyError: 'threshold'). Together with the build_log fix in the companion PR, crashes are caught from both sides.

methods/ error handling

• FileKey.get_filekey_from_pattern raises a ValueError naming the filename and pattern instead of returning None (every caller dereferenced the result); parse_keypart likewise. No caller anywhere relied on the None return.
• Fixed the latent StopIteration / dict-mutation crash in get_path_from_filekey's dict-kwarg resolution (both copies now share _resolve_dict_kwargs).
• paths.py accessors raise dataflow-config paths.<key> is not set; patterns.py raises ValueError naming the tier; pars_loading checks validity-file existence with tier context; create_pars_keylist validates catalogs have an all entry; cal_grouping raises named errors for unknown partitions/missing default, validates rNNN..rNNN ranges at load, and warns when a partition resolves to zero par files (fallbacks kept — raising would abort DAG construction for legitimately empty partitions).

Tests (62 → 100)

New console-script smoke test (imports all 21 [project.scripts] entry points and runs --help — four of the crash bugs above would have been caught by it) plus error-path tests across the methods and flow-script test modules. Helper unit tests moved to legend-dataflow-scripts with the helpers.

Verification

• pytest: 100 passed (with the scripts clone editable-installed)
• ruff check / ruff format: clean

---

Per AI_POLICY.md: this contribution was drafted with AI assistance (Claude Code) and reviewed by the submitter.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR strengthens the workflow “plumbing” layer by adding broad unit-test coverage, improving documentation/docstrings, fixing multiple latent runtime bugs in scripts and helpers, and adding a CI documentation build gate to keep Sphinx warnings from regressing.

Changes:

• Fixes and hardens tier/flow/par scripts (config handling, filelist expansion, error messages/validation, safer Snakemake “glue” imports).
• Expands unit tests substantially across methods/ and scripts/flow/, plus smoke tests for console-script entry points.
• Adds/updates contributor and user documentation, plus a new CI job to build docs with warnings treated as errors.

Reviewed changes

Copilot reviewed 55 out of 55 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
workflow/src/legenddataflow/scripts/tier/tcm.py	Support per-FCID config mappings and improve error reporting.
workflow/src/legenddataflow/scripts/tier/skm.py	Make keep_fields expansion more robust with clearer errors.
workflow/src/legenddataflow/scripts/tier/raw_orca.py	Switch to shared logging/config helpers and simplify logging setup.
workflow/src/legenddataflow/scripts/tier/raw_fcio.py	Add module docstring.
workflow/src/legenddataflow/scripts/tier/raw_blind.py	Fix imports and add validation/guardrails for channel/blinding inputs.
workflow/src/legenddataflow/scripts/tier/evt.py	Refactor channel resolution, improve argument defaults, validate TCM input, adjust jitter delay scan.
workflow/src/legenddataflow/scripts/par/spms/dsp/trigger_threshold.py	Improve warning/error messages for missing/insufficient waveform data.
workflow/src/legenddataflow/scripts/par/geds/tcm/pulser.py	Use shared filelist expansion and tighten required CLI args.
workflow/src/legenddataflow/scripts/par/geds/raw/blindcheck.py	Use shared filelist expansion and improve failure message detail.
workflow/src/legenddataflow/scripts/par/geds/raw/blindcal.py	Use shared filelist expansion.
workflow/src/legenddataflow/scripts/par/geds/psp/dplms.py	Add explicit error when no FFT inputs are found.
workflow/src/legenddataflow/scripts/par/geds/psp/average.py	Centralize config lookup, harden parsing/averaging, and improve plot/object merge logic.
workflow/src/legenddataflow/scripts/par/geds/pht/util.py	Add module docstring.
workflow/src/legenddataflow/scripts/par/geds/pht/qc.py	Centralize config lookup and channel config selection; add module docstring.
workflow/src/legenddataflow/scripts/par/geds/pht/qc_phy.py	Centralize config lookup, fix file reading scope, add input-empty guard, and correct discharge timestamp handling.
workflow/src/legenddataflow/scripts/par/geds/pht/lq.py	Refactor to shared config helpers, validate required keys, and harden pulser-mask handling; add module docstring.
workflow/src/legenddataflow/scripts/par/geds/pht/fast.py	Refactor config loading/validation, harden pulser-mask handling, and guard optional outputs; add module docstring.
workflow/src/legenddataflow/scripts/par/geds/pht/ecal_part.py	Refactor config loading/validation, improve error message clarity, and fix “common” plot merge behavior.
workflow/src/legenddataflow/scripts/par/geds/pht/aoe.py	Refactor config loading/validation and improve pulser-mask handling; add module docstring.
workflow/src/legenddataflow/scripts/flow/write_filelist.py	Make Snakemake glue import-safe for unit tests and expose a testable helper function.
workflow/src/legenddataflow/scripts/flow/utils.py	Add module docstring for shared flow helpers.
workflow/src/legenddataflow/scripts/flow/merge_in_channel.py	Add validation, improve LH5 merge semantics, and provide clearer errors.
workflow/src/legenddataflow/scripts/flow/merge_channels.py	Add validation and clearer errors; harden DB rewrite behavior.
workflow/src/legenddataflow/scripts/flow/complete_run.py	Make Snakemake glue import-safe for unit tests; refactor log scanning and FileDB config/build steps.
workflow/src/legenddataflow/scripts/flow/build_filelist.py	Fix ignore/runlist parsing bugs, improve run expansion, and harden ignore-keys handling.
workflow/src/legenddataflow/scripts/flow/build_chanlist.py	Add module docstring.
workflow/src/legenddataflow/methods/patterns.py	Raise ValueError with clearer messages for invalid tier inputs.
workflow/src/legenddataflow/methods/paths.py	Centralize path lookup with clearer paths.<key> error reporting.
workflow/src/legenddataflow/methods/pars_loading.py	Add explicit validity-file existence checks with actionable errors.
workflow/src/legenddataflow/methods/FileKey.py	Add docstrings and improve error handling for key parsing/matching; refactor dict-kwarg resolution.
workflow/src/legenddataflow/methods/create_pars_keylist.py	Add validation for required all entries in override catalogs.
workflow/src/legenddataflow/methods/cal_grouping.py	Add logging, validation, clearer errors, and docstrings for grouping/expansion logic.
tests/test_write_filelist.py	Add unit tests for helper and Snakemake glue execution.
tests/test_patterns.py	Add unit tests for tier/pars patterns and out-of-cycle behavior.
tests/test_paths.py	Add unit tests for path resolvers and symlink behavior.
tests/test_pars_loading.py	Expand unit coverage and add missing-validity error coverage.
tests/test_merge_in_channel.py	Add tests for YAML/pickle/LH5 merging and failure modes.
tests/test_merge_channels.py	Add tests for YAML/pickle/shelve/LH5 merging and DB path rewriting.
tests/test_flow_utils.py	Add unit tests for replace_path behavior.
tests/test_filekey.py	Add unit tests for parse/match errors and dict-kwarg expansion behavior.
tests/test_create_pars.py	Add coverage for missing all in run-override validity handling.
tests/test_console_scripts.py	Add smoke test to import and --help all configured console scripts.
tests/test_complete_run.py	Add tests for log scanning (including tracebacks), concat run parsing, and Snakemake glue.
tests/test_cal_grouping.py	Add tests for new cal-grouping validation and warning behavior.
tests/test_build_filelist.py	Add tests covering runlists/keylists, ignore keys, concat behavior, and wildcard selection.
tests/test_build_chanlist.py	Add tests for channel selection and output filename generation.
README.md	Link to new contributing guide.
pyproject.toml	Fix console script registration (remove broken entry point, add missing one).
docs/source/user_manual.rst	Document multiprocessing settings and correct $_ placeholder semantics.
docs/source/pipeline.rst	Fix formatting/indentation issues and improve readability.
dataflow-config.yaml	Fix config key typo and add clarifying inline comments (values unchanged).
CONTRIBUTING.md	Add contributor setup/testing/docs guidance.
AGENTS.md	Add/refresh developer/agent reference for repo structure and conventions.
.github/workflows/main.yml	Add docs build job to gate Sphinx warnings-as-errors in CI.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[gipert]

this should be imported from pylegendmeta

[gipert]

this looks good but shouldn't we also prepare the full integration tests we discussed (test production cycle with minimal L200 data to test all features) before deploying all this?