legend-exp · ggmarshall · Jun 4, 2026 · Jul 8, 2026 · Jul 8, 2026 · Jul 8, 2026
diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml
@@ -76,6 +76,23 @@ jobs:
       - name: Run data production tests
         run: ./tests/runprod/run-all.sh
 
+  build-docs:
+    name: Build documentation
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v6
+        with:
+          fetch-depth: 0
+      - uses: actions/setup-python@v6
+        with:
+          python-version: "3.12"
+      - name: Install docs dependencies
+        run: |
+          python -m pip install --upgrade pip wheel setuptools
+          python -m pip install --upgrade .[docs]
+      - name: Build docs
+        run: make -C docs
+
   test-coverage:
     name: Calculate and upload test coverage
     runs-on: ubuntu-latest

diff --git a/AGENTS.md b/AGENTS.md
@@ -0,0 +1,97 @@
+# LEGEND Dataflow — Agent / Developer Reference
+
+`legend-dataflow` orchestrates the LEGEND L200 data processing pipeline — raw
+digitiser output to physics-ready events — with
+[Snakemake](https://snakemake.readthedocs.io/). Full docs:
+<https://legend-dataflow.readthedocs.io>. AI-assisted contributions must follow
+`AI_POLICY.md` (human review + disclosure).
+
+Layout: `workflow/Snakefile` is the main workflow (tcm → … → skm);
+`workflow/Snakefile-build-raw` covers DAQ → raw. Both include rule files from
+`workflow/rules/`. The Python package lives at `workflow/src/legenddataflow/`
+(note the non-standard `package-dir` in `pyproject.toml`): `methods/` is the
+library used by rules (file keys, patterns, pars catalogs, cal grouping);
+`scripts/` holds the executables rules invoke (`flow/` plumbing, `tier/`
+builders, `par/geds|spms/` calibration).
+
+## Split Across Two Packages
+
+Many console scripts invoked by rules (`build-tier-dsp`, `build-tier-hit`,
+`build-tier-pht`, …) and the `dataflow`/`dataprod` CLI live in the external
+`legend-dataflow-scripts` package (imported as `legenddataflowscripts`), pinned in
+`pyproject.toml`. This repo's own entry points are listed under `[project.scripts]`.
+If a command's implementation isn't here, check that package before concluding it's
+missing.
+
+## Processing Tiers
+
+`raw` (DAQ→LH5, blinding) → `tcm` (time coincidence map) → `dsp` (signal
+processing) → `hit` (energy calibration, PSD) → `evt` (event building,
+cross-talk) → `skm` (physics skim). Partition-level variants, averaged over
+calibration runs: `psp` (dsp), `pht` (hit), `ann`/`pan` (ANN cuts, coax only),
+`pet` (evt). Rule files are named after their tier. Per tier, calibration
+parameters are derived from `cal` runs, then applied to `phy` data.
+
+## Key Conventions
+
+- **File keys:** `{experiment}-{period}-{run}-{datatype}-{timestamp}`, e.g.
+  `l200-p03-r001-phy-20230401T000000Z`; parsed/generated by `methods/FileKey.py`.
+- **Targets:** `[all|valid|<runlist-key>]-{experiment}-{period}-{run}-{datatype}-{tier}.gen`;
+  `*` wildcards and `_`-separated multi-selectors work for most components.
+- **Patterns/paths:** all filename patterns live in `methods/patterns.py`, path
+  resolvers in `methods/paths.py`; new tiers need entries in both plus
+  `dataflow-config.yaml`.
+- **Parameter catalogs:** per-tier `validity.yaml` files map parameter sets to time
+  ranges; resolved by `ParsKeyResolve` and `ParsCatalog` in `methods/`.
+- **Two rule→script styles:** `script:` directives (receive the `snakemake` object)
+  vs `shell:` + `execenv_pyexe(config, "<console-script>")`, which runs inside the
+  configured execenv. Don't convert between them casually.
+- **Config/environment:** the `PRODENV` env var is required. In
+  `dataflow-config.yaml`, `$_` expands to the config file's directory and other
+  `$VARS` come from the environment; inputs live under `inputs/`, outputs under
+  `generated/`. `execenv:` defines named environments (bare/lngs/sator/nersc);
+  non-bare ones run in containers.
+
+## Commands
+
+```bash
+uv venv --python 3.12 && source .venv/bin/activate
+uv pip install -e ".[dev]"          # add ".[docs]" to build docs
+
+python -m pytest                    # unit tests
+./tests/runprod/run-all.sh          # integration tests; run from repo root,
+                                    # needs LEGEND_METADATA set
+pre-commit run --all-files          # lint/format (once: pre-commit install)
+make -C docs                        # Sphinx docs (regenerates apidoc; -W: warnings are errors)
+
+# run the workflow
+snakemake --profile workflow/profiles/default all-l200-p03-r001-phy-skm.gen
+```
+
+## Style
+
+- **ruff** (lint + format), **snakefmt** for Snakefiles (`channel_merge.smk`
+  excluded), **shellcheck**, **prettier** (YAML/MD/JSON/TOML) — all via pre-commit;
+  pre-commit.ci pushes autofixes to PRs.
+- `from __future__ import annotations` is required in all Python files.
+- **NumPy-style docstrings** (Sphinx Napoleon is configured NumPy-only; apidoc
+  publishes undocumented modules as empty stubs).
+- Versioning is **setuptools_scm** from git tags — never edit `_version.py`.
+- Commit messages loosely follow conventional-commit prefixes (`fix(evt): …`).
+
+## Testing
+
+- pytest promotes warnings to errors and uses strict markers/config;
+  `xfail_strict = true`.
+- Unit tests cover only `methods/` (file keys, pars catalogs, cal grouping) and
+  load the `tests/dummy_cycle/` fixture config at module import time.
+- `tests/runprod/*.sh` are DAG smoke tests using empty `touch`ed dummy files — by
+  design: real processing is ~1 TB per run, far beyond CI capacity. Never propose
+  processing real data in GitHub Actions; test pure logic with unit tests instead.
+
+## Extending the Pipeline
+
+Step-by-step guides for adding processors, calibration scripts, tiers, and
+execution environments are in `docs/source/developer.rst` (published in the
+Developer Guide on ReadTheDocs) — follow those rather than reverse-engineering
+existing rules.
diff --git a/CLAUDE.md b/CLAUDE.md
@@ -0,0 +1 @@
+AGENTS.md
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -0,0 +1,59 @@
+# Contributing
+
+Contributions are welcome via pull requests against `main`.
+
+## Development setup
+
+```bash
+git clone https://github.com/legend-exp/legend-dataflow.git
+cd legend-dataflow
+uv venv --python 3.12
+source .venv/bin/activate
+uv pip install -e ".[dev]"
+pre-commit install
+```
+
+## Code style
+
+Linting and formatting are enforced by [pre-commit](https://pre-commit.com/):
+ruff (lint + format) for Python, snakefmt for Snakemake files, shellcheck for
+shell scripts, and prettier for YAML/Markdown/JSON/TOML. pre-commit.ci runs on
+every pull request and pushes autofix commits, but running the hooks locally
+gives faster feedback:
+
+```bash
+pre-commit run --all-files
+```
+
+Docstrings use the NumPy style — the API reference is auto-generated from them
+with Sphinx, so undocumented code shows up as empty stubs in the published docs.
+
+Commit messages loosely follow conventional-commit prefixes, e.g.
+`fix(evt): ...`, `docs: ...`, `chore: ...`.
+
+## Tests
+
+```bash
+python -m pytest              # unit tests
+./tests/runprod/run-all.sh    # integration tests (from the repo root;
+                              # requires LEGEND_METADATA to be set)
+```
+
+The integration tests validate the Snakemake DAG wiring with dummy files; real
+data processing (~1 TB per run) is far beyond CI capacity, so new processing
+logic should be covered by unit tests where practical.
+
+## Documentation
+
+```bash
+uv pip install -e ".[docs]"
+make -C docs                  # builds HTML into docs/build (warnings are errors)
+```
+
+See the [Developer Guide](https://legend-dataflow.readthedocs.io) for
+step-by-step instructions on extending the pipeline (new processors,
+calibration scripts, tiers, and execution environments).
+
+## AI-assisted contributions
+
+Permitted with human review and disclosure — see [AI_POLICY.md](AI_POLICY.md).
diff --git a/README.md b/README.md
@@ -95,6 +95,9 @@ Full documentation is available at the project's ReadTheDocs page:
 - **Developer Guide** – how to extend and contribute to the pipeline
 - **API Reference** – Python API documentation
 
+See [CONTRIBUTING.md](CONTRIBUTING.md) for development setup and contribution
+guidelines.
+
 ## Related projects
 
 - [legend-pydataobj](https://legend-pydataobj.readthedocs.io) – LEGEND data objects (LH5 format)

diff --git a/dataflow-config.yaml b/dataflow-config.yaml
@@ -1,25 +1,43 @@
+# Site configuration for the LEGEND dataflow. Key settings are documented in
+# docs/source/user_manual.rst (Configuration section).
+
+# Tag or branch of legend-metadata to check out automatically;
+# comment out to manage the metadata checkout manually.
 legend_metadata_version: v1.4.1
+# If false, abort when calibration parameter generation fails;
+# if true, continue with default or overridden parameters.
 allow_none_par: false
+# Generate pygama FileDB databases after a successful production run.
 build_file_dbs: true
+# Scan log files for errors/warnings after each run.
 check_log_files: true
+# Parallel processing within a single Snakemake job, its mode, and the
+# process cap.
 multiprocess: false
-mutliprocess_mode: max_usage
+multiprocess_mode: max_usage
 max_processes: 1
+# Input tier for pht building: dsp or psp.
 pht_intier: psp
 
+# $_ expands to the directory containing this file; other $VARS
+# (e.g. $PRODENV) are taken from the environment.
 paths:
+  # Staging area for incoming DAQ files (Snakefile-build-raw).
   sandbox_path: $_/sandbox
+  # Blinded raw output (physics data with signal region removed).
   tier_raw_blind: $_/generated/tier/raw-blind
 
   workflow: $_/workflow
 
+  # Inputs: metadata, processing configs, overrides, hardware maps.
   metadata: $_/inputs
   config: $_/inputs/dataprod/config
   par_overwrite: $_/inputs/dataprod/overrides
   chan_map: $_/inputs/hardware/configuration
   detector_status: $_/inputs/datasets
   detector_db: $_/inputs/hardware/detectors
 
+  # Data tier outputs.
   tier: $_/generated/tier
   tier_daq: $_/generated/tier/daq
   tier_raw: $_/generated/tier/raw
@@ -34,6 +52,7 @@ paths:
   tier_pet: $_/generated/tier/pet
   tier_skm: $_/generated/tier/skm
 
+  # Calibration parameter outputs (validity catalogs + par files).
   par: $_/generated/par
   par_raw: $_/generated/par/raw
   par_tcm: $_/generated/par/tcm
@@ -44,18 +63,23 @@ paths:
   par_pht: $_/generated/par/pht
   par_pet: $_/generated/par/pet
 
+  # Monitoring plots and logs.
   plt: $_/generated/plt
   log: $_/generated/log
 
+  # Scratch space, cleaned up on workflow success.
   tmp_plt: $_/generated/tmp/plt
   tmp_log: $_/generated/tmp/log
   tmp_benchmark: $_/generated/tmp/benchmark
   tmp_filelists: $_/generated/tmp/filelists
   tmp_par: $_/generated/tmp/par
 
+  # Python source checkout and venv created by `dataflow install`.
   src: $_/software/python/src
   install: $_/.snakemake/legend-dataflow/venv
 
+# LH5 table name inside output files, per tier
+# ({ch} = channel rawid, {grp} = detector group).
 table_format:
   raw: ch{ch:07d}/raw
   dsp: ch{ch:07d}/dsp
@@ -67,6 +91,9 @@ table_format:
   skm: "{grp}/skm"
   tcm: hardware_tcm_1
 
+# Named execution environments, selected with `--config system=<name>`
+# (profiles set this). Non-bare environments wrap commands in a container
+# via `cmd` + `arg`; `env` sets variables for every job.
 execenv:
   bare:
     env:

diff --git a/docs/source/pipeline.rst b/docs/source/pipeline.rst
@@ -76,12 +76,13 @@ Converts raw DAQ files to LH5 format:
 
 - **ORCA format** (``build_raw_orca``) – decodes data from ORCA.
 - **FCIO format** (``build_raw_fcio``) – decodes data from FLASHCAM
-used for some special data taking.
+  used for some special data taking.
 - **Blinding** (``build_raw_blind``) – removes events with any hit within 25keV of Qbb, it
-uses the daqenergy estimator to do this. The blinding is checked in each calibration using
-``rules/blinding_check.smk`` to verify the parameters are still valid. If any channels fail the
-processing will stop until a new calibration is put in. The calibrations are all stored in
-``legend-dataflow-overrides`` under ``raw``.
+  uses the daqenergy estimator to do this. The blinding is checked in each calibration using
+  ``rules/blinding_check.smk`` to verify the parameters are still valid. If any channels fail the
+  processing will stop until a new calibration is put in. The calibrations are all stored in
+  ``legend-dataflow-overrides`` under ``raw``.
+
 The helper rules in ``rules/blinding_calibration.smk`` can be used to derive all the
 daqenergy calibration coefficients.
 

diff --git a/docs/source/user_manual.rst b/docs/source/user_manual.rst
@@ -80,15 +80,21 @@ Key settings
 +-------------------------------+--------------------------------------------------------------+
 | ``multiprocess``              | Enable parallel processing within a single Snakemake job.    |
 +-------------------------------+--------------------------------------------------------------+
-| ``pht_intier  ``              | Which tier to use as input for pht can be ``dsp`` or ``psp`` |
+| ``multiprocess_mode``         | Strategy used to share work when ``multiprocess`` is on      |
+|                               | (default ``max_usage``).                                     |
++-------------------------------+--------------------------------------------------------------+
+| ``max_processes``             | Maximum number of processes when multiprocessing.            |
++-------------------------------+--------------------------------------------------------------+
+| ``pht_intier``                | Which tier to use as input for pht can be ``dsp`` or ``psp`` |
 +-------------------------------+--------------------------------------------------------------+
 
 Paths
 -----
 
-All path values support the ``$_`` placeholder, which is substituted with the value of
-the ``$PRODENV`` environment variable at runtime. This allows a single config file to
-be used across different machines by setting ``PRODENV`` appropriately.
+All path values support the ``$_`` placeholder, which is substituted with the directory
+containing the config file (the production cycle root). Other ``$VAR`` placeholders are
+taken from the environment, e.g. ``$PRODENV``. This allows a single config file to be
+used across different machines.
 
 The key path categories are:
 
@@ -182,7 +188,7 @@ The target format is:
 where:
 
 - ``all`` / ``valid`` – process all data, or only data selected for analysis,
-any keyword in ``runlists.yaml`` in ``legend-datasets`` is a possible option.
+  any keyword in ``runlists.yaml`` in ``legend-datasets`` is a possible option.
 - ``experiment`` – experiment name (e.g. ``l200``)
 - ``period`` – data-taking period (e.g. ``p03``)
 - ``run`` – run number (e.g. ``r001``)
@@ -268,6 +274,6 @@ Two files in the ``detector_status`` directory control which data are included:
 
 - ``ignored_daq_cycles.yaml`` – lists DAQ cycles to exclude from processing entirely
 - ``run_override.yaml`` – allows overriding the validity window for specific runs,
-e.g. to apply a previous valid calibration to a subsequent run
+  e.g. to apply a previous valid calibration to a subsequent run
 
 These files are part of the ``legend-datasets`` repository.
diff --git a/pyproject.toml b/pyproject.toml
@@ -51,7 +51,7 @@ dynamic = ["version"]
 
 dependencies = [
     "colorlog",
-    "dbetto==1.3.7",
+    "dbetto>=1.4",
     "pygama==2.4.1",
     "dspeed==2.1.4",
     "pylegendmeta==1.4.4",
@@ -95,7 +95,6 @@ docs = [
 ]
 
 [project.scripts]
-create-chankeylist      = "legenddataflow.scripts.create_chankeylist:create_chankeylist"
 merge-channels          = "legenddataflow.scripts.flow.merge_channels:merge_channels"
 merge-in-channels       = "legenddataflow.scripts.flow.merge_in_channel:merge_in_channel"
 build-tier-evt          = "legenddataflow.scripts.tier.evt:build_tier_evt"
@@ -107,6 +106,7 @@ build-tier-tcm          = "legenddataflow.scripts.tier.tcm:build_tier_tcm"
 par-geds-pht-aoe        = "legenddataflow.scripts.par.geds.pht.aoe:par_geds_pht_aoe"
 par-geds-pht-ecal-part  = "legenddataflow.scripts.par.geds.pht.ecal_part:par_geds_pht_ecal_part"
 par-geds-pht-fast       = "legenddataflow.scripts.par.geds.pht.fast:par_geds_pht_fast"
+par-geds-pht-lq         = "legenddataflow.scripts.par.geds.pht.lq:par_geds_pht_lq"
 par-geds-pht-qc-phy     = "legenddataflow.scripts.par.geds.pht.qc_phy:par_geds_pht_qc_phy"
 par-geds-pht-qc         = "legenddataflow.scripts.par.geds.pht.qc:par_geds_pht_qc"
 par-geds-psp-average    = "legenddataflow.scripts.par.geds.psp.average:par_geds_psp_average"