Commit ccb13a7
feat(custom): orfnormalise + orfmerge modules + orftable_fasta_gtf_buildorfcatalogue subworkflow (#11740)
* feat(custom/orfnormalise,orfmerge,orftable_fasta_gtf_buildorfcatalogue): scaffold ORF catalogue chain
Add a three-component upstream chain for building cross-caller Ribo-seq
ORF catalogues:
- custom/orfnormalise: dispatches on meta.caller in {ribocode, ribotish,
ribotricer, rpbp, price}, emits unified BED12 + sidecar TSV with the
harmonised orf_class vocabulary (canonical_cds / uORF / dORF / novel_u
/ smORF / other).
- custom/orfmerge: class-aware clustering across callers and samples,
records called_by_<caller> / score_<caller> provenance with
direction-aware best-score aggregation.
- orftable_fasta_gtf_buildorfcatalogue: composes normaliser x N callers
+ merger + bedtools/getfasta + seqkit/translate, emits BED12,
catalogue TSV, orf_to_gene TSV, AA FASTA and an MQC sidecar.
Templates use the same python/pandas/pyyaml Wave container as
custom/bed12codonpositions. Stub tests scaffolded; real-fixture tests
follow once the paired test-datasets PR lands.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(orf-catalogue chain): add real-fixture tests, fix ribotricer ORF_ID fallback
- orfnormalise: 5 per-caller tests + 1 stub (all green). Fall back to
parsing the ribotricer ORF_ID when the optional 'coordinate' column is
absent (detect-orfs output does not carry it). Tolerate empty GTF
inputs by requiring a real file before trying to read.
- orfmerge: chain via setup{} from orfnormalise outputs (ribotish +
ribocode on chr20); disambiguate per-caller prefix via a
bundled setup_prefix.config so the merger doesn't see name collisions.
- subworkflow: bundle nextflow.config that scopes the same caller-aware
prefix to ORFTABLE_FASTA_GTF_BUILDORFCATALOGUE:CUSTOM_ORFNORMALISE.
Test sets up GUNZIP for the chr20 FASTA since bedtools/getfasta needs
uncompressed input.
Fixtures live in nf-core/test-datasets#2070; will switch the test URLs
from the pinin4fjords fork back to params.modules_testdata_base_path
once that PR merges.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(custom/orfnormalise): correct PRICE Location regex + drop empty-md5 stubs
PRICE encodes strand as a single char appended to the chrom rather than
a separate colon-bracketed field (`19+:start-end`), so the previous
regex never matched and the parser returned an empty BED12 on real
data. Switch to a non-greedy `chrom` capture so the trailing strand
char is correctly extracted.
Drop the per-module stub tests: real-fixture tests already cover every
caller + the merge + the full subworkflow chain, and the stub snapshots
were tripping the test_snap_md5sum lint rule (empty-file md5).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(custom/orfnormalise + subworkflow): self-review pass on inherited parsers + empty-cohort handling
Inherited parser bugs (carried over from the local riboseq port):
- RiboCode: aa_length always 0 because the parser looked up an
AA_length column that does not exist in predicted_orfs.txt. Derive
from ORF_length (nt) instead.
- Rp-Bp: bf_mean read from column 15 (p_translated_var) instead of
column 18 (bayes_factor_mean), and orf_type read from column 14
(a metric, not a category). RPBP's predicted-orfs BED has no orf_type
column; default to canonical_cds for the post-selectfinalpredictionset
curated set.
- Ribo-TISH: combined p-value never read because the parser looked for
Pvalcombined/Pvalue/Pvalcom; actual columns are FisherPvalue /
RiboPvalue / TISPvalue, with "None" string sentinels. Walk the
preference list and skip None strings.
Subworkflow gaps:
- ch_orf_tables empty case crashed CUSTOM_ORFMERGE (arity '1..*'
violated by .collect() emitting an empty list). Filter out the empty
case so output channels are simply empty.
- BEDTOOLS_GETFASTA and SEQKIT_TRANSLATE got generic
${meta.id}.{fa,fasta} filenames; pin to .catalogue.nt.fa and
.catalogue.aa.fasta via bundled ext.prefix. Add ext.args = '-split -s
-nameOnly' to getfasta (splice-aware extraction of BED12 blocks) and
'--trim' to seqkit translate (drop trailing stop codons).
- meta.yml: cite all five caller tools with verified DOIs (RiboCode
Xiao 2018, Ribo-TISH Zhang 2017, Ribotricer Choudhary 2020, Rp-Bp
Malone 2017, PRICE Erhard 2018); document that ch_orf_tables empty
short-circuits; clarify GTF is required for ribocode/ribotish and
optional for the rest; note that cohort-level output meta is
hardcoded to [id:'cohort'].
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(orf-catalogue): second-review pass on parsers, clustering, fixtures, tests
Surfaced + addressed via independent third-party assessment of branch
6597190 (parsers vs. canonical upstream output formats; merger
clustering logic; test depth).
Parsers (orfnormalise):
- Ribotricer: detect-orfs output does NOT have the `coordinate` column
(that lives in the prepare-orfs index file). The previous fallback
treated the ORF_ID span as a single block, which for any multi-exon
host transcript emits a BED12 record that spans introns - biologically
nonsense and contaminates the catalogue AA FASTA produced via
bedtools/getfasta + seqkit/translate. New _ribotricer_blocks_from_id
takes the transcript map and intersects the ORF span with the host
transcript's exon structure (same pattern ribotish's fallback already
uses), recovering proper multi-exon blocks.
- PRICE: orf_id_raw.split("__", 1)[0] used double-underscore; actual ID
format is `<tid>_<type>_<index>` (single underscores). Was dead code
for fixtures with Gene populated, but still wrong; switch to single
underscore so the transcript_id lookup against the GTF works.
Merger (orfmerge):
- cluster_by_transcript collapsed every uORF / dORF / other ORF sharing
a transcript_id into one cluster. A transcript can host multiple
distinct uORFs (biologically common) so the catalogue under-reported
them. Split into cluster_by_transcript (canonical_cds only) +
cluster_by_transcript_position (uORF/dORF/other; keyed on outer span
as well so distinct positional ORFs stay separate).
- Document order-dependence of cluster_by_reciprocal_overlap at the
threshold boundary.
- Drop dead `_bed_key` helper.
Tests:
- Snapshot-only assertions caught changes but not correctness. Add
content-level checks per caller: aa_length > 0 for all rows, score
column populated, orf_class distribution not collapsed to a single
bucket, multi-block BED12 records for ribotricer and price. These
would have caught every parser bug in the assessment.
- Merger test asserts called_by_*/score_* columns are coherent
(catches the case where a parser bug populates the indicator but
the score column is empty).
Documentation (meta.yml):
- Cite all five caller tools with verified DOIs (already in previous
commit) - add note that Rp-Bp's predicted-orfs BED has no orf_type
column so this module defaults Rp-Bp calls to canonical_cds.
- Note PRICE iORF/intronic/orphan -> 'other' collapse (lossy).
- Note ribotricer multi-exon block recovery via GTF + that GTF is
strongly recommended for ribotricer (not just ribocode/ribotish).
- Subworkflow: AA fasta now lands as `${cohort}.catalogue.aa.fasta`
(from the previous commit).
Paired test-datasets PR nf-core/test-datasets#2070 also gets a small
fixture update: the ribotish/predict fixture's source run lacked a TIS
BAM so every p-value column came out as the literal string "None",
which silently masked the broken pvalue parser. Substitute the real
RiboPvalue value into the FisherPvalue column via awk so the parser
path is exercised; documented in the fixture README.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(custom/orfnormalise): per-caller field overrides via ext.args + provenance header
The source columns the parser reads for each derived TSV field
(`score`, `orf_type`, `aa_length`, length-for-aa-derivation) were
previously hardcoded in each parser. Some callers expose multiple
meaningful choices - Ribo-TISH has TISPvalue / RiboPvalue / FisherPvalue
(plus Q-values), Rp-Bp has bayes_factor_mean / chi_square_p /
p_translated_mean - and there was no way to override the default chain.
This change:
- Plumbs `task.ext.args` through the module to the template.
- Adds `--score-field`, `--orf-type-field`, `--length-field`,
`--aa-length-field` CLI options. When set, an override replaces the
per-caller default chain (so the user gets exactly what they asked
for).
- Refactors the rpbp parser from positional column indices to
csv.DictReader + named RPBP_COLUMNS so users can address columns by
name (e.g. `--score-field bayes_factor_var`).
- Centralises field-name preference chains into a DEFAULT_FIELDS
table next to the parsers.
- Writes a `# parser_columns: caller=<X> score=<col> orf_type=<col>
...` provenance header at the top of the normalised TSV, so
consumers (and reviewers) can verify which source column was
actually read for each derived field. Standard csv.DictReader and
pandas `comment='#'` skip the line automatically.
Also updates custom/orfmerge's load_normalised to skip `#`-prefixed
comment lines so the new provenance header doesn't leak into the
cohort catalogue.
Test changes:
- Existing per-caller tests gain a provenance-line assertion (verifies
`# parser_columns:` is present and reports the expected default
column for at least the ribocode case).
- New `homo_sapiens [chr20] - ribotish - score-field override` test
runs with `ext.args = '--score-field RiboPvalue'` and asserts the
provenance line reports `score=RiboPvalue` rather than the default
`FisherPvalue`.
- Tests now filter `#`-prefixed lines before parsing the TSV header.
Verified end-to-end on chr19/chr20 fixtures: default + override paths
emit the right provenance line, and all parser-specific assertions
(non-zero aa_length, populated score column, non-collapsed class
distribution, multi-exon BED12 blocks) continue to pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(custom/orfnormalise): caller as val input (not meta.caller); subworkflow channel now carries it as val too
nf-core/modules has a hardcoded `permitted_meta_keys = {"id",
"single_end"}` allow-list - any other meta.<key> reference in main.nf
fails the `main_nf_meta_key` lint check. The previous design embedded
the per-record caller id in meta.caller, which violated that policy
and tripped CI lint.
Lift the caller out of meta and into a proper val input - the pattern
used by raxmlng (val(model)), last/mafconvert (val(format)),
clair3 (val(platform)), and similar nf-core modules:
- Module input: `tuple val(meta), path(orfs_table), val(caller)` plus
the unchanged `tuple val(meta2), path(gtf)`. The `caller` value goes
into a per-caller enum in meta.yml (one of ribocode / ribotish /
ribotricer / rpbp / price).
- Module main.nf no longer references meta.caller; the script binds
`caller` directly from the input tuple.
- Subworkflow `ch_orf_tables` API also moves caller out of meta:
`[ val(meta), path(orf_table), val(caller) ]`. Subworkflow appends
caller to meta.id locally for the normalise call so the merger's
`beds/*` staging gets unique per-caller filenames; the merger then
rebuilds meta as `[id: 'cohort']` so no caller leakage downstream.
- Subworkflow's bundled `nextflow.config` drops the
`withName: CUSTOM_ORFNORMALISE` prefix override (the meta.id append
in main.nf does the work).
Pre-commit fixes:
- ruff E741: rename `l` to `line` in write_outputs (loop variable).
- ruff UP015: drop the redundant `"r"` mode arg from open() in open_text.
- ruff format: minor reformatting of long literal lists.
Tests:
- All 6 orfnormalise tests + 1 merger test + 1 subworkflow test pass
module input as the new 3-tuple (meta carries only `id`; caller is
the trailing val). Merger setup config still constructs
`meta + [caller: ...]` for prefix disambiguation - that meta key is
consumed by the *test* setup_prefix.config (not by the module),
which is fine since lint only scans modules' main.nf.
Local `nf-core modules lint custom/orfnormalise` now passes cleanly
(only the cosmetic Wave container-version warnings remain, shared
with the reference custom/bed12codonpositions module).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(custom/orfnormalise+orfmerge): prefix convention, args, ribotricer score [skip ci]
- orfnormalise prefix: follow #11955 convention — descriptor into default
prefix (`${meta.id}.normalised`) and drop it from output glob (`${prefix}.bed12`,
`${prefix}.tsv`). Default output filenames are unchanged; users who override
ext.prefix still get a correctly-named file.
- Both templates: replace `"${args}".split() if...` with `shlex.split("${args}")`
(consistent with rpbp chain modules; handles quoted arg values; empty string
returns [] without special-casing).
- orfnormalise/parse_ribotricer: fix bug where phase_score was read for the BED
score column but `raw_score` was always stored as "". All other parsers correctly
store the native score; ribotricer rows will now populate `score_ribotricer` in
the merged catalogue. Ribotricer test snapshot dropped for regeneration; score
assertion added to the ribotricer test to match the other caller tests.
- Fix meta.caller -> caller val wording in orfnormalise meta.yml tool description,
orfnormalise.py error message, and subworkflow meta.yml + main.nf comment.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* simplify Python: replace x==x NaN idiom and csv.DictReader comment-skip [skip ci]
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* refactor(custom/orf): drop pandas, simplify orfnormalise/orfmerge templates
Strip pandas from both bundled scripts: it was used only as a typed
list-of-dicts container (build DataFrame then iterrows / read_csv then
to_dict). Parsers now return plain lists of dicts and orfmerge reads
TSVs with csv.DictReader. Container drops to python + pyyaml.
orfnormalise: replace five near-identical classifier functions with a
keyword-rule table, factor shared aa-length and column-resolution
helpers, unify ribotish/ribotricer interval parsing, add a clamp helper,
and remove the unused SCORE_DIRECTIONS map.
orfmerge: collapse the two transcript-grouping functions into one
group_by helper.
Stubs write versions.yml via a one-line python -c using pyyaml.
Module output is byte-identical: all bed12/tsv/catalogue/fasta snapshot
md5s are unchanged; only versions.yml md5s differ (pandas removed). Adds
a previously-missing ribotricer snapshot entry.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(custom/orfmerge): record cross-sample recurrence in the catalogue
The normalised TSV's sample_id column was previously written by
orfnormalise but never consumed. orfmerge now aggregates it per cluster
into two catalogue columns, mirroring the existing per-caller consensus:
- n_samples: number of distinct samples contributing to the cluster.
- samples: sorted, comma-separated list of those sample ids.
This gives a cohort recurrence metric for filtering the catalogue. The
orfmerge test now merges a second sample (reusing the ribotish fixture)
so recurrence (n_samples == 2) is exercised and asserted.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>1 parent 457a7a8 commit ccb13a7
19 files changed
Lines changed: 2692 additions & 0 deletions
File tree
- modules/nf-core/custom
- orfmerge
- templates
- tests
- orfnormalise
- templates
- tests
- subworkflows/nf-core/orftable_fasta_gtf_buildorfcatalogue
- tests
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
0 commit comments