Release v1.21.0 — per-variant allele fraction#403
Merged
Conversation
Match the released v1.20.1 tag (RNG out-of-range fix). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
chore(conda-recipe): bump to v1.20.1
Update the SCN status doc to reflect completion: SV + CNV models built for both strains and rendered end-to-end (loop closed), the MM26-vs-PA3/BD3 contrast with its confound caveat, the RDCN→INFO/CN CNV format gap + fix, and the v1.20.1 RNG release. Refreshed tooling table, Delta ops notes (strain-override + verify-the-statistic), and the open items (common-reference comparison, Phase 2, harness #395, CNV biology, long reads). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Per-strain sim-vs-truth fidelity is the validation goal and stands on its own regardless of reference; the strain contrast + common-reference run are a biological virulence claim on top of the validated tool, not a validation gap. Reframe "Open / next steps" to separate tooling/validation items that can proceed (Phase 2 AF replay, harness #395) from biology extensions parked pending colleague feedback (common-ref comparison, CNV sanity, long reads). Co-Authored-By: Claude <noreply@anthropic.com>
Trace the alt-fraction mechanism end-to-end on develop. Corrects stale line numbers and identifies the real lever: the per-read alt/ref branch in generate_read (fastq_tools.rs:462), which drives both emitted reads and the AdCounter (so golden AF tracks it for free). mutate_position is a secondary SV-junction-only path. Input-VCF variants confirmed to flow through the main path via variant_map. Scope reduced to: add Variant.allele_fraction, populate in from_file (INFO/AF or FORMAT/AD), generalize the :462 branch; no golden-VCF output change needed. Notes overlap with parked polyploidy #266. Co-Authored-By: Claude <noreply@anthropic.com>
Add an optional `allele_fraction: Option<f64>` on `Variant`, parsed from an
input VCF (INFO/AF, else FORMAT/AD alt/(ref+alt)) and honored by the per-read
alt/ref decision in generate_read (fastq_tools.rs) and the SV-junction path
(mutated_map.rs). When present, the alt allele is emitted on that fraction of
overlapping reads instead of the {homozygous=1.0, het=0.5} default — letting an
input VCF drive a continuous AF spectrum (pooled/somatic data). AD is used over
GT for pooled data, where GT is not diploid-meaningful.
The golden VCF's measured AF (GT:AD:DP:AF) tracks the requested fraction for
free, since the same branch increments the AdCounter — no output change needed.
Enabling change for SCN Phase 2 (realism epic #311).
Determinism: the None (no-AF) path keeps the exact same RNG draw and
short-circuit, so default runs stay byte-identical (model_parity + full suite
green). Variant drops its unused Eq/Hash derives (f64 is only PartialEq); it's
never a map key or set member — compare_vcfs hashes a separate VariantKey.
Tests: parse (INFO/AF, AD, multiallelic, out-of-range, absent) + from_file
population; a generate_read counter test asserting the AD split tracks f; and
allele_fraction_e2e.rs driving the real binary end-to-end (input AF 0.25/0.75 →
golden AF matches, clear of the 0.5 default).
Co-Authored-By: Claude <noreply@anthropic.com>
#398) Adds the Phase 2 real-data validation for per-variant allele_fraction: - scn_af_compare.py — dependency-free per-site AF correlation of two VCFs (truth/input vs simulated golden). Matches sites by CHROM/POS/REF/ALT; reads AF from FORMAT/AF, INFO/AF, or FORMAT/AD; reports Pearson/Spearman, MAE/RMSE overall and per AF-decile, with a --min-depth gate. Mirrors sbs96_compare.py. - run_scn_af_validation.sh — SLURM driver: derive per-site AF from the staged pool VCF's AD (SNVs only), run gen-reads with it as input_vcf at mutation_rate=0 / sv_rate_scale=0 (only supplied sites placed), then compare golden AF vs pool AF. Untestable from the workstation (Delta filesystem); scn_af_compare.py verified locally on synthetic VCFs (r=1.0, per-decile MAE, depth gating). Driver has explicit preflights (FORMAT/AD present, non-zero site counts, banner) per the verify-the-statistic-not-the-PASS rule. Co-Authored-By: Claude <noreply@anthropic.com>
stage_scn.sh: add `-a FORMAT/AD` to the bcftools mpileup call so staged pool VCFs carry per-sample allele depths (additive; needed to derive per-site AF). run_scn_af_validation.sh: when the pool VCF predates that change and lacks AD, re-derive AD from the staged BAM at just the SNV sites (-R restricted) instead of forcing a full re-stage. POOL_AF is both the gen-reads input and the truth, so the comparison stays self-consistent either way. Co-Authored-By: Claude <noreply@anthropic.com>
The first Delta run (job 20270457) returned r=0.61 with the "truth" piled at 0.5 — a false negative. Root cause: the harness ran `bcftools +fill-tags -t AF`, whose AF tag is computed from GT, so a diploid-called pool collapses every het to 0.5 and hom to 1.0 — not the pooled allele fraction. Both gen-reads and the comparison then used 0.5, so we tested "reproduce 0.5", not the real spectrum. (The feature itself was correct: hom sites reproduced at exactly 1.0, hets scattered around 0.5 at textbook 50x binomial noise.) Fix: pass the AD-carrying VCF through with NO INFO/AF. Both gen-reads (from_file) and scn_af_compare.py derive the fraction from FORMAT/AD = alt/(ref+alt), which is the real pool spectrum. Added a pre-run sanity histogram of the AD-derived truth AF so a collapsed spectrum surfaces instead of hiding behind exit 0. Co-Authored-By: Claude <noreply@anthropic.com>
docs(scn): status — track complete (SV/CNV/render/contrast)
feat(gen-reads): per-variant allele_fraction from input VCF (#398)
Bump workspace version 1.20.1 → 1.21.0 and add the CHANGELOG entry for the allele_fraction feature (#398/#399), which merged to develop without a version bump — leaving develop colliding with the already-released 1.20.1. Minor bump: the feature is backward-compatible (variants without an allele fraction are byte-identical to prior output). No code changes — version + CHANGELOG + lockfile only. Co-Authored-By: Claude <noreply@anthropic.com>
Mechanical `cargo fmt --all` across the workspace — no logic changes. develop had drifted from rustfmt's output (no enforcement existed), so a local `cargo fmt` would rewrite ~38 files and bury real diffs; this one-time sweep resets the tree to the canonical format. Adds rustfmt.toml (edition 2024) to pin the style. Verify: `cargo fmt --all -- --check` is clean; full test suite green (26 binaries). Co-Authored-By: Claude <noreply@anthropic.com>
Adds a `fmt` job running `cargo fmt --all -- --check` so formatting drift fails CI instead of accumulating silently (the reason the sweep above was needed). Co-Authored-By: Claude <noreply@anthropic.com>
chore(release): v1.21.0 — per-variant allele fraction
chore: normalize formatting (cargo fmt) + CI fmt gate
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Release v1.21.0 to
main. develop is ahead of main by the work below and has no divergenceto reconcile (main carries no unique content). fmt clean, full suite green (26 binaries),
binary reports
1.21.0.Headline — per-variant allele fraction (#398 / #399)
gen-reads now honors an optional per-variant
allele_fractionfrom the input VCF (INFO/AF,else
FORMAT/AD), so an input VCF can drive a continuous allele-frequency spectrum instead ofonly
{0.5, 1.0}. Backward-compatible — variants without an allele fraction are byte-identicalto prior output. Validated on real SCN Pool-seq: simulated-vs-real per-site AF r = 0.98 at 150×,
unbiased. See CHANGELOG for detail.
Also included
(
scn_af_compare.py,run_scn_af_validation.sh), and status docs (docs(scn): status — track complete (SV/CNV/render/contrast) #397).cargo fmtsweep,rustfmt.toml,and a
cargo fmt --all -- --checkjob so drift fails CI going forward.After merge
v1.21.0on main (git tag v1.21.0 <merge sha> && git push origin v1.21.0).