Compare vcfs#299
Merged
Merged
Conversation
Compares a downstream variant caller's VCF against a NEAT-simulated truth VCF and attributes each false negative to the simulator's configuration. Variant equivalence is delegated to hap.py; NEAT contributes the FN attribution against the mutation/target BED and simulated-contig set recorded in a new simulation_summary.json artifact. Outputs in --output-dir: - comparison_summary.json: counts, precision/recall/F1, FN reasons - comparison_summary.txt: human-readable rollup - FN_with_reasons.vcf: hap.py FNs with NEAT_REASON INFO tag - fn_attribution.png: optional bar chart (--plot) The read simulator now emits simulation_summary.json alongside its other outputs (config echo + delivered counts) as a prerequisite for compare-vcfs. hap.py is an optional external dependency (conda bioconda::hap.py); compare-vcfs exits with a clear install hint when not found. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Audit found five untested paths in the compare-vcfs work:
- CLI layer: subcommand registration + argument routing
- --plot flag: runner integration (on/off)
- Stale BED path in simulation_summary: undefined failure mode
- Chrom-prefix mismatch ('1' vs 'chr1'): documents a user gotcha
- Multi-FN scaling through real hap.py
Adds 9 tests across 4 files (+1 new test_cli.py). Full suite: 727 passing.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When a mutation_bed or target_bed uses different chrom names than the reference (e.g., '1' vs 'chr1', 'MT' vs 'chrM'), NEAT previously produced silent garbage — every FN tagged 'outside_mutation_bed' even though they were actually inside. Changes: - new neat/common/chrom_names.py with prefix/mitochondrial heuristics and a user-supplied alias-file parser - simulation_summary.json now records delivered.reference_contigs (full FASTA contig set, separate from contigs_simulated) - compare_vcfs_runner runs detect_chrom_naming_mismatches before attribution; emits WARNING logs AND a 'warnings' array in comparison_summary.json with suggested aliases - new --chrom-aliases TSV flag applies user-supplied mappings to BED chrom names at load time - README + ChangeLog updated; mismatch behavior documented as non- normalizing by default NEAT does not auto-normalize: silent renaming masks real bugs. Detection + warnings + opt-in normalization is the policy. 34 new tests (chrom_names utility, schema, attribution detection, runner integration). Full suite: 761 passing. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Semantic fix: when chrom-name mismatches make a BED unusable, the runner now skips attribution against that BED entirely (via new skip_beds param on attribute_fns) instead of tagging every FN as 'outside_<bed>'. The warning in comparison_summary.json explains the cause; the fn_attribution counts now reflect what NEAT actually checked. Test additions (7): - render_summary_txt with/without warnings section (rendering was untested) - detect_chrom_naming_mismatches skips empty BEDs (no spurious warning) - detect_chrom_naming_mismatches still warns when user aliases don't resolve the mismatch (suggested mapping comes from raw BED chroms, not user input) - attribute_fns skip_beds suppresses outside_<bed> tags - attribute_fns skip_beds doesn't affect outside_simulated_contigs - Real hap.py end-to-end with mismatched mutation_bed (warning fires, outside_mutation_bed absent, --chrom-aliases silences warning) Existing test updated: - test_runner_emits_chrom_mismatch_warning_into_json_report now also asserts the semantic guarantee that outside_mutation_bed is absent. Full suite: 768 passing. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add's a compare vcf utility which uses hap.py to analyze a vcf created on fastq or bam data supplied by NEAT, then adds additional analysis via an exported json file, which gives the user a nice output and a chart showing the differences.
Compares a downstream variant caller's VCF against a NEAT-simulated truth VCF and attributes each false negative to the simulator's configuration. Variant equivalence is delegated to hap.py; NEAT contributes the FN attribution against the mutation/target BED and simulated-contig set recorded in a new simulation_summary.json artifact.
Outputs in --output-dir:
The read simulator now emits simulation_summary.json alongside its other outputs (config echo + delivered counts) as a prerequisite for compare-vcfs.
hap.py is an optional external dependency (conda bioconda::hap.py); compare-vcfs exits with a clear install hint when not found.