Fix erroneous INFO/END recalculation; New --no-realign option

pd3 · pd3 · commit d93757761b28 · 2026-04-07T13:38:32.000+01:00
- INFO/END was calculated incorrectly for <DUP> symbolic alleles - Added `--no-realign[=NUM]` to skip realignment during normalization; when NUM is given, only events longer than NUM bp are skipped. This deprecates the option `--do-not-normalize` (equivalent to `--no-realign=0`). Resolves #2500
diff --git a/NEWS b/NEWS
@@ -50,6 +50,12 @@ Changes affecting specific commands:
 
     - Fix a bug in splitting Type=String FORMAT fields via -a and -m (#2476)
 
+    - Fix erroneous INFO/END recalculation of symbolic alleles (#2500).
+
+    - Added `--no-realign[=NUM]` to skip realignment during normalization; when NUM is given,
+      only events longer than NUM bp are skipped. This deprecates the option `--do-not-normalize`
+      (equivalent to `--no-realign=0`). See also #2500.
+
 * bcftools +prune
 
     - When pruning by window length in base pairs, one out-of-window record would
diff --git a/doc/bcftools.1 b/doc/bcftools.1
diff --git a/doc/bcftools.html b/doc/bcftools.html
@@ -4,7 +4,7 @@
 <meta charset="UTF-8">
 <meta http-equiv="X-UA-Compatible" content="IE=edge">
 <meta name="viewport" content="width=device-width, initial-scale=1.0">
-<meta name="generator" content="Asciidoctor 2.0.16.dev">
+<meta name="generator" content="Asciidoctor 2.0.23">
 <title>bcftools(1)</title>
 <link rel="stylesheet" href="./docbook-xsl.css">
 </head>
@@ -50,7 +50,7 @@ <h2 id="_description">DESCRIPTION</h2>
 <div class="sect2">
 <h3 id="_version">VERSION</h3>
 <div class="paragraph">
-<p>This manual page was last updated <strong>2026-03-18</strong> and refers to bcftools git version <strong>1.23.1</strong>.</p>
+<p>This manual page was last updated <strong>2026-04-07 10:58 BST</strong> and refers to bcftools git version <strong>1.23.1-61-gb64ebe7b+</strong>.</p>
 </div>
 </div>
 <div class="sect2">
@@ -2822,7 +2822,8 @@ <h3 id="merge">bcftools merge [<em>OPTIONS</em>] <em>A.vcf.gz</em> <em>B.vcf.gz<
 </div>
 <div class="listingblock">
 <div class="content">
-<pre>-m none        ..  no new multiallelics, output multiple records instead
+<pre>-m exact       ..  require the exact same alleles
+-m none        ..  no new multiallelics, output multiple records instead
 -m snps        ..  allow multiallelic SNP records
 -m indels      ..  allow multiallelic indel records
 -m both        ..  both SNP and indel records can be multiallelic
@@ -3548,7 +3549,8 @@ <h3 id="norm">bcftools norm [<em>OPTIONS</em>] <em>file.vcf.gz</em></h3>
 cannot be stressed enough, that <em>s</em> will NOT fix strand issues in
 your VCF, do NOT use it for that purpose!!! (Instead see
 <a href="http://samtools.github.io/bcftools/howtos/plugin.af-dist.html" class="bare">http://samtools.github.io/bcftools/howtos/plugin.af-dist.html</a> and
-&lt;<a href="http://samtools.github.io/bcftools/howtos/plugin.fixref.html&gt;" class="bare">http://samtools.github.io/bcftools/howtos/plugin.fixref.html&gt;</a>.)</p>
+<a href="http://samtools.github.io/bcftools/howtos/plugin.fixref.html" class="bare">http://samtools.github.io/bcftools/howtos/plugin.fixref.html</a>.)
+See also the option <strong>-N, --no-realign</strong>.</p>
 </dd>
 <dt class="hdlist1"><strong>-d, --rm-dup</strong> <em>snps</em>|<em>indels</em>|<em>both</em>|<em>all</em>|<em>exact</em></dt>
 <dd>
@@ -3568,7 +3570,7 @@ <h3 id="norm">bcftools norm [<em>OPTIONS</em>] <em>file.vcf.gz</em></h3>
 <dt class="hdlist1"><strong>-f, --fasta-ref</strong> <em>FILE</em><a id="fasta_ref"></a></dt>
 <dd>
 <p>reference sequence. Supplying this option will turn on left-alignment
-and normalization, however, see also the <strong><a href="#do_not_normalize">--do-not-normalize</a></strong>
+and normalization, however, see also the <strong><a href="#do_not_normalize">--no-realign</a></strong>
 option below.</p>
 </dd>
 <dt class="hdlist1"><strong>--force</strong></dt>
@@ -3615,11 +3617,12 @@ <h3 id="norm">bcftools norm [<em>OPTIONS</em>] <em>file.vcf.gz</em></h3>
 <dd>
 <p>see <strong><a href="#common_options">Common Options</a></strong></p>
 </dd>
-<dt class="hdlist1"><strong>-N, --do-not-normalize</strong><a id="do_not_normalize"></a></dt>
+<dt class="hdlist1"><strong>-N, --no-realign</strong> [<em>NUM</em>]<a id="do_not_normalize"></a></dt>
 <dd>
-<p>the <em>-c s</em> option can be used to fix or set the REF allele from the
-reference <em>-f</em>. The <em>-N</em> option will not turn on indel normalisation
-as the <em>-f</em> option normally implies</p>
+<p>do not realign indels and symbolic alleles, which is normally performed
+when <strong>-f, --fasta-ref</strong> option is provided. If <em>NUM</em> is given, only
+events longer than <em>NUM</em> bp are not realigned.
+Previously named <strong>--do-not-normalize</strong>.</p>
 </dd>
 <dt class="hdlist1"><strong>--old-rec-tag</strong> <em>STR</em></dt>
 <dd>
@@ -4024,7 +4027,7 @@ <h4 id="_list_of_plugins_coming_with_the_distribution">List of plugins coming wi
 <p>Convert between similar tags, such as GL,PL,GP or QR,QA,QS or tags with localized alleles e.g. LPL,LAD.
 See <a href="http://samtools.github.io/bcftools/howtos/plugin.tag2tag.html" class="bare">http://samtools.github.io/bcftools/howtos/plugin.tag2tag.html</a> for more.</p>
 </dd>
-<dt class="hdlist1"><strong>trio-dnm2</strong></dt>
+<dt class="hdlist1"><strong>trio-dnm3</strong></dt>
 <dd>
 <p>screen variants for possible de-novo mutations in trios</p>
 </dd>
@@ -4049,6 +4052,10 @@ <h4 id="_list_of_plugins_coming_with_the_distribution">List of plugins coming wi
 <dd>
 <p>print the variants as a set of tables</p>
 </dd>
+<dt class="hdlist1"><strong>vrfs</strong></dt>
+<dd>
+<p>assess site noisiness (variant read frequency score) from a large number of reference samples</p>
+</dd>
 </dl>
 </div>
 </div>
@@ -5829,7 +5836,7 @@ <h2 id="_copying">COPYING</h2>
 </div>
 <div id="footer">
 <div id="footer-text">
-Last updated 2026-03-18 14:50:59 UTC
+Last updated 2026-04-07 10:58:12 +0100
 </div>
 </div>
 </body>
diff --git a/doc/bcftools.txt b/doc/bcftools.txt
@@ -2704,6 +2704,7 @@ the *<<fasta_ref,--fasta-ref>>* option is supplied.
     your VCF, do NOT use it for that purpose!!! (Instead see
     <http://samtools.github.io/bcftools/howtos/plugin.af-dist.html> and
     <http://samtools.github.io/bcftools/howtos/plugin.fixref.html>.)
+    See also the option *-N, --no-realign*.
 
 *-d, --rm-dup* 'snps'|'indels'|'both'|'all'|'exact'::
     If a record is present multiple times, output only the first instance.
@@ -2719,7 +2720,7 @@ the *<<fasta_ref,--fasta-ref>>* option is supplied.
 
 *-f, --fasta-ref* 'FILE'[[fasta_ref]]::
     reference sequence. Supplying this option will turn on left-alignment
-    and normalization, however, see also the *<<do_not_normalize,--do-not-normalize>>*
+    and normalization, however, see also the *<<do_not_normalize,--no-realign>>*
     option below.
 
 *--force*::
@@ -2759,10 +2760,12 @@ the *<<fasta_ref,--fasta-ref>>* option is supplied.
 *--no-version*::
     see *<<common_options,Common Options>>*
 
-*-N, --do-not-normalize*[[do_not_normalize]]::
-    the '-c s' option can be used to fix or set the REF allele from the
-    reference '-f'. The '-N' option will not turn on indel normalisation
-    as the '-f' option normally implies
+*-N, --no-realign* ['NUM'][[do_not_normalize]]::
+    do not realign indels and symbolic alleles (normally performed
+    when *-f, --fasta-ref* option is provided). If 'NUM' is given,
+    skip realignment only for events longer than 'NUM' bp. Note
+    that 'NUM' must follow with no space, e.g. *-N1000* or *--no-realign=1000*.
+    Previously named *--do-not-normalize* and did not accept an argument.
 
 *--old-rec-tag* 'STR'::
     Add INFO/STR annotation with the original record. The format of the
diff --git a/test/norm.dup-end.1.out b/test/norm.dup-end.1.out
@@ -0,0 +1,7 @@
+##fileformat=VCFv4.2
+##FILTER=<ID=PASS,Description="All filters passed">
+##contig=<ID=chr>
+##INFO=<ID=END,Number=1,Type=Integer,Description="">
+##INFO=<ID=SVLEN,Number=1,Type=Integer,Description="SVLEN">
+#CHROM	POS	ID	REF	ALT	QUAL	FILTER	INFO
+chr	56	.	G	<DUP>	.	.	END=61;SVLEN=5
diff --git a/test/norm.dup-end.2.out b/test/norm.dup-end.2.out
@@ -0,0 +1,7 @@
+##fileformat=VCFv4.2
+##FILTER=<ID=PASS,Description="All filters passed">
+##contig=<ID=chr>
+##INFO=<ID=END,Number=1,Type=Integer,Description="">
+##INFO=<ID=SVLEN,Number=1,Type=Integer,Description="SVLEN">
+#CHROM	POS	ID	REF	ALT	QUAL	FILTER	INFO
+chr	57	.	G	<DUP>	.	.	END=62;SVLEN=5
diff --git a/test/norm.dup-end.3.out b/test/norm.dup-end.3.out
@@ -0,0 +1,7 @@
+##fileformat=VCFv4.2
+##FILTER=<ID=PASS,Description="All filters passed">
+##contig=<ID=chr>
+##INFO=<ID=END,Number=1,Type=Integer,Description="">
+##INFO=<ID=SVLEN,Number=1,Type=Integer,Description="SVLEN">
+#CHROM	POS	ID	REF	ALT	QUAL	FILTER	INFO
+chr	57	.	N	<DUP>	.	.	END=62;SVLEN=5
diff --git a/test/norm.dup-end.fa b/test/norm.dup-end.fa
@@ -0,0 +1,3 @@
+>chr
+ACAAGTTTGTTCTCATCATCTAATCATGGTCCTCCCGCAAGGTGCTGCCTGATGAGGACTTGGATCATTC
+AGAGGCAGTCCCATTTTAGGCTCAGTCCTTT
diff --git a/test/norm.dup-end.vcf b/test/norm.dup-end.vcf
@@ -0,0 +1,6 @@
+##fileformat=VCFv4.2
+##contig=<ID=chr>
+##INFO=<ID=END,Number=1,Type=Integer,Description="">
+##INFO=<ID=SVLEN,Number=1,Type=Integer,Description="SVLEN">
+#CHROM	POS	ID	REF	ALT	QUAL	FILTER	INFO
+chr	57	.	N	<DUP>	.	.	END=62;SVLEN=5
diff --git a/test/test.pl b/test/test.pl
@@ -294,6 +294,12 @@
 run_test(\&test_vcf_query,$opts,in=>'query.header',out=>'query.98.2.out',args=>q[-HH -f'%CHROM %POS[ %SAMPLE][ %DP][ %GT]']);
 run_test(\&test_vcf_query,$opts,in=>'query.filter-or',out=>'query.filter-or.1.out',args=>q[-f'[%SAMPLE %DP\\n]' -i'DP=1 || DP=2']);
 run_test(\&test_vcf_query,$opts,in=>'query.filter-or',out=>'query.filter-or.2.out',args=>q[-f'[%SAMPLE %DP\\n]' -i'DP=1 |  DP=2']);
+run_test(\&test_vcf_norm,$opts,in=>'norm.dup-end',fai=>'norm.dup-end',out=>'norm.dup-end.1.out',args=>qq[-c s]);
+run_test(\&test_vcf_norm,$opts,in=>'norm.dup-end',fai=>'norm.dup-end',out=>'norm.dup-end.1.out',args=>qq[-c s -N10]);
+run_test(\&test_vcf_norm,$opts,in=>'norm.dup-end',fai=>'norm.dup-end',out=>'norm.dup-end.2.out',args=>qq[-c s -N]);
+run_test(\&test_vcf_norm,$opts,in=>'norm.dup-end',fai=>'norm.dup-end',out=>'norm.dup-end.2.out',args=>qq[-c s -N0]);
+run_test(\&test_vcf_norm,$opts,in=>'norm.dup-end',fai=>'norm.dup-end',out=>'norm.dup-end.2.out',args=>qq[-c s -N3]);
+run_test(\&test_vcf_norm,$opts,in=>'norm.dup-end',fai=>'norm.dup-end',out=>'norm.dup-end.3.out',args=>qq[-c s -e 'type~"other"']);
 run_test(\&test_vcf_norm,$opts,in=>'norm.check-ref',fai=>'norm.check-ref',out=>'norm.check-ref.1.out',args=>qq[-c s]);
 run_test(\&test_vcf_norm,$opts,in=>'norm.filter',out=>'norm.filter.1.out',args=>qq[-m +both -i 'ID=\@{PATH}/norm.filter.txt']);
 run_test(\&test_vcf_norm,$opts,in=>'norm.filter',out=>'norm.filter.1.out',args=>qq[-m +both -i 'ALT!="C"']);
@@ -318,7 +324,7 @@
 run_test(\&test_vcf_norm,$opts,in=>'norm.merge.2',out=>'norm.merge.2.out',args=>'-m+');
 run_test(\&test_vcf_norm,$opts,in=>'norm.merge.3',out=>'norm.merge.3.out',args=>'-m+');
 run_test(\&test_vcf_norm,$opts,in=>'norm.merge',out=>'norm.merge.strict.out',args=>'-m+ -s');
-run_test(\&test_vcf_norm,$opts,in=>'norm.setref',out=>'norm.setref.out',args=>'-Nc s',fai=>'norm');
+run_test(\&test_vcf_norm,$opts,in=>'norm.setref',out=>'norm.setref.out',args=>'-N -c s',fai=>'norm');
 run_test(\&test_vcf_norm,$opts,in=>'norm.telomere',out=>'norm.telomere.out',fai=>'norm');
 run_test(\&test_vcf_norm,$opts,in=>'norm.rmdup',out=>'norm.rmdup.1.out',args=>'-d snps');
 run_test(\&test_vcf_norm,$opts,in=>'norm.rmdup',out=>'norm.rmdup.2.out',args=>'-d indels');
diff --git a/vcfnorm.c b/vcfnorm.c
@@ -112,7 +112,8 @@ typedef struct
     faidx_t *fai;
     struct { uint64_t tot, set, swap; } nref;
     char **argv, *output_fname, *ref_fname, *vcf_fname, *region, *targets;
-    int argc, rmdup, output_type, n_threads, check_ref, strict_filter, do_indels, clevel;
+    uint32_t do_realign;
+    int argc, rmdup, output_type, n_threads, check_ref, strict_filter, clevel;
     uint64_t nchanged, nskipped, nsplit, njoined, ntotal, nfilter, nrmdup, mrows_op, mrows_collapse, parsimonious;
     int record_cmd_line, force, force_warned, keep_sum_ad;
     abuf_t *abuf;
@@ -708,6 +709,7 @@ static int realign(args_t *args, bcf1_t *line)
     {
         als[0].l = 1;
         als[0].s[ als[0].l ] = 0;
+        new_reflen = line->rlen;
     }
     if ( new_pos==line->pos && !strcasecmp(line->d.allele[0],als[0].s) ) return ERR_OK;
 
@@ -2345,7 +2347,7 @@ static int normalize_line(args_t *args, bcf1_t *line)
             old_rec_tag_init(args,line);
             if ( fix_ref(args,line) ) old_rec_tag_set(args,line,0);
         }
-        if ( args->do_indels )
+        if ( line->rlen < args->do_realign )
         {
             int ret = args->filter_pass ? realign(args, line) : ERR_OK;
 
@@ -2537,7 +2539,7 @@ static void usage(void)
     fprintf(stderr, "    -m, --multiallelics -|+TYPE     Split multiallelics (-) or join biallelics (+), type: snps|indels|both|any [both]\n");
     fprintf(stderr, "        --multi-overlaps 0|.        Fill in the reference (0) or missing (.) allele when splitting multiallelics [0]\n");
     fprintf(stderr, "        --no-version                Do not append version and command line to the header\n");
-    fprintf(stderr, "    -N, --do-not-normalize          Do not normalize indels (with -m or -c s)\n");
+    fprintf(stderr, "    -N, --no-realign [NUM]          Skip realignment; optionally only for events longer than NUM bp (e.g. -N1000, no space)\n");
     fprintf(stderr, "        --old-rec-tag STR           Annotate modified records with INFO/STR indicating the original variant\n");
     fprintf(stderr, "    -o, --output FILE               Write output to a file [standard output]\n");
     fprintf(stderr, "    -O, --output-type u|b|v|z[0-9]  u/b: un/compressed BCF, v/z: un/compressed VCF, 0-9: compression level [v]\n");
@@ -2577,7 +2579,6 @@ int main_vcfnorm(int argc, char *argv[])
     args->aln_win = 100;
     args->buf_win = 1000;
     args->mrows_collapse = COLLAPSE_BOTH;
-    args->do_indels = 1;
     args->ma_use_ref_allele = 1;
     args->clevel = -1;
     int region_is_file  = 0;
@@ -2587,6 +2588,7 @@ int main_vcfnorm(int argc, char *argv[])
     int regions_overlap = 1;
     int targets_overlap = 0;
     args->cmp_func = cmp_bcf_pos;
+    args->do_realign = UINT32_MAX;
 
     static struct option loptions[] =
     {
@@ -2602,7 +2604,8 @@ int main_vcfnorm(int argc, char *argv[])
         {"sort",required_argument,NULL,'S'},
         {"gff-annot",required_argument,NULL,'g'},
         {"right-align",no_argument,NULL,15},            // undocumented, only for debugging
-        {"do-not-normalize",no_argument,NULL,'N'},
+        {"do-not-normalize",no_argument,NULL,16},
+        {"no-realign",optional_argument,NULL,'N'},
         {"multiallelics",required_argument,NULL,'m'},
         {"multi-overlaps",required_argument,NULL,13},
         {"regions",required_argument,NULL,'r'},
@@ -2626,7 +2629,7 @@ int main_vcfnorm(int argc, char *argv[])
         {NULL,0,NULL,0}
     };
     char *tmp;
-    while ((c = getopt_long(argc, argv, "hr:R:f:w:Dd:o:O:c:m:t:T:sNag:W::v:S:i:e:",loptions,NULL)) >= 0) {
+    while ((c = getopt_long(argc, argv, "hr:R:f:w:Dd:o:O:c:m:t:T:sN::ag:W::v:S:i:e:",loptions,NULL)) >= 0) {
         switch (c) {
             case  10:
                 // possibly generalize this also to INFO/AD and other tags
@@ -2668,7 +2671,17 @@ int main_vcfnorm(int argc, char *argv[])
                     error("Unsupported index format '%s'\n", optarg);
                 break;
             case 15 : args->right_align = 1; break;
-            case 'N': args->do_indels = 0; break;
+            case 16 : args->do_realign = 0; break;
+            case 'N':
+                if ( !optarg ) args->do_realign = 0;
+                else
+                {
+                    errno = 0;
+                    double val = strtod(optarg,&tmp);
+                    if ( errno || tmp == optarg || *tmp || val < 0 || val > UINT32_MAX ) error("Could not parse argument: -N, --no-realign %s\n", optarg);
+                    args->do_realign = (uint32_t)val;
+                }
+                break;
             case 'd':
                 if ( !strcmp("snps",optarg) ) args->rmdup = BCF_SR_PAIR_SNPS;
                 else if ( !strcmp("indels",optarg) ) args->rmdup = BCF_SR_PAIR_INDELS;

Original file line number	Diff line number	Diff line change
`@@ -0,0 +1,3 @@`
	`1`	`+>chr`
	`2`	`+ACAAGTTTGTTCTCATCATCTAATCATGGTCCTCCCGCAAGGTGCTGCCTGATGAGGACTTGGATCATTC`
	`3`	`+AGAGGCAGTCCCATTTTAGGCTCAGTCCTTT`