Add interactive ntSynt-viz output

README.md

@@ -12,20 +12,22 @@
 4. [Installation](#install)
 5. [Usage](#usage)
 6. [Examples](#example)
-7. [Citing](#citing)
-8. [License](#license)
+7. [Interactive plots](#interactive-plots)
+8. [Citing](#citing)
+9. [License](#license)
 
 ## Credit  <a name=credit></a>
 Written by Lauren Coombe
 
 ## Description <a name=description></a>
-ntSynt-viz is an easy-to-use framework for generating ribbon plots combined with chromosome painting to visualize multi-genome synteny blocks. The tool is set-up to accept synteny blocks formatted in the [ntSynt](https://github.com/BirolLab/ntSynt) style, but any multi-genome synteny block file that adheres to the simple, BED-like TSV format of ntSynt can be visualized using ntSynt-viz.
+ntSynt-viz is an easy-to-use framework for generating static and interactive ribbon plots combined with chromosome painting to visualize multi-genome synteny blocks. The tool is set-up to accept synteny blocks formatted in the [ntSynt](https://github.com/BirolLab/ntSynt) style, but any multi-genome synteny block file that adheres to the simple, BED-like TSV format of ntSynt can be visualized using ntSynt-viz.
 
 This flexible framework implements numerous features, including:
 * Option to normalize the strands of input chromosomes based on a target assembly
 * Synteny-guided ordering of assemblies from top-to-bottom, based on an input tree structure or distance estimates from the synteny blocks
 * Sorting chromosomes right-to-left based on synteny to adjacent assemblies
 * Colouring both the ribbons and chromosomes based on the target (top) assembly chromosomes
+* Static and interactive ribbon plot outputs
 
 These features ensure that the output ribbon plots (powered by [gggenomes](https://thackl.github.io/gggenomes/)) are as easily understandable and as information-rich as possible.
 
@@ -48,6 +50,8 @@ These features ensure that the output ribbon plots (powered by [gggenomes](https
   * [stringr](https://stringr.tidyverse.org/)
   * [ggplot2](https://ggplot2.tidyverse.org)
   * [svglite](https://cran.r-project.org/web/packages/svglite/index.html)
+  * [ggiraph](https://davidgohel.github.io/ggiraph/)
+  * [biopython](https://biopython.org/)
 
 ### Installing ntSynt-viz using conda <a name=install></a>
 ```
@@ -66,8 +70,8 @@ export PATH=/path/to/ntsynt-viz/github/ntSynt-viz/bin:$PATH
 ```
 usage: ntsynt_viz.py [-h] --blocks BLOCKS --fais FAIS [FAIS ...] [--name_conversion NAME_CONVERSION] [--tree TREE] [--target-genome TARGET_GENOME] [--normalize]
                      [--indel INDEL] [--length LENGTH] [--seq_length SEQ_LENGTH] [--keep KEEP [KEEP ...]] [--centromeres CENTROMERES] [--haplotypes HAPLOTYPES]
-                     [--order ORDER] [--prefix PREFIX] [--format {png,pdf,svg}] [--scale SCALE] [--height HEIGHT] [--width WIDTH] [--dpi DPI] [--annotate-genome-info]
-                     [--no-arrow] [--ribbon_adjust RIBBON_ADJUST] [-f] [-n] [-v]
+                     [--order ORDER] [--prefix PREFIX] [--format {png,pdf,svg}] [--scale SCALE] [--height HEIGHT] [--width WIDTH] [--dpi DPI]
+                     [--annotate-genome-info] [--optimize-ordering] [--no-arrow] [--ribbon_adjust RIBBON_ADJUST] [-f] [-n] [-v]
 
 Visualizing multi-genome synteny
 
@@ -98,7 +102,6 @@ main plot formatting arguments:
                         File listing haplotype assembly names: TSV, maternal/paternal assembly file names separated by tabs.
   --order ORDER         Optional file specifying the order of genomes in the ribbon plot. If supplied, will override synteny distance-based ordering. If --tree
                         supplied, the ordering must be compatible with the phylogenetic tree.
-  --no-arrow            Only used with --normalize; do not draw arrows indicating reverse-complementation
 
 block filtering arguments:
   --indel INDEL         Indel size threshold (used in computing synteny-based distances) [50000]
@@ -118,6 +121,8 @@ output arguments:
   --dpi DPI             Resolution of output plot - png output only [300]
   --annotate-genome-info
                         Add annotations about number of sequences and genome size to the right of each genome in the ribbon plot
+  --optimize-ordering   Optimize tree-guided genome sorting using inversions. Only use with strictly bifurcating trees.
+  --no-arrow            Only used with --normalize; do not draw arrows indicating reverse-complementation
   --ribbon_adjust RIBBON_ADJUST
                         Ratio for adjusting spacing beside ribbon plot. Increase if ribbon plot labels are cut off, and decrease to reduce the white space to
                         the left of the ribbon plot [0.1]
@@ -147,6 +152,25 @@ For more information about the output files from ntSynt-viz, check out our [wiki
 ## Using ntSynt-viz with pangenome graphs or synteny blocks from tools other than ntSynt
 To visualize synteny information in pangenome graphs or from synteny block detection tools other than ntSynt, the synteny blocks simply need to be converted to the straightforward [ntSynt format](https://github.com/BirolLab/ntsynt?tab=readme-ov-file#output-files). For convenience, we also provide some scripts to do this conversion in the `conversion_scripts` directory.
 
+## Interactive plots
+As of v1.1.0, ntSynt-viz generates an HTML file to allow interactive exploration of the ribbon plot.
+
+Interactive elements include:
+* Hover/click on a ribbon to highlight the entire synteny block and display block coordinates and orientations
+* Hover/click on a chromosome to display ID and length
+* Click on one or more chromosomes in the legend to highlight the correseponding synteny blocks
+
+Example ribbon plot: XXX
+
+<video src="https://github.com/user-attachments/assets/d8263726-b1dc-48c6-8ec0-fd60021add84"
+       autoplay
+       loop
+       muted
+       playsinline
+       width="100%">
+</video>
+
+
 ## Citing <a name=citing></a>
 
 Thank you for your [![Stars](https://img.shields.io/github/stars/BirolLab/ntSynt-viz.svg)](https://github.com/BirolLab/ntSynt-viz/stargazers) and for using and promoting this free software! We hope that ntSynt-viz (& ntSynt) is useful to you and your research.

azure-pipelines.yml

@@ -28,7 +28,7 @@ jobs:
 
   - script: |
       source activate ntsynt_viz_ci
-      mamba install --yes -c conda-forge -c bioconda quicktree r-base bioconductor-treeio r-ggpubr bioconductor-ggtree r-phytools r-dplyr r-argparse r-scales r-stringr pylint intervaltree snakemake r-ggplot2 r-gggenomes r-svglite
+      mamba install --yes -c conda-forge -c bioconda quicktree r-base bioconductor-treeio r-ggpubr bioconductor-ggtree r-phytools r-dplyr r-argparse r-scales r-stringr pylint intervaltree snakemake r-ggplot2 r-gggenomes r-svglite r-ggiraph biopython
     displayName: Install dependencies
 
   - script: |
@@ -46,7 +46,7 @@ jobs:
       source activate ntsynt_viz_ci
       export PATH=$(pwd)/bin:$PATH
       cd tests
-      ntsynt_viz.py --blocks great-apes.ntSynt.synteny_blocks.tsv --fais fais.tsv --tree great-apes.mt-tree.nwk --name_conversion great-apes.name-conversions.tsv --normalize --prefix great-apes_ribbon-plots --ribbon_adjust 0.14 --scale 1e9
+      ntsynt_viz.py --blocks great-apes.ntSynt.synteny_blocks.tsv --fais fais.tsv --tree great-apes.mt-tree.nwk --name_conversion great-apes.name-conversions.tsv --normalize --prefix great-apes_ribbon-plots --ribbon_adjust 0.14 --scale 1e9 --target-genome Homo_sapiens
     displayName: Run Example 1 test (PNG output)
 
   - script: |
@@ -76,6 +76,9 @@ jobs:
   - publish: tests/great-apes_ribbon-plots_ribbon-plot_tree.png
     artifact: Example1
 
+  - publish: tests/great-apes_ribbon-plots_ribbon-plot_tree.html
+    artifact: Example1_HTML
+
   - publish: tests/great-apes_ribbon-plots_no-tree_ribbon-plot.pdf
     artifact: Example2
 
@@ -85,6 +88,9 @@ jobs:
   - publish: tests/great-apes_ribbon-plots_tree_order_ribbon-plot.png
     artifact: Example4
 
+  - publish: tests/great-apes_ribbon-plots_tree_order_ribbon-plot.html
+    artifact: Example4_HTML
+
 
 - job:
   displayName: macOS-latest
@@ -113,7 +119,7 @@ jobs:
 
   - script: |
       source activate ntsynt_viz_ci
-      mamba install --yes -c conda-forge -c bioconda quicktree r-base bioconductor-treeio r-ggpubr bioconductor-ggtree r-phytools r-dplyr r-argparse r-scales r-stringr pylint intervaltree snakemake r-ggplot2 r-gggenomes r-svglite
+      mamba install --yes -c conda-forge -c bioconda quicktree r-base bioconductor-treeio r-ggpubr bioconductor-ggtree r-phytools r-dplyr r-argparse r-scales r-stringr pylint intervaltree snakemake r-ggplot2 r-gggenomes r-svglite r-ggiraph biopython
     displayName: Install dependencies
 
   - script: |

bin/ntsynt_viz.py

@@ -67,7 +67,7 @@ def main():
                         action="store_true")
     block_filter_group.add_argument("--indel", help="Indel size threshold [50000]", default=50000, type=int)
     block_filter_group.add_argument("--length", help="Minimum synteny block length [100000]", default=100000, type=int)
-    block_filter_group.add_argument("--seq_length", help="Minimum sequence length [500000]", default=500000, type=float)
+    block_filter_group.add_argument("--seq_length", help="Minimum sequence length [500000]", default=500000, type=int)
     block_filter_group.add_argument("--keep", help="List of genome_name:chromosome to show in visualization. "
                                     "All chromosomes with links to the specified chromosomes will also be shown.",
                                     nargs="+", required=False, type=str)
@@ -100,7 +100,11 @@ def main():
                               help="Add annotations about number of sequences "
                                   "and genome size to the right of each genome in the ribbon plot",
                               action="store_true")
-    main_formatting_group.add_argument("--no-arrow", help="Only used with --normalize; "
+    output_group.add_argument("--optimize-ordering",
+                              help="Optimize tree-guided genome sorting using inversions. "
+                              "Only use with strictly bifurcating trees.",
+                              action="store_true")
+    output_group.add_argument("--no-arrow", help="Only used with --normalize; "
                         "do not draw arrows indicating reverse-complementation",
                         action="store_true")
     output_group.add_argument(
@@ -127,6 +131,8 @@ def main():
     if len(args.fais) == 1:
         args.fais = read_fai_files(args.fais[0])
 
+    print(f"Running ntSynt-viz {NTSYNT_VIZ_VERSION}...", flush=True)
+
     cmd = f"snakemake -s {base_dir}/ntsynt_viz.smk " \
             f"--cores 2 " \
             f"--config " \
@@ -162,6 +168,8 @@ def main():
         cmd += f"order={args.order} "
     if args.annotate_genome_info:
         cmd += "annotate_genome_info=True "
+    if args.optimize_ordering:
+        cmd += "optimize_ordering=True "
     if args.tree:
         cmd += f"tree={args.tree} "
         target = "gggenomes_ribbon_plot_tree"

bin/ntsynt_viz.smk

@@ -30,6 +30,8 @@ res = config.get("dpi", 300)
 orders = config.get("order", [])
 annotate_genome_info = config.get("annotate_genome_info", False)
 
+optimize_ordering = config.get("optimize_ordering", False)
+
 def sort_fais(fai_list, name_conversion, orders):
     "Based on the name conversion TSV, sort the FAIs based on orders"
     # Read in name conversions
@@ -91,16 +93,20 @@ output_files = {
     }
 
 rule all:
-    input: output_files[format_img]
+    input: output_files[format_img],
+            f"{prefix}_ribbon-plot.html",
+            f"{prefix}_ribbon-plot_LLM-info.md",
 
-ouput_files_tree = {
+output_files_tree = {
         "png": f"{prefix}_ribbon-plot_tree.png",
         "pdf": f"{prefix}_ribbon-plot_tree.pdf",
         "svg": f"{prefix}_ribbon-plot_tree.svg",
     }
 
 rule gggenomes_ribbon_plot_tree:
-    input: ouput_files_tree[format_img]
+    input: output_files_tree[format_img],
+            f"{prefix}_ribbon-plot_tree.html",
+            f"{prefix}_ribbon-plot_tree_LLM-info.md"
 
 rule renaming:
     input: 
@@ -143,15 +149,25 @@ rule make_nj_tree:
 
 rule cladogram:
     input:
-        rules.make_nj_tree.output
+        tree = rules.make_nj_tree.output,
+        blocks = rules.renaming.output.renamed_blocks
     output:
         orders_tmp = temp(f"{prefix}_est-distances.order_tmp.tsv"),
         nwk_tmp = temp(f"{prefix}_est-distances_tmp.cladogram.nwk")
     params:
         prefix = f"{prefix}_est-distances",
         target = f"--target {target_genome}" if target_genome else ""
-    shell:
-        "ntsynt_viz_distance_cladogram.R --nwk {input} -p {params.prefix} --update_nwk {params.target}" 
+    run:
+        if optimize_ordering:
+            shell("""
+                set -eux -o pipefail
+                ntsynt_viz_distance_cladogram.R --nwk {input.tree} -p {params.prefix} --update_nwk {params.target}
+                ntsynt_viz_optimize_tree_topology.py --blocks {input.blocks} --tree {output.nwk_tmp} --out-order {output.orders_tmp} \
+                    --out-tree tmp_tree.nwk {params.target}
+                mv tmp_tree.nwk {output.nwk_tmp}
+                """)
+        else:
+            shell("set -eux -o pipefail; ntsynt_viz_distance_cladogram.R --nwk {input.tree} -p {params.prefix} --update_nwk {params.target}")
 
 rule orders:
     input: 
@@ -185,7 +201,7 @@ rule sort_blocks:
     output:
         sorted_blocks = f"{blocks_no_suffix}.renamed.sorted.blocks.tsv",
         intermediate_blocks = temp(f"{blocks_no_suffix}.renamed.sorted-tmp.tsv") if normalize else [],
-        chrom_oris = temp(f"{blocks_no_suffix}.renamed.sorted.chrom-orientations.tsv") if normalize else []
+        chrom_oris = f"{blocks_no_suffix}.renamed.sorted.chrom-orientations.tsv" if normalize else []
     params:
         intermediate_blocks = f"{blocks_no_suffix}.renamed.sorted-tmp.tsv",
         name_conversion = f"-c {name_conversion}" if name_conversion else "",
@@ -250,7 +266,7 @@ rule chrom_paint:
     input: links = rules.gggenomes_files.output.links
     output: colour_feats = f"{prefix}.chrom-paint-feats.tsv"
     shell:
-        '''cat {input.links} |perl -ne 'chomp; @a=split("\t"); if(!defined $ct){{print "block_id\tseq_id\tbin_id\tstart\tend\tcolour_block\n"; $ct=1}} else {{print "$a[0]\t$a[1]\t$a[2]\t$a[3]\t$a[4]\t$a[10]\n"; print "$a[0]\t$a[5]\t$a[6]\t$a[7]\t$a[8]\t$a[10]\n";}}' > {output.colour_feats}'''
+        '''cat {input.links} |perl -ne 'chomp; @a=split("\t"); if(!defined $ct){{print "block_id\tseq_id\tbin_id\tstart\tend\tcolour_block\n"; $ct=1}} else {{print "$a[0]\t$a[1]\t$a[2]\t$a[3]\t$a[4]\t$a[12]\n"; print "$a[0]\t$a[5]\t$a[6]\t$a[7]\t$a[8]\t$a[12]\n";}}' > {output.colour_feats}'''
 
 rule ribbon_plot:
     input: 
@@ -260,7 +276,9 @@ rule ribbon_plot:
         haplotypes = rules.nudges.output.nudges if haplotypes else [],
         colour_seqs = rules.chrom_sorting.output.colour_info
     output:
-        out_img = output_files[format_img]
+        out_img = output_files[format_img] if format_img != "png" else [],
+        out_png = output_files["png"],
+        out_html = f"{prefix}_ribbon-plot.html"
     params:
         prefix = f"{prefix}_ribbon-plot",
         ratio = ribbon_ratio,
@@ -287,7 +305,9 @@ rule ribbon_plot_tree:
         haplotypes = rules.nudges.output.nudges if haplotypes else [],
         colour_seqs = rules.chrom_sorting.output.colour_info
     output:
-        out_img = ouput_files_tree[format_img]
+        out_img = output_files_tree[format_img] if format_img != "png" else [],
+        out_png = output_files_tree["png"],
+        out_html = f"{prefix}_ribbon-plot_tree.html"
     params:
         prefix = f"{prefix}_ribbon-plot_tree",
         ratio = ribbon_ratio,
@@ -303,3 +323,39 @@ rule ribbon_plot_tree:
         "ntsynt_viz_plot_synteny_blocks_ribbon_plot.R -s {input.sequences} -l {input.links} -p {params.prefix} --tree {input.tree}"
         " --ratio {params.ratio} --scale {params.scale} -c {input.colour_feats} --format {params.out_img_format}  --height {params.height} --width {params.width}"
         " --order {input.orders} {params.centromeres} {params.arrow} {params.haplotypes} --colour_indices {input.colour_seqs} {params.resolution} {params.annotate_genome_info}"
+
+rule llm_instructions:
+    input: 
+        ribbon_png = rules.ribbon_plot.output.out_png, # png
+        chrom_oris = f"{blocks_no_suffix}.renamed.sorted.chrom-orientations.tsv" if normalize else [],
+        synteny_tsv = rules.sort_blocks.output.sorted_blocks,
+        chrom_lengths = rules.chrom_sorting.output.sorted_seqs,
+    output:
+        markdown = f"{prefix}_ribbon-plot_LLM-info.md"
+    params:
+        normalize_opt = lambda wc, input: (
+            f"--normalize {input.chrom_oris}" if normalize else ""
+        ),
+        top_n = max(50, len(fais)*2)
+    shell:
+        """
+        ntsynt_viz_generate_llm_markdown.py -o {output.markdown} --top-n {params.top_n} --image {input.ribbon_png} --lengths {input.chrom_lengths} {params.normalize_opt} {input.synteny_tsv}
+        """
+
+rule llm_instructions_tree:
+    input: 
+        ribbon_png = rules.ribbon_plot_tree.output.out_png, # png
+        chrom_oris = f"{blocks_no_suffix}.renamed.sorted.chrom-orientations.tsv" if normalize else [],
+        synteny_tsv = rules.sort_blocks.output.sorted_blocks,
+        chrom_lengths = rules.chrom_sorting.output.sorted_seqs,
+    output:
+        markdown = f"{prefix}_ribbon-plot_tree_LLM-info.md"
+    params:
+        normalize_opt = lambda wc, input: (
+            f"--normalize {input.chrom_oris}" if normalize else ""
+        ),
+        top_n = max(50, len(fais)*2)
+    shell:
+        """
+        ntsynt_viz_generate_llm_markdown.py -o {output.markdown} --top-n {params.top_n} --image {input.ribbon_png} --top-n 50 --lengths {input.chrom_lengths} {params.normalize_opt} {input.synteny_tsv}
+        """

bin/ntsynt_viz_format_blocks_gggenomes.py

@@ -94,7 +94,7 @@ def make_links_file(synteny_file, prefix, valid_blocks_set, target_assembly):
     target_assembly_chrom = None
     with open(f"{prefix}.links.tsv", 'w', encoding="utf-8") as fout:
         fout.write("block_id\tseq_id\tbin_id\tstart\tend\t"\
-                    "seq_id2\tbin_id2\tstart2\tend2\tstrand\tcolour_block\n")
+                    "seq_id2\tbin_id2\tstart2\tend2\tstrand\tstrand1\tstrand2\tcolour_block\n")
         with open(synteny_file, 'r', encoding="utf-8") as fin:
             for line in fin:
                 line = line.strip().split("\t")
@@ -104,7 +104,8 @@ def make_links_file(synteny_file, prefix, valid_blocks_set, target_assembly):
                     start_curr, end_curr = curr_block.start, curr_block.end
                     relative_ori = "-" if curr_block.strand != prev_line.strand else "+"
                     out_line = f"{curr_block.id}\t{prev_line.chrom}\t{prev_line.genome}\t{start_prev}\t{end_prev}\t"\
-                                f"{curr_block.chrom}\t{curr_block.genome}\t{start_curr}\t{end_curr}\t{relative_ori}"
+                                f"{curr_block.chrom}\t{curr_block.genome}\t{start_curr}\t{end_curr}\t{relative_ori}\t"\
+                                f"{prev_line.strand}\t{curr_block.strand}"
                     lines_to_print.append(out_line)
 
                 if prev_line is not None and prev_line.id != curr_block.id: