Refresh Daijin and utility documentation

docs/Tutorial/Daijin_tutorial.rst

@@ -5,6 +5,7 @@
 .. _ERR1662533_2.fastq.gz: ftp://ftp.sra.ebi.ac.uk/vol1/fastq/ERR166/003/ERR1662533/ERR1662533_2.fastq.gz
 .. _ERR1662534_1.fastq.gz: ftp://ftp.sra.ebi.ac.uk/vol1/fastq/ERR166/003/ERR1662534/ERR1662534_1.fastq.gz
 .. _ERR1662534_2.fastq.gz: ftp://ftp.sra.ebi.ac.uk/vol1/fastq/ERR166/003/ERR1662534/ERR1662534_2.fastq.gz
+.. _UniProt Aedes aegypti proteins: https://rest.uniprot.org/uniprotkb/stream?compressed=true&format=fasta&query=%28taxonomy_id%3A7159%29
 
 
 
@@ -40,6 +41,10 @@ Mikado should be installed and configured properly (see our :ref:`installation i
 * Stringtie (v1.2.4)
 * CLASS2 (v2.12)
 * SAMtools (v1.1 or later)
+* gnuplot, used by the ``samtools plot-bamstats`` helper
+* bgzip, provided by HTSlib
+* BLAST+, required if using BLAST instead of DIAMOND or if converting BLAST ASN
+  input with ``blast_formatter``
 
 
 Input data
@@ -53,7 +58,8 @@ Throughout this tutorial, we will use data coming from EnsEMBL v89, and from the
 
      * ERX1732854_, left (`ERR1662533_1.fastq.gz`_) and right (`ERR1662533_2.fastq.gz`_) reads
      * ERX1732855_, left (`ERR1662534_1.fastq.gz`_) and right (`ERR1662534_2.fastq.gz`_) reads
- * protein sequences for the related species *Aedes aegypti*, `downloaded from Uniprot <"http://www.uniprot.org/uniprot/?sort=score&desc=&compress=yes&query=taxonomy:diptera%20NOT%20taxonomy:%22Drosophila%20(fruit%20flies)%20[7215]%22%20AND%20taxonomy:%22Aedes%20aegypti%22&fil=&format=fasta&force=yes">`_
+ * protein sequences for the related species *Aedes aegypti*, downloaded from
+   `UniProt Aedes aegypti proteins`_
 
 Preparation of the input data
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -64,7 +70,8 @@ First of all, let us set up a folder with the reference data::
     cd Reference;
     wget ftp://ftp.ensembl.org/pub/release-89/gtf/drosophila_melanogaster/Drosophila_melanogaster.BDGP6.89.gtf.gz;
     wget ftp://ftp.ensembl.org/pub/release-89/fasta/drosophila_melanogaster/dna/Drosophila_melanogaster.BDGP6.dna.toplevel.fa.gz;
-    wget "http://www.uniprot.org/uniprot/?sort=score&desc=&compress=yes&query=taxonomy:diptera%20NOT%20taxonomy:%22Drosophila%20(fruit%20flies)%20[7215]%22%20AND%20taxonomy:%22Aedes%20aegypti%22&fil=&format=fasta&force=yes" -O Aedes_aegypti.fasta.gz;
+    wget "https://rest.uniprot.org/uniprotkb/stream?compressed=true&format=fasta&query=%28taxonomy_id%3A7159%29" \
+        -O Aedes_aegypti.fasta.gz;
     gunzip *gz;
     cd ../;
 

docs/Usage/Utilities.rst

@@ -5,6 +5,10 @@ Mikado miscellaneous scripts
 
 All these utilities can be accessed with the ``mikado util`` CLI. They perform relatively minor tasks.
 
+The subcommands documented below reflect the current ``mikado util`` CLI. Older
+releases also exposed ``merge_blast`` here, but current versions load BLAST
+input during ``mikado serialise`` instead.
+
 awk_gtf
 ~~~~~~~
 
@@ -59,6 +63,34 @@ Usage::
       -c {Intronic,Match,Alternative splicing,Unknown,Fragment,Overlap,Extension,Fusion} [{Intronic,Match,Alternative splicing,Unknown,Fragment,Overlap,Extension,Fusion} ...], --category {Intronic,Match,Alternative splicing,Unknown,Fragment,Overlap,Extension,Fusion} [{Intronic,Match,Alternative splicing,Unknown,Fragment,Overlap,Extension,Fusion} ...]
       -o OUT, --out OUT
 
+.. _collect-compare-command:
+
+collect_compare
+~~~~~~~~~~~~~~~
+
+This utility collects summary statistics from one or more ``mikado compare``
+``*.stats`` files and writes collapsed tables for selected accuracy levels.
+
+Usage::
+
+    $ mikado util collect_compare --help
+    usage: mikado util collect_compare [-h] [-o OUT] [-fmt  [ ...]]
+                                       [-l {all,f1,sn,pr,matches,missed_novel}
+                                           [{all,f1,sn,pr,matches,missed_novel} ...]]
+                                       [-avf]
+                                       [stat ...]
+
+    positional arguments:
+      stat
+
+    optional arguments:
+      -h, --help            show this help message and exit
+      -o OUT, --out OUT     Prefix for the output files.
+      -fmt, --format        List of formats to print the tables into.
+      -l, --levels          Levels to print.
+      -avf, --available-formats
+                            Print out a list of available formats and exit.
+
 convert
 ~~~~~~~
 
@@ -104,28 +136,6 @@ Usage::
                   and will exclude/include all the transcripts children of the
                   selected genes.
 
-.. _merge-blast-command:
-
-merge_blast
-~~~~~~~~~~~
-
-This script merges together various XML BLAST+ files into a single entity. It might be of use when the input data has been chunked into different FASTA files for submission to a cluster queue. It is also capable of converting from ASN files and of dealing with GZipped files.
-
-Usage::
-
-    $ mikado util merge_blast --help
-    usage: mikado.py util merge_blast [-h] [-v] [-l LOG] [--out [OUT]]
-                                      xml [xml ...]
-
-    positional arguments:
-      xml
-
-    optional arguments:
-      -h, --help         show this help message and exit
-      -v, --verbose
-      -l LOG, --log LOG
-      --out [OUT]
-
 .. _metrics-command:
 
 metrics
@@ -262,6 +272,26 @@ This script is used to collect statistics obtained with from the :ref:`mikado ut
     optional arguments:
       -h, --help  show this help message and exit
 
+analyse_metrics.py
+~~~~~~~~~~~~~~~~~~
+
+This script compares a Mikado metrics table with a reference map and reports,
+for each metric, whether selected reference transcripts tend to match the
+maximum, minimum, or a target value within their loci.
+
+calculate_distances.py
+~~~~~~~~~~~~~~~~~~~~~~
+
+This script reports the nearest neighbouring genes before and after each gene in
+a GFF3 annotation, using the same reference indexing helpers as ``mikado
+compare``.
+
+create_model.py
+~~~~~~~~~~~~~~~
+
+This script builds a candidate scoring model from Mikado TMAP and metrics files.
+It is intended as an exploratory helper when designing a scoring configuration.
+
 bam2gtf.py
 ~~~~~~~~~~
 
@@ -311,6 +341,20 @@ Python3 wrapper for the CLASS [Class2]_ assembler. It will perform the necessary
                             help and exit.
       -v, --verbose
 
+extract_promoter_regions.py
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+This script extracts promoter FASTA sequences for a list of genes from a genome
+FASTA and GFF3 annotation. It can write multiple upstream distances and can
+optionally ignore neighbouring genes or compress the output.
+
+extract_test_case.py
+~~~~~~~~~~~~~~~~~~~~
+
+This script extracts a compact Mikado test case from a prepared GTF, populated
+database, reference FASTA and genomic region. It writes the region FASTA,
+shifted GTF and selected database rows under the requested output prefix.
+
 getFastaFromIds.py
 ~~~~~~~~~~~~~~~~~~
 
@@ -429,6 +473,13 @@ Quick script to remove sequences from a given organism from SwissProt files, and
                             Organism to be excluded
       --format {fasta}      Output format. Choices: fasta. Default: fasta.
 
+remove_utrs.py
+~~~~~~~~~~~~~~
+
+This script strips UTRs from a GFF3/GTF annotation by reducing coding
+transcripts to their CDS coordinates while preserving gene and transcript
+structure.
+
 sanitize_blast_db.py
 ~~~~~~~~~~~~~~~~~~~~