Merge pull request #612 from deeptools/develop

Develop

Author: joachimwolff

docs/content/News.rst

@@ -1,6 +1,29 @@
 News and Developments
 =====================
 
+
+Release 3.5.2
+-------------
+**06 October 2020**
+
+- Bug fix release: 
+   - enforcing version 15 of HiCMatrix. Version 14 had a bug concerning the application of the correction factors of cool files. See issue #595
+   - Fixing a bug in hicDetectLoops in single-core mode. Thanks @alecw
+   - Fixing a bug in hicDifferentialTAD concerning multiple chromosomes in the bed file. See issue #587
+   - Updating dependencies to newest versions, except biopython. Forcing here <1.77 because their API change is breaking our source code. See issue #609
+   - Fixing #596
+- Changing internal structure of the docs. Navigation should be better now.
+
+
+Release 3.5.1
+--------------
+**11 August 2020**
+
+- patch hicCorrelate
+- hicBuildMatrix: Better help text
+- patch for hicPlotMatrix if matrix does not start at 0 and --region is used
+- bug fix for remove option in hicAdjustMatrix
+
 Release 3.5
 -----------
 **10 July 2020**

docs/content/installation.rst

@@ -7,27 +7,31 @@ Installation
 Requirements
 -------------
 
-* Python 3.6
-* numpy >= 1.17
-* scipy >= 1.3
-* matplotlib == 3.1
-* pysam >= 0.15
-* intervaltree >= 3.0
-* biopython >= 1.74
-* pytables >= 3.5
-* pyBigWig >= 0.3
-* future >= 0.17
-* jinja2 >= 2.10
-* pandas >= 0.25
-* unidecode >= 1.1
-* hicmatrix = 11
-* pygenometracks >= 3.0
-* psutil >= 5.6
-* hic2cool >= 0.7
-* cooler >= 0.8.5
-* krbalancing >= 0.0.5 (Needs the library eigen; openmp is recommended for linux users. No openmp support on macOS.)
+* python >= 3.6
+* numpy >= 1.19.*
+* scipy >= 1.5.*
+* matplotlib-base >= 3.1.*
+* ipykernel >= 5.3.0
+* pysam >= 0.16.*
+* intervaltree >= 3.1.*
+* biopython < 1.77
+* pytables >= 3.6.*
+* pandas >= 1.1.*
+* pybigwig >= 0.3.*
+* jinja2 >= 2.11
+* unidecode >= 1.1.*
+* hicmatrix >= 15
+* hic2cool >= 0.8.3
+* psutil >= 5.7.*
+* pygenometracks >= 3.5
 * fit_nbinom >= 1.1
-* pybedtools >= 0.8
+* cooler >= 0.8.10
+* krbalancing >= 0.0.5 (Needs the library eigen; openmp is recommended for linux users. No openmp support on macOS.)
+* pybedtools >= 0.8.*
+* future >= 0.18
+* tqdm >= 4.50
+* hyperopt >= 0.2.4
+* python-graphviz >= 0.14
 
 
 **Warning:** Python 2.7 support is discontinued. Moreover, the support for pip is discontinued too. 
@@ -36,7 +40,7 @@ Requirements
 Command line installation using ``conda``
 -----------------------------------------
 
-The fastet way to obtain **Python 3.6 together with numpy and scipy** is
+The fastet way to obtain **Python 3.6 or 3.7 together with numpy and scipy** is
 via the `Anaconda Scientific Python
 Distribution <https://store.continuum.io/cshop/anaconda/>`_.
 Just download the version that's suitable for your operating system and

docs/content/list-of-tools.rst

@@ -1,6 +1,88 @@
 HiCExplorer tools
 =================
 
+Tools for Hi-C data pre-processing
+----------------------------------
+.. toctree::
+    :maxdepth: 1
+
+    tools/hicFindRestSite
+    tools/hicBuildMatrix
+    tools/hicSumMatrices
+    tools/hicMergeMatrixBins
+    tools/hicCorrectMatrix
+    tools/hicNormalize
+
+Tools for Hi-C QC
+-----------------
+.. toctree::
+    :maxdepth: 1  
+
+    tools/hicQuickQC
+    tools/hicQC
+    tools/hicCorrelate
+    tools/hicPlotDistVsCounts
+    tools/hicInfo
+
+Tools for Hi-C data analysis
+----------------------------
+.. toctree::
+    :maxdepth: 1  
+
+    tools/hicCompareMatrices
+    tools/hicPCA
+    tools/hicTransform
+    tools/hicAverageRegions
+    tools/hicDetectLoops
+    tools/hicValidateLocations
+    tools/hicMergeLoops
+    tools/hicHyperoptDetectLoops
+    tools/hicHyperoptDetectLoopsHiCCUPS
+    tools/hicCompartmentalization
+    tools/hicPlotSVL
+
+Tools for TADs processing
+-------------------------
+.. toctree::
+    :maxdepth: 1  
+
+    tools/hicFindTADs
+    tools/hicMergeDomains
+    tools/hicDifferentialTAD
+    tools/hicMergeTADbins
+
+Tools for Hi-C and TADs visualization
+-------------------------------------
+.. toctree::
+    :maxdepth: 1  
+
+    tools/hicPlotMatrix
+    tools/hicPlotTADs
+    tools/hicPlotViewpoint
+    tools/hicAggregateContacts
+    tools/hicPlotAverageRegions
+
+Hi-C contact matrix handling
+----------------------------
+.. toctree::
+    :maxdepth: 1  
+
+    tools/hicConvertFormat
+    tools/hicAdjustMatrix
+
+Capture Hi-C analysis
+---------------------
+.. toctree::
+    :maxdepth: 1  
+
+    tools/chicQualityControl
+    tools/chicViewpointBackgroundModel
+    tools/chicViewpoint
+    tools/chicSignificantInteractions
+    tools/chicAggregateStatistic
+    tools/chicDifferentialTest
+    tools/chicPlotViewpoint
+
 .. contents::
     :local:
 
@@ -77,7 +159,7 @@ HiCExplorer tools
 +--------------------------------------+------------------+-----------------------------------+---------------------------------------------+-----------------------------------------------------------------------------------+
 |:ref:`hicPlotAverageRegions`          | visualization    | one npz file                      | one image                                   | Visualization of hicAverageRegions.                                               |
 +--------------------------------------+------------------+-----------------------------------+---------------------------------------------+-----------------------------------------------------------------------------------+
-|:ref`hicPlotSVL`                      | analysis         | one / multiple Hi-C matrices      | one image, p-values file, raw data file     | Computes short/long range contacts; a box plot, a p-value and raw data file       |
+|:ref:`hicPlotSVL`                     | analysis         | one / multiple Hi-C matrices      | one image, p-values file, raw data file     | Computes short/long range contacts; a box plot, a p-value and raw data file       |
 +--------------------------------------+------------------+-----------------------------------+---------------------------------------------+-----------------------------------------------------------------------------------+
 |:ref:`hicMergeTADbins`                | preprocessing    | one Hi-C matrix, one BED file     | one Hi-C matrix                             | Uses a BED file of domains or TAD boundaries to merge the                         |
 |                                      |                  |                                   |                                             | bin counts of a Hi-C matrix.                                                      |
@@ -128,114 +210,3 @@ You can always see all available command-line options via --help:
 - Output format of plots should be indicated by the file ending, e.g. ``MyPlot.pdf`` will return a pdf file, ``MyPlot.png`` a png-file.
 - Most of the tools that produce plots can also output the underlying data - this can be useful in cases where you don't like the HiCExplorer visualization, as you can then use the data matrices produced by deepTools with your favorite plotting tool, such as R.
 - The vast majority of command line options are also available in Galaxy (in a few cases with minor changes to their naming).
-
-
-Tools for Hi-C data pre-processing
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-:ref:`hicFindRestSite`
-""""""""""""""""""""""
-:ref:`hicBuildMatrix`
-"""""""""""""""""""""
-:ref:`hicSumMatrices`
-"""""""""""""""""""""
-:ref:`hicMergeMatrixBins`
-"""""""""""""""""""""""""
-:ref:`hicCorrectMatrix`
-"""""""""""""""""""""""
-:ref:`hicNormalize`
-"""""""""""""""""""
-
-Tools for Hi-C QC
-^^^^^^^^^^^^^^^^^
-
-:ref:`hicQuickQC`
-"""""""""""""""""
-:ref:`hicQC`
-""""""""""""
-:ref:`hicCorrelate`
-"""""""""""""""""""
-:ref:`hicPlotDistVsCounts`
-""""""""""""""""""""""""""
-:ref:`hicInfo`
-""""""""""""""
-
-Tools for Hi-C data analysis
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-:ref:`hicCompareMatrices`
-"""""""""""""""""""""""""
-:ref:`hicPCA`
-"""""""""""""
-:ref:`hicTransform`
-"""""""""""""""""""
-:ref:`hicAverageRegions`
-""""""""""""""""""""""""
-:ref:`hicDetectLoops`
-""""""""""""""""""""""""
-:ref:`hicValidateLocations`
-"""""""""""""""""""""""""""
-:ref:`hicMergeLoops`
-""""""""""""""""""""
-:ref:`hicHyperoptDetectLoops`
-"""""""""""""""""""""""""""""
-:ref:`hicHyperoptDetectLoopsHiCCUPS`
-""""""""""""""""""""""""""""""""""""
-:ref:`hicCompartmentalization`
-""""""""""""""""""""""""""""""
-:ref:`hicPlotSVL`
-"""""""""""""""""
-
-Tools for TADs processing
-^^^^^^^^^^^^^^^^^^^^^^^^^
-
-:ref:`hicFindTADs`
-""""""""""""""""""
-:ref:`hicMergeDomains`
-""""""""""""""""""""""
-:ref:`hicDifferentialTAD`
-"""""""""""""""""""""""""
-:ref:`hicMergeTADbins`
-""""""""""""""""""""""
-
-Tools for Hi-C and TADs visualization
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-:ref:`hicPlotMatrix`
-""""""""""""""""""""
-:ref:`hicPlotTADs`
-""""""""""""""""""
-:ref:`hicPlotViewpoint`
-"""""""""""""""""""""""
-:ref:`hicAggregateContacts`
-"""""""""""""""""""""""""""
-:ref:`hicPlotAverageRegions`
-""""""""""""""""""""""""""""
-
-
-Hi-C contact matrix handling
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-:ref:`hicConvertFormat`
-"""""""""""""""""""""""
-:ref:`hicAdjustMatrix`
-""""""""""""""""""""""
-
-
-Capture Hi-C analysis
-^^^^^^^^^^^^^^^^^^^^^
-
-:ref:`chicQualityControl`
-"""""""""""""""""""""""""
-:ref:`chicViewpointBackgroundModel`
-"""""""""""""""""""""""""""""""""""
-:ref:`chicViewpoint`
-""""""""""""""""""""
-:ref:`chicSignificantInteractions`
-""""""""""""""""""""""""""""""""""
-:ref:`chicAggregateStatistic`
-"""""""""""""""""""""""""""""
-:ref:`chicDifferentialTest`
-"""""""""""""""""""""""""""
-:ref:`chicPlotViewpoint`
-""""""""""""""""""""""""

hicexplorer/_version.py

@@ -2,4 +2,4 @@
 # This file is originally generated from Git information by running 'setup.py
 # version'. Distribution tarballs contain a pre-generated copy of this file.
 
-__version__ = '3.5.1'
+__version__ = '3.5.2'

hicexplorer/hicCorrectMatrix.py

@@ -511,7 +511,8 @@ def plot_histogram(row_sum_values, mad_values, ax1, title=None):
 
             # high remove outliers
             row_sum = row_sum[modified_z_score < 5]
-
+            if len(row_sum) == 0:
+                log.error('No data for chromosome {}. Continue with next chromosome.'.format(chrname))
             col = plot_num % num_cols
             row = plot_num // num_cols
             ax[chrname] = fig.add_subplot(grids[row, col])
@@ -527,6 +528,9 @@ def plot_histogram(row_sum_values, mad_values, ax1, title=None):
 
         # high remove outliers
         row_sum = row_sum[modified_z_score < 5]
+        if len(row_sum) == 0:
+            log.error('No data available, exit.')
+            exit(1)
         ax = fig.add_subplot(111)
         plot_histogram(row_sum, mad, ax)
 

hicexplorer/hicDetectLoops.py

@@ -997,6 +997,8 @@ def main(args=None):
     else:
         single_core = False
 
+    fail_flag = False
+    fail_message = ''
     if single_core:
         for chromosome in chromosomes_list:
             if is_cooler:
@@ -1022,8 +1024,6 @@ def main(args=None):
         all_threads_done = False
         thread_done = [False] * args.threads
         count_call_of_read_input = 0
-        fail_flag = False
-        fail_message = ''
         while not all_data_processed or not all_threads_done:
             for i in range(args.threads):
                 if queue[i] is None and not all_data_processed: