recreate project as dirhash

Author: andhus

.coveragerc

@@ -0,0 +1,3 @@
+[run]
+branch = True
+source = dirhash

.gitignore

@@ -0,0 +1,111 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+.hypothesis/
+.pytest_cache/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# pyenv
+.python-version
+
+# celery beat schedule file
+celerybeat-schedule
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+
+# Pycharm
+.idea/
+
+
+# Project specific
+benchmark/test_cases/*

.travis.yml

@@ -0,0 +1,14 @@
+language: python
+dist: trusty
+python:
+  - "2.7"
+  - "3.6"
+install:
+  - pip install -e .
+  - pip install pytest-cov codecov
+  - pip freeze
+script:
+  - py.test --cov-config=.coveragerc --cov=dirhash tests/
+after_success:
+  - coverage report
+  - codecov

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2019 Anders Huss
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -0,0 +1,88 @@
+[![Build Status](https://travis-ci.com/andhus/dirhash.svg?branch=master)](https://travis-ci.com/andhus/dirhash)
+[![codecov](https://codecov.io/gh/andhus/dirhash/branch/master/graph/badge.svg)](https://codecov.io/gh/andhus/dirhash)
+
+# dirhash
+A lightweight python module and tool for computing the hash of any
+directory based on its files' structure and content.
+- Supports any hashing algorithm of Python's built-in `hashlib` module
+- `.gitignore` style "wildmatch" patterns for expressive filtering of files to 
+include/exclude.
+- Multiprocessing for up to [6x speed-up](#performance)
+
+## Installation
+```commandline
+git clone git@github.com:andhus/dirhash.git
+pip install dirhash/
+```
+
+## Usage
+Python module:
+```python
+from dirhash import dirhash
+
+dirpath = 'path/to/directory'
+dir_md5          = dirhash(dirpath, 'md5')
+filtered_sha1    = dirhash(dirpath, 'sha1', ignore=['.*', '.*/', '*.pyc'])
+pyfiles_sha3_512 = dirhash(dirpath, 'sha3_512', match=['*.py'])
+```
+CLI:
+```commandline
+dirhash path/to/directory -a md5
+dirhash path/to/directory -a sha1 -i ".*  .*/  *.pyc"
+dirhash path/to/directory -a sha3_512 -m "*.py"
+```
+
+## Why?
+If you (or your application) need to verify the integrity of a set of files as well
+as their name and location, you might find this useful. Use-cases range from 
+verification of your image classification dataset (before spending GPU-$$$ on 
+training your fancy Deep Learning model) to validation of generated files in
+regression-testing.
+
+There isn't really a standard way of doing this. There are plenty of recipes out 
+there (see e.g. these SO-questions for [linux](https://stackoverflow.com/questions/545387/linux-compute-a-single-hash-for-a-given-folder-contents)
+and [python](https://stackoverflow.com/questions/24937495/how-can-i-calculate-a-hash-for-a-filesystem-directory-using-python))
+but I couldn't find one that is properly tested (there are some gotcha:s to cover!) 
+and documented with a compelling user interface. `dirhash` was created with this as 
+the goal.
+
+[checksumdir](https://github.com/cakepietoast/checksumdir) is another python 
+module/tool with similar intent (that inspired this project) but it lacks much of the
+functionality offered here (most notably including file names/structure in the hash)
+and lacks tests.
+
+## Performance
+The python `hashlib` implementation of common hashing algorithms are highly
+optimised. `dirhash` mainly parses the file tree, pipes data to `hashlib` and 
+combines the output. Reasonable measures have been taken to minimize the overhead 
+and for common use-cases, the majority of time is spent reading data from disk 
+and executing `hashlib` code.
+
+The main effort to boost performance is support for multiprocessing, where the
+reading and hashing is parallelized over individual files.
+
+As a reference, let's compare the performance of the `dirhash` [CLI](https://github.com/andhus/dirhash/blob/master/dirhash/cli.py) 
+with the shell command:
+
+`find path/to/folder -type f -print0 | sort -z | xargs -0 md5 | md5` 
+
+which is the top answer for the SO-question: 
+[Linux: compute a single hash for a given folder & contents?](https://stackoverflow.com/questions/545387/linux-compute-a-single-hash-for-a-given-folder-contents)
+Results for two test cases are shown below. Both have 1 GiB of random data: in 
+"flat_1k_1MB", split into 1k files (1 MiB each) in a flat structure, and in 
+"nested_32k_32kB", into 32k files (32 KiB each) spread over the 256 leaf directories 
+in a binary tree of depth 8.
+
+Implementation      | Test Case       | Time (s) | Speed up
+------------------- | --------------- | -------: | -------:
+shell reference     | flat_1k_1MB     | 2.29     | -> 1.0
+`dirhash`           | flat_1k_1MB     | 1.67     | 1.36
+`dirhash`(8 workers)| flat_1k_1MB     | 0.48     | **4.73**
+shell reference     | nested_32k_32kB | 6.82     | -> 1.0
+`dirhash`           | nested_32k_32kB | 3.43     | 2.00
+`dirhash`(8 workers)| nested_32k_32kB | 1.14     | **6.00**
+
+The benchmark was run a MacBook Pro (2018), further details and source code [here](https://github.com/andhus/dirhash/tree/master/benchmark).
+
+## Documentation
+Please refer to `dirhash -h` and the python [source code](https://github.com/andhus/dirhash/blob/master/dirhash/__init__.py).

benchmark/README.md

@@ -0,0 +1,33 @@
+# Benchmark
+
+As a reference, the performance of `dirhash` is benchmarked against the shell command:
+
+`find path/to/folder -type f -print0 | sort -z | xargs -0 md5 | md5` 
+
+(top answer for the SO-question: 
+[Linux: compute a single hash for a given folder & contents?](https://stackoverflow.com/questions/545387/linux-compute-a-single-hash-for-a-given-folder-contents))
+
+Each test case contains 1 GiB of random data, split equally into 8, 1k or 32k files,
+in a flat or nested (binary tree of depth 8) structure.
+
+For a fair comparison, *the CLI version* of `dirhash` was used (including startup 
+time for loading of python modules etc.).
+
+For full details/reproducibility see/run the `run.py` script for which the output is 
+found in `results.csv`. These results were generated on a MacBook Pro (2018): 
+- 2,2 GHz Intel Core i7 (`sysctl -n hw.physicalcpu hw.logicalcpu`-> 6, 12)
+- 16 GB 2400 MHz DDR4
+- APPLE SSD AP0512M
+
+
+
+## Sample results:
+
+Implementation      | Test Case       | Time (s) | Speed up
+------------------- | --------------- | -------: | -------:
+shell reference     | flat_1k_1MB     | 2.29     | -> 1.0
+`dirhash`           | flat_1k_1MB     | 1.67     | 1.36
+`dirhash`(8 workers)| flat_1k_1MB     | 0.48     | **4.73**
+shell reference     | nested_32k_32kB | 6.82     | -> 1.0
+`dirhash`           | nested_32k_32kB | 3.43     | 2.00
+`dirhash`(8 workers)| nested_32k_32kB | 1.14     | **6.00**

benchmark/results.csv

@@ -0,0 +1,51 @@
+,test_case,implementation,algorithm,workers,t_best,t_median,speed-up (median)
+0,flat_8_128MB,shell reference,md5,1,2.014,2.02,1.0
+1,flat_8_128MB,dirhash,md5,1,1.602,1.604,1.2593516209476308
+2,flat_8_128MB,dirhash,md5,2,0.977,0.98,2.061224489795918
+3,flat_8_128MB,dirhash,md5,4,0.562,0.569,3.5500878734622145
+4,flat_8_128MB,dirhash,md5,8,0.464,0.473,4.2706131078224105
+5,flat_1k_1MB,shell reference,md5,1,2.263,2.268,1.0
+6,flat_1k_1MB,dirhash,md5,1,1.662,1.667,1.3605278944211157
+7,flat_1k_1MB,dirhash,md5,2,0.978,0.983,2.3072227873855544
+8,flat_1k_1MB,dirhash,md5,4,0.57,0.58,3.910344827586207
+9,flat_1k_1MB,dirhash,md5,8,0.476,0.48,4.725
+10,flat_32k_32kB,shell reference,md5,1,6.711,6.721,1.0
+11,flat_32k_32kB,dirhash,md5,1,3.329,3.354,2.003875968992248
+12,flat_32k_32kB,dirhash,md5,2,2.067,2.074,3.240597878495661
+13,flat_32k_32kB,dirhash,md5,4,1.345,1.362,4.934654919236417
+14,flat_32k_32kB,dirhash,md5,8,1.09,1.094,6.143510054844606
+15,nested_1k_1MB,shell reference,md5,1,2.296,2.306,1.0
+16,nested_1k_1MB,dirhash,md5,1,1.713,1.714,1.3453908984830805
+17,nested_1k_1MB,dirhash,md5,2,0.996,1.009,2.285431119920714
+18,nested_1k_1MB,dirhash,md5,4,0.601,0.602,3.8305647840531565
+19,nested_1k_1MB,dirhash,md5,8,0.499,0.505,4.566336633663366
+20,nested_32k_32kB,shell reference,md5,1,6.814,6.818,1.0
+21,nested_32k_32kB,dirhash,md5,1,3.376,3.426,1.9900758902510214
+22,nested_32k_32kB,dirhash,md5,2,2.147,2.153,3.166744078030655
+23,nested_32k_32kB,dirhash,md5,4,1.414,1.416,4.814971751412429
+24,nested_32k_32kB,dirhash,md5,8,1.137,1.138,5.991212653778559
+25,flat_8_128MB,shell reference,sha1,1,2.181,2.196,1.0
+26,flat_8_128MB,dirhash,sha1,1,1.214,1.225,1.7926530612244898
+27,flat_8_128MB,dirhash,sha1,2,0.768,0.774,2.8372093023255816
+28,flat_8_128MB,dirhash,sha1,4,0.467,0.474,4.632911392405064
+29,flat_8_128MB,dirhash,sha1,8,0.47,0.477,4.603773584905661
+30,flat_1k_1MB,shell reference,sha1,1,2.221,2.229,1.0
+31,flat_1k_1MB,dirhash,sha1,1,1.252,1.263,1.7648456057007127
+32,flat_1k_1MB,dirhash,sha1,2,0.774,0.777,2.8687258687258685
+33,flat_1k_1MB,dirhash,sha1,4,0.471,0.477,4.672955974842767
+34,flat_1k_1MB,dirhash,sha1,8,0.378,0.478,4.663179916317992
+35,flat_32k_32kB,shell reference,sha1,1,4.178,4.224,1.0
+36,flat_32k_32kB,dirhash,sha1,1,2.921,3.008,1.4042553191489362
+37,flat_32k_32kB,dirhash,sha1,2,1.888,1.892,2.232558139534884
+38,flat_32k_32kB,dirhash,sha1,4,1.266,1.275,3.3129411764705887
+39,flat_32k_32kB,dirhash,sha1,8,1.072,1.079,3.914735866543096
+40,nested_1k_1MB,shell reference,sha1,1,2.236,2.26,1.0
+41,nested_1k_1MB,dirhash,sha1,1,1.308,1.314,1.719939117199391
+42,nested_1k_1MB,dirhash,sha1,2,0.797,0.8,2.8249999999999997
+43,nested_1k_1MB,dirhash,sha1,4,0.501,0.509,4.4400785854616895
+44,nested_1k_1MB,dirhash,sha1,8,0.499,0.503,4.493041749502981
+45,nested_32k_32kB,shell reference,sha1,1,4.383,4.406,1.0
+46,nested_32k_32kB,dirhash,sha1,1,3.041,3.05,1.4445901639344263
+47,nested_32k_32kB,dirhash,sha1,2,1.943,1.965,2.242239185750636
+48,nested_32k_32kB,dirhash,sha1,4,1.329,1.334,3.3028485757121433
+49,nested_32k_32kB,dirhash,sha1,8,1.14,1.149,3.8346388163620535