Redesign to object-oriented structure

.github/workflows/test.yml

@@ -9,46 +9,31 @@ on:
 jobs:
   test:
     runs-on: ubuntu-latest
-    strategy:
-      matrix:
-        python-version: [3.8, 3.9, "3.10"]
     steps:
     - uses: actions/checkout@v3
     - name: Set up Python
-      uses: actions/setup-python@v3
-      with:
-        python-version: ${{ matrix.python-version }}
+      uses: pdm-project/setup-pdm@v4
     - name: Install dependencies
       run: |
+        set +x
         python -m pip install --upgrade pip
-        pip install tox
-        if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
+        pdm install -G test
     - name: Run tests
       run: |
-        tox
+        pdm run test
   publish:
     runs-on: ubuntu-latest
     needs: test
+    # Only publish on master
+    if: github.ref == 'refs/heads/master'
+    permissions:
+      id-token: write
     steps:
       - uses: actions/checkout@v3
-      - name: Set up Python
-        uses: actions/setup-python@v3
-        with:
-          python-version: "3.10"
-      - name: Install publishing dependencies
-        if: github.ref == 'refs/heads/master'
-        run: |
-          python -m pip install --upgrade pip
-          pip install setuptools wheel twine
-      - name: Install package dependencies
-        if: github.ref == 'refs/heads/master'
-        run: |
-          python -m pip install -r requirements.txt
-      - name: Build and publish
-        if: github.ref == 'refs/heads/master'
+      - name: Set up PDM
+        uses: pdm-project/setup-pdm@v4
+      - name: Publish wheel
         env:
-          TWINE_USERNAME: __token__
-          TWINE_PASSWORD: ${{ secrets.PYPI_API_TOKEN }}
-        run: |
-          python setup.py sdist bdist_wheel
-          twine upload dist/*
+          PDM_PUBLISH_USERNAME: __token__
+          PDM_PUBLISH_PASSWORD: ${{ secrets.PYPI_API_TOKEN }}
+        run: pdm publish

README.md

@@ -24,28 +24,40 @@ grant agreement No. 82201
 Please read the docs for more information.
 
 ## Quick start
-The recommended way to install demeuk is to install it in a virtual
-environment.
+Demeuk supports Python versions 3.12 and up.
+The recommended way to install demeuk is to use [pipx](https://pipx.pypa.io/stable/).
 
 ```
-# Create virtual environment
-virtualenv <virtual environment name>
-# Activate the virtual environment
-source <virtual environment name>/bin/activate
-pip3 install -r requirements.txt
+pipx install demeuk --python /usr/bin/python3.14
 ```
 
-Now you can run bin/demeuk.py:
+Now you can invoke demeuk directly from the command-line from any directory:
 
 Examples:
 ```
-    demeuk -i inputfile.tmp -o outputfile.dict -l droppedfile.txt
-    demeuk -i inputfile -o outputfile -j 24 -l logfile.log
-    demeuk -i inputfile.tmp -o outputfile.dict -l droppedfile.txt --leak
-    demeuk -i inputfile -o outputfile -j 24 -l logfile.log --leak-full
-    demeuk -i inputdir/*.txt -o outputfile.dict -l logfile.log
-    demeuk -o outputfile.dict -l logfile.log
+    demeuk -i inputfile.tmp -o outputfile.dict -l demeuk.log
+    demeuk -i inputfile -o outputfile -j 24 -l demeuk.log
+    demeuk -i inputfile.tmp -o outputfile.dict -l demeuk.log --leak
+    demeuk -i inputfile -o outputfile -j all -l demeuk.log --leak-full
+    demeuk -i inputdir/*.txt -o outputfile.dict -l demeuk.log
 ```
+Demeuk also works with pipes:
+```
+    cat wordlist | demeuk --leak-full --debug > list.out 2> list.log
+```
+
+## Development
+To make changes to demeuk, you need to run it from the source Python files.
+```
+git clone https://github.com/NetherlandsForensicInstitute/demeuk.git
+cd demeuk
+pipx install ./ --python /usr/bin/python3.14
+```
+Now you can run demeuk as in the examples. Note that the shortcut `demeuk` only updates after you
+run `pipx upgrade demeuk`.
+
+You can install demeuk through PDM with `pdm install -p ./`, this circumvents this issue although you have to
+run demeuk with `pdm run demeuk`.
 
 ## Docs
 The docs are available at: <http://demeuk.rtfd.io/>

docs/api_ref.rst

@@ -3,9 +3,53 @@ API Reference
 
 This chapter is for developers of demeuk, it contains the API functions.
 
-Demeuk-api
-==========
-.. automodule:: bin.demeuk
+demeuk
+------
+.. automodule:: demeuk.demeuk
    :members:
    :undoc-members:
    :show-inheritance:
+
+
+modules.base
+------------
+.. automodule:: demeuk.modules.base
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+modules.add
+-----------
+.. automodule:: demeuk.modules.add.add
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+modules.check
+-------------
+.. automodule:: demeuk.modules.check.check
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+modules.modify
+--------------
+.. automodule:: demeuk.modules.modify.modify
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+modules.remove
+--------------
+.. automodule:: demeuk.modules.remove.remove
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+modules.macro
+--------------
+.. automodule:: demeuk.modules.macro.macro
+   :members:
+   :undoc-members:
+   :show-inheritance:
+

docs/conf.py

@@ -13,19 +13,19 @@
 import os
 import sys
 
-sys.path.insert(0, os.path.abspath('..'))
+sys.path.insert(0, os.path.abspath('../src'))
 source_suffix = ['.rst', '.md']
 
-from bin.demeuk import version  # noqa: E402
+from demeuk.demeuk import get_version
 
 # -- Project information -----------------------------------------------------
 
 project = 'demeuk'
-copyright = '2019 - 2021, NFI'
+copyright = '2019 - 2026, NFI'
 author = 'Netherlands Forensic Institute (NFI)'
 
 # The full version, including alpha/beta/rc tags
-release = version
+release = get_version()
 
 
 # -- General configuration ---------------------------------------------------
@@ -47,6 +47,8 @@
 # This pattern also affects html_static_path and html_extra_path.
 exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']
 
+# Use the same ordering as the source file for members.
+autodoc_member_order = 'bysource'
 
 # -- Options for HTML output -------------------------------------------------
 

docs/design.rst

@@ -6,50 +6,107 @@ some insight on how the application works. Mostly it is useful in case you
 are working with a bug or don't understand why something is happening and it
 is a must read for anyone adding features to demeuk.
 
+.. _modules:
+
+Modules
+-------
+Demeuk is a modular program; you can enable functionality by specifying their respective command-line options.
+Demeuk supports four different type of modules.
+
+- Modify or Clean modules. Those modules modify something in a line. For example replace tab
+  character with ':'. The commandline parameters will have the name of the module
+  without a prefix.
+- Add modules. Those modules will modify something in a line, but keep the original
+  line as well. For example, add a lower case variant of a line. These modules will
+  have the commandline parameters start with 'add-' prefix.
+- Check modules. Those modules will check if a line passes some test. For example
+  a minimal length check. The commandline parameters start with the 'check-' prefix.
+  If a line fails the check, the line is dropped.
+- Remove modules. Those modules will remove specific parts of a line and does this
+  in place. For example punctuation needs to be removed, those modules will be used.
+  The commandline parameters will start with the 'remove-' prefix.
+
+Note that when any add option is used, any other modules (like clean, check, remove
+AND even add) will be ran on the modified line again. This might result in creating
+an loop if it keeps creating new lines. So be careful with using those options.
+
+
+There is another class of modules called macro modules. These are modules which invoke a number of
+other modules, and may provide other functionality themselves. Examples of macro modules are
+``leak`` and ``leak-full``, which are simply a sensible collection of other modules to clean up data leaks.
+
+
+.. _pipeline:
+
+Pipeline
+--------
+Demeuk runs its input through a pipeline, which is a linear sequence of modules. The action
+of a module on types determines its place in the pipeline.
+
+Input is read as bytes, at some point it is *encoded*, at which point these bytes are treated as a
+Python string. At this point we can work with the input as strings of characters. Keeping this in
+mind, a module can fall in three classes:
+
+* A module which takes bytes as input and returns a byte sequence,
+* A module which takes bytes as input and returns a string (an *encoding* module),
+* A module which takes a string as input and returns a string.
+
+A module in the pipeline will return a ``Results`` object, containing a ``status`` field. If this
+is set to ``False``, nothing happens and the line is passed to the next module in line. If however
+this field is ``True``, the ``Results`` object is passed to ``handle()`` which, depending on the
+module type, determines what concrete actions to take. These actions could be: Stop further
+processing of the line, add a variant, log a debug message, etc. This is encoded in and ``Actions``
+object which is processed by ``Pipeline.run()``.
+
 Threading
 ---------
 In cause of an input file, the file is 'chunked' by the main processes. It will split
-the input files in chunks. It does so by reading the file per 1 KB. After reading 1 KB
-it will search for the next newline after the 1 KB. It will then read again 1 KB and 
+the input files in chunks. It does so by reading the file per 1 MB. After reading 1 MB
+it will search for the next newline after the 1 MB. It will then read again 1 MB and
 search for the first new line after that. This will continue until the end of the file.
 
-The size of 1 KB is used to reduce memory load and was found to be a solid number for
+The size of 1 MB is used to reduce memory load and was found to be a solid number for
 good performance.
 
 When using stdin, the input is not chunked. This is because stdin is a stream and
 thus we can not seek to a specific offset. So the main thread will read the input
-per 1 KB and search for the first newline after that.
+per 1 MB and search for the first newline after that.
 
-Searching for the next newline is done using the python's splitlines() 
+Searching for the next newline is done using the python's splitlines()
 function. This means the line will be splitted on: line feed, carriage return,
 LF + CR, formfeeds, file separator, etc. See https://docs.python.org/3/library/stdtypes.html
 for more information.
 
 Next a thread will process the list of lines.
 
-Once all threads are done, the main thread will combine all of the results. 
+Once all threads are done, the main thread will combine all of the results.
 You should note that the order inside the final output will be completely
 un ordered and thus if you want to have a sorted list you need to sort it yourself.
 
+A note on the add modules and threading: Lines are dedicated to different
+threads based on a configured chunk size. When additional lines are added, all
+other modules will run again on the line. The thread that created the new line
+will also run those modules again. Meaning that if one thread creates a lot of
+different new lines that thread might be busier then other threads. But because
+the chunksize is quite small, this will probably not be an issue. If this is an
+issue for someone please submit a bug.
+
 Encoding detection
 ------------------
-Next, when '--tab' is enabled all tabs will be converted to ':' greedy. This is to have
-a single cut/splitting char. This is done on binary level.
-
-Next, we arrive at one of the most important things of this application. The encoding detecting
-enable this with '--encode'. Some dataset are a combination of different sources. This means
+One of the most important things of this application is the encoding detection,
+enabled with ``--encode``. Some datasets are a combination of different sources. This means
 EVERY line can have a different encoding. People or applications tend to make a lot
 of errors in encoding, as does this application. Demeuk tries its best to detect
 and correct as much as possible, but there will for sure be some weird case where it fails
 to do so. By default the application will try to decode the data using UTF-8.
 
 So we start by checking if we have a default encoding to try. This is either
-UTF-8 or supplied by the user. If the line decodes and there does not appear to be
-control character inside the line we can assume that the detection went correctly.
-Note: If you supply a list of input encodings. Put multibyte encodings first.
-Because single byte encodings will cause false positives.
+UTF-8 or supplied by the user with ``--input-encoding``. If the line decodes and there does not
+appear to be control characters inside the line we can assume that the detection went correctly.
+Note: If you supply a list of input encodings. Put multibyte encodings first, because single byte
+encodings will cause false positives.
 
-If that fails we run the detect function of the chardet library. Note: first the 
+If that fails we run the detect function of the chardet library. Note: first the
 cchardet library was implemented, but this library resulted in too many wrongly
 encoded lines. Inside the tests of demeuk there are lot of edge cases which were
 found and corrected. So if you change something in the encoding detection
@@ -58,44 +115,14 @@ please run the tests to verify that you have not broken something.
 If it managed detect any encoding, it will try to decode this line. If no unicode
 error happens we assume that we got some result.
 
-Next we try to fix mojibakes, for this enable the --mojibake option
-basically, we might have decoded the string incorrectly
-and now correct some of the common errors. For this we use the FTFY library.
-
-Modules
--------
-After a line has been decoded correctly demeuk will start to run all the modules.
-Demeuk consist of 4 different type of modules.
-
-- Clean modules. Those modules modify something in a line. For example replace tab
-  character with ':'. The commandline parameters will have the name of the module 
-  without a prefix.
-- Add modules. Those modules will modify something in a line, but keep the original
-  line aswell. For example, add a lower case variant of a line. These modules will
-  have the commandline parameters start with 'add-' prefix.
-- Check modules. Those modules will check if a line passes some test. For example
-  a minimal length check. The commandline parameters start with the 'check-' prefix.
-  If a line fails the check, the line is dropped.
-- Remove modules. Those modules will remove specific parts of a line and does this
-  in place. For example punctuation needs to be removed, those modules will be used.
-  The commandline parameters will start with the 'remove-' prefix.
-
-The name that a module has on the commandline will mean that the function inside the
-source code must also has the exact same name. Only clean module will start with the
-'clean\_' prefix to prevent name clashes with default functions.
-
-Note that when any add option is used, any other modules (like clean, check, remove
-AND even add) will be ran on the modified line again. This might result in creating
-an loop if it keeps creating new lines. So be careful with using those options.
+You can enable the ``--mojibake`` option to let demeuk try to fix mojibakes, which are
+artifacts of wrong decoding. For this we use the FTFY library.
 
-For now there is no specific order in which the module type will run. Apart from
-the add modules, which will always run last. If someone find a specific use case
-for which the order needs to be configured; please submit a bug.
 
-Another note on the add modules and threading. Lines are dedicated to different
-threads based on a configured chunk size. When additional lines are added, all
-other modules will run again on the line. The thread that created the new line
-will also run those modules again. Meaning that if one thread creates a lot of
-different new lines that thread might be busier then other threads. But because
-the chunksize is quite small, this will probably not be an issue. If this is an
-issue for someone please submit a bug.
+Module ordering
+---------------
+After successfully figuring out the decoding of the line, there are many different modules which
+can be run. All modules which function after the encoding step will be run in the order in which
+they are supplied to the program. This enables great flexibility, but it also means you need to
+think about this order. If you don't know where to start, put modify modules first, then check
+modules, remove modules and finally add modules.

docs/index.rst

@@ -16,6 +16,7 @@ Table of content
     :maxdepth: 3
 
     install
-    usage
     design
+    new_module
+    usage
     api_ref