Argparse update, parse arguments in order, and prepare for distribution as a python package

[MelvinFrederiks]

Argparse

Move from docopt to argparse. This allows more flexibility in how arguments are parsed, it adds nice error messages and we don't have to manually construct a config data structure anymore.

Arguments in order

Almost all of the modules which can be passed as command-line options are now sensitive to ordering, this allows more control over the code execution.

Improved structure

For this improved argument parsing, I've modularized parts of the code and separated the script into multiple distinct parts. It should be easier to add new modules/functionality in the future.

A Python package

Demeuk now functions more as a python package, with a standardized directory structure. I've incorporated pyproject.toml for distribution with PDM. Because of this, this version cannot be run standalone from a script but needs to be run as a module, for example by using pdm run demeuk [args] or python -m demeuk.demeuk [args].

Minor things

• Dropped support for python 3.9 and below, and support 3.10 and up.
• Added validation of modules. This is not strictly necessary for function, however this can help to catch errors early when developing new modules.

[zyronix]

I went through most of the code, generally speaking it works and implements exactly what is required. However, as we already discussed in person; it feel like it make a lot of sense to introduce proper interfaces and object oriented work flow.

I would say to do this refactoring now, because the current implementation contains too many code smells. A great example of this is the validate.py file and the demeuk.py main loop.

So to move forward, refactor the code so that it used modules with well defined interfaces. The main issue regarding the pipelining clean_up currently does three separate jobs in one loop body:

• deciding how to call a function (FLAG vs PARAM unpacking)
• deciding how to interpret the result (CHECK/MODIFY/ADD/REMOVE dispatch),
• and running a separate hardcoded fixed ordered pipeline (if args.tab, if args.encode, etc.) before the dynamic one.

The first goal is for clean_up to simply iterate a flat list of Module objects and call module.run(line) or something using a polymorphic wrapper classes. This also means the fixed order and argument based ordered pipelines should be unified into a single pipeline. This probably will required couple of modules.

The second goal should be that a developer can define a module something like:

class CheckEmail(CheckModule):
    cli_option = '--check-email'
    help_text  = 'Drop lines containing e-mail addresses.'

    def __call__(self, line: str) -> CheckResult:
        if search(EMAIL_REGEX, line):
            return Result(passed=False, reason='Check_email; found email address')
        return Result(passed=True)

Basically be able to add one file that will then be auto discovered without the need to add it everywhere. Maybe even for auto generating docs?

Thirdly the modules should be callable without initing the whole project. Aka, I want to use demeuk in my python project to cleanup bogus lines.

Something like:

from demeuk.modules.check import CheckEmail

is_clean  = CheckEmail()("user@example.com")   # False

And maybe as string operation in my project:

pipeline = demeuk.Pipeline(CheckMinLength, ModifyHex)
demeuk.clean(bytes, pipeline)

I would suggest to start designing this more from the ground up and slowly migrating features. If anything does not fit the new design, feel free to drop or refactor. My suggested concrete steps:

1. First build a PoC of the module interfaces, get some feedback.
2. Extend this by implementing the Pipeline class
3. Build a new main clean function
4. Build all the wrappers to build it as a CLI too, separate this from the rest

Besides, introduce a Config object that store stuff like: global_store_punctuation, global_store_input_encoding, global_store_delims, global_store_cut_fields, log_verbose

Regarding usage, end users should be able to still call pip install demeuk which install demeuk into what ever environment they are using and just be able to call it directly using demeuk -i etc. Usage of demeuk should no rely on pdm.

[zyronix]

I would like to suggest that we are going to use pipx, which install the application into an isolated environment and always the suer to call 'demeuk' as a CLI tool.

[zyronix]

Maye call this "development"

[zyronix]

Bump to 5.0, is this version also inside of the source code? Is there a way to maybe link this version with the version in the source code so that you only have to bump it at one place?

[zyronix]

supports

[zyronix]

Add stderr import, the is now done implicitly through utils.

[zyronix]

Describe with comments why this is needed

[zyronix]

When adding a fixed order module you now need to add it at two places.

I think it make sense to add the fixed order module to the pipeline in order that they are defined so this part of the code can be reduced. Plus it makes it simpler to add a new module.

[zyronix]

Describe better with comments what the code does. What is option_type? And doesn't it make more sense to create a better interface for those modules? So you don't need to check if you need to specific a param?

[zyronix]

multiple locations for version

[zyronix]

This parts looks overly complex. I would suggest having a handler table to solve this:

Something like:

def handle_check(rest):
    msg = rest[0]
    debug_log(msg, line_decoded)
    return True  # stop


def handle_modify_remove(rest):
    line_decoded_local, msg = rest
    debug_log(msg, line_decoded_local)
    return False


def handle_add(rest):
    result, msg = rest
    results = result if isinstance(result, list) else [result]

    for new_line in results:
        debug_log(msg, new_line)
        work_queue.append(new_line.encode())

    return False

handlers = {
    ModuleType.CHECK: handle_check,
    ModuleType.MODIFY: handle_modify_remove,
    ModuleType.REMOVE: handle_modify_remove,
    ModuleType.ADD: handle_add,
}

handler = handlers.get(module_type)
if handler is None:
    raise ValueError(f"Unhandled module type: {module_type}")

stop = handler(rest)