Merge branch 'feature/directory-mode' into develop

Author: idle-code

CONTRIBUTING.md

@@ -119,4 +119,4 @@ The only difference between those modes is that in the case of name generation,
 
 Actual renaming depends on the mode which determines what filesystem operations will be used.
 For the **path** mode, `move` filesystem operation is used, while **name** mode, utilizes the `rename` operation.
-If `--dry-run`/`-d` flag has been specified, `DryRunRenamer` is selected and no renaming is performed. All other stages are unaffected by this flag.
+If `--dry-run`/`-dr` flag has been specified, `DryRunRenamer` is selected and no renaming is performed. All other stages are unaffected by this flag.

MANUAL.md

@@ -117,13 +117,22 @@ The output of this command will list all available tag names sorted by name and
 
 
 # Modes of operation
-`tempren` have two main modes of operation: **name** and **path**.
+`tempren` have three main modes of operation: **name**, **directory** and **path**.
 
 In the (default) **name** mode (represented by `--name`/`-n` flag), the template is used for filename generation only.
 This mode is used most often as it doesn't alter directory structure and focuses just on the files. In this mode, if a path separator (i.e. `/`) is present in the generated path, an error will be reported.
 
 Name mode is used mostly when operating on a single directory or files specified directly in the command-line arguments.
 
+Similarly, with **directory** mode selected (by `--directory`/`-d` flag), template generates new names for directories only - without changing filesystem hierarchy.
+
+> Note: In directory mode only contextual and directory-enabled tags are available
+<!--
+TODO: `--directory --list` flags combination (in that order) can be used to list supported tags.
+-->
+
+> Note: Currently sorting is not available in the directory mode
+
 In the **path** mode (enabled by `--path`/`-p` flag), for each processed file, the template generates a whole, new path (relative to the input directory).
 This way you can move (sort) files into dynamically generated catalogues.
 
@@ -340,7 +349,7 @@ To create an ad-hoc tag, you will need to provide an executable path as an argum
 tempren --ad-hoc awk --help awk
 ```
 
-> Note: When executing the ad-hoc tags, even with `--dry-run`/`-d` flag, `tempren` doesn't have control of the behaviour of the invoked executables.
+> Note: When executing the ad-hoc tags, even with `--dry-run`/`-dr` flag, `tempren` doesn't have control of the behaviour of the invoked executables.
 > Care should be taken to make sure that the user-provided program doesn't create any undesirable side effects upon template execution.
 
 There are two ways in which an executable associated with an ad-hoc tag can be invoked.
@@ -373,7 +382,7 @@ Program arguments can be passed in the ad-hoc tag arguments but care should be t
 separate them correctly as spaces do not delimit the arguments like in the shell environment.
 For example, to pass two parameters to the ad-hoc `Program`, the following syntax should be used:
 ```
-%Program("--flag-parameter", "positional")
+%Program("--flag", "flag-argument")
 ```
 
 # Tag aliases
@@ -387,7 +396,7 @@ Tags created from the aliases cannot receive any arguments or context - they are
 
 # Various options
 ## Dry run
-To facilitate discovery-based usage learning, `tempren`'s `--dry-run`/`-d` flag can be used to disable the actual file renaming stage of the pipeline. This way, users can test their templates without making changes to the filesystem.
+To facilitate discovery-based usage learning, `tempren`'s `--dry-run`/`-dr` flag can be used to disable the actual file renaming stage of the pipeline. This way, users can test their templates without making changes to the filesystem.
 > Note: While dry-run is active, side effects from filtering/sorting template expressions (which are valid Python code), ad-hoc tags
 > or even tags themselves may still affect the file system.\
 > Be careful not to copy-paste templates that look suspicious.

README.md

@@ -77,7 +77,7 @@ Renamed: 1211740803547.jpg
 <summary>Sorting files into directories based on their MIME type</summary>
 
 ```commandline
-$ tempren -d --path "%Capitalize(){%Mime(subtype)}/%Name()" ~/Downloads
+$ tempren -dr --path "%Capitalize(){%Mime(subtype)}/%Name()" ~/Downloads
 Renamed: dotnet-install.sh
      to: X-shellscript/dotnet-install.sh
 Renamed: openrgb_0.7_amd64_buster_6128731.deb

pyproject.toml

@@ -43,6 +43,7 @@ coverage = "^7.2.7"
 pre-commit = "^3.5"
 mypy = "^1.4.1"
 pylint = "^3.0.2"
+black = "^24.4.0"
 
 [tool.poetry.extras]
 video = ["pymediainfo"]

tempren/cli.py

@@ -18,6 +18,7 @@
 from tempren.template.exceptions import TagError, TemplateError
 
 from .pipeline import (
+    ConfigurationError,
     ConflictResolutionStrategy,
     FilterType,
     InvalidDestinationError,
@@ -134,6 +135,7 @@ def validate_adhoc_tags(
     unique_names = set(names)
     if len(names) > len(unique_names):
         repeating_names = [name for name in unique_names if names.count(name) > 1]
+        # TODO: Test
         for duplicate_name in repeating_names:
             executables_with_the_same_name = list(
                 map(
@@ -159,6 +161,8 @@ def validate_aliases(aliases: List[List[Tuple[TagName, str]]]) -> Dict[TagName,
     unique_names = set(names)
     if len(names) > len(unique_names):
         repeating_names = [name for name in unique_names if names.count(name) > 1]
+        # TODO: Test
+        # TODO: Refactor - the same code is found in validate_adhoc_tags
         for duplicate_name in repeating_names:
             patterns_with_the_same_name = list(
                 map(
@@ -175,7 +179,7 @@ def validate_aliases(aliases: List[List[Tuple[TagName, str]]]) -> Dict[TagName,
     return dict(list_of_tuples)
 
 
-class SystemExitError(Exception):
+class SystemExitError(Exception):  # TODO: Use ConfigurationError instead
     status: int
 
     def __init__(self, status: int, message: Optional[str]):
@@ -200,9 +204,10 @@ def __call__(
             category = registry.category_map[category_name]
 
             all_category_tags = sorted(category.tag_map.items())
-            if not all_category_tags:
+            if (
+                not all_category_tags
+            ):  # NOCOVER: Empty categories should not be present in the registry
                 # There is no tags in the category
-                # TODO: Empty categories should not be present in the registry
                 continue
             max_name_length = max(
                 [len(tag_name) for tag_name, factory in all_category_tags]
@@ -245,7 +250,8 @@ def __call__(
                 tag_factory = registry.get_tag_factory(qualified_name)
             except TagError as tag_error:
                 parser.exit(ErrorCode.USAGE_ERROR, str(tag_error))
-                raise
+                # noinspection PyUnreachableCode
+                raise  # NOCOVER: required by mypy
             log.info("")
             log.info(indent(tag_factory.configuration_signature, "  "))
             log.info("")
@@ -314,7 +320,6 @@ def _get_help_string(self, action):
         return action.help
 
 
-# CHECK: use pydantic-cli for argument parsing
 def process_cli_configuration(argv: List[str]) -> RuntimeConfiguration:
     log.debug("Parsing command line arguments")
     parser = argparse.ArgumentParser(
@@ -326,7 +331,7 @@ def process_cli_configuration(argv: List[str]) -> RuntimeConfiguration:
     )
 
     parser.add_argument(
-        "-d",
+        "-dr",
         "--dry-run",
         action="store_true",
         help="Do not perform any renaming - just show expected operation results",
@@ -385,7 +390,15 @@ def process_cli_configuration(argv: List[str]) -> RuntimeConfiguration:
         dest="mode",
         const=OperationMode.name,
         default=OperationMode.name,
-        help="Use template to generate file name",
+        help="Use template to generate file name only",
+    )
+    operation_mode.add_argument(
+        "-d",
+        "--directory",
+        action="store_const",
+        dest="mode",
+        const=OperationMode.directory,
+        help="Use template to generate directory names only",
     )
     operation_mode.add_argument(
         "-p",
@@ -396,34 +409,6 @@ def process_cli_configuration(argv: List[str]) -> RuntimeConfiguration:
         help="Use template to generate relative file path",
     )
 
-    conflict_resolution_group = parser.add_argument_group("conflict resolution")
-    conflict_resolution = conflict_resolution_group.add_mutually_exclusive_group()
-    conflict_resolution.add_argument(
-        "-cs",
-        "--conflict-stop",
-        action="store_true",
-        default=True,
-        help="Keep source file name intact and stop",
-    )
-    conflict_resolution.add_argument(
-        "-ci",
-        "--conflict-ignore",
-        action="store_true",
-        help="Keep source file name intact and continue",
-    )
-    conflict_resolution.add_argument(
-        "-co",
-        "--conflict-override",
-        action="store_true",
-        help="Override destination file",
-    )
-    conflict_resolution.add_argument(
-        "-cm",
-        "--conflict-manual",
-        action="store_true",
-        help="Prompt user to resolve conflict manually (choose an option or provide new filename)",
-    )
-
     filtering_group = parser.add_argument_group("filtering")
     filter_mode = filtering_group.add_mutually_exclusive_group()
     filter_mode.add_argument(
@@ -469,6 +454,34 @@ def process_cli_configuration(argv: List[str]) -> RuntimeConfiguration:
         help="Reverse sorting order",
     )
 
+    conflict_resolution_group = parser.add_argument_group("conflict resolution")
+    conflict_resolution = conflict_resolution_group.add_mutually_exclusive_group()
+    conflict_resolution.add_argument(
+        "-cs",
+        "--conflict-stop",
+        action="store_true",
+        default=True,
+        help="Keep source file name intact and stop",
+    )
+    conflict_resolution.add_argument(
+        "-ci",
+        "--conflict-ignore",
+        action="store_true",
+        help="Keep source file name intact and continue",
+    )
+    conflict_resolution.add_argument(
+        "-co",
+        "--conflict-override",
+        action="store_true",
+        help="Override destination file",
+    )
+    conflict_resolution.add_argument(
+        "-cm",
+        "--conflict-manual",
+        action="store_true",
+        help="Prompt user to resolve conflict manually (choose an option or provide new filename)",
+    )
+
     parser.add_argument(
         "template",
         type=nonempty_string,
@@ -650,6 +663,9 @@ def main() -> int:
         if exc.status != 0:
             log.error(exc)
         return exc.status
+    except ConfigurationError as exc:
+        log.error(exc)
+        return ErrorCode.USAGE_ERROR
     except TemplateEvaluationError as template_evaluation_error:
         render_template_evaluation_error(template_evaluation_error)
         return ErrorCode.TEMPLATE_EVALUATION_ERROR

tempren/file_sorters.py

@@ -43,3 +43,8 @@ def _generate_sort_key(self, file: File) -> Tuple:
                 evaluation_error.expression,
                 evaluation_error.message,
             )
+
+
+class PathDepthSorter(FileSorter):
+    def __call__(self, files: Iterable[File]) -> Iterable[File]:
+        return sorted(files, key=lambda f: (len(f.relative_path.parts),), reverse=True)

tempren/filesystem.py

@@ -88,6 +88,26 @@ def _gather_in(self, directory: Path) -> Iterable[File]:
                 )
 
 
+class RecursiveDirectoryGatherer(FilesystemGatherer):
+    def __init__(self, start_directory: Path):
+        self._start_directory = start_directory.absolute()
+
+    def gather_files(self) -> Iterable[File]:
+        yield from self._gather_in(self._start_directory)
+
+    def _gather_in(self, directory: Path) -> Iterable[File]:
+        print(f"Gathering in {directory}")
+        for dir_entry in directory.iterdir():
+            if not self._include_path_in_result(dir_entry):
+                continue
+
+            if dir_entry.is_dir():
+                yield File(
+                    self._start_directory, dir_entry.relative_to(self._start_directory)
+                )
+                yield from self._gather_in(dir_entry)
+
+
 class ExplicitFileGatherer(FileGatherer):
     def __init__(self, files: List[Path]):
         self._files_to_provide = files

tempren/pipeline.py

@@ -15,7 +15,7 @@
     RegexPathFileFilter,
     TemplateFileFilter,
 )
-from tempren.file_sorters import TemplateFileSorter
+from tempren.file_sorters import PathDepthSorter, TemplateFileSorter
 from tempren.filesystem import (
     CombinedFileGatherer,
     DestinationAlreadyExistsError,
@@ -28,6 +28,7 @@
     FlatFileGatherer,
     InvalidDestinationError,
     PrintingRenamerWrapper,
+    RecursiveDirectoryGatherer,
     RecursiveFileGatherer,
 )
 from tempren.primitives import CategoryName, File, PathGenerator, Pattern, TagName
@@ -48,6 +49,7 @@ class FilterType(Enum):
 class OperationMode(Enum):
     name = "name"
     path = "path"
+    directory = "directory"
 
 
 # TODO: Find a way to keep documentation close to the enum values and use it in argparser/generated help
@@ -283,16 +285,28 @@ def build_pipeline(
     pipeline = Pipeline()
 
     file_gatherers: List[FileGatherer] = []
+
     input_directories = filter(lambda p: p.is_dir(), config.input_paths)
-    for directory in input_directories:
+    input_files = list(filter(lambda p: p.is_file(), config.input_paths))
+    if config.mode == OperationMode.directory:
+        if any(input_files):
+            log.warning("Ignoring file paths provided in directory mode")
+
         if config.recursive:
-            file_gatherers.append(RecursiveFileGatherer(directory))
+            for directory in input_directories:
+                file_gatherers.append(RecursiveDirectoryGatherer(directory))
         else:
-            file_gatherers.append(FlatFileGatherer(directory))
+            file_gatherers.append(ExplicitFileGatherer(input_directories))
+    else:
+        for directory in input_directories:
+            if config.recursive:
+                file_gatherers.append(RecursiveFileGatherer(directory))
+            else:
+                file_gatherers.append(FlatFileGatherer(directory))
+
+        if any(input_files):
+            file_gatherers.append(ExplicitFileGatherer(input_files))
 
-    input_files = list(filter(lambda p: p.is_file(), config.input_paths))
-    if any(input_files):
-        file_gatherers.append(ExplicitFileGatherer(input_files))
     pipeline.file_gatherer = CombinedFileGatherer(file_gatherers)
     pipeline.file_gatherer.include_hidden = config.include_hidden
 
@@ -307,7 +321,7 @@ def _compile_template(template_text: str) -> Pattern:
 
     bound_pattern = _compile_template(config.template)
 
-    if config.mode == OperationMode.name:
+    if config.mode == OperationMode.name or config.mode == OperationMode.directory:
         pipeline.path_generator = TemplateNameGenerator(bound_pattern)
         if config.filter:
             if config.filter_type == FilterType.regex:
@@ -318,7 +332,7 @@ def _compile_template(template_text: str) -> Pattern:
                 bound_filter_pattern = _compile_template(config.filter)
                 pipeline.file_filter = TemplateFileFilter(bound_filter_pattern)
             else:
-                raise NotImplementedError("Unknown filter type")
+                raise NotImplementedError("Unknown filter type: " + str(config.filter))
     elif config.mode == OperationMode.path:
         pipeline.path_generator = TemplatePathGenerator(bound_pattern)
         if config.filter:
@@ -330,14 +344,20 @@ def _compile_template(template_text: str) -> Pattern:
                 bound_filter_pattern = _compile_template(config.filter)
                 pipeline.file_filter = TemplateFileFilter(bound_filter_pattern)
             else:
-                raise NotImplementedError("Unknown filter type")
+                raise NotImplementedError("Unknown filter type: " + str(config.filter))
     else:
-        raise NotImplementedError("Unknown operation mode")
+        raise NotImplementedError("Unknown operation mode: " + str(config.mode))
 
     if config.filter_invert and config.filter is not None:
         pipeline.file_filter = FileFilterInverter(pipeline.file_filter)
 
-    if config.sort:
+    if config.mode == OperationMode.directory:
+        # FIXME: This workaround allows us to mitigate conflicts arising from
+        #        trying to rename child directories after their parent has been renamed
+        if config.sort:
+            raise ConfigurationError("Sorting is not available in directory mode")
+        pipeline.sorter = PathDepthSorter()
+    elif config.sort:
         bound_sorter_pattern = _compile_template(config.sort)
         pipeline.sorter = TemplateFileSorter(bound_sorter_pattern, config.sort_invert)
 
@@ -346,9 +366,9 @@ def _compile_template(template_text: str) -> Pattern:
 
     if config.dry_run:
         pipeline.renamer = DryRunRenamer()
-        # FIXME: Use DryRunMover for path mode?
+        # FIXME: Use custom DryRunMover for path mode?
     else:
-        if config.mode == OperationMode.name:
+        if config.mode == OperationMode.name or config.mode == OperationMode.directory:
             pipeline.renamer = FileRenamer()
         elif config.mode == OperationMode.path:
             pipeline.renamer = FileMover()