Modernise Development Workflow, Enhance Code Quality, and Refactor Core Utilities by chrimaho · Pull Request #38 · data-science-extensions/toolbox-python

chrimaho · 2025-12-28T10:01:28Z

📋 Overview

This pull request introduces a significant modernisation of the project's development workflow, code quality standards, and core utility implementations. Key enhancements include migrating the environment management to uv, replacing legacy static analysis tools with more performant alternatives, refactoring complex logic to improve maintainability, and standardising type safety across the entire codebase.

🚀 Environment and Workflow Modernisation

Migration to `uv` Ecosystem

pyproject.toml: Transition the project to use uv as the primary package and environment manager
- Define comprehensive dependency groups for dev, docs, and test within the [tool.uv] configuration
- Remove legacy pipenv and poetry configurations to simplify the build process
- Standardise the requires-python specification by removing the upper bound restriction for better forward compatibility
- Comment out project scripts in pyproject.toml to prevent command-line tool conflicts whilst preserving definitions
README.md: Update installation and contribution guides to reflect the new uv-based workflow
- Replace pip install and poetry install instructions with uv sync and uv run commands
- Document the unified task execution pattern using the centralised utility script
Cleanup: Remove redundant environment configuration files
- Delete the Makefile in preference for direct script execution via uv
- Remove the requirements/ directory as all dependencies are now managed within pyproject.toml

Centralised Task Execution

src/utils/scripts.py: Refactor the utility script to serve as a comprehensive task runner
- Implement dynamic package and directory name detection using PACKAGE_NAME and DIRECTORY_NAME constants
- Reorder check sequences to prioritise fast-failing tests and linting operations
- Integrate new quality assurance commands including check-complexity, check-ty, and check-docstrings
- Fix CLI argument handling by updating indexing from sys.argv[1] to sys.argv[2] to account for function name parameters

🛡️ Code Quality and Static Analysis

Enhanced Type Checking and Complexity Monitoring

Type Safety: Replace mypy with ty for faster and more integrated type checking
- Update the check-ty command to execute ty check across the source and test directories
- Standardise type annotations using Optional() and Union() classes for improved compatibility
- Incorporate @overload declarations for functions like print_or_log_output() to handle complex parameter combinations
- Add an assert to ensure that the print_or_log parameter is never None in the print_or_log_output() function
- Correct the tuple type hint to include the ellipsis ... to properly represent variadic tuples in the check_type parameter documentation
- Enhance type checking in checkers.py and output.py by adding more comprehensive @overload conditions
Complexity Analysis: Introduce complexipy to monitor and enforce code maintainability
- Configure [tool.complexipy] with a max-complexity-allowed threshold of 15
- Implement the check-complexity task to automate cyclomatic complexity audits during development

Documentation Standards

Docstring Validation: Integrate docstring-format-checker as a pre-commit hook
- Replace over 290 lines of custom docstring parsing and validation logic with the external dfc tool for improved maintainability
- Remove the complex FunctionAndClassDetails() class and associated validation methods
- Enforce consistent docstring formatting across all modules, excluding utility and test directories
- Refactor existing docstrings to fix misaligned parameter documentation and incorrect type hints
- Standardise admonition styles, replacing ???+ info with ???+ abstract for better visual hierarchy

⚙️ CI/CD Pipeline Enhancements

Robust Deployment Workflow

.github/workflows/cd.yml: Refactor the continuous deployment pipeline for improved reliability
- Consolidate permissions at the workflow level instead of per-job for better maintainability
- Update all action versions to the latest stable releases for improved security and features
- Include Python 3.14 in the test matrix to ensure early compatibility with upcoming releases
- Add python-version-file configuration for consistent Python version management across environments
- Correct the CI job matrix to run on specified operating systems instead of ubuntu-latest only
- Standardise script execution by invoking utility scripts directly via uv run
- Increase max-parallel execution to 30 to accelerate multi-platform installation checks
- Extend artifact retention to 5 days to facilitate better post-build analysis
- Optimise the uv publish command by using explicit token flags and disabling unnecessary caching
- Set PYTHONIOENCODING to utf-8 to ensure consistent log output across different platforms

Git and Subprocess Handling

src/utils/changelog.py: Improve git interaction and commit message processing
- Standardise text handling by enforcing utf-8 encoding during subprocess execution
- Refine commit message filtering to exclude Co-authored-by lines and empty entries
- Update the release title reference to use the name property instead of the deprecated title attribute
Git Utilities: Refactor git helper functions for better reusability
- Extract branch checkout logic into a new git_checkout_branch() function
- Introduce git_switch_to_branch() function to support branch switching via command-line arguments
- Optimise git_add_coverage_report() function by ensuring the destination directory exists before copying files
- Simplify add_git_credentials() and git_fix_tag_reference() functions by removing redundant comments

🛠️ Core Logic and Utility Refactoring

Retry Logic Optimisation

src/toolbox_python/retry.py: Perform a major refactor of the retry() decorator
- Extract core execution logic into a new internal _Retry() class to improve modularity
- Reduce cyclomatic complexity from 17 to 4 by decomposing the retry loop into specific methods
- Implement ._handle_expected_error(), ._handle_unexpected_error(), and ._handle_final_failure() methods
- Strengthen type safety by using the NoReturn() type hint for terminal failure states
- Ensure the exceptions attribute always stores a tuple() of exception types for consistent catching

New Validation Utilities

src/toolbox_python/validators.py: Introduce a new Validators() class for numeric range assertions
- Implement ._value_is_between() and ._assert_value_is_between() for individual boundary checks
- Provide ._all_values_are_between() and ._assert_all_values_are_between() for sequence validation
- Add comprehensive unit tests in src/tests/test_validators.py achieving 100% coverage

Metadata and Versioning Standardisation

src/toolbox_python/__init__.py: Modernise package metadata retrieval
- Replace hardcoded version strings with dynamic retrieval via the metadata() function
- Expose package attributes like __version__, __author__, and __email__ dynamically
- Remove secondary metadata attributes including __license__, __url__, and __description__ to organise the package root
Cleanup: Remove the custom src/utils/bump_version.py utility in favour of native uv version capabilities
- Delete the test_version.py file as hardcoded version checks are no longer necessary
- Update the CD workflow to use the native uv version command for release management

🎯 Impact and Benefits

This change improves the project's maintainability and developer experience by:

Streamlining the development environment through the adoption of uv, reducing setup time and dependency conflicts
Enforcing higher code quality standards with automated complexity monitoring and standardised docstring validation
Improving runtime reliability by refactoring complex decorators and strengthening type safety across core modules
Future-proofing the codebase by adding support for Python 3.14 and removing restrictive version bounds

… comprehensive `@overload` conditions for better clarity and functionality.

… docstrings - Details admonition: `???+ info "Details"` --> `???+ abstract "Details"` - Notes admonition: `???+ info "Notes"` --> `???+ abstract "Notes"` - Generic type list: `Type` --> `(Type)` - Typos: `Parameters` --> `Params` - Missing docstrings

- Replace custom docstring validation logic with external `dfc` tool for improved consistency and maintainability - Remove unused imports including `ast`, `re`, `math.e`, and various typing components - Fix CLI argument indexing from `sys.argv[1]` to `sys.argv[2]` across multiple functions to account for function name parameter - Remove complex `FunctionAndClassDetails` class and associated validation methods - Simplify `check_docstrings()` function to use single `dfc --output=table ./src/toolbox_python` command - Add centralised CLI execution logic with function name resolution and error handling - Eliminate over 290 lines of custom docstring parsing and validation code

- Temporarily disable all console script entry points to prevent command-line tool conflicts - Preserve script definitions for future re-enablement by commenting rather than removing - Affects syncing, linting, checking, git operations, and documentation scripts - Maintains project configuration structure whilst removing executable commands

- Consolidate permissions at workflow level instead of per-job for better maintainability - Update all action versions to latest stable releases for improved security and features - Replace hardcoded values with environment variables for better configurability - Enable PyPI package publishing by uncommenting the publish step - Improve package installation verification with explicit version constraints - Reorganise job execution order by moving tag reference fix after package upload - Add python-version-file configuration for consistent Python version management - Correct CI job matrix to run on specified operating systems instead of ubuntu-latest only - Enhance environment variable coverage for tokens, repository details, and build settings

- Replace local docstring checking implementation with external pre-commit repository - Use `docstring-format-checker` from `data-science-extensions` organisation at version 1.3.0 - Maintain same configuration and output format whilst leveraging standardised tooling - Comment out previous local implementation to preserve configuration for reference

- Replace union type syntax with `Optional` and `Literal` imports for better compatibility - Enhance commit message processing to filter out co-authored-by lines and empty lines - Update commit output format to include short SHA with link and improved author attribution - Fix release title reference to use `name` property instead of deprecated `title` - Add type ignore comment for repository retrieval to suppress type checker warnings - Include main execution guard for proper script entry point handling

- Replace `mypy` with `ty` in the `test` dependency group to standardise type checking - Remove the `[tool.mypy]` configuration block - Rename the `check_mypy()` function to the `check_ty()` function - Update the `check_ty()` function to execute the `ty check` command using a dynamic directory path - Update the `check()` function to call the `check_ty()` function

…e `src/utils/scripts.py` module

- Define the `check_complexity()` function to automate quality checks and analyse code complexity levels to guide developers - Configure the `[tool.complexipy]` section to establish analysis parameters that enforce coding standards - Set the `max-complexity-allowed` parameter to 15 to ensure code remains maintainable

- Expand the exclusion pattern in `.pre-commit-config.yaml` to include the `src/utils/` and `src/tests` directories. - Utilise a multi-line regex format to manage multiple excluded paths more effectively.

- Define `PACKAGE_NAME` and `DIRECTORY_NAME` constants to centralise configuration - Update `check_pylint()` function to use the dynamic `DIRECTORY_NAME` constant - Refine `check_pycln()` function to target the specific package directory

- Relocate the `check_docstrings()` function to improve logical file structure - Update the `check_docstrings()` function to reference the `DIRECTORY_NAME` constant - Standardise the source path within the `check_docstrings()` function to support dynamic directory names

- Move `check_pylint()` function after `check_pycln()` function to improve linting workflow - Add `check_complexity()` function to the `check()` function sequence to monitor code quality - Prioritise `check_pytest()` function execution by moving it before documentation and build checks

- Correct the `tuple` type hint to include the ellipsis `...` to properly represent variadic tuples in the docstrings. - Ensure the `check_type` parameter documentation accurately reflects that multiple types can be provided. - Standardise the docstrings for the `is_value_of_type()`, `is_all_values_of_type()`, `is_any_values_of_type()`, `assert_value_of_type()`, `assert_all_values_of_type()`, and `assert_any_values_of_type()` functions.

- Update type annotations to use the `Optional()` and `Union()` classes for better compatibility within the `Defaults()` class. - Replace the `is_type()` function call with the standard `isinstance()` function within the `.get()` method to verify string values. - Standardise method signatures for the `._validate_value_and_default()` and `._validate_type()` methods to use explicit typing constructs.

- Introduce `Validators()` class to centralise logic for checking if numeric values fall within specified bounds - Implement `._value_is_between()` method to perform individual range checks and validate that boundary arguments are logical - Provide `._assert_value_is_between()` method to raise an `AssertionError` when a value violates the defined range - Include `._all_values_are_between()` method and `._assert_all_values_are_between()` method to facilitate bulk validation of sequences - Add unit tests to verify correct behaviour and error handling for the new `Validators()` class

…es` module

- Related to the use of `*` in the Params section

…signatures

- Update the `tuple()` class type hint within the `name_func_nested_list()` function to include an ellipsis for variable-length support.

- Update `requires-python` to remove the `<4.0` restriction - Standardise the configuration to improve forward compatibility

- Replace hardcoded version strings with dynamic retrieval via the `metadata()` function - Remove the custom `bump_version.py` utility and its configuration in the `pyproject.toml` file - Update the `CD` workflow to use the native `uv version` command - Delete the `test_version.py` file as hardcoded version checks are no longer necessary - Expand the `__init__.py` file to dynamically expose package metadata such as version and author

- Add a new `@overload` for the `print_or_log_output()` function to support `Optional` parameters. - Update the `print_or_log_output()` function signature to accept an `Optional` value for the `print_or_log` argument. - Standardise the `log_level` type hint within the docstring of the `print_or_log_output()` function. - Reformat existing `@overload` declarations for the `print_or_log_output()` function to improve readability. - Add an `assert` to ensure that the `print_or_log` parameter is never `None`

- Extract the core logic from the `retry()` decorator into a new internal `_Retry()` class to improve maintainability. - Modularise error handling by introducing specific methods like `._handle_expected_error()` and `._handle_unexpected_error()` within the `_Retry()` class. - Update the `retry()` decorator to instantiate the `_Retry()` class and invoke its `.run()` method. - Strengthen type hinting by incorporating `Any` and defining explicit return types for the internal wrapper. - Reduce module complexity from >17 to ~4

- Standardise the `exceptions` attribute in the `_Retry()` class by ensuring it always stores a tuple of exception types. - Update the `._handle_final_failure()` method to use the `NoReturn` type hint as it consistently raises a `RuntimeError()` class.

- Remove `__license__`, `__url__`, and `__description__` variables - Organise the package root by removing secondary metadata attributes

- Add `encoding="utf-8"` to the `run()` function call within the `run_command()` function to standardise text handling - Extract branch checkout logic into a new `git_checkout_branch()` function to deduplicate `git_switch_to_main_branch()` and `git_switch_to_docs_branch()` functions - Introduce `git_switch_to_branch()` function to support branch switching via command-line arguments - Optimise `git_add_coverage_report()` function by ensuring the destination directory exists before copying files - Simplify `add_git_credentials()` and `git_fix_tag_reference()` functions by removing redundant comments and streamlining command execution

- Include Python `3.14` in test matrices to ensure forward compatibility. - Remove redundant `debug` jobs to reduce workflow clutter and execution time. - Standardise script execution by invoking utility scripts directly via `uv run`. - Force package reinstallation and upgrades using the `--reinstall-package` and `--upgrade` flags in the `uv sync` command. - Set `PYTHONIOENCODING` to `utf-8` to ensure consistent log output. - Increase `max-parallel` to 30 to speed up multi-platform installation checks. - Extend artifact retention to 5 days for better post-build access. - Refine the `uv publish` command to use explicit token flags and disable caching.

- Add `uv` as a supported installation method to improve package management efficiency - Replace legacy `pipenv` and `poetry` setup guides with `uv sync` workflows to simplify environment builds - Standardise development scripts using the `uv run src/utils/scripts.py` script for unified task execution - Expand quality assurance requirements to include complexity and docstring checks for better code maintainability - Update reference links to include `uv`, `ty`, `complexipy`, and `dfc` and fix the documentation URL

…onfig

Copilot

Pull request overview

This pull request modernizes the development workflow by migrating to uv for package management, enhancing code quality tooling, and refactoring core utilities for improved maintainability and type safety.

Key Changes

Environment Management: Complete migration from pipenv/poetry to uv ecosystem with centralized dependency management in pyproject.toml
Code Quality Tooling: Replaced mypy with ty for type checking, introduced complexipy for complexity monitoring, and integrated docstring-format-checker for documentation standards
Core Refactoring: Extracted retry logic into internal _Retry class (reducing complexity from 17 to 4), introduced new Validators class for numeric range validation, and improved type safety with comprehensive @overload declarations

Reviewed changes

Copilot reviewed 35 out of 35 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
`src/utils/scripts.py`	Refactored task runner with dynamic package detection, reordered check sequences, and updated CLI argument handling from `sys.argv[1]` to `sys.argv[2]`
`src/utils/changelog.py`	Improved commit message filtering to exclude `Co-authored-by` lines and empty entries, standardized UTF-8 encoding for subprocess execution
`src/utils/bump_version.py`	Deleted in favor of native `uv version` command
`src/toolbox_python/retry.py`	Major refactoring with new `_Retry` class extracting core execution logic into specific handler methods
`src/toolbox_python/validators.py`	New utility class providing numeric range validation methods with comprehensive test coverage
`src/toolbox_python/output.py`	Enhanced type safety with multiple `@overload` signatures and added assertion for type checker satisfaction
`src/toolbox_python/checkers.py`	Added support for `list[type]` parameter alongside existing `tuple[type, ...]`, expanded operators with `is` and `is not`
`src/toolbox_python/strings.py`	Standardized `Raises` section formatting in docstrings to use parentheses notation
`src/toolbox_python/lists.py`	Updated documentation admonition styles from `info` to `abstract`
`src/toolbox_python/dictionaries.py`	Added comprehensive docstrings to `DotDict` methods, improved type hints with `Any` annotations
`src/toolbox_python/defaults.py`	Replaced union type syntax `X
`src/toolbox_python/bools.py`	Fixed spelling typo in `distutils` reference
`src/toolbox_python/__init__.py`	Modernized metadata retrieval using `importlib.metadata` instead of hardcoded values
`src/tests/test_validators.py`	New comprehensive test suite for `Validators` class
`src/tests/test_version.py`	Deleted as dynamic version retrieval eliminates need for hardcoded version checks
`pyproject.toml`	Removed upper Python version bound, commented out project scripts, added configurations for `complexipy` and `dfc` tools
`.pre-commit-config.yaml`	Updated all action versions, removed `mypy` hook, added `docstring-format-checker` and `ty` hooks
`.github/workflows/cd.yml`	Consolidated permissions at workflow level, added Python 3.14 to test matrix, updated to use `uv run` for script execution
`README.md`	Updated installation instructions to include `uv`, revised contribution guide to use `uv sync` commands
`requirements/*.txt`	All deleted as dependencies now managed in `pyproject.toml` dependency groups
`Makefile`	Deleted in favor of direct script execution via `uv run`
`docs/usage/overview.md`	Simplified to include README content via snippet

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

- Use `-r` instead of `--recursive` in `check_build()` and `check_mkdocs()` functions - Standardise directory removal commands for consistency - Ensure commands can be run on all OS's consistently

- Expose the `.value_is_between()` method, `.assert_value_is_between()` method, `.all_values_are_between()` method, and `.assert_all_values_are_between()` method to make them public. - Add a class-level docstring to the `Validators()` class to summarise its purpose and available methods. - Update unit tests and internal method calls to reference the renamed public methods.

- Optimise function readability and ensure explicit return behaviour for the `print_or_log_output()` function.

chrimaho added 30 commits August 3, 2025 10:32

Fix missing type hints

1d9acdd

Remove redundant lines

2de8701

Enhance type checking in checkers.py and output.py by adding more…

5f32f50

… comprehensive `@overload` conditions for better clarity and functionality.

Run linting for better formatting

713ae0e

Bump package versions in the pre-commit config file

31dc776

Clean up outdated checks in the pre-commit config file

0609a01

Eliminate redundant scripts from the pyproject.toml file, prefer th…

167398f

…e `src/utils/scripts.py` module

Update pre-commit hook exclusions

008cdf5

- Expand the exclusion pattern in `.pre-commit-config.yaml` to include the `src/utils/` and `src/tests` directories. - Utilise a multi-line regex format to manage multiple excluded paths more effectively.

Fix missing complexipy dependency

b51e693

Add missing typehints to the functions and methods in the `dictionari…

9bb4c8b

…es` module

Fix docsting error in the dictionaries module

02acdeb

- Related to the use of `*` in the Params section

In the lists module, fix types in docstrings to match the function …

1eeacc8

…signatures

Fix tuple type hint for nested parameters

3ea392b

- Update the `tuple()` class type hint within the `name_func_nested_list()` function to include an ellipsis for variable-length support.

Fix formatting

48da0dd

Remove Python version upper bound

9ececa7

- Update `requires-python` to remove the `<4.0` restriction - Standardise the configuration to improve forward compatibility

chrimaho added 12 commits December 28, 2025 18:45

Remove redundant Makefile

2e057dd

Fix misaligned parameter docs and function signatures

857900f

Remove redundant metadata attributes

5999a79

- Remove `__license__`, `__url__`, and `__description__` variables - Organise the package root by removing secondary metadata attributes

Delete requirements/ directory in preference for pyproject.toml c…

9a8bc46

…onfig

Merge remote-tracking branch 'origin/main' into updates

f11c76b

Merge remote-tracking branch 'origin/main' into updates

9884a0c

chrimaho requested a review from Copilot December 28, 2025 10:01

chrimaho self-assigned this Dec 28, 2025

Copilot started reviewing on behalf of chrimaho December 28, 2025 10:01 View session

Copilot AI reviewed Dec 28, 2025

View reviewed changes

Comment thread pyproject.toml

Comment thread .github/workflows/cd.yml

Comment thread .github/workflows/cd.yml

Comment thread src/toolbox_python/validators.py

Comment thread src/utils/scripts.py

Comment thread src/toolbox_python/output.py

chrimaho added 4 commits December 28, 2025 21:09

Simplify directory removal flags

aad6e43

- Use `-r` instead of `--recursive` in `check_build()` and `check_mkdocs()` functions - Standardise directory removal commands for consistency - Ensure commands can be run on all OS's consistently

Merge remote-tracking branch 'origin/main' into updates

47327fa

Add explicit return to print_or_log_output()

4b24e77

- Optimise function readability and ensure explicit return behaviour for the `print_or_log_output()` function.

chrimaho merged commit 3ecb723 into main Dec 28, 2025
21 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Modernise Development Workflow, Enhance Code Quality, and Refactor Core Utilities#38

Modernise Development Workflow, Enhance Code Quality, and Refactor Core Utilities#38
chrimaho merged 46 commits into
mainfrom
updates

chrimaho commented Dec 28, 2025

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

chrimaho commented Dec 28, 2025

📋 Overview

🚀 Environment and Workflow Modernisation

Migration to uv Ecosystem

Centralised Task Execution

🛡️ Code Quality and Static Analysis

Enhanced Type Checking and Complexity Monitoring

Documentation Standards

⚙️ CI/CD Pipeline Enhancements

Robust Deployment Workflow

Git and Subprocess Handling

🛠️ Core Logic and Utility Refactoring

Retry Logic Optimisation

New Validation Utilities

Metadata and Versioning Standardisation

🎯 Impact and Benefits

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Key Changes

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Migration to `uv` Ecosystem