Upping the test coverage (#299)

* Upping the test coverage

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* README updates + add click to test requirements

* add redis pytest mark on test_missing_redis_client

* ruff fixes and pre-commit updates

* remove pull_request_target

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

Author: shaypal5

.github/workflows/ci-test.yml

@@ -3,8 +3,6 @@ name: Test
 on:
   pull_request:
     branches: [master]
-  pull_request_target:
-    branches: [master]
   push:
     branches: [master]
   schedule:

.pre-commit-config.yaml

@@ -23,7 +23,7 @@ repos:
       - id: detect-private-key
 
   - repo: https://github.com/crate-ci/typos
-    rev: v1
+    rev: v1.34.0
     hooks:
       - id: typos
         # empty to do not write fixes
@@ -55,7 +55,7 @@ repos:
         args: ["--print-width=79"]
 
   - repo: https://github.com/astral-sh/ruff-pre-commit
-    rev: v0.12.2
+    rev: v0.12.4
     hooks:
       # use black formatting
       - id: ruff-format
@@ -67,10 +67,10 @@ repos:
 
   # it needs to be after formatting hooks because the lines might be changed
   - repo: https://github.com/pre-commit/mirrors-mypy
-    rev: v1.16.1
+    rev: v1.17.0
     hooks:
       - id: mypy
-        files: "src/*"
+        files: "src/.*"
 
   - repo: https://github.com/tox-dev/pyproject-fmt
     rev: v2.6.0

README.rst

@@ -519,13 +519,20 @@ Clone:
   git clone git@github.com:python-cachier/cachier.git
 
 
-Install in development mode with test dependencies:
+Install in development mode with test dependencies for local cores (memory and pickle) only:
 
 .. code-block:: bash
 
   cd cachier
   pip install -e . -r tests/requirements.txt
 
+Each additional core (MongoDB, Redis, SQL) requires additional dependencies. To install all dependencies for all cores, run:
+
+.. code-block:: bash
+
+  pip install -r tests/mongodb_requirements.txt
+  pip install -r tests/redis_requirements.txt
+  pip install -r tests/sql_requirements.txt
 
 Running the tests
 -----------------
@@ -630,7 +637,23 @@ To test all cachier backends (MongoDB, Redis, SQL, Memory, Pickle) locally with
   # Keep containers running for debugging
   ./scripts/test-local.sh all -k
 
-The unified test script automatically manages Docker containers, installs required dependencies, and runs the appropriate test suites. See ``scripts/README-local-testing.md`` for detailed documentation.
+  # Test specific test files with selected backends
+  ./scripts/test-local.sh mongo -f tests/test_mongo_core.py
+
+  # Test multiple files across all backends
+  ./scripts/test-local.sh all -f tests/test_main.py -f tests/test_redis_core_coverage.py
+
+The unified test script automatically manages Docker containers, installs required dependencies, and runs the appropriate test suites. The ``-f`` / ``--files`` option allows you to run specific test files instead of the entire test suite. See ``scripts/README-local-testing.md`` for detailed documentation.
+
+
+Running pre-commit hooks locally
+--------------------------------
+
+After you've installed test dependencies, you can run pre-commit hooks locally by using the following command:
+
+.. code-block:: bash
+
+  pre-commit run --all-files
 
 
 Adding documentation

pyproject.toml

@@ -24,6 +24,7 @@ license = { file = "LICENSE" }
 authors = [
   { name = "Shay Palachy Affek", email = 'shay.palachy@gmail.com' },
 ]
+requires-python = ">=3.9"
 classifiers = [
   "Development Status :: 4 - Beta",
   "Intended Audience :: Developers",
@@ -188,8 +189,8 @@ branch = true
 # dynamic_context = "test_function"
 omit = [
   "tests/*",
-  "cachier/_version.py",
-  "cachier/__init__.py",
+  "src/cachier/_version.py",
+  "src/cachier/__init__.py",
   "**/scripts/**",
 ]
 [tool.coverage.report]

scripts/README-local-testing.md

@@ -40,6 +40,7 @@ This guide explains how to run cachier tests locally with Docker containers for
 - `-v, --verbose` - Show verbose pytest output
 - `-k, --keep-running` - Keep Docker containers running after tests
 - `-h, --html-coverage` - Generate HTML coverage report
+- `-f, --files` - Specify test files to run (can be used multiple times)
 - `--help` - Show help message
 
 ## Examples
@@ -86,6 +87,15 @@ make test-sql-local
 
 # Using environment variable
 CACHIER_TEST_CORES="mongo redis" ./scripts/test-local.sh
+
+# Test specific files with MongoDB backend
+./scripts/test-local.sh mongo -f tests/test_mongo_core.py
+
+# Test multiple files across all backends
+./scripts/test-local.sh all -f tests/test_main.py -f tests/test_redis_core_coverage.py
+
+# Combine file selection with other options
+./scripts/test-local.sh redis sql -f tests/test_sql_core.py -v -k
 ```
 
 ### Docker Compose

scripts/test-local.sh

@@ -25,6 +25,7 @@ COVERAGE_REPORT="term"
 KEEP_RUNNING=false
 SELECTED_CORES=""
 INCLUDE_LOCAL_CORES=false
+TEST_FILES=""
 
 # Function to print colored messages
 print_message() {
@@ -54,6 +55,7 @@ OPTIONS:
     -v, --verbose       Show verbose output
     -k, --keep-running  Keep containers running after tests
     -h, --html-coverage Generate HTML coverage report
+    -f, --files         Specify test files to run (can be used multiple times)
     --help              Show this help message
 
 EXAMPLES:
@@ -62,6 +64,7 @@ EXAMPLES:
     $0 all                      # Run all backend tests
     $0 external -k              # Run external backends, keep containers
     $0 mongo memory -v          # Run MongoDB and memory tests verbosely
+    $0 all -f tests/test_main.py -f tests/test_redis_core_coverage.py  # Run specific test files
 
 ENVIRONMENT:
     You can also set cores via CACHIER_TEST_CORES environment variable:
@@ -85,6 +88,16 @@ while [[ $# -gt 0 ]]; do
             COVERAGE_REPORT="html"
             shift
             ;;
+        -f|--files)
+            shift
+            if [[ $# -eq 0 ]] || [[ "$1" == -* ]]; then
+                print_message $RED "Error: -f/--files requires a file argument"
+                usage
+                exit 1
+            fi
+            TEST_FILES="$TEST_FILES $1"
+            shift
+            ;;
         --help)
             usage
             exit 0
@@ -193,14 +206,14 @@ check_docker() {
         echo ""
         echo "After starting Docker, wait a few seconds and try running this script again."
         echo ""
-        
+
         # Show the actual docker error for debugging
         echo "Technical details:"
         docker ps 2>&1 | sed 's/^/  /'
         echo ""
         exit 1
     fi
-    
+
     print_message $GREEN "✓ Docker is installed and running"
 }
 
@@ -473,27 +486,48 @@ main() {
     done
 
     # Run pytest
-    # Check if we selected all cores - if so, run all tests without marker filtering
-    all_cores="memory mongo pickle redis sql"
-    selected_sorted=$(echo "$SELECTED_CORES" | tr ' ' '\n' | sort | tr '\n' ' ' | xargs)
-    all_sorted=$(echo "$all_cores" | tr ' ' '\n' | sort | tr '\n' ' ' | xargs)
-
-    if [ "$selected_sorted" = "$all_sorted" ]; then
-        print_message $BLUE "Running: pytest (all tests, including unmarked)"
-        if [ "$VERBOSE" = true ]; then
-            pytest -v --cov=cachier --cov-report=$COVERAGE_REPORT
-        else
-            pytest --cov=cachier --cov-report=$COVERAGE_REPORT
+    # Build pytest command
+    PYTEST_CMD="pytest"
+
+    # Add test files if specified
+    if [ -n "$TEST_FILES" ]; then
+        PYTEST_CMD="$PYTEST_CMD $TEST_FILES"
+        print_message $BLUE "Test files specified: $TEST_FILES"
+    fi
+
+    # Add markers if needed (only if no specific test files were given)
+    if [ -z "$TEST_FILES" ]; then
+        # Check if we selected all cores - if so, run all tests without marker filtering
+        all_cores="memory mongo pickle redis sql"
+        selected_sorted=$(echo "$SELECTED_CORES" | tr ' ' '\n' | sort | tr '\n' ' ' | xargs)
+        all_sorted=$(echo "$all_cores" | tr ' ' '\n' | sort | tr '\n' ' ' | xargs)
+
+        if [ "$selected_sorted" != "$all_sorted" ]; then
+            PYTEST_CMD="$PYTEST_CMD -m \"$pytest_markers\""
         fi
     else
-        print_message $BLUE "Running: pytest -m \"$pytest_markers\""
-        if [ "$VERBOSE" = true ]; then
-            pytest -v -m "$pytest_markers" --cov=cachier --cov-report=$COVERAGE_REPORT
-        else
-            pytest -m "$pytest_markers" --cov=cachier --cov-report=$COVERAGE_REPORT
+        # When test files are specified, still apply markers if not running all cores
+        all_cores="memory mongo pickle redis sql"
+        selected_sorted=$(echo "$SELECTED_CORES" | tr ' ' '\n' | sort | tr '\n' ' ' | xargs)
+        all_sorted=$(echo "$all_cores" | tr ' ' '\n' | sort | tr '\n' ' ' | xargs)
+
+        if [ "$selected_sorted" != "$all_sorted" ]; then
+            PYTEST_CMD="$PYTEST_CMD -m \"$pytest_markers\""
         fi
     fi
 
+    # Add verbose flag if needed
+    if [ "$VERBOSE" = true ]; then
+        PYTEST_CMD="$PYTEST_CMD -v"
+    fi
+
+    # Add coverage options
+    PYTEST_CMD="$PYTEST_CMD --cov=cachier --cov-report=$COVERAGE_REPORT"
+
+    # Print and run the command
+    print_message $BLUE "Running: $PYTEST_CMD"
+    eval $PYTEST_CMD
+
     TEST_EXIT_CODE=$?
 
     if [ $TEST_EXIT_CODE -eq 0 ]; then

tests/requirements.txt

@@ -10,3 +10,7 @@ pygments
 # the memory core tests dataframe caching
 pandas
 pympler
+# for cli tests
+click
+# to run pre-commit hooks localy
+pre-commit

tests/test_base_core.py

@@ -0,0 +1,76 @@
+"""Additional tests for base core to improve coverage."""
+
+from unittest.mock import patch
+
+from cachier.cores.base import _BaseCore
+
+
+class ConcreteCachingCore(_BaseCore):
+    """Concrete implementation of _BaseCore for testing."""
+
+    def get_entry_by_key(self, key, reload=False):
+        """Retrieve an entry by its key."""
+        return key, None
+
+    def set_entry(self, key, func_res):
+        """Store an entry in the cache."""
+        return True
+
+    def mark_entry_being_calculated(self, key):
+        """Mark an entry as being calculated."""
+        pass
+
+    def mark_entry_not_calculated(self, key):
+        """Mark an entry as not being calculated."""
+        pass
+
+    def wait_on_entry_calc(self, key):
+        """Wait for an entry calculation to complete."""
+        return None
+
+    def clear_cache(self):
+        """Clear the cache."""
+        pass
+
+    def clear_being_calculated(self):
+        """Clear entries that are being calculated."""
+        pass
+
+    def delete_stale_entries(self, stale_after):
+        """Delete stale entries from the cache."""
+        pass
+
+
+def test_estimate_size_fallback():
+    """Test _estimate_size falls back to sys.getsizeof when asizeof fails."""
+    # Test lines 101-102: exception handling in _estimate_size
+    core = ConcreteCachingCore(
+        hash_func=None, wait_for_calc_timeout=10, entry_size_limit=1000
+    )
+
+    # Mock asizeof to raise exception
+    with patch(
+        "cachier.cores.base.asizeof.asizeof",
+        side_effect=Exception("asizeof failed"),
+    ):
+        # Should fall back to sys.getsizeof
+        size = core._estimate_size("test_value")
+        assert size > 0  # sys.getsizeof should return a positive value
+
+
+def test_should_store_exception():
+    """Test _should_store returns True when size estimation fails."""
+    # Test lines 109-110: exception handling in _should_store
+    core = ConcreteCachingCore(
+        hash_func=None, wait_for_calc_timeout=10, entry_size_limit=1000
+    )
+
+    # Mock both size estimation methods to fail
+    patch1 = patch(
+        "cachier.cores.base.asizeof.asizeof",
+        side_effect=Exception("asizeof failed"),
+    )
+    patch2 = patch("sys.getsizeof", side_effect=Exception("getsizeof failed"))
+    with patch1, patch2:
+        # Should return True (allow storage) when size can't be determined
+        assert core._should_store("test_value") is True

tests/test_config.py

@@ -0,0 +1,25 @@
+"""Additional tests for config module to improve coverage."""
+
+import pytest
+
+from cachier.config import get_default_params, set_default_params
+
+
+def test_set_default_params_deprecated():
+    """Test that set_default_params shows deprecation warning."""
+    # Test lines 103-111: deprecation warning
+    with pytest.warns(
+        DeprecationWarning,
+        match="set_default_params.*deprecated.*set_global_params",
+    ):
+        set_default_params(stale_after=60)
+
+
+def test_get_default_params_deprecated():
+    """Test that get_default_params shows deprecation warning."""
+    # Test lines 143-151: deprecation warning
+    with pytest.warns(
+        DeprecationWarning,
+        match="get_default_params.*deprecated.*get_global_params",
+    ):
+        assert get_default_params() is not None