test(pickle): add Linux-only test for inotify instance exhaustion (Issue #24) (#275)

* test(pickle): add Linux-only test for inotify instance exhaustion (Issue #24)\n\nAdds a test to reproduce the 'inotify instance limit reached' error on Linux by rapidly creating many concurrent cache waits using the pickle backend. The test is skipped on non-Linux systems and is designed to be informative, not flaky.\n\nAlso updates CI to lower the inotify instance limit on Linux/local jobs, increasing the likelihood of hitting the error in CI.\n\nReferences: https://github.com/python-cachier/cachier/issues/24\n\n- Follows project and Python best practices for test isolation, OS-specific logic, and CI configuration.\n- Test is marked with @pytest.mark.pickle and uses pytest.skip for clarity.\n- CI change is Linux/local only, with clear comments.\n\nThis helps document and track the resource exhaustion issue, and provides a reproducible test for future backend or watchdog improvements.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* style(tests): fix long lines for Ruff E501 compliance in test_pickle_core.py

* style(tests): split long lines in test_pickle_core.py to satisfy Ruff E501

* Make inotify test more aggressive and force failure instead of skip

- Increase thread count to 4x the system limit (up to 4096)
- Make the slow function slower (0.5s instead of 0.1s)
- Increase wait_for_calc_timeout to 0.1s
- Force test to fail instead of skip when limit not hit
- Add more debugging output to understand what's happening

This should help reproduce the inotify instance limit issue in CI.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Reverse inotify test logic: now fails when bug exists, passes when fixed

- Test now FAILS when inotify instance limit is reached (bug exists)
- Test now PASSES when no inotify errors occur (bug is fixed)
- This makes the test useful for verifying fixes to the inotify issue
- Updated error messages to reflect the new logic

* Add xfail marker to inotify test

- Test is now marked as expected to fail with @pytest.mark.xfail
- Will show as XFAIL when bug exists (expected)
- Will show as XPASS when bug is fixed (pleasant surprise)
- Provides clear reason for expected failure

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix all ruff lint errors in inotify test (line length, subprocess path, ternary, string wrapping)

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

Author: shaypal5

.github/workflows/ci-test.yml

@@ -56,6 +56,11 @@ jobs:
           python -m pip install --upgrade pip
           python -m pip install -e . -r tests/requirements.txt
 
+      - name: Lower inotify instance limit (Linux only, for Issue #24 test)
+        if: runner.os == 'Linux' && matrix.backend == 'local'
+        run: sudo sysctl -w fs.inotify.max_user_instances=128
+        # This helps the test_inotify_instance_limit_reached hit the limit in CI
+
       - name: Unit tests (local)
         if: matrix.backend == 'local'
         run: pytest -m "not mongo and not sql and not redis" --cov=cachier --cov-report=term --cov-report=xml:cov.xml

tests/test_pickle_core.py

@@ -26,6 +26,7 @@
     import Queue as queue  # type: ignore
 
 import hashlib
+import sys
 
 import pandas as pd
 
@@ -607,3 +608,108 @@ def _params_with_dataframe(*args, **kwargs):
     value_b = _params_with_dataframe(1, df=df_b)
 
     assert value_a == value_b  # same content --> same key
+
+
+@pytest.mark.pickle
+@pytest.mark.skipif(
+    not sys.platform.startswith("linux"),
+    reason="inotify instance limit is only relevant on Linux",
+)
+@pytest.mark.xfail(
+    reason=(
+        "inotify instance limit issue not yet fixed - test will pass "
+        "when issue is resolved"
+    )
+)
+def test_inotify_instance_limit_reached():
+    """Reproduces the inotify instance exhaustion issue (see Issue #24).
+
+    Rapidly creates many cache waits to exhaust inotify instances.
+    Reference: https://github.com/python-cachier/cachier/issues/24
+
+    """
+    import queue
+    import subprocess
+    import time
+
+    # Try to get the current inotify limit
+    try:
+        result = subprocess.run(
+            ["/bin/cat", "/proc/sys/fs/inotify/max_user_instances"],
+            capture_output=True,
+            text=True,
+            timeout=5,
+        )
+        if result.returncode == 0:
+            current_limit = int(result.stdout.strip())
+            print(f"Current inotify max_user_instances limit: {current_limit}")
+        else:
+            current_limit = None
+            print("Could not determine inotify limit")
+    except Exception as e:
+        current_limit = None
+        print(f"Error getting inotify limit: {e}")
+
+    @cachier(backend="pickle", wait_for_calc_timeout=0.1)
+    def slow_func(x):
+        time.sleep(0.5)  # Make it slower to increase chance of hitting limit
+        return x
+
+    # Start many threads to trigger wait_on_entry_calc
+    threads = []
+    errors = []
+    results = queue.Queue()
+
+    # Be more aggressive - try to exhaust the limit
+    N = (
+        min(current_limit * 4, 4096) if current_limit is not None else 4096
+    )  # Try to exceed the limit more aggressively
+    print(f"Starting {N} threads to test inotify exhaustion")
+
+    def call():
+        try:
+            results.put(slow_func(1))
+        except OSError as e:
+            errors.append(e)
+        except Exception as e:
+            # Capture any other exceptions for debugging
+            errors.append(e)
+
+    for i in range(N):
+        t = threading.Thread(target=call)
+        threads.append(t)
+        t.start()
+        if i % 100 == 0:
+            print(f"Started {i} threads...")
+
+    print("Waiting for all threads to complete...")
+    for t in threads:
+        t.join()
+
+    print(
+        f"Test completed. Got {len(errors)} errors, {results.qsize()} results"
+    )
+
+    # If any OSError with "inotify instance limit reached" is raised,
+    # the test FAILS (expected failure due to the bug)
+    if any("inotify instance limit reached" in str(e) for e in errors):
+        print(
+            "FAILURE: Hit inotify instance limit - this indicates the bug "
+            "still exists"
+        )
+        raise AssertionError(
+            "inotify instance limit reached error occurred. "
+            f"Got {len(errors)} errors with inotify limit issues."
+        )
+
+    # If no inotify errors but other errors, fail
+    if errors:
+        print(f"Unexpected errors occurred: {errors}")
+        raise AssertionError(f"Unexpected OSErrors: {errors}")
+
+    # If no errors at all, the test PASSES (issue is fixed!)
+    print(
+        "SUCCESS: No inotify instance limit errors occurred - the issue "
+        "appears to be fixed!"
+    )
+    # No need to return - test passes naturally