Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,8 @@ wheels/
.venv
.idea/

.coverage
.coverage*
coverage.xml
logs/
.envrc
cdisplayagain.egg-info
Expand Down
12 changes: 5 additions & 7 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,16 +30,14 @@ Use git worktrees to work on multiple cards in parallel without branch conflicts
- WIP limit: 3 cards total in progress across all worktrees.

## Test Coverage Requirements
- Current target: 90% coverage threshold (configured in `pyproject.toml`)
- Coverage milestones: 68% ✅ → 74% ✅ → 80% ✅ → 85% ✅ → 90% ✅ → 95% 🎯 → 100%
- Current target: 96% coverage threshold (configured in `pyproject.toml`)
- Coverage milestones: 68% ✅ → 74% ✅ → 80% ✅ → 85% ✅ → 90% ✅ → 95% ✅ → 96% 🎯 → 100%
- Always run `uv run pytest --cov=cdisplayagain --cov-report=term-missing` to check missing coverage
- When touching logic or input handling, ensure tests are added to maintain coverage
- Strategies for increasing coverage:
- Add tests for placeholder methods (11 methods in cdisplayagain.py)
- Add tests for UI features (info screen, F1 help, configuration screen)
- Add tests for configuration options
- Add tests for edge cases (empty archives, corrupted files, large archives)
- Add integration tests for full workflows
- Add tests for remaining uncovered edge cases
- Add tests for complex error handling paths
- Add tests for platform-specific code paths

## Performance Guidelines
- Image resizing (LANCZOS resampling) is the primary bottleneck (~65% of CPU time)
Expand Down
153 changes: 153 additions & 0 deletions SESSION_SUMMARY.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,153 @@
# CI Threading Crash Fix - Session Summary

## Problem
CI tests were failing with threading crashes during pytest collection/execution. Threads were stuck in `queue.get()` calls causing Python to abort.

```
Fatal Python error: Aborted
Thread X: File "queue.py", line 180 in get
File "cdisplayagain.py", line 424 in _run
```

## Attempted Solutions

### 1. Try/Finally Blocks ✅ (Commit: e179690)
- Added `try/finally` blocks to 16 tests across 3 files
- Ensures `worker.stop()` is called before Tk root destruction
- **Result**: Tests pass locally, CI still crashes
- **Issue**: Too verbose, not Pythonic

### 2. Context Manager ✅ (Commit: c19ea81)
- Converted `ImageWorker` to a context manager
- Added `__enter__` and `__exit__` methods
- Updated tests to use `with ImageWorker() as worker:` pattern
- **Result**: Tests pass locally (411 passed, 95.99% coverage)
- **Issue**: CI still crashes with same threading errors

### 3. Worker Thread Safety Improvements
- Added `_stopped` flag to `ImageWorker`
- Added early exit checks in `_run()` loop
- Set `self._app = None` in `stop()` to prevent access after cleanup
- Added exception handling in `thread.join()`
- **Current State**: Tests pass locally, but:
- Pyright type checking fails on 2 lines (app._app could be None)
- CI is likely still failing

## Root Cause Analysis

The issue appears to be that even with context managers, worker threads are processing tasks when the Tk root is destroyed during test teardown. The threads are:
1. Stuck in `queue.get()` waiting for work
2. Or in the middle of processing (calling `get_resized_pil()`)
3. Trying to access Tk (`after_idle`) when Tk is already destroyed

## Current Code State

### ImageWorker Changes
```python
class ImageWorker:
_app: ComicViewer | None = None

def __init__(self, app, num_workers: int = 4):
self._app = app
self._stopped: bool = False
# ... start threads

def stop(self):
self._stopped = True
self._app = None # Prevent access after cleanup
# ... send stop signals to queue

def __enter__(self):
return self

def __exit__(self, exc_type, exc_val, exc_tb):
self.stop()
return False
```

### ComicViewer Changes
```python
class ComicViewer(tk.Frame):
def __del__(self):
self.cleanup()

def cleanup(self):
if hasattr(self, "_worker") and self._worker:
self._worker.stop()
```

## Remaining Issues

### Type Checking Errors ✅ FIXED
- Added `assert app is not None` after null check to satisfy pyright
- Added `if source is None: break` check to handle None source gracefully
- All type checking errors resolved (0 errors, 0 warnings, 0 informations)

### Coverage ✅ FIXED
- Current: 96.00%
- Target: 96%
- Gap: 0% (target reached!)
- Added test `test_worker_handles_none_source_gracefully` to cover source=None path

## Files Modified

### cdisplayagain.py
- Added context manager methods to `ImageWorker`
- Added `_stopped` flag for graceful shutdown
- Added `cleanup()` method to `ComicViewer`
- Modified `_run()` to check `_stopped` flag multiple times
- Modified `stop()` to handle `queue.Full` and set `_app = None`
- ✅ Added `assert app is not None` after null check (fixes pyright errors)
- ✅ Added `if source is None: break` check (handles shutdown gracefully)

### tests/test_parallel_workers.py
- Converted 10 tests from `try/finally` to context manager pattern
- ✅ Added `test_worker_handles_none_source_gracefully` to increase coverage to 96%

### tests/test_threading.py
- Converted 4 tests from `try/finally` to context manager pattern

### tests/test_benchmark_parallel.py
- Converted 2 tests from `try/finally` to context manager pattern

### tests/conftest.py
- Tried adding autouse fixture to track/cleanup all viewers (reverted - caused issues)

## Next Steps

1. ~~**Fix type checking errors**: Use local variable `app = self._app` after null check to satisfy pyright~~ ✅ DONE
2. **Investigate CI-specific issue**: The threading crash still occurs in CI but not locally - suggests:
- Timing difference in CI environment
- Different garbage collection behavior
- xvfb-run virtual display interaction
- ✅ Type checking and coverage are now fixed, ready to test on CI
3. **Alternative approaches to consider if CI still fails**:
- Make workers non-daemon and ensure explicit cleanup
- Add a test-wide fixture that ensures all workers are stopped before any Tk root is destroyed
- Increase timeout in `thread.join()`
- Use a different shutdown mechanism (e.g., event instead of queue sentinel)

## Commands to Run

```bash
# Run tests
uv run pytest tests/ --ignore=tests/test_benchmark_parallel.py

# Run linting
make lint

# Check type errors specifically
uv run pyright cdisplayagain.py
```

## Git Status

- Branch: `move-test-coverage-marker`
- Last commits:
- e179690: Fix CI threading crash by stopping worker threads in tests (try/finally)
- c19ea81: Refactor worker cleanup to use context manager
- Working tree: Dirty (uncommitted changes to cdisplayagain.py and tests/test_parallel_workers.py)
- Fixed type checking errors (assert app is not None)
- Added source=None check in worker loop
- Added test_worker_handles_none_source_gracefully test
- Coverage now at 96.00%, all linting passes
97 changes: 88 additions & 9 deletions cdisplayagain.py
Original file line number Diff line number Diff line change
Expand Up @@ -380,11 +380,14 @@ def get_bytes(_: str) -> bytes:
class ImageWorker:
"""Background thread pool for image processing."""

_app: ComicViewer | None = None

def __init__(self, app, num_workers: int = 4):
"""Initialize worker pool with app reference and start daemon threads."""
self._app = app
self._queue = queue.PriorityQueue(maxsize=4)
self._threads: list[threading.Thread] = []
self._stopped: bool = False
for i in range(num_workers):
thread = threading.Thread(target=self._run, daemon=True, name=f"ImageWorker-{i}")
thread.start()
Expand All @@ -394,6 +397,8 @@ def request_page(
self, index: int, width: int, height: int, preload: bool = False, render_generation: int = 0
):
"""Request a page be processed in background."""
if self._stopped or not self._app:
return
try:
priority = 1 if preload else 0
self._queue.put_nowait((priority, index, width, height, preload, render_generation))
Expand All @@ -408,40 +413,104 @@ def preload(self, index: int):
ch = max(1, self._app.canvas.winfo_height())
self.request_page(index, cw, ch, preload=True)

def stop(self):
"""Signal all worker threads to stop and wait for them to exit."""
self._stopped = True

self._app = None

for _ in self._threads:
try:
self._queue.put_nowait((2, None, None, None, None, None))
except queue.Full:
break

for thread in self._threads:
try:
thread.join(timeout=1.0)
except Exception:
pass
self._threads.clear()

def __enter__(self):
"""Context manager entry."""
return self

def __exit__(self, exc_type, exc_val, exc_tb):
"""Context manager exit - stop workers."""
self.stop()
return False

def _should_stop(self) -> bool:
"""Check if worker should stop processing."""
return self._stopped

def _run(self):
"""Process resize requests in background."""
while True:
while not self._stopped:
priority = None
try:
priority, index, width, height, preload, render_generation = self._queue.get()
priority, index, width, height, preload, render_generation = self._queue.get(
timeout=0.1
)

if priority == 2:
break

if self._should_stop() or not self._app:
break

app = self._app
if not app:
break
assert app is not None

if preload:
logging.info("Worker preloading page %d at %dx%d", index, width, height)
else:
logging.info("Worker processing page %d at %dx%d", index, width, height)

if not preload and render_generation != self._app._render_generation:
if not preload and render_generation != app._render_generation:
logging.info(
"Worker cancelling stale render for page %d (gen %d != %d)",
index,
render_generation,
self._app._render_generation,
app._render_generation,
)
continue

raw = self._app.source.get_bytes(self._app.source.pages[index])
if self._should_stop():
break

source = app.source
if source is None:
break
raw = source.get_bytes(source.pages[index])
resized_pil = get_resized_pil(raw, width, height)

if self._should_stop():
break

if preload:
logging.info("Worker finished preloading page %d", index)
else:
logging.info("Worker finished page %d, scheduling callback", index)

self._app.after_idle(
lambda idx=index, img=resized_pil: self._app._update_from_cache(idx, img)
)

if app and hasattr(app, "after_idle"):
try:
app.after_idle(
lambda idx=index, img=resized_pil, app=app: app._update_from_cache(
idx, img
)
)
except Exception:
pass

except queue.Empty:
continue
except Exception as e:
logging.error("Image worker error: %s", e)
break


def load_comic(path: Path) -> PageSource:
Expand All @@ -468,6 +537,15 @@ def load_comic(path: Path) -> PageSource:
class ComicViewer(tk.Frame):
"""Tk viewer for comic archives and image folders."""

def __del__(self):
"""Cleanup worker threads on garbage collection."""
self.cleanup()

def cleanup(self):
"""Stop worker threads to prevent threading crashes during shutdown."""
if hasattr(self, "_worker") and self._worker:
self._worker.stop()

def __init__(self, master: tk.Tk, comic_path: Path):
"""Initialize the viewer frame and load the initial comic."""
init_start = time.perf_counter()
Expand Down Expand Up @@ -976,6 +1054,7 @@ def _quit(self):
try:
if self.source and self.source.cleanup:
self.source.cleanup()
self._worker.stop()
finally:
logging.info("Destroying app window.")
self.master.destroy()
Expand Down
Loading