Add contract-level test suite with CI matrix (Ubuntu/Windows)

20 pytest contract tests over the stable behavior layer: SQLite
repositories, pure utils, and chat API request construction. Tests pin
behavioral promises, never implementation structure, so forks can
refactor internals freely (philosophy documented in TESTING.md).

- tests/ run hermetically: import-time side effects are sandboxed into
  a temp dir, network access is blocked at the socket level
- GitHub Actions matrix runs the suite on ubuntu + windows (Python 3.10)
- UI, CSS, audio transcription and live model calls deliberately stay
  untested/manual (see TESTING.md)

Author: Leon Sander

.github/workflows/tests.yml

@@ -0,0 +1,41 @@
+name: tests
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+    branches: [main]
+
+jobs:
+  tests:
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-latest, windows-latest]
+        python-version: ["3.10"]
+    runs-on: ${{ matrix.os }}
+    timeout-minutes: 20
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+          cache: pip
+          cache-dependency-path: requirements*.txt
+
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install -r requirements.txt -r requirements-dev.txt
+
+      # Catches import-time breakage (config paths, heavy deps) with a clear
+      # error before the test run. torch is deliberately NOT installed: only
+      # audio_handler needs it, and audio stays manual (see TESTING.md).
+      - name: Smoke-import the modules under test
+        run: python -c "import chat_api_handler, database_operations, utils"
+        env:
+          PYTHONPATH: src
+
+      - name: Run tests
+        run: pytest

README.md

@@ -116,6 +116,17 @@ Running Ollama inside Docker is slow on Windows (system calls are translated bet
 
 Key settings live in `config.yaml` (Ollama base URL and embedding model, Whisper model, Chroma path/collection, chat-sessions DB path). Runtime options (endpoint, model, chunk size/overlap, retrieved chunks, chat memory) are adjustable in the sidebar and persisted per user.
 
+## Testing
+
+A small contract-level test suite covers the SQLite layer, utility functions, and API request construction (UI, audio, and real model calls stay manual by design):
+
+```bash
+pip install -r requirements-dev.txt
+pytest
+```
+
+CI runs the suite on Ubuntu and Windows. See [TESTING.md](./TESTING.md) for the philosophy and what is deliberately not tested.
+
 ## Roadmap
 
 - [ ] Additional model providers (Gemini, others)

TESTING.md

@@ -0,0 +1,58 @@
+# Testing
+
+This repo has a small, deliberate test suite: **20 contract tests** over the parts of the code whose behavior is stable by design. It is meant to stay cheap to maintain — including for forks.
+
+## Philosophy: contracts, not structure
+
+A test here pins a **behavioral promise** (what a function guarantees to its callers), never an implementation detail. The intended property: **a test only fails when behavior actually changed.** If you fork this repo, rewrite internals freely — a red test means you changed something user-visible, which is exactly when you want to know.
+
+A few places accept **medium churn** deliberately, each documented inline where it occurs:
+
+- `test_pdf_chat_stuffs_retrieved_context_into_prompt` patches the internal `load_vectordb` seam (the alternative is a real Chroma + Ollama instance).
+- `test_command_dispatch` patches the internal `pull_model_in_background` seam (the alternative is asyncio + a live Ollama pull).
+- The handler tests patch two values that src binds **at import time** (`chat_api_handler.config`, `chat_api_handler.openai_api_key`). A behavior-preserving refactor to call-time lookups would require updating the two fixtures at the top of `tests/test_chat_api_handler.py` — a two-line fix, accepted as the cost of avoiding real env vars and network.
+
+## What is covered
+
+| File | Contracts |
+|------|-----------|
+| `tests/test_database_operations.py` (9) | message roundtrips (text + binary), LLM-context feed ordering/filtering, per-session delete isolation, session id listing, settings read-through defaults + persistence, restart durability |
+| `tests/test_utils.py` (3) | base64/data-URL wire format, `/command` dispatch, config save/load roundtrip |
+| `tests/test_chat_api_handler.py` (8) | endpoint routing (ollama/openai/unknown), request bodies + auth headers, image wire formats (Ollama bare-b64 vs OpenAI data URL), RAG context stuffing, error-shape handling (string vs nested object) |
+
+## What deliberately stays untested (and why)
+
+- **UI structure and CSS** (`app.py`, `html_templates.py`) — Streamlit class names churn every release; tests would break without any behavior change.
+- **Audio transcription** (`audio_handler.py`) — needs Whisper model downloads; the browser microphone path can't be unit-tested anyway.
+- **Real model calls / PDF ingestion** — network- and model-dependent; the request *construction* is tested, the live call is not.
+- **Trivial helpers** (`convert_ns_to_seconds`, `get_avatar`, timestamp formatting) — near-zero regression value for the count they add.
+- **The manual smoke test remains**: launch the app, send a text message, record audio once. That is the irreducible browser part.
+
+## Running
+
+From the **repo root**:
+
+```bash
+pip install -r requirements-dev.txt   # includes requirements.txt
+pytest
+```
+
+CI runs the same suite on Ubuntu and Windows (Python 3.10) on every push/PR to `main` — which also doubles as a "does it still install on Windows" check.
+
+## Gotchas when writing new tests
+
+The src modules bind several things **at import time** — patch the right namespace:
+
+| To fake | Patch | Why |
+|---------|-------|-----|
+| OpenAI key | `chat_api_handler.openai_api_key` | `os.getenv` ran at import; `monkeypatch.setenv` does nothing |
+| Ollama base URL | `chat_api_handler.config` | module-level `load_config()` froze it |
+| Vector DB | `chat_api_handler.load_vectordb` | from-import binding; patching `vectordb_handler` does nothing |
+| HTTP | `requests.post` (module attribute) | call-time lookup, reaches all src modules |
+| Session state | `streamlit.session_state` → `FakeSessionState` (conftest) | never rely on bare-mode session_state |
+
+Other invariants:
+
+- An autouse `no_network` fixture replaces `requests.get/post` and blocks raw socket connects (which also covers `aiohttp` and `requests.Session`) — a test that reaches for the network fails loudly instead of silently hitting a real endpoint.
+- DB tests build their own `DatabaseManager` on `tmp_path` and must `close()` it (Windows cannot delete open sqlite files). Never assert through the module-level singleton in `database_operations`.
+- `tests/conftest.py` chdirs into a throwaway **sandbox** (with its own minimal `config.yaml` and `chat_sessions/`) before anything imports the src modules — their import-time side effects (config read, sqlite creation) land in the sandbox, the repo's real `config.yaml` and `chat_sessions/` are never read or written, and pytest works from any invocation directory.

pytest.ini

@@ -0,0 +1,3 @@
+[pytest]
+pythonpath = src
+testpaths = tests

requirements-dev.txt

@@ -0,0 +1,2 @@
+-r requirements.txt
+pytest==9.1.1

tests/conftest.py

@@ -0,0 +1,149 @@
+"""Shared fixtures for the contract-level test suite.
+
+IMPORTANT - import-time side effects in src/ are load-bearing: utils.py,
+chat_api_handler.py and database_operations.py read the CWD-relative
+config.yaml when first imported, and database_operations.py opens a sqlite
+database at the configured path at import time. The module-level code below
+therefore chdirs into a throwaway sandbox seeded with a minimal config.yaml
+BEFORE any test module (and hence any src module) is imported: every
+import-time side effect lands in the sandbox, the repo's real config.yaml and
+chat_sessions/ are never read or written, and the suite works from any
+invocation directory.
+"""
+import atexit
+import os
+import shutil
+import socket
+import tempfile
+from pathlib import Path
+
+import pytest
+import requests
+
+_SANDBOX = Path(tempfile.mkdtemp(prefix="local-multimodal-chat-tests-"))
+(_SANDBOX / "chat_sessions").mkdir()
+(_SANDBOX / "config.yaml").write_text(
+    "\n".join(
+        [
+            "ollama:",
+            '  embedding_model: "sandbox-embed"',
+            "  base_url: http://sandbox-ollama:11434",
+            "",
+            'whisper_model: "sandbox-whisper"',
+            "",
+            "chromadb:",
+            '  chromadb_path: "chroma_db"',
+            '  collection_name: "pdfs"',
+            "",
+            'chat_sessions_database_path: "./chat_sessions/chat_sessions.db"',
+            "",
+        ]
+    )
+)
+# Best-effort cleanup; ignore_errors because the module-level singleton in
+# database_operations holds its sandbox db open for the whole session.
+atexit.register(shutil.rmtree, _SANDBOX, ignore_errors=True)
+
+
+def pytest_configure(config):
+    # chdir in the configure hook, not at conftest import: pytest resolves
+    # testpaths against the CWD first, and collection (which imports the test
+    # modules and therefore the src modules) only starts afterwards - so the
+    # import-time side effects still land in the sandbox.
+    os.chdir(_SANDBOX)
+
+
+class FakeSessionState(dict):
+    """Stand-in for st.session_state covering the four access patterns used in
+    src/: st.session_state["key"], .get("key", default), attribute access, and
+    membership tests. Never rely on Streamlit's bare-mode session_state - it is
+    a process-global singleton with undocumented semantics."""
+
+    def __getattr__(self, name):
+        try:
+            return self[name]
+        except KeyError:
+            raise AttributeError(name) from None
+
+    def __setattr__(self, name, value):
+        self[name] = value
+
+
+class DummyResponse:
+    """Minimal requests.Response stand-in; .json() is repeatably callable."""
+
+    def __init__(self, payload):
+        self._payload = payload
+
+    def json(self):
+        return self._payload
+
+
+class RecordingPost:
+    """Callable that records every requests.post(...) call and returns a
+    canned response. The signature mirrors the real
+    requests.post(url, data=None, json=None, ...) so that a refactor in src to
+    positional arguments cannot silently record the body under the wrong key."""
+
+    def __init__(self, response_payload):
+        self.response_payload = response_payload
+        self.calls = []
+
+    def __call__(self, url=None, data=None, json=None, headers=None, **kwargs):
+        self.calls.append({"url": url, "data": data, "json": json, "headers": headers})
+        return DummyResponse(self.response_payload)
+
+
+@pytest.fixture(autouse=True)
+def no_network(monkeypatch):
+    """Fail loudly if any test hits the network without explicitly patching
+    it: requests.get/post are replaced for a clear error message, and raw
+    socket connects are blocked as a catch-all (this also covers aiohttp and
+    requests.Session). captured_post overrides requests.post for tests that
+    expect an HTTP call."""
+
+    def _blocked(*args, **kwargs):
+        raise RuntimeError(
+            "Network access is disabled in tests; use the captured_post "
+            "fixture or patch the network call explicitly."
+        )
+
+    monkeypatch.setattr(requests, "get", _blocked)
+    monkeypatch.setattr(requests, "post", _blocked)
+    monkeypatch.setattr(socket.socket, "connect", _blocked)
+
+
+@pytest.fixture
+def captured_post(monkeypatch):
+    """Factory: captured_post(payload) patches requests.post to record calls
+    and answer with `payload`. Works because src modules do `import requests`
+    and look up requests.post at call time."""
+
+    def _install(response_payload):
+        recorder = RecordingPost(response_payload)
+        monkeypatch.setattr(requests, "post", recorder)
+        return recorder
+
+    return _install
+
+
+@pytest.fixture
+def fake_session_state(monkeypatch):
+    """Replace st.session_state for every src module at once (they all do
+    `import streamlit as st`, so patching the module attribute reaches all)."""
+    import streamlit as st
+
+    state = FakeSessionState()
+    monkeypatch.setattr(st, "session_state", state)
+    return state
+
+
+@pytest.fixture
+def db_manager(tmp_path):
+    """A DatabaseManager on a throwaway sqlite file. close() in teardown is
+    mandatory: an open handle makes tmp_path cleanup fail on Windows."""
+    from database_operations import DatabaseManager
+
+    manager = DatabaseManager(str(tmp_path / "test.db"))
+    yield manager
+    manager.close()