Skip to content

Commit 1804c8e

Browse files
feat(tests): add E2E tests with PostgreSQL support (#2847)
## Summary - Move API tests from `tests/e2e/` to `tests/integration/api/` (SQLite in-memory) - Add new E2E tests that run against real PostgreSQL in CI - Update CI workflow with PostgreSQL service container ## Test Structure ``` tests/ ├── unit/ # Fast tests with mocks ├── integration/ # SQLite in-memory (repositories + API) └── e2e/ # Real PostgreSQL (full stack) ``` ## Changes - Rename `ci-unittest.yml` → `ci-tests.yml` with PostgreSQL service - Fix test isolation (cache clearing, session expire) - Remove tests for non-existent endpoints (robots.txt, code download) - Update documentation in CLAUDE.md ## Test plan - [x] All 446 unit/integration tests pass locally - [ ] CI workflow runs with PostgreSQL service - [ ] E2E tests execute against real database in CI 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
1 parent b0bc886 commit 1804c8e

12 files changed

Lines changed: 697 additions & 324 deletions

File tree

.github/copilot-instructions.md

Lines changed: 10 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -46,7 +46,7 @@ This file provides guidance to GitHub Copilot when working with code in this rep
4646
# Install dependencies (uses uv - fast Python package manager)
4747
uv sync --all-extras
4848

49-
# Run all tests (unit + integration)
49+
# Run all tests (unit + integration + e2e)
5050
uv run pytest
5151

5252
# Run only unit tests
@@ -55,13 +55,16 @@ uv run pytest tests/unit
5555
# Run only integration tests (uses SQLite in-memory)
5656
uv run pytest tests/integration
5757

58-
# Check code formatting and linting
58+
# Run only E2E tests (requires DATABASE_URL)
59+
uv run pytest tests/e2e
60+
61+
# Linting (required for CI)
5962
uv run ruff check .
6063

61-
# Auto-fix issues
64+
# Auto-fix linting issues
6265
uv run ruff check . --fix
6366

64-
# Format code
67+
# Formatting (required for CI)
6568
uv run ruff format .
6669
```
6770

@@ -164,9 +167,9 @@ Examples: `scatter-basic`, `scatter-color-mapped`, `bar-grouped-horizontal`, `he
164167
- **Naming**: `test_{what_it_does}`
165168
- **Fixtures**: Use pytest fixtures in `tests/conftest.py`
166169
- **Test Types**:
167-
- **Unit tests** (`tests/unit/`): Fast, isolated, mocked dependencies - run in CI
168-
- **Integration tests** (`tests/integration/`): Real database with SQLite in-memory - run in CI
169-
- **E2E tests** (`tests/e2e/`): Full stack with FastAPI TestClient - not yet in CI
170+
- **Unit tests** (`tests/unit/`): Fast, isolated, mocked dependencies
171+
- **Integration tests** (`tests/integration/`): SQLite in-memory for API tests
172+
- **E2E tests** (`tests/e2e/`): Real PostgreSQL with isolated `test_e2e` schema (skipped if DATABASE_URL not set)
170173
- **Markers**: Use `@pytest.mark.unit`, `@pytest.mark.integration`, `@pytest.mark.e2e`
171174

172175
### Plot Implementation Style (KISS)
Lines changed: 18 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
name: "CI: Unit Tests"
1+
name: "CI: Tests"
22
run-name: "Tests: ${{ github.ref_name }}"
33

44
on:
@@ -16,11 +16,16 @@ on:
1616

1717
jobs:
1818
test:
19-
name: Run Unit Tests
19+
name: Run Tests
2020
runs-on: ubuntu-latest
2121
permissions:
2222
contents: read
2323

24+
env:
25+
# E2E tests use isolated test_e2e schema (production data unaffected)
26+
DATABASE_URL: ${{ secrets.DATABASE_URL }}
27+
ENVIRONMENT: test
28+
2429
steps:
2530
- uses: actions/checkout@v6
2631
with:
@@ -66,11 +71,18 @@ jobs:
6671
if: steps.check.outputs.should_test == 'true'
6772
run: uv sync --extra test
6873

69-
- name: Run unit and integration tests with coverage
74+
- name: Run all tests with coverage
7075
if: steps.check.outputs.should_test == 'true'
71-
run: uv run pytest tests/unit tests/integration -v --tb=short --cov=core --cov=api --cov-report=term-missing --cov-report=xml
72-
# Note: E2E tests require database dependency override (not implemented yet)
73-
# Integration tests now work with SQLite using custom database types
76+
run: |
77+
uv run pytest tests/unit tests/integration tests/e2e \
78+
-v --tb=short \
79+
--cov=core --cov=api \
80+
--cov-report=term-missing \
81+
--cov-report=xml
82+
# Tests include:
83+
# - Unit tests: Fast, mocked dependencies
84+
# - Integration tests: SQLite in-memory for repository layer
85+
# - E2E tests: Real PostgreSQL with isolated test_e2e schema
7486

7587
- name: Upload coverage to GitHub Artifacts
7688
if: steps.check.outputs.should_test == 'true' && always()

CLAUDE.md

Lines changed: 16 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -129,7 +129,7 @@ uv run alembic upgrade head
129129
### Testing
130130

131131
```bash
132-
# Run all tests (unit + integration)
132+
# Run all tests (unit + integration + e2e)
133133
uv run pytest
134134

135135
# Run only unit tests
@@ -138,6 +138,9 @@ uv run pytest tests/unit
138138
# Run only integration tests (uses SQLite in-memory)
139139
uv run pytest tests/integration
140140

141+
# Run only E2E tests (requires DATABASE_URL)
142+
uv run pytest tests/e2e
143+
141144
# Run with coverage
142145
uv run pytest --cov=. --cov-report=html
143146

@@ -149,22 +152,27 @@ uv run pytest tests/unit/api/test_routers.py::test_get_specs
149152
```
150153

151154
**Test Infrastructure**:
152-
- **Unit tests** (`tests/unit/`): Fast, mocked dependencies, run in CI
153-
- **Integration tests** (`tests/integration/`): Real database operations with SQLite, run in CI
154-
- **E2E tests** (`tests/e2e/`): Full stack with FastAPI TestClient, not yet in CI
155+
- **Unit tests** (`tests/unit/`): Fast, mocked dependencies
156+
- **Integration tests** (`tests/integration/`): SQLite in-memory for API tests
157+
- **E2E tests** (`tests/e2e/`): Real PostgreSQL with isolated `test_e2e` schema
155158

156-
**Database for Tests**: Integration tests use SQLite in-memory (via custom types in `core/database/types.py`). Production uses PostgreSQL native types (ARRAY, JSONB, UUID), tests use compatible fallbacks (JSON, String).
159+
**Database for Tests**:
160+
- **Unit/Integration**: SQLite in-memory (via custom types in `core/database/types.py`)
161+
- **E2E**: PostgreSQL with `test_e2e` schema (auto-created, auto-dropped)
162+
- E2E tests are skipped if `DATABASE_URL` is not set
157163

158164
### Code Quality
159165

166+
**Both linting and formatting must pass for CI.**
167+
160168
```bash
161-
# Check code formatting and linting
169+
# Linting (required for CI)
162170
uv run ruff check .
163171

164-
# Auto-fix issues
172+
# Auto-fix linting issues
165173
uv run ruff check . --fix
166174

167-
# Format code
175+
# Formatting (required for CI)
168176
uv run ruff format .
169177
```
170178

README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
[![Python 3.13+](https://img.shields.io/badge/python-3.13+-blue.svg)](https://www.python.org/)
44
[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](LICENSE)
5-
[![Tests](https://github.com/MarkusNeusinger/pyplots/actions/workflows/ci-unittest.yml/badge.svg?branch=main)](https://github.com/MarkusNeusinger/pyplots/actions/workflows/ci-unittest.yml)
5+
[![Tests](https://github.com/MarkusNeusinger/pyplots/actions/workflows/ci-tests.yml/badge.svg?branch=main)](https://github.com/MarkusNeusinger/pyplots/actions/workflows/ci-tests.yml)
66
[![Ruff](https://github.com/MarkusNeusinger/pyplots/actions/workflows/ci-lint.yml/badge.svg?branch=main)](https://github.com/MarkusNeusinger/pyplots/actions/workflows/ci-lint.yml)
77
[![codecov](https://codecov.io/github/MarkusNeusinger/pyplots/graph/badge.svg?token=4EGPSHH0H0)](https://codecov.io/github/MarkusNeusinger/pyplots)
88

tests/conftest.py

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -154,4 +154,7 @@ async def test_db_with_data(test_session):
154154
test_session.add_all([scatter_matplotlib, scatter_seaborn, bar_matplotlib])
155155
await test_session.commit()
156156

157+
# Expire all cached objects to ensure fresh loading with relationships
158+
test_session.expire_all()
159+
157160
yield test_session

tests/e2e/__init__.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
"""End-to-End tests for pyplots API."""
1+
"""E2E tests with real PostgreSQL database (isolated test_e2e schema)."""

tests/e2e/conftest.py

Lines changed: 173 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,173 @@
1+
"""
2+
E2E test fixtures with real PostgreSQL database.
3+
4+
Uses a separate 'test_e2e' schema to isolate test data from production.
5+
Tests are skipped if DATABASE_URL is not set.
6+
7+
Note: These tests must NOT be run with pytest-xdist parallelization
8+
as multiple workers would conflict on the shared test_e2e schema.
9+
"""
10+
11+
import os
12+
13+
import pytest
14+
import pytest_asyncio
15+
from sqlalchemy import text
16+
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker, create_async_engine
17+
18+
from core.database.models import Base
19+
20+
21+
TEST_SCHEMA = "test_e2e"
22+
23+
# Test data constants
24+
TEST_IMAGE_URL = "https://storage.googleapis.com/pyplots-images/test/plot.png"
25+
TEST_THUMB_URL = "https://storage.googleapis.com/pyplots-images/test/thumb.png"
26+
27+
28+
def _get_database_url():
29+
"""Get DATABASE_URL from environment, loading .env if needed."""
30+
from dotenv import load_dotenv
31+
32+
load_dotenv()
33+
return os.environ.get("DATABASE_URL")
34+
35+
36+
@pytest_asyncio.fixture(scope="function")
37+
async def pg_engine():
38+
"""
39+
Create PostgreSQL engine and setup test schema.
40+
41+
Creates a separate 'test_e2e' schema to isolate tests from production data.
42+
The schema is dropped and recreated for each test.
43+
"""
44+
database_url = _get_database_url()
45+
if not database_url:
46+
pytest.skip("DATABASE_URL not set - skipping PostgreSQL E2E tests")
47+
48+
# First create schema with a temporary engine (no search_path yet)
49+
temp_engine = create_async_engine(database_url, echo=False)
50+
async with temp_engine.begin() as conn:
51+
await conn.execute(text(f"DROP SCHEMA IF EXISTS {TEST_SCHEMA} CASCADE"))
52+
await conn.execute(text(f"CREATE SCHEMA {TEST_SCHEMA}"))
53+
await temp_engine.dispose()
54+
55+
# Create engine with search_path set at connection level (handles pooling correctly)
56+
engine = create_async_engine(
57+
database_url, echo=False, connect_args={"server_settings": {"search_path": TEST_SCHEMA}}
58+
)
59+
60+
# Create tables in test schema
61+
async with engine.begin() as conn:
62+
await conn.run_sync(Base.metadata.create_all)
63+
64+
yield engine
65+
66+
# Cleanup: Drop entire test schema
67+
async with engine.begin() as conn:
68+
await conn.execute(text(f"DROP SCHEMA IF EXISTS {TEST_SCHEMA} CASCADE"))
69+
await engine.dispose()
70+
71+
72+
@pytest_asyncio.fixture
73+
async def pg_session(pg_engine):
74+
"""Create session with test schema (search_path set at engine level)."""
75+
async_session = async_sessionmaker(pg_engine, class_=AsyncSession, expire_on_commit=False)
76+
async with async_session() as session:
77+
yield session
78+
await session.rollback()
79+
80+
81+
@pytest_asyncio.fixture
82+
async def pg_db_with_data(pg_session):
83+
"""
84+
Seed test schema with sample data.
85+
86+
Creates the same test data as tests/conftest.py:test_db_with_data
87+
but in the PostgreSQL test schema.
88+
"""
89+
from core.database.models import Impl, Library, Spec
90+
91+
# Create libraries
92+
matplotlib_lib = Library(
93+
id="matplotlib",
94+
name="Matplotlib",
95+
version="3.10.0",
96+
documentation_url="https://matplotlib.org",
97+
description="Comprehensive library for visualizations",
98+
)
99+
seaborn_lib = Library(
100+
id="seaborn",
101+
name="Seaborn",
102+
version="0.13.0",
103+
documentation_url="https://seaborn.pydata.org",
104+
description="Statistical data visualization",
105+
)
106+
pg_session.add_all([matplotlib_lib, seaborn_lib])
107+
108+
# Create specs
109+
scatter_spec = Spec(
110+
id="scatter-basic",
111+
title="Basic Scatter Plot",
112+
description="A basic scatter plot",
113+
applications=["data visualization", "correlation analysis"],
114+
data=["numeric"],
115+
notes=["Use for 2D data"],
116+
tags={"plot_type": ["scatter"], "domain": ["statistics"], "data_type": ["numeric"], "features": ["basic"]},
117+
issue=42,
118+
suggested="contributor",
119+
)
120+
bar_spec = Spec(
121+
id="bar-grouped",
122+
title="Grouped Bar Chart",
123+
description="A grouped bar chart",
124+
applications=["comparisons"],
125+
data=["categorical", "numeric"],
126+
notes=["Good for comparing categories"],
127+
tags={"plot_type": ["bar"], "domain": ["statistics"], "data_type": ["categorical"], "features": ["grouped"]},
128+
issue=43,
129+
suggested="contributor2",
130+
)
131+
pg_session.add_all([scatter_spec, bar_spec])
132+
await pg_session.commit()
133+
134+
# Create implementations
135+
scatter_matplotlib = Impl(
136+
spec_id="scatter-basic",
137+
library_id="matplotlib",
138+
code="import matplotlib.pyplot as plt\n# scatter plot code",
139+
preview_url=TEST_IMAGE_URL.replace("plot", "scatter-matplotlib"),
140+
preview_thumb=TEST_THUMB_URL.replace("thumb", "scatter-matplotlib-thumb"),
141+
quality_score=92.5,
142+
generated_by="claude",
143+
python_version="3.13",
144+
library_version="3.10.0",
145+
)
146+
scatter_seaborn = Impl(
147+
spec_id="scatter-basic",
148+
library_id="seaborn",
149+
code="import seaborn as sns\n# scatter plot code",
150+
preview_url=TEST_IMAGE_URL.replace("plot", "scatter-seaborn"),
151+
preview_thumb=TEST_THUMB_URL.replace("thumb", "scatter-seaborn-thumb"),
152+
quality_score=95.0,
153+
generated_by="claude",
154+
python_version="3.13",
155+
library_version="0.13.0",
156+
)
157+
bar_matplotlib = Impl(
158+
spec_id="bar-grouped",
159+
library_id="matplotlib",
160+
code="import matplotlib.pyplot as plt\n# bar chart code",
161+
preview_url=TEST_IMAGE_URL.replace("plot", "bar-matplotlib"),
162+
quality_score=88.0,
163+
generated_by="claude",
164+
python_version="3.13",
165+
library_version="3.10.0",
166+
)
167+
pg_session.add_all([scatter_matplotlib, scatter_seaborn, bar_matplotlib])
168+
await pg_session.commit()
169+
170+
# Expire all cached objects to ensure fresh loading with relationships
171+
pg_session.expire_all()
172+
173+
yield pg_session

0 commit comments

Comments
 (0)