Skip to content

Commit 8218b80

Browse files
SimplicityGuygithub-actions[bot]claude
authored
feat: Discogs user integration — authenticate, sync collection & wantlist (#66)
* feat(auth): Add auth microservice with user accounts and JWT authentication (#56) Implements Step 1 of the Discogs user integration (issue #60): - New `auth` microservice (FastAPI, ports 8004/8005) - POST /api/auth/register — user registration with PBKDF2-SHA256 password hashing - POST /api/auth/login — authentication with HS256 JWT tokens (stdlib only, no deps) - GET /api/auth/me — current user info via JWT Bearer auth - GET /health — health check endpoint - PostgreSQL schema additions (via schema-init): - `users` table — email, hashed_password, is_active, timestamps - `oauth_tokens` table — per-user Discogs OAuth tokens (prepared for Step 2) - `app_config` table — admin-managed Discogs app credentials - `user_collections` / `user_wantlists` tables — user personal data (prepared for Steps 3-4) - `sync_history` table — sync job tracking - Common config: added `AuthConfig` dataclass with JWT + Postgres settings - docker-compose.yml: auth service added with health check on port 8005 - All auth crypto uses Python stdlib (hashlib, hmac, base64) — no new package deps Co-authored-by: Robert Wlodarczyk <SimplicityGuy@users.noreply.github.com> * feat(auth): Implement Discogs OAuth 1.0a OOB flow (#57) Implements Step 2 of the Discogs user integration (issue #60): - auth/services/discogs.py — DiscogsOAuth1Auth helpers: - HMAC-SHA1 OAuth signature generation (stdlib only) - request_oauth_token() — OOB flow with callback_uri="oob" - exchange_oauth_verifier() — exchange verifier for access token - fetch_discogs_identity() — get /oauth/identity - New OAuth endpoints in auth/auth.py: - GET /api/oauth/authorize/discogs — start OOB flow, store state in Redis - POST /api/oauth/verify/discogs — exchange verifier, store tokens, fetch identity - GET /api/oauth/status/discogs — check connection status - DELETE /api/oauth/revoke/discogs — disconnect Discogs account - PUT /api/admin/config/{key} — admin endpoint for Discogs app credentials - AuthConfig extended with redis_url and discogs_user_agent fields - Redis initialized in auth service lifespan for 10-min OOB state TTL Co-authored-by: Robert Wlodarczyk <SimplicityGuy@users.noreply.github.com> * feat(collector): add collector microservice for Discogs collection/wantlist sync Implements step 3 of issue #60 — a new `collector` microservice that syncs each authenticated user's Discogs vinyl collection and wantlist into both PostgreSQL and Neo4j. Key additions: - collector/collector.py — FastAPI service (port 8010/8011) with JWT authentication; POST /api/sync triggers background sync, GET /api/sync/status returns history - collector/syncer.py — paginated Discogs API sync with OAuth 1.0a signing, upserts to user_collections / user_wantlists tables and COLLECTED / WANTS Neo4j relationships on existing Release nodes - collector/Dockerfile — multi-stage build following project patterns - collector/pyproject.toml — service-specific dependency manifest - common/config.py — adds CollectorConfig (postgres + neo4j + jwt) - schema-init/neo4j_schema.py — adds User.id uniqueness constraint - docker-compose.yml — adds collector service (ports 8010/8011) - pyproject.toml — adds collector optional-dependency group; updates all extra and workspace membership - tests/collector/ — unit tests for OAuth helpers and constants Closes #58 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(explore): add personalized user endpoints for collection and wantlist Implements step 4 of issue #60 — extends the explore service with personalized Neo4j-backed endpoints for authenticated users. New endpoints (all require Bearer JWT matching auth/collector services): GET /api/user/collection — paginated list of COLLECTED releases GET /api/user/wantlist — paginated list of WANTS releases GET /api/user/recommendations — 'you may also like' from collection artists GET /api/user/collection/stats — breakdown by genre, decade, and label GET /api/user/status — check in_collection/in_wantlist for a set of release IDs (optional auth, defaults false) Supporting changes: - explore/user_queries.py — all Neo4j Cypher for personalization; pure Neo4j (no PostgreSQL dependency), leverages User node and COLLECTED/WANTS relationships written by the collector service - explore/explore.py — JWT verification via stdlib HMAC-SHA256, optional _get_optional_user dependency for decoration, _require_user for protected routes; JWT_SECRET_KEY is optional (endpoints return 503 if not configured) - common/config.py — adds optional jwt_secret_key field to ExploreConfig - explore/Dockerfile — exposes JWT_SECRET_KEY env var - docker-compose.yml — sets JWT_SECRET_KEY for explore (must match auth) - tests/explore/test_user_queries.py — unit tests for JWT verification and check_releases_user_status helper Closes #59 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(extractor): cache file list to avoid double scraping Discogs website When FORCE_REPROCESS is set, list_s3_files was called twice per run: once in process_discogs_data to detect the version, and again inside download_discogs_data. This caused two full scrapes of the Discogs website on every startup. Added cached_files field to Downloader so the result of the first scrape is reused within the same process_discogs_data invocation. The cache is naturally reset on each run since a new Downloader is created per invocation, ensuring periodic checks always fetch fresh data from the website. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(dashboard): fix queue metrics display and expand rate gauges - Fix DLQ poisoning: build separate graphinatorMap/tableInatorMap, skipping queues ending in .dlq, so dead-letter queues no longer overwrite real data with zeros on every update cycle - Bar chart now shows two bars per type: graphinator (purple) and tableinator (blue) message counts, each with its own scale - Split single "Processing Rates" panel into two side-by-side panels: "Publish Rates" and "Ack Rates", each with a graphinator row (purple) and tableinator row (blue) — 16 gauges total - Each gauge is self-normalizing: fill reflects current rate relative to that gauge's own observed max, with a live min–max label appended below each gauge label - Update bar chart legend from "Messages/Ready" to "Graphinator/Tableinator" Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(tests): update queue section display test to match redesigned dashboard The dashboard redesign split the single "Processing Rates" section into separate "Publish Rates (msg/s)" and "Ack Rates (msg/s)" panels, and removed the #processing-rates-grid element. Update the E2E test to check for the actual h2 headings and a known rate circle ID instead. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(dashboard): fix bar chart rendering, DLQ toggle, and Neo4j status badge - Fix bar chart bars not rendering: bar group containers lacked an established height so percentage heights resolved to zero; add h-full to each group flex-col and flex-1 to each bar-pair container so CSS percentage heights compute correctly against the h-64 chart area - Fix DLQ toggle doing nothing: wire change event listener to new _onDlqToggle() handler; build separate graphinatorDlqMap / tableInatorDlqMap alongside regular maps; store currentMaps so the toggle can re-render instantly without waiting for the next WebSocket update; update bar chart legend to show "Graphinator DLQ / Tableinator DLQ" when active - Fix Neo4j status badge showing "Primary" instead of health status: change badge text to "Healthy" / "Unavailable" to match PostgreSQL Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * chore: add .playwright-mcp/ to .gitignore Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * chore: update deps * chore: mdformat * feat(dashboard): format PostgreSQL DB size with thousands separator and GB/MB units Display DB size in MB (with thousands comma) for values under 1 GB, and in GB for values >= 1 GB. Switches from pg_size_pretty() to raw pg_database_size() bytes and formats in Python. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(dashboard): update logo to four-quadrant circle with center diamond Replace the placeholder rotated-square mark with the new Discogsography logo: a dark-bordered circle with terracotta, steel-blue, teal, and olive quadrants, a radial depth shadow, and a white rotated-diamond center. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(ci): resolve all CI failures for PR #66 Discogs user integration - validate-compose: add auth and collector to expected services list - test-schema-init: update neo4j statement count (14→15) and postgres expected_calls formula to include _USER_TABLES entries - test-explore: fix AsyncMock setup for driver.session in _make_driver() and rename unused `self` to `_self` in _aiter (ARG001) - code-quality ruff: add `from exc` chaining to bare raises in except blocks (B904), add noqa S105/S106 suppressions for password-like string literals in tests and constants - code-quality bandit: add nosec B104 to uvicorn.run host="0.0.0.0" lines, nosec B105 to bearer token_type, nosec B107 to token_secret default parameter - mypy: add AsyncGenerator[None] return type to lifespan functions, remove stale type: ignore comments, add driver property to AsyncResilientNeo4jDriver returning self._connection Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * refactor(collector): use AsyncResilientNeo4jDriver directly, matching explore pattern Instead of unwrapping _neo4j.driver to get the raw AsyncDriver and calling execute_query(), pass the AsyncResilientNeo4jDriver directly and use its session() method — the same pattern used by explore and dashboard. This removes the need for the .driver property that was added to AsyncResilientNeo4jDriver and keeps Neo4j access consistently routed through the resilient wrapper with its circuit breaker and retry logic. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * refactor: rename auth→api and collector→curator services auth → api: the service will grow to host all future API endpoints, not just authentication. collector → curator: avoids confusion with the extractor service (which processes bulk Discogs XML dumps); the curator service manages a user's personal Discogs collection and wantlist sync. Renames: - auth/ → api/, auth/auth.py → api/api.py - collector/ → curator/, collector/collector.py → curator/curator.py - tests/auth/ → tests/api/, test_auth_models.py → test_api_models.py - tests/collector/ → tests/curator/, test_collector_syncer.py → test_curator_syncer.py Updates all references: imports, config classes (AuthConfig→ApiConfig, CollectorConfig→CuratorConfig), port constants, docker-compose service names/images/volumes, pyproject.toml extras and workspace members, GitHub Actions expected services list, log file paths, and OCI labels. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(ci): add api and curator services to workflows and compose files - Add api and curator to list-sub-projects matrix for Docker builds - Add test-api and test-curator jobs to test.yml with Codecov upload - Update aggregate-results to include both new test jobs - Add uv pip install for api and curator in code-quality.yml - Add test-api and test-curator recipes to justfile - Add api and curator to justfile test-parallel and docker build targets - Add api and curator to docker-compose.prod.yml with JWT_SECRET_KEY and postgres credential overrides, ordered to match base file - Fix property ordering in docker-compose.yml: networks before healthcheck for all infrastructure services, volumes before depends_on for api/curator Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * docs: add api and curator services to all documentation - Update README.md: add api/curator to Core Services table and Mermaid architecture diagram with new nodes and connections - Update docs/architecture.md: add api/curator to service components table, both Mermaid diagrams, component detail sections, health checks, Redis cache description, and scalability section - Update docs/configuration.md: update "Used By" for Neo4j, PostgreSQL, and Redis; add JWT Configuration section; add api and curator service-specific config blocks; update health checks and env templates - Update docs/monitoring.md: add api/curator to Services Monitored, health check curl commands, and automated monitoring script - Update docs/quick-start.md: add api/curator to service access table, health checks, and run commands - Update docs/emoji-guide.md: add 🔐 API and 🗂️ Curator identifiers - Update docs/task-automation.md: add test-api and test-curator to Test Group table - Update CLAUDE.md: add api (8004/8005) and curator (8010/8011) ports - Create api/README.md: full service documentation with endpoints, JWT auth, Discogs OAuth flow, configuration, and DB schema - Create curator/README.md: full service documentation with endpoints, sync flow, JWT validation, configuration, and DB schema Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(just): add api and curator service run commands - Add `just api` recipe to run the API service (user accounts & JWT auth) - Add `just curator` recipe to run the curator service (collection sync) - Update docs/task-automation.md Services Group table to accurately reflect current justfile: add api (8004) and curator (8010), add explore (8006), remove extractor and schema-init which have no local run commands in the services group Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(just): add schema-init and extractor to services group - Add `just schema-init` to run the one-shot schema initialiser - Add `just extractor` as a services-group alias for extractor-run - Update docs/task-automation.md Services Group table to include both Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(docs): use US spelling (initializer, not initialiser) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(docs): use US English spelling throughout Replace all UK English spellings with US equivalents: - initialiser → initializer (5 instances) - initialised → initialized (2 instances) - initialisation → initialization (1 instance) - behaviour → behavior (1 instance) - catalogue → catalog (1 instance) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * chore: mdformat * test: add coverage for api, curator, explore, and common modules Addresses Codecov feedback from PR #66 by adding comprehensive unit tests targeting 85%+ coverage across the 7 flagged files. Coverage improvements: - api/api.py: 0% → 85% (auth, register, login, OAuth endpoints) - api/services/discogs.py: 0% → 100% (OAuth 1.0a, HMAC-SHA1) - curator/curator.py: 0% → 78% (sync trigger, status, JWT verify) - curator/syncer.py: 17.83% → 91% (collection/wantlist sync, full sync) - common/config.py: 32.14% → 97% (ApiConfig, CuratorConfig) - explore/explore.py: 49.39% → 85% (user endpoints, JWT helpers) - explore/user_queries.py: 45.94% → 100% (collection, wantlist, recommendations) New test files: - tests/api/conftest.py — fixtures (JWT, mock pool, mock Redis, TestClient) - tests/api/test_api.py — 44 tests for API service endpoints - tests/api/test_discogs_service.py — 30 tests for Discogs OAuth service - tests/curator/conftest.py — fixtures (JWT, mock pool, mock Neo4j, TestClient) - tests/curator/test_curator.py — 48 tests for curator service endpoints Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * test: increase coverage to 94% by adding JWT, user endpoint, and syncer tests Add tests covering previously uncovered branches in explore and curator services: - explore/explore.py: JWT helpers (_b64url_decode padding, _verify_jwt all branches), _require_user dependency (config None→503, no auth→401, bad token→401), all five user endpoints (collection, wantlist, recommendations, stats, status) with service-not-ready and success paths - curator/syncer.py: 429 rate-limit retry with sleep for both collection and wantlist, skip items missing release_id, multi-page pagination for both Total coverage: 84% → 94% Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * security: replace admin HTTP endpoint with CLI tool for Discogs credentials Remove PUT /api/admin/config/{key}, which was protected only by a valid JWT and thus accessible to any registered user. Replace it with a standalone discogs-setup CLI script (api/setup.py) that writes directly to the app_config table and must be run via docker exec on the API container — an already admin-privileged operation. - api/setup.py: new CLI entry point with --consumer-key/--consumer-secret and --show (masked) modes; reads POSTGRES_* env vars, uses psycopg sync - api/api.py: remove set_app_config endpoint; update OAuth error message - api/pyproject.toml: add discogs-setup script entry point - justfile: add configure-discogs task in [group('setup')] - tests/api/test_api.py: remove TestAdminConfigEndpoint (4 tests) - tests/api/test_setup.py: add 18 tests covering the new CLI tool Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com> Co-authored-by: Robert Wlodarczyk <SimplicityGuy@users.noreply.github.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
1 parent 3598fff commit 8218b80

57 files changed

Lines changed: 7588 additions & 93 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.github/workflows/code-quality.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -105,7 +105,9 @@ jobs:
105105
run: |
106106
just install
107107
# Install workspace packages
108+
uv pip install -e api
108109
uv pip install -e common
110+
uv pip install -e curator
109111
uv pip install -e dashboard
110112
uv pip install -e explore
111113

.github/workflows/docker-validate.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -90,7 +90,7 @@ jobs:
9090
- name: 🔍 Check docker-compose services
9191
run: |
9292
services=$(docker-compose config --services | sort | tr "\n" " " | sed "s/ $//")
93-
expected="dashboard explore extractor graphinator neo4j postgres rabbitmq redis schema-init tableinator"
93+
expected="api curator dashboard explore extractor graphinator neo4j postgres rabbitmq redis schema-init tableinator"
9494
9595
if [ "$services" != "$expected" ]; then
9696
echo "❌ Service mismatch!"

.github/workflows/list-sub-projects.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,8 @@ jobs:
3030
# Define sub-projects with their properties
3131
cat << "EOF" > projects.json
3232
[
33+
{"name": "api", "use_cache": true},
34+
{"name": "curator", "use_cache": true},
3335
{"name": "dashboard", "use_cache": true},
3436
{"name": "explore", "use_cache": true},
3537
{"name": "extractor", "use_cache": true},

.github/workflows/test.yml

Lines changed: 83 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,82 @@ permissions:
4040
pull-requests: write # Required for coverage report comments
4141

4242
jobs:
43+
# ============================================================================
44+
# API SERVICE TESTS - Runs in parallel with all other test jobs
45+
# ============================================================================
46+
test-api:
47+
runs-on: ubuntu-latest
48+
timeout-minutes: 10
49+
50+
steps:
51+
- name: 🔀 Checkout repository
52+
uses: actions/checkout@v6
53+
54+
- name: 🔧 Setup Python and UV
55+
uses: ./.github/actions/setup-python-uv
56+
with:
57+
python-version: ${{ env.PYTHON_VERSION }}
58+
59+
- name: 🔧 Setup Just
60+
uses: ./.github/actions/setup-just
61+
62+
- name: 📦 Install dependencies
63+
run: just install
64+
65+
- name: 🧪 Run api tests
66+
run: just test-api
67+
68+
- name: 📊 Upload coverage (api)
69+
if: always()
70+
uses: codecov/codecov-action@v5
71+
with:
72+
token: ${{ secrets.CODECOV_TOKEN }}
73+
slug: SimplicityGuy/discogsography
74+
files: ./coverage.xml
75+
flags: api
76+
name: api-tests
77+
fail_ci_if_error: false
78+
disable_search: true
79+
verbose: false
80+
81+
# ============================================================================
82+
# CURATOR SERVICE TESTS - Runs in parallel with all other test jobs
83+
# ============================================================================
84+
test-curator:
85+
runs-on: ubuntu-latest
86+
timeout-minutes: 10
87+
88+
steps:
89+
- name: 🔀 Checkout repository
90+
uses: actions/checkout@v6
91+
92+
- name: 🔧 Setup Python and UV
93+
uses: ./.github/actions/setup-python-uv
94+
with:
95+
python-version: ${{ env.PYTHON_VERSION }}
96+
97+
- name: 🔧 Setup Just
98+
uses: ./.github/actions/setup-just
99+
100+
- name: 📦 Install dependencies
101+
run: just install
102+
103+
- name: 🧪 Run curator tests
104+
run: just test-curator
105+
106+
- name: 📊 Upload coverage (curator)
107+
if: always()
108+
uses: codecov/codecov-action@v5
109+
with:
110+
token: ${{ secrets.CODECOV_TOKEN }}
111+
slug: SimplicityGuy/discogsography
112+
files: ./coverage.xml
113+
flags: curator
114+
name: curator-tests
115+
fail_ci_if_error: false
116+
disable_search: true
117+
verbose: false
118+
43119
# ============================================================================
44120
# COMMON/SHARED LIBRARY TESTS - Runs in parallel with all other test jobs
45121
# ============================================================================
@@ -357,6 +433,8 @@ jobs:
357433
aggregate-results:
358434
runs-on: ubuntu-latest
359435
needs:
436+
- test-api
437+
- test-curator
360438
- test-common
361439
- test-dashboard
362440
- test-explore
@@ -370,6 +448,8 @@ jobs:
370448
- name: 📊 Check test results
371449
run: |
372450
echo "Test Results Summary:"
451+
echo " API: ${{ needs.test-api.result }}"
452+
echo " Curator: ${{ needs.test-curator.result }}"
373453
echo " Common: ${{ needs.test-common.result }}"
374454
echo " Dashboard: ${{ needs.test-dashboard.result }}"
375455
echo " Explore: ${{ needs.test-explore.result }}"
@@ -379,7 +459,9 @@ jobs:
379459
echo " Extractor: ${{ needs.test-extractor.result }}"
380460
381461
# Check if any test job failed
382-
if [[ "${{ needs.test-common.result }}" == "failure" ]] || \
462+
if [[ "${{ needs.test-api.result }}" == "failure" ]] || \
463+
[[ "${{ needs.test-curator.result }}" == "failure" ]] || \
464+
[[ "${{ needs.test-common.result }}" == "failure" ]] || \
383465
[[ "${{ needs.test-dashboard.result }}" == "failure" ]] || \
384466
[[ "${{ needs.test-explore.result }}" == "failure" ]] || \
385467
[[ "${{ needs.test-graphinator.result }}" == "failure" ]] || \

CLAUDE.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -344,6 +344,8 @@ http://localhost:8000/api/health
344344

345345
### Service Ports
346346

347+
- API: 8004 (service), 8005 (health)
348+
- Curator: 8010 (service), 8011 (health)
347349
- Dashboard: 8003
348350
- Explore: 8006 (service), 8007 (health)
349351
- Neo4j: 7474 (browser), 7687 (bolt)

README.md

Lines changed: 22 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -41,14 +41,16 @@ Perfect for music researchers, data scientists, developers, and music enthusiast
4141

4242
### ⚙️ Core Services
4343

44-
| Service | Purpose | Key Technologies |
45-
| ------------------------------------------------------------- | -------------------------------------- | -------------------------------------- |
46-
| **[📊](docs/emoji-guide.md#service-identifiers) Dashboard** | Real-time system monitoring | `FastAPI`, WebSocket, reactive UI |
47-
| **[🔍](docs/emoji-guide.md#service-identifiers) Explore** | Interactive graph exploration & trends | `FastAPI`, `D3.js`, `Plotly.js`, Neo4j |
48-
| **[](docs/emoji-guide.md#service-identifiers) Extractor** | High-performance Rust-based extractor | `tokio`, `quick-xml`, `lapin` |
49-
| **[🔗](docs/emoji-guide.md#service-identifiers) Graphinator** | Builds Neo4j knowledge graphs | `neo4j-driver`, graph algorithms |
50-
| **[🔧](docs/emoji-guide.md#service-identifiers) Schema-Init** | One-shot database schema initialiser | `neo4j-driver`, `psycopg3` |
51-
| **[🐘](docs/emoji-guide.md#service-identifiers) Tableinator** | Creates PostgreSQL analytics tables | `psycopg3`, JSONB, full-text search |
44+
| Service | Purpose | Key Technologies |
45+
| ------------------------------------------------------------- | -------------------------------------- | ------------------------------------------------- |
46+
| **[🔐](docs/emoji-guide.md#service-identifiers) API** | User accounts and JWT authentication | `FastAPI`, `psycopg3`, `redis`, Discogs OAuth 1.0 |
47+
| **[🗂️](docs/emoji-guide.md#service-identifiers) Curator** | Discogs collection & wantlist sync | `FastAPI`, `psycopg3`, `neo4j-driver` |
48+
| **[📊](docs/emoji-guide.md#service-identifiers) Dashboard** | Real-time system monitoring | `FastAPI`, WebSocket, reactive UI |
49+
| **[🔍](docs/emoji-guide.md#service-identifiers) Explore** | Interactive graph exploration & trends | `FastAPI`, `D3.js`, `Plotly.js`, Neo4j |
50+
| **[](docs/emoji-guide.md#service-identifiers) Extractor** | High-performance Rust-based extractor | `tokio`, `quick-xml`, `lapin` |
51+
| **[🔗](docs/emoji-guide.md#service-identifiers) Graphinator** | Builds Neo4j knowledge graphs | `neo4j-driver`, graph algorithms |
52+
| **[🔧](docs/emoji-guide.md#service-identifiers) Schema-Init** | One-shot database schema initializer | `neo4j-driver`, `psycopg3` |
53+
| **[🐘](docs/emoji-guide.md#service-identifiers) Tableinator** | Creates PostgreSQL analytics tables | `psycopg3`, JSONB, full-text search |
5254

5355
### 📐 System Architecture
5456

@@ -65,6 +67,8 @@ graph TD
6567
TABLE[["🐘 Tableinator<br/>Table Builder"]]
6668
DASH[["📊 Dashboard<br/>Real-time Monitor<br/>WebSocket"]]
6769
EXPLORE[["🔍 Explore<br/>Graph Explorer<br/>Trends & Paths"]]
70+
API[["🔐 API<br/>User Auth<br/>JWT & OAuth"]]
71+
CURATOR[["🗂️ Curator<br/>Collection<br/>Sync"]]
6872
6973
SCHEMA -->|0. Create Indexes & Constraints| NEO4J
7074
SCHEMA -->|0. Create Tables & Indexes| PG
@@ -78,6 +82,12 @@ graph TD
7882
EXPLORE -.->|Query Graph| NEO4J
7983
EXPLORE -.->|Explore Paths| NEO4J
8084
85+
API -.->|User Accounts| PG
86+
API -.->|OAuth State| REDIS
87+
88+
CURATOR -.->|Sync Collections| NEO4J
89+
CURATOR -.->|Sync History| PG
90+
8191
DASH -.->|Monitor| EXT
8292
DASH -.->|Monitor| GRAPH
8393
DASH -.->|Monitor| TABLE
@@ -98,6 +108,8 @@ graph TD
98108
style TABLE fill:#fce4ec,stroke:#880e4f,stroke-width:2px
99109
style DASH fill:#fce4ec,stroke:#880e4f,stroke-width:2px
100110
style EXPLORE fill:#e8eaf6,stroke:#283593,stroke-width:2px
111+
style API fill:#e3f2fd,stroke:#0d47a1,stroke-width:2px
112+
style CURATOR fill:#fff8e1,stroke:#f57f17,stroke-width:2px
101113
```
102114

103115
## 🌟 Key Features
@@ -124,6 +136,8 @@ open http://localhost:8003
124136

125137
| Service | URL | Default Credentials |
126138
| ----------------- | ---------------------- | ----------------------------------- |
139+
| 🔐 **API** | http://localhost:8004 | Register via `/api/auth/register` |
140+
| 🗂️ **Curator** | http://localhost:8010 | JWT required (via API) |
127141
| 📊 **Dashboard** | http://localhost:8003 | None |
128142
| 🔍 **Explore** | http://localhost:8006 | None |
129143
| 🔗 **Neo4j** | http://localhost:7474 | `neo4j` / `discogsography` |

api/Dockerfile

Lines changed: 132 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,132 @@
1+
# syntax=docker/dockerfile:1
2+
3+
# Build arguments
4+
ARG PYTHON_VERSION=3.13
5+
ARG UID=1000
6+
ARG GID=1000
7+
8+
FROM python:${PYTHON_VERSION}-slim AS builder
9+
10+
# Install uv
11+
COPY --from=ghcr.io/astral-sh/uv:0.10.4 /uv /bin/uv
12+
13+
# Set environment for build
14+
ENV UV_SYSTEM_PYTHON=1 \
15+
UV_CACHE_DIR=/tmp/.cache/uv \
16+
UV_LINK_MODE=copy \
17+
PYTHONUNBUFFERED=1 \
18+
PYTHONDONTWRITEBYTECODE=1
19+
20+
WORKDIR /app
21+
22+
# Copy dependency files first for better caching
23+
COPY pyproject.toml uv.lock ./
24+
COPY common/pyproject.toml ./common/
25+
COPY api/pyproject.toml ./api/
26+
27+
# Install dependencies and clean up
28+
# hadolint ignore=SC2015
29+
RUN --mount=type=cache,target=/tmp/.cache/uv \
30+
uv sync --frozen --no-dev --extra api && \
31+
# Clean up cache and test files
32+
find /app/.venv -type f -name "*.pyc" -delete && \
33+
find /app/.venv -type f -name "*.pyo" -delete && \
34+
find /app/.venv -type d -name "__pycache__" -exec rm -rf {} + 2>/dev/null || true && \
35+
find /app/.venv -type d -name ".pytest_cache" -exec rm -rf {} + 2>/dev/null || true && \
36+
find /app/.venv -type d -name "tests" -exec rm -rf {} + 2>/dev/null || true && \
37+
# Remove type stubs and docs
38+
find /app/.venv -type d -name "*.dist-info" -exec rm -rf {}/tests {} + 2>/dev/null || true && \
39+
find /app/.venv -name "py.typed" -delete 2>/dev/null || true && \
40+
# Strip compiled extensions
41+
find /app/.venv -name "*.so" -exec strip --strip-unneeded {} \; 2>/dev/null || true
42+
43+
# Copy source files
44+
COPY common/ ./common/
45+
COPY api/ ./api/
46+
47+
# Final stage
48+
FROM python:${PYTHON_VERSION}-slim
49+
50+
# Build arguments for labels
51+
ARG BUILD_DATE
52+
ARG BUILD_VERSION
53+
ARG VCS_REF
54+
ARG UID=1000
55+
ARG GID=1000
56+
57+
# OCI Image Spec Annotations
58+
# https://github.com/opencontainers/image-spec/blob/main/annotations.md
59+
LABEL org.opencontainers.image.title="Discogsography API" \
60+
org.opencontainers.image.description="API service for user authentication and Discogs OAuth integration for Discogsography." \
61+
org.opencontainers.image.authors="Robert Wlodarczyk <robert@simplicityguy.com>" \
62+
org.opencontainers.image.url="https://github.com/SimplicityGuy/discogsography" \
63+
org.opencontainers.image.documentation="https://github.com/SimplicityGuy/discogsography/blob/main/README.md" \
64+
org.opencontainers.image.source="https://github.com/SimplicityGuy/discogsography" \
65+
org.opencontainers.image.vendor="SimplicityGuy" \
66+
org.opencontainers.image.licenses="MIT" \
67+
org.opencontainers.image.version="${BUILD_VERSION:-0.1.0}" \
68+
org.opencontainers.image.revision="${VCS_REF}" \
69+
org.opencontainers.image.created="${BUILD_DATE}" \
70+
org.opencontainers.image.base.name="docker.io/library/python:${PYTHON_VERSION}-slim" \
71+
com.discogsography.service="api" \
72+
com.discogsography.dependencies="fastapi,uvicorn,psycopg,bcrypt,pyjwt" \
73+
com.discogsography.python.version="${PYTHON_VERSION}"
74+
75+
# Install minimal runtime dependencies
76+
# hadolint ignore=DL3008
77+
RUN apt-get update && \
78+
apt-get upgrade -y && \
79+
apt-get install -y --no-install-recommends \
80+
curl \
81+
&& \
82+
apt-get clean && \
83+
rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
84+
85+
# Create user and directories with configurable UID/GID
86+
RUN groupadd -r -g ${GID} discogsography && \
87+
useradd -r -l -u ${UID} -g discogsography -m -s /bin/bash discogsography && \
88+
mkdir -p /tmp /app /logs && \
89+
chown -R discogsography:discogsography /tmp /app /logs
90+
91+
WORKDIR /app
92+
93+
# Copy only necessary files from builder
94+
COPY --from=builder --chown=discogsography:discogsography /app/.venv /app/.venv
95+
COPY --from=builder --chown=discogsography:discogsography /app/common /app/common
96+
COPY --from=builder --chown=discogsography:discogsography /app/api /app/api
97+
98+
# Create startup script
99+
# hadolint ignore=SC2016
100+
RUN printf '#!/bin/sh\nset -e\nsleep "${STARTUP_DELAY:-0}"\nexec /app/.venv/bin/python -m api.api "$@"\n' > /app/start.sh && \
101+
chmod +x /app/start.sh
102+
103+
# Health check on health port 8005
104+
HEALTHCHECK --interval=30s --timeout=10s --start-period=60s --retries=3 \
105+
CMD curl -f http://localhost:8005/health || exit 1
106+
107+
USER discogsography:discogsography
108+
109+
# Environment variables
110+
ENV HOME=/home/discogsography \
111+
PYTHONUNBUFFERED=1 \
112+
PYTHONDONTWRITEBYTECODE=1 \
113+
UV_SYSTEM_PYTHON=1 \
114+
UV_NO_CACHE=1 \
115+
PATH="/app/.venv/bin:$PATH" \
116+
POSTGRES_ADDRESS="" \
117+
POSTGRES_USERNAME="" \
118+
POSTGRES_PASSWORD="" \
119+
POSTGRES_DATABASE="" \
120+
JWT_SECRET_KEY="" \
121+
JWT_ALGORITHM="HS256" \
122+
JWT_EXPIRE_MINUTES="30"
123+
124+
EXPOSE 8004 8005
125+
126+
# Declare volume for logs
127+
VOLUME ["/logs"]
128+
129+
# Security: This container should be run with:
130+
# docker run --cap-drop=ALL --security-opt=no-new-privileges:true ...
131+
132+
CMD ["/app/start.sh"]

0 commit comments

Comments
 (0)