Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
46 commits
Select commit Hold shift + click to select a range
9299303
docs: spec and implementation plan for v1 API overhaul
marevol May 21, 2026
c212eaf
test: POC for AIP-136 colon-verb paths in FastAPI
marevol May 21, 2026
dc31752
docs: fix POC deletion task number, spec external refs, OpenAPI path
marevol May 21, 2026
a377e56
feat(serving): add v1 schema module (Pydantic v2)
marevol May 21, 2026
d48ccbd
feat(serving): expose model_version, loaded_at, kind, supported_verbs
marevol May 21, 2026
5f2901a
feat(serving): add v1 request counter + latency + batch_size metrics
marevol May 21, 2026
a7ca066
feat(serving): skeleton v1_router factory
marevol May 21, 2026
11fa153
feat(serving): port health + metrics into v1_router
marevol May 21, 2026
0afe00a
feat(serving): POST /v1/recipes/{name}:recommend
marevol May 21, 2026
e55a7ce
feat(serving): POST /v1/recipes/{name}:recommend-related
marevol May 21, 2026
e1de8c9
feat(serving): POST /v1/recipes/{name}:batch-recommend
marevol May 21, 2026
20ec9e3
feat(serving): POST /v1/recipes/{name}:batch-recommend-related
marevol May 21, 2026
9f65e78
feat(serving): GET /v1/recipes and /v1/recipes/{name}
marevol May 21, 2026
7c54512
feat(serving): mount v1 router under /v1, retire legacy make_router
marevol May 21, 2026
99525bd
test(integration): retarget e2e suite at v1 endpoints
marevol May 21, 2026
813a9f3
docs: refresh README + getting-started + add api-reference and migrat…
marevol May 21, 2026
8f4e595
style: apply ruff lint + format fixes across v1 modules
marevol May 21, 2026
2a63d3e
docs: record v1 API overhaul in CHANGELOG
marevol May 21, 2026
d29b1aa
fix(serving): populate v1 fields in startup-scan load path
marevol May 21, 2026
ea78f7f
test(e2e): retarget run.sh at v1 endpoints
marevol May 21, 2026
585ff8a
fix(serving): address v1 API review feedback
marevol May 21, 2026
0a0edf6
chore: stop tracking local working docs under docs/plans and docs/specs
marevol May 21, 2026
6960aa3
refactor(serving): consolidate v1 router into routes.py
marevol May 21, 2026
dc9cb9a
ci: drop flaky MovieLens100K nightly workflow
marevol May 21, 2026
eb1d84b
docs: drop migration-v1 guide
marevol May 21, 2026
e0c9dab
fix(serving): v1 API parity gaps + flat error body envelope
marevol May 21, 2026
70adaeb
fix(serving): address v1 API review findings (auth, schemas, batch pa…
marevol May 21, 2026
b948f07
test(serving): expand v1 API coverage and align test fixture with pro…
marevol May 21, 2026
b7143ae
fix(serving): tighten v1 API error codes, metrics labels, and batch v…
marevol May 21, 2026
ba4d60f
fix(serving): address multi-agent v1 API review findings
marevol May 22, 2026
26fa91b
docs(serving): add alpha → v1 migration guide
marevol May 22, 2026
f6d0a46
docs(serving): note /v1/metrics auth requirement
marevol May 22, 2026
265a898
fix(serving): preserve pydantic validation details in batch elements
marevol May 22, 2026
81c2632
docs(serving): note X-Recotem-Metadata-Degraded removal
marevol May 22, 2026
522ed53
fix(serving): correct artifact-load failure reason taxonomy
marevol May 22, 2026
261b9b2
fix(metadata): preserve row-flatten exception type/message
marevol May 22, 2026
eb63924
fix(serving): isolate per-item metadata serialization failures
marevol May 22, 2026
ae37a8a
test(serving): assert model_version rotates after hot-swap
marevol May 22, 2026
3eb630c
test(serving): cover wrong-API-key + key rotation across v1 verbs
marevol May 22, 2026
2e20ad9
test(serving): cover dev-bypass on v1 recommend verbs
marevol May 22, 2026
bbdd177
test(serving): YAML deleted at runtime returns 404/503 via v1
marevol May 22, 2026
e46297c
test(serving): cover :recommend-related metadata enrichment + denylist
marevol May 22, 2026
e49f6c0
test(serving): assert /v1/metrics label cardinality bounds
marevol May 22, 2026
15c88d6
test(serving): assert /v1/health/details per-recipe shape
marevol May 22, 2026
5a092ae
refactor(serving): minor cleanups from v1 API review
marevol May 22, 2026
2e0f52d
refactor(serving): address v1 API review findings
marevol May 22, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
94 changes: 0 additions & 94 deletions .github/workflows/nightly.yml

This file was deleted.

13 changes: 10 additions & 3 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -145,10 +145,17 @@ jobs:

echo "Scanning log files in $LOG_DIR ..."

# Pattern 1: raw sha256 hex digest (signing key or hash leaked)
if grep -rEq 'sha256:[0-9a-f]{64}' "$LOG_DIR"; then
# Pattern 1: raw sha256 hex digest (signing key or hash leaked).
# Excludes the public X-Recotem-Model-Version header and the
# corresponding JSON "model_version" response field, which carry
# the artifact content hash by design and are not secrets.
pat1_hits=$(grep -rEn 'sha256:[0-9a-f]{64}' "$LOG_DIR" \
| grep -v '"model_version"[[:space:]]*:[[:space:]]*"sha256:' \
| grep -vi 'x-recotem-model-version:[[:space:]]*sha256:' \
|| true)
if [ -n "$pat1_hits" ]; then
echo "FAIL: Found sha256:<hex64> in log output."
grep -rEn 'sha256:[0-9a-f]{64}' "$LOG_DIR" | head -5
echo "$pat1_hits" | head -5
FAIL=1
fi

Expand Down
5 changes: 5 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -51,3 +51,8 @@ az:/

# Tools
.serena/

# Local working docs (plans, specs) — not for the public repo
docs/plans/
docs/specs/

26 changes: 14 additions & 12 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ single Python package (`pip install recotem`) plus a single Docker image.
├──────────────────────────────────────────────────────────────────┤
│ CLI (Typer) │
│ ├─ recotem train <recipe.yaml> batch: fetch→train→sign │
│ ├─ recotem serve --recipes <dir> FastAPI /predict
│ ├─ recotem serve --recipes <dir> FastAPI /v1/recipes/:*
│ ├─ recotem inspect <artifact> read header (no payload) │
│ ├─ recotem validate <recipe.yaml> schema + connectivity │
│ ├─ recotem schema emit JSON Schema for IDEs │
Expand Down Expand Up @@ -52,12 +52,12 @@ src/recotem/

tests/
├── unit/ per-module tests (recipe, artifact, training, ...)
├── integration/ in-process train + serve + predict
├── integration/ in-process train + serve + recommend
├── fuzz/ hypothesis byte mutations on artifact / recipe loaders
└── e2e/ bash script: train → serve → curl /predict
└── e2e/ bash script: train → serve → curl /v1/recipes/{name}:recommend

docs/
├── getting-started.md Docker / pip walkthrough → train → /predict
├── getting-started.md Docker / pip walkthrough → train → /v1/recipes/{name}:recommend
├── recipe-reference.md every recipe field, type, default, validation
├── data-sources/ bigquery.md, csv.md, ga4.md, sql.md
├── deployment/ docker.md, k8s.md, cron.md
Expand Down Expand Up @@ -90,16 +90,16 @@ uv run recotem train examples/tutorial-purchase-log/recipe.yaml
# Serve from a directory of recipes
uv run recotem serve --recipes ./recipes/ --port 8080

# Predict
curl -X POST http://localhost:8080/predict/news_articles \
# Recommend
curl -X POST http://localhost:8080/v1/recipes/news_articles:recommend \
-H "X-API-Key: <plaintext>" \
-H "Content-Type: application/json" \
-d '{"user_id":"u1","cutoff":10}'
-d '{"user_id":"u1","limit":10}'
```

## Recipe model

A recipe is the single source of truth: 1 YAML = 1 model = 1 `/predict/{name}`.
A recipe is the single source of truth: 1 YAML = 1 model = 1 `/v1/recipes/{name}:recommend` (plus the related/batch verbs).
See `docs/recipe-reference.md` for the full schema. Highlights:

- `source.type` is a discriminator (`csv` | `parquet` | `bigquery` | `sql` | `ga4` | plugins).
Expand Down Expand Up @@ -146,9 +146,11 @@ Binary container `magic | version | reserved | kid | hmac | header_json | payloa
`uv run ruff format src tests`). Line-length 88. Selected rules in
`pyproject.toml`.
- pytest 8 + hypothesis 6. `@pytest.mark.slow` deselected by default.
- `from __future__ import annotations` is used everywhere except where it
breaks FastAPI dependency introspection (e.g. `routes.py` uses
`kid: str = Depends(_require_auth)` instead of `Annotated[...]`).
- `from __future__ import annotations` is used everywhere, including the
serving router. FastAPI dependency arguments are written as
`kid: str = Depends(_require_auth)` (not `Annotated[...]`) in
`serving/routes.py` so that `Depends` is resolved as a runtime
default rather than a stringified annotation.
- structlog logger per module; the redaction processor in
`recotem.log_redaction` is first in the chain and strips API keys, signing
keys, and cloud creds. Lives at the top level so `train`-only invocations do
Expand Down Expand Up @@ -208,7 +210,7 @@ uv run ruff format --check src tests
| `RECOTEM_MAX_PAYLOAD_BYTES` | 512 MiB | Per-payload cap (post-HMAC-verify) for serve-side deserialization. Clamped [1 MiB, 16 GiB]. Smaller than `RECOTEM_MAX_ARTIFACT_BYTES` to bound deserialization memory expansion. |
| `RECOTEM_ARTIFACT_ROOT` | (empty) | If set, local `output.path` must lie under it. |
| `RECOTEM_RECIPE_*` | — | Allow-listed for `${...}` recipe expansion. |
| `RECOTEM_METADATA_FIELD_DENY` | (empty) | Comma-separated columns stripped from `/predict` responses. |
| `RECOTEM_METADATA_FIELD_DENY` | (empty) | Comma-separated columns stripped from `/v1/recipes/{name}:recommend` and `:recommend-related` responses. |
| `RECOTEM_METRICS_ENABLED` | (empty) | Opt-in Prometheus `/metrics` endpoint. Truthy values: `1`, `true`, `yes`, `on`. Requires `recotem[metrics]` extra. |
| `RECOTEM_LOCK_DIR` | (empty) | Override directory for per-recipe training lock files. Local outputs always lock at `<output_path>.lock`; remote outputs (`s3://`, `gs://`, ...) need a host-local path and fall back to `<tempdir>/recotem-locks/`. `flock` is host-local — across hosts use scheduler-level mutex (`concurrencyPolicy: Forbid`). |
| `RECOTEM_BQ_REQUIRE_STORAGE_API` | (empty) | When truthy (`1`/`true`/`yes`/`on`), the BigQuery source raises `DataSourceError` instead of falling back to the REST path when the Storage Read API fails. Requires the service account to hold `bigquery.readSessions.create`. |
Expand Down
2 changes: 1 addition & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ Recotem is a single Python package with two execution modes:

- `recotem train recipe.yaml` — fetch → train → write a signed artifact.
- `recotem serve --recipes <dir>` — FastAPI server that watches the dir and
serves `/predict/{name}` for every loaded recipe.
serves `/v1/recipes/{name}:*` for every loaded recipe.

The two modes communicate only via the signed artifact file format, so the
trainer and the server can run on completely separate hosts.
Expand Down
36 changes: 22 additions & 14 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,9 @@ Recipe-driven recommender training and serving, built on
[irspack](https://github.com/tohtsky/irspack). One YAML recipe describes
where the data lives, how to train, and where to write the result —
`recotem train` produces a signed binary artifact, `recotem serve`
mounts it as a `/predict/{name}` HTTP endpoint and hot-swaps when a new
artifact appears. No database, no message broker, no admin UI.
mounts it under `/v1/recipes/{name}:recommend` (plus `:recommend-related`
and batch verbs) and hot-swaps when a new artifact appears. No database,
no message broker, no admin UI.

## Why Recotem

Expand All @@ -32,7 +33,7 @@ moving parts to a recipe file and a binary artifact:

## Features

- Recipe-driven: 1 YAML = 1 model = 1 `/predict/{name}` endpoint
- Recipe-driven: 1 YAML = 1 model = 1 `/v1/recipes/{name}:recommend` endpoint (with related/batch verbs)
- Hyperparameter search across irspack algorithms via Optuna
- Pluggable data sources (built-in: CSV / Parquet / BigQuery / SQL / GA4; extend via Python entry points)
- HMAC-signed artifacts with multi-key rotation and a deterministic
Expand Down Expand Up @@ -84,21 +85,28 @@ recotem train examples/quickstart/recipe.yaml
recotem serve --recipes examples/quickstart/ &

# Wait for the server to become ready before sending traffic.
until curl -s -o /dev/null -w "%{http_code}" http://localhost:8080/health | grep -q "200"; do sleep 1; done
until curl -s -o /dev/null -w "%{http_code}" http://localhost:8080/v1/health | grep -q "200"; do sleep 1; done

# 3. Predict
curl -X POST http://localhost:8080/predict/top_picks \
# 3. Recommend
# 3a. Recommend for a known user
curl -X POST http://localhost:8080/v1/recipes/top_picks:recommend \
-H "X-API-Key: $RECOTEM_API_PLAINTEXT" \
-H "Content-Type: application/json" \
-d '{"user_id": "u01", "cutoff": 5}'
-d '{"user_id": "u01", "limit": 5}'

# 3b. Recommend items related to a seed item
curl -X POST http://localhost:8080/v1/recipes/top_picks:recommend-related \
-H "X-API-Key: $RECOTEM_API_PLAINTEXT" \
-H "Content-Type: application/json" \
-d '{"seed_items": ["i00"], "limit": 5}'
```

```json
{
"items": [{"item_id": "i00", "score": 0.91}],
"model": {"recipe": "top_picks", "trained_at": "...",
"best_class": "TopPopRecommender", "kid": "dev"},
"request_id": "..."
"request_id": "req_01HZX...",
"recipe": "top_picks",
"model_version": "sha256:abc...",
"items": [{"item_id": "i00", "score": 0.91}]
}
```

Expand All @@ -112,8 +120,8 @@ for the source of truth and
| Variable | Required by | Purpose |
|---|---|---|
| `RECOTEM_SIGNING_KEYS` | `train` and `serve` | HMAC sign / verify artifact files (server keeps plaintext; needed for both sides) |
| `RECOTEM_API_KEYS` | `serve` | Authenticate `/predict` callers (server keeps **hash** only) |
| `X-API-Key: <plaintext>` | HTTP clients | Sent by clients on every `/predict` call; server re-hashes and compares |
| `RECOTEM_API_KEYS` | `serve` | Authenticate `/v1/recipes/*` callers (server keeps **hash** only) |
| `X-API-Key: <plaintext>` | HTTP clients | Sent by clients on every `/v1/recipes/*` call; server re-hashes and compares |

Both variables accept multiple comma-separated entries (`kid:value,kid2:value,…`)
to enable zero-downtime key rotation — that is why they are pluralised.
Expand All @@ -129,7 +137,7 @@ to enable zero-downtime key rotation — that is why they are pluralised.
│ (batch job) (HMAC-signed) (FastAPI, │
│ hot-swap) │
│ │
│ any scheduler local FS, S3, POST /predict/{name}│
│ any scheduler local FS, S3, POST /v1/recipes/{name}
│ (cron / k8s / …) GCS, fsspec X-API-Key auth │
│ │
└────────────────────────────────────────────────────────────────────────┘
Expand Down
4 changes: 2 additions & 2 deletions compose.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -20,10 +20,10 @@
# 3. Serve + curl
# $ RECOTEM_SIGNING_KEYS="..." RECOTEM_API_KEYS="..." \
# docker compose up -d serve
# $ curl -sX POST http://localhost:8080/predict/purchase_log \
# $ curl -sX POST http://localhost:8080/v1/recipes/purchase_log:recommend \
# -H "X-API-Key: <api plaintext>" \
# -H "Content-Type: application/json" \
# -d '{"user_id": "1", "cutoff": 5}'
# -d '{"user_id": "1", "limit": 5}'

x-recotem-image: &recotem-image
image: ghcr.io/codelibs/recotem:latest
Expand Down
2 changes: 1 addition & 1 deletion docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

## Getting started

- [Getting started](getting-started.md) — install (Docker or pip), train from a public CSV, curl `/predict`
- [Getting started](getting-started.md) — install (Docker or pip), train from a public CSV, curl `/v1/recipes/{name}:recommend`

## Reference

Expand Down
Loading