@@ -4,8 +4,8 @@ Ingests public mailing lists (e.g. lore.kernel.org / LKML-style) into CDP,
44following the same worker pattern as ` services/apps/git_integration ` :
55FastAPI lifespan + async poll worker + asyncpg + aiokafka.
66
7- Status: under construction (CM-1318). See the plan/step tracker referenced
8- in the PR for build progress.
7+ Status: hardcoded lists only, no onboarding UI (CM-1318). See the plan/step
8+ tracker referenced in the PR for build progress.
99
1010## Architecture (planned)
1111
@@ -33,3 +33,43 @@ endpoint on host port `8086`.
3333
3434Other targets: ` make lint ` , ` make format ` , ` make test ` , ` make rebuild ` . See
3535` make help ` for the full list.
36+
37+ ## Onboarding a list (dev, hardcoded)
38+
39+ There is no UI yet — a list is onboarded by inserting its rows directly.
40+ ` dev/seed.sql ` does this for one example list (` lore.kernel.org/git ` ):
41+
42+ ``` bash
43+ psql " $CROWD_DB_WRITE_URI " -f dev/seed.sql
44+ ```
45+
46+ It creates a ` public.integrations ` row (` platform='groupsio' ` ), a
47+ ` mailinglist.lists ` row pointing at an existing segment, and a
48+ ` mailinglist."listProcessing" ` row in state ` pending ` — which the worker's
49+ ` acquire_onboarding_list() ` will pick up on its next poll. Edit the ` name `
50+ and ` "sourceUrl" ` values in the file to onboard a different lore list.
51+
52+ ## End-to-end verification
53+
54+ 1 . Apply ` backend/src/database/migrations/V1784048135__mailinglist-schema.sql `
55+ and run ` dev/seed.sql ` against the target Postgres.
56+ 2 . Start Kafka and a ` data_sink_worker ` consumer on the ` data-sink-worker ` topic.
57+ 3 . ` make run ` (or ` uv run uvicorn crowdmail.server:app ` ) — worker polls, mirrors
58+ the seeded list via ` public-inbox-clone ` , parses new commits, and:
59+ - inserts rows into ` integration.results ` with ` state='pending' ` and
60+ ` data.type='activity' ` ,
61+ - emits one Kafka message per result (` process_integration_result ` ,
62+ ` platform=groupsio ` ),
63+ - advances ` mailinglist."listProcessing"."lastProcessedHeads" ` and sets
64+ ` state='completed' ` .
65+ 4 . Confirm ` data_sink_worker ` consumes the message, resolves the integration's
66+ ` platform='groupsio' ` , and creates the member (email-based verified
67+ ` username ` +` email ` identities) and activity.
68+ 5 . ` GET http://localhost:8086/ ` returns the FastAPI health check.
69+
70+ ## Out of scope (CM-1318)
71+
72+ - Onboarding UI/backend to create ` integrations ` + ` mailinglist.lists ` rows
73+ (this is what ` dev/seed.sql ` stands in for).
74+ - Whether ` member_join ` /` member_leave ` activities are derivable from lore, or
75+ only ` message ` initially.
0 commit comments