Skip to content

Commit 7e086b0

Browse files
committed
chore: add mailing list dev seed and docs (CM-1318)
Signed-off-by: Uroš Marolt <uros@marolt.me>
1 parent 9fd947f commit 7e086b0

2 files changed

Lines changed: 87 additions & 2 deletions

File tree

services/apps/mailing_list_integration/README.md

Lines changed: 42 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,8 +4,8 @@ Ingests public mailing lists (e.g. lore.kernel.org / LKML-style) into CDP,
44
following the same worker pattern as `services/apps/git_integration`:
55
FastAPI lifespan + async poll worker + asyncpg + aiokafka.
66

7-
Status: under construction (CM-1318). See the plan/step tracker referenced
8-
in the PR for build progress.
7+
Status: hardcoded lists only, no onboarding UI (CM-1318). See the plan/step
8+
tracker referenced in the PR for build progress.
99

1010
## Architecture (planned)
1111

@@ -33,3 +33,43 @@ endpoint on host port `8086`.
3333

3434
Other targets: `make lint`, `make format`, `make test`, `make rebuild`. See
3535
`make help` for the full list.
36+
37+
## Onboarding a list (dev, hardcoded)
38+
39+
There is no UI yet — a list is onboarded by inserting its rows directly.
40+
`dev/seed.sql` does this for one example list (`lore.kernel.org/git`):
41+
42+
```bash
43+
psql "$CROWD_DB_WRITE_URI" -f dev/seed.sql
44+
```
45+
46+
It creates a `public.integrations` row (`platform='groupsio'`), a
47+
`mailinglist.lists` row pointing at an existing segment, and a
48+
`mailinglist."listProcessing"` row in state `pending` — which the worker's
49+
`acquire_onboarding_list()` will pick up on its next poll. Edit the `name`
50+
and `"sourceUrl"` values in the file to onboard a different lore list.
51+
52+
## End-to-end verification
53+
54+
1. Apply `backend/src/database/migrations/V1784048135__mailinglist-schema.sql`
55+
and run `dev/seed.sql` against the target Postgres.
56+
2. Start Kafka and a `data_sink_worker` consumer on the `data-sink-worker` topic.
57+
3. `make run` (or `uv run uvicorn crowdmail.server:app`) — worker polls, mirrors
58+
the seeded list via `public-inbox-clone`, parses new commits, and:
59+
- inserts rows into `integration.results` with `state='pending'` and
60+
`data.type='activity'`,
61+
- emits one Kafka message per result (`process_integration_result`,
62+
`platform=groupsio`),
63+
- advances `mailinglist."listProcessing"."lastProcessedHeads"` and sets
64+
`state='completed'`.
65+
4. Confirm `data_sink_worker` consumes the message, resolves the integration's
66+
`platform='groupsio'`, and creates the member (email-based verified
67+
`username`+`email` identities) and activity.
68+
5. `GET http://localhost:8086/` returns the FastAPI health check.
69+
70+
## Out of scope (CM-1318)
71+
72+
- Onboarding UI/backend to create `integrations` + `mailinglist.lists` rows
73+
(this is what `dev/seed.sql` stands in for).
74+
- Whether `member_join`/`member_leave` activities are derivable from lore, or
75+
only `message` initially.
Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
-- Dev seed: one hardcoded mailing list (CM-1318), no onboarding UI yet.
2+
-- Run against a local CDP postgres. Picks an arbitrary existing segment —
3+
-- replace the subquery with a real segmentId for anything beyond local testing.
4+
5+
BEGIN;
6+
7+
WITH target_segment AS (
8+
SELECT id AS "segmentId" FROM segments ORDER BY "createdAt" LIMIT 1
9+
),
10+
new_integration AS (
11+
INSERT INTO public.integrations (
12+
id, platform, status, settings, "tenantId", "createdAt", "updatedAt"
13+
)
14+
SELECT
15+
uuid_generate_v4(),
16+
'groupsio',
17+
'done',
18+
'{}'::jsonb,
19+
'875c38bd-2b1b-4e91-ad07-0cfbabb4c49f', -- DEFAULT_TENANT_ID
20+
NOW(),
21+
NOW()
22+
RETURNING id AS "integrationId"
23+
),
24+
new_list AS (
25+
INSERT INTO mailinglist.lists (
26+
id, name, "sourceUrl", "segmentId", "integrationId", "createdAt", "updatedAt"
27+
)
28+
SELECT
29+
uuid_generate_v4(),
30+
'kernel/git',
31+
'https://lore.kernel.org/git',
32+
target_segment."segmentId",
33+
new_integration."integrationId",
34+
NOW(),
35+
NOW()
36+
FROM target_segment, new_integration
37+
RETURNING id AS "listId"
38+
)
39+
INSERT INTO mailinglist."listProcessing" (
40+
"listId", state, priority, "lastProcessedHeads", "createdAt", "updatedAt"
41+
)
42+
SELECT "listId", 'pending', 2, '{}'::jsonb, NOW(), NOW()
43+
FROM new_list;
44+
45+
COMMIT;

0 commit comments

Comments
 (0)