Merge pull request #447 from dahlia/main

Move remote reply scraping to a worker

Author: dahlia

AGENTS.md

@@ -382,21 +382,26 @@ STORAGE_URL_BASE=https://your-bucket.s3.amazonaws.com
 
 ### Optional variables
 
-| Variable                       | Default | Description                              |
-| ------------------------------ | ------- | ---------------------------------------- |
-| `PORT`                         | 3000    | Server port                              |
-| `BIND`                         | -       | Bind address                             |
-| `NODE_TYPE`                    | all     | Node type: `all`, `web`, or `worker`     |
-| `BEHIND_PROXY`                 | false   | Trust proxy headers                      |
-| `LOG_LEVEL`                    | info    | Logging level                            |
-| `LOG_QUERY`                    | false   | Log database queries                     |
-| `LOG_FILE`                     | -       | JSON log file path                       |
-| `SENTRY_DSN`                   | -       | Sentry error tracking                    |
-| `HOME_URL`                     | -       | Home page redirect URL                   |
-| `ALLOW_PRIVATE_ADDRESS`        | false   | Disable SSRF protection                  |
-| `REMOTE_ACTOR_FETCH_POSTS`     | 10      | Posts to fetch from remote actors        |
-| `REMOTE_ACTOR_STALENESS_DAYS`  | 7       | Days before remote actor data is stale   |
-| `REFRESH_ACTORS_ON_INTERACTION`| false   | Refresh actors on all activity types     |
+| Variable                                 | Default | Description                              |
+| ---------------------------------------- | ------- | ---------------------------------------- |
+| `PORT`                                   | 3000    | Server port                              |
+| `BIND`                                   | -       | Bind address                             |
+| `NODE_TYPE`                              | all     | Node type: `all`, `web`, or `worker`     |
+| `BEHIND_PROXY`                           | false   | Trust proxy headers                      |
+| `LOG_LEVEL`                              | info    | Logging level                            |
+| `LOG_QUERY`                              | false   | Log database queries                     |
+| `LOG_FILE`                               | -       | JSON log file path                       |
+| `SENTRY_DSN`                             | -       | Sentry error tracking                    |
+| `HOME_URL`                               | -       | Home page redirect URL                   |
+| `ALLOW_PRIVATE_ADDRESS`                  | false   | Disable SSRF protection                  |
+| `REMOTE_ACTOR_FETCH_POSTS`               | 10      | Posts to fetch from remote actors        |
+| `REMOTE_ACTOR_STALENESS_DAYS`            | 7       | Days before remote actor data is stale   |
+| `REFRESH_ACTORS_ON_INTERACTION`          | false   | Refresh actors on all activity types     |
+| `REMOTE_REPLIES_SCRAPE_DEPTH`            | 2       | Reply scraping depth for remote posts    |
+| `REMOTE_REPLIES_SCRAPE_MAX_ITEMS`        | 100     | Replies to process per scraping job      |
+| `REMOTE_REPLIES_SCRAPE_INTERVAL_SECONDS` | 5       | Delay between scrape requests per origin |
+| `REMOTE_REPLIES_SCRAPE_BACKOFF_SECONDS`  | 300     | Backoff for 429 without `Retry-After`    |
+| `REMOTE_REPLIES_SCRAPE_COOLDOWN_SECONDS` | 300     | Completed scrape deduplication window    |
 
 
 Adding new environment variables

CHANGES.md

@@ -38,6 +38,20 @@ To be released.
      -  Preview card scraping, media processing, and authentication helpers now
         load on demand instead of eagerly during route registration.
 
+ -  Moved remote replies scraping from synchronous post ingestion to a
+    rate-limited background worker.  Remote posts now enqueue reply collection
+    scraping jobs instead of fetching nested replies inline, which prevents
+    slow or very large remote reply collections from delaying federation
+    processing.  [[#445], [#447]]
+
+     -  Added per-origin throttling and `429 Too Many Requests` backoff for
+        remote replies scraping.
+     -  Added bounded scraping controls:
+        `REMOTE_REPLIES_SCRAPE_DEPTH`, `REMOTE_REPLIES_SCRAPE_MAX_ITEMS`,
+        `REMOTE_REPLIES_SCRAPE_INTERVAL_SECONDS`,
+        `REMOTE_REPLIES_SCRAPE_BACKOFF_SECONDS`, and
+        `REMOTE_REPLIES_SCRAPE_COOLDOWN_SECONDS`.
+
  -  Added automatic refresh of stale remote actor profiles.  When receiving
     activities like `Announce` or `Create(Note)`, Hollo now checks if the
     actor's cached data is stale and asynchronously refreshes their profile
@@ -137,6 +151,8 @@ To be released.
 [#425]: https://github.com/fedify-dev/hollo/issues/425
 [#435]: https://github.com/fedify-dev/hollo/issues/435
 [#436]: https://github.com/fedify-dev/hollo/pull/436
+[#445]: https://github.com/fedify-dev/hollo/issues/445
+[#447]: https://github.com/fedify-dev/hollo/pull/447
 [Fedify debugger]: https://fedify.dev/manual/debug
 
 

bin/server.ts

@@ -71,14 +71,17 @@ if (NODE_TYPE === "worker" || NODE_TYPE === "all") {
     { federation },
     { startImportWorker, stopImportWorker },
     { startCleanupWorker, stopCleanupWorker },
+    { startRemoteReplyScrapeWorker, stopRemoteReplyScrapeWorker },
   ] = await Promise.all([
     import("../src/federation"),
     import("../src/import/worker"),
     import("../src/cleanup/worker"),
+    import("../src/federation/replies-worker"),
   ]);
   stopWorkers = () => {
     stopImportWorker();
     stopCleanupWorker();
+    stopRemoteReplyScrapeWorker();
   };
 
   // Start the Fedify message queue
@@ -93,8 +96,11 @@ if (NODE_TYPE === "worker" || NODE_TYPE === "all") {
   // Start the workers for background job processing
   startImportWorker();
   startCleanupWorker();
+  startRemoteReplyScrapeWorker();
 
-  console.log("Worker started (Fedify queue + Import worker + Cleanup worker)");
+  console.log(
+    "Worker started (Fedify queue + Import worker + Cleanup worker + Remote reply scrape worker)",
+  );
 }
 
 // Graceful shutdown handling

docs/src/content/docs/install/env.mdx

@@ -46,9 +46,11 @@ e.g., `UTC`, `America/New_York`, `Asia/Tokyo`.
 
 Controls which components run in this process.  Valid values are:
 
- -  `all` (default): Run web server, Fedify message queue, import worker and cleanup worker
+ -  `all` (default): Run web server, Fedify message queue, import worker,
+    cleanup worker and remote replies scrape worker
  -  `web`: Run only the web server (HTTP API)
- -  `worker`: Run only workers (Fedify message queue + import worker + cleanup worker)
+ -  `worker`: Run only workers (Fedify message queue + import worker +
+    cleanup worker + remote replies scrape worker)
 
 This allows separating the web server from background workers for better
 scalability.  When running high-traffic instances with many followers,
@@ -107,6 +109,40 @@ encountered first time.
 
 `10` by default.
 
+### `REMOTE_REPLIES_SCRAPE_DEPTH` <Badge text="Optional" />
+
+The number of remote reply levels to scrape in background worker jobs.
+Set this to `0` to disable remote replies scraping.
+
+`2` by default.
+
+### `REMOTE_REPLIES_SCRAPE_MAX_ITEMS` <Badge text="Optional" />
+
+The maximum number of reply items to persist from a single remote replies
+scraping job.
+
+`100` by default.
+
+### `REMOTE_REPLIES_SCRAPE_INTERVAL_SECONDS` <Badge text="Optional" />
+
+The minimum delay between remote replies scraping requests to the same origin.
+
+`5` by default.
+
+### `REMOTE_REPLIES_SCRAPE_BACKOFF_SECONDS` <Badge text="Optional" />
+
+The fallback delay before retrying a remote replies scraping job after an HTTP
+429 response when the remote server does not provide `Retry-After`.
+
+`300` by default.
+
+### `REMOTE_REPLIES_SCRAPE_COOLDOWN_SECONDS` <Badge text="Optional" />
+
+The time window during which completed remote replies scraping jobs suppress
+duplicate jobs for the same replies collection.
+
+`300` by default.
+
 ### `REMOTE_ACTOR_STALENESS_DAYS` <Badge text="Optional" />
 
 The number of days after which a remote actor's cached data is considered stale.

docs/src/content/docs/install/workers.mdx

@@ -30,21 +30,22 @@ Worker separation is beneficial when:
 How it works
 ------------
 
-Hollo has three main components:
+Hollo has five main components:
 
  1. *Web server*: Handles HTTP requests (API, web UI)
  2. *Fedify message queue*: Processes ActivityPub inbox/outbox messages
  3. *Import worker*: Handles background data import jobs
  4. *Cleanup worker*: Handles background cleanup jobs
+ 5. *Remote replies scrape worker*: Fetches remote replies collections slowly
 
-By default (`NODE_TYPE=all`), all three run in a single process. You can
+By default (`NODE_TYPE=all`), all five run in a single process. You can
 separate them using the `NODE_TYPE` environment variable:
 
-| `NODE_TYPE` | Web server | Fedify queue | Import worker | Cleanup worker |
-| ----------- | ---------- | ------------ | ------------- | -------------- |
-| `all` (default) | ✓ | ✓ | ✓ | ✓ |
-| `web` | ✓ | ✗ | ✗ | ✗ |
-| `worker` | ✗ | ✓ | ✓ | ✓ |
+| `NODE_TYPE` | Web server | Fedify queue | Import worker | Cleanup worker | Remote replies scrape worker |
+| ----------- | ---------- | ------------ | ------------- | -------------- | ---------------------------- |
+| `all` (default) | ✓ | ✓ | ✓ | ✓ | ✓ |
+| `web` | ✓ | ✗ | ✗ | ✗ | ✗ |
+| `worker` | ✗ | ✓ | ✓ | ✓ | ✓ |
 
 All nodes share the same PostgreSQL database, which acts as the message queue
 backend using `LISTEN`/`NOTIFY` for real-time message delivery.
@@ -253,11 +254,11 @@ sudo journalctl -u hollo-worker -f
 When a worker node starts, you should see:
 
 ```
-Worker started (Fedify queue + Import worker + Cleanup worker)
+Worker started (Fedify queue + Import worker + Cleanup worker + Remote reply scrape worker)
 ```
 
-Watch for messages about processing activities and import and cleanup jobs to 
-confirm the worker is functioning correctly.
+Watch for messages about processing activities, imports, cleanups, and remote
+replies scraping to confirm the worker is functioning correctly.
 
 
 Troubleshooting

docs/src/content/docs/ja/install/env.mdx

@@ -47,9 +47,11 @@ openssl rand -hex 32
 
 このプロセスで実行するコンポーネントを制御します。有効な値は：
 
- -  `all`（デフォルト）：Webサーバー、Fedifyメッセージキュー、インポートワーカーをすべて実行
+ -  `all`（デフォルト）：Webサーバー、Fedifyメッセージキュー、
+    インポートワーカー、クリーンアップワーカー、リモート返信取得ワーカーをすべて実行
  -  `web`：Webサーバー（HTTP API）のみ実行
- -  `worker`：ワーカーのみ実行（Fedifyメッセージキュー + インポートワーカー）
+ -  `worker`：ワーカーのみ実行（Fedifyメッセージキュー + インポートワーカー +
+    クリーンアップワーカー + リモート返信取得ワーカー）
 
 これにより、スケーラビリティを向上させるためにWebサーバーとバックグラウンドワーカーを分離できます。
 フォロワーが多い高トラフィックインスタンスを運営する場合、ワーカーを分離することでパフォーマンスが向上します。
@@ -106,6 +108,39 @@ HolloがL7ロードバランサーの後ろにある場合（通常はそうす
 
 デフォルトは`10`です。
 
+### `REMOTE_REPLIES_SCRAPE_DEPTH` <Badge text="オプション" />
+
+バックグラウンドワーカージョブで取得するリモート返信の階層数。
+`0`に設定すると、リモート返信の取得を無効にします。
+
+デフォルトは`2`です。
+
+### `REMOTE_REPLIES_SCRAPE_MAX_ITEMS` <Badge text="オプション" />
+
+1つのリモート返信取得ジョブから保存する返信アイテムの最大数。
+
+デフォルトは`100`です。
+
+### `REMOTE_REPLIES_SCRAPE_INTERVAL_SECONDS` <Badge text="オプション" />
+
+同じoriginへのリモート返信取得リクエスト間の最小待機秒数。
+
+デフォルトは`5`です。
+
+### `REMOTE_REPLIES_SCRAPE_BACKOFF_SECONDS` <Badge text="オプション" />
+
+リモートサーバーが`Retry-After`を返さずHTTP 429を返した場合に、
+リモート返信取得ジョブを再試行するまでの待機秒数。
+
+デフォルトは`300`です。
+
+### `REMOTE_REPLIES_SCRAPE_COOLDOWN_SECONDS` <Badge text="オプション" />
+
+完了したリモート返信取得ジョブが同じ返信コレクションの重複ジョブを
+抑制する時間。
+
+デフォルトは`300`です。
+
 ### `REMOTE_ACTOR_STALENESS_DAYS` <Badge text="オプション" />
 
 リモートアクターのキャッシュされたデータが古いと見なされるまでの日数。

docs/src/content/docs/ja/install/workers.mdx

@@ -30,20 +30,22 @@ import { Aside, Code, Tabs, TabItem } from '@astrojs/starlight/components';
 動作の仕組み
 ------------
 
-Holloは3つの主要コンポーネントで構成されています:
+Holloは5つの主要コンポーネントで構成されています:
 
  1. *Webサーバー*: HTTPリクエスト(API、Web UI)を処理
  2. *Fedifyメッセージキュー*: ActivityPub inbox/outboxメッセージを処理
  3. *インポートワーカー*: バックグラウンドデータインポートジョブを処理
+ 4. *クリーンアップワーカー*: バックグラウンドクリーンアップジョブを処理
+ 5. *リモート返信取得ワーカー*: リモート返信コレクションをゆっくり取得
 
-デフォルト(`NODE_TYPE=all`)では、3つすべてが単一プロセスで実行されます。
+デフォルト(`NODE_TYPE=all`)では、5つすべてが単一プロセスで実行されます。
 `NODE_TYPE`環境変数を使用して分離できます:
 
-| `NODE_TYPE` | Webサーバー | Fedifyキュー | インポートワーカー |
-| ----------- | ----------- | ------------ | ------------------ |
-| `all` (デフォルト) | ✓ | ✓ | ✓ |
-| `web` | ✓ | ✗ | ✗ |
-| `worker` | ✗ | ✓ | ✓ |
+| `NODE_TYPE` | Webサーバー | Fedifyキュー | インポートワーカー | クリーンアップワーカー | 返信ワーカー |
+| ----------- | ----------- | ------------ | ------------------ | ------------------------ | ------------ |
+| `all` (デフォルト) | ✓ | ✓ | ✓ | ✓ | ✓ |
+| `web` | ✓ | ✗ | ✗ | ✗ | ✗ |
+| `worker` | ✗ | ✓ | ✓ | ✓ | ✓ |
 
 すべてのノードは同じPostgreSQLデータベースを共有し、
 `LISTEN`/`NOTIFY`を使用してリアルタイムメッセージ配信のための
@@ -251,11 +253,12 @@ sudo journalctl -u hollo-worker -f
 ワーカーノードが起動すると、次のように表示されます:
 
 ```
-Worker started (Fedify queue + Import worker)
+Worker started (Fedify queue + Import worker + Cleanup worker + Remote reply scrape worker)
 ```
 
 ワーカーが正常に機能していることを確認するには、
-アクティビティ処理とインポートジョブに関するメッセージを確認してください。
+アクティビティ処理、インポート、クリーンアップ、リモート返信取得に関する
+メッセージを確認してください。
 
 
 トラブルシューティング

docs/src/content/docs/ko/install/env.mdx

@@ -46,9 +46,11 @@ openssl rand -hex 32
 
 이 프로세스에서 실행할 구성 요소를 제어합니다. 유효한 값은:
 
- -  `all` (기본값): 웹 서버, Fedify 메시지 큐, 임포트 워커를 모두 실행
+ -  `all` (기본값): 웹 서버, Fedify 메시지 큐, 임포트 워커, 정리 워커,
+    원격 답글 가져오기 워커를 모두 실행
  -  `web`: 웹 서버(HTTP API)만 실행
- -  `worker`: 워커만 실행 (Fedify 메시지 큐 + 임포트 워커)
+ -  `worker`: 워커만 실행 (Fedify 메시지 큐 + 임포트 워커 + 정리 워커 +
+    원격 답글 가져오기 워커)
 
 이를 통해 더 나은 확장성을 위해 웹 서버와 백그라운드 워커를 분리할 수 있습니다.
 팔로워가 많은 고트래픽 인스턴스를 운영할 때 워커를 분리하면 성능이 향상될 수 있습니다.
@@ -104,6 +106,39 @@ Hollo가 L7 로드 밸런서 뒤에 위치할 경우 (일반적으로 그래야
 
 기본값은 `10`입니다.
 
+### `REMOTE_REPLIES_SCRAPE_DEPTH` <Badge text="선택" />
+
+백그라운드 워커 작업에서 가져올 원격 답글의 계층 수입니다.
+`0`으로 설정하면 원격 답글 가져오기를 비활성화합니다.
+
+기본값은 `2`입니다.
+
+### `REMOTE_REPLIES_SCRAPE_MAX_ITEMS` <Badge text="선택" />
+
+원격 답글 가져오기 작업 하나에서 저장할 답글 항목의 최대 수입니다.
+
+기본값은 `100`입니다.
+
+### `REMOTE_REPLIES_SCRAPE_INTERVAL_SECONDS` <Badge text="선택" />
+
+같은 origin에 대한 원격 답글 가져오기 요청 사이의 최소 대기 시간(초)입니다.
+
+기본값은 `5`입니다.
+
+### `REMOTE_REPLIES_SCRAPE_BACKOFF_SECONDS` <Badge text="선택" />
+
+원격 서버가 `Retry-After` 없이 HTTP 429를 반환했을 때 원격 답글
+가져오기 작업을 다시 시도하기 전까지 기다릴 시간(초)입니다.
+
+기본값은 `300`입니다.
+
+### `REMOTE_REPLIES_SCRAPE_COOLDOWN_SECONDS` <Badge text="선택" />
+
+완료된 원격 답글 가져오기 작업이 같은 답글 컬렉션의 중복 작업을
+억제하는 시간(초)입니다.
+
+기본값은 `300`입니다.
+
 ### `REMOTE_ACTOR_STALENESS_DAYS` <Badge text="선택" />
 
 원격 액터의 캐시된 데이터가 오래된 것으로 간주되기까지의 일수.

docs/src/content/docs/ko/install/workers.mdx

@@ -30,20 +30,22 @@ import { Aside, Code, Tabs, TabItem } from '@astrojs/starlight/components';
 작동 방식
 ---------
 
-Hollo는 세 가지 주요 구성 요소로 이루어져 있습니다:
+Hollo는 다섯 가지 주요 구성 요소로 이루어져 있습니다:
 
  1. *웹 서버*: HTTP 요청(API, 웹 UI) 처리
  2. *Fedify 메시지 큐*: ActivityPub inbox/outbox 메시지 처리
  3. *임포트 워커*: 백그라운드 데이터 가져오기 작업 처리
+ 4. *정리 워커*: 백그라운드 정리 작업 처리
+ 5. *원격 답글 가져오기 워커*: 원격 답글 컬렉션을 천천히 가져오기
 
-기본적으로(`NODE_TYPE=all`) 세 가지 모두 단일 프로세스에서 실행됩니다.
+기본적으로(`NODE_TYPE=all`) 다섯 가지 모두 단일 프로세스에서 실행됩니다.
 `NODE_TYPE` 환경 변수를 사용하여 분리할 수 있습니다:
 
-| `NODE_TYPE` | 웹 서버 | Fedify 큐 | 임포트 워커 |
-| ----------- | ------- | --------- | ----------- |
-| `all` (기본값) | ✓ | ✓ | ✓ |
-| `web` | ✓ | ✗ | ✗ |
-| `worker` | ✗ | ✓ | ✓ |
+| `NODE_TYPE` | 웹 서버 | Fedify 큐 | 임포트 워커 | 정리 워커 | 답글 워커 |
+| ----------- | ------- | --------- | ----------- | --------- | --------- |
+| `all` (기본값) | ✓ | ✓ | ✓ | ✓ | ✓ |
+| `web` | ✓ | ✗ | ✗ | ✗ | ✗ |
+| `worker` | ✗ | ✓ | ✓ | ✓ | ✓ |
 
 모든 노드는 동일한 PostgreSQL 데이터베이스를 공유하며,
 `LISTEN`/`NOTIFY`를 사용하여 실시간 메시지 전달을 위한 메시지 큐 백엔드로 작동합니다.
@@ -250,10 +252,11 @@ sudo journalctl -u hollo-worker -f
 워커 노드가 시작되면 다음과 같이 표시됩니다:
 
 ```
-Worker started (Fedify queue + Import worker)
+Worker started (Fedify queue + Import worker + Cleanup worker + Remote reply scrape worker)
 ```
 
-워커가 정상적으로 작동하는지 확인하려면 액티비티 처리 및 가져오기 작업에 대한 메시지를 확인하세요.
+워커가 정상적으로 작동하는지 확인하려면 액티비티 처리, 가져오기, 정리,
+원격 답글 가져오기에 대한 메시지를 확인하세요.
 
 
 문제 해결