Commit f4fa24b
feat: add ParquetMergePipeline supervisor and publisher feedback (Phase 3d+3e)
Phase 3 pipeline integration, combined supervisor and feedback PR:
- ParquetMergePipeline supervisor: spawns all merge actors (publisher →
sequencer → uploader → executor → downloader → planner), health-checks
with periodic supervision loop, respawn on failure with backoff,
graceful shutdown via FinishPendingMergesAndShutdownPipeline that
disconnects feedback and runs finalize policy. 3 tests.
- Publisher feedback: add parquet_merge_planner_mailbox_opt to Publisher
(feature-gated behind cfg(feature = "metrics")). After successful
ParquetSplitsUpdate publish of new ingested splits, sends ParquetNewSplits
to the planner. Merge outputs (non-empty replaced_split_ids) are not
fed back to avoid infinite loops.
- DisconnectMergePlanner extended to clear both Tantivy and Parquet planner
mailboxes, supporting shutdown drain for both pipeline types.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>1 parent 5937440 commit f4fa24b
5 files changed
Lines changed: 607 additions & 0 deletions
File tree
- quickwit/quickwit-indexing/src/actors
- metrics_pipeline
Lines changed: 4 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
41 | 41 | | |
42 | 42 | | |
43 | 43 | | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
44 | 48 | | |
45 | 49 | | |
46 | 50 | | |
| |||
Lines changed: 2 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
27 | 27 | | |
28 | 28 | | |
29 | 29 | | |
| 30 | + | |
30 | 31 | | |
31 | 32 | | |
32 | 33 | | |
| |||
50 | 51 | | |
51 | 52 | | |
52 | 53 | | |
| 54 | + | |
53 | 55 | | |
54 | 56 | | |
55 | 57 | | |
| |||
0 commit comments