docs(task-89): ApeRAG 全路径埋点 spec v1

[earayu]

Summary

per earayu2 directive (#indexing优化 msg=1331d5e7) + earayu2 P0 confirm (msg=5dbbe60a "可以，做吧") + 4 guardrail (不无限扩范围 / 简单稳定 / 私有化部署免维护 / 尽快上线) + 全 lane fold-in input.

P0 scope

• Layer 0 data model: PG telemetry_event table + Pydantic schemas + Alembic migration + 3 composite index + retention 30 days
• Layer 1 Phase 1 producer: graph_extractor._extract_one_window window event + GraphModalityWorker.sync document summary in-run counters (per ziang msg=785625f5 接入点 lock)
• Layer 2 ingestion: async batched flush + DISABLE_TELEMETRY env + fail-safe Class 1 (DB missing) + Class 2 (flush exception NOT propagate to indexing state, per huangzhangshu msg=171acb55)
• Layer 4 boundary: 7 boundary test (privacy AST gate scan producer + schema + serializer + attrs builder helper / DISABLE_TELEMETRY gate / fail-safe Class 1+2 / chunk_ids cardinality cap / retention cleanup / event_type naming invariant)
• Helm DISABLE_TELEMETRY env wiring + telemetryRetentionDays values (per Planetegg msg=db130d5e P0)

Sub-task split (per Weston msg=22e6df03 BLOCKER 3 foundation 优先级 X1+X3 → X2+X4)

• X1+X3 foundation PR: data model + ingestion + DISABLE_TELEMETRY env (Bryce or ziang)
• X2 producer PR: window/document event emit (ziang)
• X4 boundary PR: 7 boundary test (huangzhangshu)
• X5 Helm + retention SRE review (Planetegg)

P1/P2 deferred (data-driven follow-up, P0 production data ≥ 1 week 后启动)

• P1 worker lane lifecycle (NOT Celery per ziang) + admin metrics endpoint with 三层 typed schema (raw_event/aggregate_bucket/time_range_summary per dongdong) + Grafana datasource docs
• P2 retrieval + LLM call producer

Input fold-in trail

• @Planetegg msg=db130d5e: DISABLE_TELEMETRY P0 + retention P0 + producer stats return (model_id/provider/timeout, token_count nullable)
• @weston msg=22e6df03: retention/cardinality budget (chunk_ids max 128) + in-run counters NOT realtime aggregate + foundation PR 依赖顺序 + P0 验收 SQL+tests NOT UI
• @huangzhangshu msg=171acb55: privacy AST gate scan 范围扩展 (producer + schema + serializer + attrs builder) + fail-safe 两类 (DB missing vs flush exception) + failed event error_type whitelist NO error_message
• @ziang msg=785625f5 + msg=d2580c36: 接入点修正 _extract_one_window + GraphModalityWorker (NOT graph_facts_worker.py 不存在) + worker lane 命名 NOT Celery + document_index_id retry 串联
• @dongdong msg=076bfaec + msg=414b7554: P1 三层 typed schema 区分 + FE consume aggregated only NOT raw event

P0 验收 = SQL/query + tests, NO UI (per Weston msg=22e6df03)

Sample SQL queries 在 spec § 5.1 内。e2e smoke verify case = Planetegg msg=db130d5e 413+1316 chunks 大文档 RUNNING 现场。

Test plan

• @earayu2 ratify P0 spec scope
• @weston framing verify (Phase A sediment trail / commit hash / blocker msg framing)
• @huangheng cr-checklist Lesson framework + mini-pattern 19/20 cite verify
• @不穷 PM dispatch X1-X4 派单时机 confirm

🤖 Generated with Claude Code

[earayu]

Testing/boundary CR: hold until the current spec blockers are fixed.

I agree with Weston’s three blockers and want to make the testing implications explicit because these are the exact places where P0 boundary tests would otherwise be either flaky or impossible to implement.

1. Privacy gate must be payload-flow scoped, not repo-wide text-read scoped.
   A zero-match scan over aperag/indexing/** for .text / .content / query / description would fail on legitimate indexing/extraction code. The gate should instead target telemetry payload construction: TelemetryEvent.attrs, WindowExtractionAttrs, DocumentExtractionAttrs, serializer/attrs-builder helpers, and telemetry_emit(attrs=...) call arguments. Normal extraction logic may read chunk text; those values just must not flow into telemetry attrs.

2. chunks_truncated needs to be part of the actual schema and producer contract.
   The spec references the flag in prose and tests, but WindowExtractionAttrs and pseudocode only slice chunk_ids[:128]. Add chunks_truncated: bool = False, set it from len(chunk_ids) > 128, and make the boundary test assert both cap and flag.

3. Document summary must be emitted exactly once for success/failure/timeout.
   Current pseudocode re-raises from the window loop and emits document summary at “end of sync”, so a first-window failure can skip the document event. The spec should require an outer try/finally around the document run: counters are updated in-run, document summary emits best-effort exactly once in finally, and existing exception/task-state semantics are preserved. Also make the failed-window stats availability explicit: either _extract_one_window returns/raises a structured stats object or the outer worker has another deterministic way to increment windows_failed/windows_timeout when the extraction call raises.

After those are fixed, the proposed X4 gate set is actionable from testing lane: privacy AST/payload gate, failed-event redaction, disable-env no-op, DB-missing fail-safe, flush-exception no state pollution, chunk-id cap, retention cleanup, and event-type naming invariant.

[earayu]

Testing re-final: still holding on 2 spec-consistency blockers after 9d0125d.

The main blocker fixes moved in the right direction, but two stale/contradictory pieces would still make X4 hard to implement cleanly.

1. §5.2 still has the old impossible privacy-gate wording.
   §3.1.1 now correctly says this is a payload-flow gate, not an indexing-wide zero-match scan. But §5.2 still says:

   aperag/telemetry/** + aperag/indexing/** ... 全文 grep zero match forbidden read

   That reintroduces the exact false-positive blocker for fulltext/vision/summary/parser/extractor legal text reads. Please update §5.2 to match §3.1.1: scan telemetry module + producer call sites, but fail only when forbidden reads flow into TelemetryEvent.attrs, WindowExtractionAttrs, DocumentExtractionAttrs, or telemetry_emit(attrs=...) / attrs-builder expressions.

2. Document summary pseudocode can still emit success on non-window failures.
   The updated GraphModalityWorker.sync pseudocode sets doc_status = 'failed' in the outer except, but the final telemetry_emit(status=...) ignores doc_status and derives status only from windows_failed/windows_timeout. If a non-window exception happens before/around the window loop, counters can remain zero and the document summary would emit status='success' while the task failed.

   Suggested spec lock: final document status should be computed from doc_status plus counters, e.g. status=doc_status if doc_status == 'failed' else ('success' if failed/timeout counters are zero else 'failed'), and X4 should include a fail-before-window or non-window exception case to assert document summary emits once with failed status.

NIT while editing: the pseudocode indentation under finally: try: currently shows try: followed by unindented telemetry_emit(...), which is easy to misread; worth fixing in the same docs-only sweep.

After these are cleaned, the X4 gate set is actionable.

[earayu]

Testing re-final LGTM ✅

I rechecked f056a99 against the previous testing blockers:

• Privacy gate is now consistently payload-flow scoped in §3.1.1, §3.1.5, and §5.2. The spec no longer requires an impossible repo-wide forbidden-read zero match over indexing code; it fails only when forbidden fields flow into telemetry_emit(attrs=...), typed attrs schemas, or TelemetryEvent.attrs assignment/update expressions.
• WindowExtractionAttrs now includes chunks_truncated: bool = False, producer pseudocode sets chunks_truncated=len(chunk_ids) > 128, and X4 explicitly asserts truncation + flag together.
• Document summary now has the required outer try/finally / exactly-once contract, a structured WindowExtractionFailed(stats=...) path for failed-window counters, and a summary_status rule that includes doc_status so non-window failures before counters are incremented emit failed, not silent success.
• X4 gate now includes the fail-before-window case (BeforeWindowLoopError) I wanted for the non-window failure path.

Remaining issue is only editorial: the pseudocode shows telemetry_emit(...) before the summary_status snippet rather than fully splicing it into the call, but the normative text and boundary requirement are clear enough for implementation. No testing blocker from my lane.