test(compat): task #61 P1 — bulk_upsert_entity_with_lineage_parts cross-backend by earayu · Pull Request #1927 · apecloud/ApeRAG

earayu · 2026-04-30T04:52:31Z

Summary

task #61 DB 兼容审计 P1 testing-lane slice (task #67，claimed in msg=e02c3028)。补 bulk_upsert_entity_with_lineage_parts 跨 backend test — 这是我 task #67 active scan 发现的 P0 protocol 覆盖缺口（msg=3e93bb64），PM @不穷在 msg=10b753e8 elevate 为 P0：

P0 新候选（@ziang task #64 重点看）：bulk_upsert_entity_with_lineage_parts — bulk write，3 后端 batch 限制/原子性/error handling 差异最大，完全无 test cover

改动

1 文件 / +147 / -0 — tests/integration/compat/test_lineage_graph_compat.py 加 7 cases：

Test	Pin
`…_empty_is_noop`	empty parts MUST NOT 隐式 create row
`…_rejects_mixed_names`	mixed names MUST raise `ValueError` (atomicity guarantee)
`…_round_trip`	3 distinct (doc_id, parse_version) parts visible after one bulk
`…_dedup_last_wins_within_bulk`	same-key parts in same bulk collapse, last description wins
`…_replaces_existing_same_key`	bulk strips matching pre-existing rows then appends
`…_appends_distinct_keys`	bulk MUST NOT wipe unrelated pre-existing lineage
`…_entity_type_last_wins`	last bulk part's `entity_type` is post-write value

每条 test 直接 cite aperag/indexing/graph.py:575+ Protocol contract docstring 作 truth source。

覆盖度

baseline: pytest --collect-only -q = 30 cases
post: 37 cases ✅
7 个新 case 完全 align 现有 parametrized fixture（PG / Neo4j / Nebula × 7 case = 21 真实 backend 验证 once env vars 配齐）

跟 chenyexuan PR #1926 关系

PR #1926 修 compat-test.yml paths filter（dead reference 指向 Wave 7 已删 aperag/domains/knowledge_graph/graphindex/，真实路径 aperag/indexing/graph_storage/）。没有 #1926 的 paths fix，本 PR 的测试在 CI 上不会触发 — 任何修改 graph_storage/*.py 的 PR 都不 trigger Backend-Compat-Test workflow。

两 PR 配对：

ci(compat): trigger Backend-Compat-Test on real graph_storage path #1926 (chenyexuan)：让 compat-test workflow 真触发
本 PR (#待编号)：让 compat-test 钉 bulk_upsert_entity_with_lineage_parts 跨 backend 一致性

CR 请求

@符炫炜 architect ratify (cross-link task [Features] integrate Feishu bot as a frontend of kubechat #72 spec v1 P0「必须一致」清单)
@ziang implementation CR (确认 7 case 跟 graph_storage/{neo4j,nebula,postgres}.py:bulk_upsert 实现 contract align — task [Features] A collection supports multi source of corpse #64 lane 自然交集)
@huangzhangshu testing lane primary CR (test 设计 + 7 case 覆盖度)
@huangheng line-level CR + Lesson framework 应用 (Lesson feat: socket message #13 v3 dual-side rewrite 钉 cross-backend invariant)

Test plan

pytest tests/integration/compat/test_lineage_graph_compat.py --collect-only -q → 37 collected (was 30)
ruff check ✅
make lint (pre-commit) ✅
CI: lint-and-unit + Backend-Compat-Test (after ci(compat): trigger Backend-Compat-Test on real graph_storage path #1926 merges, real backend run)
等 ci(compat): trigger Backend-Compat-Test on real graph_storage path #1926 merge → 本 PR rebase → 跑 compat-test 实证 7/7 pass on PG/Neo4j/Nebula backends（如有 backend 行为差异，作为 task [Features] add API /api/v1/kbclusters to list KubeBlocks clusters #61 P0 fix candidate 反向 surface）

后续

我 task #67 next slice：补 remove_relation_lineage_member (P1 — counterpart entity 已 test) + list_entities (P1 — pagination/sort cross-backend)。等本 PR merge 后跟进。

🤖 Generated with Claude Code

@huangheng

…tests Two non-blocking NITs from @huangheng msg=99b5ffd5 + @ziang msg=84f5c3cc re-CR on PR #1927 — fold-in to land more complete test: * `_rejects_mixed_names` now also asserts post-raise zero-side-effect (`get_entity("Alice") is None` + `get_entity("Bob") is None`) — pins Lesson #12 v6.4 aggregation-chain invariant: a backend that swapped validation order to raise AFTER the first row write would silently leak partial state. * New `_replay_is_idempotent` case — pins the Protocol's "Forward-only retry safety: per-part dedup so replays are idempotent" contract. A backend that appended on replay (instead of dedup-then-replace) would silently duplicate lineage members under retry. Coverage delta: 37 → 38 cross-backend cases. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Weston msg=13dd5e91 BLOCKER (score normalization severity drift): 保持 P0-V3+V4 P0 across §1.1 / §2.2 / §5.3 — score 方向是 caller 语义硬契约，不能在 PGVector/Qdrant 间显示反向。§2.2 加 P0-V3+V4 显式行 + §5.3 加 test_score_normalization_in_vector.py boundary test (跨 metric × 跨 adapter 全 6 cell parametrize). Streaming integrations (5 lane): 1. Bryce msg=23a2f514 P0-V1 first-principles 重新定性 — Qdrant legacy mode tenant isolation 是 collection name level 不是 query filter level (verify qdrant_connector.py:442-446)，下沉 P1-V4 defense-in-depth (legacy mode deprecation follow-up 候选). 2. Bryce msg=8e895471 11 vector findings — 4 P0 (cross-tenant 下沉 / filter silent / score V3+V4) + 3 P1 (collection init / batch atomicity / filter Or 语义) + 4 P2. 3. dongdong msg=4201465a + PR #1929 + cuiwenbo msg=bcec38ad — P0-D1 Helm worker Neo4j env missing (Singapore graph viz root-cause); P1-D1 e2e shape matrix gap; P1-D2 Nebula no Helm first-class; P1-D3 typed schema 缺 vector backend exposure. 4. chenyexuan NIT — Lesson #16 candidate cite added §6. 5. Planetegg msg=eb9de4b0 NIT — P2-S1 量化 max_nodes*2 default 1000→2000 / hybrid default 1000 max 5000; msg ID corrections §7 (msg=41665d7e Singapore multitenant verify, msg=eb9de4b0 P2-S1 quantification, dropped invalid msg=ec358a3e). 冬柏 PR #1927 commit b2234ae fold-in §5.3 (38 cases incl zero-side-effect + replay idempotency post-NIT). P0 list final: P0-V2 (filter silent, Bryce P0-A) + P0-V3+V4 (score normalization, Bryce P0-B) + P0-G1 (bulk_upsert, 冬柏 PR #1927) + P0-W1 (compat-test paths, chenyexuan PR #1926) + P0-D1 (Helm Neo4j env, dongdong PR #1929).

…ss-backend PM @不穷 elevated this Protocol method as a P0 audit gap (msg=10b753e8). Until now ``bulk_upsert_entity_with_lineage_parts`` (Wave 8 W8-2) had no cross-backend test in `tests/integration/compat/`, even though all three production backends (Postgres / Neo4j / Nebula) implement it and the indexing worker uses it for the LineageEntityMerger merge step. Bulk write paths are exactly where backend differences emerge — batch size limits, transaction atomicity, error handling, dedup contract — and the lack of a parametrized matrix here meant any silent drift in the bulk semantics would survive merge. This adds 7 new parametrized cases that pin the Protocol contract declared in `aperag/indexing/graph.py:575+`: * empty parts is a no-op (no implicit row creation) * mixed-name parts raise ValueError (atomicity guarantee) * round-trip: 3 distinct (document_id, parse_version) parts visible after * dedup last-wins within a single bulk call * bulk replaces existing rows on matching key (same as single upsert) * bulk with distinct keys appends, never wipes pre-existing lineage * per-part entity_type follows last-wins rule Coverage delta: 30 → 37 cross-backend cases (collect-only verified). Sister to chenyexuan PR #1926 — without that workflow path fix, this test never triggered on PRs that touch `aperag/indexing/graph_storage/*`. Both PRs together restore real CI gating on cross-backend regressions for the LineageGraphStore Protocol surface. Part of task #61 DB compat audit (earayu2 directive msg=f26b703e), testing-lane slice (task #67, claimed via msg=e02c3028). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

@huangheng

…tests Two non-blocking NITs from @huangheng msg=99b5ffd5 + @ziang msg=84f5c3cc re-CR on PR #1927 — fold-in to land more complete test: * `_rejects_mixed_names` now also asserts post-raise zero-side-effect (`get_entity("Alice") is None` + `get_entity("Bob") is None`) — pins Lesson #12 v6.4 aggregation-chain invariant: a backend that swapped validation order to raise AFTER the first row write would silently leak partial state. * New `_replay_is_idempotent` case — pins the Protocol's "Forward-only retry safety: per-part dedup so replays are idempotent" contract. A backend that appended on replay (instead of dedup-then-replace) would silently duplicate lineage members under retry. Coverage delta: 37 → 38 cross-backend cases. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

…audit (#1928) * docs(task-61): DB adapter compat spec v1 — vector + graph cross-impl audit Architect spec v1 起草 per earayu2 directive (msg=8b989470 / msg=2bad8e75 / msg=f26b703e) + PM 不穷 task #72 dispatch. Streaming evidence integration from 8 lanes: - huangheng msg=ed2f2973: 3 vector P0 candidates (cross-tenant / filter silent / collection init) - Bryce msg=8e895471 task #69: 11 vector findings (4 P0 + 3 P1 + 4 P2, including upgraded score normalization P0-V3/V4) - 冬柏 msg=3e93bb64 task #67: 3 missing Protocol method tests (bulk_upsert_entity_with_lineage_parts P0 + remove_relation_lineage P1 + list_entities P1) - chenyexuan msg=f298011e + PR #1926: workflow paths filter dead reference P0-W1 (in flight) - cuiwenbo msg=dfebf706 task #70: FE/UX 3 candidates (score, viz error vs empty, confidence_score) - Planetegg msg=db7fb085 + msg=41906f4 + msg=41665d7e task #65: alias resolution gather P2-S1 + Singapore QDRANT_MULTITENANT=True (no hot-fix needed) + env shape verify - ziang task #64 graph store audit (in_progress, will fold-in) - dongdong task #71 deploy/typed schema (in_progress, will fold-in) Spec structure: - §1 inventory by lane with file:line evidence - §2 缺口 by severity (P0 CRITICAL hot-fix candidate / P0 必修 / P1 允许差异 declare / P2 性能优化 / YAGNI) - §3 三层 design direction per Weston msg=85e527e3 framework - §4 sub-task dispatch (Phase A 8 lane parallel + Phase B per-P0 three-PR-pattern + Phase C P2 + Phase D PR #1926 unblock) - §5 acceptance: P0/P1 standards + boundary test gate + e2e + sample limitation免责 - §6 CR mandatory checklist citing Lesson #11-#16 family from PR #1916/#1924/#1922 sediment + new Lesson #16 candidate (workflow paths dead reference) Sample limitation: spec evidence from streaming surface, not huangzhangshu collected gap list — fix-forward amend after huangzhangshu lane completes + Bryce/ziang audit slice输出. Not blocking: PR #1925 task #30 B3 default=2, PR #1926 compat-test paths filter, Singapore 2pm release (env fix separate lane), task #31 graph node merge / task #33 P3 workflow gate. * docs(task-61): fix-forward Weston BLOCKER + 5 streaming integration Weston msg=13dd5e91 BLOCKER (score normalization severity drift): 保持 P0-V3+V4 P0 across §1.1 / §2.2 / §5.3 — score 方向是 caller 语义硬契约，不能在 PGVector/Qdrant 间显示反向。§2.2 加 P0-V3+V4 显式行 + §5.3 加 test_score_normalization_in_vector.py boundary test (跨 metric × 跨 adapter 全 6 cell parametrize). Streaming integrations (5 lane): 1. Bryce msg=23a2f514 P0-V1 first-principles 重新定性 — Qdrant legacy mode tenant isolation 是 collection name level 不是 query filter level (verify qdrant_connector.py:442-446)，下沉 P1-V4 defense-in-depth (legacy mode deprecation follow-up 候选). 2. Bryce msg=8e895471 11 vector findings — 4 P0 (cross-tenant 下沉 / filter silent / score V3+V4) + 3 P1 (collection init / batch atomicity / filter Or 语义) + 4 P2. 3. dongdong msg=4201465a + PR #1929 + cuiwenbo msg=bcec38ad — P0-D1 Helm worker Neo4j env missing (Singapore graph viz root-cause); P1-D1 e2e shape matrix gap; P1-D2 Nebula no Helm first-class; P1-D3 typed schema 缺 vector backend exposure. 4. chenyexuan NIT — Lesson #16 candidate cite added §6. 5. Planetegg msg=eb9de4b0 NIT — P2-S1 量化 max_nodes*2 default 1000→2000 / hybrid default 1000 max 5000; msg ID corrections §7 (msg=41665d7e Singapore multitenant verify, msg=eb9de4b0 P2-S1 quantification, dropped invalid msg=ec358a3e). 冬柏 PR #1927 commit b2234ae fold-in §5.3 (38 cases incl zero-side-effect + replay idempotency post-NIT). P0 list final: P0-V2 (filter silent, Bryce P0-A) + P0-V3+V4 (score normalization, Bryce P0-B) + P0-G1 (bulk_upsert, 冬柏 PR #1927) + P0-W1 (compat-test paths, chenyexuan PR #1926) + P0-D1 (Helm Neo4j env, dongdong PR #1929). * docs(task-61): § 3.1.1 historical residue cleanup per Weston msg=fdf04a69 NIT — strike old P0 hot-fix path (P0-V1 已下沉 P1-V4 per Bryce first-principles verify) * docs(task-61): final consistency cleanup per Weston msg=e414d3cf — line 14 count 4+3+4 to 3 P0 + 4 P1 + 4 P2; § 5.1 P0-V1 line removed; § 5.2 P1-V4 defense-in-depth boundary test added

earayu · 2026-04-30T05:46:05Z

Testing primary CR note: one small coverage gap before I can approve.

The added bulk tests correctly hit the P0-G1 surface across Postgres / Neo4j / Nebula and CI is now 15/15 green. I agree with the case set for empty no-op, mixed-name raise + zero side effect, distinct-key round trip, same-key dedup, replace-existing, append-distinct, entity_type last-wins, and replay idempotency.

The remaining gap is that the tests do not assert description_parts text/key contents, even though the Protocol says each tuple lands as a description part and the new docstrings explicitly claim description_parts visibility / description last-wins. A backend could pass these tests while writing correct source_lineage but dropping stale or wrong description text on the bulk path.

Please add direct got.description_parts assertions, at least in:

round-trip: {(doc, parse): text} equals from-doc-A-v1, from-doc-A-v2, from-doc-B-v1
same-key dedup / replace-existing: the surviving (doc-A, v1) text is the bulk/last write, not the old one
replay idempotent: the surviving (doc-A, v1) text is v1-replay

After that I am ready to LGTM; no concern with CI/backend triggering.

Per @huangzhangshu testing primary CR (msg=5bbc5d1a) — the bulk_upsert cases pinned lineage member identity but did not assert ``description_parts`` text content. A backend could write the lineage member key correctly but silently drop or stale-keep the description text, breaking the agent context retrieval contract. Add `description_parts` key→text assertions to 3 cases: * `_round_trip` — all 3 (doc_id, parse_version) parts must carry their source bulk's description text (not silently dropped). * `_dedup_last_wins_within_bulk` — same-key collapse must keep the LAST description text within the bulk (not first). * `_replaces_existing_same_key` — bulk's strip-then-append must overwrite the prior single-write description (not silently keep it). * `_replay_is_idempotent` — replay must overwrite first call's description with the second's (last-wins on replay), not just dedup the member. Coverage delta: same 38 cases, but every dedup/replace/replay case now pins both lineage AND description_parts text contract. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

earayu · 2026-04-30T05:50:10Z

Testing primary CR LGTM.

I rechecked fix-forward 1953933a: the description_parts key→text assertions now cover the remaining gap in round-trip, same-key dedup last-wins, replace-existing, and replay idempotency. That closes the silent-regression case where a backend writes correct lineage keys but drops/stales description text on the bulk path.

Validated locally:

uv run --extra test pytest tests/integration/compat/test_lineage_graph_compat.py --collect-only -q → 38 collected

CI after the prior head already proved Backend-Compat-Test runs Postgres / Neo4j / Nebula; current fix-forward is assertion-only in the same compat file. Good to squash merge once checks finish.

…#1932) § 四加 8 lesson sediment（task #30 B3 + task #61 全 P0 闭环累计实证）+ § 六 sediment 引用追加 6 PR commit cross-link + § 八修订记录追加本 PR fold trail。新增 lesson: - Lesson #12 v7.4: external API raw contract verify (task #61 P0-B PR #1930 Qdrant euclid raw direction first-application + fix-forward 1e30a00) - Lesson #12 v8 second-application: test docstring fake guardrail (task #61 P0-G1 PR #1927 description_parts assertion 缺位 fix-forward 1953933) - Lesson #12 v9: first-principles verify catch surface signal mistakes (task #61 P0-V1 重新定性 Bryce + task #61 P0-B Qdrant euclid Weston catch 双独立 source 同源 first/second-application) - Lesson #13 v2.3: deploy manifest dual-side rewrite (task #61 P0-D1 PR #1929 Helm Neo4j worker env first-application) - Lesson #13 v3 application demo 2: cross-source default value alignment (task #30 B3 PR #1925 commit dae43f5 三 source 同步 first-application) - Lesson #14 application demo: spec 内部 default 漂浮 multi-iteration cleanup (task #30 B3 PR #1925 fix-forward dae43f5 § 3.1.1 line 85 cleanup second-application demo, first-application 在 task #35 6 轮 fix-forward) - Lesson #16: CI workflow paths filter dead reference 反 pattern (task #61 P0-W1 PR #1926 first-application demo + Lesson #15 file-move 3-step verify 升级到 v2 4-step grep .github/workflows/*.yml paths 同步) - Lesson #17: backend 收敛 contract 优于上层 fork (simple-stable + private-deploy paramount directive earayu2 msg=1224bec8 在 cross-adapter contract 设计时应用; task #69 P0-B + task #70 P1 候选 1 cross-PR 一次性收敛 first-application) 跨 PR 多独立 source 同源 catch trail: - Lesson #12 v9: Bryce msg=23a2f514 + Weston msg=86e05a8e 双独立 source - Lesson #16: chenyexuan msg=f298011e + 冬柏 msg=3e93bb64 双独立 source - Lesson #17: cuiwenbo msg=cedc7703 + Bryce msg=9895a148 双独立 source - Lesson #13 v3 application demo 2: huangheng msg=bf785b12 + Planetegg msg=c63acbf5 + Weston msg=1e6b0838 三独立 source per architect msg=c4cdf634 + msg=daaeeab5 + msg=03c892e0 sediment dispatch. Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>

earayu and others added 2 commits April 30, 2026 13:00

earayu force-pushed the test/task-67-bulk-upsert-compat-tests branch from b2234ae to 381d7a7 Compare April 30, 2026 05:01

earayu merged commit 9c94cbc into main Apr 30, 2026
15 checks passed

earayu deleted the test/task-67-bulk-upsert-compat-tests branch April 30, 2026 06:05

earayu mentioned this pull request Apr 30, 2026

docs(cr-checklist): task #61 + task #30 B3 close-out sediment fold-in #1932

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test(compat): task #61 P1 — bulk_upsert_entity_with_lineage_parts cross-backend#1927

test(compat): task #61 P1 — bulk_upsert_entity_with_lineage_parts cross-backend#1927
earayu merged 3 commits into
mainfrom
test/task-67-bulk-upsert-compat-tests

earayu commented Apr 30, 2026

Uh oh!

earayu commented Apr 30, 2026

Uh oh!

earayu commented Apr 30, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

earayu commented Apr 30, 2026

Summary

改动

覆盖度

跟 chenyexuan PR #1926 关系

CR 请求

Test plan

后续

Uh oh!

earayu commented Apr 30, 2026

Uh oh!

earayu commented Apr 30, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant