Skip to content

Commit 472ed55

Browse files
earayuclaude
andauthored
feat(celery Wave 6 #33 chunk 2): cross-backend keyword + traversal + chunk 1 amend (#1741)
* feat(celery Wave 6 #33 chunk 2): cross-backend keyword + traversal impls + chunk 1 amend (drop vector-recall) Per architect Option A ruling msg=54eac595 — chunk 2 is narrowed to `query_entities_by_keyword` + `expand_neighbors_n_hops` across all three backends. `query_entities_by_vector` is dropped: the Wave 4 lineage schema has no entity-vector column, and adding either an entity vector index or a Qdrant cross-concern coupling violates the simple-stable-directive separation of concerns. Wave 7 may re-introduce vector recall via a separate `LightRAGQueryService` if real evidence surfaces. `aperag/indexing/graph.py`: * Drop `query_entities_by_vector` Protocol method declaration (was in chunk 1) + replace with explicit dropped-by-design comment pointing at the architect ruling. * `InMemoryLineageGraphStore` ships REAL implementations of the two remaining methods (substring match + BFS frontier traversal) so unit tests can exercise the canonical semantics without standing up a database. `aperag/indexing/graph_storage/postgres.py`: * `query_entities_by_keyword`: `WHERE name ILIKE '%query%'` ordered by name, limit `top_k`. ILIKE is the canonical case-insensitive substring idiom on Postgres. * `expand_neighbors_n_hops`: BFS-style frontier expansion. Per hop: fetch relations where `source IN $names OR target IN $names`, collect unseen endpoints as next frontier, fetch their entity rows. Bounded by `hops`. Single connection across all hops. `aperag/indexing/graph_storage/neo4j.py`: * `query_entities_by_keyword`: `MATCH (n:aperag_LineageEntity ...)` + `WHERE toLower(n.name) CONTAINS toLower($query)` + `ORDER BY n.name LIMIT $top_k`. Cypher 5.x has no built-in case-insensitive CONTAINS, so this is the canonical idiom. * `expand_neighbors_n_hops`: Same BFS pattern as Postgres but expressed as two Cypher MATCH queries per hop (entities by name IN list, relations by source/target IN list). Relations are stored as labelled nodes (not edges) per the Wave 4 schema, so traversal is JOIN-style not Cypher-pattern-style. `aperag/indexing/graph_storage/nebula.py`: * `query_entities_by_keyword`: enumerate all entity VIDs via the existing `_list_all_entity_vids` helper, read each lineage by VID, filter substring-match in Python, sort by name, return top_k. Nebula has no JSON / text-search index on tag string properties so this is necessarily a scan — acceptable per simple-stable directive (no per-collection text-index infra to manage). * `expand_neighbors_n_hops`: same scan-and-filter pattern as `find_relation_keys_with_lineage`. Per hop, scan all relation VIDs and filter by source/target ∈ current frontier. Read each matching relation + its endpoints. Wrapped in `asyncio.to_thread` since Nebula's Python driver is sync. `tests/unit_test/indexing/test_lineage_query_protocol.py`: replaces the chunk-1 Protocol-stub tests with real semantic tests against `InMemoryLineageGraphStore` (10 total): * Protocol surface declares 2 read methods (vector-recall removed). * Signature contract: kw-only args + `hops=1` default. * `query_entities_by_keyword`: substring matches (Alice / Bob / Carol fixture) + `top_k <= 0` and empty/whitespace query return `[]`. * `expand_neighbors_n_hops`: 1-hop / 2-hop / 0-hop / empty seeds / unknown seed cases. Production-readiness 三类 (chunk 2): - must-be-real: real keyword + real BFS traversal across 3 production backends; in-memory store ships matching semantics. - may-be-gated: vector recall NOT shipped (per architect Option A); Protocol surface intentionally narrowed. - fully-resolves: §K.11.11 chunk 2 narrowed acceptance + chunk 1 amend rolled into same commit per architect msg=54eac595. Tests + local gates: * 169/169 indexing unit tests pass (10 query-protocol + 159 existing). * ruff check + format clean. chunk 3 (caller migration: 5 files in retrieval/pipeline.py + knowledge_graph/service.py + graph_curation/* + legacy graphindex package hard-cut delete + grep-zero verify) lands separately. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * docs(celery Wave 6 §K.11): chunk 2 spec amend — drop query_entities_by_vector + add Pattern 3 Per architect Option A ruling msg=54eac595, fold spec amend into Bryce's chunk 2 PR #1741 (per "由 architect 在 Bryce chunk 2 PR push 后 同 chunk 2 fold 进 spec amend (single PR-A continuation)"). Three §K.11 sub-section edits: 1. §K.11.4 row #33 must-be-real layer: - Removed "vector-recall" from query layer scope - Added explicit "Graph entity vector recall NOT in Wave 6 scope — deferred to Wave 7+ if real evidence" with link to ruling - Cited reason: lineage schema 没承担 entity vector storage; inline coupling 违反 simple-stable directive #3 2. §K.11.5 pre-check pattern lock (renamed from "双 pattern" to "三 pattern"): - Added Pattern 3 (state binding — Protocol method reads from state X) - Pattern 3 trigger: "spec acceptance lock form 'Protocol method reads from state X' must grep-verify state X schema 现存在" - Documented #33 chunk 1 retrospective miss as lesson - Forward template: 4 sub-options when state X schema absent (in-chunk add / prior chunk add / separate service taking state dep / defer to next Wave) 3. §K.11.11 PR-A 3-chunk decomposition: - Added Status column (chunk 1 ✅ MERGED `20b9071b` / chunk 2 🔄 PR #1741 / chunk 3 ⏳ post-chunk-2) - Added "Chunk 1 → chunk 2 amend note" explicit subsection citing architect ruling msg=54eac595 + Pattern 3 sediment - Updated PR-A acceptance items reference: §K.10 (1-5) + amended #33 must-be-real per §K.11.4 Companion memory sediment in `feedback_spec_lock_grep_verify_caller.md` (Pattern 3 added) — keeps spec ↔ memory feedback aligned per `feedback_announce_equals_landed.md` narrative-truth invariant. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
1 parent 0c73b91 commit 472ed55

6 files changed

Lines changed: 624 additions & 100 deletions

File tree

aperag/indexing/graph.py

Lines changed: 92 additions & 52 deletions
Original file line numberDiff line numberDiff line change
@@ -508,25 +508,29 @@ async def get_relation(self, source: str, target: str, type: str) -> RelationWit
508508
"""Read-path helper for relations."""
509509

510510
# ------------------------------------------------------------------
511-
# Wave 6 #33 chunk 1 — LightRAG-style retrieval query layer (Protocol
512-
# stubs; cross-backend implementations land in chunk 2 per
513-
# §K.11.11). The retrieval pipeline (§G.5) and graph-curation flows
514-
# consume these to compose a graph-recall context for a user query
515-
# without going through the legacy ``GraphIndexService.query_context``.
511+
# Wave 6 #33 — LightRAG-style retrieval query layer. The retrieval
512+
# pipeline (§G.5) and graph-curation flows consume these to compose
513+
# a graph-recall context for a user query without going through the
514+
# legacy ``GraphIndexService.query_context``.
516515
#
517516
# Design notes:
518517
# * Returns are the canonical :class:`EntityWithLineage` /
519518
# :class:`RelationWithLineage` shapes already used by ``get_*`` —
520519
# keeps the read surface uniform; callers compose context text
521520
# themselves rather than relying on a backend-formatted string.
522-
# * ``query_by_vector`` takes a pre-computed embedding so backends
523-
# stay agnostic of the embedding model; the retrieval pipeline
524-
# already owns the embedder.
525521
# * ``expand_neighbors_n_hops`` returns BOTH neighbour entities and
526522
# the relation edges that connect them (1-hop or multi-hop). Per
527523
# simple-stable directive the multi-hop frontier is bounded by
528524
# ``hops`` (caller picks; default 1) so backends don't need a
529525
# query planner.
526+
# * Vector-recall (``query_entities_by_vector``) was originally
527+
# declared in chunk 1 but dropped per architect ruling
528+
# msg=54eac595 (Option A): the Wave 4 lineage schema does not
529+
# carry an entity-vector column, so a Protocol method on this
530+
# store would either invite a Qdrant cross-concern coupling or
531+
# silently no-op. Wave 7+ may add it via a separate
532+
# ``LightRAGQueryService`` if real evidence surfaces; until then
533+
# no dead code lives here. See §K.11 amend.
530534
# ------------------------------------------------------------------
531535

532536
async def query_entities_by_keyword(
@@ -541,30 +545,8 @@ async def query_entities_by_keyword(
541545
Match semantics are backend-defined (case-insensitive substring
542546
on Postgres / Cypher CONTAINS on Neo4j / etc.); the contract
543547
is "best-effort lexical recall, not strict equality". An empty
544-
``query`` returns ``[]`` (no spurious recall).
545-
546-
Wave 6 #33 chunk 1 — Protocol stub; backend implementations
547-
land chunk 2.
548-
"""
549-
550-
async def query_entities_by_vector(
551-
self,
552-
*,
553-
embedding: list[float],
554-
top_k: int,
555-
) -> list[EntityWithLineage]:
556-
"""Return up to ``top_k`` entities ranked by similarity of
557-
``embedding`` against an entity-vector index the backend
558-
maintains. Backends without a native vector index may compose
559-
this against the collection's existing Qdrant collection
560-
(entity-name-keyed points) — chunk 2 picks the per-backend
561-
approach.
562-
563-
Empty ``embedding`` is invalid and raises ``ValueError`` (the
564-
retrieval pipeline always passes a real vector — the embedder
565-
has already validated the query upstream).
566-
567-
Wave 6 #33 chunk 1 — Protocol stub.
548+
or whitespace-only ``query`` returns ``[]`` (no spurious recall).
549+
``top_k <= 0`` returns ``[]``.
568550
"""
569551

570552
async def expand_neighbors_n_hops(
@@ -585,9 +567,9 @@ async def expand_neighbors_n_hops(
585567
Implementations MUST bound the result by ``hops`` to keep the
586568
traversal cost predictable; per simple-stable directive
587569
operators don't manage tunable per-collection traversal
588-
budgets — this is the only knob.
589-
590-
Wave 6 #33 chunk 1 — Protocol stub.
570+
budgets — this is the only knob. ``hops <= 0`` returns just
571+
the requested seed entities (no relations). Empty
572+
``entity_names`` returns ``([], [])``.
591573
"""
592574

593575

@@ -789,13 +771,7 @@ async def get_relation(self, source: str, target: str, type: str) -> RelationWit
789771
)
790772

791773
# ------------------------------------------------------------------
792-
# Wave 6 #33 chunk 1 — LightRAG-style query layer stubs.
793-
# Real implementations land chunk 2 (per §K.11.11 3-chunk
794-
# decomposition). The in-memory store also defers to chunk 2 so
795-
# tests that assert the Protocol surface can pin the
796-
# NotImplementedError contract today and switch to the real
797-
# behaviour when chunk 2 ships — without rewriting the test
798-
# scaffolding.
774+
# Wave 6 #33 chunk 2 — LightRAG-style query layer real impls.
799775
# ------------------------------------------------------------------
800776

801777
async def query_entities_by_keyword(
@@ -804,23 +780,87 @@ async def query_entities_by_keyword(
804780
query: str,
805781
top_k: int,
806782
) -> list[EntityWithLineage]:
807-
raise NotImplementedError("Wave 6 #33 chunk 2 — query_entities_by_keyword pending implementation")
808-
809-
async def query_entities_by_vector(
810-
self,
811-
*,
812-
embedding: list[float],
813-
top_k: int,
814-
) -> list[EntityWithLineage]:
815-
raise NotImplementedError("Wave 6 #33 chunk 2 — query_entities_by_vector pending implementation")
783+
if not query or not query.strip() or top_k <= 0:
784+
return []
785+
needle = query.strip().lower()
786+
matches: list[EntityWithLineage] = []
787+
async with self._guard:
788+
for name, row in self._entities.items():
789+
if needle not in name.lower():
790+
continue
791+
matches.append(
792+
EntityWithLineage(
793+
name=row.name,
794+
type=row.type,
795+
source_lineage=tuple(_sorted_lineage(row.source_lineage.values())),
796+
description_parts=tuple(_sorted_description_parts(row.description_parts.values())),
797+
)
798+
)
799+
# Deterministic ordering for stable test output.
800+
matches.sort(key=lambda e: e.name)
801+
return matches[:top_k]
816802

817803
async def expand_neighbors_n_hops(
818804
self,
819805
*,
820806
entity_names: list[str],
821807
hops: int = 1,
822808
) -> tuple[list[EntityWithLineage], list[RelationWithLineage]]:
823-
raise NotImplementedError("Wave 6 #33 chunk 2 — expand_neighbors_n_hops pending implementation")
809+
if not entity_names:
810+
return ([], [])
811+
812+
seen_entities: dict[str, EntityWithLineage] = {}
813+
seen_relations: dict[tuple[str, str, str], RelationWithLineage] = {}
814+
frontier: set[str] = {n for n in entity_names if n}
815+
816+
async with self._guard:
817+
# Always include the seed entities (whether or not they
818+
# exist in the store).
819+
for name in frontier:
820+
row = self._entities.get(name)
821+
if row is None:
822+
continue
823+
seen_entities[name] = EntityWithLineage(
824+
name=row.name,
825+
type=row.type,
826+
source_lineage=tuple(_sorted_lineage(row.source_lineage.values())),
827+
description_parts=tuple(_sorted_description_parts(row.description_parts.values())),
828+
)
829+
830+
current = set(frontier)
831+
for _ in range(max(hops, 0)):
832+
next_frontier: set[str] = set()
833+
for rel_key, rel_row in self._relations.items():
834+
src, tgt, rtype = rel_key
835+
if src in current or tgt in current:
836+
if rel_key not in seen_relations:
837+
seen_relations[rel_key] = RelationWithLineage(
838+
source=rel_row.source,
839+
target=rel_row.target,
840+
type=rel_row.type,
841+
evidence_lineage=tuple(_sorted_lineage(rel_row.evidence_lineage.values())),
842+
description_parts=tuple(_sorted_description_parts(rel_row.description_parts.values())),
843+
)
844+
for neighbour in (src, tgt):
845+
if neighbour in seen_entities:
846+
continue
847+
row = self._entities.get(neighbour)
848+
if row is None:
849+
continue
850+
seen_entities[neighbour] = EntityWithLineage(
851+
name=row.name,
852+
type=row.type,
853+
source_lineage=tuple(_sorted_lineage(row.source_lineage.values())),
854+
description_parts=tuple(_sorted_description_parts(row.description_parts.values())),
855+
)
856+
next_frontier.add(neighbour)
857+
if not next_frontier:
858+
break
859+
current = next_frontier
860+
861+
entities = sorted(seen_entities.values(), key=lambda e: e.name)
862+
relations = sorted(seen_relations.values(), key=lambda r: (r.source, r.target, r.type))
863+
return (entities, relations)
824864

825865

826866
def _sorted_lineage(members: Iterable[LineageMember]) -> list[LineageMember]:

aperag/indexing/graph_storage/nebula.py

Lines changed: 113 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -788,6 +788,119 @@ def _read() -> RelationWithLineage | None:
788788

789789
return await asyncio.to_thread(_read)
790790

791+
# -- LightRAG-style query layer (Wave 6 #33 chunk 2) --------------
792+
793+
async def query_entities_by_keyword(
794+
self,
795+
*,
796+
query: str,
797+
top_k: int,
798+
) -> list[EntityWithLineage]:
799+
if not query or not query.strip() or top_k <= 0:
800+
return []
801+
await self.ensure_schema()
802+
803+
needle = query.strip().lower()
804+
805+
def _scan() -> list[EntityWithLineage]:
806+
# Nebula has no JSON / text-search index on tag string
807+
# properties; we enumerate all entity VIDs via LOOKUP and
808+
# filter substring-match in Python. For collections with
809+
# very large entity counts this is slower than the
810+
# SQL/Cypher equivalents — acceptable per simple-stable
811+
# directive (no per-collection text-index infra to manage).
812+
matches: list[EntityWithLineage] = []
813+
for vid in self._list_all_entity_vids():
814+
row = self._read_entity_lineage_by_vid(vid)
815+
if row is None:
816+
continue
817+
name, type_value, members, parts = row
818+
if needle not in name.lower():
819+
continue
820+
matches.append(
821+
EntityWithLineage(
822+
name=name,
823+
type=type_value,
824+
source_lineage=tuple(members),
825+
description_parts=tuple(parts),
826+
)
827+
)
828+
matches.sort(key=lambda e: e.name)
829+
return matches[:top_k]
830+
831+
return await asyncio.to_thread(_scan)
832+
833+
async def expand_neighbors_n_hops(
834+
self,
835+
*,
836+
entity_names: list[str],
837+
hops: int = 1,
838+
) -> tuple[list[EntityWithLineage], list[RelationWithLineage]]:
839+
if not entity_names:
840+
return ([], [])
841+
await self.ensure_schema()
842+
843+
def _walk() -> tuple[list[EntityWithLineage], list[RelationWithLineage]]:
844+
seen_entities: dict[str, EntityWithLineage] = {}
845+
seen_relations: dict[tuple[str, str, str], RelationWithLineage] = {}
846+
847+
def _add_entity(name: str) -> None:
848+
if name in seen_entities:
849+
return
850+
row = self._read_entity_lineage(name)
851+
if row is None:
852+
return
853+
type_value, members, parts = row
854+
seen_entities[name] = EntityWithLineage(
855+
name=name,
856+
type=type_value,
857+
source_lineage=tuple(members),
858+
description_parts=tuple(parts),
859+
)
860+
861+
current = {n for n in entity_names if n}
862+
for name in current:
863+
_add_entity(name)
864+
865+
# Nebula has no edge type for relations (relations are
866+
# tag-vertices). Walk by enumerating all relation VIDs and
867+
# filtering by source/target ∈ current — same approach as
868+
# ``find_relation_keys_with_lineage``.
869+
for _ in range(max(hops, 0)):
870+
next_frontier: set[str] = set()
871+
if not current:
872+
break
873+
for vid in self._list_all_relation_vids():
874+
row = self._read_relation_lineage_by_vid(vid)
875+
if row is None:
876+
continue
877+
src, tgt, rtype, members, parts = row
878+
if src not in current and tgt not in current:
879+
continue
880+
key = (src, tgt, rtype)
881+
if key not in seen_relations:
882+
seen_relations[key] = RelationWithLineage(
883+
source=src,
884+
target=tgt,
885+
type=rtype,
886+
evidence_lineage=tuple(members),
887+
description_parts=tuple(parts),
888+
)
889+
for endpoint in (src, tgt):
890+
if endpoint not in seen_entities and endpoint not in next_frontier:
891+
next_frontier.add(endpoint)
892+
if not next_frontier:
893+
break
894+
for name in next_frontier:
895+
_add_entity(name)
896+
current = next_frontier
897+
898+
entities = sorted(seen_entities.values(), key=lambda e: e.name)
899+
relations = sorted(seen_relations.values(), key=lambda r: (r.source, r.target, r.type))
900+
return (entities, relations)
901+
902+
return await asyncio.to_thread(_walk)
903+
791904

792905
__all__ = [
793906
"NebulaLineageGraphStore",

0 commit comments

Comments
 (0)