feat: add document-graph — structured data ETL for graph-enhanced GenAI#355
Open
evanerwee wants to merge 2 commits into
Open
feat: add document-graph — structured data ETL for graph-enhanced GenAI#355evanerwee wants to merge 2 commits into
evanerwee wants to merge 2 commits into
Conversation
Document-graph extends graphrag-toolkit with typed node support for structured formats (CSV, Excel, JSON, Parquet, XML). It complements lexical-graph by handling deterministic ETL while lexical-graph handles LLM-based extraction. Key features: - Schema providers (CSV, JSON, S3, Static, Glue) with auto-discovery - 20+ transformers (normalizers, field, document, filter, graph, truncators) - Cypher generation with tenant-scoped labels - Query engine for typed node retrieval - Multi-tenancy (same Neptune cluster, isolated by labels) - Hybrid integration: index structured data into lexical-graph for semantic search Coexists with lexical-graph in the same Neptune database: - lexical-graph: unstructured text → Source/Chunk/Topic/Entity - document-graph: structured data → typed nodes (__User__, __Account__, etc.) 59 tests passing. 100% docstring coverage. Sphinx docs included.
- 11 documentation pages (getting-started, lexical-graph-integration, pipeline, ingestors, schema-providers, transformers, graph-build, query-engine, multi-tenancy, examples index, API reference) - 19 tested SageMaker notebooks covering all features - Includes hybrid integration demos (document-graph + lexical-graph)
Collaborator
|
This looks very interesting! Thank you for contributing. Could I ask you to post an Issue with what we're aiming to achieve with the document-graph and we can link the issue for discussion. |
Contributor
Author
|
I will certainly do the write-up for you. When Ian was still around, we discussed the need, but Ian left. I will redo the description. |
Collaborator
related issue: #356 Thank you so much! |
Collaborator
|
@evanerwee is this PR obsolete since there is a fresh one #365 ? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds document-graph as a sibling package to lexical-graph, providing structured data ETL (CSV, Excel, JSON, Parquet, XML) that complements the existing unstructured text processing.
What it does
Both coexist in the same Neptune database with tenant-scoped label isolation.
Key Features
Testing
Context
I previously collaborated with the team on this integration (before team changes). The intent was always to contribute document-graph as a companion to lexical-graph. Happy to discuss any architectural concerns or adjustments needed to align with the project's direction.
Checklist