Skip to content

Commit af1ccdd

Browse files
committed
docs: update roadmap to move entity resolution from Meteor to Compass
Entity resolution now belongs in Compass since it has the full graph context needed for matching incoming observations against existing entities. Meteor delivers raw observations; Compass resolves, deduplicates, and constructs the graph. Updated architecture diagram, role descriptions, and priorities to reflect this boundary change.
1 parent d8362d9 commit af1ccdd

File tree

1 file changed

+17
-8
lines changed

1 file changed

+17
-8
lines changed

docs/docs/roadmap.md

Lines changed: 17 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -10,14 +10,16 @@ This document describes where Compass goes next.
1010

1111
## The Role
1212

13-
Meteor and Compass are two halves of one system. Meteor goes wide — connecting to every source, resolving entities, building the graph. Compass goes deep — storing the graph, making it queryable, serving it to every consumer.
13+
Meteor and Compass are two halves of one system. Meteor goes wide — connecting to every source, extracting rich metadata, and delivering it. Compass goes deep — resolving entities, storing the graph, making it queryable, and serving it to every consumer.
1414

15-
Meteor is the agent that roams your infrastructure. Compass is the brain that remembers what the agent found and answers questions about it.
15+
Meteor is the agent that roams your infrastructure. Compass is the brain that remembers what the agent found, figures out what's the same thing across sources, and answers questions about it.
1616

17-
As Meteor evolves from a flat metadata collector into a context graph builder, Compass evolves from a search catalog into the **persistence, query, and serving layer** for that graph. The division of responsibility:
17+
As Meteor evolves from a flat metadata collector into a richer extraction and delivery pipeline, Compass evolves from a search catalog into the **graph construction, persistence, query, and serving layer**. The division of responsibility:
1818

19-
- **Meteor** owns collection, entity resolution, and graph construction.
20-
- **Compass** owns storage, indexing, querying, and serving.
19+
- **Meteor** owns collection, enrichment, and delivery of raw observations from sources.
20+
- **Compass** owns entity resolution, graph construction, storage, indexing, querying, and serving.
21+
22+
Meteor sends raw observations — "I found this thing at this source with these properties." Compass resolves those observations against what it already knows, deduplicates, constructs the graph, and serves it. This keeps Meteor stateless and simple, while Compass — which already has the full graph context needed for resolution — owns the intelligence.
2123

2224
The consumer base expands from humans using a UI to AI agents calling APIs. Both remain first-class. But the AI serving path is where most of the new investment goes.
2325

@@ -58,7 +60,7 @@ Each edge type carries its own semantics and should be queryable independently.
5860

5961
The graph query API should support expressive traversal patterns, not just single-node neighborhood lookups.
6062

61-
**Entity as a first-class concept.** Meteor resolves entities across sources — recognizing that a BigQuery table, a dbt model, a Tableau datasource, and an Airflow task reference the same logical thing. Compass should store and expose these resolved identities. One entity, multiple representations, queryable as either the unified entity or any individual facet.
63+
**Entity resolution and identity.** The same logical thing appears across multiple sources — a BigQuery table, a dbt model, a Tableau datasource, and an Airflow task may all reference the same entity. Compass owns entity resolution. When Meteor delivers raw observations from different sources, Compass matches them against its existing graph, recognizes duplicates, merges facets, and maintains a unified entity identity. One entity, multiple source representations, queryable as either the unified entity or any individual facet. Compass is the right place for this because it has the full graph context needed to make resolution decisions — it can compare incoming observations against everything it already knows.
6264

6365
**Graph-aware ranking.** Assets that are highly connected — many dependents, many consumers, central in the lineage graph — are more important than orphaned tables nobody uses. Search ranking should factor in graph centrality and connectivity, not just text relevance.
6466

@@ -111,11 +113,17 @@ Technical metadata — schemas, lineage, service information — is necessary bu
111113
```
112114
Meteor
113115
114-
resolved assets + relationships
116+
raw observations + relationships
115117
v
116118
┌──────────────────────────────────────────────────┐
117119
│ Compass │
118120
│ │
121+
│ ┌──────────────────┐ │
122+
│ │ Entity Resolver │ │
123+
│ │ • deduplication │ │
124+
│ │ • identity merge │ │
125+
│ └────────┬─────────┘ │
126+
│ v │
119127
│ ┌─────────────┐ ┌──────────────┐ │
120128
│ │ Graph Store │ │ Vector Index │ │
121129
│ │ (Postgres) │ │ (Semantic) │ │
@@ -145,6 +153,7 @@ Technical metadata — schemas, lineage, service information — is necessary bu
145153

146154
The key additions over today's architecture:
147155

156+
- **Entity resolver** that matches incoming observations from Meteor against the existing graph, deduplicates, and merges identities.
148157
- **Vector index** alongside Elasticsearch for semantic search.
149158
- **Query engine** that can do graph traversal, text search, and semantic search in a single query.
150159
- **MCP server** as a first-class API surface alongside gRPC and REST.
@@ -160,7 +169,7 @@ The existing gRPC/REST APIs, Postgres storage, and Elasticsearch indexing remain
160169

161170
**Third: Semantic search.** Add vector embeddings to the indexing pipeline. When an asset is upserted, Compass indexes both the text and a vector embedding. Search becomes hybrid. This requires adding embedding generation and a vector index but doesn't change the data model.
162171

163-
**Fourth: Rich graph model.** Expand relationship types beyond lineage. Add multi-hop traversal queries. Implement graph-aware ranking. Store resolved entity identities from Meteor. This requires schema evolution and new query capabilities — it's the deepest change.
172+
**Fourth: Rich graph model.** Expand relationship types beyond lineage. Add entity resolution to match and merge observations from Meteor into unified entities. Add multi-hop traversal queries. Implement graph-aware ranking. This requires schema evolution and new query capabilities — it's the deepest change.
164173

165174
**Fifth: Active knowledge layer.** Change feeds, subscriptions, freshness scoring, usage tracking, business glossary. This turns Compass from a store you query into a system that actively keeps consumers informed and provides business context alongside technical metadata.
166175

0 commit comments

Comments
 (0)