Skip to content

Commit 4b8d9e7

Browse files
kurtstohrerkurtstohrer
committed
Add docs/data-source-discovery.md
1 parent 66bf38c commit 4b8d9e7

1 file changed

Lines changed: 215 additions & 0 deletions

File tree

‎docs/data-source-discovery.md‎

Lines changed: 215 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,215 @@
1+
# Data Source Discovery
2+
3+
Annotask scans a project's data-fetching libraries and local code to build a catalog of data sources used by:
4+
5+
- Audit > Data in the shell
6+
- `GET /__annotask/api/data-sources`
7+
- `GET /__annotask/api/data-source-examples/:name`
8+
- `GET /__annotask/api/data-source-details/:name`
9+
- `GET /__annotask/api/data-source-bindings/:name`
10+
- `annotask_get_data_sources`, `annotask_get_data_source_examples`, `annotask_get_data_source_details`
11+
- the `annotask data-sources`, `data-source-examples`, `data-source-details` CLI commands
12+
- per-task `data_context` enrichment (the primary source a task is bound to)
13+
14+
The scanner is workspace-aware, so in monorepos it aggregates data libraries and definitions across sibling packages and MFEs rather than only the current package.
15+
16+
## Two Layers
17+
18+
The catalog distinguishes **libraries** (data-fetching packages found in `package.json`) from **project entries** (concrete hooks, stores, fetch wrappers, GraphQL operations, and tRPC routers defined in `src/`).
19+
20+
A library entry only appears if it is both installed *and* at least one of its recognized identifiers is actually used somewhere in `src/`. This avoids listing packages that are declared but never imported.
21+
22+
## What Gets Extracted
23+
24+
### Library entries
25+
26+
```ts
27+
interface DataSourceLibrary {
28+
name: string
29+
version?: string
30+
detected_patterns: string[] // identifiers this library exports that we recognize
31+
}
32+
```
33+
34+
### Project entries
35+
36+
```ts
37+
interface ProjectDataEntry {
38+
kind: 'composable' | 'signal' | 'store' | 'fetch' | 'graphql' | 'loader' | 'rpc'
39+
name: string
40+
display_name?: string // e.g. "localhost:4320 /api/health" for inline fetches
41+
file: string // workspace-relative
42+
line?: number // 1-based definition line
43+
endpoint?: string // literal endpoint or query key when extractable
44+
resolved_endpoint?: string // rewritten through the nearest vite.config proxy
45+
used_count: number // ranking signal (non-definition references across src/)
46+
hint_symbols?: string[] // local vars holding the fetch result (for the binding analyzer)
47+
}
48+
```
49+
50+
### Catalog shape
51+
52+
```ts
53+
interface DataSourceCatalog {
54+
libraries: DataSourceLibrary[]
55+
project_entries: ProjectDataEntry[] // sorted by used_count desc
56+
scannedAt: number
57+
}
58+
```
59+
60+
### Runtime reference (on tasks)
61+
62+
The per-task `data_context.sources[]` uses a narrower `DataSource` shape — identifier, kind, module, endpoint, method, line, dynamic-endpoint flag, and (when the endpoint matches an API schema) a `response_schema_ref` + `schema_in_repo` pair. See `src/schema.ts` for the canonical definitions.
63+
64+
## Current Shell UX
65+
66+
The Audit > Data page lives in `src/shell/components/DataSourcesPage.vue` and is driven by `src/shell/composables/useDataSources.ts`.
67+
68+
Three sub-tabs:
69+
70+
- `hooks` — project data sources (composables, stores, signals, fetches, GraphQL, tRPC)
71+
- `libraries` — detected data-fetching libraries
72+
- `apis` — discovered API schemas (OpenAPI, GraphQL, tRPC) — served by a separate scanner (see `api-schema-scanner.ts`), but shown alongside for context
73+
74+
Filters:
75+
76+
- free-text search across name / file / endpoint
77+
- `All` vs `On page` (driven by live highlight rects from the iframe)
78+
- MFE filter when workspace MFEs are present
79+
80+
Detail pane shows signature, return type, body excerpt, leading imports, co-located siblings, and the binding graph (rendering sites where this source is consumed). Highlights in the app iframe come from the binding-graph `sites`, not from grep.
81+
82+
## Workspace Behavior
83+
84+
`scanDataSources()` uses `resolveWorkspace()` to read dependencies and walk `src/` across every workspace package. File paths are relativized against the workspace root so cross-MFE references stay unambiguous, and `useWorkspace()` in the shell maps those paths back to MFE ids for filtering.
85+
86+
Path-only endpoints like `/api/health` are resolved through the nearest `vite.config`'s `server.proxy` — so a Vue MFE proxying `/api` to a FastAPI service at :4320 doesn't get its highlights attributed to a Go service at :4330 that happens to expose the same path.
87+
88+
## Scanner Strategies
89+
90+
The scanner runs in order (see `scanDataSourcesUncached` in `src/server/data-source-scanner.ts`):
91+
92+
### 1. Library detection
93+
94+
Reads `dependencies` + `devDependencies` from every workspace `package.json` and cross-checks against `DATA_LIB_PATTERNS` — the hand-curated map of package name → identifiers we know how to recognize. A library only survives if at least one of its identifiers actually appears in source.
95+
96+
### 2. Project entry detection
97+
98+
Walks each package's `src/` (capped at 5000 files) and matches `ENTRY_PATTERNS` — specificity-ordered regexes for:
99+
100+
- named composables / hooks (`export function useX(...)`, `export const useX = (...)`)
101+
- Pinia stores (`defineStore`)
102+
- Zustand stores (`create(...)`)
103+
- Jotai atoms
104+
- Svelte stores (`writable`, `readable`, `derived`)
105+
- Solid primitives (`createSignal`, `createResource`, `createStore`)
106+
- GraphQL operations (tagged `` gql`...` ``)
107+
- tRPC routers (`createTRPCRouter`)
108+
- Fetch wrappers in API-ish directories
109+
- Inline fetches in component files (`fetch()`, `axios.*()`, `ofetch()`, `$fetch()`, `new URL()`, htmx `hx-*` attributes)
110+
111+
First match wins, so more specific patterns take precedence over generic ones.
112+
113+
### 3. Endpoint resolution
114+
115+
Literal endpoints extracted from the definition body are run through `parseViteProxy()` against the nearest `vite.config` so `/api/health` becomes `http://localhost:4320/api/health`.
116+
117+
### 4. Usage counting
118+
119+
One combined alternation regex per file counts non-definition references to each entry's name. Definition lines are excluded by line number. `used_count` drives the sort order and the `used_only` filter.
120+
121+
### 5. Hint symbols for inline fetches
122+
123+
Inline fetches like `const health = await fetch('/api/health').then(r => r.json())` get an endpoint-derived name (`apiHealth`) that never appears verbatim in source, which would leave the binding analyzer with nothing to match. `collectHintSymbols()` captures the local variables that hold the fetch result (`health`) so the analyzer can still trace them into templates / JSX.
124+
125+
## Recognized Libraries
126+
127+
| Library | Kind |
128+
|---------|------|
129+
| `@tanstack/{react,vue,solid,svelte}-query`, `swr` | `composable` |
130+
| `@apollo/client`, `urql`, `@urql/{vue,svelte}`, `graphql-request` | `composable` / `graphql` |
131+
| `axios`, `ofetch`, `htmx.org` | `fetch` |
132+
| `pinia`, `vuex`, `zustand`, `@reduxjs/toolkit`, `react-redux`, `jotai`, `valtio`, `mobx`, `svelte`, `svelte/store`, `$app/stores` | `store` |
133+
| `solid-js` (`createSignal`, `createMemo`, `createEffect`) | `signal` |
134+
| `solid-js` (`createResource`) | `composable` |
135+
| `solid-js` (`createStore`) | `store` |
136+
| `vue-router`, `react-router{,-dom}`, `@remix-run/react`, `next`, `@solidjs/router` | `loader` |
137+
| `@trpc/client`, `@trpc/react-query`, `@trpc/next` | `rpc` |
138+
139+
Full map in `DATA_LIB_PATTERNS` (`src/server/data-source-scanner.ts`).
140+
141+
## Binding Graph
142+
143+
The second layer — `resolveBindingGraph()` in `src/server/binding-analysis/` — traces where a data source is rendered. Two-pass:
144+
145+
1. **Seed pass** — walk every supported file and run the framework analyzer (Vue, Svelte, JSX/TSX, plus a regex fallback) with `sourceName` as the taint seed. Record render sites and component prop edges.
146+
2. **Prop propagation** — for each prop edge, re-analyze the child file with the prop name seeded as tainted. This picks up patterns like `<PlanetCard :planet="planet" />` → `{{ planet.moons }}` in the child.
147+
148+
Returns a `SourceBindingGraph` with `sites` (file + line + tainted symbols), `prop_edges`, a `partial` flag when any file fell back to file-level heuristics, and per-file diagnostics. Cached 60s, keyed by `projectRoot::sourceName::hintSymbols::scopeFile`.
149+
150+
The shell uses `sites` as DOM highlight targets via the `data-annotask-file` / `data-annotask-line` attributes the transform injects on every rendered element.
151+
152+
## Data-Context vs Data-Sources
153+
154+
The catalog is project-wide. The per-task `data_context` is a narrower slice — "which sources power *this* element?" — resolved at task-create time by `src/server/data-context.ts`:
155+
156+
| | Data-Source Catalog | Data Context |
157+
|---|---|---|
158+
| Scope | project-wide | per-file / per-task |
159+
| Driver | filesystem scan + pattern matching | task file + line + binding graph |
160+
| Returns | `DataSourceCatalog` | `DataContext` (sources + rendered_identifiers + route_bindings) |
161+
| Caching | 60s TTL, coalesced | probe cache keyed by realpath+mtime, FIFO evicted at 500 |
162+
| Used by | Data tab, agent exploration | task enrichment, agent anchor |
163+
164+
`resolveDataContext()` also cross-references `scanApiSchemas()` to populate `response_schema_ref` on sources whose endpoints match a known operation. `resolveElementDataContext()` uses the binding graph with a ±3-line tolerance for element-level precision.
165+
166+
`sources[0]` on a task is the one an agent should anchor on — nearest to `task.line`, with ties broken in the order `composable > signal > store > fetch > graphql > loader > rpc`.
167+
168+
## Caching
169+
170+
Catalog scans are cached in memory with a 60-second TTL. Concurrent scans are coalesced behind a single in-flight promise. `clearDataSourceCache()` also clears the vite-proxy lookup cache used by endpoint resolution.
171+
172+
The binding graph has its own 60s TTL keyed per source + hint + scope.
173+
174+
The data-context probe cache is keyed by realpath + mtime and evicts FIFO at 500 entries — it backs the fast UX path that needs a boolean + primary signal per file without a full resolve.
175+
176+
## HTTP, MCP, And CLI Access
177+
178+
### HTTP
179+
180+
```bash
181+
curl http://localhost:5173/__annotask/api/data-sources
182+
curl http://localhost:5173/__annotask/api/data-sources?kind=composable&used_only=1
183+
curl http://localhost:5173/__annotask/api/data-source-examples/useUserQuery?limit=5
184+
curl http://localhost:5173/__annotask/api/data-source-details/useUserQuery
185+
curl http://localhost:5173/__annotask/api/data-source-bindings/useUserQuery
186+
```
187+
188+
### MCP
189+
190+
- `annotask_get_data_sources` — supports `kind`, `library`, `search`, `used_only`
191+
- `annotask_get_data_source_examples` — supports `name`, `kind`, `limit`
192+
- `annotask_get_data_source_details` — supports `name`, `kind`, `file`, `context_lines`
193+
194+
When multiple definitions share a name, `annotask_get_data_source_details` returns `{ error: 'ambiguous', candidates: [...] }` — re-call with `file` and/or `kind` to disambiguate.
195+
196+
### CLI
197+
198+
```bash
199+
annotask data-sources
200+
annotask data-sources --kind=composable --used-only --mcp
201+
annotask data-source-examples useUserQuery --limit=5 --mcp
202+
annotask data-source-details useUserQuery --file=src/composables/useUserQuery.ts --mcp
203+
```
204+
205+
## When Agents Should Use It
206+
207+
Reach for the data-source catalog when a task asks to:
208+
209+
- modify a fetch contract, query, mutation, or store — this is the `api_update` task type's primary surface
210+
- add a new UI that needs to bind to existing data — start from `used_only=true` to see what's already fetched on this page
211+
- rewire a component to a different hook or store that matches project conventions
212+
- understand what data an element on the current page depends on — use the per-task `data_context`, then call `annotask_get_data_source_details` on the primary source for its shape
213+
- trace a source end-to-end through prop chains — the bindings endpoint returns the full render-site graph
214+
215+
For API-contract work, pair data sources with `annotask_get_api_operation` / `annotask_resolve_endpoint` — the `response_schema_ref` on a source points directly at the schema the fetch returns.

0 commit comments

Comments
 (0)