|
| 1 | +# Metadata Index and Shared Go Search/Install Design |
| 2 | + |
| 3 | +Date: 2026-06-17 |
| 4 | + |
| 5 | +## Goal |
| 6 | + |
| 7 | +CAM will add a shared Go metadata layer for agents, skills, and plugins. The layer will ingest repository metadata configured in `~/.config/code-agent-manager/*_repos.json` and `~/.config/code-agent-manager/config.yaml`, store normalized records in SQLite, and expose local-first search, refresh, and install operations to both the Go CLI and desktop sidecar. |
| 8 | + |
| 9 | +Python CLI code is not extended for this feature. Python removal is a later follow-up after the Go implementation is complete. |
| 10 | + |
| 11 | +## Architecture |
| 12 | + |
| 13 | +The feature adds one shared Go metadata subsystem used by both CLI and desktop APIs: |
| 14 | + |
| 15 | +```text |
| 16 | +~/.config/code-agent-manager/config.yaml |
| 17 | +~/.config/code-agent-manager/*_repos.json |
| 18 | + │ |
| 19 | + ▼ |
| 20 | + Go metadata refresh service |
| 21 | + │ |
| 22 | + ▼ |
| 23 | + SQLite metadata index |
| 24 | + │ |
| 25 | + ┌──────┴──────┐ |
| 26 | + ▼ ▼ |
| 27 | + Go CLI Desktop sidecar API |
| 28 | +``` |
| 29 | + |
| 30 | +The metadata subsystem is responsible for loading source configuration, refreshing item metadata, storing normalized records, searching SQLite first, and installing selected items into target coding agents through shared Go code. |
| 31 | + |
| 32 | +## Components |
| 33 | + |
| 34 | +### Shared metadata package |
| 35 | + |
| 36 | +A new Go package owns metadata models, refresh logic, search logic, SQLite persistence, and install orchestration. It defines the supported item kinds: `agent`, `skill`, and `plugin`. |
| 37 | + |
| 38 | +The package exposes methods equivalent to: |
| 39 | + |
| 40 | +- `Refresh(ctx, options)` |
| 41 | +- `Search(ctx, query, filters)` |
| 42 | +- `Get(ctx, id)` |
| 43 | +- `Install(ctx, id, targetApp, options)` |
| 44 | + |
| 45 | +### Config/source loader |
| 46 | + |
| 47 | +The loader reads user configuration first and falls back to bundled repository defaults where existing CAM behavior expects it. Inputs include: |
| 48 | + |
| 49 | +- `~/.config/code-agent-manager/config.yaml` |
| 50 | +- `~/.config/code-agent-manager/agent_repos.json` |
| 51 | +- `~/.config/code-agent-manager/skill_repos.json` |
| 52 | +- `~/.config/code-agent-manager/plugin_repos.json` |
| 53 | +- bundled fallback repo metadata files |
| 54 | + |
| 55 | +User configuration has priority over bundled defaults. |
| 56 | + |
| 57 | +### SQLite store |
| 58 | + |
| 59 | +The store persists configured sources and normalized items. It runs migrations automatically before metadata operations and uses upserts so refresh can be repeated safely. |
| 60 | + |
| 61 | +### Discovery/refresh adapters |
| 62 | + |
| 63 | +Separate adapters handle each source type because conventions differ: |
| 64 | + |
| 65 | +- agents: markdown files under configured `agentsPath` |
| 66 | +- skills: skill directories or markdown files with frontmatter |
| 67 | +- plugins: plugin or marketplace metadata files |
| 68 | + |
| 69 | +### CLI integration |
| 70 | + |
| 71 | +The Go CLI gains a metadata command group for the shared workflow: |
| 72 | + |
| 73 | +```bash |
| 74 | +cam metadata refresh |
| 75 | +cam metadata search <query> |
| 76 | +cam metadata search <query> --type agent |
| 77 | +cam metadata search <query> --type skill |
| 78 | +cam metadata search <query> --type plugin |
| 79 | +cam metadata install <item-id> --target claude |
| 80 | +``` |
| 81 | + |
| 82 | +Domain-specific aliases such as `cam skill search` can be added later, but the first implementation should avoid duplicate behavior. |
| 83 | + |
| 84 | +### Desktop sidecar integration |
| 85 | + |
| 86 | +The sidecar exposes metadata endpoints for refresh, search, and install. The frontend can use these APIs to show local SQLite results first, then offer an explicit refresh action. |
| 87 | + |
| 88 | +## Data Flow |
| 89 | + |
| 90 | +### Refresh |
| 91 | + |
| 92 | +```text |
| 93 | +User clicks refresh / runs CLI refresh |
| 94 | + │ |
| 95 | + ▼ |
| 96 | +Load config.yaml + *_repos.json |
| 97 | + │ |
| 98 | + ▼ |
| 99 | +Resolve enabled agent/skill/plugin repos |
| 100 | + │ |
| 101 | + ▼ |
| 102 | +Fetch or read source metadata |
| 103 | + │ |
| 104 | + ▼ |
| 105 | +Parse agents / skills / plugins |
| 106 | + │ |
| 107 | + ▼ |
| 108 | +Upsert normalized records into SQLite |
| 109 | + │ |
| 110 | + ▼ |
| 111 | +Return refresh summary |
| 112 | +``` |
| 113 | + |
| 114 | +Refresh is explicit. Search stays local and fast by default. |
| 115 | + |
| 116 | +### Search |
| 117 | + |
| 118 | +```text |
| 119 | +User searches query |
| 120 | + │ |
| 121 | + ▼ |
| 122 | +Search SQLite first |
| 123 | + │ |
| 124 | + ▼ |
| 125 | +Return local results immediately |
| 126 | + │ |
| 127 | + ▼ |
| 128 | +Optional refresh action |
| 129 | + │ |
| 130 | + ▼ |
| 131 | +Pull configured online/GitHub sources |
| 132 | + │ |
| 133 | + ▼ |
| 134 | +Update SQLite |
| 135 | + │ |
| 136 | + ▼ |
| 137 | +Search again |
| 138 | +``` |
| 139 | + |
| 140 | +This implements the selected behavior: SQLite first, then refresh. |
| 141 | + |
| 142 | +### Install |
| 143 | + |
| 144 | +```text |
| 145 | +User selects item |
| 146 | + │ |
| 147 | + ▼ |
| 148 | +Load item from SQLite by ID |
| 149 | + │ |
| 150 | + ▼ |
| 151 | +Resolve target coding agent |
| 152 | + │ |
| 153 | + ▼ |
| 154 | +Fetch source repo if needed |
| 155 | + │ |
| 156 | + ▼ |
| 157 | +Copy/install item into target agent location/config |
| 158 | + │ |
| 159 | + ▼ |
| 160 | +Update installed status metadata |
| 161 | +``` |
| 162 | + |
| 163 | +CLI and desktop both use the same Go install path. |
| 164 | + |
| 165 | +## SQLite Schema |
| 166 | + |
| 167 | +### `metadata_sources` |
| 168 | + |
| 169 | +Stores configured repositories and source metadata. |
| 170 | + |
| 171 | +Fields: |
| 172 | + |
| 173 | +- `id` |
| 174 | +- `kind`: `agent`, `skill`, or `plugin` |
| 175 | +- `source_key`: stable key such as `owner/repo` |
| 176 | +- `owner` |
| 177 | +- `repo` |
| 178 | +- `branch` |
| 179 | +- `path` |
| 180 | +- `enabled` |
| 181 | +- `source_file` |
| 182 | +- `last_refreshed_at` |
| 183 | +- `created_at` |
| 184 | +- `updated_at` |
| 185 | + |
| 186 | +### `metadata_items` |
| 187 | + |
| 188 | +Stores searchable installable records. |
| 189 | + |
| 190 | +Fields: |
| 191 | + |
| 192 | +- `id` |
| 193 | +- `kind`: `agent`, `skill`, or `plugin` |
| 194 | +- `name` |
| 195 | +- `description` |
| 196 | +- `source_id` |
| 197 | +- `repo_owner` |
| 198 | +- `repo_name` |
| 199 | +- `repo_branch` |
| 200 | +- `item_path` |
| 201 | +- `install_key` |
| 202 | +- `target_apps` |
| 203 | +- `metadata_json` |
| 204 | +- `installed` |
| 205 | +- `installed_targets` |
| 206 | +- `last_seen_at` |
| 207 | +- `created_at` |
| 208 | +- `updated_at` |
| 209 | + |
| 210 | +### Search strategy |
| 211 | + |
| 212 | +The first implementation uses indexed `LIKE` searches across `name`, `description`, `repo_owner`, `repo_name`, and `install_key`. SQLite FTS can be added later if needed. |
| 213 | + |
| 214 | +## Online/GitHub Behavior |
| 215 | + |
| 216 | +Search is local-only by default. The online switch means refresh from configured remote/GitHub sources, update SQLite, then search SQLite again. It does not mean broad, arbitrary GitHub search in this feature. |
| 217 | + |
| 218 | +This keeps behavior deterministic and aligned with CAM's configured marketplace/repository model. |
| 219 | + |
| 220 | +## Error Handling |
| 221 | + |
| 222 | +Refresh is resilient. One broken repository does not fail the whole refresh. Refresh returns a summary with sources scanned, items indexed, items updated, stale records, failed sources, and warnings. |
| 223 | + |
| 224 | +Install reports clear errors for missing item IDs, unsupported targets, failed downloads, existing installed items, and partial failures. It does not mark an item installed if installation fails. |
| 225 | + |
| 226 | +SQLite migration or database errors stop the command/API request with the database path and actionable context. |
| 227 | + |
| 228 | +## Testing |
| 229 | + |
| 230 | +Unit tests cover: |
| 231 | + |
| 232 | +- config loading priority |
| 233 | +- parsing agent, skill, and plugin repo files |
| 234 | +- metadata normalization |
| 235 | +- SQLite migrations |
| 236 | +- SQLite upsert/search behavior |
| 237 | +- stale item handling |
| 238 | +- install target validation |
| 239 | + |
| 240 | +Integration-style tests use temporary directories for fake CAM config, fake source repositories, a temporary SQLite database, and fake target agent directories. They cover refresh, search, install, and refresh-after-source-change flows. |
| 241 | + |
| 242 | +CLI tests cover the new `cam metadata refresh`, `cam metadata search`, and `cam metadata install` command behavior where practical. |
| 243 | + |
| 244 | +Because the implementation changes Go code, repository test requirements apply after coding. |
| 245 | + |
| 246 | +## Scope |
| 247 | + |
| 248 | +Included: |
| 249 | + |
| 250 | +- Shared Go metadata service |
| 251 | +- SQLite metadata index |
| 252 | +- Config/repo loading from user and bundled sources |
| 253 | +- Local SQLite search |
| 254 | +- Explicit refresh from configured local/remote sources |
| 255 | +- Shared install path for selected agents, skills, and plugins |
| 256 | +- Go CLI metadata commands |
| 257 | +- Desktop sidecar metadata APIs |
| 258 | + |
| 259 | +Excluded: |
| 260 | + |
| 261 | +- Deleting Python CLI files |
| 262 | +- Full arbitrary GitHub global search |
| 263 | +- Replacing every existing domain-specific command |
| 264 | +- Major frontend redesign beyond wiring search/refresh/install if supported by the current UI |
| 265 | +- New external registry service |
0 commit comments