feat(browser): rewrite network for agent-native discovery (#1100)

* feat(browser): rewrite network command for agent-native discovery

Replace the index-based list + pretty-printed --detail flow with a
structured JSON interface built around stable keys, body-shape previews,
and a persistent capture cache. Agents can now reference captured
requests by operationName (GraphQL) or `METHOD host+pathname` (REST)
instead of array indexes that shift on every rerun.

- `browser network` now emits JSON: `{workspace, captured_at, count,
  filtered_out, entries: [{key, method, status, url, ct, size, shape}],
  detail_hint}` — no body payloads by default
- Shape inference (src/browser/shape.ts) walks response JSON into a
  flat path -> descriptor map with depth cap 6 and a 2KB budget per
  entry, so agents see structure without paying body tokens
- Stable key generator (src/browser/network-key.ts) derives
  `operationName` from graphql URLs and `METHOD host+pathname`
  elsewhere, disambiguating collisions with `#N` suffixes
- Persistent cache (src/browser/network-cache.ts) snapshots every
  capture to `~/.opencli/cache/browser-network/<workspace>.json` with
  a 24h TTL, so `--detail <key>` survives later commands
- `--detail <key>` returns `{key, url, method, status, ct, size, shape,
  body}` with structured error codes (cache_missing / cache_expired /
  cache_corrupt / key_not_found, the latter including available_keys)
- Add `--raw` for agents that want every full body inline, `--ttl` for
  cache lookups
- Update opencli-adapter-author + opencli-autofix skill docs to
  reference `--detail <key>` and the shape-first discovery flow

Supersedes the cache prototype in #1051.

Co-authored-by: freemandealer <freeman.zhang1992@gmail.com>

* fix(browser): structured errors for capture/save, shape budget guard

Self-review findings on the network refactor:

- captureNetworkItems throwing (browser crashed / CDP dropped) now emits
  `error.code: capture_failed` on stdout rather than leaking a bare
  stderr line from browserAction's generic handler — agents get a
  parseable JSON blob on every failure path, matching the design goal.
- saveNetworkCache throwing (disk full, read-only path) is a soft
  failure: the captured data is already in hand, so surface a
  `cache_warning` field in the envelope and keep going instead of
  aborting. `--detail` lookups on that run will miss the cache but the
  listing still reaches the agent.
- shape.ts: guard the sub-walk on `add()`'s return value so the
  "budget hits on the array/object descriptor itself" path can never
  emit a stray child without its parent marker.
- network-key.ts: document that `#N` suffixes start at `#2` — the first
  occurrence stays bare, there is no `#1`. Matches test + code.

Added regression tests: `capture_failed` on readNetworkCapture throw,
`cache_warning` on persistence failure, shape budget hit on array descriptor.

Co-authored-by: freemandealer <freeman.zhang1992@gmail.com>

---------

Co-authored-by: freemandealer <freeman.zhang1992@gmail.com>

Author: jackwener

skills/opencli-adapter-author/references/api-discovery.md

@@ -14,26 +14,34 @@
 opencli browser network
 ```
 
-输出里过掉这些噪音：
-- `.css / .js / .woff / .png / .svg / .webp / .mp4` — 静态资源
-- `googletagmanager / sentry / crazyegg / doubleclick / tracking / beacon` — 埋点
-- `/healthz / /ping / /heartbeat` — 健康检查
+默认输出是 JSON，每个候选都带：
+- `key` — 稳定引用（GraphQL 的 `operationName` 或 `METHOD host+pathname`）
+- `shape` — response body 的路径→类型映射（不含原 body，省 token）
+- `status / url / method / ct / size`
 
-剩下的每一条都是候选。挑 URL 里含业务词（`list / detail / quote / feed / timeline / stock / user` 等）的优先看：
+静态资源 / 埋点 / 追踪默认已过滤；需要全量看用 `--all`。
+
+### 按 shape 初筛
+
+挑 `key` 里含业务词（`list / detail / Timeline / User / Tweets / Quote`）的优先看 `shape`：
+
+- `$.data` 是 `object` 且下面出现 `array(N)` / `total` / `page` → 基本是它
+- 路径里出现 `nickname / avatar / title / price / tweets / items` → 就是它
+- shape 只有 `$: string` 或全是 HTML 噪音 → 下一条
+
+### 拉完整 body
+
+候选定了再拉完整 body（by key，不是 index — 数组顺序会随每次 capture 变）：
 
 ```bash
-opencli browser network --detail <N>
+opencli browser network --detail <key>
 ```
 
-看 response body 前 200 字节：
-
-- 含数组 / `total` / `page` 字段 → 基本是它
-- 含 `nickname / avatar / title / price` → 就是它
-- 是 HTML 或纯广告 → 下一条
+capture 会持久化到 `~/.opencli/cache/browser-network/<workspace>.json`（默认 TTL 24h），所以 `--detail` 即使跨多条其他命令也还在。
 
 ### 关键 request headers
 
-找到候选后，`--detail <N>` 里看请求头：
+`browser network` 当前只抓响应（body + status + ct），抓不到请求头。要看请求头就在 DevTools Network 面板里点这条 request，或用 `browser eval` 手动 `fetch(url)` 复现一次观察浏览器发出去的头：
 
 | 看到 | 含义 | 对应策略 |
 |------|------|---------|

skills/opencli-autofix/SKILL.md

@@ -130,8 +130,8 @@ opencli browser open https://example.com/target-page && opencli browser state
 # Interact to trigger API calls
 opencli browser click <N> && opencli browser network
 
-# Inspect specific API response
-opencli browser network --detail <index>
+# Inspect specific API response (key is the `key` field from the default JSON output)
+opencli browser network --detail <key>
 ```
 
 ## Step 4: Patch the Adapter

src/browser/network-cache.test.ts

@@ -0,0 +1,76 @@
+import { afterEach, beforeEach, describe, expect, it } from 'vitest';
+import * as fs from 'node:fs';
+import * as os from 'node:os';
+import * as path from 'node:path';
+import {
+    DEFAULT_TTL_MS,
+    findEntry,
+    getCachePath,
+    loadNetworkCache,
+    saveNetworkCache,
+    type CachedNetworkEntry,
+    type NetworkCacheFile,
+} from './network-cache.js';
+
+function makeEntry(key: string, body: unknown = { ok: true }): CachedNetworkEntry {
+    return { key, url: `https://x.com/${key}`, method: 'GET', status: 200, size: 2, ct: 'application/json', body };
+}
+
+describe('network-cache', () => {
+    let baseDir: string;
+
+    beforeEach(() => {
+        baseDir = fs.mkdtempSync(path.join(os.tmpdir(), 'opencli-netcache-'));
+    });
+    afterEach(() => {
+        fs.rmSync(baseDir, { recursive: true, force: true });
+    });
+
+    it('sanitizes workspace names into safe filenames', () => {
+        const p = getCachePath('browser:default', baseDir);
+        expect(path.basename(p)).toBe('browser_default.json');
+    });
+
+    it('round-trips entries through save + load', () => {
+        saveNetworkCache('ws', [makeEntry('UserTweets'), makeEntry('UserByScreenName')], baseDir);
+        const res = loadNetworkCache('ws', { baseDir });
+        expect(res.status).toBe('ok');
+        expect(res.file?.entries).toHaveLength(2);
+        expect(res.file?.entries[0].key).toBe('UserTweets');
+    });
+
+    it('reports missing when cache file does not exist', () => {
+        expect(loadNetworkCache('nope', { baseDir }).status).toBe('missing');
+    });
+
+    it('reports expired when the cache is older than ttl', () => {
+        saveNetworkCache('ws', [makeEntry('A')], baseDir);
+        const future = Date.now() + DEFAULT_TTL_MS + 60_000;
+        const res = loadNetworkCache('ws', { baseDir, now: future });
+        expect(res.status).toBe('expired');
+        expect(res.file?.entries).toHaveLength(1);
+    });
+
+    it('reports corrupt for malformed json', () => {
+        const file = getCachePath('ws', baseDir);
+        fs.mkdirSync(path.dirname(file), { recursive: true });
+        fs.writeFileSync(file, '{not json');
+        expect(loadNetworkCache('ws', { baseDir }).status).toBe('corrupt');
+    });
+
+    it('reports corrupt for wrong schema version', () => {
+        const file = getCachePath('ws', baseDir);
+        fs.mkdirSync(path.dirname(file), { recursive: true });
+        fs.writeFileSync(file, JSON.stringify({ version: 0, entries: [] }));
+        expect(loadNetworkCache('ws', { baseDir }).status).toBe('corrupt');
+    });
+
+    it('findEntry returns matching entry or null', () => {
+        const file: NetworkCacheFile = {
+            version: 1, workspace: 'ws', savedAt: new Date().toISOString(),
+            entries: [makeEntry('A'), makeEntry('B')],
+        };
+        expect(findEntry(file, 'B')?.key).toBe('B');
+        expect(findEntry(file, 'missing')).toBeNull();
+    });
+});

src/browser/network-cache.ts

@@ -0,0 +1,102 @@
+/**
+ * Persistent cache for browser network captures.
+ *
+ * The live capture buffer (JS interceptor / daemon ring) can be cleared
+ * by navigation or lost between CLI invocations. Agents still need
+ * stable references to request bodies after running other commands,
+ * so every `browser network` call snapshots its results to disk.
+ *
+ * Layout: <cacheDir>/browser-network/<workspace>.json
+ * Entries expire after DEFAULT_TTL_MS (24h).
+ */
+
+import * as fs from 'node:fs';
+import * as os from 'node:os';
+import * as path from 'node:path';
+
+export const DEFAULT_TTL_MS = 24 * 60 * 60 * 1000;
+
+export interface CachedNetworkEntry {
+    key: string;
+    url: string;
+    method: string;
+    status: number;
+    size: number;
+    ct: string;
+    body: unknown;
+}
+
+export interface NetworkCacheFile {
+    version: 1;
+    workspace: string;
+    savedAt: string;
+    entries: CachedNetworkEntry[];
+}
+
+function getDefaultCacheDir(): string {
+    return process.env.OPENCLI_CACHE_DIR || path.join(os.homedir(), '.opencli', 'cache');
+}
+
+export function getCachePath(workspace: string, baseDir: string = getDefaultCacheDir()): string {
+    const safe = workspace.replace(/[^a-zA-Z0-9_-]+/g, '_');
+    return path.join(baseDir, 'browser-network', `${safe}.json`);
+}
+
+export function saveNetworkCache(
+    workspace: string,
+    entries: CachedNetworkEntry[],
+    baseDir?: string,
+): void {
+    const target = getCachePath(workspace, baseDir);
+    fs.mkdirSync(path.dirname(target), { recursive: true });
+    const payload: NetworkCacheFile = {
+        version: 1,
+        workspace,
+        savedAt: new Date().toISOString(),
+        entries,
+    };
+    fs.writeFileSync(target, JSON.stringify(payload), 'utf-8');
+}
+
+export interface LoadOptions {
+    baseDir?: string;
+    ttlMs?: number;
+    now?: number;
+}
+
+export interface LoadResult {
+    status: 'ok' | 'missing' | 'expired' | 'corrupt';
+    file?: NetworkCacheFile;
+    ageMs?: number;
+}
+
+export function loadNetworkCache(workspace: string, opts: LoadOptions = {}): LoadResult {
+    const target = getCachePath(workspace, opts.baseDir);
+    let raw: string;
+    try { raw = fs.readFileSync(target, 'utf-8'); }
+    catch { return { status: 'missing' }; }
+
+    let parsed: NetworkCacheFile;
+    try {
+        const obj = JSON.parse(raw);
+        if (!obj || obj.version !== 1 || !Array.isArray(obj.entries)) {
+            return { status: 'corrupt' };
+        }
+        parsed = obj as NetworkCacheFile;
+    } catch {
+        return { status: 'corrupt' };
+    }
+
+    const ttl = opts.ttlMs ?? DEFAULT_TTL_MS;
+    const now = opts.now ?? Date.now();
+    const savedAt = Date.parse(parsed.savedAt);
+    if (!Number.isFinite(savedAt)) return { status: 'corrupt' };
+    const ageMs = now - savedAt;
+    if (ageMs > ttl) return { status: 'expired', file: parsed, ageMs };
+
+    return { status: 'ok', file: parsed, ageMs };
+}
+
+export function findEntry(file: NetworkCacheFile, key: string): CachedNetworkEntry | null {
+    return file.entries.find((e) => e.key === key) ?? null;
+}

src/browser/network-key.test.ts

@@ -0,0 +1,55 @@
+import { describe, expect, it } from 'vitest';
+import { assignKeys, deriveKey } from './network-key.js';
+
+describe('deriveKey', () => {
+    it('extracts operationName from Twitter-style graphql URLs', () => {
+        expect(deriveKey({
+            method: 'GET',
+            url: 'https://x.com/i/api/graphql/6fWQaBPK51aGyC_VC7t9GQ/UserTweets?variables=...',
+        })).toBe('UserTweets');
+    });
+
+    it('handles graphql URLs without a query id', () => {
+        expect(deriveKey({
+            method: 'POST',
+            url: 'https://example.com/graphql/MyOp?vars=1',
+        })).toBe('MyOp');
+    });
+
+    it('uses METHOD host+pathname for REST calls', () => {
+        expect(deriveKey({
+            method: 'get',
+            url: 'https://api.example.com/v1/users?page=1',
+        })).toBe('GET api.example.com/v1/users');
+    });
+
+    it('falls back to truncated raw url when URL parsing fails', () => {
+        const key = deriveKey({ method: 'GET', url: 'not-a-valid-url' });
+        expect(key.startsWith('GET ')).toBe(true);
+        expect(key).toContain('not-a-valid-url');
+    });
+});
+
+describe('assignKeys', () => {
+    it('disambiguates collisions with #N suffixes', () => {
+        const out = assignKeys([
+            { url: 'https://x.com/i/api/graphql/a/UserTweets', method: 'GET' },
+            { url: 'https://x.com/i/api/graphql/b/UserTweets', method: 'GET' },
+            { url: 'https://api.example.com/v1/u', method: 'GET' },
+            { url: 'https://api.example.com/v1/u', method: 'GET' },
+            { url: 'https://api.example.com/v1/u', method: 'GET' },
+        ]);
+        expect(out.map(o => o.key)).toEqual([
+            'UserTweets',
+            'UserTweets#2',
+            'GET api.example.com/v1/u',
+            'GET api.example.com/v1/u#2',
+            'GET api.example.com/v1/u#3',
+        ]);
+    });
+
+    it('preserves extra fields on each request', () => {
+        const out = assignKeys([{ url: 'https://a.com/x', method: 'GET', status: 200 }]);
+        expect(out[0]).toMatchObject({ status: 200, key: 'GET a.com/x' });
+    });
+});

src/browser/network-key.ts

@@ -0,0 +1,69 @@
+/**
+ * Stable keys for network capture entries.
+ *
+ * Agents reference entries by key (e.g. `UserTweets`, `GET api.x.com/1.1/home`)
+ * instead of array index, so the mapping survives new captures.
+ *
+ * Rules:
+ *   GraphQL (URL contains `/graphql/`): key = operationName derived from URL path
+ *                                       (the segment after a 22-char query id, or the last segment)
+ *   Everything else: key = `METHOD host+pathname`
+ *
+ * On collision assignKeys suffixes duplicates as `base#2`, `base#3`, ... —
+ * the first occurrence stays bare (there is no `#1`).
+ */
+
+export interface KeyableRequest {
+    url: string;
+    method: string;
+}
+
+export function deriveKey(req: KeyableRequest): string {
+    const parsed = safeParseUrl(req.url);
+    if (!parsed) return `${req.method.toUpperCase()} ${truncate(req.url, 120)}`;
+
+    const path = parsed.pathname;
+    if (path.includes('/graphql/')) {
+        const op = graphqlOperationName(path);
+        if (op) return op;
+    }
+
+    return `${req.method.toUpperCase()} ${parsed.host}${path}`;
+}
+
+export function assignKeys<T extends KeyableRequest>(requests: T[]): Array<T & { key: string }> {
+    const counts = new Map<string, number>();
+    const out: Array<T & { key: string }> = [];
+    for (const req of requests) {
+        const base = deriveKey(req);
+        const n = counts.get(base) ?? 0;
+        counts.set(base, n + 1);
+        const key = n === 0 ? base : `${base}#${n + 1}`;
+        out.push({ ...req, key });
+    }
+    return out;
+}
+
+function graphqlOperationName(pathname: string): string | null {
+    // Patterns we've seen in the wild:
+    //   /i/api/graphql/<queryId>/UserTweets
+    //   /graphql/<queryId>/SomeOp
+    //   /graphql/SomeOp                       (rare, no id)
+    const segments = pathname.split('/').filter(Boolean);
+    const idx = segments.indexOf('graphql');
+    if (idx < 0) return null;
+    const tail = segments.slice(idx + 1);
+    if (tail.length === 0) return null;
+    if (tail.length === 1) return tail[0];
+    // tail[0] is usually a query id; the operation name is the next segment.
+    return tail[1] || tail[0];
+}
+
+function safeParseUrl(url: string): URL | null {
+    try { return new URL(url); }
+    catch { return null; }
+}
+
+function truncate(s: string, max: number): string {
+    return s.length <= max ? s : `${s.slice(0, max - 1)}…`;
+}