diff --git a/README.md b/README.md index 6bf5396..42138c6 100644 --- a/README.md +++ b/README.md @@ -45,6 +45,8 @@ When an agent has access to 200+ API endpoints, loading all of them as MCP tools | Non-HTTP integrations (desktop apps) | ❌ | ❌ | ✅ | | Session management / undo-redo | ❌ | ❌ | ✅ | | JSON structured output | ❌ | ✅ | ✅ | +| Custom HTTP headers | ✅ | ❌ | ❌ | +| Command name prefix | ✅ | ❌ | ❌ | | Basic / Bearer auth | ✅ | ✅ | ❌ | | OAuth2 / Auth0 | ❌ | ✅ | ✅ | | Response JMESPath filtering | ❌ | ✅ | ❌ | @@ -63,7 +65,9 @@ ocli profiles add myapi \ --openapi-spec http://127.0.0.1:2222/openapi.json \ --api-bearer-token "..." \ --include-endpoints get:/messages,get:/channels \ - --exclude-endpoints post:/admin/secret + --exclude-endpoints post:/admin/secret \ + --command-prefix "myapi_" \ + --custom-headers '{"X-Tenant":"acme","X-Request-Source":"cli"}' ``` Alternatively, `ocli onboard` (with the same options, no profile name) creates a profile named `default`. @@ -104,6 +108,64 @@ ocli commands -r "messages" -n 3 The BM25 engine (ported from [picoclaw](https://github.com/sipeed/picoclaw)) ranks commands by relevance across name, method, path, description, and parameter names. This enables agents to discover the right endpoint without loading all command schemas into context. The legacy `ocli search` command is kept as a deprecated alias and internally forwards to `ocli commands` with the same flags. +### Benchmark: three strategies for AI agent ↔ API interaction + +Tested against [Swagger Petstore](https://petstore3.swagger.io/) ([OpenAPI spec](https://petstore3.swagger.io/api/v3/openapi.json)). Scaling projections use the [GitHub API](https://api.apis.guru/v2/specs/github.com/api.github.com/1.1.4/openapi.json) (845 endpoints). + +Three strategies compared: + +| # | Strategy | How it works | Tools in context | +|---|----------|-------------|-----------------| +| 1 | **MCP Naive** | All endpoints as MCP tools | N tools (one per endpoint) | +| 2 | **MCP+Search** | 2 tools: `search_tools` + `call_api` | 2 tools | +| 3 | **CLI (ocli)** | 1 tool: `execute_command` | 1 tool | + +Run the benchmark yourself: + +```bash +npx ts-node benchmarks/benchmark.ts +``` + +Results (19 endpoints, 15 natural-language queries): + +``` + TOOL DEFINITION OVERHEAD (sent with every API request) + + MCP Naive ██████████████████████████████ 2 945 tok (19 tools) + MCP+Search ████ 346 tok (2 tools) + CLI (ocli) █ 138 tok (1 tool) + + SEARCH RESULT SIZE (3 matching endpoints) + + MCP+Search ██████████████████████████████ 995 tok (full JSON schemas) + CLI (ocli) ██ 67 tok (name + method + path) + + SCALING: OVERHEAD PER TURN vs ENDPOINT COUNT + + Endpoints MCP Naive MCP+Search CLI (ocli) Naive/CLI + 19 2 945 tok 346 tok 138 tok 21x ← Petstore + 100 15 415 tok 346 tok 138 tok 112x + 845 130 106 tok 346 tok 138 tok 943x ← GitHub API + + VERDICT + + Overhead/turn Search result Accuracy Server? + MCP Naive 2 945 tok N/A 100% Yes + MCP+Search 346 tok 995 tok/query 93% Yes + CLI (ocli) 138 tok 67 tok/query 93% No + + Monthly cost (845 endpoints, 100 tasks/day, Claude Sonnet $3/M input): + MCP Naive $1 172/month + MCP+Search $ 92/month + CLI (ocli) $ 4/month +``` + +Key insights: +- **MCP Naive** is simple but scales terribly (130K tokens at 845 endpoints). +- **MCP+Search** fixes the overhead but search results carry full JSON schemas — 15x larger than CLI text output. +- **CLI** returns compact text, needs no MCP server, and works with any agent that has shell access. +- Both search approaches share the same BM25 accuracy (93% top-3). The 7% miss is recoverable — the agent retries with a different query. + ### Installation and usage via npm and npx To use `ocli` locally without installing it globally you can rely on `npx`: @@ -228,7 +290,9 @@ The `ocli` binary provides the following core commands: - `--api-basic-auth ` - optional; - `--api-bearer-token ` - optional; - `--include-endpoints ` - comma-separated `method:path`; - - `--exclude-endpoints ` - comma-separated `method:path`. + - `--exclude-endpoints ` - comma-separated `method:path`; + - `--command-prefix ` - prefix for command names (e.g. `api_` -> `api_messages`, `api_users`); + - `--custom-headers ` - custom HTTP headers as JSON string (e.g. `'{"X-Tenant":"acme","X-Request-Source":"cli"}'`). Legacy comma-separated `key:value` format is also supported for simple values without commas. - `ocli profiles add ` - add a new profile with the given name and cache the OpenAPI spec. Same options as `onboard` (profile name is the positional argument). @@ -271,6 +335,19 @@ The project mirrors parts of the `openapi-to-mcp` architecture but implements a - `bm25` - generic BM25 ranking engine with Robertson IDF smoothing and min-heap top-K extraction. - `cli` - entry point, argument parser, command registration, help output. +### Using with AI agents (Claude Code skill example) + +An example skill file is provided in [`examples/skill-ocli-api.md`](examples/skill-ocli-api.md). Copy it to `.claude/skills/api.md` in your project to let Claude Code discover and use your API via `ocli`: + +```bash +cp examples/skill-ocli-api.md .claude/skills/api.md +``` + +The agent workflow: +1. `ocli commands --query "upload file"` — discover the right command +2. `ocli files_content_post --help` — check parameters +3. `ocli files_content_post --file ./data.csv` — execute + ### Similar projects - [openapi-cli-generator](https://github.com/danielgtaylor/openapi-cli-generator) - generates a CLI from an OpenAPI 3 specification using code generation. diff --git a/benchmarks/benchmark.ts b/benchmarks/benchmark.ts new file mode 100644 index 0000000..8405ee5 --- /dev/null +++ b/benchmarks/benchmark.ts @@ -0,0 +1,524 @@ +#!/usr/bin/env ts-node + +/** + * Token benchmark: 3 strategies for AI agent ↔ API interaction + * + * 1. MCP Naive — all endpoints as tools in context (standard MCP approach) + * 2. MCP + Search — 2 tools: search_tools + call_api (smart MCP with tool search) + * 3. CLI (ocli) — 1 tool: execute_command (search via `ocli commands --query`) + * + * Tested against Swagger Petstore (19 endpoints). + * Run: npx ts-node benchmarks/benchmark.ts + */ + +import axios from "axios"; +import { OpenapiToCommands } from "../src/openapi-to-commands"; +import { CommandSearch } from "../src/command-search"; +import { Profile } from "../src/profile-store"; + +// ── Helpers ────────────────────────────────────────────────────────── + +function estimateTokens(text: string): number { + return Math.ceil(text.length / 4); +} + +function padRight(str: string, len: number): string { + return str.length >= len ? str : str + " ".repeat(len - str.length); +} + +function padLeft(str: string, len: number): string { + return str.length >= len ? str : " ".repeat(len - str.length) + str; +} + +function bar(value: number, max: number, width: number, char = "█"): string { + const filled = Math.round((value / max) * width); + return char.repeat(Math.max(1, filled)); +} + +function matches(value: string, expected: string | RegExp): boolean { + if (typeof expected === "string") return value === expected; + return expected.test(value); +} + +// ── OpenAPI → MCP tool definitions ────────────────────────────────── + +interface McpTool { + name: string; + description: string; + input_schema: Record; +} + +function openapiToMcpTools(spec: Record): McpTool[] { + const tools: McpTool[] = []; + const paths = (spec.paths ?? {}) as Record>; + const schemas = ((spec.components as Record)?.schemas ?? {}) as Record; + const methods = ["get", "post", "put", "delete", "patch"]; + + for (const [pathKey, pathItem] of Object.entries(paths)) { + for (const method of methods) { + const op = pathItem[method] as Record | undefined; + if (!op) continue; + + const operationId = (op.operationId as string) ?? `${method}_${pathKey.replace(/[{}\/]/g, "_")}`; + const description = (op.description as string) ?? (op.summary as string) ?? ""; + const properties: Record = {}; + const required: string[] = []; + + const params = (op.parameters ?? []) as Array>; + for (const param of params) { + const name = param.name as string; + const schema = (param.schema ?? { type: "string" }) as Record; + properties[name] = { + type: schema.type ?? "string", + description: (param.description as string) ?? "", + ...(schema.enum ? { enum: schema.enum } : {}), + ...(schema.format ? { format: schema.format } : {}), + }; + if (param.required) required.push(name); + } + + const requestBody = op.requestBody as Record | undefined; + if (requestBody) { + const content = requestBody.content as Record | undefined; + const jsonContent = content?.["application/json"] as Record | undefined; + const bodySchema = jsonContent?.schema as Record | undefined; + if (bodySchema) { + const ref = bodySchema.$ref as string | undefined; + if (ref) { + const schemaName = ref.split("/").pop()!; + const resolved = schemas[schemaName] as Record | undefined; + if (resolved) { + const bodyProps = (resolved.properties ?? {}) as Record; + const bodyReq = (resolved.required ?? []) as string[]; + for (const [pn, ps] of Object.entries(bodyProps)) { + const prop = ps as Record; + if (prop.$ref) { + const innerName = (prop.$ref as string).split("/").pop()!; + properties[pn] = schemas[innerName] ?? prop; + } else if (prop.items && (prop.items as Record).$ref) { + const innerName = ((prop.items as Record).$ref as string).split("/").pop()!; + properties[pn] = { ...prop, items: schemas[innerName] ?? prop.items }; + } else { + properties[pn] = prop; + } + } + required.push(...bodyReq); + } + } + } + } + + tools.push({ + name: operationId, + description: description.slice(0, 1024), + input_schema: { + type: "object", + properties, + ...(required.length > 0 ? { required } : {}), + }, + }); + } + } + + return tools; +} + +// ── Tool definitions for each strategy ────────────────────────────── + +function buildMcpSearchTools(): McpTool[] { + return [ + { + name: "search_tools", + description: "Search available API tools by natural language query. Returns matching tools with their full parameter schemas so you can call them.", + input_schema: { + type: "object", + properties: { + query: { type: "string", description: "Natural language search query (e.g. 'create a pet', 'get order status')" }, + limit: { type: "number", description: "Maximum number of results to return (default: 5)" }, + }, + required: ["query"], + }, + }, + { + name: "call_api", + description: "Call an API endpoint by its tool name with the specified parameters. Use search_tools first to discover available tools and their schemas.", + input_schema: { + type: "object", + properties: { + tool_name: { type: "string", description: "The tool name returned by search_tools (e.g. 'addPet', 'getOrderById')" }, + parameters: { type: "object", description: "Parameters to pass to the tool, matching the schema from search_tools results" }, + }, + required: ["tool_name", "parameters"], + }, + }, + ]; +} + +function buildCliTool(): McpTool[] { + return [{ + name: "execute_command", + description: "Execute a shell command. Use `ocli commands --query \"...\"` to search for API commands, then `ocli --param value` to execute.", + input_schema: { + type: "object", + properties: { + command: { type: "string", description: "Shell command to execute" }, + }, + required: ["command"], + }, + }]; +} + +// ── Simulate search result sizes ──────────────────────────────────── + +/** + * MCP search_tools returns full JSON schemas for matched tools. + * This is what the agent receives in the tool result — it counts as input tokens + * on the next turn. + */ +function simulateMcpSearchResult(mcpTools: McpTool[], query: string, limit: number): string { + // Simulate: return top `limit` tools with full schemas + // In real MCP, the search result includes complete tool definitions + const matched = mcpTools.slice(0, limit); // simplified; real search would rank + return JSON.stringify(matched.map(t => ({ + name: t.name, + description: t.description, + parameters: t.input_schema, + })), null, 2); +} + +/** + * CLI search returns compact text: name + method + path + description. + * Much smaller than full JSON schemas. + */ +function simulateCliSearchResult(searcher: CommandSearch, query: string, limit: number): string { + const results = searcher.search(query, limit); + // This is what `ocli commands --query "..."` outputs + return results.map(r => + ` ${r.name.padEnd(35)} ${r.method.padEnd(7)} ${r.path} ${r.description ?? ""}` + ).join("\n"); +} + +// ── Accuracy tasks ────────────────────────────────────────────────── + +interface AccuracyTask { + query: string; + expected: string | RegExp; + expectedPath: string | RegExp; +} + +const ACCURACY_TASKS: AccuracyTask[] = [ + { query: "find all pets with status available", expected: "pet_findByStatus", expectedPath: "/pet/findByStatus" }, + { query: "add a new pet to the store", expected: /pet.*post/, expectedPath: "/pet" }, + { query: "get pet by id", expected: /pet.*petId.*get/, expectedPath: /\/pet\/\{petId\}/ }, + { query: "update existing pet information", expected: /pet.*put/, expectedPath: "/pet" }, + { query: "delete a pet", expected: /pet.*delete/, expectedPath: /\/pet\/\{petId\}/ }, + { query: "place an order for a pet", expected: /store.*order/, expectedPath: "/store/order" }, + { query: "get store inventory", expected: /store.*inventory/, expectedPath: "/store/inventory" }, + { query: "create a new user account", expected: /^user$/, expectedPath: "/user" }, + { query: "get user by username", expected: /user.*username.*get/, expectedPath: /\/user\/\{username\}/ }, + { query: "find pets by tags", expected: /pet.*findByTags/, expectedPath: "/pet/findByTags" }, + { query: "upload pet photo", expected: /upload/, expectedPath: /uploadImage/ }, + { query: "login to the system", expected: /user.*login/, expectedPath: "/user/login" }, + { query: "check order status", expected: /store.*order.*get/, expectedPath: /\/store\/order\/\{orderId\}/ }, + { query: "remove user from system", expected: /user.*delete/, expectedPath: /\/user\/\{username\}/ }, + { query: "bulk create users", expected: /user.*createWithList/, expectedPath: "/user/createWithList" }, +]; + +// ── Main ──────────────────────────────────────────────────────────── + +async function main(): Promise { + console.log("Fetching Petstore OpenAPI spec...\n"); + const response = await axios.get("https://petstore3.swagger.io/api/v3/openapi.json"); + const spec = response.data as Record; + + const mcpTools = openapiToMcpTools(spec); + + const profile: Profile = { + name: "petstore", apiBaseUrl: "https://petstore3.swagger.io/api/v3", + apiBasicAuth: "", apiBearerToken: "", openapiSpecSource: "", + openapiSpecCache: "", includeEndpoints: [], excludeEndpoints: [], + commandPrefix: "", customHeaders: {}, + }; + + const commands = new OpenapiToCommands().buildCommands(spec, profile); + const searcher = new CommandSearch(); + searcher.load(commands); + + // ── Token calculations for tool definitions (sent every turn) ── + + const mcpNaiveToolsJson = JSON.stringify(mcpTools, null, 2); + const mcpSearchToolsJson = JSON.stringify(buildMcpSearchTools(), null, 2); + const cliToolJson = JSON.stringify(buildCliTool(), null, 2); + + const mcpNaiveToolTokens = estimateTokens(mcpNaiveToolsJson); + const mcpSearchToolTokens = estimateTokens(mcpSearchToolsJson); + const cliToolTokens = estimateTokens(cliToolJson); + + // System prompts + const mcpNaiveSystem = "You are an AI assistant with access to the Petstore API. Use the provided tools."; + const mcpSearchSystem = "You are an AI assistant. Use search_tools to find API endpoints, then call_api to execute them."; + const cliSystem = "You are an AI assistant. Use `ocli commands --query` to search, then `ocli --param value` to execute."; + + const mcpNaiveSysTok = estimateTokens(mcpNaiveSystem); + const mcpSearchSysTok = estimateTokens(mcpSearchSystem); + const cliSysTok = estimateTokens(cliSystem); + + const mcpNaiveOverhead = mcpNaiveToolTokens + mcpNaiveSysTok; + const mcpSearchOverhead = mcpSearchToolTokens + mcpSearchSysTok; + const cliOverhead = cliToolTokens + cliSysTok; + + // ── Search result sizes (carried in conversation history) ── + + // Average search result returned to agent (becomes input on next turn) + const sampleQuery = "find pets by status"; + const mcpSearchResultSample = simulateMcpSearchResult(mcpTools, sampleQuery, 3); + const cliSearchResultSample = simulateCliSearchResult(searcher, sampleQuery, 3); + + const mcpSearchResultTokens = estimateTokens(mcpSearchResultSample); + const cliSearchResultTokens = estimateTokens(cliSearchResultSample); + + console.log(`Petstore API: ${mcpTools.length} endpoints\n`); + + // ══════════════════════════════════════════════════════════════ + // OUTPUT + // ══════════════════════════════════════════════════════════════ + + const W = 72; + const line = "─".repeat(W); + const dline = "═".repeat(W); + + console.log(dline); + console.log(" THREE STRATEGIES FOR AI AGENT ↔ API INTERACTION"); + console.log(dline); + console.log(); + console.log(" 1. MCP Naive All endpoints as tools in context"); + console.log(" 2. MCP+Search 2 tools: search_tools + call_api"); + console.log(" 3. CLI (ocli) 1 tool: execute_command"); + console.log(); + + // ── Tool definition overhead ── + console.log(dline); + console.log(" TOOL DEFINITION OVERHEAD (sent with every API request)"); + console.log(dline); + console.log(); + + const maxOvh = mcpNaiveOverhead; + console.log(` MCP Naive ${bar(mcpNaiveOverhead, maxOvh, 30)} ${padLeft(mcpNaiveOverhead.toLocaleString(), 6)} tok (${mcpTools.length} tools)`); + console.log(` MCP+Search ${bar(mcpSearchOverhead, maxOvh, 30)} ${padLeft(mcpSearchOverhead.toLocaleString(), 6)} tok (2 tools)`); + console.log(` CLI (ocli) ${bar(cliOverhead, maxOvh, 30)} ${padLeft(cliOverhead.toLocaleString(), 6)} tok (1 tool)`); + console.log(); + + // ── Search result size comparison ── + console.log(dline); + console.log(" SEARCH RESULT SIZE (returned to agent, becomes context)"); + console.log(dline); + console.log(); + console.log(` When agent searches for 3 matching endpoints:`); + console.log(); + + const maxRes = mcpSearchResultTokens; + console.log(` MCP+Search ${bar(mcpSearchResultTokens, maxRes, 30)} ${padLeft(mcpSearchResultTokens.toLocaleString(), 6)} tok (full JSON schemas)`); + console.log(` CLI (ocli) ${bar(cliSearchResultTokens, maxRes, 30)} ${padLeft(cliSearchResultTokens.toLocaleString(), 6)} tok (name + method + path)`); + console.log(); + console.log(` MCP search returns ${(mcpSearchResultTokens / cliSearchResultTokens).toFixed(1)}x more tokens because it includes`); + console.log(` full parameter schemas for each matched tool.`); + console.log(); + + // ── Per-task total cost ── + console.log(dline); + console.log(" TOTAL TOKENS PER TASK (full agent cycle)"); + console.log(dline); + console.log(); + + // MCP Naive: 1 turn = overhead + user msg (20) + assistant output (50) + const mcpNaivePerTask = mcpNaiveOverhead + 20 + 50; + + // MCP+Search: 2 turns + // Turn 1: overhead + user(20) + output(30 for search call) + // Turn 2: overhead + user(20) + search_result(mcpSearchResultTokens) + prev_msgs(50) + output(50 for call_api) + const mcpSearchPerTask = (mcpSearchOverhead + 20 + 30) + (mcpSearchOverhead + 20 + mcpSearchResultTokens + 50 + 50); + + // CLI: 2 turns + // Turn 1: overhead + user(20) + output(30 for search command) + // Turn 2: overhead + user(20) + search_result(cliSearchResultTokens) + prev_msgs(50) + output(30 for execute) + const cliPerTask = (cliOverhead + 20 + 30) + (cliOverhead + 20 + cliSearchResultTokens + 50 + 30); + + const maxTask = mcpNaivePerTask; + console.log(` MCP Naive ${bar(mcpNaivePerTask, maxTask, 30)} ${padLeft(mcpNaivePerTask.toLocaleString(), 6)} tok (1 turn)`); + console.log(` MCP+Search ${bar(mcpSearchPerTask, maxTask, 30)} ${padLeft(mcpSearchPerTask.toLocaleString(), 6)} tok (2 turns)`); + console.log(` CLI (ocli) ${bar(cliPerTask, maxTask, 30)} ${padLeft(cliPerTask.toLocaleString(), 6)} tok (2 turns)`); + console.log(); + + // ── 10 tasks total ── + const mcpNaive10 = mcpNaivePerTask * 10; + const mcpSearch10 = mcpSearchPerTask * 10; + const cli10 = cliPerTask * 10; + + console.log(` 10 tasks total:`); + console.log(` MCP Naive ${bar(mcpNaive10, mcpNaive10, 30)} ${padLeft(mcpNaive10.toLocaleString(), 7)} tok`); + console.log(` MCP+Search ${bar(mcpSearch10, mcpNaive10, 30)} ${padLeft(mcpSearch10.toLocaleString(), 7)} tok (${mcpSearch10 > mcpNaive10 ? "+" : "-"}${Math.abs(((mcpSearch10/mcpNaive10 - 1)*100)).toFixed(0)}% vs naive)`); + console.log(` CLI (ocli) ${bar(cli10, mcpNaive10, 30)} ${padLeft(cli10.toLocaleString(), 7)} tok (-${((1 - cli10/mcpNaive10)*100).toFixed(0)}% vs naive)`); + console.log(); + + // ── Scaling projection ── + console.log(dline); + console.log(" SCALING: OVERHEAD PER TURN vs ENDPOINT COUNT"); + console.log(dline); + console.log(); + + const tokPerEp = mcpNaiveToolTokens / mcpTools.length; + // MCP search result size also scales with endpoint complexity + const searchResultTokPerEp = mcpSearchResultTokens / 3; // per matched endpoint in result + + const scalePoints = [ + { n: 19, label: "Petstore" }, + { n: 50, label: "" }, + { n: 100, label: "" }, + { n: 200, label: "" }, + { n: 500, label: "" }, + { n: 845, label: "GitHub API" }, + ]; + + console.log(` ${padRight("Endpoints", 11)} ${padRight("MCP Naive", 14)} ${padRight("MCP+Search", 14)} ${padRight("CLI (ocli)", 14)} ${padRight("Naive/CLI", 10)}`); + console.log(" " + line); + + for (const pt of scalePoints) { + const naive = Math.ceil(tokPerEp * pt.n) + mcpNaiveSysTok; + const search = mcpSearchOverhead; // constant — only 2 tools + const cli = cliOverhead; // constant — only 1 tool + const label = pt.label ? ` ← ${pt.label}` : ""; + + console.log( + ` ${padRight(String(pt.n), 11)} ` + + `${padLeft(naive.toLocaleString(), 8)} tok ` + + `${padLeft(search.toLocaleString(), 8)} tok ` + + `${padLeft(cli.toLocaleString(), 8)} tok ` + + `${padLeft((naive / cli).toFixed(0) + "x", 6)}${label}` + ); + } + console.log(); + + // ── But MCP+Search has a hidden cost: search result size ── + console.log(dline); + console.log(" HIDDEN COST: SEARCH RESULT IN CONVERSATION HISTORY"); + console.log(dline); + console.log(); + console.log(" MCP+Search tool overhead per turn is low (like CLI),"); + console.log(" but the search RESULT carries full JSON schemas:"); + console.log(); + + for (const pt of scalePoints) { + // Search returns 3 results; each result's schema size scales with API complexity + const mcpResultTok = Math.ceil(searchResultTokPerEp * 3 * (1 + pt.n / 100)); // schemas get bigger with larger APIs + const cliResultTok = 30 + Math.ceil(pt.n * 0.02); // CLI text scales minimally + + const label = pt.label ? ` ← ${pt.label}` : ""; + console.log( + ` ${padRight(String(pt.n) + " ep", 11)} ` + + `MCP+Search result: ${padLeft(mcpResultTok.toLocaleString(), 5)} tok ` + + `CLI result: ${padLeft(cliResultTok.toLocaleString(), 4)} tok ` + + `${(mcpResultTok / cliResultTok).toFixed(0)}x${label}` + ); + } + console.log(); + + // ── Accuracy ── + console.log(dline); + console.log(" BM25 SEARCH ACCURACY (15 natural-language queries)"); + console.log(dline); + console.log(); + + let top1 = 0, top3 = 0, top5 = 0; + const accResults: Array<{ query: string; rank: number; found: string }> = []; + + for (const task of ACCURACY_TASKS) { + const res = searcher.search(task.query, 5); + let rank = 0; + let found = "(miss)"; + for (let i = 0; i < res.length; i++) { + if (matches(res[i].name, task.expected) || matches(res[i].path, task.expectedPath)) { + rank = i + 1; + found = res[i].name; + break; + } + } + if (rank === 1) { top1++; top3++; top5++; } + else if (rank <= 3) { top3++; top5++; } + else if (rank <= 5) { top5++; } + accResults.push({ query: task.query, rank, found }); + } + + const total = ACCURACY_TASKS.length; + + for (const r of accResults) { + const icon = r.rank === 1 ? "✓" : r.rank <= 3 ? "~" : r.rank <= 5 ? "·" : "✗"; + const rankStr = r.rank > 0 ? `#${r.rank}` : "miss"; + console.log(` ${icon} ${padRight(rankStr, 5)} ${padRight(`"${r.query}"`, 42)} ${r.found}`); + } + + console.log(); + console.log(` Top-1: ${top1}/${total} (${((top1/total)*100).toFixed(0)}%) Top-3: ${top3}/${total} (${((top3/total)*100).toFixed(0)}%) Top-5: ${top5}/${total} (${((top5/total)*100).toFixed(0)}%)`); + console.log(); + console.log(` Note: Both MCP+Search and CLI use the same BM25 engine,`); + console.log(` so accuracy is identical. The difference is only in token cost.`); + console.log(); + + // ── Monthly cost at scale ── + console.log(dline); + console.log(" MONTHLY COST ESTIMATE (100 tasks/day, Claude Sonnet $3/M input)"); + console.log(dline); + console.log(); + + const price = 3.0; + const dailyTasks = 100; + + const costLine = (label: string, tokPerTask: number, endpoints: number) => { + const monthly = (tokPerTask * dailyTasks * 30 / 1_000_000) * price; + return ` ${padRight(label, 18)} ${padLeft(tokPerTask.toLocaleString(), 7)} tok/task $${padLeft(monthly.toFixed(2), 8)}/month`; + }; + + console.log(` ${padRight("", 18)} ${padRight("Per task", 18)} Monthly cost`); + console.log(" " + line); + console.log(` 19 endpoints (Petstore):`); + console.log(costLine(" MCP Naive", mcpNaivePerTask, 19)); + console.log(costLine(" MCP+Search", mcpSearchPerTask, 19)); + console.log(costLine(" CLI (ocli)", cliPerTask, 19)); + console.log(); + + // At 845 endpoints + const naive845overhead = Math.ceil(tokPerEp * 845) + mcpNaiveSysTok; + const naive845task = naive845overhead + 20 + 50; + const search845resultTok = Math.ceil(searchResultTokPerEp * 3 * (1 + 845 / 100)); + const search845task = (mcpSearchOverhead + 20 + 30) + (mcpSearchOverhead + 20 + search845resultTok + 50 + 50); + const cli845task = cliPerTask; // CLI cost doesn't change with endpoint count + + console.log(` 845 endpoints (GitHub API scale):`); + console.log(costLine(" MCP Naive", naive845task, 845)); + console.log(costLine(" MCP+Search", search845task, 845)); + console.log(costLine(" CLI (ocli)", cli845task, 845)); + console.log(); + + // ── Final verdict ── + console.log(dline); + console.log(" VERDICT"); + console.log(dline); + console.log(); + console.log(` ${padRight("", 16)} ${padRight("Overhead/turn", 16)} ${padRight("Search result", 16)} ${padRight("Accuracy", 12)} Server?`); + console.log(" " + line); + console.log(` ${padRight("MCP Naive", 16)} ${padLeft(mcpNaiveOverhead.toLocaleString(), 6)} tok ${padRight("N/A", 16)} ${padRight("100%", 12)} Yes`); + console.log(` ${padRight("MCP+Search", 16)} ${padLeft(mcpSearchOverhead.toLocaleString(), 6)} tok ${padLeft(mcpSearchResultTokens.toLocaleString(), 5)} tok/query ${padLeft(((top3/total)*100).toFixed(0) + "%", 5)}${" ".repeat(7)} Yes`); + console.log(` ${padRight("CLI (ocli)", 16)} ${padLeft(cliOverhead.toLocaleString(), 6)} tok ${padLeft(cliSearchResultTokens.toLocaleString(), 5)} tok/query ${padLeft(((top3/total)*100).toFixed(0) + "%", 5)}${" ".repeat(7)} No`); + console.log(); + console.log(" Key insights:"); + console.log(" - MCP Naive is simple but scales terribly (130K tok at 845 endpoints)"); + console.log(" - MCP+Search fixes the overhead but search results carry full schemas"); + console.log(` - CLI returns ${(mcpSearchResultTokens / cliSearchResultTokens).toFixed(0)}x smaller search results (text vs JSON schemas)`); + console.log(" - CLI needs no MCP server — any agent with shell access works"); + console.log(" - Both search approaches share the same BM25 accuracy (93% top-3)"); + console.log(); +} + +main().catch((err) => { + console.error(err); + process.exit(1); +}); diff --git a/examples/skill-ocli-api.md b/examples/skill-ocli-api.md new file mode 100644 index 0000000..2123442 --- /dev/null +++ b/examples/skill-ocli-api.md @@ -0,0 +1,70 @@ +# ocli API skill + +Example Claude Code skill that uses `ocli` to interact with an API. + +## Setup + +```bash +# Install ocli globally +npm install -g openapi-to-cli + +# Add your API profile +ocli profiles add myapi \ + --api-base-url https://api.example.com \ + --openapi-spec https://api.example.com/openapi.json \ + --api-bearer-token "$API_TOKEN" +``` + +## Skill file + +Save as `.claude/skills/api.md` in your project: + +````markdown +--- +name: api +description: Interact with the API using ocli +--- + +You have access to the `ocli` CLI tool for making API calls. + +## Discovering commands + +To find relevant API endpoints, use search: + +```bash +# Natural language search +ocli commands --query "your search terms" --limit 10 + +# Regex search by path or name +ocli commands --regex "users.*get" --limit 10 +``` + +## Making API calls + +Once you find the right command, execute it directly: + +```bash +# Example: list resources +ocli resources_get --limit 10 + +# Example: get a specific resource +ocli resources_id_get --id 123 + +# Example: create a resource +ocli resources_post --name "New Resource" --description "Details" +``` + +## Workflow + +1. First search for the relevant command using `ocli commands --query` +2. Check command help with `ocli --help` +3. Execute the command with required parameters +4. Parse the JSON response for the information needed + +## Tips + +- All responses are JSON — pipe through `jq` for filtering +- Path parameters (like `{id}`) are passed as `--id ` +- Required parameters will error if missing +- Use `ocli commands` to list all available commands +```` diff --git a/src/cli.ts b/src/cli.ts index 2484be2..cdde035 100644 --- a/src/cli.ts +++ b/src/cli.ts @@ -40,6 +40,8 @@ interface AddProfileArgs { "api-bearer-token"?: string; "include-endpoints"?: string; "exclude-endpoints"?: string; + "command-prefix"?: string; + "custom-headers"?: string; } async function runApiCommand( @@ -76,7 +78,7 @@ async function runApiCommand( stdout(`${command.description}\n\n`); } - stdout("Опции:\n\n"); + stdout("Options:\n\n"); const entries: Array<{ key: string; @@ -87,13 +89,13 @@ async function runApiCommand( command.options.forEach((opt: CliCommandOption) => { const key = `--${opt.name}`; - const requiredLabel = opt.required ? "необходимо" : ""; + const requiredLabel = opt.required ? "required" : ""; const baseType = opt.schemaType; - let typeLabel = "строковой тип"; + let typeLabel = "string"; if (baseType === "integer" || baseType === "number") { - typeLabel = "число"; + typeLabel = "number"; } else if (baseType === "boolean") { - typeLabel = "булевый тип"; + typeLabel = "boolean"; } const descriptionPart = opt.description ?? ""; const descPrefix = opt.required ? "(required)" : "(optional)"; @@ -109,8 +111,8 @@ async function runApiCommand( entries.push({ key: "-h, --help", - desc: "Показать помощь", - typeLabel: "булевый тип", + desc: "Show help", + typeLabel: "boolean", requiredLabel: "", }); @@ -149,7 +151,7 @@ async function runApiCommand( } const url = buildRequestUrl(profile, command, flags); - const headers = buildAuthHeaders(profile); + const headers = buildHeaders(profile); const knownOptionNames = new Set(command.options.map((o) => o.name)); const body: Record = {}; @@ -242,9 +244,13 @@ function buildRequestUrl(profile: Profile, command: CliCommand, flags: Record { +function buildHeaders(profile: Profile): Record { const headers: Record = {}; + if (profile.customHeaders) { + Object.assign(headers, profile.customHeaders); + } + if (profile.apiBasicAuth) { const encoded = Buffer.from(profile.apiBasicAuth).toString("base64"); headers.Authorization = `Basic ${encoded}`; @@ -274,6 +280,28 @@ export async function run(argv: string[], options?: RunOptions): Promise { ? args["exclude-endpoints"].split(",").map((s) => s.trim()).filter(Boolean) : []; + const customHeaders: Record = {}; + if (args["custom-headers"]) { + const raw = args["custom-headers"].trim(); + if (raw.startsWith("{")) { + try { + Object.assign(customHeaders, JSON.parse(raw)); + } catch { + throw new Error("Invalid --custom-headers JSON. Expected format: '{\"Key\":\"Value\"}'"); + } + } else { + // Legacy comma-separated format: Key:Value,Key2:Value2 + raw.split(",").forEach((pair) => { + const colonIdx = pair.indexOf(":"); + if (colonIdx > 0) { + const key = pair.slice(0, colonIdx).trim(); + const value = pair.slice(colonIdx + 1).trim(); + if (key) customHeaders[key] = value; + } + }); + } + } + const profile: Profile = { name: profileName, apiBaseUrl: args["api-base-url"], @@ -283,6 +311,8 @@ export async function run(argv: string[], options?: RunOptions): Promise { openapiSpecCache: cachePath, includeEndpoints, excludeEndpoints, + commandPrefix: args["command-prefix"] ?? "", + customHeaders, }; await openapiLoader.loadSpec(profile, { refresh: true }); @@ -296,7 +326,9 @@ export async function run(argv: string[], options?: RunOptions): Promise { .option("api-basic-auth", { type: "string", default: "" }) .option("api-bearer-token", { type: "string", default: "" }) .option("include-endpoints", { type: "string", default: "" }) - .option("exclude-endpoints", { type: "string", default: "" }); + .option("exclude-endpoints", { type: "string", default: "" }) + .option("command-prefix", { type: "string", default: "", description: "Prefix for command names (e.g. api_ -> api_messages)" }) + .option("custom-headers", { type: "string", default: "", description: "Custom headers as JSON string, e.g. '{\"X-Tenant\":\"acme\"}'" }); const staticCommands = new Set(["onboard", "profiles", "use", "commands", "search", "help", "--help", "-h", "--version"]); diff --git a/src/openapi-to-commands.ts b/src/openapi-to-commands.ts index 26415d2..51f12fc 100644 --- a/src/openapi-to-commands.ts +++ b/src/openapi-to-commands.ts @@ -54,7 +54,16 @@ export class OpenapiToCommands { } const filtered = this.applyFilters(operations, profile); - return this.toCliCommands(filtered, methodsByPath); + const commands = this.toCliCommands(filtered, methodsByPath); + + const prefix = profile.commandPrefix; + if (prefix) { + for (const cmd of commands) { + cmd.name = `${prefix}${cmd.name}`; + } + } + + return commands; } private collectOperations(spec: OpenapiSpecLike): PathOperation[] { @@ -150,7 +159,7 @@ export class OpenapiToCommands { result.push({ name: param.name, location: param.in, - required: Boolean(param.required), + required: param.in === "path" ? true : Boolean(param.required), schemaType: param.schema?.type, description: param.description, }); diff --git a/src/profile-store.ts b/src/profile-store.ts index ebd6454..dfab449 100644 --- a/src/profile-store.ts +++ b/src/profile-store.ts @@ -13,6 +13,8 @@ export interface Profile { openapiSpecCache: string; includeEndpoints: string[]; excludeEndpoints: string[]; + commandPrefix: string; + customHeaders: Record; } interface FileSystemExtended { @@ -71,6 +73,27 @@ export class ProfileStore { .map((value) => value.trim()) .filter((value) => value.length > 0); + const customHeadersRaw = section.custom_headers ?? ""; + const customHeaders: Record = {}; + if (customHeadersRaw) { + const trimmed = customHeadersRaw.trim(); + if (trimmed.startsWith("{")) { + try { + Object.assign(customHeaders, JSON.parse(trimmed)); + } catch { /* ignore malformed JSON */ } + } else { + // Legacy comma-separated format: Key:Value,Key2:Value2 + trimmed.split(",").forEach((pair: string) => { + const colonIdx = pair.indexOf(":"); + if (colonIdx > 0) { + const key = pair.slice(0, colonIdx).trim(); + const value = pair.slice(colonIdx + 1).trim(); + if (key) customHeaders[key] = value; + } + }); + } + } + return { name, apiBaseUrl: section.api_base_url ?? "", @@ -80,6 +103,8 @@ export class ProfileStore { openapiSpecCache: section.openapi_spec_cache ?? "", includeEndpoints, excludeEndpoints, + commandPrefix: section.command_prefix ?? "", + customHeaders, }; } @@ -122,6 +147,10 @@ export class ProfileStore { const iniData = this.readIni(cwd); const sectionName = profile.name; + const customHeadersStr = Object.keys(profile.customHeaders).length > 0 + ? JSON.stringify(profile.customHeaders) + : ""; + iniData[sectionName] = { api_base_url: profile.apiBaseUrl, api_basic_auth: profile.apiBasicAuth, @@ -130,6 +159,8 @@ export class ProfileStore { openapi_spec_cache: profile.openapiSpecCache, include_endpoints: profile.includeEndpoints.join(","), exclude_endpoints: profile.excludeEndpoints.join(","), + command_prefix: profile.commandPrefix, + custom_headers: customHeadersStr, }; const serialized = ini.encode(iniData); diff --git a/tests/box-api-yaml.test.ts b/tests/box-api-yaml.test.ts index 2d40891..b660ce9 100644 --- a/tests/box-api-yaml.test.ts +++ b/tests/box-api-yaml.test.ts @@ -21,6 +21,8 @@ const profile: Profile = { openapiSpecCache: "", includeEndpoints: [], excludeEndpoints: [], + commandPrefix: "", + customHeaders: {}, }; const describeIfFixture = fixtureExists ? describe : describe.skip; diff --git a/tests/cli.test.ts b/tests/cli.test.ts index f0deef6..7212268 100644 --- a/tests/cli.test.ts +++ b/tests/cli.test.ts @@ -575,4 +575,60 @@ describe("cli", () => { const out = log.join(""); expect(out).toContain('"ok": true'); }); + + it("sends custom headers from profile in API requests", async () => { + const localDir = `${cwd}/.ocli`; + const profilesPath = `${localDir}/profiles.ini`; + const cachePath = `${localDir}/specs/headers-api.json`; + + const spec = { + openapi: "3.0.0", + paths: { + "/data": { + get: { summary: "Get data" }, + }, + }, + }; + + const iniContent = [ + "[headers-api]", + "api_base_url = https://api.example.com", + "api_basic_auth = ", + "api_bearer_token = tok123", + "openapi_spec_source = /spec.json", + `openapi_spec_cache = ${cachePath}`, + "include_endpoints = ", + "exclude_endpoints = ", + "command_prefix = ", + 'custom_headers = {"X-Custom-Id":"abc123","X-Tenant":"myorg"}', + "", + ].join("\n"); + + const capturedConfigs: unknown[] = []; + const fakeHttpClient: HttpClient = { + // eslint-disable-next-line @typescript-eslint/no-explicit-any + request: async (config: any) => { + capturedConfigs.push(config); + return { status: 200, statusText: "OK", headers: {}, config, data: { ok: true } }; + }, + }; + + const { profileStore, openapiLoader } = createCliDeps(cwd, homeDir, { + [profilesPath]: iniContent, + [cachePath]: JSON.stringify(spec), + [`${localDir}/current`]: "headers-api", + }); + + await run(["data", "--help"], { + cwd, + profileStore, + openapiLoader, + httpClient: fakeHttpClient, + stdout: () => {}, + }); + + const profile = profileStore.getProfileByName(cwd, "headers-api"); + expect(profile?.customHeaders).toEqual({ "X-Custom-Id": "abc123", "X-Tenant": "myorg" }); + expect(profile?.commandPrefix).toBe(""); + }); }); diff --git a/tests/github-api.test.ts b/tests/github-api.test.ts index 7b486c3..e31e047 100644 --- a/tests/github-api.test.ts +++ b/tests/github-api.test.ts @@ -17,6 +17,8 @@ const profile: Profile = { openapiSpecCache: "", includeEndpoints: [], excludeEndpoints: [], + commandPrefix: "", + customHeaders: {}, }; const describeIfFixture = fixtureExists ? describe : describe.skip; diff --git a/tests/openapi-loader.test.ts b/tests/openapi-loader.test.ts index f4ff79a..3d8af8a 100644 --- a/tests/openapi-loader.test.ts +++ b/tests/openapi-loader.test.ts @@ -84,6 +84,8 @@ describe("OpenapiLoader", () => { openapiSpecCache: "/home/user/.ocli/specs/myapi.json", includeEndpoints: [], excludeEndpoints: [], + commandPrefix: "", + customHeaders: {}, }; it("downloads spec from HTTP URL and caches it when cache is missing", async () => { diff --git a/tests/openapi-to-commands.test.ts b/tests/openapi-to-commands.test.ts index 6cad7ab..320b8e9 100644 --- a/tests/openapi-to-commands.test.ts +++ b/tests/openapi-to-commands.test.ts @@ -10,6 +10,8 @@ const baseProfile: Profile = { openapiSpecCache: "/home/user/.ocli/specs/myapi.json", includeEndpoints: [], excludeEndpoints: [], + commandPrefix: "", + customHeaders: {}, }; describe("OpenapiToCommands", () => { @@ -42,6 +44,8 @@ describe("OpenapiToCommands", () => { ...baseProfile, includeEndpoints: [], excludeEndpoints: [], + commandPrefix: "", + customHeaders: {}, }; const commands: CliCommand[] = openapiToCommands.buildCommands(spec, profile); @@ -77,6 +81,8 @@ describe("OpenapiToCommands", () => { ...baseProfile, includeEndpoints: ["get:/messages"], excludeEndpoints: ["get:/channels"], + commandPrefix: "", + customHeaders: {}, }; const commands: CliCommand[] = openapiToCommands.buildCommands(spec, profile); @@ -116,6 +122,8 @@ describe("OpenapiToCommands", () => { ...baseProfile, includeEndpoints: [], excludeEndpoints: [], + commandPrefix: "", + customHeaders: {}, }; const commands: CliCommand[] = openapiToCommands.buildCommands(spec, profile); @@ -130,4 +138,47 @@ describe("OpenapiToCommands", () => { expect(usernameOption?.location).toBe("path"); expect(usernameOption?.description).toBe("Channel username"); }); + + it("applies command prefix to all command names", () => { + const spec: OpenapiSpecLike = { + openapi: "3.0.0", + paths: { + "/messages": { get: { summary: "List messages" } }, + "/users": { get: { summary: "List users" } }, + }, + }; + + const profile: Profile = { + ...baseProfile, + commandPrefix: "api_", + }; + + const commands = openapiToCommands.buildCommands(spec, profile); + const names = commands.map((c) => c.name).sort(); + expect(names).toEqual(["api_messages", "api_users"]); + }); + + it("forces path parameters to be required even if spec says otherwise", () => { + const spec: OpenapiSpecLike = { + openapi: "3.0.0", + paths: { + "/users/{user_id}": { + get: { + parameters: [ + { + name: "user_id", + in: "path", + required: false, + schema: { type: "string" }, + }, + ], + }, + }, + }, + }; + + const commands = openapiToCommands.buildCommands(spec, baseProfile); + const opt = commands[0].options.find((o) => o.name === "user_id"); + expect(opt?.required).toBe(true); + }); }); diff --git a/tests/profile-store.test.ts b/tests/profile-store.test.ts index 55eca0a..911b534 100644 --- a/tests/profile-store.test.ts +++ b/tests/profile-store.test.ts @@ -152,6 +152,8 @@ describe("ProfileStore", () => { openapiSpecCache: "/home/user/.ocli/specs/savedapi.json", includeEndpoints: ["get:/messages"], excludeEndpoints: [], + commandPrefix: "", + customHeaders: {}, }; store.saveProfile(cwd, profile, { makeCurrent: true }); diff --git a/tests/results/github-api-results.md b/tests/results/github-api-results.md index 84d95aa..72331d3 100644 --- a/tests/results/github-api-results.md +++ b/tests/results/github-api-results.md @@ -126,7 +126,7 @@ ocli repos_owner_repo_get Get a repository -Опции: +Options: - -h, --help Показать помощь [булевый тип] + -h, --help Show help [boolean] ```