From 81e83a2ad984f164eacc5adf3897eabecca15f22 Mon Sep 17 00:00:00 2001 From: Adam Altman Date: Wed, 11 Mar 2026 20:09:39 -0500 Subject: [PATCH 01/22] feat: add score command --- .changeset/add-score-command.md | 5 + docs/@v2/commands/index.md | 5 +- docs/@v2/commands/score.md | 152 ++++++++ .../score/__tests__/example-coverage.test.ts | 183 ++++++++++ .../score/__tests__/schema-walker.test.ts | 334 ++++++++++++++++++ .../commands/score/__tests__/scoring.test.ts | 284 +++++++++++++++ .../score/collectors/document-metrics.ts | 261 ++++++++++++++ .../score/collectors/schema-walker.ts | 165 +++++++++ .../score/collectors/workflow-graph.ts | 71 ++++ packages/cli/src/commands/score/constants.ts | 57 +++ .../cli/src/commands/score/formatters/json.ts | 60 ++++ .../src/commands/score/formatters/stylish.ts | 149 ++++++++ packages/cli/src/commands/score/hotspots.ts | 115 ++++++ packages/cli/src/commands/score/index.ts | 145 ++++++++ packages/cli/src/commands/score/scoring.ts | 199 +++++++++++ packages/cli/src/commands/score/types.ts | 140 ++++++++ packages/cli/src/index.ts | 25 ++ 17 files changed, 2348 insertions(+), 2 deletions(-) create mode 100644 .changeset/add-score-command.md create mode 100644 docs/@v2/commands/score.md create mode 100644 packages/cli/src/commands/score/__tests__/example-coverage.test.ts create mode 100644 packages/cli/src/commands/score/__tests__/schema-walker.test.ts create mode 100644 packages/cli/src/commands/score/__tests__/scoring.test.ts create mode 100644 packages/cli/src/commands/score/collectors/document-metrics.ts create mode 100644 packages/cli/src/commands/score/collectors/schema-walker.ts create mode 100644 packages/cli/src/commands/score/collectors/workflow-graph.ts create mode 100644 packages/cli/src/commands/score/constants.ts create mode 100644 packages/cli/src/commands/score/formatters/json.ts create mode 100644 packages/cli/src/commands/score/formatters/stylish.ts create mode 100644 packages/cli/src/commands/score/hotspots.ts create mode 100644 packages/cli/src/commands/score/index.ts create mode 100644 packages/cli/src/commands/score/scoring.ts create mode 100644 packages/cli/src/commands/score/types.ts diff --git a/.changeset/add-score-command.md b/.changeset/add-score-command.md new file mode 100644 index 0000000000..e118660343 --- /dev/null +++ b/.changeset/add-score-command.md @@ -0,0 +1,5 @@ +--- +'@redocly/cli': minor +--- + +Added new `score` command that analyzes OpenAPI 3.x descriptions and produces integration simplicity and agent readiness scores (0-100). Reports normalized subscores, raw per-operation metrics, and top hotspot operations with human-readable explanations. Supports `--format=stylish` (default) and `--format=json` output. diff --git a/docs/@v2/commands/index.md b/docs/@v2/commands/index.md index 8dafe429df..aad01badaf 100644 --- a/docs/@v2/commands/index.md +++ b/docs/@v2/commands/index.md @@ -13,10 +13,11 @@ Documentation commands: API management commands: -- [`stats`](stats.md) Gather statistics for a document. - [`bundle`](bundle.md) Bundle API description. -- [`split`](split.md) Split API description into a multi-file structure. - [`join`](join.md) Join API descriptions [experimental feature]. +- [`score`](score.md) Score an API for integration simplicity and agent readiness. +- [`split`](split.md) Split API description into a multi-file structure. +- [`stats`](stats.md) Gather statistics for a document. Linting commands: diff --git a/docs/@v2/commands/score.md b/docs/@v2/commands/score.md new file mode 100644 index 0000000000..cdea1bcb9b --- /dev/null +++ b/docs/@v2/commands/score.md @@ -0,0 +1,152 @@ +# `score` + +## Introduction + +The `score` command analyzes an OpenAPI description and produces two composite scores: + +- **Integration Simplicity** (0–100): How easy is this API to integrate? Higher is better. +- **Agent Readiness** (0–100): How usable is this API by AI agents and LLM-based tooling? Higher is better. + +In addition to the top-level scores, the command reports normalized subscores, raw metrics for every operation, and a list of **hotspot operations** — the endpoints most likely to cause integration friction — along with human-readable explanations. + +{% admonition type="warning" name="OpenAPI 3.x only" %} +The `score` command supports OpenAPI 3.0, 3.1, and 3.2 descriptions. OpenAPI 2.0 (Swagger) and AsyncAPI are not currently supported. +{% /admonition %} + +### Metrics + +The following raw metrics are collected per operation and aggregated across the document: + +| Metric | Description | +| ---------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ | +| Parameter count | Total parameters (path, query, header, cookie) per operation. | +| Required parameter count | How many of those parameters are required. | +| Request body presence | Whether the operation defines a request body. | +| Top-level writable field count | Number of non-`readOnly` top-level properties in request schemas. | +| Max request/response schema depth | Deepest nesting level in request and response schemas. | +| Polymorphism count | Number of `oneOf`, `anyOf`, and `allOf` usages. `anyOf` is penalized more heavily. | +| Property count | Total schema properties across request and response. | +| Description coverage | Fraction of operations, parameters, and schema properties that have descriptions. | +| Ambiguous identifier count | Parameters with generic names (e.g. `id`, `name`, `type`) and no description. | +| Constraint coverage | Count of constraining keywords (`enum`, `format`, `pattern`, `minimum`, `maximum`, `minLength`, `maxLength`, `discriminator`, etc.). | +| Request/response example coverage | Whether request and response media types include `example` or `examples`. | +| Structured error response coverage | How many 4xx/5xx responses include a content schema or meaningful description. | +| Workflow dependency depth | Inferred from shared `$ref` usage across operations; deeper shared-schema graphs indicate more tightly coupled workflows. | + +### Subscores + +Subscores are normalized to 0–1 and grouped into two categories: + +**Integration Simplicity subscores:** `parameterSimplicity`, `schemaSimplicity`, `documentationQuality`, `constraintClarity`, `exampleCoverage`, `errorClarity`, `workflowClarity`. + +**Agent Readiness subscores:** `documentationQuality`, `constraintClarity`, `exampleCoverage`, `errorClarity`, `identifierClarity`, `workflowClarity`, `polymorphismClarity`. + +### Hotspots + +The command identifies the operations with the lowest scores and provides reasons such as: + +- "High parameter count (N)" +- "Deep schema nesting (depth M)" +- "Polymorphism (anyOf) without discriminator" +- "Missing request and response examples" +- "No structured error responses (4xx/5xx)" +- "Missing operation description" + +## Usage + +```bash +redocly score +redocly score [--format=