Open
Conversation
🦋 Changeset detectedLatest commit: a7a0f2a The changes in this PR will be included in the next version bump. This PR includes changesets to release 3 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Contributor
Contributor
|
RomanHotsiy
reviewed
Mar 12, 2026
- Add "AI" before "agent readiness" in changeset and docs for clarity - Replace <pre> block with fenced code block in score.md - Add security scheme coverage to metrics documentation - Remove resolveIfRef helper, replace with resolveNode that falls back to the original node when resolution fails - Refactor to use walkDocument visitor approach (matching stats command pattern) instead of manually iterating the document tree - Use resolveDocument + normalizeVisitors + walkDocument from openapi-core for proper $ref resolution and spec-format extensibility - Update index.test.ts to mock the new walk infrastructure Made-with: Cursor
adamaltman
force-pushed
the
aa/api-score
branch
from
79078d6 to
0e328aa
Compare
JLekawa
reviewed
Mar 12, 2026
Co-authored-by: Jacek Łękawa <164185257+JLekawa@users.noreply.github.com>
tatomyr
reviewed
Mar 12, 2026
Collaborator
tatomyr
left a comment
tatomyr
left a comment
There was a problem hiding this comment.
Left a couple of comments. I haven'd fully reviewed the scoring and collectors though as it takes time.
packages/cli/src/commands/score/__tests__/document-metrics.test.ts
Outdated
Show resolved
Hide resolved
tatomyr
requested changes
Mar 16, 2026
Collaborator
tatomyr
left a comment
tatomyr
left a comment
There was a problem hiding this comment.
I found some dead code. Please check if it's needed and remove if not. Also, I'm not sure what tests to review as many appear to only test that dead code, so let's handle that first.
packages/cli/src/commands/score/__tests__/document-metrics.test.ts
Outdated
Show resolved
Hide resolved
- Inline collect-metrics.ts test helper into document-metrics.test.ts - Use parseYaml as yaml directly instead of wrapper function - Remove default parameter from getStylishOutput in formatter tests - Use getMajorSpecVersion + exitWithError for spec version check - Add explicit case 'stylish' before default in format switch - Remove unsupported 'markdown' from score command format choices - Add comment explaining depth=-1 initialization - Clarify anyOf penalty and dependency terminology in docs - Update non-oas3 rejection test for exitWithError (throws) Made-with: Cursor
- Add type cast for parseYaml (returns unknown) in document-metrics tests - Inline collectDocumentMetrics helper into example-coverage tests - Remove collect-metrics.js import from scoring tests, use direct metrics - Add missing debugLogs property to accumulator mock in index tests Made-with: Cursor
adamaltman
commented
Mar 24, 2026
packages/cli/src/commands/score/__tests__/document-metrics.test.ts
Outdated
Show resolved
Hide resolved
Extract the metric-collection pipeline from handleScore into a standalone collect-metrics.ts module with two exports: - collectMetrics(): low-level function used by handleScore - collectDocumentMetrics(): high-level convenience used by tests This eliminates ~300 lines of duplicated walker setup across three test files (document-metrics, example-coverage, index) and ensures tests exercise the same code path as the production command. Add $ref-keyed memoization to walkSchema so repeated references to the same component schema return cached stats instead of re-walking. Stripe API: 37.6s → 11.3s (3.3× faster). Made-with: Cursor
Add median alongside averages for parameters, schema depth, polymorphism, and properties in the stylish output. Rename the misleading "Avg max schema depth" to "Schema depth". Made-with: Cursor
Rename workflowClarity to dependencyClarity, workflowDepths to dependencyDepths, and workflow-graph to dependency-graph to align code naming with the "Dependency Clarity" display label. Also adds discoverability subscore, recursive composition keyword stripping for accurate property counting, and updates e2e snapshots. Made-with: Cursor
tatomyr
reviewed
Mar 24, 2026
- Use isPlainObject from openapi-core for schema cycle detection instead of raw typeof checks that match arrays - Remove formatter unit tests in favor of e2e snapshot coverage - Clarify makeScores() purpose in hotspot tests Made-with: Cursor
tatomyr
reviewed
Mar 31, 2026
tatomyr
reviewed
Mar 31, 2026
tatomyr
reviewed
Mar 31, 2026
- Add experimental admonition to score command docs - Remove --config from usage examples - Remove redundant mockClear and process.exitCode in tests - Simplify JSON formatter by using spread instead of manual field mapping - Use isRef from openapi-core for $ref detection in collect-metrics - Use isPlainObject and isNotEmptyObject from openapi-core in document-metrics - Replace != null with Array.isArray for discriminatorRefs check - Remove unnecessary non-null assertions on discriminatorRefs - Lower branch coverage threshold to 70% per reviewer guidance Made-with: Cursor
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 4 potential issues.
Autofix Details
Bugbot Autofix prepared fixes for all 4 issues found in the latest run.
- ✅ Fixed: Discriminator flag lost with oneOf/anyOf polymorphism
- polyStats now sets hasDiscriminator when the parent schema has a discriminator (propertyName or non-empty mapping), merged with branch stats via OR.
- ✅ Fixed: Structured error count double-incremented per media type
- Added errorResponseStructuredCounted so structuredErrorResponseCount increases at most once per error response (description-only in Response.enter, or the first MediaType.enter that counts).
- ✅ Fixed: Independent max of correlated metrics across media types
- totalSchemaProperties, schemaPropertiesWithDescription, and constraintCount are now summed across media types instead of maxed independently.
- ✅ Fixed: Document parsed object mutated during metric collection
- collectMetrics now uses try/finally so restoreCompositionKeywords always runs after stripCompositionKeywords, even when inner work throws.
Or push these changes by commenting:
@cursor push b6568647da
Preview (b6568647da)
diff --git a/packages/cli/src/commands/score/collect-metrics.ts b/packages/cli/src/commands/score/collect-metrics.ts
--- a/packages/cli/src/commands/score/collect-metrics.ts
+++ b/packages/cli/src/commands/score/collect-metrics.ts
@@ -122,6 +122,30 @@
}: CollectMetricsOptions): CollectMetricsResult {
const removedComposition = stripCompositionKeywords(document.parsed);
+ try {
+ return collectMetricsInner({
+ document,
+ types,
+ resolvedRefMap,
+ ctx,
+ debugOperationId,
+ removedComposition,
+ });
+ } finally {
+ restoreCompositionKeywords(removedComposition);
+ }
+}
+
+function collectMetricsInner({
+ document,
+ types,
+ resolvedRefMap,
+ ctx,
+ debugOperationId,
+ removedComposition,
+}: CollectMetricsOptions & {
+ removedComposition: Map<object, StrippedComposition>;
+}): CollectMetricsResult {
const schemaWalkState = createSchemaWalkState();
const schemaVisitor = createSchemaMetricVisitor(schemaWalkState);
const normalizedSchemaVisitors = normalizeVisitors(
@@ -236,6 +260,10 @@
: null;
const hasDiscriminatorBranches =
Array.isArray(discriminatorRefs) && discriminatorRefs.length > 0;
+ const hasParentDiscriminator = !!(
+ disc?.propertyName ||
+ (isPlainObject(disc?.mapping) && Object.keys(disc.mapping).length > 0)
+ );
let result: SchemaStats;
@@ -258,6 +286,7 @@
...maxBranch,
polymorphismCount: maxBranch.polymorphismCount + polyBranches.length,
anyOfCount: maxBranch.anyOfCount + (polyKeyword === 'anyOf' ? polyBranches.length : 0),
+ hasDiscriminator: maxBranch.hasDiscriminator || hasParentDiscriminator,
};
}
@@ -323,8 +352,6 @@
ctx,
});
- restoreCompositionKeywords(removedComposition);
-
return {
metrics: getDocumentMetrics(accumulator),
debugLogs: accumulator.debugLogs,
diff --git a/packages/cli/src/commands/score/collectors/document-metrics.ts b/packages/cli/src/commands/score/collectors/document-metrics.ts
--- a/packages/cli/src/commands/score/collectors/document-metrics.ts
+++ b/packages/cli/src/commands/score/collectors/document-metrics.ts
@@ -212,6 +212,8 @@
inRequestBody: boolean;
inResponse: boolean;
currentResponseCode: string;
+ /** True once structured error counting ran for the current error response (Response or first MediaType). */
+ errorResponseStructuredCounted: boolean;
refsUsed: Set<string>;
}
@@ -278,6 +280,7 @@
inRequestBody: false,
inResponse: false,
currentResponseCode: '',
+ errorResponseStructuredCounted: false,
refsUsed: new Set(),
};
@@ -337,8 +340,10 @@
if (isErrorCode(code)) {
current.totalErrorResponses++;
+ current.errorResponseStructuredCounted = false;
if (!response.content && response.description) {
current.structuredErrorResponseCount++;
+ current.errorResponseStructuredCounted = true;
}
}
},
@@ -359,8 +364,13 @@
if (current.inResponse) current.responseExamplePresent = true;
}
- if (current.inResponse && isErrorCode(current.currentResponseCode)) {
+ if (
+ current.inResponse &&
+ isErrorCode(current.currentResponseCode) &&
+ !current.errorResponseStructuredCounted
+ ) {
current.structuredErrorResponseCount++;
+ current.errorResponseStructuredCounted = true;
}
if (mediaType.schema) {
@@ -372,15 +382,9 @@
const stats = accumulator.walkSchema(mediaType.schema, isDebugTarget);
current.propertyCount = Math.max(current.propertyCount, stats.propertyCount);
- current.totalSchemaProperties = Math.max(
- current.totalSchemaProperties,
- stats.totalSchemaProperties
- );
- current.schemaPropertiesWithDescription = Math.max(
- current.schemaPropertiesWithDescription,
- stats.schemaPropertiesWithDescription
- );
- current.constraintCount = Math.max(current.constraintCount, stats.constraintCount);
+ current.totalSchemaProperties += stats.totalSchemaProperties;
+ current.schemaPropertiesWithDescription += stats.schemaPropertiesWithDescription;
+ current.constraintCount += stats.constraintCount;
current.polymorphismCount = Math.max(current.polymorphismCount, stats.polymorphismCount);
current.anyOfCount = Math.max(current.anyOfCount, stats.anyOfCount);
if (stats.hasDiscriminator) current.hasDiscriminator = true;This Bugbot Autofix run was free. To enable autofix for future PRs, go to the Cursor dashboard.
- Fix duplicate import from @redocly/openapi-core (CI lint error) - Propagate hasDiscriminator flag for oneOf/anyOf + discriminator schemas - Prevent structured error response double-counting per media type - Keep totalSchemaProperties and schemaPropertiesWithDescription paired from the same schema to avoid misleading documentation quality ratios - Wrap composition stripping in try/finally to guarantee restoration - Export isMappingRef from openapi-core and use in collect-metrics - Rename bfsMaxDepth to computeLongestBfsPath for clarity Made-with: Cursor
- Add allOf member count to polymorphismCount for parity with oneOf/anyOf branch counting - Handle RFC 6901 JSON Pointer escaping (~0 → ~ and ~1 → /) in resolveJsonPointer - Remove unused scores parameter from getHotspotReasons Made-with: Cursor
tatomyr
reviewed
Apr 3, 2026
Collaborator
tatomyr
left a comment
tatomyr
left a comment
There was a problem hiding this comment.
These metrics look like a duplication:
Is it intended?
packages/cli/src/commands/score/__tests__/document-metrics.test.ts
Outdated
Show resolved
Hide resolved
Member
Author
|
Yes, I think there is too much overlap between the two scores so I'm going to consolidate it into one. It will make docs easier, result understanding easier, etc... Addressing other comments too. |
… single score Merge the two overlapping composite scores into a single "Agent Readiness" score with all 9 subscores. This eliminates duplication in weights, scoring functions, and output. Also addresses code review feedback: use unescapePointerFragment, simplify resetSchemaWalkState via Object.assign, consolidate SchemaStats into types.ts, remove collect() test wrapper, simplify hasExample return, and use spread for walkSchemaRaw return. Made-with: Cursor
- Remove global stripCompositionKeywords that silently lost nested oneOf/anyOf within properties/items — the parentOnly shallow copy already prevents double-counting at the top level while preserving nested composition for walkSchemaRaw to count naturally - Remove unused maxPropertiesGood threshold (dead code) - Extract duplicate median function to shared utils.ts Made-with: Cursor
Member
Author
|
Addressed all remaining review comments. The two scores have been consolidated into a single Agent Readiness score with all 9 subscores. Also removed the global |
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 2 potential issues.
Autofix Details
Bugbot Autofix prepared fixes for both issues found in the latest run.
- ✅ Fixed: Constraint clarity mixes metrics from different media types
- Media-type stats are now chosen together via a combined bundle score so constraintCount, totalSchemaProperties, and schemaPropertiesWithDescription always refer to the same walk.
- ✅ Fixed: Redundant
propertyCountandtotalSchemaPropertiesalways equal- Removed propertyCount from SchemaStats and OperationMetrics and use totalSchemaProperties everywhere, including polymorphic branch selection in collect-metrics.
Or push these changes by commenting:
@cursor push dfca434d5c
Preview (dfca434d5c)
diff --git a/packages/cli/src/commands/score/__tests__/dependency-graph.test.ts b/packages/cli/src/commands/score/__tests__/dependency-graph.test.ts
--- a/packages/cli/src/commands/score/__tests__/dependency-graph.test.ts
+++ b/packages/cli/src/commands/score/__tests__/dependency-graph.test.ts
@@ -15,7 +15,6 @@
polymorphismCount: 0,
anyOfCount: 0,
hasDiscriminator: false,
- propertyCount: 0,
operationDescriptionPresent: false,
schemaPropertiesWithDescription: 0,
totalSchemaProperties: 0,
diff --git a/packages/cli/src/commands/score/__tests__/document-metrics.test.ts b/packages/cli/src/commands/score/__tests__/document-metrics.test.ts
--- a/packages/cli/src/commands/score/__tests__/document-metrics.test.ts
+++ b/packages/cli/src/commands/score/__tests__/document-metrics.test.ts
@@ -131,7 +131,7 @@
const op = metrics.operations.get('createItem')!;
expect(op.requestBodyPresent).toBe(true);
expect(op.requestExamplePresent).toBe(true);
- expect(op.propertyCount).toBe(2);
+ expect(op.totalSchemaProperties).toBe(2);
expect(op.constraintCount).toBe(2);
expect(op.schemaPropertiesWithDescription).toBe(1);
});
diff --git a/packages/cli/src/commands/score/__tests__/hotspots.test.ts b/packages/cli/src/commands/score/__tests__/hotspots.test.ts
--- a/packages/cli/src/commands/score/__tests__/hotspots.test.ts
+++ b/packages/cli/src/commands/score/__tests__/hotspots.test.ts
@@ -16,7 +16,6 @@
polymorphismCount: 0,
anyOfCount: 0,
hasDiscriminator: false,
- propertyCount: 0,
operationDescriptionPresent: true,
schemaPropertiesWithDescription: 0,
totalSchemaProperties: 0,
diff --git a/packages/cli/src/commands/score/__tests__/index.test.ts b/packages/cli/src/commands/score/__tests__/index.test.ts
--- a/packages/cli/src/commands/score/__tests__/index.test.ts
+++ b/packages/cli/src/commands/score/__tests__/index.test.ts
@@ -51,7 +51,6 @@
polymorphismCount: 0,
anyOfCount: 0,
hasDiscriminator: false,
- propertyCount: 1,
operationDescriptionPresent: true,
schemaPropertiesWithDescription: 0,
totalSchemaProperties: 1,
diff --git a/packages/cli/src/commands/score/__tests__/scoring.test.ts b/packages/cli/src/commands/score/__tests__/scoring.test.ts
--- a/packages/cli/src/commands/score/__tests__/scoring.test.ts
+++ b/packages/cli/src/commands/score/__tests__/scoring.test.ts
@@ -23,7 +23,6 @@
polymorphismCount: 0,
anyOfCount: 0,
hasDiscriminator: false,
- propertyCount: 0,
operationDescriptionPresent: true,
schemaPropertiesWithDescription: 0,
totalSchemaProperties: 0,
@@ -132,7 +131,6 @@
parameterCount: 1,
requiredParameterCount: 1,
paramsWithDescription: 0,
- propertyCount: 1,
totalSchemaProperties: 1,
maxResponseSchemaDepth: 1,
}),
@@ -146,7 +144,6 @@
requestBodyPresent: true,
requestExamplePresent: true,
constraintCount: 1,
- propertyCount: 2,
totalSchemaProperties: 2,
maxRequestSchemaDepth: 1,
maxResponseSchemaDepth: 1,
diff --git a/packages/cli/src/commands/score/collect-metrics.ts b/packages/cli/src/commands/score/collect-metrics.ts
--- a/packages/cli/src/commands/score/collect-metrics.ts
+++ b/packages/cli/src/commands/score/collect-metrics.ts
@@ -87,7 +87,6 @@
polymorphismCount: 0,
anyOfCount: 0,
hasDiscriminator: false,
- propertyCount: 0,
totalSchemaProperties: 0,
schemaPropertiesWithDescription: 0,
constraintCount: 0,
@@ -102,7 +101,6 @@
polymorphismCount: a.polymorphismCount + b.polymorphismCount,
anyOfCount: a.anyOfCount + b.anyOfCount,
hasDiscriminator: a.hasDiscriminator || b.hasDiscriminator,
- propertyCount: a.propertyCount + b.propertyCount,
totalSchemaProperties: a.totalSchemaProperties + b.totalSchemaProperties,
schemaPropertiesWithDescription:
a.schemaPropertiesWithDescription + b.schemaPropertiesWithDescription,
@@ -172,7 +170,7 @@
let maxBranch = walkSchema(polyBranches[0], debug);
for (let i = 1; i < polyBranches.length; i++) {
const branchStats = walkSchema(polyBranches[i], debug);
- if (branchStats.propertyCount > maxBranch.propertyCount) {
+ if (branchStats.totalSchemaProperties > maxBranch.totalSchemaProperties) {
maxBranch = branchStats;
}
}
@@ -191,7 +189,7 @@
let maxBranch = walkSchema(discriminatorRefs[0], debug);
for (let i = 1; i < discriminatorRefs.length; i++) {
const branchStats = walkSchema(discriminatorRefs[i], debug);
- if (branchStats.propertyCount > maxBranch.propertyCount) {
+ if (branchStats.totalSchemaProperties > maxBranch.totalSchemaProperties) {
maxBranch = branchStats;
}
}
diff --git a/packages/cli/src/commands/score/collectors/document-metrics.ts b/packages/cli/src/commands/score/collectors/document-metrics.ts
--- a/packages/cli/src/commands/score/collectors/document-metrics.ts
+++ b/packages/cli/src/commands/score/collectors/document-metrics.ts
@@ -49,7 +49,6 @@
polymorphismCount: number;
anyOfCount: number;
hasDiscriminator: boolean;
- propertyCount: number;
totalSchemaProperties: number;
schemaPropertiesWithDescription: number;
constraintCount: number;
@@ -68,7 +67,6 @@
polymorphismCount: 0,
anyOfCount: 0,
hasDiscriminator: false,
- propertyCount: 0,
totalSchemaProperties: 0,
schemaPropertiesWithDescription: 0,
constraintCount: 0,
@@ -122,7 +120,6 @@
if (isPlainObject(schema.properties)) {
const props = Object.entries(schema.properties) as [string, any][];
state.totalSchemaProperties += props.length;
- state.propertyCount += props.length;
for (const [name, prop] of props) {
localPropertyNames.push(name);
@@ -166,7 +163,6 @@
maxRequestSchemaDepth: number;
maxResponseSchemaDepth: number;
- propertyCount: number;
totalSchemaProperties: number;
schemaPropertiesWithDescription: number;
constraintCount: number;
@@ -187,6 +183,9 @@
currentResponseCode: string;
errorStructuredCounted: boolean;
+ /** Best-scoring media type for constraint + property metrics (see `mediaTypeBundleScore`). */
+ mediaTypeBundleScore: number;
+
refsUsed: Set<string>;
}
@@ -233,7 +232,6 @@
maxRequestSchemaDepth: 0,
maxResponseSchemaDepth: 0,
- propertyCount: 0,
totalSchemaProperties: 0,
schemaPropertiesWithDescription: 0,
constraintCount: 0,
@@ -254,6 +252,8 @@
currentResponseCode: '',
errorStructuredCounted: false,
+ mediaTypeBundleScore: -1,
+
refsUsed: new Set(),
};
}
@@ -273,7 +273,6 @@
polymorphismCount: ctx.polymorphismCount,
anyOfCount: ctx.anyOfCount,
hasDiscriminator: ctx.hasDiscriminator,
- propertyCount: ctx.propertyCount,
operationDescriptionPresent: ctx.operationDescriptionPresent,
schemaPropertiesWithDescription: ctx.schemaPropertiesWithDescription,
totalSchemaProperties: ctx.totalSchemaProperties,
@@ -350,12 +349,13 @@
const stats = accumulator.walkSchema(mediaType.schema, isDebugTarget);
- current.propertyCount = Math.max(current.propertyCount, stats.propertyCount);
- if (stats.totalSchemaProperties > current.totalSchemaProperties) {
+ const bundleScore = mediaTypeBundleScore(stats);
+ if (bundleScore > current.mediaTypeBundleScore) {
+ current.mediaTypeBundleScore = bundleScore;
current.totalSchemaProperties = stats.totalSchemaProperties;
current.schemaPropertiesWithDescription = stats.schemaPropertiesWithDescription;
+ current.constraintCount = stats.constraintCount;
}
- current.constraintCount = Math.max(current.constraintCount, stats.constraintCount);
current.polymorphismCount = Math.max(current.polymorphismCount, stats.polymorphismCount);
current.anyOfCount = Math.max(current.anyOfCount, stats.anyOfCount);
if (stats.hasDiscriminator) current.hasDiscriminator = true;
@@ -388,7 +388,7 @@
accumulator.debugLogs.push({
context,
entries: stats.debugEntries,
- totalProperties: stats.propertyCount,
+ totalProperties: stats.totalSchemaProperties,
totalPolymorphism: stats.polymorphismCount,
totalConstraints: stats.constraintCount,
maxDepth: stats.maxDepth,
@@ -475,3 +475,13 @@
const num = parseInt(code, 10);
return num >= 400 && num < 600;
}
+
+/** Matches `propCount` in scoring.ts so operation-level rollup stays from one media type. */
+function propCountForScoring(totalSchemaProperties: number): number {
+ return totalSchemaProperties || 1;
+}
+
+function mediaTypeBundleScore(stats: SchemaStats): number {
+ const p = propCountForScoring(stats.totalSchemaProperties);
+ return stats.constraintCount / p + stats.schemaPropertiesWithDescription / p;
+}
diff --git a/packages/cli/src/commands/score/formatters/stylish.ts b/packages/cli/src/commands/score/formatters/stylish.ts
--- a/packages/cli/src/commands/score/formatters/stylish.ts
+++ b/packages/cli/src/commands/score/formatters/stylish.ts
@@ -85,7 +85,7 @@
params.push(m.parameterCount);
depths.push(Math.max(m.maxRequestSchemaDepth, m.maxResponseSchemaDepth));
polys.push(m.polymorphismCount);
- props.push(m.propertyCount);
+ props.push(m.totalSchemaProperties);
if (m.requestExamplePresent) opsWithReqExample++;
if (m.responseExamplePresent) opsWithResExample++;
if (m.operationDescriptionPresent) opsWithDescription++;
@@ -140,7 +140,7 @@
out(cyan(' ' + '─'.repeat(header.length - 2)));
const entries = [...result.rawMetrics.operations.entries()].sort(
- ([, a], [, b]) => b.propertyCount - a.propertyCount
+ ([, a], [, b]) => b.totalSchemaProperties - a.totalSchemaProperties
);
for (const [, m] of entries) {
@@ -149,7 +149,7 @@
const line =
' ' +
label.padEnd(50) +
- String(m.propertyCount).padStart(7) +
+ String(m.totalSchemaProperties).padStart(7) +
String(m.polymorphismCount).padStart(7) +
String(depth).padStart(7) +
String(m.parameterCount).padStart(8) +
diff --git a/packages/cli/src/commands/score/types.ts b/packages/cli/src/commands/score/types.ts
--- a/packages/cli/src/commands/score/types.ts
+++ b/packages/cli/src/commands/score/types.ts
@@ -15,7 +15,6 @@
polymorphismCount: number;
anyOfCount: number;
hasDiscriminator: boolean;
- propertyCount: number;
operationDescriptionPresent: boolean;
schemaPropertiesWithDescription: number;
@@ -125,7 +124,6 @@
polymorphismCount: number;
anyOfCount: number;
hasDiscriminator: boolean;
- propertyCount: number;
totalSchemaProperties: number;
schemaPropertiesWithDescription: number;
constraintCount: number;
diff --git a/tests/e2e/score/score-json/snapshot.txt b/tests/e2e/score/score-json/snapshot.txt
--- a/tests/e2e/score/score-json/snapshot.txt
+++ b/tests/e2e/score/score-json/snapshot.txt
@@ -4,8 +4,8 @@
"subscores": {
"parameterSimplicity": 0.8625,
"schemaSimplicity": 0.796875,
- "documentationQuality": 0.9017857142857143,
- "constraintClarity": 0.5833333333333333,
+ "documentationQuality": 0.8988095238095238,
+ "constraintClarity": 0.6,
"exampleCoverage": 0.875,
"errorClarity": 1,
"dependencyClarity": 0.7291666666666669,
@@ -29,7 +29,6 @@
"polymorphismCount": 0,
"anyOfCount": 0,
"hasDiscriminator": false,
- "propertyCount": 3,
"operationDescriptionPresent": true,
"schemaPropertiesWithDescription": 2,
"totalSchemaProperties": 3,
@@ -54,7 +53,6 @@
"polymorphismCount": 0,
"anyOfCount": 0,
"hasDiscriminator": false,
- "propertyCount": 6,
"operationDescriptionPresent": true,
"schemaPropertiesWithDescription": 6,
"totalSchemaProperties": 6,
@@ -79,7 +77,6 @@
"polymorphismCount": 0,
"anyOfCount": 0,
"hasDiscriminator": false,
- "propertyCount": 6,
"operationDescriptionPresent": true,
"schemaPropertiesWithDescription": 6,
"totalSchemaProperties": 6,
@@ -104,7 +101,6 @@
"polymorphismCount": 0,
"anyOfCount": 0,
"hasDiscriminator": false,
- "propertyCount": 6,
"operationDescriptionPresent": true,
"schemaPropertiesWithDescription": 6,
"totalSchemaProperties": 6,
@@ -129,7 +125,6 @@
"polymorphismCount": 0,
"anyOfCount": 0,
"hasDiscriminator": false,
- "propertyCount": 2,
"operationDescriptionPresent": true,
"schemaPropertiesWithDescription": 0,
"totalSchemaProperties": 2,
@@ -154,7 +149,6 @@
"polymorphismCount": 0,
"anyOfCount": 0,
"hasDiscriminator": false,
- "propertyCount": 6,
"operationDescriptionPresent": true,
"schemaPropertiesWithDescription": 6,
"totalSchemaProperties": 6,
@@ -179,10 +173,9 @@
"polymorphismCount": 0,
"anyOfCount": 0,
"hasDiscriminator": false,
- "propertyCount": 6,
"operationDescriptionPresent": true,
- "schemaPropertiesWithDescription": 5,
- "totalSchemaProperties": 6,
+ "schemaPropertiesWithDescription": 4,
+ "totalSchemaProperties": 5,
"constraintCount": 4,
"requestExamplePresent": true,
"responseExamplePresent": true,
@@ -204,7 +197,6 @@
"polymorphismCount": 0,
"anyOfCount": 0,
"hasDiscriminator": false,
- "propertyCount": 0,
"operationDescriptionPresent": true,
"schemaPropertiesWithDescription": 0,
"totalSchemaProperties": 0,
@@ -303,12 +295,12 @@
}
},
"buyMuseumTickets": {
- "agentReadiness": 88.8,
+ "agentReadiness": 90,
"subscores": {
"parameterSimplicity": 1,
"schemaSimplicity": 0.875,
- "documentationQuality": 0.8571428571428571,
- "constraintClarity": 0.6666666666666666,
+ "documentationQuality": 0.8333333333333334,
+ "constraintClarity": 0.8,
"exampleCoverage": 1,
"errorClarity": 1,
"dependencyClarity": 0.6666666666666667,
diff --git a/tests/e2e/score/score-stylish/snapshot.txt b/tests/e2e/score/score-stylish/snapshot.txt
--- a/tests/e2e/score/score-stylish/snapshot.txt
+++ b/tests/e2e/score/score-stylish/snapshot.txt
@@ -8,7 +8,7 @@
Parameter Simplicity [█████████████████░░░] 86%
Schema Simplicity [████████████████░░░░] 80%
Documentation Quality [██████████████████░░] 90%
- Constraint Clarity [████████████░░░░░░░░] 58%
+ Constraint Clarity [████████████░░░░░░░░] 60%
Example Coverage [██████████████████░░] 88%
Error Clarity [████████████████████] 100%
Dependency Clarity [███████████████░░░░░] 73%
@@ -22,7 +22,7 @@
Parameters/operation: avg 1.4 median 1.0 min 0 max 4
Schema depth: avg 1.6 median 2.0 min 0 max 3
Polymorphism/operation: avg 0.0 median 0.0 min 0 max 0
- Properties/operation: avg 4.4 median 6.0 min 0 max 6
+ Properties/operation: avg 4.3 median 5.5 min 0 max 6
Operations with request examples: 3/8 (38%)
Operations with response examples: 7/8 (88%)
Operations with description: 8/8 (100%)This Bugbot Autofix run was free. To enable autofix for future PRs, go to the Cursor dashboard.
- Remove as any casts from normalizeVisitors — types infer correctly - Remove redundant propertyCount (always equals totalSchemaProperties) - Fix constraintCount mixing across media types — keep all three correlated metrics from the same schema walk - Consolidate CurrentOperationContext with OperationMetrics field names, simplify buildOperationMetrics to destructuring spread - Export Oas3MediaType from openapi-core, use in hasExample signature - Remove any type on MediaType.enter — visitor infers type by default - Move collectDocumentMetrics to __tests__/collect-metrics-helper.ts since it is test-only code Made-with: Cursor
Member
Author
|
Addressed all review comments in fdb8c94:
|
| result.polymorphismClarity /= n; | ||
|
|
||
| return result; | ||
| } |
There was a problem hiding this comment.
Document subscores and composite score use inconsistent aggregation
Medium Severity
aggregateSubscores computes the arithmetic mean of per-operation subscores, while computeDocumentScores computes the median of per-operation composite scores. Because median-of-weighted-sums ≠ weighted-sum-of-means, applying computeAgentReadiness to the displayed aggregate subscores yields a different number than the displayed agentReadiness. The subscores shown to the user don't actually explain the composite score, which is misleading.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit fdb8c94. Configure here.
| current.totalSchemaProperties = stats.totalSchemaProperties; | ||
| current.schemaPropertiesWithDescription = stats.schemaPropertiesWithDescription; | ||
| current.constraintCount = stats.constraintCount; | ||
| } |
There was a problem hiding this comment.
Max-wins schema replacement discards smaller schema quality metrics
Medium Severity
When multiple media types are processed for an operation, schemaPropertiesWithDescription and constraintCount are replaced atomically based on which schema has the highest totalSchemaProperties. If a response schema has more properties but fewer descriptions/constraints than the request body schema, the request body's quality metrics are silently discarded. This causes documentationQuality and constraintClarity subscores to be inaccurately low for operations where the request schema is better documented than the response schema.
Reviewed by Cursor Bugbot for commit fdb8c94. Configure here.
tatomyr
approved these changes
Apr 8, 2026
Collaborator
tatomyr
left a comment
tatomyr
left a comment
There was a problem hiding this comment.
LGTM. There are a few bugbot comments left -- please review whether they make sense.
- Preserve top-level $ref in refsUsed when composition branch is taken, so computeDependencyDepths correctly tracks cross-operation schema sharing for oneOf/anyOf/allOf/discriminator schemas - Return constraintClarity=1 when totalSchemaProperties is 0, matching errorClarity's "no data" pattern — avoids unfairly penalizing DELETE and binary-returning endpoints that have no schema properties Made-with: Cursor
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 2 potential issues.
There are 4 total unresolved issues (including 2 from previous reviews).
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit a7a0f2a. Configure here.
| prop && '$ref' in prop && prop.$ref ? (ctx.resolve(prop)?.node ?? prop) : prop; | ||
| if (res?.description) state.schemaPropertiesWithDescription++; | ||
| if (res?.example !== undefined || res?.examples) state.hasPropertyExamples = true; | ||
| if (!res?.readOnly) state.writableTopLevelFields++; |
There was a problem hiding this comment.
Writable field counter ignores depth, miscounts properties
Medium Severity
The writableTopLevelFields counter in createSchemaMetricVisitor increments for every non-readOnly property at every nesting depth, not just at depth 0. Since Schema.enter is called recursively for nested schemas, a schema like { name: string, address: { street: string, city: string } } would report 3 writable "top-level" fields instead of 2. This incorrect value flows through SchemaStats.writableTopLevelFields → OperationMetrics.topLevelWritableFieldCount and appears in the JSON output, giving users misleading data.
Reviewed by Cursor Bugbot for commit a7a0f2a. Configure here.
| result.polymorphismClarity /= n; | ||
|
|
||
| return result; | ||
| } |
There was a problem hiding this comment.
Empty document shows perfect score with zero subscores
Low Severity
When an API has zero operations, computeDocumentScores returns agentReadiness: 100, but aggregateSubscores returns all-zero subscores (dividing zero sums by 1). The stylish output displays a perfect 100/100 Agent Readiness alongside every subscore bar at 0%, which is contradictory and confusing to users.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit a7a0f2a. Configure here.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What/Why/How?
What: Adds a new score command to Redocly CLI that analyzes OpenAPI 3.x descriptions and produces two composite scores: Integration Simplicity (0-100, how easy is this API to integrate) and Agent Readiness (0-100, how usable is this API by AI agents/LLM tooling).
Why: API producers currently lack a quick, deterministic way to assess how developer-friendly or AI-agent-friendly their API descriptions are. The existing stats command counts structural elements but doesn't evaluate quality signals like documentation coverage, example presence, schema complexity, or error response structure. This command fills that gap with actionable, explainable scores.
How: The implementation follows the same pattern as the
statscommand (bundle+analyze), with a clean separation between metric collection and score calculation:collectors/): Walks the bundled document, resolving internal $refs, to gather per-operation raw metrics (parameter counts, schema depth, polymorphism, description/constraint/example coverage, structured error responses, workflow dependency depth via shared schema refs).scoring.ts): Pure functions that normalize raw metrics into subscores and compute weighted composite scores. Thresholds and weights are configurable constants. anyOf is penalized more heavily than oneOf/allOf; discriminator presence improves the agent readiness polymorphism clarity subscore.hotspots.ts): Identifies the operations with the most issues, sorted by number of reasons, with human-readable explanations.--format=stylish(default, with color bar charts) and--format=json(machine-readable for CI/dashboards).Reference
Related to API governance and developer experience tooling. No existing issue -- this is a new feature.
Testing
$refresolution, polymorphism counting (oneOf/anyOf/allOf), constraint detection (including const), example coverage scoring,anyOfpenalty multiplier,discriminatorimpact on agent readiness, deterministic output, and score range validation.tsc --noEmit), all existing tests continue to pass.Screenshots (optional)
Stylish output for Redocly Cafe:
Check yourself
Security
Note
Medium Risk
Adds a new CLI command with non-trivial OpenAPI traversal/scoring logic and a small core export change (
isMappingRef,Oas3MediaType), which could affect downstream consumers and output stability, though it’s mostly additive.Overview
Adds a new experimental
redocly scoreCLI command that bundles an OpenAPI 3.x description, collects per-operation complexity/documentation/example/error metrics, computes a composite Agent Readiness score (plus normalized subscores), and reports top “hotspot” operations with reasons. Output supports--format=stylish(default) and--format=json, plus optional--operation-detailsand--debug-operation-idschema breakdowns.Implements new metric collection/walking, dependency-depth inference via shared
$refs, scoring/weighting constants, hotspot selection, and dedicated formatters underpackages/cli/src/commands/score. Updates CLI wiring, docs/sidebars, adds unit + e2e snapshot coverage for the new command, exportsisMappingRef/Oas3MediaTypefrom@redocly/openapi-core, and slightly lowers the global branch coverage threshold.Reviewed by Cursor Bugbot for commit a7a0f2a. Bugbot is set up for automated code reviews on this repo. Configure here.