Skip to content

Publish a JSON Schema for coverage.json#1184

Open
sferik wants to merge 3 commits into
mainfrom
schema
Open

Publish a JSON Schema for coverage.json#1184
sferik wants to merge 3 commits into
mainfrom
schema

Conversation

@sferik
Copy link
Copy Markdown
Collaborator

@sferik sferik commented May 14, 2026

Since we want third-party tools to use the coverage.json (and not .resultset.json), we should encourage that by versioning the document and publishing a JSON Schema. This was suggested by @keithrbennett in #1143 (comment) and I agree that it’s a good idea.

This PR adds:

  • schemas/coverage.schema.json, which includes meta, total, per-file coverage, groups, and all four errors shapes. The gemspec has also been modified to add the schemas directory.
  • That file contains a meta.schema_version (currently "1.0"), which is intentionally independent of the gem version, so we can bump the gem version without bumping the schema version.
  • A spec that validates JSONFormatter output against the schema.
  • json_schemer as a development dependency to validate the schema.

This comment was marked as resolved.

@keithrbennett
Copy link
Copy Markdown

keithrbennett commented May 17, 2026

@sferik Thanks for doing this. Regarding versioned and unversioned schema files, I think the versioned files need to be primary/canonical and the 'latest' or 'current' version a convenience. I think the generated result set data file needs to contain a schema spec that includes the schema version so that it will always be usable when future schema versions are released. I may be misunderstanding though.

Here is a ChatGPT discussion of their recommended version strategy and implementation:

https://chatgpt.com/share/6a095730-57d0-8322-aac5-38f90035c2ac

(References in that chat to 'cov-loupe' are unintentional and should be 'simplecov' instead.)

Comment thread README.md
"groups": { "<group name>": { /* per-group stats + files */ } },
"errors": { /* minimum_coverage, minimum_coverage_by_file, minimum_coverage_by_group, maximum_coverage_drop violations */ }
}
```
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would be nice to get full example for some minimal 'Hello World' script (that covers all features).

Comment thread README.md Outdated
Comment thread schemas/coverage.schema.json Outdated
Comment thread README.md Outdated
Comment thread simplecov.gemspec
gem.required_ruby_version = ">= 3.1"

gem.files = Dir["lib/**/*.*", "exe/*", "LICENSE", "CHANGELOG.md", "README.md", "doc/*"]
gem.files = Dir["{lib,schemas}/**/*.*", "exe/*", "LICENSE", "CHANGELOG.md", "README.md", "doc/*"]
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While we are at it, maybe it is possible to generate human readable documentation with real world examples to close #1030 (comment) ?

There are a couple of documentation generators listed https://json-schema.org/tools#documentation but I haven't found a nice one yet.

@sferik
Copy link
Copy Markdown
Collaborator Author

sferik commented May 27, 2026

@keithrbennett I’ve incorporated your feedback in 590da07. Please have another look and let me know if you have any more feedback before I merge this in.

@sferik
Copy link
Copy Markdown
Collaborator Author

sferik commented May 27, 2026

@abitrolly I’ve incorporated your feedback in dcf5e59. Please have another look and let me know if you have any more feedback before I merge this in.

@sferik sferik force-pushed the schema branch 2 times, most recently from bf7b0a1 to 071579e Compare May 27, 2026 17:32
@sferik sferik force-pushed the main branch 2 times, most recently from fc484f1 to ad30894 Compare May 27, 2026 18:41
sferik added 3 commits May 27, 2026 11:41
…yload

Move the canonical schema to schemas/coverage-v1.0.schema.json with a
versioned $id, so each version is immutable. Keep
schemas/coverage.schema.json as a convenience alias for "the latest"
that mirrors the canonical except for $id, title, and description. A new
spec asserts the two stay structurally identical so the alias cannot
drift.

Add a top-level $schema field to every coverage.json holding the
versioned canonical URL, so each emitted document is self-describing and
consumers can resolve the exact contract without out-of-band knowledge.
meta.schema_version stays as the human-readable companion.
Draft-07 is from 2018. 2020-12 is the current published draft, supported
by every modern validator (including json_schemer in the dev Gemfile)
and by every IDE that auto-resolves $schema URLs. There is no
compatibility reason to ship a 2018-era contract as the public schema.

Update the meta-schema URI in both schema files (the versioned canonical
and the unversioned alias) and switch the spec assertion from
JSONSchemer.draft7 to JSONSchemer.draft202012. Retitle the README
section to "JSON Schema for coverage.json" so it is searchable for
someone looking for "JSON schema simplecov" rather than for the internal
filename, and tighten the opening paragraph (the second bullet was
redundant with the prose that followed).
@abitrolly
Copy link
Copy Markdown

@sferik looks good. Copying standard when a new version is released seems like a good way to preserve backward compatibility.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants