Skip to content

Commit fcb3718

Browse files
Add conformance.toml manifest and CI guard (#77)
* Add conformance.toml manifest and CI guard Records which openarmature-spec proposals are implemented in this package and at which release version. The openarmature-spec docs build fetches this file from raw.githubusercontent.com to render a per-implementation status column on the proposals index page. scripts/check_conformance_manifest.py validates the file against the pinned spec submodule's proposals/ on every PR and release build, so a stale entry can't quietly serve wrong data to the docs site. Seeded from CHANGELOG: 23 implemented, 3 textual-only (0019, 0026, 0030). Proposal 0031 is intentionally absent: it was accepted on spec main after the v0.9.0 cut and isn't in the pinned submodule. * Tighten conformance guard input validation Two structured errors instead of AttributeError / TypeError tracebacks when conformance.toml has a malformed shape: - Entry that's a scalar instead of a table (e.g., accidentally writing `proposals."0001" = "oops"` at the top level). - `since` value that's an int because the version was written without quotes (e.g., `since = 1` instead of `since = "1.0.0"`). Both are plausible hand-edit typos. The guard now emits the script's standard ::error::-style messages for either, matching the rest of the validator's failure output. Addresses CoPilot PR review feedback on #77.
1 parent f1fa5c1 commit fcb3718

5 files changed

Lines changed: 328 additions & 0 deletions

File tree

.github/workflows/ci.yml

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -57,6 +57,14 @@ jobs:
5757
# ``tests/unit/test_observability_otel.py``.
5858
run: uv sync --frozen --all-extras --group examples
5959

60+
- name: Validate conformance.toml against pinned spec
61+
# Catches drift between conformance.toml and the pinned spec
62+
# submodule's proposals/. External readers (notably the
63+
# openarmature-spec docs build) fetch this manifest to surface
64+
# python's impl status on the proposals index page, so a stale
65+
# entry quietly serves wrong data to the docs site.
66+
run: uv run python scripts/check_conformance_manifest.py
67+
6068
- name: Lint (ruff check)
6169
run: uv run ruff check .
6270

.github/workflows/release.yml

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -84,6 +84,13 @@ jobs:
8484
)
8585
PY
8686
87+
- name: Validate conformance.toml against pinned spec
88+
# Mirrors the equivalent step in ci.yml. Catches drift between
89+
# conformance.toml and the pinned spec submodule's proposals/
90+
# at release time as well as PR time, so a tag push that
91+
# bypassed PR review still fails before publishing.
92+
run: uv run python scripts/check_conformance_manifest.py
93+
8794
- name: Lint (ruff check)
8895
run: uv run ruff check .
8996

RELEASING.md

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,14 @@ changelog entry.
4040
release that changed user-visible behavior is reflected in the
4141
upcoming version's section. The date matches the day the rc tag
4242
is pushed (refresh it again at the real-release step).
43+
- [ ] **`conformance.toml` is current.** Any proposal whose impl
44+
landed in this cycle has its `[proposals."NNNN"]` entry — either
45+
newly added (set `since` to the version about to ship) or
46+
adjusted (e.g., `not-yet``implemented`). If the pinned spec
47+
submodule was bumped, also bump `[manifest].spec_pin` to the new
48+
tag. The CI guard at `scripts/check_conformance_manifest.py`
49+
enforces structural consistency, but it can't check that
50+
semantic status reflects reality — read the diff manually.
4351
- [ ] **Docs sweep for stale references.** For each behavior change in
4452
the upcoming release, grep the docs for the old wording, file
4553
paths, and flag descriptions; reconcile in the same PR as the

conformance.toml

Lines changed: 151 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,151 @@
1+
# Conformance manifest for openarmature-python.
2+
#
3+
# Records which openarmature-spec proposals are implemented in this
4+
# package, and starting at which release version. Intended to be read
5+
# by external consumers (notably the openarmature-spec docs build,
6+
# which surfaces a per-implementation status column on the proposals
7+
# index page).
8+
#
9+
# Stable URL (read-only, fetched at docs-build time):
10+
# https://raw.githubusercontent.com/LunarCommand/openarmature-python/main/conformance.toml
11+
#
12+
# Maintenance: keep in sync with CHANGELOG.md. The CI guard at
13+
# scripts/check_conformance_manifest.py validates this file against the
14+
# pinned spec submodule's proposals/ directory on every PR and release
15+
# build; any Accepted proposal lacking an entry (or any entry pointing
16+
# at a non-existent / non-Accepted proposal) fails the build.
17+
#
18+
# Scope: this file lists ONLY proposals visible in the pinned spec
19+
# submodule (openarmature-spec at the SHA pinned by this repo's
20+
# submodule). Proposals accepted on the spec's main branch after this
21+
# repo's last spec bump are intentionally absent — surfacing the gap
22+
# between pinned-spec and spec-head is the consumer's job (e.g., the
23+
# spec docs site computes the difference and renders accordingly).
24+
#
25+
# Convention: this file is only updated as part of release PRs. Between
26+
# releases, the manifest reflects the most-recently-published version
27+
# so external readers never see a `since` referring to an unreleased
28+
# pre-tag commit.
29+
30+
[manifest]
31+
implementation = "openarmature-python"
32+
spec_pin = "v0.22.1"
33+
34+
# Status values:
35+
# implemented — shipped behavior matches the proposal's contract
36+
# partial — partial impl; consult `note` for what's missing
37+
# textual-only — accepted proposal is purely textual (reframe,
38+
# clarification, template) with no module-level
39+
# change required; CHANGELOG note explains why
40+
# not-yet — accepted in spec, not yet shipped in this package
41+
#
42+
# Drafts and Superseded proposals are deliberately absent from this
43+
# file. The CI guard requires entries only for proposals whose spec
44+
# header reads `Status: Accepted`.
45+
46+
[proposals."0001"]
47+
status = "implemented"
48+
since = "0.5.0"
49+
50+
[proposals."0002"]
51+
status = "implemented"
52+
since = "0.5.0"
53+
54+
[proposals."0003"]
55+
status = "implemented"
56+
since = "0.5.0"
57+
58+
[proposals."0004"]
59+
status = "implemented"
60+
since = "0.5.0"
61+
62+
[proposals."0005"]
63+
status = "implemented"
64+
since = "0.5.0"
65+
66+
[proposals."0006"]
67+
status = "implemented"
68+
since = "0.5.0"
69+
70+
[proposals."0007"]
71+
status = "implemented"
72+
since = "0.5.0"
73+
74+
[proposals."0008"]
75+
status = "implemented"
76+
since = "0.5.0"
77+
78+
[proposals."0009"]
79+
status = "implemented"
80+
since = "0.9.0"
81+
82+
[proposals."0010"]
83+
status = "implemented"
84+
since = "0.9.0"
85+
86+
[proposals."0011"]
87+
status = "implemented"
88+
since = "0.6.0"
89+
90+
[proposals."0012"]
91+
status = "implemented"
92+
since = "0.5.0"
93+
94+
[proposals."0013"]
95+
status = "implemented"
96+
since = "0.5.0"
97+
98+
[proposals."0014"]
99+
status = "implemented"
100+
since = "0.6.0"
101+
102+
[proposals."0015"]
103+
status = "implemented"
104+
since = "0.6.0"
105+
106+
[proposals."0016"]
107+
status = "implemented"
108+
since = "0.6.0"
109+
110+
[proposals."0017"]
111+
status = "implemented"
112+
since = "0.6.0"
113+
114+
[proposals."0018"]
115+
status = "implemented"
116+
since = "0.6.0"
117+
118+
[proposals."0019"]
119+
status = "textual-only"
120+
since = "0.9.0"
121+
note = "Purely textual reframe of llm-provider §8 as a catalog of wire-format mappings (OpenAI-compatible body nested under §8.1). No module-level change required."
122+
123+
[proposals."0024"]
124+
status = "implemented"
125+
since = "0.8.0"
126+
127+
[proposals."0025"]
128+
status = "implemented"
129+
since = "0.9.0"
130+
131+
[proposals."0026"]
132+
status = "textual-only"
133+
since = "0.9.0"
134+
note = "Purely textual §8 framing paragraph; the existing OpenAI §8.1 mapping is the template's reference shape, so no module-level work was needed."
135+
136+
[proposals."0027"]
137+
status = "implemented"
138+
since = "0.9.0"
139+
140+
[proposals."0028"]
141+
status = "implemented"
142+
since = "0.9.0"
143+
144+
[proposals."0029"]
145+
status = "implemented"
146+
since = "0.9.0"
147+
148+
[proposals."0030"]
149+
status = "textual-only"
150+
since = "0.9.0"
151+
note = "Drain snapshot semantic and timeout-input validation already implemented as part of the proposal 0010 impl PR (v0.9.0); no additional module-level work needed."
Lines changed: 154 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,154 @@
1+
"""Validate conformance.toml against the pinned spec submodule's proposals.
2+
3+
Failure modes caught:
4+
5+
- Accepted proposal in the spec has no entry in conformance.toml.
6+
- Entry in conformance.toml refers to a proposal that doesn't exist in
7+
the spec, or refers to a proposal whose Status is not "Accepted"
8+
(e.g., Draft / Superseded — those are deliberately excluded from the
9+
manifest so the docs site doesn't claim impl status for unsettled
10+
proposals).
11+
- Entry has an unknown `status` value or a malformed `since` version.
12+
13+
Read-only; intended for CI. Non-zero exit on any failure with a
14+
human-readable diff. Runs under the repo's stdlib Python (>=3.12, so
15+
`tomllib` is always available).
16+
"""
17+
18+
from __future__ import annotations
19+
20+
import re
21+
import sys
22+
import tomllib
23+
from pathlib import Path
24+
from typing import Any, cast
25+
26+
REPO_ROOT = Path(__file__).resolve().parent.parent
27+
MANIFEST_PATH = REPO_ROOT / "conformance.toml"
28+
PROPOSALS_DIR = REPO_ROOT / "openarmature-spec" / "proposals"
29+
30+
ALLOWED_STATUSES = frozenset({"implemented", "partial", "textual-only", "not-yet"})
31+
PROPOSAL_FILENAME_RE = re.compile(r"^(\d{4})-[a-z0-9-]+\.md$")
32+
STATUS_LINE_RE = re.compile(r"^- \*\*Status:\*\*\s*(.+?)\s*$", re.MULTILINE)
33+
SINCE_RE = re.compile(r"^\d+\.\d+\.\d+$")
34+
35+
36+
def parse_spec_proposals() -> dict[str, str]:
37+
# Returns {proposal_id: status} for every proposal markdown file in
38+
# the pinned spec submodule. proposal_id is the 4-digit string used
39+
# as the manifest key; status is the literal value from the file's
40+
# `- **Status:** ...` header line.
41+
if not PROPOSALS_DIR.is_dir():
42+
sys.exit(
43+
f"::error::proposals dir not found at {PROPOSALS_DIR} — "
44+
"is the openarmature-spec submodule checked out?"
45+
)
46+
47+
result: dict[str, str] = {}
48+
for path in sorted(PROPOSALS_DIR.iterdir()):
49+
m = PROPOSAL_FILENAME_RE.match(path.name)
50+
if not m:
51+
continue
52+
proposal_id = m.group(1)
53+
text = path.read_text(encoding="utf-8")
54+
status_match = STATUS_LINE_RE.search(text)
55+
if not status_match:
56+
sys.exit(f"::error::proposal {proposal_id} ({path.name}) has no `- **Status:** ...` header line")
57+
result[proposal_id] = status_match.group(1).strip()
58+
return result
59+
60+
61+
def load_manifest() -> dict[str, Any]:
62+
# Returns {proposal_id: entry} for every [proposals."NNNN"] section
63+
# in conformance.toml. Each `entry` is typed as Any rather than
64+
# dict because TOML lets a user accidentally write a scalar value
65+
# under [proposals]; the caller validates the type per-entry and
66+
# emits a structured error on shape drift.
67+
if not MANIFEST_PATH.is_file():
68+
sys.exit(f"::error::manifest not found at {MANIFEST_PATH}")
69+
70+
with MANIFEST_PATH.open("rb") as f:
71+
data = tomllib.load(f)
72+
73+
proposals = data.get("proposals", {})
74+
if not isinstance(proposals, dict):
75+
sys.exit("::error::conformance.toml [proposals] table malformed")
76+
return cast(dict[str, Any], proposals)
77+
78+
79+
def main() -> int:
80+
spec = parse_spec_proposals()
81+
manifest = load_manifest()
82+
83+
accepted_ids = {pid for pid, status in spec.items() if status == "Accepted"}
84+
manifest_ids = set(manifest.keys())
85+
86+
errors: list[str] = []
87+
88+
missing = sorted(accepted_ids - manifest_ids)
89+
for pid in missing:
90+
errors.append(f"Accepted spec proposal {pid} has no entry in conformance.toml")
91+
92+
extra = sorted(manifest_ids - accepted_ids)
93+
for pid in extra:
94+
if pid not in spec:
95+
errors.append(
96+
f"conformance.toml entry {pid} refers to a proposal that "
97+
f"doesn't exist in openarmature-spec/proposals/"
98+
)
99+
else:
100+
errors.append(
101+
f"conformance.toml entry {pid} refers to a proposal whose "
102+
f"spec Status is {spec[pid]!r}, not 'Accepted' — "
103+
f"drafts and superseded proposals should be omitted"
104+
)
105+
106+
for pid in sorted(manifest_ids):
107+
raw_entry = manifest[pid]
108+
if not isinstance(raw_entry, dict):
109+
errors.append(
110+
f"conformance.toml entry {pid} is not a table "
111+
f'(got {type(raw_entry).__name__}); check the [proposals."{pid}"] '
112+
f"section is a table, not a scalar"
113+
)
114+
continue
115+
entry = cast(dict[str, Any], raw_entry)
116+
status = entry.get("status")
117+
if status not in ALLOWED_STATUSES:
118+
errors.append(
119+
f"conformance.toml entry {pid} has unknown status {status!r} "
120+
f"(allowed: {sorted(ALLOWED_STATUSES)})"
121+
)
122+
# `since` is required for every status except `not-yet`.
123+
since = entry.get("since")
124+
if status == "not-yet":
125+
if since is not None:
126+
errors.append(
127+
f"conformance.toml entry {pid} has status=not-yet but "
128+
f"also a `since` field — drop `since` for not-yet entries"
129+
)
130+
else:
131+
if since is None:
132+
errors.append(f"conformance.toml entry {pid} has status={status!r} but no `since` field")
133+
elif not isinstance(since, str):
134+
errors.append(
135+
f"conformance.toml entry {pid} `since` value {since!r} is not a string "
136+
f"(got {type(since).__name__}); did you forget to quote it?"
137+
)
138+
elif not SINCE_RE.match(since):
139+
errors.append(
140+
f"conformance.toml entry {pid} `since` value {since!r} is not in MAJOR.MINOR.PATCH form"
141+
)
142+
143+
if errors:
144+
print("conformance.toml validation failed:", file=sys.stderr)
145+
for err in errors:
146+
print(f" - {err}", file=sys.stderr)
147+
return 1
148+
149+
print(f"OK: {len(accepted_ids)} accepted proposals, {len(manifest_ids)} manifest entries, all consistent")
150+
return 0
151+
152+
153+
if __name__ == "__main__":
154+
raise SystemExit(main())

0 commit comments

Comments
 (0)