|
| 1 | +// SPDX-License-Identifier: CC-BY-SA-4.0 |
| 2 | += Flagship Rubric — the measuring-stick for hyperpolymath Julia libraries |
| 3 | +Jonathan D.A. Jewell <j.d.a.jewell@open.ac.uk> |
| 4 | +:revdate: 2026-07-01 |
| 5 | +:toc: left |
| 6 | +:toclevels: 3 |
| 7 | + |
| 8 | +The gradable standard every estate Julia library is measured against. |
| 9 | +`Axiom.jl` is the reference instance (the flagship), defined as 100% on this |
| 10 | +rubric; every other estate Julia library is scored relative to it. The |
| 11 | +machine-checkable subset of this rubric is what `just quality` runs. |
| 12 | + |
| 13 | +[.lead] |
| 14 | +Design principle: *a claim counts only if a tool can be run to confirm it.* |
| 15 | +Doctrine — ground-truth by running the tool, not trusting a status doc; report |
| 16 | +faithfully, no overclaim; holes before features. |
| 17 | + |
| 18 | +== Grades |
| 19 | + |
| 20 | +Adapted from the Component Readiness Grade (CRG) ladder, specialised for a |
| 21 | +registered Julia library. |
| 22 | + |
| 23 | +[cols="1,2,4"] |
| 24 | +|=== |
| 25 | +| Grade | Name | Bar |
| 26 | + |
| 27 | +| D | Alpha | Loads and runs: `using Pkg; Pkg.test()` completes without erroring |
| 28 | +on the LTS floor. At least one *meaningful* assertion (not `@test true`). |
| 29 | + |
| 30 | +| C | Beta | Correct on representative input; `Aqua.test_all` passes; closed |
| 31 | +`[compat]`; README.adoc + a building Documenter site. |
| 32 | + |
| 33 | +| B | RC | Edge cases + multiple input types covered; scoped `JET.test_package` |
| 34 | +returns zero package-origin reports; doctests pass; `panic-attack assail` = 0 |
| 35 | +unsuppressed critical; registry name↔UUID/repo-slug agree. |
| 36 | + |
| 37 | +| A | Production | All of B green in CI across `[LTS, stable]`; property/fuzz |
| 38 | +tests where the domain admits them; maintenance axes documented; SPDX + licence |
| 39 | +correct; A2ML dogfood gate + estate-rules green. |
| 40 | + |
| 41 | +| ★ | Flagship | All of A, *plus*: verification claims formally honest (no |
| 42 | +overclaimed proofs; certificates truthfully labelled; FFI residuals audited via |
| 43 | +`assail-classifications.a2ml`); zero-warning build; and it is itself a |
| 44 | +*reference* others are measured against. `Axiom.jl` holds ★. |
| 45 | +|=== |
| 46 | + |
| 47 | +An instance's grade is the highest ladder rung for which *every* row below at or |
| 48 | +below that rung passes. One failing B-row caps you at C regardless of A-rows. |
| 49 | + |
| 50 | +== The scored dimensions |
| 51 | + |
| 52 | +Each dimension lists its machine check (what `just quality` runs) and the flagship |
| 53 | +bar. `[auto]` = enforced by `just quality`; `[review]` = human/agent-judged |
| 54 | +(e.g. licence classification — never automated, per doctrine #6). |
| 55 | + |
| 56 | +=== 1. Correctness & tests `[auto]` |
| 57 | +* `julia --project=. -e 'using Pkg; Pkg.instantiate(); Pkg.build(); Pkg.precompile(); Pkg.test()'` green. |
| 58 | +* Assertions are behavioural, not tautological. Grep guard: fail if the only |
| 59 | + assertions are `@test true` / type-only `@test x isa T` (mirrors panic-attack |
| 60 | + MutationGap PA025 "all-type-only assertions"). |
| 61 | +* Flagship bar: green on LTS *and* stable in CI; nightly allow-fail present. |
| 62 | + |
| 63 | +=== 2. Static quality — Aqua `[auto]` |
| 64 | +* `Aqua.test_all(<Pkg>)` passes with NO overrides. |
| 65 | +* An override is a finding unless it carries an inline written rationale for a |
| 66 | + genuine false positive. Flagship bar: zero overrides (fix source, as Axiom did |
| 67 | + for the normalization unbound-typeparam bug + the stale-JSON3 dep). |
| 68 | + |
| 69 | +=== 3. Static quality — JET `[auto]` |
| 70 | +* `JET.test_package(<Pkg>; target_modules = (<Pkg>,))` returns zero |
| 71 | + package-origin reports. |
| 72 | +* Scoping uses JET's native `target_modules` (not a file-path hack), and a |
| 73 | + written note explains any borderline candidate as non-package-origin. |
| 74 | +* Flagship bar: zero, with the honesty note ("count is 0 today, not an |
| 75 | + allowlist"). |
| 76 | + |
| 77 | +=== 4. Compat closure & registry `[auto]` |
| 78 | +* Every `[deps]`/`[weakdeps]` entry + stdlibs + `julia` bounded in `[compat]`. |
| 79 | +* `name`/`uuid`/`repo` agree across `Project.toml`, the registry `Package.toml`, |
| 80 | + the `Registry.toml` line, and the GitHub slug (registry `ci/validate.jl`). |
| 81 | +* Flagship bar: AutoMerge-eligible; upper-bounded compat everywhere (registry |
| 82 | + warns on lower-bound-only). |
| 83 | + |
| 84 | +=== 5. Documentation `[auto]` + `[review]` |
| 85 | +* `julia --project=docs docs/make.jl` builds; `doctest = true` passes `[auto]`. |
| 86 | +* `README.adoc` (AsciiDoc, estate default) + `EXPLAINME.adoc` present `[auto]`. |
| 87 | +* API-doc coverage of exported symbols; no dangling doc references `[review]`. |
| 88 | +* Flagship bar: doctests green in CI; docs deploy on main. |
| 89 | + |
| 90 | +=== 6. Security & soundness `[auto]` |
| 91 | +* `panic-attack assail .` = 0 *unsuppressed* critical. |
| 92 | +* FFI/`unsafe` residuals are audited via `audits/assail-classifications.a2ml` |
| 93 | + (TOML-A2ML conformant) + a companion `audits/audit-*.adoc` — suppressions are |
| 94 | + non-gameable (registry lives separately from the code under scan). |
| 95 | +* No MD5/SHA-1 for security (SHA-256+); HTTPS-only; no hardcoded secrets; |
| 96 | + SHA-pinned CI actions. |
| 97 | +* Flagship bar: every suppression cites a current audit; secret-scanner + |
| 98 | + static-analysis-gate green. |
| 99 | + |
| 100 | +=== 7. Verification honesty `[review]` + partial `[auto]` |
| 101 | +* No overclaimed proofs: a `@prove`/proof-import path labelled as formal must be |
| 102 | + formal; heuristics labelled as heuristics (Axiom G03/G04). |
| 103 | +* Certificates truthfully labelled (no fake "authenticated" without a real MAC/ |
| 104 | + signature; Axiom G01). |
| 105 | +* Proof-drift honest: proof escape hatches visible, not hidden (panic-attack |
| 106 | + ProofDrift PA021 left un-suppressed unless audited). |
| 107 | +* Flagship bar: verification claims survive an adversarial re-read; no |
| 108 | + soundness hole outranked by a feature. |
| 109 | + |
| 110 | +=== 8. Estate compliance `[auto]` + `[review]` |
| 111 | +* A2ML substrate present and *valid TOML-A2ML* — dogfood gate `Validate A2ML |
| 112 | + manifests` green (every `.a2ml` self-identifies; even a panic-attack |
| 113 | + S-expression registry is TOML-A2ML-wrapped, per Axiom PR #52). |
| 114 | +* `estate-rules` green: no Python; AsciiDoc-by-default under `docs/` (except |
| 115 | + `docs/wiki/` + `docs/src/`); no V-lang (Zig is allowed and encouraged). |
| 116 | +* Licence correct for the repo's classification `[review]` — MPL-2.0 sole-owner |
| 117 | + default; NEVER an automated licence edit. |
| 118 | +* SPDX headers on all source files (authored from birth, never bulk-swept). |
| 119 | + |
| 120 | +=== 9. Maintenance & lifecycle `[review]` |
| 121 | +* Corrective / adaptive / perfective / preventive maintenance axes documented |
| 122 | + (`.machine_readable/policies/MAINTENANCE-AXES.a2ml`) with a checklist. |
| 123 | +* `CHANGELOG.md` current; ADRs under `docs/decisions/` for load-bearing choices. |
| 124 | +* Flagship bar: a living gap register; every "fixed" reaches "verified" via an |
| 125 | + independent adversarial pass before it counts. |
| 126 | + |
| 127 | +=== 10. Wire-first / no-dead-scaffolding `[review]` |
| 128 | +* Every claimed feature is reachable and tested; no orphaned modules that the |
| 129 | + README advertises (Axiom G06 huggingface lesson). Unwired is not done. |
| 130 | +* Placeholders (`{{...}}`) rendered or removed before release. |
| 131 | +* Flagship bar: green from birth on instantiation; docs and code agree. |
| 132 | + |
| 133 | +== `just quality` — the machine-checkable bar |
| 134 | + |
| 135 | +The recipe runs every `[auto]` row and fails on the first breach. Sketch (the |
| 136 | +template ships the real recipe wired to the instance): |
| 137 | + |
| 138 | +[source,makefile] |
| 139 | +---- |
| 140 | +# Justfile (excerpt) — the flagship rubric's automated subset. |
| 141 | +quality: |
| 142 | + # 1,2,3 correctness + Aqua + JET (Aqua/JET run inside Pkg.test) |
| 143 | + julia --project=. -e 'using Pkg; Pkg.instantiate(); Pkg.build(); Pkg.precompile(); Pkg.test()' |
| 144 | + # 5 docs build + doctests |
| 145 | + julia --project=docs -e 'using Pkg; Pkg.instantiate()' |
| 146 | + julia --project=docs docs/make.jl |
| 147 | + # 6 static analysis: 0 unsuppressed critical |
| 148 | + panic-attack assail . --output /tmp/assail.json |
| 149 | + # 4,8 compat/registry + A2ML validity are enforced by CI gates |
| 150 | + # (registry ci/validate.jl; dogfood-gate Validate A2ML manifests) |
| 151 | + @echo "quality gates passed" |
| 152 | +
|
| 153 | +test: |
| 154 | + julia --project=. -e 'using Pkg; Pkg.test()' |
| 155 | +---- |
| 156 | + |
| 157 | +`just test && just quality` is the golden path; both green is the minimum for a |
| 158 | +release claim (release-claim-requires-hard-pass). |
| 159 | + |
| 160 | +== Scoring (relative to the flagship) |
| 161 | + |
| 162 | +For cross-library comparison, score each of the 10 dimensions 0–3 (0 = absent, |
| 163 | +1 = partial, 2 = meets grade-A bar, 3 = meets flagship bar). Max 30. |
| 164 | + |
| 165 | +* `Axiom.jl` = 30/30 by definition (the reference). |
| 166 | +* A library is *flagship-track* at ≥ 27 with no dimension below 2. |
| 167 | +* Publish the vector, not just the total — a 24 with a 0 in "Security & |
| 168 | + soundness" is not a 24-quality library; the hole caps it (holes-first). |
| 169 | + |
| 170 | +== Provenance |
| 171 | + |
| 172 | +Every bar here was met, and the tool run, on `Axiom.jl` during the flagship |
| 173 | +initiative (gaps G01–G18 verified). This rubric is the generalisation of that |
| 174 | +work into a reusable standard; the template's `just quality` is its executable |
| 175 | +form. |
0 commit comments