Skip to content

Add --self-test flag to bin/validate-patterns (closes #71)#95

Merged
etagwerker merged 1 commit intomainfrom
feature/validate-patterns-self-test
May 5, 2026
Merged

Add --self-test flag to bin/validate-patterns (closes #71)#95
etagwerker merged 1 commit intomainfrom
feature/validate-patterns-self-test

Conversation

@JuanVqz
Copy link
Copy Markdown
Member

@JuanVqz JuanVqz commented May 5, 2026

Summary

Closes #71. Adds a --self-test flag to bin/validate-patterns that writes valid + invalid fixture YAMLs to a tmpdir and asserts the validator's behavior end-to-end. Wired into CI alongside the existing file-validation step. Single-file, stdlib-only, no Bundler / no test framework — matches the validator's existing philosophy.

Closes the test-coverage gap that was surfaced during PR #70 review: the validator's rejection paths (missing required keys, broken regex, unknown kind: value) are exercised manually but never asserted in CI. Without this, someone could refactor the validator and silently drop a guard while the 12 in-repo files keep passing.

What --self-test does

  1. Builds a base valid pattern doc and 8 case variants.
  2. Writes each case to Dir.mktmpdir.
  3. Calls validate(path) in-process (not via subprocess).
  4. Asserts:
    • Valid cases → errors.empty?
    • Invalid cases → errors contains a substring matching the expected rejection
  5. Prints OK self-test (N/N passed) and exits 0; or prints which assertions failed and exits 1.

The 8 cases

# Case Expect
1 Valid: kind: breaking pass
2 Valid: kind: deprecation pass
3 Valid: kind: migration pass
4 Valid: kind: optional pass
5 Invalid: missing upgrade_findings: top-level key fail with missing top-level key: upgrade_findings
6 Invalid: missing kind: per-pattern key fail with missing key: kind
7 Invalid: pattern: regex [unclosed fail with pattern regex does not compile
8 Invalid: kind: bogus fail with invalid kind: "bogus"

Files changed

  • bin/validate-patterns — added self_test function (~60 lines) and a --self-test dispatch branch. Updated usage docstring.
  • .github/workflows/validate-patterns.yml — added ruby bin/validate-patterns --self-test step after the existing validation step.
  • CLAUDE.md — documented the --self-test invocation under Repository tooling.
  • rails-upgrade/CHANGELOG.md — Unreleased entry.

Test plan

  • bin/validate-patterns --self-test exits 0 with OK self-test (8/8 passed)
  • bin/validate-patterns (no flag) still validates all 12 in-repo files clean — no regression
  • Manually verified one rejection case by introducing a bad fixture: validator's expected error string matches

Why this approach

The original issue body proposed a separate bin/validate-patterns-test script. Switched to a --self-test flag on the same binary because:

  • One file, one entry point — no new binary to discover.
  • Test logic lives next to the validator logic it tests.
  • Calls validate(path) in-process — cleaner than shelling out and re-parsing exit codes.
  • ~60 line addition rather than a 50-line standalone script.

Linked issues

Out of scope

  • Adopting RSpec / Minitest (the proposed flag deliberately uses no framework)
  • Reorganizing bin/validate-patterns into a class
  • Tests for behavior outside the rejection paths (e.g., file glob behavior, paths-empty warning)

Closes the test-coverage gap surfaced during PR #70 review. Until now
the validator's rejection paths (missing required keys, broken regex,
unknown kind: value) had no automated coverage — CI ran the validator
on the 12 in-repo files which all pass, so a future refactor could
silently drop a guard without breaking CI.

`bin/validate-patterns --self-test` builds 8 fixture cases in a tmpdir
and calls validate(path) in-process for each:

- 4 valid cases (one per kind: value)
- 4 invalid cases:
  - missing top-level upgrade_findings: key
  - missing required pattern key (kind:)
  - pattern: regex that does not compile
  - unknown kind: value (bogus)

Asserts valid → errors.empty?; invalid → errors contains the expected
substring. Prints "OK self-test (N/N passed)" and exits 0/1.

Wired into .github/workflows/validate-patterns.yml as an additional
step after the existing file-validation step.

Single file, stdlib-only — no Bundler, no test framework. Matches the
validator's existing philosophy.

Refs #53, closes #71
@JuanVqz JuanVqz self-assigned this May 5, 2026
@JuanVqz JuanVqz requested review from arielj and etagwerker May 5, 2026 00:32
@JuanVqz JuanVqz marked this pull request as ready for review May 5, 2026 00:32
Copy link
Copy Markdown
Member

@etagwerker etagwerker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@JuanVqz Looks good, thanks! 👍🏻

@etagwerker etagwerker merged commit ed731ff into main May 5, 2026
1 check passed
@etagwerker etagwerker deleted the feature/validate-patterns-self-test branch May 5, 2026 00:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

Add --self-test flag to bin/validate-patterns (cover rejection paths)

2 participants