feat: auto-retry-on-flaky for mix test.json (v0.5.0)#2
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a783fae3c0
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
|
||
| {flaky_tests, kept_tests} = | ||
| Enum.split_with(run1_tests, fn test -> | ||
| failed?(test) and id(test) in run2_passed_ids |
There was a problem hiding this comment.
When auto-retry is used with --all and a setup_all failure heals on the retry, the first run includes each test as state: "invalid" while the module failure is moved to flaky. This predicate only removes tests whose first-run state was failed, so those invalid entries remain in tests even though merge_summary/4 sets invalid to 0 and result to passed, producing a contradictory green result containing invalid tests. This affects mix test.json --all suites with flaky setup_all callbacks.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Pull request overview
Adds default automatic retry-on-flaky behavior for mix test.json, including merge logic to classify healed failures as flaky while preserving confirmed failures.
Changes:
- Adds
ExUnitJSON.Retry.merge/2and retry orchestration inMix.Tasks.Test.Json. - Adds
--no-retry/config :ex_unit_json, retry: falsesupport and tests. - Updates README, moduledocs, changelog, version, and roadmap metadata for v0.5.0.
Reviewed changes
Copilot reviewed 13 out of 13 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
lib/ex_unit_json/retry.ex |
Adds retry merge/classification logic. |
lib/mix/tasks/test_json.ex |
Adds retry enablement, buffering, subprocess retry, and final output handling. |
lib/ex_unit_json/config.ex |
Adds retry config option and retry?/0. |
lib/ex_unit_json.ex |
Documents retry behavior and schema additions. |
test/ex_unit_json/retry_test.exs |
Adds unit coverage for retry merge behavior. |
test/ex_unit_json/config_test.exs |
Adds config tests for retry option handling. |
test/mix/tasks/test_json_test.exs |
Adds parsing and integration tests for retry behavior. |
README.md |
Documents user-facing retry behavior and opt-out. |
CHANGELOG.md |
Adds v0.5.0 release notes. |
ROADMAP.md |
Marks retry task as shipped. |
roadmap/tasks.toml |
Adds completed roadmap task metadata. |
roadmap/data.json |
Adds generated roadmap task data. |
.version |
Bumps package version to 0.5.0. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| # When everything healed, any run-1 `invalid` count (from a setup_all that | ||
| # has since recovered) is stale — passing means nothing invalid remains. | ||
| if result == "passed", do: Map.put(summary, "invalid", 0), else: summary |
…al-reviewer pass Reports: .audit/<sha>-<slug>.md × 23 (14 full audits, 9 fast-path) Auto-applied (bugs): - retry.ex: setup_all casualties (invalid tests) now resolve against the retry run — healed → passed, re-failed → confirmed, never silently dropped (4-reasoner: Codex GH bot P2 + Copilot + Codex CLI + Claude, PR #2) - formatter.ex: umbrella --compact merge crash → JSONL concatenation - formatter.ex: --first-failure capped across umbrella merge - formatter.ex: merged error_groups re-sorted count-desc - test_json.ex: stale-output guard (rm → truncate fallback → raise) - test_json.ex: --cover precompile honors --no-compile / --warnings-as-errors - test_json.ex: coverage start failures warn on stderr - test_json.ex: --cover no longer suppresses the --failed TIP - test_json_test.exs: parse tests exercise production parser (70-line copy deleted) Auto-applied (docs): - CHANGELOG: [Unreleased] entry; v0.4.1 --cover-threshold; v0.4.3 hint fix - Schema docs: "invalid" state, hint key, module_failures shape, coverage examples - ROADMAP: schema section synced, Jason-fallback wording, phase prose - roadmap/tasks.toml: AGENT.md → AGENTS.md (rmap re-rendered) - config/filters/formatter inline doc drift (:all → :debug, defaults) Discarded with rationale: Codex "~> 0.4 excludes v0.5.0" (false — includes it), 4 cosmetic single-reasoner items. No discuss-design divergence; no rmap follow-ups. Quality gates: 349/349 tests, dialyzer 0, credo --strict 0, sobelow clean, doctor 100%
Summary
Bare
mix test.jsonnow auto-retries failed tests once and distinguishes confirmed failures from flakes — so an AI agent running the default command isn't blocked by 1–2 intermittent async/GenServer/LiveView reds it can't be told to re-run.On failures, the task re-runs only the previously-failed tests (ExUnit-native
--failed --all) in a subprocess, merges both runs by{module, name}, and classifies:tests, exit non-zeroflaky[]array (named, never hidden) and no longer blocksWhen every first-run failure heals,
summary.resultbecomes"passed"and the exit code is0(System.halt(0)override). Aretrymetadata object (retried/confirmed/flaky) is added whenever a retry runs. Honors NEVER HIDE TEST FAILURES — flakes are always surfaced.Opt-out
--no-retryflagconfig :ex_unit_json, retry: false--failed,--summary-only,--first-failure,--compact,--group-by-error,--filter-out, afile:linetarget, umbrella projects, and any already-green suite.Changes
ExUnitJSON.Retry— puremerge/2core (100% coverage)ExUnitJSON.Config—retry?/0+:retryoptionMix.Tasks.Test.Json—--no-retryparse, generalized temp-buffer, retry orchestration, heal-to-green exit override.version→0.5.0Tests
12
Retryunit + 4 retry integration + 4Configtests. Full suite green. format / credo --strict / dialyzer / sobelow / doctor all clean.