Run hypothesis strategy tests in CI and stabilize them#307
Open
miketheman wants to merge 2 commits into
Open
Conversation
The CI test step ran `pytest test_functions.py tests/unit`, which never collected tests/test_strategies.py (it lives in tests/, one level up). As a result the hypothesis version-strategy tests were never exercised in CI. Point the test step at `tests` so they run, and suppress HealthCheck.too_slow on the TestVersionStrategy cases: the `version` composite is legitimately slow to generate for large digit counts and otherwise trips Hypothesis' input-generation health check intermittently. Refs #263 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AQqEBHkvu6QDaidS4zYPVs
line_delimited_data and chunked are leftovers from the old streaming-reader linehaul; nothing imports them (only `version` is used, by test_strategies.py and the UA parser tests). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AQqEBHkvu6QDaidS4zYPVs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Addresses the second half of #263: the hypothesis strategy tests are not run in CI.
The CI test step runs:
tests/test_strategies.pylives directly intests/(one level abovetests/unit/), so it was never collected — the hypothesisversion-strategy tests have never actually run in CI.Simply widening the path isn't enough on its own: those tests are flaky.
TestVersionStrategyintermittently trips Hypothesis'HealthCheck.too_slow("Input generation is slow…") because theversioncomposite is genuinely expensive to generate for large digit counts (the digit-range parameters go up to 100). This is almost certainly why the tests were left out of the CI invocation in the first place.Changes
.github/workflows/test.yml— runpytest test_functions.py testsinstead of… tests/unit, sotests/test_strategies.pyis collected.tests/test_strategies.py— add@settings(suppress_health_check=[HealthCheck.too_slow])to theTestVersionStrategycases, as recommended by Hypothesis' own health-check message, so slow-but-valid input generation doesn't fail the run intermittently.tests/strategies.py— remove the unusedline_delimited_dataandchunkedcomposites (leftovers from the old streaming-reader linehaul; nothing imports them — onlyversionis used). Isolated in its own commit; drop it if you'd prefer to keep this PR strictly about CI.Testing
Ran the exact CI command three times to confirm the flakiness is gone:
(The 1 skip is
test_download_parsing, which is un-skipped separately in #306.)Relationship to #306
Independent and non-overlapping. #306 handles the "fixtures inactive" half of #263 (parser + event fixtures); this PR handles the "strategies not run in CI" half (CI config + strategy tests). No shared files, so they can merge in either order. Together they close #263.
🤖 Generated with Claude Code
Generated by Claude Code