Skip to content

feat: Optimize CI by conditionally running schema generation tests#1146

Closed
nathan-stender wants to merge 36 commits into
mainfrom
optimize-ci-conditional-schema-tests
Closed

feat: Optimize CI by conditionally running schema generation tests#1146
nathan-stender wants to merge 36 commits into
mainfrom
optimize-ci-conditional-schema-tests

Conversation

@nathan-stender

Copy link
Copy Markdown
Collaborator

Summary

Optimize CI performance by only running schema generation tests when relevant files change.

Problem

  • Schema generation test takes 3+ minutes (35% of test suite time)
  • Most PRs don't modify schemas or schema generation code
  • Running this test for every PR wastes CI resources

Solution

Implemented conditional test execution using GitHub Actions path filtering:

  • Schema generation test only runs when schema-related files change
  • Always runs on main branch to ensure consistency
  • Test runs on only one Python version (output is the same for all)

Changes

  1. Added dorny/paths-filter to detect changed files
  2. Excluded schema generation test from regular test runs
  3. Created separate job for schema generation that runs conditionally
  4. Reduced timeout from 15 to 10 minutes for regular tests

Files that trigger schema generation test

  • src/allotropy/allotrope/schemas/** - Schema files
  • src/allotropy/allotrope/schema_parser/** - Schema generation code
  • src/allotropy/allotrope/models/** - Generated models
  • tests/allotrope/schema_parser/generate_schemas_test.py - The test itself
  • pyproject.toml - Dependencies

Expected Impact

  • Most PRs: Save ~3 minutes per Python version (9+ minutes total)
  • Schema-related PRs: No change (test still runs)
  • Overall CI: ~50% faster for typical PRs

Testing

This PR itself demonstrates the optimization:

  • Schema generation test should NOT run (no schema files changed)
  • All other tests should run normally
  • CI should complete faster than usual

Notes

  • If schema generation test becomes a required status check, we may need to adjust the approach
  • Consider documenting this behavior in contributing guidelines

Co-Authored-By: Claude Opus 4.1 noreply@anthropic.com

nathan-stender and others added 30 commits March 12, 2026 15:45
Problem:
- Tests were timing out after 10 minutes in GitHub Actions
- Root cause: chardet.detect() spending 20+ seconds on large test files

Solution:
- Try UTF-8 encoding first (works for 95% of files, nearly instant)
- Fall back to chardet only when UTF-8 decode fails
- Increase timeout to 15 minutes as safety measure

Results:
- 4x speedup for large files (10s → 2.5s)
- 2x speedup for full test suites
- Tests now complete well within time limits

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The luminex CSV files contain corrupted micro (µ) symbols that were replaced
with UTF-8 replacement characters. When parsed with ISO-8859-1 (via chardet),
these got incorrectly displayed as '�'. With the UTF-8 optimization, we now
correctly get '�' instead.

Updated the expected JSON files to reflect the correct encoding behavior.
Similar to the previous luminex fix, the CSV file contains a registered
trademark symbol (®) that was incorrectly displayed as '®' when parsed
with ISO-8859-1. With UTF-8 parsing, we now correctly get '®'.
- AppBio QuantStudio example01.txt: 3.5MB → 7KB (99.8% reduction)
- Biorad Bioplex example01.xml: 6.0MB → 1.9MB (68.4% reduction)

Reduces test execution time while maintaining full test coverage.
All tests passing with regenerated output files.
Successfully optimized:
- appbio_quantstudio_example05.txt: 228KB → 5KB (97.9% reduction)
- appbio_quantstudio_example07.txt: 284KB → 6KB (97.9% reduction)

Total files optimized so far: 4
All tests passing with regenerated output files.
Test suite improvement: 1.1x faster (10.8% reduction)
- Baseline: 341 seconds
- Optimized: 304 seconds
- 37 seconds saved

4 files optimized with 68-99% size reductions
All tests passing ✅
AppBio QuantStudio:
- example02.txt: 717KB → 54KB (92.5% reduction)
- example03.txt: 302KB → 40KB (86.7% reduction)
- example04.txt: 124KB → 28KB (77.2% reduction)

MolDev SoftMax Pro:
- partial_plate_with_empty_values.txt: 226KB → 7KB (96.8% reduction)
- ACSINS_absorbance_timeformat_spectrum.txt: 223KB → 79KB (64.7% reduction)

Agilent:
- TapeStation example_01.xml: 161KB → 18KB (89.0% reduction)
- Gen5 kinetic_helper_gene_growth_curve.txt: 120KB → 7KB (93.8% reduction)

Roche CEDEX BioHT:
- example03.txt: 104KB → 103KB (minimal change)

Preserves test coverage while significantly reducing CI execution time.

Co-Authored-By: Claude Opus 4.1 <noreply@anthropic.com>
Luminex xPONENT:
- NaN_and_not_reported_values.csv: 44KB → 3KB (92.1% reduction)

Roche CEDEX HiRes:
- example_3.csv: 40KB → 4KB (89.7% reduction)

Removed JSON outputs for affected files to be regenerated on next test run.

Co-Authored-By: Claude Opus 4.1 <noreply@anthropic.com>
Deleted JSON outputs from previous batches that will be regenerated
when tests are run with the optimized input files.

Co-Authored-By: Claude Opus 4.1 <noreply@anthropic.com>
Reverted 3 files that caused test failures:
- luminex_xPONENT_NaN_and_not_reported_values.csv
- ACSINS_absorbance_timeformat_spectrum.txt
- partial_plate_with_empty_values.txt

These files have complex formatting requirements that were broken
by the optimization. Restoring to original versions.

Co-Authored-By: Claude Opus 4.1 <noreply@anthropic.com>
- Reduced from 7 tubes to 3 tubes (keeping test coverage)
- Simplified polygon points in gate regions (max 5 points per polygon)
- File lines reduced from 11,629 to 7,965
- Test passes with optimized data

Co-Authored-By: Claude Opus 4.1 <noreply@anthropic.com>
- Reduced from 16 samples to 6 samples (keeping test coverage)
- File lines reduced from ~1500 to 767
- Test passes with optimized data

Co-Authored-By: Claude Opus 4.1 <noreply@anthropic.com>
…: 38KB → 16KB (58% reduction)

- Reduced from 16 rows (A-P) to 6 rows (A-F) per data section
- Kept 144 wells instead of 384 wells (maintaining test coverage)
- File lines reduced from 240 to 160
- Test passes with optimized data

Co-Authored-By: Claude Opus 4.1 <noreply@anthropic.com>
…: 43KB → 18KB (58% reduction)

- Reduced from 16 rows (A-P) to 6 rows (A-F) per data section
- Kept 144 wells instead of 384 wells (maintaining test coverage)
- File lines reduced from 260 to 170
- Test passes with optimized data

Co-Authored-By: Claude Opus 4.1 <noreply@anthropic.com>
- Reduced from 16 samples to 5 samples (A1, A2, B1, C1, D1)
- Maintains diversity of test data across all rows
- File lines reduced from 257 to 81
- Test passes with optimized data

Co-Authored-By: Claude Opus 4.1 <noreply@anthropic.com>
…: 23KB → 10KB (57% reduction)

- Reduced from 16 rows (A-P) to 6 rows (A-F) per data section
- Kept 144 wells instead of 384 wells (maintaining test coverage)
- File lines reduced from 157 to 117
- Test passes with optimized data

Co-Authored-By: Claude Opus 4.1 <noreply@anthropic.com>
… 38KB → 11KB (71% reduction)

- Reduced kinetic time points from 61 to 13 (keeping every 5th point)
- Maintains kinetic profile for testing purposes
- File lines reduced from 134 to 86
- Test passes with optimized data

Co-Authored-By: Claude Opus 4.1 <noreply@anthropic.com>
- Reduced ladder sample peaks from 14 to 5
- Kept all peaks for other samples (3 each)
- File lines reduced from 766 to 586
- Test passes with optimized data

Co-Authored-By: Claude Opus 4.1 <noreply@anthropic.com>
…ction)

- Reduced wavelength points from 41 to 14 (keeping every 3rd point)
- Maintains spectral profile with NaN and OVRFLW values
- File lines reduced from 82 to 55
- Test passes with optimized data

Co-Authored-By: Claude Opus 4.1 <noreply@anthropic.com>
…KB → 10KB (60% reduction)

- Reduced wavelength points from 41 to 14 (keeping every 3rd point)
- Maintains spectral profile for testing
- File lines reduced from 85 to 58
- Test passes with optimized data

Co-Authored-By: Claude Opus 4.1 <noreply@anthropic.com>
…(70% reduction)

MAJOR PERFORMANCE IMPROVEMENT:
- Test runtime reduced from 1.75s to 0.60s (66% faster!)
- Reduced from 93 wells to 24 wells
- File lines reduced from 112 to 43
- One of the slowest tests now runs much faster

Co-Authored-By: Claude Opus 4.1 <noreply@anthropic.com>
…peedup)

- Reduced tubes from 7 to 3
- Simplified polygon points in gates (max 5 points)
- File size: 1.4MB → 690KB
- Test time: 2.19s → 1.03s (53% improvement)

Co-authored-by: Claude <assistant@anthropic.com>
- Kept every 3rd wavelength point (14 of 41)
- File size: 2.8KB → 2.4KB
- Test time: stable at ~0.74s

Co-authored-by: Claude <assistant@anthropic.com>
nathan-stender and others added 6 commits March 14, 2026 01:22
- Kept every 3rd wavelength point (14 of 41)
- File size: 2.5KB → 2.1KB
- Handles OVRFLW values correctly

Co-authored-by: Claude <assistant@anthropic.com>
- Reduced wells from 93 to 24
- File size: 19KB (unchanged but less data)
- Test time: 1.75s → 0.73s (58% improvement)

Co-authored-by: Claude <assistant@anthropic.com>
- Reduced from 8 well rows (A-H) to 4 (A-D)
- File size: 21KB → 13KB
- Test time: 3.41s → 1.85s (46% improvement)

Co-authored-by: Claude <assistant@anthropic.com>
- Reduced samples from 6 to 3
- File size: 25KB → 16.5KB (34% reduction)
- Test time: ~1.37s

Co-authored-by: Claude <assistant@anthropic.com>
- Reduced from 96 to 24 wells (for both fluorophores)
- File size: 13KB → 2.5KB (81% reduction)
- Test time: ~0.38s

Co-authored-by: Claude <assistant@anthropic.com>
- Schema generation test only runs when relevant files change
- Saves ~3 minutes per Python version (9+ minutes total) for most PRs
- Always runs on main branch to ensure consistency
- Reduces timeout from 15 to 10 minutes for regular tests

The schema generation test accounts for 35% of test suite time but is only
needed when schemas or generation code changes.

Co-Authored-By: Claude Opus 4.1 <noreply@anthropic.com>
@nathan-stender
nathan-stender requested review from a team and slopez-b as code owners March 14, 2026 19:07
@nathan-stender

Copy link
Copy Markdown
Collaborator Author

Closing this PR as it contained unrelated test file optimization changes. Created a clean PR with only the CI workflow changes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant