Skip to content

test: Optimize test file sizes for faster CI execution (batch 2)#1151

Open
nathan-stender wants to merge 6 commits into
mainfrom
optimize-test-files-batch2-v2
Open

test: Optimize test file sizes for faster CI execution (batch 2)#1151
nathan-stender wants to merge 6 commits into
mainfrom
optimize-test-files-batch2-v2

Conversation

@nathan-stender

Copy link
Copy Markdown
Collaborator

Summary

This PR continues the test file optimization effort to reduce CI execution time by optimizing large test data files while maintaining test coverage.

(Replaces #1150 - rebased onto latest main to fix workflow issues)

Changes

Optimized 5 large test files by reducing redundant data rows while keeping representative samples:

Files Optimized

  1. Perkin Elmer Envision luminescence_example01.csv

    • Reduced from 263 to 151 lines (kept rows A, E, I, M from 16-row plates)
    • JSON: 3.95MB → 965KB (76% reduction)
  2. Methodical Mind methodical_test_2.txt

    • Reduced from 8 to 4 representative rows (A, C, E, G)
    • JSON: 1.77MB → 867KB (51% reduction)
  3. Agilent Gen5 endpoint_singleplate_filter.txt

    • Reduced 384-well plate from 16 to 5 rows (A, D, H, L, P)
    • JSON: 1.8MB → 567KB (69% reduction)
  4. Agilent Gen5 endpoint_fluorescence_polarization.txt

    • Reduced 384-well plate from 16 to 5 rows
    • JSON: 3.6MB → 1.1MB (69% reduction)
  5. BMG MARS 24-10-16 15-09-52 Luminescence 1536.csv

    • Reduced 1536-well plate from 32 to 6 rows (A, G, M, S, Y, e)
    • JSON: 2.4MB → 460KB (81% reduction)

Impact

  • Total JSON size reduced: ~14.6MB reduced to ~3.9MB (73% overall reduction)
  • Test coverage maintained: All tests pass with optimized files
  • CI speed improvement: Significantly faster test execution due to smaller files

Testing

  • ✅ All parser tests pass with optimized files
  • ✅ Representative data patterns preserved for proper test coverage
  • ✅ Linting and type checking pass

Related

- Reduced PE_Envision_luminescence_example01.csv from 263 to 151 lines
- Kept representative rows (A, E, I, M) from 384-well plates
- JSON file reduced from 3.95MB to 965KB (76% reduction)
- Maintains test coverage with representative data samples
- Reduced data rows from 8 (A-H) to 4 representative rows (A, C, E, G)
- TXT file reduced from 7.9KB to 4.9KB (38% reduction)
- JSON file reduced from 1.77MB to 867KB (51% reduction)
- Maintains test coverage with representative data patterns
- endpoint_singleplate_filter.txt: 127 → 71 lines, JSON 1.8MB → 567KB (69% reduction)
- endpoint_fluorescence_polarization.txt: 152 → 72 lines, JSON 3.6MB → 1.1MB (69% reduction)
- Kept representative rows (A, D, H, L, P) from 384-well plates while maintaining test coverage
- Reduced from 32 rows to 6 representative rows (A, G, M, S, Y, e)
- CSV: 41 → 16 lines
- JSON: 2.4MB → 460KB (81% reduction)
- Maintains test coverage for 1536-well plate data
@nathan-stender
nathan-stender requested review from a team and slopez-b as code owners March 16, 2026 17:59
…e modified

The detect-changes job was failing when only test files (not source files) were
changed because grep returns exit code 1 when no matches are found, and the
workflow runs with set -e (exit on error).

Added || true to grep commands to prevent the script from exiting when no
source files match the pattern.

Co-Authored-By: Claude Opus 4.1 <noreply@anthropic.com>
@nathan-stender

Copy link
Copy Markdown
Collaborator Author

Fixed the detect-changes job failure. The issue was that grep returns exit code 1 when no matches are found, and since the workflow runs with set -e, it was causing the script to exit when only test files (not source files) were modified. Added || true to the grep commands to handle this case properly.

The CI should now run correctly.

ajcariaga16
ajcariaga16 previously approved these changes Mar 17, 2026
# Conflicts:
#	tests/parsers/perkin_elmer_envision/testdata/PE_Envision_luminescence_example01.json
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants