feat: Cytiva T200 - Implement streaming decoder to reduce memory usage by 55%

[nathan-stender]

Summary

Implement streaming architecture for Cytiva Biacore T200 Evaluation parser to reduce peak memory usage from 3.5 GB to 1.6 GB (55% reduction).

Problem

The original parser was using 3.5 GB peak memory (28.8x file size amplification) which caused OOM errors in 6 GB ECS containers.

Solution

Key insight: Label streams are tiny (23 KB total) - pre-load them all to solve flow cell tracking, then stream cycles.

Architecture:

1. Pre-load ALL label streams (23 KB) to build (cycle, curve) → flow_cell_id mapping
2. Stream cycles one at a time using the label mapping
3. Each cycle: 0.5 MB numpy → 10 MB DataFrame
4. All 131 cycles collected: 1.35 GB (vs 3.5 GB before)

Results

Memory Reduction:

• Before: 3.5 GB peak (28.8x amplification) ❌ OOMing
• After: 1.6 GB peak (12.9x amplification) ✅ Comfortable margin
• Reduction: 1.9 GB (55.3%)

Production Viability (6 GB Container):

Container size:     6144 MB
OS overhead:        1024 MB
Parser peak:        1576 MB
Worker overhead:     500 MB
Total used:         3100 MB
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Remaining buffer:   3044 MB (49.5%)

✅ SUCCESS - Comfortable margin, should not OOM

Changes

Core Implementation

• cytiva_biacore_t200_evaluation_decoder.py

  • Add decode_data_streaming() generator that yields one cycle at a time
  • Pre-load all label streams to build flow cell mapping
  • Stream cycles during decode using label mapping
  • decode_data() wraps streaming decoder for test compatibility

• cytiva_biacore_t200_evaluation_data_creator.py

  • Add explicit gc.collect() for cleanup
  • Keep numpy arrays instead of converting to Python lists early

• data_cube.py (previously optimized)

  • Detect numpy arrays and use efficient .tolist() conversion
  • Still unavoidable Python float conversion (8x memory cost)

Testing

• Add new large test file: 25June2025 QMGG256 FcyRI.bme (122 MB, 131 cycles, 17.5M data points)
• Add golden file: 25June2025 QMGG256 FcyRI.json (156 KB)
• All tests passing ✅
• Output unchanged (bit-for-bit identical)

Technical Details

Memory Profile Breakdown:

Phase 1: Decoding (decode_data)
  - Labels + metadata: 0.5 MB
  - Streaming builds cycles one-by-one
  - Peak during build: ~50 MB per cycle
  - Returns all 131 DataFrames: 1.35 GB
  - Peak: 1.43 GB

Phase 2: ASM Conversion (create_data)
  - DataFrames → ASM models
  - data_cube.py calls .tolist() on 17.5M points
  - numpy float32 (4 bytes) → Python float (32 bytes)
  - Conversion overhead: 1.07 GB
  - Peak: 1.62 GB

Top Memory Allocations:

• 533 MB: data_cube.py:122 (.tolist() for measures)
• 533 MB: data_cube.py:70 (.tolist() for dimensions)
• 9 MB: dataclasses (ASM model instances)

Test Results

$ pytest tests/parsers/cytiva_biacore_t200_evaluation/ -xvs
======================== 2 passed in 11.88s ========================

All tests passing with streaming implementation ✅

Documentation

See STREAMING_RESULTS.md for:

• Detailed memory analysis
• Implementation architecture
• Further optimization options
• Comparison with original approach

Future Optimization

If we need further reduction (unlikely given 49% buffer):

• Option A: True end-to-end streaming (1.6 GB → ~200 MB)
  • Process cycle → ASM → discard → next cycle
  • Requires test refactoring
  • 4-6 hour effort

🤖 Generated with Claude Code