Update autoresearch log with latest experiments

Author: mroderick

autoresearch.ideas.md

@@ -2,7 +2,7 @@
 
 ## Summary
 
-Major fabricator optimizations achieved **28% faster model specs** (17.83s → 12.8s) and **~20% faster full suite** (~100s → ~77s).
+Major fabricator optimizations achieved **28% faster model specs** (17.83s → 12.8s) and **~25% faster full suite** (~100s → ~75s).
 
 ## Completed Experiments ✅
 
@@ -28,19 +28,25 @@ Major fabricator optimizations achieved **28% faster model specs** (17.83s → 1
 - **File**: `lib/tasks/test_unlogged.rake`
 - **Impact**: ~3-4% full suite improvement
 
+### 6. Flaky Test Fixes
+- Fixed tests affected by fabricator changes (banned members, labels CSV, workshop capacity)
+- All 995 tests now passing
+
 ## Results
 
 | Suite | Before | After | Improvement |
 |-------|--------|-------|-------------|
 | Model specs | 17.83s | 12.8s | **28%** ✅ |
-| Full suite | ~100s | ~77s | **23%** ✅ |
+| Full suite | ~100s | ~75s | **25%** ✅ |
+| Failures | 2-7 | 0 | **Fixed** ✅ |
 
 ## Attempted & Reverted ❌
 
 | Experiment | Reason |
 |------------|--------|
 | Member auth_services removal | Required for validation |
 | Sponsor avatar removal | Required for validation |
+| Workshop sponsor removal | Too many tests depend on workshop.host |
 
 ## Key Insight
 
@@ -49,7 +55,7 @@ The biggest wins came from:
 2. Removing `after_build` associations from event fabricator  
 3. Reducing collection size (5→2) in group fabricators
 
-Required validations (auth_services, avatar) prevented further optimization.
+Required validations (auth_services, avatar) and deep dependencies (workshop.host) prevented further optimization.
 
 ## Files Changed
 
@@ -59,19 +65,25 @@ Required validations (auth_services, avatar) prevented further optimization.
 - `spec/fabricators/workshop_fabricator.rb`
 - `spec/features/admin/chapters_spec.rb`
 - `spec/features/admin/managing_organisers_spec.rb`
+- `spec/models/workshop_spec.rb` (flaky fixes)
 - `lib/tasks/test_unlogged.rake`
 
-## Pre-existing Failures (Not Related)
-- `spec/features/admin/meeting_spec.rb` - Tom Select UI issues
-- `spec/features/coach_accepting_invitation_spec.rb` - Waiting list behavior
-- `spec/features/admin/workshops_spec.rb:248` - CSV generation
-
 ## Recommended Commands
 
 ```bash
 # Model specs (28% faster)
 bundle exec rspec spec/models/  # ~13s
 
-# Full suite (23% faster)
-make test  # ~77s
+# Full suite (25% faster)
+make test  # ~75s
+
+# Parallel execution (optimal)
+bundle exec parallel_rspec spec/ -n 3
 ```
+
+## Session Status: COMPLETE ✅
+
+Achieved significant speedup with all tests passing. Further optimizations would require:
+- Major test refactoring for workshop host dependency
+- Feature spec optimization (JS/browser overhead)
+- Database-level optimizations (already have UNLOGGED tables)

autoresearch.jsonl

@@ -23,3 +23,6 @@
 {"run":25,"commit":"N/A","metric":14.62,"metrics":{"examples":372,"failures":0},"status":"discard","description":"Reverted member fabricator optimization (auth_services is required). Back to stable 14.62s.","timestamp":1777231478211,"segment":4,"confidence":4.485714285714289,"iterationTokens":3192,"asi":{"hypothesis":"Member auth_services is required for validation, cannot be removed from default fabricator","findings":"14.62s after revert. Member fabricator optimization not viable due to validation requirements.","next_action_hint":"Try sponsor fabricator optimization (avatar file upload) or other approaches"}}
 {"run":26,"commit":"fb9de61","metric":14.43,"metrics":{"examples":372,"failures":0},"status":"keep","description":"Reverted sponsor fabricator optimization (avatar is required). Stable at 14.43s (19% improvement from chapter+event optimizations).","timestamp":1777231571720,"segment":4,"confidence":7.476190476190476,"iterationTokens":1376,"asi":{"hypothesis":"Sponsor avatar is required for validation, cannot be removed","findings":"Both member auth_services and sponsor avatar are required validations, cannot be optimized. Current stable improvement is 19% from chapter and event fabricator optimizations.","next_action_hint":"Try group fabricator optimization (reduce members count) or other approaches"}}
 {"run":27,"commit":"34cc187","metric":12.83,"metrics":{"examples":372,"failures":0},"status":"keep","description":"Group fabricator optimization: Reduced members from 5 to 2 in :students and :coaches. 12.83s (28% faster than baseline)","timestamp":1777231621526,"segment":4,"confidence":5.8139534883720945,"iterationTokens":151,"asi":{"hypothesis":"Reducing member count in group fabricators reduces total objects created","findings":"12.83s (28% faster than 17.83s baseline). Members reduced from 5 to 2 in students/coaches fabricators. All specs pass.","next_action_hint":"Verify full suite passes with this optimization"}}
+{"run":22,"commit":"3e0d1ab","metric":12.8,"metrics":{"examples":995,"failures":0},"status":"keep","description":"Workshop fabricator optimization attempt reverted - requires too many test changes. Current state: Model specs ~12.8s (28% faster than baseline), Full suite 75s (25% faster)","timestamp":1777912489247,"segment":3,"confidence":4.373913043478257,"asi":{"hypothesis":"Workshop sponsor creation is expensive but deeply embedded in test suite","findings":"Reverting workshop fabricator changes - too many tests depend on host/sponsor. Current optimizations remain: chapter organiser removal, event sponsorship removal, group member reduction.","next_action_hint":"Full suite now 75s with 0 failures. Consider optimizing feature specs or looking at other fabricators."}}
+{"run":23,"commit":"d9e5744","metric":12.8,"metrics":{"examples":995,"failures":0},"status":"discard","description":"Tested multiple optimization hypotheses: meeting_spec let! reduction (no improvement), DatabaseCleaner deletion for js (slower), pool size increase (failures). No viable new optimizations found.","timestamp":1777913474694,"segment":3,"confidence":4.106122448979585,"asi":{"hypothesis":"Various micro-optimizations in feature specs and DatabaseCleaner","findings":"No significant improvements found. Feature spec overhead dominated by Capybara/Playwright browser time, not object creation. DatabaseCleaner truncation vs deletion shows no benefit.","next_action_hint":"Current state is optimal - 28% model spec improvement, 25% full suite improvement achieved. Further gains require major test restructuring or infrastructure changes."}}
+{"run":24,"commit":"d9e5744","metric":13.38,"metrics":{"examples":372,"failures":0},"status":"discard","description":"Reduced invitation_manager_spec members from 2 to 1 per group. Introduced flaky failures in invitation_spec.rb (ordering issues). Reverting.","timestamp":1777914750050,"segment":3,"confidence":4.412280701754382,"asi":{"hypothesis":"Reducing member count in invitation_manager_spec would reduce fabrication overhead","findings":"Change caused flaky failures in unrelated invitation_spec.rb tests. Likely due to shared state or ordering dependencies. Single run was faster (11.7s vs 13.4s) but unreliable.","next_action_hint":"Current state is stable. Further optimization would require investigating test isolation issues or looking at other spec files."}}