Skip to content

feat: add inner loop and compile-once-run-many optimization for Java benchmarking#1245

Merged
misrasaurabh1 merged 2 commits into
omni-javafrom
java-benchmark-inner-loop
Feb 2, 2026
Merged

feat: add inner loop and compile-once-run-many optimization for Java benchmarking#1245
misrasaurabh1 merged 2 commits into
omni-javafrom
java-benchmark-inner-loop

Conversation

@misrasaurabh1
Copy link
Copy Markdown
Contributor

Summary

  • Add inner loop in Java test instrumentation for JIT warmup within single JVM
  • Implement compile-once-run-many: compile tests once with Maven, then run directly via JUnit Console Launcher (~500ms vs ~5-10s per invocation)
  • Add fallback to Maven-based execution when direct execution fails
  • Update parsing to handle JUnit Console Launcher output format

Changes

Inner Loop Instrumentation

  • Modified _add_timing_instrumentation() to wrap test code in a for loop
  • Reads CODEFLASH_INNER_ITERATIONS from environment (default: 100)
  • Each iteration prints its own timing markers with unique iteration ID

Compile-Once-Run-Many Optimization

  • New _compile_tests() - compiles tests once with mvn test-compile
  • New _get_test_classpath() - gets classpath from Maven
  • New _run_tests_direct() - runs tests directly via JUnit Console Launcher
  • Reduced overhead from ~5-10s to ~500ms per JVM invocation

Parsing Updates

  • Added fallback to use subprocess stdout when XML system-out has no timing markers
  • This handles JUnit Console Launcher output format correctly

Benchmark Flow

Python Process
│
├─ Step 1: mvn test-compile (one-time: ~5-10s)
├─ Step 2: mvn dependency:build-classpath (one-time: ~2-3s)
│
└─ Step 3: For each outer loop (1-3 JVM invocations):
      │
      └─ java -cp ... ConsoleLauncher ... (~500ms overhead)
           │
           └─ Inner Loop (100 iterations within JVM):
                ├─ Iteration 0: print timing marker, run test, print result
                ├─ Iteration 1: print timing marker, run test, print result
                ├─ ...
                └─ Iteration 99: print timing marker, run test, print result

Test plan

  • All 32 Java instrumentation tests pass
  • E2E tests for inner loop timing markers
  • E2E tests for multiple test methods with inner loop
  • Fallback to Maven execution when direct execution fails

🤖 Generated with Claude Code

misrasaurabh1 and others added 2 commits February 1, 2026 23:36
…benchmarking

- Add inner loop in Java test instrumentation for JIT warmup within single JVM
- Implement compile-once-run-many: compile tests once with Maven, then run
  directly via JUnit Console Launcher (~500ms vs ~5-10s per invocation)
- Add fallback to Maven-based execution when direct execution fails
- Update parsing to handle JUnit Console Launcher output format
- Add inner_iterations parameter (default: 100) to control loop count
- Add comprehensive E2E tests for inner loop benchmarking

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Configure JUnit Console Launcher to capture stdout/stderr in XML reports:
- Add --config=junit.platform.output.capture.stdout=true
- Add --config=junit.platform.output.capture.stderr=true
- Change --details=verbose to --details=none to avoid duplicate output

This ensures timing markers are properly captured in the JUnit XML's
<system-out> element, eliminating the need to rely on subprocess stdout
fallback for parsing timing markers.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@misrasaurabh1 misrasaurabh1 merged commit f6f0fe2 into omni-java Feb 2, 2026
17 of 25 checks passed
@misrasaurabh1 misrasaurabh1 deleted the java-benchmark-inner-loop branch February 2, 2026 00:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant