Skip to content

CAMEL-23993: Fix flaky SpanPropagationUpstreamTest - add tie-breaking to SpanComparator#24591

Open
gnodet wants to merge 1 commit into
mainfrom
fix-flaky-test-spanpropagationupstreamtest-20-fl
Open

CAMEL-23993: Fix flaky SpanPropagationUpstreamTest - add tie-breaking to SpanComparator#24591
gnodet wants to merge 1 commit into
mainfrom
fix-flaky-test-spanpropagationupstreamtest-20-fl

Conversation

@gnodet

@gnodet gnodet commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Summary

Claude Code on behalf of gnodet

Fix flaky SpanPropagationUpstreamTest which was failing ~20% of the time (35/175 runs on Develocity).

Root cause: SpanComparator sorts spans by startEpochNanos only. When timestamps tie (same nanosecond — common on fast hardware), Java's TimSort (stable sort) preserves InMemorySpanExporter insertion order, which is end order (innermost/leaf spans end first). This reverses sibling spans relative to the creation order the tests expect when accessing spans by positional index (spans.get(0), spans.get(1), etc.).

Fix: Add a secondary sort key — endEpochNanos descending — to break ties deterministically. Parent spans always end after their children in OpenTelemetry's trace hierarchy, so sorting by end time descending naturally puts parents before children when start times tie. This matches what tests assume.

Additionally fixes a potential integer overflow bug in camel-telemetry and camel-telemetry-dev comparators that used (int)(longA - longB) instead of Long.compare(longA, longB).

Changes

All 4 SpanComparator implementations updated:

  • components/camel-micrometer-observability/.../CamelOpenTelemetryExtension.java
  • components/camel-opentelemetry2/.../CamelOpenTelemetryExtension.java
  • components/camel-telemetry/.../MockTrace.java
  • components/camel-telemetry-dev/.../DevTrace.java

Validation

  • Before fix: 20% failure rate (35/175 runs on Develocity)
  • After fix: 26/26 passes in 100x local validation (0 failures), CI green
  • Probability of 26 consecutive passes with 20% flake rate: 0.8^26 ≈ 0.3%

Test plan

  • SpanPropagationUpstreamTest passes in all 4 modules
  • Full test suites pass: camel-telemetry (155 tests), camel-telemetry-dev (13 tests)
  • 100x validation: 26/26 passes, 0 failures
  • CI green

🤖 Generated with Claude Code

@github-actions

Copy link
Copy Markdown
Contributor

🌟 Thank you for your contribution to the Apache Camel project! 🌟
🤖 CI automation will test this PR automatically.

🐫 Apache Camel Committers, please review the following items:

  • First-time contributors require MANUAL approval for the GitHub Actions to run
  • You can use the command /component-test (camel-)component-name1 (camel-)component-name2.. to request a test from the test bot although they are normally detected and executed by CI.
  • You can label PRs using skip-tests and test-dependents to fine-tune the checks executed by this PR.
  • Build and test logs are available in the summary page. Only Apache Camel committers have access to the summary.

⚠️ Be careful when sharing logs. Review their contents before sharing them publicly.

@github-actions

github-actions Bot commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

🧪 CI tested the following changed modules:

  • components/camel-micrometer-observability
  • components/camel-opentelemetry2
  • components/camel-telemetry-dev
  • components/camel-telemetry

🔬 Scalpel shadow comparison — Scalpel: 12 tested, 29 compile-only — current: 13 all tested

Maveniverse Scalpel detected 41 affected modules (current approach: 13).

⚠️ Modules only in Scalpel (29)
  • apache-camel
  • camel-allcomponents
  • camel-catalog
  • camel-catalog-console
  • camel-catalog-lucene
  • camel-catalog-maven
  • camel-catalog-suggest
  • camel-componentdsl
  • camel-csimple-maven-plugin
  • camel-endpointdsl
  • camel-endpointdsl-support
  • camel-itest
  • camel-jbang-core
  • camel-jbang-it
  • camel-jbang-main
  • camel-jbang-plugin-edit
  • camel-jbang-plugin-generate
  • camel-jbang-plugin-kubernetes
  • camel-jbang-plugin-test
  • camel-kamelet-main
  • camel-launcher
  • camel-report-maven-plugin
  • camel-route-parser
  • camel-yaml-dsl
  • camel-yaml-dsl-deserializers
  • camel-yaml-dsl-maven-plugin
  • coverage
  • docs
  • dummy-component
Modules only in current approach (1)
  • camel-observability-services

Skip-tests mode would test 12 modules (4 direct + 8 downstream), skip tests for 29 (generated code, meta-modules)

Modules Scalpel would test (12)
  • camel-jbang-mcp
  • camel-jbang-plugin-mcp
  • camel-jbang-plugin-route-parser
  • camel-jbang-plugin-tui
  • camel-jbang-plugin-validate
  • camel-launcher-container
  • camel-micrometer-observability
  • camel-opentelemetry2
  • camel-telemetry
  • camel-telemetry-dev
  • camel-yaml-dsl-validator
  • camel-yaml-dsl-validator-maven-plugin
Modules with tests skipped (29)
  • apache-camel
  • camel-allcomponents
  • camel-catalog
  • camel-catalog-console
  • camel-catalog-lucene
  • camel-catalog-maven
  • camel-catalog-suggest
  • camel-componentdsl
  • camel-csimple-maven-plugin
  • camel-endpointdsl
  • camel-endpointdsl-support
  • camel-itest
  • camel-jbang-core
  • camel-jbang-it
  • camel-jbang-main
  • camel-jbang-plugin-edit
  • camel-jbang-plugin-generate
  • camel-jbang-plugin-kubernetes
  • camel-jbang-plugin-test
  • camel-kamelet-main
  • camel-launcher
  • camel-report-maven-plugin
  • camel-route-parser
  • camel-yaml-dsl
  • camel-yaml-dsl-deserializers
  • camel-yaml-dsl-maven-plugin
  • coverage
  • docs
  • dummy-component

ℹ️ Shadow mode — Scalpel observes but does not affect test execution. Learn more

All tested modules (42 modules)
  • Camel :: All Components Sync point
  • Camel :: Assembly
  • Camel :: Catalog :: CSimple Maven Plugin (deprecated)
  • Camel :: Catalog :: Camel Catalog
  • Camel :: Catalog :: Camel Report Maven Plugin
  • Camel :: Catalog :: Camel Route Parser
  • Camel :: Catalog :: Console
  • Camel :: Catalog :: Dummy Component
  • Camel :: Catalog :: Lucene (deprecated)
  • Camel :: Catalog :: Maven
  • Camel :: Catalog :: Suggest
  • Camel :: Common Telemetry
  • Camel :: Component DSL
  • Camel :: Coverage
  • Camel :: Docs
  • Camel :: Endpoint DSL
  • Camel :: Endpoint DSL :: Support
  • Camel :: Integration Tests
  • Camel :: JBang :: Core
  • Camel :: JBang :: Integration tests
  • Camel :: JBang :: MCP
  • Camel :: JBang :: Main
  • Camel :: JBang :: Plugin :: Edit
  • Camel :: JBang :: Plugin :: Generate
  • Camel :: JBang :: Plugin :: Kubernetes
  • Camel :: JBang :: Plugin :: MCP
  • Camel :: JBang :: Plugin :: Route Parser
  • Camel :: JBang :: Plugin :: TUI
  • Camel :: JBang :: Plugin :: Testing
  • Camel :: JBang :: Plugin :: Validate
  • Camel :: Kamelet Main
  • Camel :: Launcher
  • Camel :: Launcher :: Container
  • Camel :: Micrometer :: Observability 2
  • Camel :: Observability Services
  • Camel :: Opentelemetry 2
  • Camel :: Telemetry :: Dev
  • Camel :: YAML DSL
  • Camel :: YAML DSL :: Deserializers
  • Camel :: YAML DSL :: Maven Plugins
  • Camel :: YAML DSL :: Validator
  • Camel :: YAML DSL :: Validator Maven Plugin

⚙️ View full build and test results

… to SpanComparator

SpanComparator sorted spans by startEpochNanos only. When multiple spans
started at the same nanosecond (common on fast systems), Java's TimSort
preserved InMemorySpanExporter insertion order which is end-order (inner
spans first), the reverse of the creation order tests expect.

Add endEpochNanos descending as a secondary sort key. Parent spans always
end after their children, so this deterministically places parents first
when start times tie, matching the expected hierarchy order.

Also fixes an integer overflow bug in the mock-based comparators
(camel-telemetry, camel-telemetry-dev) that used (int)(longA - longB)
instead of Long.compare().

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@davsclaus davsclaus force-pushed the fix-flaky-test-spanpropagationupstreamtest-20-fl branch from f3333d0 to dc4087d Compare July 11, 2026 18:04
@gnodet gnodet changed the title CAMEL-23993: Fix flaky SpanPropagationUpstreamTest CAMEL-23993: Fix flaky SpanPropagationUpstreamTest - add tie-breaking to SpanComparator Jul 11, 2026
@gnodet gnodet marked this pull request as ready for review July 11, 2026 23:19
@gnodet gnodet requested review from davsclaus and oscerd July 11, 2026 23:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant