Support Multithreaded DD4hep Simulation #2397
Draft
rahmans1 wants to merge 7 commits into
Draft
Conversation
added 5 commits
May 11, 2026 11:34
Add the low-level step data needed to record terminal track states for the MT DD4hep port. Wire the StepInterface track_death filter through StepParams, add death record storage to StepStateData and StepStateDataImpl, and teach the step gathering/copyout path to capture killed tracks into compact death records. Enable GeantSd to request primary IDs and death records so downstream replay code can reconstruct the terminal state without changing the existing event-scoped primary propagation model. Prompt: "Implement the core step/death-record plumbing for the MT DD4hep port without changing the existing event-scoped primary-ID model. Add a StepInterface filter for track_death, propagate that flag through StepParams, extend StepStateDataImpl and StepStateData with death_track_id, death_primary_id, death_particle, death_pos, death_dir, death_energy, death_time, and death_valid_id, and update the resize/assignment/bool logic accordingly. Add TrackDeathRecord and death output storage to DetectorSteps, implement copy_deaths on host and device, and update StepGatherExecutor to clear stale death slots and capture terminal state for killed tracks independently of detector filtering. Update GeantSd so the step selection includes primary_id and track_death, and add or adjust tests as needed for the new fields and copyout behavior." Assisted-by: Codex (GPT-5.5)
Extend GeantTrackReconstruction so it can preserve and restore the initial handover state of offloaded tracks while keeping the existing event-scoped PrimaryId model. Capture the particle ID and initial kinematics when a track is acquired, restore the full track state for downstream processing, and add the helper methods needed by replay. Update the GeantTrackReconstruction unit tests to cover the new acquisition signature and the event-scoped replay behavior. Prompt: "Implement the GeantTrackReconstruction part of the MT DD4hep port while preserving the existing event-scoped primary propagation model. Change acquire(G4Track&) to acquire(G4Track&, ParticleId), store the particle ID and initial kinematics in AcquiredData, add restore_initial(), view_initial(), is_generator_primary(), and particle_id() support, and keep start_/init_event()-based event-scoped indexing internally. Update the GeantTrackReconstruction tests so they pass ParticleId explicitly and still verify multi-flush/event-scoped behavior." Assisted-by: Codex (GPT-5.5)
Wire the tracking replay path that reconstitutes offloaded primaries and fires Pre/PostUserTrackingAction callbacks during Flush. Accumulate death records in the hit processor, replay tracking callbacks using the reconstructed event-scoped PrimaryIds, and apply terminal death state to PostUserTrackingAction when a matching death record exists. Prompt: "Implement the core tracking replay path for the MT DD4hep port without changing the existing primary-ID model. Add death accumulation to HitProcessor, replay Pre/PostUserTrackingAction from LocalTransporter::Flush using the reconstructed event-scoped PrimaryIds, apply death records to the post-state before firing PostUserTrackingAction, and add the IsTrackOffloadedToCeleritas helper. Keep the DD4hep initialization changes out of this commit." Assisted-by: Codex (GPT-5.5)
Replace the DD4hep run-action path with the StateDependent-based lifecycle hook in CelerPhysics, remove the separate CelerRun plugin, and wire the DD4hep build to the new initialization path. Prompt: "Implement the DD4hep integration layer for the MT port using the Geant4 StateDependent mechanism instead of a dedicated run action. Update CelerPhysics to register the state-dependent hooks, remove the CelerRun plugin and its CMake wiring, and keep the DD4hep-specific changes isolated from the core library commits." Assisted-by: Codex (GPT-5.5)
Update the DD4hep preshower example so it runs against the external EIC input file instead of the particle gun, and add the tracker-region constants required by the Geant4 particle handler for MC-truth linking. Prompt: "Update the DD4hep example configuration so it uses the external EIC HepMC3 input file instead of the particle gun, keeps the revised EDM4HEP output configuration, and adds the tracker_region_rmax and tracker_region_zmax constants to the Preshower XML so the Geant4 particle handler can build MC-truth links without errors." Assisted-by: Codex (GPT-5.5)
Test summary 5 334 files 8 641 suites 6m 55s ⏱️ Results for commit 2ed8d77. ♻️ This comment has been updated with latest results. |
added 2 commits
May 11, 2026 15:49
Reconstruct Geant4 touchable state for synthetic tracks used when replaying DD4hep tracking callbacks after Celeritas transport. The DD4hep particle handler queries `G4Track::GetVolume()` during begin/end tracking callbacks, but reconstructed tracks only restored identity, user information, creator process, position, direction, time, and energy. Without a touchable, `GetVolume()` returns null and DD4hep can dereference it while building MC-truth state, causing the observed SIGSEGV during the multithreaded preshower run. Use the Geant4 tracking navigator to locate the restored track position and attach a fresh touchable before invoking DD4hep callbacks. This is done for the initial handoff state used by `PreUserTrackingAction` and again after applying the Celeritas terminal state used by `PostUserTrackingAction`. Terminal Celeritas states can legitimately be on the world boundary or just outside the world, so failure to locate the terminal state is non-fatal and keeps the initial touchable for DD4hep bookkeeping. This avoids copying original `G4TouchableHandle` values, which may be null or unsafe for handed-over tracks, while preserving the callback replay path needed for DD4hep MC-truth bookkeeping and future Celeritas-owned secondaries. Prompt: "Fix the DD4hep preshower crash in multithreaded Celeritas offload by making reconstructed Geant4 tracks used for DD4hep tracking callback replay provide valid touchable/volume state, while handling terminal Celeritas states that land on or outside the world boundary without aborting." Assisted-by: Codex (GPT-5.5)
Install the DD4hep worker state-dependent initializer only when Geant4 is running as a multithreaded application. In single-threaded runs the master thread also transports tracks, so registering both the master and worker hooks on that thread calls Celeritas BeginOfRunAction twice. Prompt: "Fix the DD4hep -j 1 crash caused by duplicate StateDependent begin-run hooks calling Celeritas BeginOfRunAction twice." Assisted-by: Codex (GPT-5.5)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Title
Support Multithreaded DD4hep Simulation
Summary
This branch adds the multithreaded DD4hep support for tracks offloaded to Celeritas, including partial truth info propagation
StepInterface,StepParams,StepStateData,DetectorSteps, and host/device copyout so terminal track state canbe captured independently of detector hit filtering.
GeantTrackReconstructionto preserve initial offload state, restore event-scoped primaries, expose particle IDs, and support replay of acquiredtracks without changing the existing primary-ID model.
HitProcessorand replayPreUserTrackingAction/PostUserTrackingActionfromLocalTransporter::Flush, applying terminalCeleritas state before post-tracking callbacks when available.
StateDependentlifecycle hooks registered fromCelerPhysics, including worker initialization for MT runs.Testing
GeantTrackReconstructionunit coverage for the acquisition signature and event-scoped replay behavior.