Skip to content

Mutex-free TGeo material budget in Propagator for multithreaded users - #16146

Merged
sawenzel merged 1 commit into
AliceO2Group:devfrom
shahor02:pr_propMT
Nov 17, 2026
Merged

sawenzel merged 1 commit into
AliceO2Group:devfrom
shahor02:pr_propMT

Conversation

@shahor02

@shahor02 shahor02 commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

Actually, further reviews revealed some corner cases to avoid, moving here the final comment of the commit.

Mutex-free TGeo material budget for multithreaded users GeometryManager::meanMaterialBudget(Ext) used the shared TGeo navigator under sTGMutex, serializing all callers (Propagator TGeo branch, LUT building).
Add opt-in per-thread navigators:

  • GeometryManager::setMaxThreads(n): wraps TGeoManager::SetMaxThreads and enables per-thread navigation; returns the effective capacity.
  • GeometryManager::getThreadNavigator(): thread_local navigator, created on first use, removed at thread exit. nullptr if not enabled. Throws std::runtime_error if more threads than declared ask for one.
  • meanMaterialBudget / meanMaterialBudgetExt (and thus Propagator, unchanged) take the calling thread's navigator when enabled, otherwise the shared one under the mutex, as before. meanMaterialBudgetExt now navigates via a TGeoNavigator instead of the gGeoManager global API.
  • getOriginalMatrix / getOriginalMatrixFromPath navigate the same way (they also no longer re-lock sTGMutex when getOriginalMatrix(symname) falls back to getOriginalMatrixFromPath, which self-deadlocked).
  • rebuildVoxels(): the voxels, shared by all navigators and rebuilt lazily after TGeoPhysicalNode::Align, are rebuilt serially (also in setMaxThreads).
  • MatLayerCylSet LUT build uses the same mechanism and limits its parallelism to the enabled capacity.
  • Unit test testGeometryManagerMT.

How to use in code calling Propagator (or meanMaterialBudget) from threads:

  1. Load the geometry (and apply alignment).
  2. Serially, BEFORE starting the worker threads, call once o2::base::GeometryManager::setMaxThreads(nThreads);
    with nThreads >= the maximum number of threads which will ever use TGeo in the process (workers other than the calling thread).
  3. Call Propagator methods (USEMatCorrTGeo or TGeo fall-back without LUT) from the threads as usual: no further change needed.
    Without step 2 behaviour is unchanged (thread-safe but mutexed).
    MatCorrType::USEMatCorrLUT never touches TGeo.

Corner cases to avoid (TGeoManager is not designed for them):

  • setMaxThreads can be called effectively once per geometry: TGeo would reset the thread slots of threads already navigating. Repeated calls with n <= capacity are no-ops, a larger n is refused with an error.
  • TGeo thread slots (TGeoManager::ThreadId) are never released: every distinct thread which navigated keeps its slot after exit. With thread churn (short-lived threads) use a thread pool, otherwise slots run out and getThreadNavigator throws. Never silently fall back to the shared navigator in multithread mode: it would index the per-volume thread data with the calling thread's slot, possibly out of range.
  • Direct use of the gGeoManager navigation API (cd, InitTrack, GetCurrentNode, ...) from a worker thread is safe only after that thread has called getThreadNavigator() (or called setMaxThreads): GetCurrentNavigator returns nullptr for threads without a navigator. Prefer calling the methods of the navigator returned by getThreadNavigator(). Any TGeo navigation from a new thread consumes a slot.
  • TGeoPhysicalNode::Align after setMaxThreads: call rebuildVoxels() while no thread navigates, otherwise the lazy rebuild races between the threads.
  • Replacing gGeoManager disables multithreading (detected via IsMultiThread(), not only the pointer) and the stale navigators are dropped. The slot map is static in TGeo and not reset: threads which navigated before must not be reused and enabling again may find slots already consumed (warning issued).
  • setMaxThreads, rebuildVoxels and geometry replacement need all other threads to be idle.

Co-Authored-By: Claude Sonnet 5.5

@shahor02
shahor02 requested a review from sawenzel as a code owner November 17, 2026 13:57
GeometryManager::meanMaterialBudget(Ext) used the shared TGeo navigator under
sTGMutex, serializing all callers (Propagator TGeo branch, LUT building).
Add opt-in per-thread navigators:

 * GeometryManager::setMaxThreads(n): wraps TGeoManager::SetMaxThreads and
   enables per-thread navigation; returns the effective capacity.
 * GeometryManager::getThreadNavigator(): thread_local navigator, created on
   first use, removed at thread exit. nullptr if not enabled. Throws
   std::runtime_error if more threads than declared ask for one.
 * meanMaterialBudget / meanMaterialBudgetExt (and thus Propagator, unchanged)
   take the calling thread's navigator when enabled, otherwise the shared
   one under the mutex, as before. meanMaterialBudgetExt now navigates via a
   TGeoNavigator instead of the gGeoManager global API.
 * getOriginalMatrix / getOriginalMatrixFromPath navigate the same way (they
   also no longer re-lock sTGMutex when getOriginalMatrix(symname) falls back
   to getOriginalMatrixFromPath, which self-deadlocked).
 * rebuildVoxels(): the voxels, shared by all navigators and rebuilt lazily
   after TGeoPhysicalNode::Align, are rebuilt serially (also in setMaxThreads).
 * MatLayerCylSet LUT build uses the same mechanism and limits its
   parallelism to the enabled capacity.
 * Unit test testGeometryManagerMT.

How to use in code calling Propagator (or meanMaterialBudget) from threads:

  1. Load the geometry (and apply alignment).
  2. Serially, BEFORE starting the worker threads, call once
       o2::base::GeometryManager::setMaxThreads(nThreads);
     with nThreads >= the maximum number of threads which will ever use TGeo
     in the process (workers other than the calling thread).
  3. Call Propagator methods (USEMatCorrTGeo or TGeo fall-back without LUT)
     from the threads as usual: no further change needed.
  Without step 2 behaviour is unchanged (thread-safe but mutexed).
  MatCorrType::USEMatCorrLUT never touches TGeo.

Corner cases to avoid (TGeoManager is not designed for them):
  * setMaxThreads can be called effectively once per geometry: TGeo would
    reset the thread slots of threads already navigating. Repeated calls with
    n <= capacity are no-ops, a larger n is refused with an error.
  * TGeo thread slots (TGeoManager::ThreadId) are never released: every
    distinct thread which navigated keeps its slot after exit. With thread
    churn (short-lived threads) use a thread pool, otherwise slots run out and
    getThreadNavigator throws. Never silently fall back to the shared
    navigator in multithread mode: it would index the per-volume thread data
    with the calling thread's slot, possibly out of range.
  * Direct use of the gGeoManager navigation API (cd, InitTrack, GetCurrentNode,
    ...) from a worker thread is safe only after that thread has called
    getThreadNavigator() (or called setMaxThreads): GetCurrentNavigator
    returns nullptr for threads without a navigator. Prefer calling the
    methods of the navigator returned by getThreadNavigator(). Any TGeo
    navigation from a new thread consumes a slot.
  * TGeoPhysicalNode::Align after setMaxThreads: call rebuildVoxels() while no
    thread navigates, otherwise the lazy rebuild races between the threads.
  * Replacing gGeoManager disables multithreading (detected via
    IsMultiThread(), not only the pointer) and the stale navigators are
    dropped. The slot map is static in TGeo and not reset: threads which
    navigated before must not be reused and enabling again may find slots
    already consumed (warning issued).
  * setMaxThreads, rebuildVoxels and geometry replacement need all other
    threads to be idle.

Co-Authored-By: Claude Sonnet 5.5
@sawenzel
sawenzel merged commit 6de3742 into AliceO2Group:dev Nov 17, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants