[Gate-users] segfault ITK multithreader

Fri Sep 29 12:14:10 CEST 2017

Dear fellow GATE users,

I have a problem with running multiple parallel GATE simulations. After a
while (around 10 minutes), simulations start segfaulting (stacktrace
attached). This does not happen every or at the same time; sometimes all
simulations finish correctly. It also happens if I don"t start as many
simulations, but I feel not as frequently.

I use the FixedForcedDetection Actor and start around 32 simulations at
once on a 64 thread system. The simulation is attached.

AFAIK, I did not enable multithread mode on any GATE-related program by
hand when installing them.

I am using
 - GATE 7.2
 - root 5.34/30
 - simpleITK 4.10.1
 - RTK 1.3.0

Thank you for your help!

Kind regards
Clemens Schmid
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opengatecollaboration.org/pipermail/gate-users/attachments/20170929/b8bab851/attachment-0001.html>
-------------- next part --------------
 *** Break *** segmentation violation

===========================================================
There was a crash.
This is the entire stack trace of all threads:
===========================================================
#0  0x00007f5974b990cb in __GI___waitpid (pid=5778, stat_loc=stat_loc
entry=0x7fffcfcf4240, options=options
entry=0) at ../sysdeps/unix/sysv/linux/waitpid.c:29
#1  0x00007f5974b11fbb in do_system (line=<optimized out>) at ../sysdeps/posix/system.c:148
#2  0x00007f597bcc9dd4 in TUnixSystem::StackTrace() () from /usr/lib/x86_64-linux-gnu/libCore.so.5.34
#3  0x00007f597bccc03c in TUnixSystem::DispatchSignals(ESignals) () from /usr/lib/x86_64-linux-gnu/libCore.so.5.34
#4  <signal handler called>
#5  0x00007f597a1558d9 in pthread_join (threadid=7410523124669075712, thread_return=0x0) at pthread_join.c:45
#6  0x0000000000bbf7fd in itk::MultiThreader::SpawnWaitForSingleMethodThread(unsigned long) ()
#7  0x0000000000bc00f4 in itk::MultiThreader::SingleMethodExecute() ()
#8  0x0000000000906253 in itk::ImageSource<itk::Image<float, 3u> >::GenerateData() ()
#9  0x0000000000bbc945 in itk::ProcessObject::UpdateOutputData(itk::DataObject*) ()
#10 0x0000000000bc5aad in itk::DataObject::Update() ()
#11 0x0000000000933507 in void GateFixedForcedDetectionActor::ForceDetectionOfInteraction<(ProcessType)0, GateFixedForcedDetectionProjector<GateFixedForcedDetectionFunctor::ComptonValueAccumulation> >(GateFixedForcedDetectionProjector<Gate
FixedForcedDetectionFunctor::ComptonValueAccumulation>*, itk::SmartPointer<itk::Image<float, 3u> >&) ()
#12 0x00000000008ed773 in GateFixedForcedDetectionActor::ForceDetectionOfInteraction(int, int, int, G4String, G4String, G4String, G4String, CLHEP::Hep3Vector, CLHEP::Hep3Vector, double, double, G4String, int) ()
#13 0x00000000008e4d39 in GateFixedForcedDetectionActor::UserSteppingAction(GateVVolume const*, G4Step const*) ()
#14 0x00000000007ac382 in GateVActor::ProcessHits(G4Step*, G4TouchableHistory*) ()
#15 0x00007f5976fa3ea8 in G4VPrimitiveScorer::HitPrimitive (ROhis=0x0, aStep=0x27b3b00, this=0x2c64110) at /tmp/gate-install-build/geant4/source/digits_hits/detector/include/G4VPrimitiveScorer.hh:119
#16 G4MultiFunctionalDetector::ProcessHits (this=<optimized out>, aStep=0x27b3b00, aTH=0x0) at /tmp/gate-install-build/geant4/source/digits_hits/detector/src/G4MultiFunctionalDetector.cc:52
#17 0x000000000075281f in GateMultiSensitiveDetector::ProcessHits(G4Step*, G4TouchableHistory*) ()
#18 0x00007f597901687d in G4VSensitiveDetector::Hit (aStep=0x27b3b00, this=0x2c92e00) at /tmp/gate-install-build/geant4/source/digits_hits/detector/include/G4VSensitiveDetector.hh:122
#19 G4SteppingManager::Stepping (this=0x27b3960) at /tmp/gate-install-build/geant4/source/tracking/src/G4SteppingManager.cc:237
#20 0x00007f597902110d in G4TrackingManager::ProcessOneTrack (this=0x27b3920, apValueG4Track=apValueG4Track
entry=0x52fee20) at /tmp/gate-install-build/geant4/source/tracking/src/G4TrackingManager.cc:126
#21 0x00007f597925c4e6 in G4EventManager::DoProcessing (this=0x27b3880, anEvent=<optimized out>) at /tmp/gate-install-build/geant4/source/event/src/G4EventManager.cc:184
#22 0x00007f597925ccdc in G4EventManager::ProcessOneEvent (this=<optimized out>, anEvent=<optimized out>) at /tmp/gate-install-build/geant4/source/event/src/G4EventManager.cc:335
#23 0x00007f59794e2ec7 in G4RunManager::ProcessOneEvent (this=0x2639f80, i_event=245240) at /tmp/gate-install-build/geant4/source/run/src/G4RunManager.cc:399
#24 0x00007f59794e16c3 in G4RunManager::DoEventLoop (this=0x2639f80, n_event=2147483647, macroFile=<optimized out>, n_select=<optimized out>) at /tmp/gate-install-build/geant4/source/run/src/G4RunManager.cc:367
#25 0x00007f59794e13ee in G4RunManager::BeamOn (this=0x2639f80, n_event=2147483647, macroFile=0x0, n_select=-1) at /tmp/gate-install-build/geant4/source/run/src/G4RunManager.cc:273
#26 0x0000000000aee0a0 in GateApplicationMgr::StartDAQ() ()
#27 0x0000000000b08018 in GateApplicationMgrMessenger::SetNewValue(G4UIcommand*, G4String) ()
#28 0x00007f5975ea3a3d in G4UIcommand::DoIt (this=0x2b23b60, parameterList=...) at /tmp/gate-install-build/geant4/source/intercoms/src/G4UIcommand.cc:230
#29 0x00007f5975eb72ac in G4UImanager::ApplyCommand (this=0x2055350, aCmd=<optimized out>) at /tmp/gate-install-build/geant4/source/intercoms/src/G4UImanager.cc:523
#30 0x00007f5975eb7978 in G4UImanager::ApplyCommand (this=<optimized out>, aCmd=...) at /tmp/gate-install-build/geant4/source/intercoms/src/G4UImanager.cc:441
#31 0x00007f5975e94817 in G4UIbatch::ExecCommand (this=this
entry=0x2b29700, command=...) at /tmp/gate-install-build/geant4/source/intercoms/src/G4UIbatch.cc:170
#32 0x00007f5975e95a0e in G4UIbatch::SessionStart (this=0x2b29700) at /tmp/gate-install-build/geant4/source/intercoms/src/G4UIbatch.cc:215
#33 0x00007f5975eb7cc2 in G4UImanager::ExecuteMacroFile (this=this
entry=0x2055350, fileName=0x7fffcfcf8980 "mac/main.mac") at /tmp/gate-install-build/geant4/source/intercoms/src/G4UImanager.cc:296
#34 0x00007f5975eaada0 in G4UIcontrolMessenger::SetNewValue (this=0x2055780, command=command
entry=0x2055fd0, newValue=...) at /tmp/gate-install-build/geant4/source/intercoms/src/G4UIcontrolMessenger.cc:313
#35 0x00007f5975ea3a3d in G4UIcommand::DoIt (this=0x2055fd0, parameterList=...) at /tmp/gate-install-build/geant4/source/intercoms/src/G4UIcommand.cc:230
#36 0x00007f5975eb72ac in G4UImanager::ApplyCommand (this=0x2055350, aCmd=<optimized out>) at /tmp/gate-install-build/geant4/source/intercoms/src/G4UImanager.cc:523
#37 0x0000000000621463 in main ()
===========================================================

The lines below might hint at the cause of the crash.
If they do not help you then please submit a bug report at
http://root.cern.ch/bugs. Please post the ENTIRE stack trace
from above as an attachment in addition to anything else
that might help us fixing this issue.
===========================================================
#5  0x00007f597a1558d9 in pthread_join (threadid=7410523124669075712, thread_return=0x0) at pthread_join.c:45
#6  0x0000000000bbf7fd in itk::MultiThreader::SpawnWaitForSingleMethodThread(unsigned long) ()
#7  0x0000000000bc00f4 in itk::MultiThreader::SingleMethodExecute() ()
#8  0x0000000000906253 in itk::ImageSource<itk::Image<float, 3u> >::GenerateData() ()
#9  0x0000000000bbc945 in itk::ProcessObject::UpdateOutputData(itk::DataObject*) ()
#10 0x0000000000bc5aad in itk::DataObject::Update() ()
#11 0x0000000000933507 in void GateFixedForcedDetectionActor::ForceDetectionOfInteraction<(ProcessType)0, GateFixedForcedDetectionProjector<GateFixedForcedDetectionFunctor::ComptonValueAccumulation> >(GateFixedForcedDetectionProjector<GateFixedForcedDetectionFunctor::ComptonValueAccumulation>*, itk::SmartPointer<itk::Image<float, 3u> >&) ()
#12 0x00000000008ed773 in GateFixedForcedDetectionActor::ForceDetectionOfInteraction(int, int, int, G4String, G4String, G4String, G4String, CLHEP::Hep3Vector, CLHEP::Hep3Vector, double, double, G4String, int) ()
#13 0x00000000008e4d39 in GateFixedForcedDetectionActor::UserSteppingAction(GateVVolume const*, G4Step const*) ()
#14 0x00000000007ac382 in GateVActor::ProcessHits(G4Step*, G4TouchableHistory*) ()
#15 0x00007f5976fa3ea8 in G4VPrimitiveScorer::HitPrimitive (ROhis=0x0, aStep=0x27b3b00, this=0x2c64110) at /tmp/gate-install-build/geant4/source/digits_hits/detector/include/G4VPrimitiveScorer.hh:119
#16 G4MultiFunctionalDetector::ProcessHits (this=<optimized out>, aStep=0x27b3b00, aTH=0x0) at /tmp/gate-install-build/geant4/source/digits_hits/detector/src/G4MultiFunctionalDetector.cc:52
#17 0x000000000075281f in GateMultiSensitiveDetector::ProcessHits(G4Step*, G4TouchableHistory*) ()
#18 0x00007f597901687d in G4VSensitiveDetector::Hit (aStep=0x27b3b00, this=0x2c92e00) at /tmp/gate-install-build/geant4/source/digits_hits/detector/include/G4VSensitiveDetector.hh:122
#19 G4SteppingManager::Stepping (this=0x27b3960) at /tmp/gate-install-build/geant4/source/tracking/src/G4SteppingManager.cc:237
#20 0x00007f597902110d in G4TrackingManager::ProcessOneTrack (this=0x27b3920, apValueG4Track=apValueG4Track
entry=0x52fee20) at /tmp/gate-install-build/geant4/source/tracking/src/G4TrackingManager.cc:126
#21 0x00007f597925c4e6 in G4EventManager::DoProcessing (this=0x27b3880, anEvent=<optimized out>) at /tmp/gate-install-build/geant4/source/event/src/G4EventManager.cc:184
#22 0x00007f597925ccdc in G4EventManager::ProcessOneEvent (this=<optimized out>, anEvent=<optimized out>) at /tmp/gate-install-build/geant4/source/event/src/G4EventManager.cc:335
#23 0x00007f59794e2ec7 in G4RunManager::ProcessOneEvent (this=0x2639f80, i_event=245240) at /tmp/gate-install-build/geant4/source/run/src/G4RunManager.cc:399
#24 0x00007f59794e16c3 in G4RunManager::DoEventLoop (this=0x2639f80, n_event=2147483647, macroFile=<optimized out>, n_select=<optimized out>) at /tmp/gate-install-build/geant4/source/run/src/G4RunManager.cc:367
#25 0x00007f59794e13ee in G4RunManager::BeamOn (this=0x2639f80, n_event=2147483647, macroFile=0x0, n_select=-1) at /tmp/gate-install-build/geant4/source/run/src/G4RunManager.cc:273
#26 0x0000000000aee0a0 in GateApplicationMgr::StartDAQ() ()
#27 0x0000000000b08018 in GateApplicationMgrMessenger::SetNewValue(G4UIcommand*, G4String) ()
#28 0x00007f5975ea3a3d in G4UIcommand::DoIt (this=0x2b23b60, parameterList=...) at /tmp/gate-install-build/geant4/source/intercoms/src/G4UIcommand.cc:230
#29 0x00007f5975eb72ac in G4UImanager::ApplyCommand (this=0x2055350, aCmd=<optimized out>) at /tmp/gate-install-build/geant4/source/intercoms/src/G4UImanager.cc:523
#30 0x00007f5975eb7978 in G4UImanager::ApplyCommand (this=<optimized out>, aCmd=...) at /tmp/gate-install-build/geant4/source/intercoms/src/G4UImanager.cc:441
#31 0x00007f5975e94817 in G4UIbatch::ExecCommand (this=this
entry=0x2b29700, command=...) at /tmp/gate-install-build/geant4/source/intercoms/src/G4UIbatch.cc:170
#32 0x00007f5975e95a0e in G4UIbatch::SessionStart (this=0x2b29700) at /tmp/gate-install-build/geant4/source/intercoms/src/G4UIbatch.cc:215
#33 0x00007f5975eb7cc2 in G4UImanager::ExecuteMacroFile (this=this
entry=0x2055350, fileName=0x7fffcfcf8980 "mac/main.mac") at /tmp/gate-install-build/geant4/source/intercoms/src/G4UImanager.cc:296
#34 0x00007f5975eaada0 in G4UIcontrolMessenger::SetNewValue (this=0x2055780, command=command
entry=0x2055fd0, newValue=...) at /tmp/gate-install-build/geant4/source/intercoms/src/G4UIcontrolMessenger.cc:313
#35 0x00007f5975ea3a3d in G4UIcommand::DoIt (this=0x2055fd0, parameterList=...) at /tmp/gate-install-build/geant4/source/intercoms/src/G4UIcommand.cc:230
#36 0x00007f5975eb72ac in G4UImanager::ApplyCommand (this=0x2055350, aCmd=<optimized out>) at /tmp/gate-install-build/geant4/source/intercoms/src/G4UImanager.cc:523
#37 0x0000000000621463 in main ()
===========================================================

-------------- next part --------------
A non-text attachment was scrubbed...
Name: simu.zip
Type: application/zip
Size: 238381 bytes
Desc: not available
URL: <http://lists.opengatecollaboration.org/pipermail/gate-users/attachments/20170929/b8bab851/attachment-0001.zip>