[Gate-users] Frustrating Error, Fwd: Simulation crashes unexpectedly

Fri Oct 15 03:19:32 CEST 2021

Hi Matthew,
Thank you for the info. That sounds a lot like what I'm experiencing.

I got my sim to run without error finally. It's probably unrelated, but the only thing I did differently was to run "Gate MacroName.mac" from bash instead of redirecting to a log file like I usually do ("Gate MacroName.mac > log.txt")

I'm not using actors or aliases, but I am using Mersenne Twister with an automatic seed. I'll try changing that if I can replicate it.

Thanks again!
Josh

________________________________
From: Matthew Strugari <matthew.strugari at dal.ca>
Sent: Thursday, October 14, 2021 9:10:55 PM
To: Gate-users <gate-users at lists.opengatecollaboration.org>; Josh Knowland <jknowland at lucernodynamics.com>
Subject: Re: [Gate-users] Frustrating Error, Fwd: [Gate-users] Simulation crashes unexpectedly

Hey Josh,

Cassandra and I had noted similar behaviour in the past with GATE (see thread below.) I was using GATE v9.0 on Ubuntu 18.04 with identical observations. Multiple sources and random crashes when running on multiple CPUs. There’s some info below but a bug was never submitted since I have not had time to creat a minimum working macro that could reproduce the crash.

Hope this helps,
Matthew

Begin forwarded message:

From: David Sarrut <David.Sarrut at creatis.insa-lyon.fr>
Date: February 3, 2021 at 3:44:58 AM AST
Subject: Re: [Gate-users] Simulation crashes unexpectedly
Reply-To: David.Sarrut at creatis.insa-lyon.fr

CAUTION: The Sender of this email is not from within Dalhousie.
Hello,

thanks for the information. The feedback from Cassandra scares me a bit ;)
As DavidB said, could you try to have a minimal example with the latest version of Gate (develop branch from the github + G4 10.7) ?
My two cents guess is it may be an issue during the digitization process for SPECT that involve root output.

thanks
David

On Wed, Feb 3, 2021 at 12:09 AM David Boersma <david.boersma at acmit.at<mailto:david.boersma at acmit.at>> wrote:
Hi Matthew and Cassandra,

Segfaults are embarrassing and should be fixed. The backtrace sometimes helps, but I didn't see any clear hints in the one you provided. In order to have a chance to solve it we'd need a minimal Gate macro that enables us to reproduce this problem. Use a fixed seed and tell us after how long it runs before it crashes for you. It can sometimes help if in addition the versions of Gate, Geant4 and ROOT, you also specify which OS (name and release), which compiler and some specs of the hardware you run it on. Please post all these details in a GitHub #issue so that it won't get buried in history.

Debugging this sort of problems is very necessary (because they are so embarrassing!), but it is also hard and time consuming (run it under "valgrind" and study the very verbose logs), so I don't guarantee that it will be fixed any time soon, even if we have a GitHub #issue with the full information. Sorry for the inconvenience!

/David

________________________________
From: Gate-users <gate-users-bounces at lists.opengatecollaboration.org<mailto:gate-users-bounces at lists.opengatecollaboration.org>> on behalf of Matthew Strugari <matthew.strugari at dal.ca<mailto:matthew.strugari at dal.ca>>
Sent: Tuesday, February 2, 2021 4:23 PM
To: Cassandra Miller <cassandramiller at phas.ubc.ca<mailto:cassandramiller at phas.ubc.ca>>; gate-users at lists.opengatecollaboration.org<mailto:gate-users at lists.opengatecollaboration.org> <gate-users at lists.opengatecollaboration.org<mailto:gate-users at lists.opengatecollaboration.org>>
Subject: Re: [Gate-users] Simulation crashes unexpectedly

Hi Cassandra,

Very strange. I also think it may be a memory issue, although maybe not related to simulation size in my case. My total simulation consumes ~4.6G of RAM out of 64G available. The issue has also never occurred during dosimetry-related experiments.

I noticed that the issue began after I made some changes to my macro file in order to ensure that whole runs were distributed amongst the jobs. This required GATE to perform a number of internal alias comparisons and assignments. I have since cleaned up the alias and doif commands so hopefully that eliminates the issue since there were never any problems before adding those commands. Does your SPECT configuration utilize a lot of alias and doif commands as well?

As another developer pointed out to me, the stderr printout means that a pointer to a physical volume pointed to an area of unused memory. I have always used the auto seed generator with the MersenneTwister option. If the issue persists, then I will manually set the seed to check if it crashes repeatedly. If it doesn’t crash with the same seed, then I am told it may be a problem with my computer.

Thank you for your input!

Matthew

From: Cassandra Miller <cassandramiller at phas.ubc.ca<mailto:cassandramiller at phas.ubc.ca>>
Date: Monday, February 1, 2021 at 19:35
To: Matthew Strugari <matthew.strugari at dal.ca<mailto:matthew.strugari at dal.ca>>
Subject: Re: [Gate-users] Simulation crashes unexpectedly

CAUTION: The Sender of this email is not from within Dalhousie.

Hi Matthew,

I don't have any insight into why this happens, but it happens to me ALL the time. The frequency it happens depends on the size of the simulation, I think it's a memory thing somehow. It's never happened to me using GATE actors for dosimetry, but it seems to exclusively happen when I'm doing SPECT imaging.

The more jobs I split my simulation into, the less this happens. So sometimes in the past when I've ran simulations that are ridiculous in size or when I'm using specific sources (for example it happens way more with the fastY90 source than an ion source) I've split my sim into a stupid amount of jobs and then summed them up at the end. And wrote some scripts to automatically open and list the jobs that have crashed, go inside and change the engine seed number (I find if I change the engine seed it will not crash when I rerun it, but if I don't change the engine seed it will crash again), and re-run them.

It sounds a bit ridiculous but.. I've opted to do that instead of figure out what causes the problem in the first place! So I'm sorry I can't help you with the root of the problem but there are ways around it at least.

Best,
Cassandra

--

Cassandra Miller

PhD Student, Physics

Quantitative Radiomolecular Imaging and Therapy Lab

Department of Physics and Astronomy, University of British Columbia

On 2021-02-01 2:36 p.m., Matthew Strugari wrote:

Hi all,

I am having difficulty determining why my simulation is crashing (see attached log file). I am using the task spooler to split my simulation into 12 jobs. Sometimes, one or more jobs will crash at the beginning of a run, but the run that GATE crashes on seems to be random (jobs have either 5 or 6 runs/projection images). Other times, all of the jobs will run to completion. After a job has crashed, I can reissue the command and the job will run to completion without error.

I cannot figure out why the job(s) seemingly crash at random. Does anyone have any insight into the issue? Your assistance is greatly appreciated.

Regards,

Matthew

--

Matthew Strugari

Biomedical Translational Imaging Centre - BIOTIC,
5890 University Ave,
Halifax, NS, B3K 6R8

_______________________________________________

Gate-users mailing list

Gate-users at lists.opengatecollaboration.org<mailto:Gate-users at lists.opengatecollaboration.org>

http://lists.opengatecollaboration.org/mailman/listinfo/gate-users

_______________________________________________
Gate-users mailing list
Gate-users at lists.opengatecollaboration.org<mailto:Gate-users at lists.opengatecollaboration.org>
http://lists.opengatecollaboration.org/mailman/listinfo/gate-users

--
David Sarrut, Phd
Directeur de recherche CNRS
CREATIS, UMR CNRS 5220, Inserm U1294
Centre de lutte contre le cancer Léon Bérard
28 rue Laënnec, 69373 Lyon cedex 08
Tel : 04 78 78 51 51 / 06 74 72 05 42
http://dsarrut.github.io<http://www.creatis.insa-lyon.fr/~dsarrut>
_________________________________
 "2 + 2 = 5,  for extremely large values of 2"
_________________________________
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opengatecollaboration.org/pipermail/gate-users/attachments/20211015/6e35eaa1/attachment-0001.html>