<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body dir="auto">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">Hey Josh,</div>
<div dir="ltr"><br>
</div>
<div dir="ltr">Cassandra and I had noted similar behaviour in the past with GATE (see thread below.) I was using GATE v9.0 on Ubuntu 18.04 with identical observations. Multiple sources and random crashes when running on multiple CPUs. There’s some info below
but a bug was never submitted since I have not had time to creat a minimum working macro that could reproduce the crash. </div>
<div dir="ltr"><br>
</div>
<div dir="ltr">Hope this helps,<br>
<div dir="ltr">Matthew</div>
<div id="AppleMailSignature" dir="ltr"><br>
</div>
<div dir="ltr">Begin forwarded message:</div>
<div dir="ltr"><br>
</div>
<div dir="ltr"><br>
<br>
</div>
<blockquote type="cite">
<div dir="ltr"><b>From:</b> David Sarrut <David.Sarrut@creatis.insa-lyon.fr><br>
<b>Date:</b> February 3, 2021 at 3:44:58 AM AST<br>
<b>Subject:</b> <b>Re: [Gate-users] Simulation crashes unexpectedly</b><br>
<b>Reply-To:</b> David.Sarrut@creatis.insa-lyon.fr<br>
<br>
</div>
</blockquote>
<blockquote type="cite">
<div dir="ltr"><!-- START CAUTION Box Code -->
<table align="center" border="0" cellpadding="0" cellspacing="0" style="padding:10px 0 10px 0" width="100%">
<tbody>
<tr>
<td style="line-height:0px;font-size:0px;mso-line-height-rule:exactly;">
<table align="center" border="0" cellpadding="0" cellspacing="0" style="background:#707372;background-color:#707372;width:100%;border-radius:5px;overflow:hidden;">
<tbody>
<tr>
<td style="border-top:solid 8px #fbe122;padding:4px 8px;text-align:left;vertical-align:top;">
<table role="presentation" border="0" cellpadding="0" cellspacing="0">
<tbody>
<tr>
<td align="left">
<div style="font-family:Arial,sans-serif;font-size:12px;line-height:16px;text-align:left;color:#ffffff;">
<span style="font-weight:bold;font-size:12px;">CAUTION:</span> The Sender of this email is not from within Dalhousie.</div>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
<!-- END CAUTION Box Code -->
<div>
<div dir="ltr">Hello,
<div><br>
</div>
<div>thanks for the information. The feedback from Cassandra scares me a bit ;) </div>
<div>As DavidB said, could you try to have a minimal example with the latest version of Gate (develop branch from the github + G4 10.7) ? </div>
<div>My two cents guess is it may be an issue during the digitization process for SPECT that involve root output. </div>
<div><br>
</div>
<div>thanks</div>
<div>David</div>
<div><br>
</div>
<div> </div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Wed, Feb 3, 2021 at 12:09 AM David Boersma <<a href="mailto:david.boersma@acmit.at">david.boersma@acmit.at</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
Hi Matthew and Cassandra,</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
Segfaults are embarrassing and should be fixed. The backtrace sometimes helps, but I didn't see any clear hints in the one you provided. In order to have a chance to solve it we'd need a minimal Gate macro that enables us to reproduce this problem. Use a fixed
seed and tell us after how long it runs before it crashes for you. It can sometimes help if in addition the versions of Gate, Geant4 and ROOT, you also specify which OS (name and release), which compiler and some specs of the hardware you run it on. Please
post all these details in a GitHub #issue so that it won't get buried in history.<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
Debugging this sort of problems is very necessary (because they are so embarrassing!), but it is also hard and time consuming (run it under "valgrind" and study the very verbose logs), so I don't guarantee that it will be fixed any time soon, even if we have
a GitHub #issue with the full information. Sorry for the inconvenience!</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
/David<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
<br>
</div>
<div id="gmail-m_-4306214783058277778appendonsend"></div>
<hr style="display:inline-block;width:98%">
<div id="gmail-m_-4306214783058277778divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Gate-users <<a href="mailto:gate-users-bounces@lists.opengatecollaboration.org" target="_blank">gate-users-bounces@lists.opengatecollaboration.org</a>>
on behalf of Matthew Strugari <<a href="mailto:matthew.strugari@dal.ca" target="_blank">matthew.strugari@dal.ca</a>><br>
<b>Sent:</b> Tuesday, February 2, 2021 4:23 PM<br>
<b>To:</b> Cassandra Miller <<a href="mailto:cassandramiller@phas.ubc.ca" target="_blank">cassandramiller@phas.ubc.ca</a>>;
<a href="mailto:gate-users@lists.opengatecollaboration.org" target="_blank">gate-users@lists.opengatecollaboration.org</a> <<a href="mailto:gate-users@lists.opengatecollaboration.org" target="_blank">gate-users@lists.opengatecollaboration.org</a>><br>
<b>Subject:</b> Re: [Gate-users] Simulation crashes unexpectedly</font>
<div> </div>
</div>
<div lang="EN-CA" style="overflow-wrap: break-word;">
<div>
<p><span lang="EN-US" style="font-size:11pt">Hi Cassandra,</span></p>
<p><span lang="EN-US" style="font-size:11pt"> </span></p>
<p><span lang="EN-US" style="font-size:11pt">Very strange. I also think it may be a memory issue, although maybe not related to simulation size in my case. My total simulation consumes ~4.6G of RAM out of 64G available. The issue has also never occurred during
dosimetry-related experiments.</span></p>
<p><span lang="EN-US" style="font-size:11pt"> </span></p>
<p><span lang="EN-US" style="font-size:11pt">I noticed that the issue </span><span style="font-size:11pt;color:black">began after I made some changes to my macro file in order to ensure that whole runs were distributed amongst the jobs. This required GATE to
perform a number of internal alias comparisons and assignments. I have since cleaned up the alias and doif commands so hopefully that eliminates the issue since there were never any problems before adding those commands. Does your SPECT configuration utilize
a lot of alias and doif commands as well?</span></p>
<p><span style="font-size:11pt;color:black"> </span></p>
<p><span style="font-size:11pt;color:black">As another developer pointed out to me, the stderr printout means that a pointer to a physical volume pointed to an area of unused memory. I have always used the auto seed generator with the MersenneTwister option.
If the issue persists, then I will manually set the seed to check if it crashes repeatedly. If it doesn’t crash with the same seed, then I am told it may be a problem with my computer.</span></p>
<p><span style="font-size:11pt;color:black"> </span></p>
<p><span style="font-size:11pt;color:black">Thank you for your input!</span></p>
<p><span style="font-size:11pt;color:black">Matthew</span></p>
<p><span lang="EN-US" style="font-size:11pt"> </span></p>
<p><span lang="EN-US" style="font-size:11pt"> </span></p>
<div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb(181,196,223);padding:3pt 0cm 0cm">
<p style="margin-bottom:12pt"><b><span style="color:black">From: </span></b><span style="color:black">Cassandra Miller <<a href="mailto:cassandramiller@phas.ubc.ca" target="_blank">cassandramiller@phas.ubc.ca</a>><br>
<b>Date: </b>Monday, February 1, 2021 at 19:35<br>
<b>To: </b>Matthew Strugari <<a href="mailto:matthew.strugari@dal.ca" target="_blank">matthew.strugari@dal.ca</a>><br>
<b>Subject: </b>Re: [Gate-users] Simulation crashes unexpectedly</span></p>
</div>
<div align="center">
<table border="0" cellspacing="0" cellpadding="0" width="100%" style="width:100%;border-radius:5px;overflow:hidden">
<tbody>
<tr>
<td style="padding:7.5pt 0cm">
<div align="center">
<table border="0" cellspacing="0" cellpadding="0" width="100%" style="width:100%;background:rgb(112,115,114)">
<tbody>
<tr>
<td valign="top" style="border-right:none;border-bottom:none;border-left:none;border-top:6pt solid rgb(251,225,34);padding:3pt 6pt">
<table border="0" cellspacing="0" cellpadding="0">
<tbody>
<tr>
<td style="padding:0cm">
<p style="line-height:12pt"><b><span style="font-size:9pt;font-family:Arial,sans-serif;color:white">CAUTION:</span></b><span style="font-size:9pt;font-family:Arial,sans-serif;color:white"> The Sender of this email is not from within Dalhousie.</span></p>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</div>
</td>
</tr>
</tbody>
</table>
</div>
<div>
<p><span style="color:white">Hi Matthew, </span><br>
<br>
I don't have any insight into why this happens, but it happens to me ALL the time. The frequency it happens depends on the size of the simulation, I think it's a memory thing somehow. It's never happened to me using GATE actors for dosimetry, but it seems to
exclusively happen when I'm doing SPECT imaging.</p>
<p>The more jobs I split my simulation into, the less this happens. So sometimes in the past when I've ran simulations that are ridiculous in size or when I'm using specific sources (for example it happens way more with the fastY90 source than an ion source)
I've split my sim into a stupid amount of jobs and then summed them up at the end. And wrote some scripts to automatically open and list the jobs that have crashed, go inside and change the engine seed number (I find if I change the engine seed it will not
crash when I rerun it, but if I don't change the engine seed it will crash again), and re-run them.<br>
<br>
It sounds a bit ridiculous but.. I've opted to do that instead of figure out what causes the problem in the first place! So I'm sorry I can't help you with the root of the problem but there are ways around it at least.<br>
<br>
Best,<br>
Cassandra</p>
<pre>-- </pre>
<pre> </pre>
<pre>Cassandra Miller</pre>
<pre>PhD Student, Physics </pre>
<pre>Quantitative Radiomolecular Imaging and Therapy Lab</pre>
<pre>Department of Physics and Astronomy, University of British Columbia</pre>
<div>
<p>On 2021-02-01 2:36 p.m., Matthew Strugari wrote:</p>
</div>
<blockquote style="margin-top:5pt;margin-bottom:5pt">
<p><span lang="EN-US" style="font-size:11pt">Hi all,</span></p>
<p><span lang="EN-US" style="font-size:11pt"> </span></p>
<p><span lang="EN-US" style="font-size:11pt">I am having difficulty determining why my simulation is crashing (see attached log file). I am using the task spooler to split my simulation into 12 jobs. Sometimes, one or more jobs will crash at the beginning of
a run, but the run that GATE crashes on seems to be random (jobs have either 5 or 6 runs/projection images). Other times, all of the jobs will run to completion. After a job has crashed, I can reissue the command and the job will run to completion without
error.</span></p>
<p><span lang="EN-US" style="font-size:11pt"> </span></p>
<p><span lang="EN-US" style="font-size:11pt">I cannot figure out why the job(s) seemingly crash at random. Does anyone have any insight into the issue? Your assistance is greatly appreciated.</span></p>
<p><span lang="EN-US" style="font-size:11pt"> </span></p>
<p><span lang="EN-US" style="font-size:11pt">Regards,</span></p>
<p><span lang="EN-US" style="font-size:11pt">Matthew</span></p>
<p><span style="font-size:10.5pt;color:black"> </span></p>
<p><span style="font-size:10.5pt;color:black">--</span></p>
<p><span style="font-size:10.5pt;color:black">Matthew Strugari</span></p>
<p><span lang="EN-US" style="font-size:10.5pt;color:black">Biomedical Translational Imaging Centre - BIOTIC,<br>
5890 University Ave, <br>
Halifax, NS, B3K 6R8</span></p>
<p><span style="font-size:11pt"><br>
<br>
</span></p>
<pre>_______________________________________________</pre>
<pre>Gate-users mailing list</pre>
<pre><a href="mailto:Gate-users@lists.opengatecollaboration.org" target="_blank">Gate-users@lists.opengatecollaboration.org</a></pre>
<pre><a href="http://lists.opengatecollaboration.org/mailman/listinfo/gate-users" target="_blank">http://lists.opengatecollaboration.org/mailman/listinfo/gate-users</a></pre>
</blockquote>
</div>
</div>
</div>
</div>
_______________________________________________<br>
Gate-users mailing list<br>
<a href="mailto:Gate-users@lists.opengatecollaboration.org" target="_blank">Gate-users@lists.opengatecollaboration.org</a><br>
<a href="http://lists.opengatecollaboration.org/mailman/listinfo/gate-users" rel="noreferrer" target="_blank">http://lists.opengatecollaboration.org/mailman/listinfo/gate-users</a></blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr" class="gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">David Sarrut, Phd<br>
Directeur de recherche CNRS<br>
CREATIS, UMR CNRS 5220, Inserm U1294
<div>Centre de lutte contre le cancer Léon Bérard<br>
28 rue Laënnec, 69373 Lyon cedex 08<br>
Tel : 04 78 78 51 51 / 06 74 72 05 42<br>
<a href="http://www.creatis.insa-lyon.fr/~dsarrut" target="_blank">http://dsarrut.github.io</a><br>
_________________________________</div>
<div> "2 + 2 = 5, for extremely large values of 2"<br>
_________________________________</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</body>
</html>