<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">Hi Mark,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">This is something I’ve worked on a fair bit and here’s what I came up with. I run my simulations remotely on a 64-core virtual machine through Azure, but the process should be the same.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Note that this method works well with the new unified Tree output in GATE 9, but also works with the ‘old’ output files and previous versions of GATE.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Please note! I do not use ROOT file outputs. I have started using the NUMPY file outputs for singles and hits so that I can do custom analysis. This process should work just fine with ROOT files, though, and I talk about that a bit at the
end of the email.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">The overall idea is to split your macro in time and then combine the results from each. Generate a macro that does what I want. Create multiple copies of the macro,
<b>each with a different Tree output filename</b>. All other simulation parameters can be the same (volume names, etc. do not need to change). Run all the macros concurrently using a shell script like below. Each process will be put on a different core and
htop will display CPU usage (F10 to exit htop).<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal" style="margin-left:.25in"><span style="font-size:10.0pt;font-family:"Courier New"">#!/bin/bash<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.25in"><span style="font-size:10.0pt;font-family:"Courier New"">(<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.25in;text-indent:.25in"><span style="font-size:10.0pt;font-family:"Courier New"">(<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New"">Gate macro_00.mac > macro_00_log.txt &<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New"">Gate macro_01.mac > macro_01_log.txt &<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New"">Gate macro_02.mac > macro_02_log.txt &<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New"">Gate macro_03.mac > macro_03_log.txt &<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New"">Gate macro_04.mac > macro_04_log.txt &<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New"">Gate macro_05.mac > macro_05_log.txt &<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New"">Gate macro_06.mac > macro_06_log.txt &<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.0in"><span style="font-size:10.0pt;font-family:"Courier New"">Gate macro_07.mac > macro_07_log.txt &<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.75in;text-indent:.25in"><span style="font-size:10.0pt;font-family:"Courier New"">wait<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.25in;text-indent:.25in"><span style="font-size:10.0pt;font-family:"Courier New"">)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.25in"><span style="font-size:10.0pt;font-family:"Courier New"">) &<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.25in"><span style="font-size:10.0pt;font-family:"Courier New"">htop<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal">An alternative approach is to use parameterized macros. For this, you’d create one macro with
<b>GATE parameters</b> as place holders using braces. Then, the bash script file would call that macro multiple times while changing the parameters. For instance, you could run a simulation and split it over time by parameterizing
<b>timestart</b> and <b>timestop</b> like this in the GATE macro: <o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Courier New"">/gate/application/setTimeSlice 0.125 s<br>
/gate/application/setTimeStart {timestart} s<br>
/gate/application/setTimeStop {timestop} s<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal">Then the Bash script would define the parameters for each process:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:10.0pt;font-family:"Courier New"">#!/bin/bash<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:10.0pt;font-family:"Courier New"">(<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-indent:.5in"><span style="font-size:10.0pt;font-family:"Courier New"">(<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.5in"><span style="font-size:10.0pt;font-family:"Courier New"">Gate -a [timestart,0.000][timestop,0.125] paramMacro.mac > output_00_log.txt &
<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.5in"><span style="font-size:10.0pt;font-family:"Courier New"">Gate -a [timestart,0.125][timestop,0.250] paramMacro.mac > output_01_log.txt &
<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.5in"><span style="font-size:10.0pt;font-family:"Courier New"">Gate -a [timestart,0.250][timestop,0.375] paramMacro.mac > output_02_log.txt &
<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.5in"><span style="font-size:10.0pt;font-family:"Courier New"">Gate -a [timestart,0.375][timestop,0.500] paramMacro.mac > output_03_log.txt &
<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.5in"><span style="font-size:10.0pt;font-family:"Courier New"">Gate -a [timestart,0.500][timestop,0.625] paramMacro.mac > output_04_log.txt &
<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.5in"><span style="font-size:10.0pt;font-family:"Courier New"">Gate -a [timestart,0.625][timestop,0.750] paramMacro.mac > output_05_log.txt &
<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.5in"><span style="font-size:10.0pt;font-family:"Courier New"">Gate -a [timestart,0.750][timestop,0.875] paramMacro.mac > output_06_log.txt &
<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.5in"><span style="font-size:10.0pt;font-family:"Courier New"">Gate -a [timestart,0.875][timestop,1.000] paramMacro.mac > output_07_log.txt &
<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.5in"><span style="font-size:10.0pt;font-family:"Courier New"">wait<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-indent:.5in"><span style="font-size:10.0pt;font-family:"Courier New"">)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:10.0pt;font-family:"Courier New"">wait<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:10.0pt;font-family:"Courier New"">) &<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:10.0pt;font-family:"Courier New"">htop<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Either way, you will end up with multiple output files:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal" style="text-indent:.5in"><span style="font-size:10.0pt;font-family:"Courier New"">output_00_log.txt<o:p></o:p></span></p>
<p class="MsoNormal" style="text-indent:.5in"><span style="font-size:10.0pt;font-family:"Courier New"">output_00.Run.dat<o:p></o:p></span></p>
<p class="MsoNormal" style="text-indent:.5in"><span style="font-size:10.0pt;font-family:"Courier New"">output_00.root<o:p></o:p></span></p>
<p class="MsoNormal" style="text-indent:.5in"><span style="font-size:10.0pt;font-family:"Courier New"">etc...</span><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">If you’re using text-based output files, there are several ways within Ubuntu to combine them.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">To combine root files, you can use the ‘hadd’ application which should be available to you:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> hadd - Merge ROOT histogram files into one file<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> hadd outputfile inputfiles ...<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">
<o:p></o:p></span></p>
<p class="MsoNormal" style="text-indent:.5in"><span style="font-size:10.0pt;font-family:"Courier New"">This program will add histograms from a list of ROOT files and write them to a target ROOT<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> file. The target file is newly created and must not be identical to one of the source files.<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I hope that helps! Feel free to ask questions.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> Gate-users <gate-users-bounces@lists.opengatecollaboration.org>
<b>On Behalf Of </b>BAKER, Mark (THE CLATTERBRIDGE CANCER CENTRE NHS FOUNDATION TRUST)<br>
<b>Sent:</b> Thursday, November 18, 2021 8:04 AM<br>
<b>To:</b> gate-users@lists.opengatecollaboration.org<br>
<b>Subject:</b> [Gate-users] Job splitter etc. on vGate<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span lang="EN-GB">Hi everyone<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">I have a macro up and running using vGate 9.0 which imports PET and CT data to generate a dose distribution. It seems to be working fine, but only uses one CPU core (which I believe is a known “feature” of Gate). My computer
has 8 cores acting as 16 logical processors and Oracle VM currently has access to 8 cores. I’m not sure if this is the full 8 physical (one logical from each core) or 4 physical (two logical from each). An obvious speeding up of the process would be to use
the jobsplitter and filemerger in order to utilise more cores, but I’m not sure if this is a possibility within vGate 9.0.
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">In the ReadTheDocs documentation, there is a section on using job splitter etc. on a cluster but this requires additional installation, plus a “cluster aware version of Gate”. My gut instinct is that vGate 9.0 isn’t one
of those versions but I was wondering if anyone had been successful or has any other tips on getting vGate to utilise my cpu power more effectively?<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">Thank you<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span lang="EN-GB" style="color:#006871;mso-fareast-language:EN-GB">Mark Baker<o:p></o:p></span></b></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#006871;mso-fareast-language:EN-GB">Senior Clinical Scientist<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#006871;mso-fareast-language:EN-GB">Imaging Physics (Ionising)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#006871;mso-fareast-language:EN-GB">The Clatterbridge Cancer Centre NHS Foundation Trust<o:p></o:p></span></p>
<p class="MsoNormal"><b><span lang="EN-GB" style="color:#006871;mso-fareast-language:EN-GB"><o:p> </o:p></span></b></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#006871;mso-fareast-language:EN-GB">CCCW Tel: 0151 556 5030
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#006871;mso-fareast-language:EN-GB">CCCL Tel: 0151 318 8438<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#006871;mso-fareast-language:EN-GB">Email:
</span><span lang="EN-GB" style="color:#4472C4;mso-fareast-language:EN-GB"><a href="mailto:mark.baker23@nhs.net"><span style="color:#4472C4">mark.baker23@nhs.net</span></a></span><span lang="EN-GB" style="color:#006871;mso-fareast-language:EN-GB">
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#4472C4;mso-fareast-language:EN-GB"><a href="https://teams.microsoft.com/l/chat/0/0?users=mark.baker23@nhs.net"><span style="color:#4472C4">Microsoft Teams (click here)</span></a><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="mso-fareast-language:EN-GB"> <span style="color:#006871"><o:p></o:p></span></span></p>
<p class="MsoNormal"><i><span lang="EN-GB" style="color:#006871;mso-fareast-language:EN-GB">Honorary Lecturer, Dept. of Physics<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span lang="EN-GB" style="color:#006871;mso-fareast-language:EN-GB">University of Liverpool<o:p></o:p></span></i></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#006871;mso-fareast-language:EN-GB"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="mso-fareast-language:EN-GB"><img border="0" width="624" height="163" style="width:6.5in;height:1.6979in" id="Picture_x0020_3" src="cid:image001.jpg@01D7DC95.A53598F0" alt="CCC email footer April 2021"></span><span lang="EN-GB" style="mso-fareast-language:EN-GB"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="mso-fareast-language:EN-GB"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span lang="EN-GB"><br>
</span><span lang="EN-GB" style="font-size:10.0pt;font-family:"Courier New";color:black"><br>
************************************************************************************** ******************************<br>
<br>
This message may contain confidential information. If you are not the intended recipient please:<br>
i) inform the sender that you have received the message in error before deleting it; and
<br>
ii) do not disclose, copy or distribute information in this e-mail or take any action in relation to its content (to do so is strictly prohibited and may be unlawful).
<br>
Thank you for your co-operation.<br>
<br>
NHSmail is the secure email, collaboration and directory service available for all NHS staff in England. NHSmail is approved for exchanging patient data and other sensitive information with NHSmail and other accredited email services.<br>
<br>
For more information and to find out how you can switch visit <a href="https://support.nhs.net/article-categories/joining-nhsmail/">
Joining NHSmail – NHSmail Support</a></span><span lang="EN-GB"><o:p></o:p></span></p>
</div>
</body>
</html>