[Gate-users] Running Gate on a Cluster (Mijat Paunovic)

Mon Feb 14 11:07:58 CET 2022

Hi Mijat

I cannot take credit for this solution (thanks to Josh Knowland who sent me the instructions on how to do this), but in essence you can run Gate several times in parallel. The downside of this is that each version of Gate then needs full resources (so you might hit a RAM limit depending on your setup) but it does utilise as many cores as you like then. Make sure you point your outputs to different folders, and then obviously they need merging afterwards.

I’ve copied Josh’s reply to me from when I asked a similar question

This is something I’ve worked on a fair bit and here’s what I came up with. I run my simulations remotely on a 64-core virtual machine through Azure, but the process should be the same.

Note that this method works well with the new unified Tree output in GATE 9, but also works with the ‘old’ output files and previous versions of GATE.

Please note! I do not use ROOT file outputs. I have started using the NUMPY file outputs for singles and hits so that I can do custom analysis. This process should work just fine with ROOT files, though, and I talk about that a bit at the end of the email.

The overall idea is to split your macro in time and then combine the results from each. Generate a macro that does what I want. Create multiple copies of the macro, each with a different Tree output filename. All other simulation parameters can be the same (volume names, etc. do not need to change). Run all the macros concurrently using a shell script like below. Each process will be put on a different core and htop will display CPU usage (F10 to exit htop).

#!/bin/bash
(
(
Gate macro_00.mac > macro_00_log.txt &
Gate macro_01.mac > macro_01_log.txt &
Gate macro_02.mac > macro_02_log.txt &
Gate macro_03.mac > macro_03_log.txt &
Gate macro_04.mac > macro_04_log.txt &
Gate macro_05.mac > macro_05_log.txt &
Gate macro_06.mac > macro_06_log.txt &
Gate macro_07.mac > macro_07_log.txt &
wait
)
) &
htop

An alternative approach is to use parameterized macros. For this, you’d create one macro with GATE parameters as place holders using braces. Then, the bash script file would call that macro multiple times while changing the parameters. For instance, you could run a simulation and split it over time by parameterizing timestart and timestop like this in the GATE macro:

/gate/application/setTimeSlice 0.125 s
/gate/application/setTimeStart {timestart} s
/gate/application/setTimeStop {timestop} s

Then the Bash script would define the parameters for each process:

#!/bin/bash
(
(
Gate -a [timestart,0.000][timestop,0.125] paramMacro.mac > output_00_log.txt &
Gate -a [timestart,0.125][timestop,0.250] paramMacro.mac > output_01_log.txt &
Gate -a [timestart,0.250][timestop,0.375] paramMacro.mac > output_02_log.txt &
Gate -a [timestart,0.375][timestop,0.500] paramMacro.mac > output_03_log.txt &
Gate -a [timestart,0.500][timestop,0.625] paramMacro.mac > output_04_log.txt &
Gate -a [timestart,0.625][timestop,0.750] paramMacro.mac > output_05_log.txt &
Gate -a [timestart,0.750][timestop,0.875] paramMacro.mac > output_06_log.txt &
Gate -a [timestart,0.875][timestop,1.000] paramMacro.mac > output_07_log.txt &
wait
)
wait
) &
htop

Either way, you will end up with multiple output files:

output_00_log.txt
output_00.Run.dat
output_00.root
etc...

If you’re using text-based output files, there are several ways within Ubuntu to combine them.

To combine root files, you can use the ‘hadd’ application which should be available to you:

   hadd - Merge ROOT histogram files into one file
      hadd outputfile inputfiles ...

This program will add histograms from a list of ROOT files and write them to a target ROOT
      file. The target file is newly created and must not be identical to one of the source files.

Hope this helps you

Mark Baker
Principal Clinical Scientist (Interim)
Imaging Physics (Ionising)
The Clatterbridge Cancer Centre NHS Foundation Trust
(he, him)

CCCW Tel:  0151 556 5030
CCCL Tel: 0151 318 8438
Email: mark.baker23 at nhs.net<mailto:mark.baker23 at nhs.net>
Microsoft Teams (click here)<https://teams.microsoft.com/l/chat/0/0?users=mark.baker23@nhs.net>

From: Gate-users <gate-users-bounces at lists.opengatecollaboration.org> On Behalf Of olie pickford scienti
Sent: 14 February 2022 08:27
To: gate-users at lists.opengatecollaboration.org
Subject: Re: [Gate-users] Running Gate on a Cluster (Mijat Paunovic)

Hello Mijat,

From my experience Gate runs on a single core by default, and I do not think you can run it in parallel on multiple cores from a single file though am happy to be corrected about this. What I do to use multiple cores on my laptop when running simulations (I also have 8 cores) is to copy the mac file but with different start and finish times and different random seeds. So for example if running for 1 second on 4 cores, have mac files which run 0-0.25 s, 0.25-0.5 s, 0.5-0.75 s and 0.75-1 s, each with a different random seed. Then I would open 4 terminals and run 4 instances of Gate, 1 for each mac script. You should then be able to combine the resulting root files, but I process my results outside of root so just export each file individually to that framework.

In your case I think, so long as random seed is different and you're just looking for testing your reconstruction rather than making statistical assessments of the data, you should be fine to use (10^13)/n primaries in each of n simulations then combine them. If n becomes very large, or you do this a lot, you may wish to write your own script for splitting the jobs if you can't get gjs working (I don't use gjs so can't advise). Hopefully someone else on here can explain why gjs doesn't work for you though.

I hope this helps.

Olie

On 13 Feb 2022 12:00, gate-users-request at lists.opengatecollaboration.org<mailto:gate-users-request at lists.opengatecollaboration.org> wrote:

Send Gate-users mailing list submissions to
gate-users at lists.opengatecollaboration.org<mailto:gate-users at lists.opengatecollaboration.org>

To subscribe or unsubscribe via the World Wide Web, visit
https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.opengatecollaboration.org%2Fmailman%2Flistinfo%2Fgate-users&data=04%7C01%7C%7Cc6c4214ec02b4feee08e08d9eee56f6b%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637803491457006083%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=p3da2LQQypsTp4beX1sPcTnu7HwV85pAFvkUYNpnbWI%3D&reserved=0
or, via email, send a message with subject or body 'help' to
gate-users-request at lists.opengatecollaboration.org<mailto:gate-users-request at lists.opengatecollaboration.org>

You can reach the person managing the list at
gate-users-owner at lists.opengatecollaboration.org<mailto:gate-users-owner at lists.opengatecollaboration.org>

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Gate-users digest..."

Today's Topics:

   1. Running Gate on a Cluster (Mijat Paunovic)

----------------------------------------------------------------------

Message: 1
Date: Sat, 12 Feb 2022 21:22:17 +0100
From: Mijat Paunovic <paunovicjata at gmail.com<mailto:paunovicjata at gmail.com>>
To: gate-users at lists.opengatecollaboration.org<mailto:gate-users at lists.opengatecollaboration.org>
Subject: [Gate-users] Running Gate on a Cluster
Message-ID:
<CAKS_kRr=q-NXBcH_E9Bzc+EryFpT_GOps16eqy1kvZavCLCh3g at mail.gmail.com<mailto:CAKS_kRr=q-NXBcH_E9Bzc+EryFpT_GOps16eqy1kvZavCLCh3g at mail.gmail.com>>
Content-Type: text/plain; charset="utf-8"

Dear Gate Community,
I have created a simple Cone Beam CT simulator in order to obtain
projections which will be fed to a reconstruction algorithm. In order for
the algorithm to work, I need to have a large number of detected photons in
each pixel of the detector. In order to obtain this large number, I wanted
to run the simulation for ~10^13 primaries. This is of course an impossible
job for my computer which has only 8 logical cores (4 after assigning half
to the virtual machine on which Gate is run) so I will get access to a
supercomputer which will be able to do this.

In the meantime, I wonder how I could maximize the performance on my
computer? When I run the Gate simulation and monitor the CPU performance, I
can see that only 1/4 logical cores is being used for the simulation.

In Gate documentation I found the '6.1. How to use Gate on a Cluster'
explaining how I should use the *gjs *(Gate job splitter) which will split
my work on 4 jobs which I can independently run on each node of the
cluster, after which I should use the *gjm *(Gate file merger) in order to
combine the results.

Several questions arose from this:
1. Is it possible to make Gate make the use of additional 3 cores that I
have assigned to the virtual machine without doing the job splitting?
2. If I perform the job splitting, can my 4 cores be considered a cluster
of it's own?
3. When running *gjs*, I am only able to get an output when am using
*-numberofsplits* of 10 or more jobs. If I try to run it for 9 or less, I
get 'floating point error'.
4. Since I am only running a simulation of 10^13 independent photon
histories (only interested in photon interactions within a phantom and the
count of the photons on the detector) would it be the same if I just fed
the same simulation to on *n *nodes, with the number of primaries being
10^13/n?

For the first test for this simulator, I will be using it on a cylindrical
phantom. Instead of making a large number of projections which will be fed
to a reconstruction algorithm, I will be generating particles from only one
angle and the result (2D matrix with the photon count) will be copied
several hundred times and hopefully, upon reconstruction, we will get a
circular cross-section.

All in all, I need a way to speed up the process of obtaining this 2D
matrix if I am going to make the simulator more realistic.

Any suggestions on this matter would be extremely helpful. Thanks in
advance!

Kind regards,
Mijat Paunović
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.opengatecollaboration.org%2Fpipermail%2Fgate-users%2Fattachments%2F20220212%2Fe318e521%2Fattachment-0001.html&data=04%7C01%7C%7Cc6c4214ec02b4feee08e08d9eee56f6b%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637803491457006083%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=mpq2bZwjMGSTedKw2QWhGIUeWkfXQ1%2B%2BwoExIP1xiw0%3D&reserved=0>

------------------------------

Subject: Digest Footer

_______________________________________________
Gate-users mailing list
Gate-users at lists.opengatecollaboration.org<mailto:Gate-users at lists.opengatecollaboration.org>
https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.opengatecollaboration.org%2Fmailman%2Flistinfo%2Fgate-users&data=04%7C01%7C%7Cc6c4214ec02b4feee08e08d9eee56f6b%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637803491457006083%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=p3da2LQQypsTp4beX1sPcTnu7HwV85pAFvkUYNpnbWI%3D&reserved=0

------------------------------

End of Gate-users Digest, Vol 189, Issue 13
*******************************************

This message originated from outside of NHSmail. Please do not click links or open attachments unless you recognise the sender and know the content is safe.

************************************************************************************** ******************************

This message may contain confidential information. If you are not the intended recipient please:
i) inform the sender that you have received the message in error before deleting it; and
ii) do not disclose, copy or distribute information in this e-mail or take any action in relation to its content (to do so is strictly prohibited and may be unlawful).
Thank you for your co-operation.

NHSmail is the secure email, collaboration and directory service available for all NHS staff in England. NHSmail is approved for exchanging patient data and other sensitive information with NHSmail and other accredited email services.

For more information and to find out how you can switch visit Joining NHSmail – NHSmail Support<https://support.nhs.net/article-categories/joining-nhsmail/>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opengatecollaboration.org/pipermail/gate-users/attachments/20220214/cf8e8cce/attachment-0001.html>