[Gate-users] Job splitter etc. on vGate

Thu Nov 18 22:18:45 CET 2021

Hi Mark,

This is something I've worked on a fair bit and here's what I came up with. I run my simulations remotely on a 64-core virtual machine through Azure, but the process should be the same.

Note that this method works well with the new unified Tree output in GATE 9, but also works with the 'old' output files and previous versions of GATE.

Please note! I do not use ROOT file outputs. I have started using the NUMPY file outputs for singles and hits so that I can do custom analysis. This process should work just fine with ROOT files, though, and I talk about that a bit at the end of the email.

The overall idea is to split your macro in time and then combine the results from each. Generate a macro that does what I want. Create multiple copies of the macro, each with a different Tree output filename. All other simulation parameters can be the same (volume names, etc. do not need to change). Run all the macros concurrently using a shell script like below. Each process will be put on a different core and htop will display CPU usage (F10 to exit htop).

#!/bin/bash
(
(
Gate macro_00.mac > macro_00_log.txt &
Gate macro_01.mac > macro_01_log.txt &
Gate macro_02.mac > macro_02_log.txt &
Gate macro_03.mac > macro_03_log.txt &
Gate macro_04.mac > macro_04_log.txt &
Gate macro_05.mac > macro_05_log.txt &
Gate macro_06.mac > macro_06_log.txt &
Gate macro_07.mac > macro_07_log.txt &
wait
)
) &
htop

An alternative approach is to use parameterized macros. For this, you'd create one macro with GATE parameters as place holders using braces. Then, the bash script file would call that macro multiple times while changing the parameters. For instance, you could run a simulation and split it over time by parameterizing timestart and timestop like this in the GATE macro:

/gate/application/setTimeSlice 0.125 s
/gate/application/setTimeStart {timestart} s
/gate/application/setTimeStop {timestop} s

Then the Bash script would define the parameters for each process:

#!/bin/bash
(
(
Gate -a [timestart,0.000][timestop,0.125] paramMacro.mac > output_00_log.txt &
Gate -a [timestart,0.125][timestop,0.250] paramMacro.mac > output_01_log.txt &
Gate -a [timestart,0.250][timestop,0.375] paramMacro.mac > output_02_log.txt &
Gate -a [timestart,0.375][timestop,0.500] paramMacro.mac > output_03_log.txt &
Gate -a [timestart,0.500][timestop,0.625] paramMacro.mac > output_04_log.txt &
Gate -a [timestart,0.625][timestop,0.750] paramMacro.mac > output_05_log.txt &
Gate -a [timestart,0.750][timestop,0.875] paramMacro.mac > output_06_log.txt &
Gate -a [timestart,0.875][timestop,1.000] paramMacro.mac > output_07_log.txt &
wait
)
wait
) &
htop

Either way, you will end up with multiple output files:

output_00_log.txt
output_00.Run.dat
output_00.root
etc...

If you're using text-based output files, there are several ways within Ubuntu to combine them.

To combine root files, you can use the 'hadd' application which should be available to you:

   hadd - Merge ROOT histogram files into one file
      hadd outputfile inputfiles ...

This program will add histograms from a list of ROOT files and write them to a target ROOT
      file. The target file is newly created and must not be identical to one of the source files.

I hope that helps! Feel free to ask questions.

From: Gate-users <gate-users-bounces at lists.opengatecollaboration.org> On Behalf Of BAKER, Mark (THE CLATTERBRIDGE CANCER CENTRE NHS FOUNDATION TRUST)
Sent: Thursday, November 18, 2021 8:04 AM
To: gate-users at lists.opengatecollaboration.org
Subject: [Gate-users] Job splitter etc. on vGate

Hi everyone

I have a macro up and running using vGate 9.0 which imports PET and CT data to generate a dose distribution. It seems to be working fine, but only uses one CPU core (which I believe is a known "feature" of Gate). My computer has 8 cores acting as 16 logical processors and Oracle VM currently has access to 8 cores. I'm not sure if this is the full 8 physical (one logical from each core) or 4 physical (two logical from each). An obvious speeding up of the process would be to use the jobsplitter and filemerger in order to utilise more cores, but I'm not sure if this is a possibility within vGate 9.0.

In the ReadTheDocs documentation, there is a section on using job splitter etc. on a cluster but this requires additional installation, plus a "cluster aware version of Gate". My gut instinct is that vGate 9.0 isn't one of those versions but I was wondering if anyone had been successful or has any other tips on getting vGate to utilise my cpu power more effectively?

Thank you

Mark Baker
Senior Clinical Scientist
Imaging Physics (Ionising)
The Clatterbridge Cancer Centre NHS Foundation Trust

CCCW Tel:  0151 556 5030
CCCL Tel: 0151 318 8438
Email: mark.baker23 at nhs.net<mailto:mark.baker23 at nhs.net>
Microsoft Teams (click here)<https://teams.microsoft.com/l/chat/0/0?users=mark.baker23@nhs.net>

Honorary Lecturer, Dept. of Physics
University of Liverpool

[CCC email footer April 2021]

************************************************************************************** ******************************

This message may contain confidential information. If you are not the intended recipient please:
i) inform the sender that you have received the message in error before deleting it; and
ii) do not disclose, copy or distribute information in this e-mail or take any action in relation to its content (to do so is strictly prohibited and may be unlawful).
Thank you for your co-operation.

NHSmail is the secure email, collaboration and directory service available for all NHS staff in England. NHSmail is approved for exchanging patient data and other sensitive information with NHSmail and other accredited email services.

For more information and to find out how you can switch visit Joining NHSmail - NHSmail Support<https://support.nhs.net/article-categories/joining-nhsmail/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opengatecollaboration.org/pipermail/gate-users/attachments/20211118/34169b5d/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.jpg
Type: image/jpeg
Size: 13489 bytes
Desc: image001.jpg
URL: <http://lists.opengatecollaboration.org/pipermail/gate-users/attachments/20211118/34169b5d/attachment-0001.jpg>