[Gate-users] How to use multithread on Gate 7.2

[David Boersma]

Hi Marc,

We do not need any reminders of the benefits of being able to run in MT 
mode. Several GATE collaboration members are yearning for MT GATE. :)

However, many of us are aware that MT programming is extremely tricky, 
so even if it compiles and runs it will take a long time to fix all the 
new bugs and to convince ourselves that MT results will always be 
compatible with the validated single thread results.

I do not think we made a systematic list of points that prevent MT in 
GATE, but I myself am thinking of the following issues:

* we lack MT expertise

I think that most of the active GATE developers/maintainers (at least 
the ones that I talked to) have no significant prior experience in MT 
programming. (I would be very happy if I am wrong about this.) I have 
started reading the "C++ concurrency in action" book by Anthony 
Williams, one of the experts in the field, and my progress is slow and 
the main thing that I have learned so far is that MT programming is very 
nontrivial. I guess that that book is a bit of overkill, it could also 
be instructive to just look at practical examples of successful open 
source MT geant4 programs.

* GATE has lots of static and global variables

... and probably several other things that are bad for MT programming. A 
large part of the code was written about 10 years ago, with a very 
sequential single thread mind set. All code in GATE needs to be reviewed 
for its MT-readiness / thread safety, that is going to be a lot of work.

* Geometry may *not* be static

I think that for several typical GATE use cases the geometry actually moves.
- CT beam and detector rotate around subject
- VMAT treatment linac rotates around subject
- patient deforms during PET scan
- ...
I guess that this can all still be solved in a MT program. But it 
probably requires a refinement of the simple [1]-[2]-[3] approach, 
because I think that the statement '[1] is "almost constant" (constant 
after initialization)' is not entirely valid for the above cases.

The nice thing about CMS is that it has no mechanically moving parts 
during normal detector operation (right?).

* random number generation

RNG in MT programming is a special art, but fortunately geant4 took care 
of that. Here is described what we need to change:

https://twiki.cern.ch/twiki/bin/view/Geant4/QuickMigrationGuideForGeant4V10#Random_numbers

We did not do that yet. But because it's so well described it might be 
the easiest thing to fix, or at least easier than the previous issues in 
the list.

Cheers,
David


Den 26/06/2017 kl. 16:04, skrev Marc Verderi:
> Dear all,
> 
>      Are the points that prevent GATE to adopt the MT identified ? 
> Making a G4 code thread-safe must definitely be done with care, but for 
> examples, TOPAS is fully MT compliant, the CMS LHC experiment has 
> started production in MT mode and ALICE is on the way, etc. Assuming 
> things will be much easier with a yet-to-be-defined product "geant5" 
> (you meant Geant-V ?) is somewhat a bet. ;)
> 
>      Let me remind that the benefit GATE could have from running in MT 
> mode is when/if using a large geometry (eg : geometry from CT scan 
> system). Assuming the memory usage of the application is :
> 
>      [1] (geometry + cross-section tables) + [2] (G4 memory work for 
> tracking) + [3] (hit collection / scoring data),
> 
> then the G4 MT brings a real advantage if [1] is of the order or larger 
> than [2] + [3] (and the more [1] is larger, the larger the advantage 
> is). As [1] is "almost constant" (constant after initialization) it is 
> shared among the threads during the events processing. In the case of 
> the 24 cores mentioned, [1] will be present only one time in memory in 
> MT mode, while it would be present 24 times if using independent 
> applications. In this case, you may reach the point that you don't have 
> enough memory to serve all the cores (hence you loose computing power). 
> This is not a theoretical point, but experienced.
> 
>      To give some idea, in a test we run with a simplified CMS geometry, 
> [1] is ~ 200 MB, [2] ~ 10 MB and [3] is ~ 0 (no scoring). This, of 
> course, totally depends on the application, and it would be interesting 
> to have an idea of the memory pattern(s) for GATE.
> 
> Cheers,
> Marc
> 
> On 26/06/2017 12:01, David Boersma wrote:
>> Hi Mery MB,
>>
>> We would love to run GATE in multithreaded mode, but for the time 
>> being that is not yet working. We think we may have to wait until 
>> Geant5 (no typo: Geant version five) is released. That will not happen 
>> any time soon. Also GATE internally needs a lot of work to be 
>> "thread-safe".
>>
>> So in order to make full use of your multicore system, I would advise 
>> to treat it as a "cluster". You can install your favorite job 
>> submission system (I am using HTCondor, which is very easy to install 
>> on Ubuntu) and then just split up your simulation into many jobs that 
>> can be run in parallel. Make sure that you never exceed the available 
>> RAM (by being careful with your GATE macro, configuring your job 
>> management system to not allow more jobs when the RAM is almost 
>> completely used, or by throwing more money at it and installing more 
>> RAM, or etc). Also be careful with hyperthreading. On my own "number 
>> cruncher", the "htop" program reports 16 cores, but physically it has 
>> actually only 8 cores. I noticed that hyperthreading slows down GATE, 
>> even if there is enough RAM. So on my system I configure HTCondor to 
>> *not* use hyperthreading, running 8 jobs in parallel gives a slightly 
>> higher throughput than running 16 jobs in parallel: with 16, each 
>> individual job runs a bit more that 2x slower than with 8. YMMV.
>>
>> There are several tools and scripts/programs floating around to assist 
>> you running GATE on clusters. AFAIK, all these tools (including "gjs" 
>> that is included in the GATE source tree) are tailored for specific 
>> use cases. If you tell us about your own intended use case then maybe 
>> some other mailing list members with similar use cases can advertise 
>> their favorite cluster tool to you.
>>
>> (BTW: if anyone with some real concurrent programming experience 
>> happens to read this message, please feel free to have a look at our 
>> code, and share your thoughts and ideas about the way forward to 
>> MT-GATE  on the gate-devel mailing list.)
>>
>> Good luck,
>> David Boersma
>>
>>
>> Den 25/06/2017 kl. 11:26, skrev Mery MB:
>>> Dear All Gate Developer and User
>>>
>>> I want to run my Gate program on a 24 core PC. How can I use all core 
>>> for one program in Gate?
>>>
>>> How we can use the Multi threaded option in Gate Programs?
>>> we should add some sentences to the program?
>>>
>>> I will thank you if some one can help me in this issue.
>>>
>>> Best Regards
>>>
>>>
>>> _______________________________________________
>>> Gate-users mailing list
>>> Gate-users at lists.opengatecollaboration.org
>>> http://lists.opengatecollaboration.org/mailman/listinfo/gate-users
>>>
>> _______________________________________________
>> Gate-users mailing list
>> Gate-users at lists.opengatecollaboration.org
>> http://lists.opengatecollaboration.org/mailman/listinfo/gate-users
> 
> _______________________________________________
> Gate-users mailing list
> Gate-users at lists.opengatecollaboration.org
> http://lists.opengatecollaboration.org/mailman/listinfo/gate-users