[Gate-users] Adding G4MTRunManager Support to GATE

[Marc Verderi]

Dear All,

     The interest of G4MTRunManager is that the geometry and the 
cross-section tables are shared among the threads. For big applications 
-and simulation of phantom irradiation is one example- this represents a 
large memory. For machines with many cores, spawning N jobs of such 
application may exhaust the memory, preventing to use all the available 
cores. By sharing geometry and cross-section tables, the G4MTRunManager 
saves a large fraction of memory, allowing to use much more cores. Some 
tests have been done by Geant4, on Xeon Phi, see for example
https://twiki.cern.ch/twiki/bin/view/Geant4/MultiThreadingTaskForce#CPU_and_Memory_Performances
and one single application of high energy physics type (simplified CMS 
simulation) could run smoothly with 240 threads, the maximum available 
(the machine has 60 user cores, up to 4 thread/core). Without MT, just 
spawning jobs, only ~30 jobs could have been run in parallel, leaving 30 
cores unoccupied, because of lack of memory !

     Moving to multi-threading has some constraints. Each thread 
processes a bunch of events. Events are hence generated and processed in 
parallel, independently. This means that primary generator action, event 
action, stepping action have to have independent instances in each 
thread. This is the very purpose of the new class 
G4VUserActionInitialization : the method "Build()" is called for each 
thread, to instantiate in each of these the above actions. For the run 
action it is a bit more complicated : a run action may be for the entire 
application, or may be for each thread. For an "all application" action, 
BuildForMaster() has to be used.
     This independence of threads has a similar impact on sensitive 
detectors : for these, the G4VUserDetectorConstruction class has a new 
method : ConstructSDandField(). Again, this method is called for each 
thread, so that sensitive detectors and fields live independent lives in 
the various threads.
     This looks quite work, but is not that heavy in practice.

     In practice also, what has to be taken care of in your code are 
"static" variables : at each occurence of a static variable, you have to 
think if this variable has to be common to the entire application -a 
"true" static-, or if it is common the thread only : "thread local" 
static. In most of the cases, static variables are static to the thread. 
For the case of a true "static", be aware that this means that each 
thread may access the variable, at any time. If this variable is read & 
write during the processing, it will have a quite unpredictable 
behavior, and this is a source of debugging headaches ;) . Any random 
crash -which are often non-reproducible between two processing- signs 
this sort of conflict.

     Most of the G4 examples (basic, extended) are provided in MT mode, 
and are good starting points.

     Hope this helps.

Cheers,
Marc (a G4 member)


On 03/26/2015 01:02 PM, Alex Vergara Gil wrote:
> Dear All
>
> I started this thread to unify all those enthusiast people who want to
> add G4MTRunManager support into GATE. The advantages of a Multi
> Threading Run Manager are obvious but I will explain it here anyway, I
> will send you also my first patch and the problems I am facing.
>
> Advantages
> 1. You will not depend on an external cluster software to run on a
> single multi cpu PC.
> 2. The time you need for simulation scales linearly with the number of cpus
> 3. You doesn't need to merge the outputs, since this is performed automatically
> 4. Any other you may add
>
> My first patch
>
> <start of the code>
>
> Autor: Alex Vergara Gil <alexvergaragil at gmail.com>  2015-03-25 17:13:45
> Committer: Alex Vergara Gil <alexvergaragil at gmail.com>  2015-03-25 17:13:45
> Padre: db6875e64d60ad1e0f2d100c496843632acb23c8 (Merge
> https://github.com/OpenGATE/Gate)
> Hija:  28c338cd3263108df3927db14c6975f4cdcc31b4 (Agregado el
> UserActionInitialization)
> Rama: partopc
> Sigue-a:
> Precede-a:
>
>      trying g4mtRunManager
>
> ------------------- source/general/include/GateRunManager.hh -------------------
> index c4164d9..b72327b 100644
> @@ -28,12 +28,19 @@
>   #define GateRunManager_h 1
>
>   #include "G4RunManager.hh"
> +#ifdef G4MULTITHREADED
> + #include "G4MTRunManager.hh"
> +#endif
>   #include "GateHounsfieldToMaterialsBuilder.hh"
>
>   class GateRunManagerMessenger;
>   class GateDetectorConstruction;
>
> +#ifdef G4MULTITHREADED
> +class GateRunManager : public G4MTRunManager
> +#else
>   class GateRunManager : public G4RunManager
> +#endif
>   {
>   public:
>     //! Constructor
> @@ -60,8 +67,11 @@ public:
>
>     //! Return the instance of the run manager
>     static GateRunManager* GetRunManager()
> +  #ifdef G4MULTITHREADED
> +  {	return dynamic_cast<GateRunManager*>(G4MTRunManager::GetRunManager()); }
> +  #else
>     {	return dynamic_cast<GateRunManager*>(G4RunManager::GetRunManager()); }
> -
> +  #endif
>     bool GetGlobalOutputFlag() { return mGlobalOutputFlag; }
>     void EnableGlobalOutput(bool b) { mGlobalOutputFlag = b; }
>     void SetUserPhysicList(G4VUserPhysicsList * m) { mUserPhysicList = m; }
>
> --------------------- source/general/src/GateRunManager.cc ---------------------
> index 2604e47..75b3fb5 100644
> @@ -8,6 +8,9 @@
>
>
>   #include "GateRunManager.hh"
> +#ifdef G4MULTITHREADED
> + #include "G4MTRunManager.hh"
> +#endif
>   #include "GateDetectorConstruction.hh"
>   #include "GateRunManagerMessenger.hh"
>   #include "GateHounsfieldToMaterialsBuilder.hh"
> @@ -27,7 +30,11 @@
>   #endif
>
>   //----------------------------------------------------------------------------------------
> +#ifdef G4MULTITHREADED
> +GateRunManager::GateRunManager():G4MTRunManager()
> +#else
>   GateRunManager::GateRunManager():G4RunManager()
> +#endif
>   {
>     pMessenger = new GateRunManagerMessenger(this);
>     mHounsfieldToMaterialsBuilder = new GateHounsfieldToMaterialsBuilder();
> @@ -112,7 +119,11 @@ void GateRunManager::InitializeAll()
>
> G4ProductionCutsTable::GetProductionCutsTable()->GetHighEdgeEnergy());
>
>       // Initialization
> +#ifdef G4MULTITHREADED
> +    G4MTRunManager::SetUserInitialization(mUserPhysicList);
> +#else
>       G4RunManager::SetUserInitialization(mUserPhysicList);
> +#endif
>
>       //To take into account the user cuts (steplimiter and special cuts)
>   #if (G4VERSION_MAJOR > 9)
> @@ -126,7 +137,11 @@ void GateRunManager::InitializeAll()
>     } // End if (mUserPhysicListName != "")
>
>     // InitializePhysics
> +#ifdef G4MULTITHREADED
>     G4RunManager::InitializePhysics();
> +#else
> +  G4MTRunManager::InitializePhysics();
> +#endif
>
>     // Take into account the em option set by the user (dedx bin etc)
>     GatePhysicsList::GetInstance()->SetEmProcessOptions();
> @@ -169,7 +184,11 @@ void GateRunManager::InitGeometryOnly()
>     if (!geometryInitialized)
>       {
>         GateMessage("Core", 1, "Initialization of geometry" << G4endl);
> +#ifdef G4MULTITHREADED
> +      G4MTRunManager::InitializeGeometry();
> +#else
>         G4RunManager::InitializeGeometry();
> +#endif
>       }
>     else
>       {
> @@ -189,7 +208,11 @@ void GateRunManager::InitGeometryOnly()
>   //----------------------------------------------------------------------------------------
>   void GateRunManager::InitPhysics()
>   {
> +  #ifdef G4MULTITHREADED
> +  G4MTRunManager::InitializePhysics();
> +#else
>     G4RunManager::InitializePhysics();
> +#endif
>   }
>   //----------------------------------------------------------------------------------------
>
> @@ -205,7 +228,11 @@ void GateRunManager::RunInitialization()
>
>     // GateMessage("Core", 0, "Initialization of the run " << G4endl);
>     // Perform a regular initialisation
> +  #ifdef G4MULTITHREADED
> +  G4MTRunManager::RunInitialization();
> +#else
>     G4RunManager::RunInitialization();
> +#endif
>
>     // Initialization of the atom deexcitation processes
>     // must be done after all other initialization
>
> </end of the code>
>
> This patch compiles without any special warnings, however when I try
> to run it it explodes with the following message
>
> <start of message>
> [G4]
> [G4] *************************************************************
> [G4]  Geant4 version Name: geant4-10-01 [MT]   (5-December-2014)
> [G4]   << in Multi-threaded mode >>
> [G4]                       Copyright : Geant4 Collaboration
> [G4]                       Reference : NIM A 506 (2003), 250-303
> [G4]                             WWW : http://cern.ch/geant4
> [G4] *************************************************************
> [G4]
>          [G4-cerr]
> -------- EEEE ------- G4Exception-START -------- EEEE -------
> *** G4Exception : Run3011
>        issued by : G4MTRunManager::SetUserAction()
> For multi-threaded version, define G4UserEventAction in
> G4VUserActionInitialization.
> *** Fatal Exception *** core dump ***
> -------- EEEE -------- G4Exception-END --------- EEEE -------
>
>          [G4-cerr]
>          [G4-cerr] *** G4Exception: Aborting execution ***
> Abortado (`core' generado)
> </end of message>
>
> So I wonder if some of you have ever face this situation and how to help me
>
> Best Regards
> Alex
> _______________________________________________
> Gate-users mailing list
> Gate-users at lists.opengatecollaboration.org
> http://lists.opengatecollaboration.org/mailman/listinfo/gate-users