[Gate-users] Gate 8.2 with job splitter on Cluster

[Andreas Resch]

Hi Nikta,

I think you need to set the environment variables/path to Gate, root, and
G4 within your script/ on each node first. Then run Gate, simplest version
could be: "Gate main.mac | tee test.log"

After splitting the Gate jobs, I start the jobs like this in slurm (using
the module package to load Gate, but you could also set the paths
classically as described in the compilation instructions):

cat <<EOF | tee submitSlurmGateFile.sh
#!/bin/bash
#SBATCH -J ${JOBNAME}
#SBATCH -N 1
#SBATCH --ntasks-per-node=16
#SBATCH --ntasks-per-core=1
#SBATCH --cpus-per-task=1
#SBATCH --time=${RUNTIMEREQUEST}
#SBATCH --mail-user="$emailAddress"
#SBATCH --mail-type=FAIL
#SBATCH --output=.Gate/${dotMainDir}/main%a_Job%A.out
#SBATCH --error=.Gate/${dotMainDir}/main%a_Job%A.err
#SBATCH --array=0-${NODESmin1}:1
#SBATCH --partition=${RUNONPARTITION}
#SBATCH --qos=${RUNONNODES}

#STARTt=\$(date +%s)
module purge
export MODULEPATH=\$MODULEPATH
module load ${GATEVERSION}

export SLURM_STEP_GRES=none

for i in {1..16}
do
 echo "000 (\${SLURM_ARRAY_JOB_ID}.000 )"
>.Gate/${dotMainDir}/main\$((i+SLURM_ARRAY_TASK_ID*16)).log  &
 Gate .Gate/${dotMainDir}/main\$((i+SLURM_ARRAY_TASK_ID*16)).mac >>
.Gate/${dotMainDir}/main\$((i+SLURM_ARRAY_TASK_ID*16)).log &
done
wait

EOF

RES=$(sbatch submitSlurmGateFile.sh)
echo $RES
JOBID=${RES##*}
JOBID=${RES//[!0-9]}
echo "Detected JobID: ${JOBID}"

cheers,
andreas

Am Do., 30. Mai 2019 um 17:49 Uhr schrieb Nikta Zarifyussefian <
Nikta.Zarifyussefian at usherbrooke.ca>:

>
> Hello,
>
>
> We have tried to run Gate with jobsplitter on our cluster.  for this, we
> have replaced the jobsplitter with the one provided in the link below to be
> compatible with  SLURM:
>
>
>
> https://github.com/OpenGATE/Gate/commit/fe0f56b7843b2896942dda0f3220271de0543baa
>
>
> and GATE v 8.2 has been compiled with below dependencies:
>
>
>     ('Geant4', '10.05.p01', '-seq'),
>     ('expat', '2.2.5'),
>     ('CLHEP', '2.4.1.0'),
>     ('ROOT', '5.34.36'),
>
> builddependencies = [('CMake', '3.12.3')]
>
> preconfigopts  = " export Geant4_DIR=$EBROOTGEANT4 && "
> preconfigopts += " export ROOTCINT_EXECUTABLE=$EBROOTROOT/bin/rootcint && "
> preconfigopts += " export
> ROOT_CONFIG_EXECUTABLE=$EBROOTROOT/bin/root-config && "
> preconfigopts += " export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$EBROOTROOT/lib
> && "
> preconfigopts += " export CPATH=$CPATH:$EBROOTROOT/include && "
>
> configopts  = '-DBUILD_TESTING=ON '
> configopts += '-DCMAKE_GATE_DOWNLOAD_BENCHMARKS_DATA=ON '
> configopts += '-DCMAKE_GATE_USE_ECAT7=ON '
> configopts += '-DCMAKE_GATE_USE_GPU=OFF '
> configopts += '-DCMAKE_GATE_USE_ITK=ON '
> configopts += '-DCMAKE_GATE_USE_RTK=ON '
> configopts += '-DCMAKE_GATE_USE_STDC11=ON '
> configopts += '-DCMAKE_GATE_USE_DAVIS=ON '
> configopts += '-DCMAKE_GEANT4_USE_SYSTEM_CLHEP=OFF '
>
>
> the slurm script is as follows:
>
>
> !/bin/bash
>
> #SBATCH --time=120:00:00
> #SBATCH --nodes=10
> #SBATCH --ntasks=10
> #SBATCH --cpus-per-task=12
> #SBATCH --mem=31000M
>
>
> gjs -n 10 main.mac-> test.txt
>
>
> the jobs are cancelled immediately with no output file. The test.txt file
> is as follows:
>
>
> Summary of all outputs:
>   ROOT       output is enabled
>   ASCII      output is disabled
>   ARF        output is disabled
>   PROJECTION output is disabled
>   ECAT7      output is disabled
>   SINOGRAM   output is disabled
>   SINOACCEL  output is disabled
>   LMF        output is disabled
>   CT         output is disabled
>   GPUSPECT   output is disabled
> Summary of all actors:
>   ~~~
> Number of enabled output: 1
> Number of enabled actors: 0
>
>
> Any help or comment is highly appreciated.
>
>
> Regards
>
> Nikta
> _______________________________________________
> Gate-users mailing list
> Gate-users at lists.opengatecollaboration.org
> http://lists.opengatecollaboration.org/mailman/listinfo/gate-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opengatecollaboration.org/pipermail/gate-users/attachments/20190531/71fd4d80/attachment.html>