[Gate-users] Using gjs while running Gate on Docker

Timothée ZARAGORI zaragori.tim at gmail.com
Tue Apr 5 16:46:58 CEST 2022


Dear Mijat,

I faced the same problem months ago and the only solution I found was to
install HT Condor on the computer and interface it with Docker. The idea is
to make HT Condor run one docker for each split but we need to specify
which data to transfer to the docker as well as scripts to define
environment variables and how to execute the GATE macro. I made a
little text file explaining the steps at the time I was working on it.
There is also a python script at the end of the file which allows to
automate the pipeline. However, I didn't find any solution to merge the
root files without gjm.
I hope it is still working and it will be useful, don't hesitate if you
have any questions.
Kind regards,
Timothée Zaragori

On Tue, Apr 5, 2022 at 3:37 PM Mijat Paunovic <paunovicjata at gmail.com>
wrote:

> Dear Gate Community,
> I am using Docker to run Gate 9.2. I want to split jobs by using gjs. I
> used gjs back when I was using pre-installed Gate on Virtual Box, and by
> following instructions on Gate Collaboration documentation, I managed to
> split my jobs. Now, I am using Gate on Docker, so I do not have Gate
> installation directory on my computer so I cannot follow the instructions
> for gjs.
> Has anyone had the chance to split jobs while using Gate on Docker?
> Kind regards,
> Mijat Paunović
> _______________________________________________
> Gate-users mailing list
> Gate-users at lists.opengatecollaboration.org
> http://lists.opengatecollaboration.org/mailman/listinfo/gate-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opengatecollaboration.org/pipermail/gate-users/attachments/20220405/ef19277a/attachment.html>
-------------- next part --------------
HT Condor + Gate Docker : 
Prerequisites: install HT Condor + Pull docker gate image

1. Configure HT Condor Docker Universe
https://htcondor.readthedocs.io/en/latest/admin-manual/setting-up-vm-docker-universes.html#the-docker-universe

2. Create script to define the environment variables in the docker that will be executed by condor
https://opengate.readthedocs.io/en/latest/compilation_instructions.html#environment-configuration-and-starting-gate

Example setup_gate_container_env.sh for docker GATE 9.0: 
source /cern/root-install/bin/thisroot.sh
source /geant4/geant4.10.06.p01-install/bin/geant4.sh
export PATH=/cern/root-install/bin:/geant4/geant4.10.06.p01-install/bin:/gate/gate_9.0-install/bin/:$PATH

3. Create csv file with all the information for the jobs.

Example of csv file job_info.csv:
Job_ID,Start_Time,Stop_Time,Output_path,Macro_path
0,0.0,1e-05,output/NECR_split0,mac/NECR.mac
1,1e-05,2e-05,output/NECR_split1,mac/NECR.mac
2,2e-05,3.0000000000000004e-05,output/NECR_split2,mac/NECR.mac
3,3.0000000000000004e-05,4e-05,output/NECR_split3,mac/NECR.mac
4,4e-05,5e-05,output/NECR_split4,mac/NECR.mac
5,5e-05,6e-05,output/NECR_split5,mac/NECR.mac
6,6.000000000000001e-05,7.000000000000001e-05,output/NECR_split6,mac/NECR.mac
7,7.000000000000001e-05,8e-05,output/NECR_split7,mac/NECR.mac
8,8e-05,9e-05,output/NECR_split8,mac/NECR.mac
9,9e-05,0.0001,output/NECR_split9,mac/NECR.mac

4. Create the script that will be run in the docker container and the the condor submit file to execute the script
https://htcondor.readthedocs.io/en/latest/users-manual/docker-universe-applications.html
http://lists.opengatecollaboration.org/pipermail/gate-users/2019-January/010683.html

Example NECR.submit for a split with 10 jobs: 
universe = docker
executable = /bin/bash
arguments = NECR.sh $(Process) # Argument to indicate job ID
docker_image = opengatecollaboration/gate:latest
output  = log_HTC/NECR_split$(Process).out
error   = log_HTC/NECR_split$(Process).err
log     = log_HTC/NECR_split$(Process).log
should_transfer_files = YES
when_to_transfer_output = ON_EXIT
transfer_input_files = setup_gate_container_env.sh,NECR.sh,job_info.csv,mac,data # Will transfer whole mac and data directory in docker container as is thus relative paths in macro can be kept
transfer_output_files = output # Will transfer back whole output directory created in docker container
queue 10

Example NECR.sh :
#!/usr/bin/bash --login
echo running $0 on `hostname` at `date`
cd `dirname $0`
echo pwd = `pwd`
source setup_gate_container_env.sh
# Get the information for the current job in csv file
while IFS=, read -r job_ID start_time stop_time output macro; do
if [ ${job_ID} = $1 ]; then
START=$start_time; STOP=$stop_time; OUTPUT=$output; MACRO=$macro; fi
done < job_info.csv
# Create output directory needed in macro
mkdir -p $OUTPUT
echo "Gate -a [SimulationStartTime,${START}][SimulationStopTime,${STOP}][OutputBaseName,${OUTPUT}] ${MACRO}"
Gate -a [SimulationStartTime,${START}][SimulationStopTime,${STOP}][OutputBaseName,${OUTPUT}] ${MACRO}

5. Run the jobs:
condor_submit NECR.submit

Possibility to automate file creation and job submission with python script:
# -*- coding: utf-8 -*-
"""
Created on Tue Apr 20 15:11:48 2021

@author: TZAR
"""

import os
import stat
import subprocess
import argparse
import csv

if __name__ == '__main__':

    # Parse command line arguments
    parser = argparse.ArgumentParser(description='HTC jobs generator for Gate docker')
    parser.add_argument("--root", "-r", help='Root directory of simulation data', required=True, type=str)
    parser.add_argument("--macro", "-m", help='Relative path of the macro file to run from root', required=True, type=str)
    parser.add_argument("--image", "-i", help='Name of the GATE docker image', default='opengatecollaboration/gate:9.0', type=str)
    parser.add_argument("--start", help='Start time simulation', default=0, type=float)
    parser.add_argument("--stop", help='Stop time simulation', required=True, type=float)
    parser.add_argument("--split", help='Number of splits', required=True, type=int)
    args = parser.parse_args()
    
    # Set current working directory to root
    root = os.path.abspath(args.root)
    os.chdir(root)

    # Simulation start time [s]
    start_time = args.start
    
    # Simulation stop time [s]
    stop_time =  args.stop
    
    # Number of jobs to submit to condor
    Nsplits = args.split
    
    split_duration = (stop_time - start_time)/Nsplits
    
    basename = os.path.splitext(os.path.basename(args.macro))[0]
    
    GATEMacro = args.macro
    
    submit_file = '%s.submit' % basename
    
    script_file = '%s.sh' % basename
    
    # Write the script that setup env variables in gate docker
    # Must change with geant4 and gate versions (here for gate 9.0)
    with open('setup_gate_container_env.sh', 'w') as f:
        f.write("source /cern/root-install/bin/thisroot.sh\n")
        f.write("source /geant4/geant4.10.06.p01-install/bin/geant4.sh\n")
        f.write("export PATH=/cern/root-install/bin:/geant4/geant4.10.06.p01-install/bin:/gate/gate_9.0-install/bin/:$PATH")
    
    csv_rows = [['Job_ID', 'Start_Time', 'Stop_Time', 'Output_path', 'Macro_path']]
    for i in range(Nsplits):
        t1 = i*split_duration # Virtual start time
        t2 = t1 + split_duration # Virtual stop time
        if t2 > stop_time:
            t2 = stop_time
        # Path of output directory. In macro /gate/output/root/setFileName {OutputBaseName}/FileName    
        OutputBaseName = '%s_split%d' % (basename, i)
        csv_rows.append([i, t1, t2, "output/%s" % OutputBaseName, GATEMacro])
        
    # Write csv file with information needed in scripts
    with open('job_info.csv', mode='w') as csv_file:
        writer = csv.writer(csv_file, delimiter=',', lineterminator='\n')
        writer.writerows(csv_rows)
    
    # Write the script that will be run in the docker container:
    print("Writing %s" % script_file)
    f = open(script_file, 'w')
    with open(script_file, 'w') as f:
        f.write("#!/usr/bin/bash --login\n")
        f.write("echo running $0 on `hostname` at `date`\n")
        f.write("cd `dirname $0`\n")
        f.write("echo pwd = `pwd`\n")
        f.write("source setup_gate_container_env.sh\n")
        f.write("while IFS=, read -r job_ID start_time stop_time output macro; do\n")
        f.write("if [ ${job_ID} = $1 ]; then\n")
        f.write("START=$start_time; STOP=$stop_time; OUTPUT=$output; MACRO=$macro; fi\n")
        f.write("done < job_info.csv\n")
        f.write("mkdir -p $OUTPUT\n")
        command = "Gate -a [SimulationStartTime,${START}][SimulationStopTime,${STOP}][OutputBaseName,${OUTPUT}] ${MACRO}"
        f.write('echo "%s"\n' % command)
        f.write(command)
    # Set permissions so that condor can execute the script:
    stats = os.stat(script_file)
    os.chmod(script_file, stats.st_mode | stat.S_IXUSR | stat.S_IXGRP | stat.S_IXOTH)
    
    # Make a list of files/dirs to be transfered for the condor jobs.
    # Since condor's transfer_input_files cannot handle wild cards,
    # we'll create a comma-separated list here assuming that 
    # there is a data and mac directory where everything is stored:
    input_files = ['setup_gate_container_env.sh', script_file, 
                   'job_info.csv', 'mac', 'data']
    # Create logdir
    os.makedirs(os.path.join(root, 'log_HTC'), exist_ok=True)
    # Write the condor submit file to execute the script:
    print("Writing %s" % submit_file)
    with open(submit_file, 'w') as f_submit:
        f_submit.write("universe = docker\n")
        f_submit.write("executable = /bin/bash\n")
        f_submit.write("arguments = %s $(Process)\n" % script_file)
        f_submit.write("docker_image = %s\n" % args.image)
        f_submit.write("output  = log_HTC/%s_split$(Process).out\n" % basename)
        f_submit.write("error   = log_HTC/%s_split$(Process).err\n" % basename)
        f_submit.write("log     = log_HTC/%s_split$(Process).log\n" % basename)
        f_submit.write("should_transfer_files = YES\n")
        f_submit.write("when_to_transfer_output = ON_EXIT\n")
        f_submit.write("transfer_input_files = " + ','.join(input_files) + "\n")
        f_submit.write("transfer_output_files = output\n")
        f_submit.write("queue %d\n" % Nsplits)
    
    # Submit the job to condor:
    print("Submitting %s" % submit_file)
    subprocess.call(["condor_submit", submit_file])


More information about the Gate-users mailing list