[Gate-users] job splitting takes too much time

[Ashok Tiwari]

 Hi Martin, 
I am wondering whether your gjs is working fine. What do you see when you run gjs command? Do you see the following usage message:   +-------------------------------------------+  | gjs -- The GATE cluster job macro spliter |  +-------------------------------------------+
  Usage: gjs [-options] your_file.mac
  Options (in any order):  -a value alias             : use any alias  -numberofsplits, -n   n    : the number of job splits; default=1  -clusterplatform, -c  name : the cluster platform, name is one of the following:                               openmosix - condor - openPBS - SGE - xgrid                               This executable is compiled with SGE as default
  -openPBSscript, -os script : template for an openPBS script                               see the example that comes with the source code (script/openPBS.script)                               overrules the environment variable below
  -SGEscript, -ss script     : template for an SGE script                               see the example that comes with the source code (script/SGE.script)                               overrules the environment variable below
  -condorscript, -cs script  : template for a condor submit file                               see the example that comes with the source code (script/condor.script)  -v                         : verbosity 0 1 2 3 - 1 default
.............................etc?
I think if you see this message then it should run in principle, if not then there might be something wrong with the installation or job submission! I am not familiar with the HT condor cluster, so I cannot give you the specific information but in the SGE cluster, I normally submit the job using the following script: (this is the copy-paste of the script)
#!/bin/bash#
# Queue to submit job#$ -q CCOM,UI   
# batch job stderr and stdout#$ -o GC_WORKDIR/GC_LOG#$ -e GC_WORKDIR/GC_ERR
# Job name#$ -N GC_JOBNAME# Use current working directory
#$ -cwd
# print date and timedate
# I want SGE cluster to send me an email# when the job begins and when it ends#$ -M ashok-tiwari at uiowa.edu#$ -m be
# -l h_vmem=20G# Set simulation time
# -l h_rt=24:00:00#$ -pe smp 8
## -l mf=20G

# executableGC_GATE
To use the jobsplitter functionality I submit the job using the following command in the command line:$ gjs -numberofsplits 10 -clusterplatform SGE  ../somedir/script/main.mac
After I hit enter in the command line, it will generate the main.submit file in the same directory within ~seconds. Also at the same time, 10 split macros will be created in the GC_DOT_GATE directory. Then you are supposed to run the main.submit executable file using: ./main.submit to perform a simulation.
Hope this is a little bit helpful.
Best, Ashok





    On Friday, 23 October 2020, 02:46:04 pm GMT-4, Martin Sower <melkatib1 at gmail.com> wrote:  
 
 Thank you Ashok for your response,
effectively, I had a .split file with one of the expected .mac files (mysim1.mac but no mysim2.mac !) in the DOT_GATE directory but no .submit file in the current directory (where gjs is launched),
I will appreciate if you could share with me your scripts so that I can compare with mines,
thank you for your help.Martin

Le ven. 23 oct. 2020 à 19:17, Ashok Tiwari <tiwarias at yahoo.com> a écrit :

Hi Martin,
I don't have experience with the HT condor platform but after running the gjs command you should have a .split file in the current directory where you ran the gjs command and split macros in your .GC_DOT_GATE directory based on your installation. It should not take such a long time, I think it is a matter of ~secs (based on my experience). I don't have an idea about why it is taking too much time, but I suggest you to check the condor script based on available scripts in GATE. I am happy to send you my SGE cluster setting files if you want to compare the scripts to find out the culprit.
Thanks,Ashok Tiwari


   On Friday, 23 October 2020, 06:41:19 am GMT-4, Martin Sower <melkatib1 at gmail.com> wrote:  
 
 Hi,
I installed the gjs (job splitter) and gjm (job merger) as in the Gate User's guide and installed a Personal HTcondor as in   https://htcondor.readthedocs.io/en/stable/cloud-computing/using-annex-first-time.html#install-a-personal-htcondor with the tests passed successfully, and now I'm trying to split my simulation to 2 jobs, with the command   gjs  -numberofsplits 2 -clusterplatform condor -condorscript /home/..../Gate-8.2/cluster_tools/jobsplitter/script/condor.script mysim.mac, I'm getting the .split and mysim1.mac in the .GATE directory but nothing else, and the problem is that this command take too much long (1h before I force them to stop), so my question is: is it normal that this gjs command take such a long time? and what can be the origin of this problem?
thank you in advance for your help.Martin
_______________________________________________
Gate-users mailing list
Gate-users at lists.opengatecollaboration.org
http://lists.opengatecollaboration.org/mailman/listinfo/gate-users  
  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opengatecollaboration.org/pipermail/gate-users/attachments/20201023/f8f1866f/attachment.html>