[Gate-users] Notes on GATE + Docker + HT Condor

[Christopher Watanabe]

Hey Mike, 

This is super neat! I wondered though about the interactivity between the python for loop and the call “subprocess.call” of your condor submit file. It seems the for loop doesn’t have to wait for the submitted condor file to finish running, otherwise this would be as good as running on a single core. I wanted to first confirm this, though the answer seems fairly clear to me. 

This is asked in the context of using a cluster managed by Polyaxon, rather than HTC. The splitting on Polyaxon is meant more for deep learning applications, but this would fall into the realm of loading data to train a single network, i.e. splitting the data loading and training over cores rather than separate time intervals of a single GATE simulation. Your solution is a really nice method for using the existing structure even for Polyaxon. 

Thanks so much for the post! 

Chris
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opengatecollaboration.org/pipermail/gate-users/attachments/20191216/5c0324f6/attachment.html>