[Gate-users] Multiple GATE simulations scheduling on cluster

[David Boersma]

Dear Francesca,


I had intended to reply to you via the mailing list, now I see that I misclicked. With this message the other gate users and developers (some have experience with docker) can follow the conversation. As I wrote in my first mail, it would be nice if we could include useful hints in our user docs about running GATE in polyaxon.


/David

________________________________
Von: Francesca De Benetti <francesca.debenetti at gmail.com>
Gesendet: Dienstag, 25. August 2020 18:11:10
An: David Boersma
Betreff: Re: [Gate-users] Multiple GATE simulations scheduling on cluster

Dear David,

thank you for your nice reply!
I am working on it with our “polyaxon masters” and I will also ask on the polyaxon slack support forum. I will let you know if we find a way to do that!

Best,

Francesca

Il giorno 24 ago 2020, alle ore 11:14, David Boersma <david.boersma at acmit.at<mailto:david.boersma at acmit.at>> ha scritto:

Dear Francesca,

There are several attempts at generalizing the task for submitting many GATE simulations to a cluster (gjs, GateLab). I hope that the developers/maintainers of some of those schemes can respond to your question (but several live in France, where it's still vacation time, I think, so you may need to be a bit patient).

I had never heard of "polyaxon" so I googled it and learned that it is some docker/kubernetes based framework for setting up very complex data science systems. So the first thing you'll need is a docker container with GATE. If you are working with a standard release, then you can download this from the GATE website. If you work with your own tweaked version of GATE, then it shouldn't be too hard to make a docker container yourself, maybe the GATE docker container maintainer (Maxime) can help you with that.

Since it looks like "polyaxon" is new to the GATE community (we are usually discussing how to make it work with very old cluster job management systems like SLURM, HTCondor, OpenPBS, etc), it might be a good idea to also ask your question on a "polyaxon" support forum (that seems to be hosted on "Slack", see https://polyaxon.com/docs/resources/). When you do that, make clear that you are not running a deep learning project, that you just want to submit a large number of single core number crunching jobs and analyze the results manually. And please let us know about your progress, so that we can try to include some information on this system in our docs.

Hope that helps,
David Boersma
________________________________
Von: Gate-users <gate-users-bounces at lists.opengatecollaboration.org<mailto:gate-users-bounces at lists.opengatecollaboration.org>> im Auftrag von Francesca De Benetti <francesca.debenetti at gmail.com<mailto:francesca.debenetti at gmail.com>>
Gesendet: Freitag, 21. August 2020 15:36:53
An: gate-users at lists.opengatecollaboration.org<mailto:gate-users at lists.opengatecollaboration.org>
Betreff: [Gate-users] Multiple GATE simulations scheduling on cluster

Dear GATE users,

in my current project, I need to run a huge number of GATE simulations on a cluster (polyaxon). I would like not to start each experiment manually, because it would be too time consuming.
At the moment I have some code to run one experiment group for each simulation on the cluster, and it works fine. My goal would be to schedule or run in parallel more than one simulation.
Is there anyone here expert with running GATE on polyaxon who has an idea how to do that?

Thank you very much for you help!

Best regards,

Francesca De Benetti
_______________________________________________
Gate-users mailing list
Gate-users at lists.opengatecollaboration.org<mailto:Gate-users at lists.opengatecollaboration.org>
http://lists.opengatecollaboration.org/mailman/listinfo/gate-users

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opengatecollaboration.org/pipermail/gate-users/attachments/20200825/f9f5e4ac/attachment.html>