Environment for running over multiple SSH machines?

(Samuel Thiriot) #1

Hello,

just wondering if you have an environment for running remotely tasks not only on one remote machine contacted by SSH, but several ones - typically, when you have about 10 colleagues having good computers, then you can start playing already without shifting to a cluster.

Maybe there is already some kind of a meta-environment which proposes to dispatch over several environments?

Thanks !
S.

(Jonathan Passerat Palmbach) #2

Hey @samthiriot,

Somebody asked the very same thing on the chat a couple of days ago, exciting times!
Basically, that’s something that I’d started playing with a couple of years ago with a student of mine, but we never really rolled it out to production.

At the time, there was little demand for this use case, and it was mostly us at Imperial with multiple clusters to gather together.

The code is still there https://github.com/openmole/hybrid_environment

Happy to assist and tell you more about it (I still have the associated thesis) if that’s something you’d be interested in rebooting!

Cheers
J

#3

For me the way to achieve this functionality is by deploy a SLURM cluster on top of the group of machine. @jopasserat has started something like that using Ansible. @jopasserat do you have a link to the script ?

(Jonathan Passerat Palmbach) #4

Yeah that’s another way of doing it, provided that you have access to Docker on their machine, and ideally root.

This other project is here, has only been tested on a small cloud deployment so far: https://github.com/AlterEgo7/slurm-kubernetes/tree/gluster-storage-fix

(also have the associated thesis :slight_smile: )

#5

I just forked it in the openmole org: https://github.com/openmole/slurm-kubernetes

(Samuel Thiriot) #6

Ok, thanks a lot to you both!

@reuillon in my context, the idea would be not to install anything. I already have access to a SLURM cluster. SSH would more stand like a 2min-solution for quick testing before submitting the tasks to a cluster (with quiet of a bit of waiting in the queue). So I guess I would not invest in deploying a SLURM for my specific case. Still it’s good to know it can be done!

@jopasserat The hybrid environment sounds nice. I’ll give it a try, but I’m not sure I’ve a lot of time to devote to test it. I’ll keep you posted in case I can devote more efforts to it.

Tks !