[slime-devel] Slime and clusters

[Zach Kost-Smith]

I wonder if anybody has yet to tread this path.

I have been using CL at on my school's cluster.  I usually develop within
Slime locally, then move it to the cluster's master node and have a trial
run over a remote Slime session, then once everything works, I ditch Slime
and invoke my programs on the worker nodes using a bash or Lisp script (yes,
we are running without a queue; crazy, huh?).  I would very much like to use
dispatch the jobs to the worker nodes from within Slime.  More to the point,
I would like to be able to start jobs on the worker nodes from my Slime
connection to the master node.  Now, of course I can connect to each node
using a tunnel and start the jobs that way, but I want an automated
interface as sketched below.

I was thinking about developing something that would work for me and perhaps
others.  I just wanted to put this out there first to make sure it hasn't
already been done and see if there are any suggests or comments on an
undertaking like this.  Basically all I would do is define a CL client for
swank and use that to connect to worker Lisps running on each worker node
from the one instance on the master.  Then, if I want to send a job to the
first node, all I have to do is send some code over the swank connection for
the worker to evaluate and return the value.  I also thought I'd write a
kind of forwarder for the Swank connection so I can jump to a worker node
from the master by setting the master to forward all data to the worker, and
return anything coming from the worker down its connect to Slime (although,
I suppose this could be arbitrarily deep, this is analogous to jumping from
server to server using ssh but without the secure-ness).  I am fairly
convinced that this will also work (sans persistent worker Lisps) for any
cluster with a queuing system that allows for interactive jobs (like the
Torque/Moab combo that we use on another cluster here) and probably in a EC2
Cloud like environment as well.  I'm just saying that it is possible that
this could be generally useful for other people as well.

Almost all of this would be on the Swank side (and thank goodness because I
feel like a fish out of water in ELisp).  However, I have already had
trouble understanding the Slime message protocol.  I'll figure it out
eventually, but are there any good internals guides for Slime/Swank?

Any advice is appreciated,

Zach KS
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.common-lisp.net/pipermail/slime-devel/attachments/20111019/5f82fac9/attachment.html>