[slime-devel] Partial multiprocessing support on CMUCL

[Luke Gorrie]

Ahoy,

I checked in some multiprocessing support under CMUCL.

[While writing this mail, I realised that it's seriously broken --
you'll notice my tone changes towards the bottom of the mail :-)]

With respect to what I said a "version one" should be, this is about
half finished. It might at least be a major improvement for people
already living on the edge and using threads :-)

Obligatory screenshot, debugging three McCLIM Listener processes here:
http://www.bluetail.com/~luke/misc/lisp/slime-threads.png

To use it, add this to your .emacs:

  (setq slime-multiprocessing t)
  (setq slime-global-debugger-hook t)

(That second line installs the slime debugger hook globally -- this is
a convenience to automatically setup the hook that Dan made for
Aranaida use.)

And this to your ~/.swank.lisp:

  (setq swank::*start-swank-in-background* t)   ;; Use SERVE-EVENT

The good parts:

  Threads that "asynchronously" hit an error -- i.e. not while
  performing an RPC for Emacs -- will suspend until Emacs decides to
  debug them.

  The Lisp side uses locks so that only one thread will enter a
  conversation with Emacs at a time. If several threads hit the
  debugger at once, they will always be debugged one-at-a-time.

  The Emacs side has a new "thread control" buffer (top right of the
  screenshot) listing all suspended threads. You press RET to "give
  the goahead" to one of them so that it enters the debugger.

The sharp edges and bare wires:

  I haven't done anything about input and output streams, aside from
  using a lock to atomic'ify all reads and writes to the Emacs socket.

  There an obscure-ish race condition: if you "gohead" a thread into
  the debugger, but send an RPC before it actually hits the debugger,
  the protocol will reach an inconsistent state and bail out.

Also, something really horrible that just occured while writing this
mail: there is no coherent scheme to say which thread is supposed to
be reading from the Emacs socket. There is a 'conversation-lock' that
prevents two threads from actually entering into talks with Emacs at
the same time, but nothing to tell the SERVE-EVENT loop that "hey,
this other thread is being debugged just now, so don't try to take the
I/O-lock and read from Emacs just because the file descriptor becomes
readable."

It seems to work anyway, but I can't say why. Shades of Java.

This is likely to be a problem to port to OpenMCL just now. OpenMCL
doesn't use SERVE-EVENT (which mysteriously seems to work) but instead
has a dedicated thread calling READ-FROM-EMACS to get and evaluate
requests. That thread should typically be blocking in a read with the
I/O-lock held, preventing any other thread (e.g. one trying to do
debugging) from reading from Emacs. At the moment the only workaround
I can think of is a bit.. unspeakable.

HELP, HELP! SAVE ME! I don't know how to write correct multithreaded
programs!

How do people do synchronization in Lisp? Do you use higher-level
abstractions than locks and shared variables, or is there some book I
can read to actually understand how to program with those things?

The mind boggles. Please someone stop me before I write a homebrew
message-passing system.

Cheers,
Luke