Erlang style processes in Common Lisp

[shenanigans at sonic.net]

Thanks to all who responded!

Here's one message replying to highlights, yet as always very informed 
answers all around!




On 08/03/2015 12:36 PM, Attila Lendvai wrote:
>>> How might we get equivalent cheap ephemeral processes into a
>>> contemporary Common Lisp implementation?
>> In short, you need to write from scratch a new CL implementation. Current ones are not designed with the Erlang constraints in mind.
> well, Nikodemus had some plans for green threads for SBCL and it
> didn't sound like a rewrite.
>
> and adding first class heaps would be a very useful addition with or
> without any of the other erlang stuff. i don't know how hard that is,
> but i assume it should be rather simple if the responsibility is
> pushed over to the user to make sure that there are no dangling
> pointers after destroying a heap.
>
> or am i missing something?

This would give a lot of traction for my projects, especially for 
*tasks* that generate a lot of garbage to be otherwise collected. (e.g., 
one use case: lots of temporary hash tables that grow to many GB within 
a minute, queried a few dozen times, then discarded.  I've previously 
used 100 line Perl scripts for this, just to avoid the gc hit.  Called 
it via file based queue and/or pipe.  Everything else in SBCL or CCL.)



On 08/03/2015 08:18 AM, Scott McKay wrote:
> Have you looks at Lisp-Flavored Erlang ("LFE")?
>  - http://lfe.io
>
> It's really quite interesting, IMO.
>
> --Scott

Yes, I have worked with LFE, Elixir and Erlang.  I was hoping to get 
away from their single mailbox per process limitation, among other issues.

Robert & Duncan are doing an excellent job on LFE.  It's come a long way 
in the past year or so.  If anyone hasn't tried it lately, please do! 
For instance, they added a macro for those who were put off by the colon 
operator, so more conventional module:fn has been usable for over a 
year.  And Jose has a very extensive macro system for Elixir that's 
possibly closest to Common Lisp's for a non-Lisp.



On 08/03/2015 01:19 PM, Stelian Ionescu wrote:
> 2) It has a bytecode-interpreting VM in which each instruction is a safepoint and all I/O is non-blocking, basically implementing scheduler preemption in user space. This allow Erlang watcher processes to randomly kill worker processes and restart them.

Ah, that's the insight to their details I was unaware of. Thanks!

In earlier CL approaches, I just wrapped the main loop in a handler-case 
and found that sufficient for my needs, but others may find my approach 
lacking...  And you don't want to know about the external watchdog for 
something similar but in C.

Erlang's VM has a very large set of use cases that they are supporting, 
far larger than the proposed set of features from the original posting.

As you also stated (but trimmed here), yes I can make do without 
Erlang's robustness guarantees. A tens of thousands of workers without a 
heavy thread pool and fast/cheap workers are my main use cases.



On 08/03/2015 12:46 PM, Faré wrote:
> Not Common Lisp (yet), but Racket has custodians for first-class 
> resource allocation pool, and all kinds of concurrency primitives. 
> Since it's a programming language designed to implement other 
> programming languages on top of it, it would make a great basis for a 
> "new" common lisp implementation.

Thanks for that.  I've been looking for an excuse to delve deeper into 
Racket, but since I started with scheme R3 in college, that pesky 
explicit value for false still feels wrong to me.  ;^)


Thanks again everyone,
-Daniel