[eurolisp] Re: http://www.dirkgerrits.com/erlisp/roadmap.html

[Dirk Gerrits]

"Hoehle, Joerg-Cyril" <Joerg-Cyril.Hoehle at t-systems.com> writes:

> Therefore I was disappointed by your roadmap which talks a lot about
> OS threads and special (thread-local) variables.  

Sorry to hear that, care to elaborate?

> Recently I came again across Lightweight Languages Workshop 2002
> http://ll2.ai.mit.edu/ You'll notice that Joe Armstrong's web server
> starts where Apache has already broken down due to too many concurrent
> users. I you go with OS threads, you'll never get there.

Well OS threads are just the easiest way that Lisp offers me to
implement these abstractions.  As I wrote in my roadmap, I see two ways
to later improve performance:

  * Using continuations as green threads.  This would limit the points
    at which processes can be preempted, but that can be partially
    alleviated by assigning an operating system thread to every few
    green threads.  Also, less preemption is not necessarily a problem.

I've already done some work on this, but I'm currently swamped in school
work.  

  * Hacking the Common Lisp implementation.  If SBCL would have more
    light-weight threads, then so would Erlisp.  Alternatively, Erlisp
    could be ported to CLISP, where light-weight processes could be
    added to the virtual machine from scratch.  CLISP also has a socket
    library so porting Erlisp's distribution aspects should be
    straightforward."

This is something I'd prefer not to do, but I see no way around it if I
want to match or surpass Erlang's performance.  (And I /do/ want to.  I
apologize if my website gave you the wrong idea.)

> I reread part's of Graham's On Lisp yesterday. There's one section
> where he mentions that a completely unperformant but working thing is
> still valuable to start with because it allows experimentation
> (similar topic idea in the article about the failure of the NIL
> machine/project).

That's the reason I am using OS threads at all.  On Lisp's CPS macros
are just not satisfactory, and a CPS code walker is more work than using
an OS thread library.  So I'm going to get something up and running with
threads first (roadmap step 1) and then make it fast with a CPS code
walker / CL implementation hacking / whatever (roadmap step 3).  In
fact, I think I might swap step 2 and 3 as performance in parallelism is
currently more important to me than distribution.

> The reason why I want to depart from OS threads is because I expect
> different ways of computing arise when "threads" are extremely light
> weight (as in Erlang or Oz, not in UNIX), like different ways of
> computing arose when stack space became cheap or when memory became
> cheap. 

I am convinced of that as well.  Otherwise I wouldn't see a real need
for Erlisp, as a standard OS threading API would have been enough.

> I.e. these days, many GNU programs allocate a few MB on the
> stack without shame, and programs long do in memory only what once
> were expensive disk-bound operations. This allows different algorithms
> to be used to address problems.
>
> OTOH maybe one can wait until a typical Linux can fork a 300000
> threads on a 256MB machine (I haven't checked how far StateThreads and
> the other approaches can go). That's ~1KB per thread!

That would be a great thing for the GNU/Linux crowd, myself included,
but it wouldn't really solve anything for the rest of the world.

> Cf. http://c2.com/cgi/wiki?MessagingAsAlternativeToMultiThreading

Thanks for the reference, I'll check it out.

> PS: I didn't see a comp.lang.lisp thread, nor a wiki/message forum on
> your site and I still think eurolisp is not the right forum for this.

I did post at comp.lang.lisp:
http://groups.google.com/groups?selm=87acurudww.fsf%40dirkgerrits.com&rnum=1
but there have been absolutely no replies there.

I agree that a better place for discussion is needed.  How about a
common-lisp.net mailing list?

Kind regards,

Dirk Gerrits