[eurolisp] Re: http://www.dirkgerrits.com/erlisp/roadmap.html

[Dirk Gerrits]

Hi guys.  

First of all, thanks for your input.  Second, I owe you an apology for
wasting your time.  A lot of what you have supplied me with in this
thread was already known to me, but I failed to make that evident on my
site.

Faré <fahree at gmail.com> writes:

>>>: "Hoehle, Joerg-Cyril" <Joerg-Cyril.Hoehle at t-systems.com>
>>: Dirk Gerrits <dirk at dirkgerrits.com>
>
>>> Therefore I was disappointed by your roadmap which talks a lot about
>>> OS threads and special (thread-local) variables.
>> 
>> Sorry to hear that, care to elaborate?
>>
> After a few tens of thousands of threads (may hundreds of thousands,
> on modern machines with several gibibytes of RAM?), the performance of
> most (all) Unix kernels becomes very bad. If only because each thread
> eats at the very least a few kilobytes of kernel memory, and
> ultimately the machine runs out of memory. Green thread have a
> potentially much lower overhead.

I was more interested in the special variables comment, and about my
roadmap talking "a lot" about OS threads.  I realize now that I did not
make that clear *at all* in my reply.  Sorry.

>>   * Using continuations as green threads.  This would limit the points
>>     at which processes can be preempted, but that can be partially
>>     alleviated by assigning an operating system thread to every few
>>     green threads.  Also, less preemption is not necessarily a problem.
>>
> With sufficient engineering and code-walking, preemption can remain
> possible. 

I do not see how a process calling a Common Lisp sequence function on a
large array can be preempted by sufficient engineering and
code-walking.  But please enlighten me, I'm just a newbie.

If you meant that preemption in user functions is possible, then I think
I know what you mean.  In Erlang implementation, it is customary (as far
as I have been able to tell so far) to have a "reduction counter" that
gets decremented on every function call.  When it reaches zero, the
process is preempted and another process gets a chance to run, with a
reset reduction counter.  This shouldn't be too hard to implement in a
CPS code walker, but I worry about its performance.  Nevertheless, it is
something I'm planning to try, as only trying it out can give me any
real data on its performance.

> However, preemption necessitates proper handling of invariants, with
> locking, roll-back and/or roll-forward.  As a cheaper way to do
> things, cooperative multithreading can do, and can be used
> automatically or semi-automatically. (Only potentially long-running
> routines need have (yield) statements inserted, and with proper
> code-walking, this can be done automatically.)

What I have considered doing is to have cooperative multithreading that
yields /only/ on Erlang's receive statements.  I'm curious as to whether
that is really as bad as it sounds.  

I could indeed have extra yields be added automatically, but that may be
more tricky than you make it sound (or maybe I'm misinterpreting).  For
example, putting a yield in a tight loop might kill performance, and not
putting it there might kill cooperation.  Trying to solve this would
bring us pretty close to the above reduction counter, wouldn't you
agree? 

>> and a CPS code walker is more work than using an OS thread library.
> There already are CPS code walkers, in Screamer and in UCW.

I've looked at those in Arnesi, UCW, and Screamer in the past and know
that I can't use them out-of-the-box.  But I could "borrow" some code
and adapt it to my purposes, is that what you suggest?  I am not opposed
to that, althoug I think that may deprive me of a valuable Common Lisp
learning experience.

>>> Cf. http://c2.com/cgi/wiki?MessagingAsAlternativeToMultiThreading
>
>> http://groups.google.com/groups?selm=87acurudww.fsf%40dirkgerrits.com&rnum=1
>
>> I agree that a better place for discussion is needed.  How about a
>> common-lisp.net mailing list?
> If you create it, please subscribe me :)

After Edi's suggestion, I subscribed myself to CLUMP
(http://www.caddr.com/clump/).  Do you think that is a good place to
continue this discussion (and future Erlisp discussions?).

Kind regards,

Dirk Gerrits