[erlang-in-lisp-devel] event loop

[Faré]

I forgot to add the following:

4- By managing OS processes, I mean make sure that around our event
loop we promptly waitpid(2) for any process we fork(2) that may die,
so as to
 a- avoid zombie processes
 b- be able to handle process death the Erlang way, by killing the
rest of the process tree and/or otherwise sending the process manager
a death notification message (if enabled).

5- We WILL need an efficient event loop internally to solve the "C10K"
problem. http://www.kegel.com/c10k.html
 a- that's why I want the project to use and/or enhance IOlib, that
tries to provide a useful abstraction for an event-loop.
 b- on linux, we'll probably want to use epoll and sigfd's (through IOlib).
 c- on bsd/osx, we'll probably want to use kqueue (also through IOlib).
 d- on windows - we'll figure it out later (unless you're a windows
expert yourself or recruit one to this group)
 e- BTW, an "interesting" bug I once hit while using IOlib was trying
to persist an event loop in a saved lisp image -- oops, it contains
process-specific file descriptors, and you need to reinitialize it
when you restore the lisp image.

6- Note that you'll have to become familiar with the semantics of many
system calls as described in section 2 of the unix man(1) pages, even
though you may or may not call them directly but hopefully through
IOlib (thin or thick) abstractions.
 a- a short list of interesting man pages that you'll read before this
project is completely started: fork(2), wait(2), pipe(2),
socketpair(2), socket(7), send(2), recv(2), cmsg(3), bind(2),
listen(2), accept(2), shutdown(2), socket(2), ip(7), tcp(7), udp(7),
unix(7), select(2), epoll(2)(linux), kqueue(2)(bsd)...
 b- you don't have to know all the content of these manpages by heart.
However,  a quick read or two to know what exists can be good. Then
you'll come back to the man pages as you need them.

7- Do not hesitate to ask questions. We're here to help.
 a- We're experienced, and we might be right when you are wrong
 b- We're not infallible, and you might be right when we are wrong
 c- The ultimate test of who's right is running code. Don't accept any
other standard.
 d- Existing Erlang implementations contain a lot of such running
code. I don't have much knowledge about that body of code, but Robert
does, and you may want to dig into it at times.

8- When in doubt, write a simple program and test your hypotheses.
Ideally, the Lisp REPL should help make such testing relatively easy.

9- This data, too, might be usefully added to documentation.

10- Apart from that, I'm glad you can start working on this as soon as
wednesday. It would be very nice to receive a daily report on this
list and/or a blog (did you start one? or just add a category to your
existing blog?)

[ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ]
Invent a clever saying, and your name shall live forever.
        -- Anonymous


2008/5/20 Faré <fahree at gmail.com>:
> (oops, I initially sent this answer only to Matt)
>
> 1- There are several semi-portable posix libraries for CL. Osicat and
> IOlib are two of them. I happen to like IOlib, or at least think it's
> a good platform to start from and hack further.
>
> 2- as for fork, yes, you should start with just a simple CL-based
> wrapper to posix fork, and build around it.
>  a- see however how in philip-jose I provide (at least in SBCL) a
> mechanism to proactively GC before you fork to avoid an early GC in
> the child.
>
> 3- the purpose of my heavily-frobbing fork was to provide a portable
> way to fork-and-exec even when you can't control the GC.
>  a- If a GC happens in the child in between the fork and the exec, the
> result may be serious lossage, especially if the parent had multiple
> system threads running. It is important then to frob and exec
> "atomically" with respect to the Lisp heap.
>  b- This isn't necessary for Erlang-like programming, only for CLSH
> shell-like activity.
>
> 4- writing a portable "run-program" however is a good way to test the
> (OS)process-tracking and fd-frobbing mechanisms.
>  a- such a step I think is essential before we attempt to implement
> fancy communication protocols between processes.
>  b- Having all the bells-and-whistles of my attempted frob-and-fork
> implementation is not necessary, and a first pure-lisp attempt is
> probably preferrable
>
> In short, start simple. If you don't understand why I (or someone
> else) wrote some code in philip-jose or some other code, then it is
> good to
>  a- start by trying without the complexity (maybe doing things better
> indeed, or maybe learning why and how the complexity has to be added)
>  b- inquire (as you just did) why the complexity was added.
>
> [ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ]
> When your hammer is C++, everything begins to look like a thumb.
>       -- Steve Hoflich on comp.lang.c++
>
>
> 2008/5/18 Matt Bone <thatmattbone at gmail.com>:
>> Ok, so I've been getting up to speed on my systems programming in
>> common lisp.  I've actually been a bit surprised that there is no
>> portable posix interface (though it shouldn't be too hard to make the
>> things we're interested in work on various implementations).
>>
>> As far as pulling in the fork stuff from philip-jose, I've been
>> screwing around with a lot of different things.  If it's alright with
>> everyone I think I'd prefer to abandon the C file and just use
>> implementation specific posix calls (like sb-posix:fork) when
>> available and CFFI when not.
>>
>> To be honest I'm still not understanding how far we need to take the
>> fork mechanism.  Are we trying to match, for example, the features in
>> SCSH so this can be a useful standalone component?  Or is this just
>> the concurrency mechanism we're looking to use as a starting point (if
>> so, I have a few other questions/ideas that I'll need to
>> ask/consider).
>>
>> I'll be away at a conference to present a paper tomorrow and tuesday.
>> After that, all my non-SoC obligations are complete until the fall.
>>
>> --matt