From fahree at gmail.com Thu May 8 05:40:30 2008 From: fahree at gmail.com (=?ISO-8859-1?Q?Far=E9?=) Date: Thu, 8 May 2008 01:40:30 -0400 Subject: [erlang-in-lisp-devel] Erlang in Lisp In-Reply-To: <653bea160805072237m55cd061ekcd4646b255e46a8e@mail.gmail.com> References: <653bea160804251601j4698718u273c4367db1af4c4@mail.gmail.com> <653bea160805072237m55cd061ekcd4646b255e46a8e@mail.gmail.com> Message-ID: <653bea160805072240s179dbc3wb0d802df2002bc69@mail.gmail.com> Dear Matt, dear Robert, LIST I propose we continue these conversations on the public mailing-list erlang-in-lisp-devel at common-lisp.net: http://common-lisp.net/cgi-bin/mailman/listinfo/erlang-in-lisp-devel REPO OK, I pulled a copy of the git repo, and see you've been active importing stuff. I think that a shell script to checkout and/or update the existing code would have been better, but let's say what's there is good where it is. You may want to try the other approach for pulling the erlang sources, or the sources of other CL libraries we'll need: iolib and its dependencies, etc. PLANNING as far as google is concerned, we're still in a planning phase, not the mad coding phase yet, but "learn about what exists, learn about each other, draw a plan". FORK Since the first step is forking, you can see in philip-jose what I was thinking of regarding the fork thing -- grep for fork there, except that the code isn't very developed. The idea would be that there should be a DSL to express all the parameters that you may want to preserve or tweak when you fork: shuffling fd's, closing some, changing directory, dropping privileges, changing process group, and all kind of weird things. Then, either you exec as part of the same C call (so as to not risk any lisp runtime confusion because of GC or other interruptions), or you immediately do a lot of adjustments before/after you get back to the lisp side in the child: frob the fd streams, reset the thread/lock structures and/or if any (hopefully none), etc. See also how the lisp-based fork tries to GC before to fork if tight in memory. But for the very first step, I think that a portable run-program implementation based on a simple C fork with a trivial DSL for shuffling fd's would be a good first goal. We should be using iolib and synchronous wait for the process management, not signal handlers. Luigi said he had such an implementation - we may ask him about it, or start from the run-program present in an existing CL implementation (not SBCL or CMUCL - it's broken with broken signal handling race conditions). YOUR TURN sorry for my long silence. I was busy at work. I'll try to be on track now. Matt: what about posting regular updates on your blog and/or on this list? [ Fran?ois-Ren? ?VB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] The Constitution may not be perfect, but it's a lot better than what we've got! 2008/4/26 Matt Bone : > I have my cl.net account and I created a git repository (empty at the > moment). I'll write this up, but for now we can: > git clone http://common-lisp.net/project/erlang-in-lisp/git > > and to push a patch: > git push /project/erlang-in-lisp/public_html/git master > > --matt > > > > On Fri, Apr 25, 2008 at 6:01 PM, Far? wrote: > > Robert Virding was making lots of interesting suggestions as to which > > level of Erlang implementation we want so as to leverage as much as > > possible of the Erlang front-end and support. It looks like Core > > Erlang or Kernel Erlang would be most fit. > > > > Drew: can you give an ETA for creating those common-lisp.net accounts, > > GIT repository and mailing-list? I suppose Robert should get an > > account, too. > > > > Matt: can you download all relevant software/documents and make a list > > of URLs/checkout commands so everyone else can do it, too? > > > > [ Fran?ois-Ren? ?VB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] > > Most people think they need a ruler. Perhaps we should give them a fake > > one that doesn't actually do anything, and then they won't think about > > it. It is sort of like giving an infant a pacifier. -- Perry Metzger From thatmattbone at gmail.com Sun May 18 22:14:19 2008 From: thatmattbone at gmail.com (Matt Bone) Date: Sun, 18 May 2008 17:14:19 -0500 Subject: [erlang-in-lisp-devel] Re: Erlang in Lisp In-Reply-To: <3dbc6d1c0805081710k4727a65fn96d804c04c8f3f9e@mail.gmail.com> References: <653bea160804251601j4698718u273c4367db1af4c4@mail.gmail.com> <653bea160805072237m55cd061ekcd4646b255e46a8e@mail.gmail.com> <3dbc6d1c0805081710k4727a65fn96d804c04c8f3f9e@mail.gmail.com> Message-ID: Ok, so I've been getting up to speed on my systems programming in common lisp. I've actually been a bit surprised that there is no portable posix interface (though it shouldn't be too hard to make the things we're interested in work on various implementations). As far as pulling in the fork stuff from philip-jose, I've been screwing around with a lot of different things. If it's alright with everyone I think I'd prefer to abandon the C file and just use implementation specific posix calls (like sb-posix:fork) when available and CFFI when not. To be honest I'm still not understanding how far we need to take the fork mechanism. Are we trying to match, for example, the features in SCSH so this can be a useful standalone component? Or is this just the concurrency mechanism we're looking to use as a starting point (if so, I have a few other questions/ideas that I'll need to ask/consider). I'll be away at a conference to present a paper tomorrow and tuesday. After that, all my non-SoC obligations are complete until the fall. --matt From rvirding at gmail.com Sun May 18 23:06:14 2008 From: rvirding at gmail.com (Robert Virding) Date: Mon, 19 May 2008 01:06:14 +0200 Subject: [erlang-in-lisp-devel] Re: Erlang in Lisp In-Reply-To: References: <653bea160804251601j4698718u273c4367db1af4c4@mail.gmail.com> <653bea160805072237m55cd061ekcd4646b255e46a8e@mail.gmail.com> <3dbc6d1c0805081710k4727a65fn96d804c04c8f3f9e@mail.gmail.com> Message-ID: <3dbc6d1c0805181606o5667bcf7p820fa7fc636c92af@mail.gmail.com> I saw the reference to fork in the earlier message. Seeing I have come in a little late you will have to excuse my asking. What do you intend to use fork for in erlang-in-lisp? Erlang processes? Many Erlang processes can generate many processes. Robert 2008/5/19 Matt Bone : > Ok, so I've been getting up to speed on my systems programming in > common lisp. I've actually been a bit surprised that there is no > portable posix interface (though it shouldn't be too hard to make the > things we're interested in work on various implementations). > > As far as pulling in the fork stuff from philip-jose, I've been > screwing around with a lot of different things. If it's alright with > everyone I think I'd prefer to abandon the C file and just use > implementation specific posix calls (like sb-posix:fork) when > available and CFFI when not. > > To be honest I'm still not understanding how far we need to take the > fork mechanism. Are we trying to match, for example, the features in > SCSH so this can be a useful standalone component? Or is this just > the concurrency mechanism we're looking to use as a starting point (if > so, I have a few other questions/ideas that I'll need to > ask/consider). > > I'll be away at a conference to present a paper tomorrow and tuesday. > After that, all my non-SoC obligations are complete until the fall. > > --matt > _______________________________________________ > erlang-in-lisp-devel mailing list > erlang-in-lisp-devel at common-lisp.net > http://common-lisp.net/cgi-bin/mailman/listinfo/erlang-in-lisp-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fahree at gmail.com Tue May 20 00:33:29 2008 From: fahree at gmail.com (=?ISO-8859-1?Q?Far=E9?=) Date: Mon, 19 May 2008 20:33:29 -0400 Subject: [erlang-in-lisp-devel] Using fork Message-ID: <653bea160805191733i81e1c47sdbd354896a140090@mail.gmail.com> My plan for Erlang-in-Lisp was to start with fork as the primary method for spawning language-level processes using a "one OS process per Erlang process" model indeed. The reasons follow. 1- Yes, the performance of fork doesn't scale and thus severely limits the applicability of the actor paradigm, but scaling can be achieved later by other means: portable lisp-in-lisp green threads (as in the Tube or philip-jose), or implementation-specific green threads. 2- fork is portable (to any unix/cygwin), and provides desirable isolation properties that would require a large number of hours of debugging to enforce using other methods (especially when Lisp implementations and libraries were not designed to be thread-safe). Yes, these other methods will have to be implemented eventually, but fork-based concurrency will allow us to have a reference implementation by then that can help discriminate between implementation bugs vs application bugs. 3- Fork-based concurrency also provides for very desirable isolation from bugs in third-party libraries, and for native compilation of lisp code without an intermediate translation layer - two properties that we will certainly want to preserve even when we have other implementation strategies available for multiprogramming. 4- last but not least, fork-based concurrency seems like it's the easiest, cheapest and fastest way to get an initial working system. So no, fork-based concurrency not the be-all end-all of Erlang-in-Lisp, but I really think it's the best way to start. Once we have a simple working implementation that doesn't scale, we can add new strategies: * lisp threads as in cl-muproc, whether the underlying lisp uses system threads (as SBCL does) or green threads (as CMUCL does), * a portable but very slow meta-programming green threads as in philip-jose (uses arnesi's slow ((lisp with continuations) in lisp) interpreter), that should allow migration of processes (within some restrictions). * a similar portable but less slow method using a lisp-in-lisp compiler instead like the one in Screamer, which might not allow for migration (or might, if we use the same tricks as the Tube). * pushing the code to a remote Erlang implementation (whether Erlang-in-Lisp or plain Erlang), if it doesn't use any of our Lisp-specific "extensions", but only things available with say LFE. * any other method du jour. Hopefully, we can make it easy to select what strategy we use to spawn any particular language-level process. (Hum, such design rationale should probably be pushed into a design document included with the code.) [ Fran?ois-Ren? ?VB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] I don't program enough to be called a "programmer", and I don't non-program enough to be called a "non-programmer". So I'm a non-non-programmer, which, in intuitionistic logic, is weaker than a programmer. -- Far? 2008/5/18 Robert Virding : > I saw the reference to fork in the earlier message. Seeing I have come in a > little late you will have to excuse my asking. What do you intend to use > fork for in erlang-in-lisp? Erlang processes? Many Erlang processes can > generate many processes. > > Robert From fahree at gmail.com Tue May 20 10:29:53 2008 From: fahree at gmail.com (=?ISO-8859-1?Q?Far=E9?=) Date: Tue, 20 May 2008 06:29:53 -0400 Subject: [erlang-in-lisp-devel] Re: Erlang in Lisp In-Reply-To: <653bea160805191754t2d1ec91aoab3eed4bda746110@mail.gmail.com> References: <653bea160804251601j4698718u273c4367db1af4c4@mail.gmail.com> <653bea160805072237m55cd061ekcd4646b255e46a8e@mail.gmail.com> <3dbc6d1c0805081710k4727a65fn96d804c04c8f3f9e@mail.gmail.com> <653bea160805191754t2d1ec91aoab3eed4bda746110@mail.gmail.com> Message-ID: <653bea160805200329v6049403am855a08257a09886d@mail.gmail.com> (oops, I initially sent this answer only to Matt) 1- There are several semi-portable posix libraries for CL. Osicat and IOlib are two of them. I happen to like IOlib, or at least think it's a good platform to start from and hack further. 2- as for fork, yes, you should start with just a simple CL-based wrapper to posix fork, and build around it. a- see however how in philip-jose I provide (at least in SBCL) a mechanism to proactively GC before you fork to avoid an early GC in the child. 3- the purpose of my heavily-frobbing fork was to provide a portable way to fork-and-exec even when you can't control the GC. a- If a GC happens in the child in between the fork and the exec, the result may be serious lossage, especially if the parent had multiple system threads running. It is important then to frob and exec "atomically" with respect to the Lisp heap. b- This isn't necessary for Erlang-like programming, only for CLSH shell-like activity. 4- writing a portable "run-program" however is a good way to test the (OS)process-tracking and fd-frobbing mechanisms. a- such a step I think is essential before we attempt to implement fancy communication protocols between processes. b- Having all the bells-and-whistles of my attempted frob-and-fork implementation is not necessary, and a first pure-lisp attempt is probably preferrable In short, start simple. If you don't understand why I (or someone else) wrote some code in philip-jose or some other code, then it is good to a- start by trying without the complexity (maybe doing things better indeed, or maybe learning why and how the complexity has to be added) b- inquire (as you just did) why the complexity was added. [ Fran?ois-Ren? ?VB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] When your hammer is C++, everything begins to look like a thumb. -- Steve Hoflich on comp.lang.c++ 2008/5/18 Matt Bone : > Ok, so I've been getting up to speed on my systems programming in > common lisp. I've actually been a bit surprised that there is no > portable posix interface (though it shouldn't be too hard to make the > things we're interested in work on various implementations). > > As far as pulling in the fork stuff from philip-jose, I've been > screwing around with a lot of different things. If it's alright with > everyone I think I'd prefer to abandon the C file and just use > implementation specific posix calls (like sb-posix:fork) when > available and CFFI when not. > > To be honest I'm still not understanding how far we need to take the > fork mechanism. Are we trying to match, for example, the features in > SCSH so this can be a useful standalone component? Or is this just > the concurrency mechanism we're looking to use as a starting point (if > so, I have a few other questions/ideas that I'll need to > ask/consider). > > I'll be away at a conference to present a paper tomorrow and tuesday. > After that, all my non-SoC obligations are complete until the fall. > > --matt From fahree at gmail.com Tue May 20 11:06:34 2008 From: fahree at gmail.com (=?ISO-8859-1?Q?Far=E9?=) Date: Tue, 20 May 2008 07:06:34 -0400 Subject: [erlang-in-lisp-devel] event loop Message-ID: <653bea160805200406r355b577doc8c04114bce94e0e@mail.gmail.com> I forgot to add the following: 4- By managing OS processes, I mean make sure that around our event loop we promptly waitpid(2) for any process we fork(2) that may die, so as to a- avoid zombie processes b- be able to handle process death the Erlang way, by killing the rest of the process tree and/or otherwise sending the process manager a death notification message (if enabled). 5- We WILL need an efficient event loop internally to solve the "C10K" problem. http://www.kegel.com/c10k.html a- that's why I want the project to use and/or enhance IOlib, that tries to provide a useful abstraction for an event-loop. b- on linux, we'll probably want to use epoll and sigfd's (through IOlib). c- on bsd/osx, we'll probably want to use kqueue (also through IOlib). d- on windows - we'll figure it out later (unless you're a windows expert yourself or recruit one to this group) e- BTW, an "interesting" bug I once hit while using IOlib was trying to persist an event loop in a saved lisp image -- oops, it contains process-specific file descriptors, and you need to reinitialize it when you restore the lisp image. 6- Note that you'll have to become familiar with the semantics of many system calls as described in section 2 of the unix man(1) pages, even though you may or may not call them directly but hopefully through IOlib (thin or thick) abstractions. a- a short list of interesting man pages that you'll read before this project is completely started: fork(2), wait(2), pipe(2), socketpair(2), socket(7), send(2), recv(2), cmsg(3), bind(2), listen(2), accept(2), shutdown(2), socket(2), ip(7), tcp(7), udp(7), unix(7), select(2), epoll(2)(linux), kqueue(2)(bsd)... b- you don't have to know all the content of these manpages by heart. However, a quick read or two to know what exists can be good. Then you'll come back to the man pages as you need them. 7- Do not hesitate to ask questions. We're here to help. a- We're experienced, and we might be right when you are wrong b- We're not infallible, and you might be right when we are wrong c- The ultimate test of who's right is running code. Don't accept any other standard. d- Existing Erlang implementations contain a lot of such running code. I don't have much knowledge about that body of code, but Robert does, and you may want to dig into it at times. 8- When in doubt, write a simple program and test your hypotheses. Ideally, the Lisp REPL should help make such testing relatively easy. 9- This data, too, might be usefully added to documentation. 10- Apart from that, I'm glad you can start working on this as soon as wednesday. It would be very nice to receive a daily report on this list and/or a blog (did you start one? or just add a category to your existing blog?) [ Fran?ois-Ren? ?VB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] Invent a clever saying, and your name shall live forever. -- Anonymous 2008/5/20 Far? : > (oops, I initially sent this answer only to Matt) > > 1- There are several semi-portable posix libraries for CL. Osicat and > IOlib are two of them. I happen to like IOlib, or at least think it's > a good platform to start from and hack further. > > 2- as for fork, yes, you should start with just a simple CL-based > wrapper to posix fork, and build around it. > a- see however how in philip-jose I provide (at least in SBCL) a > mechanism to proactively GC before you fork to avoid an early GC in > the child. > > 3- the purpose of my heavily-frobbing fork was to provide a portable > way to fork-and-exec even when you can't control the GC. > a- If a GC happens in the child in between the fork and the exec, the > result may be serious lossage, especially if the parent had multiple > system threads running. It is important then to frob and exec > "atomically" with respect to the Lisp heap. > b- This isn't necessary for Erlang-like programming, only for CLSH > shell-like activity. > > 4- writing a portable "run-program" however is a good way to test the > (OS)process-tracking and fd-frobbing mechanisms. > a- such a step I think is essential before we attempt to implement > fancy communication protocols between processes. > b- Having all the bells-and-whistles of my attempted frob-and-fork > implementation is not necessary, and a first pure-lisp attempt is > probably preferrable > > In short, start simple. If you don't understand why I (or someone > else) wrote some code in philip-jose or some other code, then it is > good to > a- start by trying without the complexity (maybe doing things better > indeed, or maybe learning why and how the complexity has to be added) > b- inquire (as you just did) why the complexity was added. > > [ Fran?ois-Ren? ?VB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] > When your hammer is C++, everything begins to look like a thumb. > -- Steve Hoflich on comp.lang.c++ > > > 2008/5/18 Matt Bone : >> Ok, so I've been getting up to speed on my systems programming in >> common lisp. I've actually been a bit surprised that there is no >> portable posix interface (though it shouldn't be too hard to make the >> things we're interested in work on various implementations). >> >> As far as pulling in the fork stuff from philip-jose, I've been >> screwing around with a lot of different things. If it's alright with >> everyone I think I'd prefer to abandon the C file and just use >> implementation specific posix calls (like sb-posix:fork) when >> available and CFFI when not. >> >> To be honest I'm still not understanding how far we need to take the >> fork mechanism. Are we trying to match, for example, the features in >> SCSH so this can be a useful standalone component? Or is this just >> the concurrency mechanism we're looking to use as a starting point (if >> so, I have a few other questions/ideas that I'll need to >> ask/consider). >> >> I'll be away at a conference to present a paper tomorrow and tuesday. >> After that, all my non-SoC obligations are complete until the fall. >> >> --matt From thatmattbone at gmail.com Fri May 30 14:05:39 2008 From: thatmattbone at gmail.com (Matt Bone) Date: Fri, 30 May 2008 09:05:39 -0500 Subject: [erlang-in-lisp-devel] fare-match question Message-ID: This is not a huge deal, and probably a bit out of the order we discussed. I started thinking about pattern matching yesterday, though, and I was trying to do the simple example in fare-matcher (a simple modification of what's in the docs): (defun test-match (param) (fare-matcher:match param (:ping (print :pong)) (:quit (print :bye)) ((:add a b) (print :add)))) Matching with the simple symbols ping and quit works just fine, but things go south when trying to match with add and bind a and b. I was attempting to do: (test-match '(:add 1 2)) But something is wrong in the macroexpansion. Am I even using my test-match function properly? Does match handle quoted sexpr properly or was I just masking this problem with :ping and :quit by handing it symbols that evaluate to themselves? Also, since I believe both authors are on the list, should I consider using cl-match instead? Or are our pattern matching requirements so minimal that a) it does not matter and b) we could even roll our own ala some of the Norvig examples? --matt From sionescu at common-lisp.net Fri May 30 14:22:42 2008 From: sionescu at common-lisp.net (Stelian Ionescu) Date: Fri, 30 May 2008 16:22:42 +0200 Subject: [erlang-in-lisp-devel] fare-match question In-Reply-To: References: Message-ID: <20080530142242.GA69752@universe.org> On Fri, May 30, 2008 at 09:05:39AM -0500, Matt Bone wrote: >This is not a huge deal, and probably a bit out of the order we >discussed. I started thinking about pattern matching yesterday, >though, and I was trying to do the simple example in fare-matcher (a >simple modification of what's in the docs): >(defun test-match (param) > (fare-matcher:match param > (:ping (print :pong)) > (:quit (print :bye)) > ((:add a b) (print :add)))) that should be (list :add a b) -- Stelian Ionescu a.k.a. fe[nl]ix Quidquid latine dictum sit, altum videtur. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: not available URL: From fahree at gmail.com Fri May 30 14:26:40 2008 From: fahree at gmail.com (=?ISO-8859-1?Q?Far=E9?=) Date: Fri, 30 May 2008 10:26:40 -0400 Subject: [erlang-in-lisp-devel] fare-match question In-Reply-To: <20080530142242.GA69752@universe.org> References: <20080530142242.GA69752@universe.org> Message-ID: <653bea160805300726p4bd4b046mb962c1ae05d78dbc@mail.gmail.com> 2008/5/30 Stelian Ionescu : > On Fri, May 30, 2008 at 09:05:39AM -0500, Matt Bone wrote: >>This is not a huge deal, and probably a bit out of the order we >>discussed. I started thinking about pattern matching yesterday, >>though, and I was trying to do the simple example in fare-matcher (a >>simple modification of what's in the docs): >>(defun test-match (param) >> (fare-matcher:match param >> (:ping (print :pong)) >> (:quit (print :bye)) >> ((:add a b) (print :add)))) > > that should be (list :add a b) Or alternatively, `(:add ,a ,b) using fare-quasiquote (that I don't recommend at this point, because its support for nested ,@ is buggy and necessitates a rewrite). [ Fran?ois-Ren? ?VB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] The American Republic will endure, until politicians realize they can bribe the people with their own money. -- Alexis de Tocqueville