From lavigne.eric at gmail.com Mon Jul 11 04:54:07 2005 From: lavigne.eric at gmail.com (Eric Lavigne) Date: Mon, 11 Jul 2005 04:54:07 +0000 Subject: [erlisp-devel] mistake in with-timeout? Message-ID: <51e1bc3005071021546cc3010a@mail.gmail.com> Dear Erlisp developers, I need some help to figure out a problem with my code. I am trying to add Allegro support to Erlisp, but I ran into some trouble as shown in this test result: ...........f........................ Did 36 checks. Pass: 35 (97%) Skip: 0 (0%) Fail: 1 (2%) Failure Details: RECEIVE-WITH-TIMEOUT []: NIL. I suspect my mistake is in erlisp:with-timeout. Here is a copy of that macro: (defmacro with-timeout ((milliseconds &rest after-timeout) &body body) #+sbcl `(sb-ext:with-timeout (/ ,milliseconds 1000) (handler-case (progn , at body) (sb-ext:timeout () , at after-timeout))) #+allegro `(mp:with-timeout ((/ ,milliseconds 1000) , at after-timeout) , at body) #-(or sbcl allegro) (error "Timeouts are not yet implemented for your compiler.")) The part I added (probably broken) is prefaced by #+allegro. Dirk's working code is prefaced by #+sbcl. Documentation on the allegro with-timeout macro is available here: http://www.franz.com/support/documentation/6.2/doc/pages/operators/mp/with-timeout.htm A complete copy of compatibility.lisp (the only modified file) is available here: http://plaza.ufl.edu/lavigne/compatibility.lisp Hopefully you can figure out my mistake so Erlisp can work on one more compiler. I would appreciate any ideas about what might be wrong. Eric -- plaza.ufl.edu/lavigne/ From lavigne.eric at gmail.com Mon Jul 11 20:35:32 2005 From: lavigne.eric at gmail.com (Eric Lavigne) Date: Mon, 11 Jul 2005 20:35:32 +0000 Subject: [erlisp-devel] mistake in with-timeout? In-Reply-To: References: <51e1bc3005071021546cc3010a@mail.gmail.com> Message-ID: <51e1bc30050711133545dafa0@mail.gmail.com> > > I suspect my mistake is in erlisp:with-timeout. Here is a copy of that macro: > > > > (defmacro with-timeout ((milliseconds &rest after-timeout) &body body) > > #+sbcl `(sb-ext:with-timeout (/ ,milliseconds 1000) > > (handler-case (progn , at body) > > (sb-ext:timeout () , at after-timeout))) > > #+allegro `(mp:with-timeout ((/ ,milliseconds 1000) , at after-timeout) > > , at body) > > #-(or sbcl allegro) (error > > "Timeouts are not yet implemented for your compiler.")) > > > > The part I added (probably broken) is prefaced by #+allegro. > > Dirk's working code is prefaced by #+sbcl. > > That macro looks good to me. I tried it on the allegro cl trial 6.2 > and it works as expected. It is got to be something else. > > Regards, > DiG Thanks, Dimitry. I assumed there must be something wrong with my with-timeout because 1) I didn't really understand it when I wrote it. 2) The SBCL version looked more complicated, so I might be missing something. But if with-timeout works correctly, then something must have actually timed out. In other words, I had a speed problem. Once I understood what to look for, the fix was easy. I already knew my code was slow (particularly event-wait) but had ignored that because I wanted to focus on making it work first. All tests pass now, so I'm ready to send a patch. Regards, Eric From digash at gmail.com Mon Jul 11 20:11:07 2005 From: digash at gmail.com (Dimitry Gashinsky) Date: Mon, 11 Jul 2005 16:11:07 -0400 Subject: [erlisp-devel] mistake in with-timeout? In-Reply-To: <51e1bc3005071021546cc3010a@mail.gmail.com> References: <51e1bc3005071021546cc3010a@mail.gmail.com> Message-ID: On 7/11/05, Eric Lavigne wrote: > I suspect my mistake is in erlisp:with-timeout. Here is a copy of that macro: > > (defmacro with-timeout ((milliseconds &rest after-timeout) &body body) > #+sbcl `(sb-ext:with-timeout (/ ,milliseconds 1000) > (handler-case (progn , at body) > (sb-ext:timeout () , at after-timeout))) > #+allegro `(mp:with-timeout ((/ ,milliseconds 1000) , at after-timeout) > , at body) > #-(or sbcl allegro) (error > "Timeouts are not yet implemented for your compiler.")) > > The part I added (probably broken) is prefaced by #+allegro. > Dirk's working code is prefaced by #+sbcl. That macro looks good to me. I tried it on the allegro cl trial 6.2 and it works as expected. It is got to be something else. Regards, DiG -- ((lambda (lambda) `(,lambda ',lambda)) '(lambda (lambda) `(,lambda ',lambda))) From dirk at dirkgerrits.com Mon Jul 11 19:43:55 2005 From: dirk at dirkgerrits.com (dirk at dirkgerrits.com) Date: Mon, 11 Jul 2005 21:43:55 +0200 Subject: [erlisp-devel] Description of pushed patches. Message-ID: <42d2cbfb.smakygs/fqdgRQPO%dirk@dirkgerrits.com> Expanded timeout clause syntax to support time-units. Dirk Gerrits **20050615235950] allegro-threads Eric Lavigne **20050711110548 Modified threading procedures in compatibility.lisp to support Allegro Common Lisp. Added structure erlisp-event to emulate SBCL's wait-queue. Added low-level mutex operations (lock-mutex and unlock-mutex) to support erlisp-event. From dirk at dirkgerrits.com Tue Jul 12 05:11:21 2005 From: dirk at dirkgerrits.com (Dirk Gerrits) Date: Tue, 12 Jul 2005 07:11:21 +0200 Subject: [erlisp-devel] Re: Description of pushed patches. In-Reply-To: <42d2cbfb.smakygs/fqdgRQPO%dirk@dirkgerrits.com> References: <42d2cbfb.smakygs/fqdgRQPO%dirk@dirkgerrits.com> Message-ID: <42D350F9.1070605@dirkgerrits.com> dirk at dirkgerrits.com wrote: > Expanded timeout clause syntax to support time-units. > Dirk Gerrits **20050615235950] Hmm, I'm guessing my script isn't completely fool proof. The above is an artifact caused by darcs's inventory format not really being line based. This patch has been in the repository for quite a while already. Kind regards, Dirk Gerrits From lavigne.eric at gmail.com Tue Jul 12 14:45:11 2005 From: lavigne.eric at gmail.com (Eric Lavigne) Date: Tue, 12 Jul 2005 14:45:11 +0000 Subject: [erlisp-devel] Re: Description of pushed patches. In-Reply-To: <42D350F9.1070605@dirkgerrits.com> References: <42d2cbfb.smakygs/fqdgRQPO%dirk@dirkgerrits.com> <42D350F9.1070605@dirkgerrits.com> Message-ID: <51e1bc3005071207452c7f2fcd@mail.gmail.com> > > Expanded timeout clause syntax to support time-units. > > Dirk Gerrits **20050615235950] > > Hmm, I'm guessing my script isn't completely fool proof. The above is > an artifact caused by darcs's inventory format not really being line > based. This patch has been in the repository for quite a while already. > > Kind regards, > > Dirk Gerrits > I don't think that patch was part of my working copy, though. The with-timeout that I worked on used only seconds. Some manual effort is probably required to merge those two patches. Eric From lavigne.eric at gmail.com Tue Jul 12 16:43:17 2005 From: lavigne.eric at gmail.com (Eric Lavigne) Date: Tue, 12 Jul 2005 16:43:17 +0000 Subject: [erlisp-devel] darcs repository outdated? In-Reply-To: <51e1bc3005071207452c7f2fcd@mail.gmail.com> References: <42d2cbfb.smakygs/fqdgRQPO%dirk@dirkgerrits.com> <42D350F9.1070605@dirkgerrits.com> <51e1bc3005071207452c7f2fcd@mail.gmail.com> Message-ID: <51e1bc300507120943601546dd@mail.gmail.com> Dear Dirk, I used this command to get a fresh copy of the erlisp repository, as described on the erlisp download page: darcs get http://www.dirkgerrits.com/darcs/erlisp The result looked a lot like what I downloaded a week ago - no indication of Allegro compatibility. I tried using "darcs changes" and it looks like the most recent patch was on June 15, moving the license to a different file. "darcs pull" indicated that there were no additional patches available. Am I looking at the wrong repository? Regards, Eric From dirk at dirkgerrits.com Tue Jul 12 17:18:40 2005 From: dirk at dirkgerrits.com (Dirk Gerrits) Date: Tue, 12 Jul 2005 19:18:40 +0200 Subject: [erlisp-devel] Re: Description of pushed patches. In-Reply-To: <51e1bc3005071207452c7f2fcd@mail.gmail.com> References: <42d2cbfb.smakygs/fqdgRQPO%dirk@dirkgerrits.com> <42D350F9.1070605@dirkgerrits.com> <51e1bc3005071207452c7f2fcd@mail.gmail.com> Message-ID: <42D3FB70.5010404@dirkgerrits.com> Eric Lavigne wrote: >>>Expanded timeout clause syntax to support time-units. >>>Dirk Gerrits **20050615235950] >> >>Hmm, I'm guessing my script isn't completely fool proof. The above is >>an artifact caused by darcs's inventory format not really being line >>based. This patch has been in the repository for quite a while already. > > I don't think that patch was part of my working copy, though. The > with-timeout that I worked on used only seconds. Some manual effort is > probably required to merge those two patches. WITH-TIMEOUT has always used milliseconds. The handling of (different) time units is all in the syntax of RECEIVE. Are you sure you didn't have that patch? Because I'm positive it has been in the dirkgerrits.com darcs repository for as long as that repository has been there... In any case, darcs on my laptop doesn't see any conflicts with your patch. Kind regards, Dirk Gerrits From dirk at dirkgerrits.com Tue Jul 12 17:23:36 2005 From: dirk at dirkgerrits.com (Dirk Gerrits) Date: Tue, 12 Jul 2005 19:23:36 +0200 Subject: [erlisp-devel] darcs repository outdated? In-Reply-To: <51e1bc300507120943601546dd@mail.gmail.com> References: <42d2cbfb.smakygs/fqdgRQPO%dirk@dirkgerrits.com> <42D350F9.1070605@dirkgerrits.com> <51e1bc3005071207452c7f2fcd@mail.gmail.com> <51e1bc300507120943601546dd@mail.gmail.com> Message-ID: <42D3FC98.9040500@dirkgerrits.com> Eric Lavigne wrote: > Dear Dirk, > > I used this command to get a fresh copy of the erlisp repository, as > described on the erlisp download page: > darcs get http://www.dirkgerrits.com/darcs/erlisp > > The result looked a lot like what I downloaded a week ago - no > indication of Allegro compatibility. I tried using "darcs changes" and > it looks like the most recent patch was on June 15, moving the license > to a different file. "darcs pull" indicated that there were no > additional patches available. Am I looking at the wrong repository? Hey Eric, are you still seeing this problem? I can't seem to reproduce it on my laptop. Fresh checkouts get all my patches and your 'allegro-threads' one. I can try a friend's computer tomorrow if you're still having this problem. Kind regards, Dirk Gerrits From lavigne.eric at gmail.com Wed Jul 13 03:16:49 2005 From: lavigne.eric at gmail.com (Eric Lavigne) Date: Wed, 13 Jul 2005 03:16:49 +0000 Subject: [erlisp-devel] darcs repository outdated? In-Reply-To: <42D3FC98.9040500@dirkgerrits.com> References: <42d2cbfb.smakygs/fqdgRQPO%dirk@dirkgerrits.com> <42D350F9.1070605@dirkgerrits.com> <51e1bc3005071207452c7f2fcd@mail.gmail.com> <51e1bc300507120943601546dd@mail.gmail.com> <42D3FC98.9040500@dirkgerrits.com> Message-ID: <51e1bc30050712201677f5df2f@mail.gmail.com> > > Dear Dirk, > > > > I used this command to get a fresh copy of the erlisp repository, as > > described on the erlisp download page: > > darcs get http://www.dirkgerrits.com/darcs/erlisp > > > > The result looked a lot like what I downloaded a week ago - no > > indication of Allegro compatibility. I tried using "darcs changes" and > > it looks like the most recent patch was on June 15, moving the license > > to a different file. "darcs pull" indicated that there were no > > additional patches available. Am I looking at the wrong repository? > > Hey Eric, > > are you still seeing this problem? I can't seem to reproduce it on my > laptop. Fresh checkouts get all my patches and your 'allegro-threads' > one. I can try a friend's computer tomorrow if you're still having this > problem. > > Kind regards, > > Dirk Gerrits > > Dirk, Problem solved. There seems to be a rather long delay between committing a patch and being able to download that patch. I just cut and pasted the same "darcs pull" command that I used before, in the same directory, and now it works. Eric From dirk at dirkgerrits.com Tue Jul 12 18:18:48 2005 From: dirk at dirkgerrits.com (Dirk Gerrits) Date: Tue, 12 Jul 2005 20:18:48 +0200 Subject: [erlisp-devel] darcs repository outdated? In-Reply-To: <51e1bc30050712201677f5df2f@mail.gmail.com> References: <42d2cbfb.smakygs/fqdgRQPO%dirk@dirkgerrits.com> <42D350F9.1070605@dirkgerrits.com> <51e1bc3005071207452c7f2fcd@mail.gmail.com> <51e1bc300507120943601546dd@mail.gmail.com> <42D3FC98.9040500@dirkgerrits.com> <51e1bc30050712201677f5df2f@mail.gmail.com> Message-ID: <42D40988.6030406@dirkgerrits.com> Eric Lavigne wrote: > There seems to be a rather long delay between committing a patch and > being able to download that patch. Well that really shouldn't be the case, but if you experience this problem again, let us know. Kind regards, Dirk Gerrits From lavigne.eric at gmail.com Wed Jul 13 03:22:04 2005 From: lavigne.eric at gmail.com (Eric Lavigne) Date: Wed, 13 Jul 2005 03:22:04 +0000 Subject: [erlisp-devel] darcs repository outdated? In-Reply-To: <42D40988.6030406@dirkgerrits.com> References: <42d2cbfb.smakygs/fqdgRQPO%dirk@dirkgerrits.com> <42D350F9.1070605@dirkgerrits.com> <51e1bc3005071207452c7f2fcd@mail.gmail.com> <51e1bc300507120943601546dd@mail.gmail.com> <42D3FC98.9040500@dirkgerrits.com> <51e1bc30050712201677f5df2f@mail.gmail.com> <42D40988.6030406@dirkgerrits.com> Message-ID: <51e1bc3005071220224c9c3282@mail.gmail.com> > > There seems to be a rather long delay between committing a patch and > > being able to download that patch. > > Well that really shouldn't be the case, but if you experience this > problem again, let us know. > > Kind regards, > > Dirk Gerrits > It seems strange to me also. I will let you know if I see this again. Eric From lavigne.eric at gmail.com Wed Jul 13 03:18:00 2005 From: lavigne.eric at gmail.com (Eric Lavigne) Date: Wed, 13 Jul 2005 03:18:00 +0000 Subject: [erlisp-devel] Re: Description of pushed patches. In-Reply-To: <42D3FB70.5010404@dirkgerrits.com> References: <42d2cbfb.smakygs/fqdgRQPO%dirk@dirkgerrits.com> <42D350F9.1070605@dirkgerrits.com> <51e1bc3005071207452c7f2fcd@mail.gmail.com> <42D3FB70.5010404@dirkgerrits.com> Message-ID: <51e1bc300507122018e06c783@mail.gmail.com> On 7/12/05, Dirk Gerrits wrote: > Eric Lavigne wrote: > >>>Expanded timeout clause syntax to support time-units. > >>>Dirk Gerrits **20050615235950] > >> > >>Hmm, I'm guessing my script isn't completely fool proof. The above is > >>an artifact caused by darcs's inventory format not really being line > >>based. This patch has been in the repository for quite a while already. > > > > I don't think that patch was part of my working copy, though. The > > with-timeout that I worked on used only seconds. Some manual effort is > > probably required to merge those two patches. > > WITH-TIMEOUT has always used milliseconds. The handling of (different) > time units is all in the syntax of RECEIVE. > > Are you sure you didn't have that patch? Because I'm positive it has > been in the dirkgerrits.com darcs repository for as long as that > repository has been there... In any case, darcs on my laptop doesn't > see any conflicts with your patch. > > Kind regards, > > Dirk Gerrits Dirk, Yes, I have that patch. I assumed that I didn't have it because with-timeout only used milliseconds. Eric -- plaza.ufl.edu/lavigne/ From dirk at dirkgerrits.com Sat Jul 23 21:15:01 2005 From: dirk at dirkgerrits.com (Dirk Gerrits) Date: Sat, 23 Jul 2005 23:15:01 +0200 Subject: [erlisp-devel] Description of pushed patches. Message-ID: <42e2b355.edchJrLxiEWXz9c4%dirk@dirkgerrits.com> cmucl-threads Eric Lavigne **20050723181545 Modified threading procedures in compatibility.lisp to support CMU Common Lisp. From lavigne.eric at gmail.com Sun Jul 24 14:02:07 2005 From: lavigne.eric at gmail.com (Eric Lavigne) Date: Sun, 24 Jul 2005 14:02:07 +0000 Subject: [erlisp-devel] Re: patch for cmucl compatibility In-Reply-To: <42E2B7B2.7000104@dirkgerrits.com> References: <51e1bc30050723205962f4b167@mail.gmail.com> <42E2B7B2.7000104@dirkgerrits.com> Message-ID: <51e1bc3005072407021cee7e8c@mail.gmail.com> > > This patch adds support for cmucl. All tests pass in cmucl. I have not > > retested under allegro or sbcl, but it should not be an issue since > > all changes are prefaced by #+cmu. > > All tests pass for me too, in both CMUCL 19a and SBCL 0.9.2. Your patch > looks good so I've pushed it. About 'CMUCL gates' though, I don't know. > Taking a quick look at code/multi-proc.lisp I'd say enable-/ > disable-process or process-add/revoke-run-reason might be related, but > I'm just not sure. I'd have to read it more carefully. If wait queues stored lists of waiting processes, then notify could involve adding run reasons to all waiting processes. Sounds complicated, but in terms of speed it's probably a lot better than letting those waiting threads remain active. > > Oh BTW, SBCL 0.9.3 will probably break Erlisp, as they've switched from > thread IDs to thread objects in the current CVS, as far as I've been > told. Should be a rather simple fix though, when the time comes. It doesn't sound too difficult. Where did you find information on future versions, though? > > So what's on your schedule now? Threads for more compilers? Process > linking? Distributed programming? Just let the mailing list know, and > I'll write up a short blog post. (That reminds me, I still need to > forward the soc-erlisp e-mails...) Threads for one more compiler (clisp), then distributed programming. What is process linking? Do you have any ideas about what the interface should look like for distributed programming? I've seen some confusing material on erlang websites which suggested that an erlang programmer could write a concurrent program without knowing whether it would be run on one computer (threads) or spread over several. Sounds nice but I never figured out how that worked. > > Oh BTW, could you check whether you can checkout the patch now? I still > have no idea what went wrong last time... > I tried pulling the latest patch. No problems. Eric -- http://plaza.ufl.edu/lavigne/ http://www.lispnyc.org/summerofcode.html From dirk at dirkgerrits.com Sun Jul 24 14:52:25 2005 From: dirk at dirkgerrits.com (Dirk Gerrits) Date: Sun, 24 Jul 2005 07:52:25 -0700 Subject: [erlisp-devel] Re: patch for cmucl compatibility In-Reply-To: <51e1bc3005072407021cee7e8c@mail.gmail.com> References: <51e1bc30050723205962f4b167@mail.gmail.com> <42E2B7B2.7000104@dirkgerrits.com> <51e1bc3005072407021cee7e8c@mail.gmail.com> Message-ID: <42E3AB29.4010504@dirkgerrits.com> Eric Lavigne wrote: >>Taking a quick look at code/multi-proc.lisp I'd say enable-/ >>disable-process or process-add/revoke-run-reason might be related, but >>I'm just not sure. I'd have to read it more carefully. > > If wait queues stored lists of waiting processes, then notify could > involve adding run reasons to all waiting processes. Sounds > complicated, but in terms of speed it's probably a lot better than > letting those waiting threads remain active. The way Erlisp uses wait queues, only one process will ever be waiting on it (the one doing the RECEIVE), though many processes can make it wake up (the ones doing SEND). Does that help? Again, I haven't studied CMUCL's threading mechanisms. >>Oh BTW, SBCL 0.9.3 will probably break Erlisp, as they've switched from >>thread IDs to thread objects in the current CVS, as far as I've been >>told. Should be a rather simple fix though, when the time comes. > > It doesn't sound too difficult. Where did you find information on > future versions, though? Mailing list: http://lists.sourceforge.net/lists/listinfo/sbcl-devel IRC: #lisp at irc.freenode.net >>So what's on your schedule now? Threads for more compilers? Process >>linking? Distributed programming? Just let the mailing list know, and >>I'll write up a short blog post. (That reminds me, I still need to >>forward the soc-erlisp e-mails...) > > > Threads for one more compiler (clisp), then distributed programming. Fine with me, though you'd then have to do process linking for both parallel /and/ distributed programming afterwards. You may want to work on 'parallel process linking' before working on the distributed programming stuff. > What is process linking? Process linking is one of Erlang's most important features. Far? said he had discussed this with you. This is from the soc-erlisp e-mail http://parenpower.com/mailman/private/soc-erlisp/2005-July/000015.html: (BTW, why are these archives password protected?) Dirk Gerrits wrote: >>As for non-distributed Erlisp, "creating a new thread in the existing >>Lisp image" is already supported on SBCL 0.9.1, and shouldn't be too >>hard to port to other Lisps and upcoming SBCL releases. There is >>still at least one very big missing feature though, and that's linking >>of processes. (See http://dirkgerrits.com/programming/erlisp/roadmap/ >>under "Error handling".) Far? wrote: > Linking of processes is a very important feature of the Erlang > programming model. The difficulty being that of killing a thread when > it's in the middle of some observable side-effect. This require proper > locking and/or proper recovery mechanisms (roll back or roll forward). > I've discussed the issue with Eric. Basically, we can either refrain > from killing outside of safe-points (notably invoked at Erlisp > function calls), or we can require processes to hold a lock while they > are doing any kind of non-thread-local side-effect. > Do you have any ideas about what the interface should look like for > distributed programming? I've seen some confusing material on erlang > websites which suggested that an erlang programmer could write a > concurrent program without knowing whether it would be run on one > computer (threads) or spread over several. Sounds nice but I never > figured out how that worked. I suggest getting a hold of (borrowing / buying second hand) the book Concurrent Programming in Erlang, Second Edition. Anyway, the way that works is that Erlang processes live on /nodes/. If you send a message to a process on the same node, what will happen is probably similar to what Erlisp does now. If you send a message to a process on a different node, the message will go over the network. The syntax is identical, so that's probably what those websites mean. And you're supposed to assume the semantics are identical too, that is, message passing can fail in either case. In practice I'd think that message passing /actually/ failing is very, very rare unless the receiver is on a different node though. So you still need to design for distributed programming, but when you do, your code can run both locally and distributedly. If you look at src/messaging.lisp, you'll see that the mechanisms for this are already there. SEND calls SEND-USING-NODES to do the work, which has the following specializations: * from a remote node Gives the error "Can only send messages from a local process." * to a remote node Gives the error "Distribution is not implemented yet." * to a threaded process on a local node Does the whole mutex/mailbox/wait-queue thing. You'd want to replace the second one. ;) - Dirk From fahree at gmail.com Sun Jul 24 15:19:51 2005 From: fahree at gmail.com (=?ISO-8859-1?Q?Far=E9?=) Date: Sun, 24 Jul 2005 11:19:51 -0400 Subject: [erlisp-devel] Erlisp and uptick In-Reply-To: <42e2b355.edchJrLxiEWXz9c4%dirk@dirkgerrits.com> References: <42e2b355.edchJrLxiEWXz9c4%dirk@dirkgerrits.com> Message-ID: <653bea16050724081969b1dc54@mail.gmail.com> Relevant discussion on the uptick blog: http://wagerlabs.com/uptick/2005/07/very-frustrating-saturday.html The author has rival concerns about speed and memory-efficiency of threads. My take: The idea is that there should be a standard API, and then multiple ways of implementing it. Providing a light-weight implementation in terms of green threads (Eric Lavigne just ported erlisp to cmucl), or of a continuation passing transform (Screamer style), are definitely implementation strategies for Erlisp. And if all goes well, we can mix and match implementation strategies within a same Erlisp universe: gimme slower memory-efficient threads for this, faster memory-wasting threads for that, migratable (Tube-like) threads for that other thing, etc. [ Fran?ois-Ren? ?VB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] Proprietary software is dead software. Using it is a form of necrophilia. -- Far? From dirk at dirkgerrits.com Sun Jul 24 16:05:50 2005 From: dirk at dirkgerrits.com (Dirk Gerrits) Date: Sun, 24 Jul 2005 09:05:50 -0700 Subject: [erlisp-devel] Erlisp and uptick In-Reply-To: <653bea16050724081969b1dc54@mail.gmail.com> References: <42e2b355.edchJrLxiEWXz9c4%dirk@dirkgerrits.com> <653bea16050724081969b1dc54@mail.gmail.com> Message-ID: <42E3BC5E.2040801@dirkgerrits.com> Far? wrote: > Relevant discussion on the uptick blog: > http://wagerlabs.com/uptick/2005/07/very-frustrating-saturday.html > The author has rival concerns about speed and memory-efficiency of threads. > > My take: > > The idea is that there should be a standard API, and then multiple > ways of implementing it. Providing a light-weight implementation in > terms of green threads (Eric Lavigne just ported erlisp to cmucl), or > of a continuation passing transform (Screamer style), are definitely > implementation strategies for Erlisp. > And if all goes well, we can mix and match implementation strategies > within a same Erlisp universe: gimme slower memory-efficient threads > for this, faster memory-wasting threads for that, migratable > (Tube-like) threads for that other thing, etc. That has been my take all along! And the code to do this is already there. (That is, there is a PROCESS class with THREADED-PROCESS as a subclass and more subclasses planned.) Guys like Joel make me wonder if people actually read the Erlisp roadmap [0]. Is it too fluffy? Not clear enough? Too long? Not structured properly? I'm willing to rewrite it if that can prevent things like these... - Dirk [0] http://dirkgerrits.com/programming/erlisp/roadmap/ From lavigne.eric at gmail.com Sun Jul 24 17:38:38 2005 From: lavigne.eric at gmail.com (Eric Lavigne) Date: Sun, 24 Jul 2005 17:38:38 +0000 Subject: [erlisp-devel] Erlisp and uptick In-Reply-To: <42E3BC5E.2040801@dirkgerrits.com> References: <42e2b355.edchJrLxiEWXz9c4%dirk@dirkgerrits.com> <653bea16050724081969b1dc54@mail.gmail.com> <42E3BC5E.2040801@dirkgerrits.com> Message-ID: <51e1bc3005072410385ee9375d@mail.gmail.com> > > That has been my take all along! And the code to do this is already > there. (That is, there is a PROCESS class with THREADED-PROCESS as a > subclass and more subclasses planned.) > > Guys like Joel make me wonder if people actually read the Erlisp roadmap > [0]. Is it too fluffy? Not clear enough? Too long? Not structured > properly? I'm willing to rewrite it if that can prevent things like > these... > > - Dirk > > [0] http://dirkgerrits.com/programming/erlisp/roadmap/ Your interest in speed is very clear (and very emphasized) in your roadmap, but only at the end. If I read the first two pages, I would get the impression that you didn't even notice that Erlang has fast threading (and that you had already implemented processes as threads). Then at the end I would see a long discussion of how to make Erlisp fast, including such extreme measures as hacking the CLisp backend or submitting CLRFIs. If all I cared about was speed, I might be inclined to stop reading after the first or second page and not see the third. That is probably what happened to Joel. Possible ways to change this: Add a goals section at the top, maybe only 10 lines long but listing what you expect from ErLisp when it is completed. Add speed related items to the list of Erlang features, such as green threads or send-and-pray networking. This is not the first time that Joel has started developing something in Lisp, expressed his desire to continue working in Lisp, but left to Erlang for its networking/threading features. If he could be persuaded to make some small contribution to ErLisp, I suspect he would remain a regular contributor for years to come. The obvious candidate would be light-weight threading, but that sounds rather difficult. Do we have any quick and easy tasks that might interest him? Eric -- http://plaza.ufl.edu/lavigne/ http://www.lispnyc.org/summerofcode.html From lavigne.eric at gmail.com Sun Jul 24 21:28:41 2005 From: lavigne.eric at gmail.com (Eric Lavigne) Date: Sun, 24 Jul 2005 21:28:41 +0000 Subject: [erlisp-devel] Re: patch for cmucl compatibility In-Reply-To: <42E3AB29.4010504@dirkgerrits.com> References: <51e1bc30050723205962f4b167@mail.gmail.com> <42E2B7B2.7000104@dirkgerrits.com> <51e1bc3005072407021cee7e8c@mail.gmail.com> <42E3AB29.4010504@dirkgerrits.com> Message-ID: <51e1bc3005072414287c7a4fe7@mail.gmail.com> > >>Taking a quick look at code/multi-proc.lisp I'd say enable-/ > >>disable-process or process-add/revoke-run-reason might be related, but > >>I'm just not sure. I'd have to read it more carefully. > > > > If wait queues stored lists of waiting processes, then notify could > > involve adding run reasons to all waiting processes. Sounds > > complicated, but in terms of speed it's probably a lot better than > > letting those waiting threads remain active. > > The way Erlisp uses wait queues, only one process will ever be waiting > on it (the one doing the RECEIVE), though many processes can make it > wake up (the ones doing SEND). Does that help? Again, I haven't > studied CMUCL's threading mechanisms. Yes, that should make it easier. Unfortunately, I haven't found any good documentation on CMUCL's threading mechanisms, and I don't understand them very well myself. Everything I know about CMUCL threading comes either from reading the compiler source code (which is well commented) or from comparison to Allegro (which CMUCL threading is loosely based on). > > >>Oh BTW, SBCL 0.9.3 will probably break Erlisp, as they've switched from > >>thread IDs to thread objects in the current CVS, as far as I've been > >>told. Should be a rather simple fix though, when the time comes. > > > > It doesn't sound too difficult. Where did you find information on > > future versions, though? > > Mailing list: > http://lists.sourceforge.net/lists/listinfo/sbcl-devel > > IRC: > #lisp at irc.freenode.net > > >>So what's on your schedule now? Threads for more compilers? Process > >>linking? Distributed programming? Just let the mailing list know, and > >>I'll write up a short blog post. (That reminds me, I still need to > >>forward the soc-erlisp e-mails...) > > > > > > Threads for one more compiler (clisp), then distributed programming. > > Fine with me, though you'd then have to do process linking for both > parallel /and/ distributed programming afterwards. You may want to work > on 'parallel process linking' before working on the distributed > programming stuff. > > > What is process linking? > > Process linking is one of Erlang's most important features. Far? said > he had discussed this with you. This is from the soc-erlisp e-mail > http://parenpower.com/mailman/private/soc-erlisp/2005-July/000015.html: > (BTW, why are these archives password protected?) Yes, Far? explained this to me, in addition to the related topic of error-handling. I forgot the term, though. Thanks for the reminder. > > Dirk Gerrits wrote: > >>As for non-distributed Erlisp, "creating a new thread in the existing > >>Lisp image" is already supported on SBCL 0.9.1, and shouldn't be too > >>hard to port to other Lisps and upcoming SBCL releases. There is > >>still at least one very big missing feature though, and that's linking > >>of processes. (See http://dirkgerrits.com/programming/erlisp/roadmap/ > >>under "Error handling".) > > Far? wrote: > > Linking of processes is a very important feature of the Erlang > > programming model. The difficulty being that of killing a thread when > > it's in the middle of some observable side-effect. This require proper > > locking and/or proper recovery mechanisms (roll back or roll forward). > > I've discussed the issue with Eric. Basically, we can either refrain > > from killing outside of safe-points (notably invoked at Erlisp > > function calls), or we can require processes to hold a lock while they > > are doing any kind of non-thread-local side-effect. > > > Do you have any ideas about what the interface should look like for > > distributed programming? I've seen some confusing material on erlang > > websites which suggested that an erlang programmer could write a > > concurrent program without knowing whether it would be run on one > > computer (threads) or spread over several. Sounds nice but I never > > figured out how that worked. > > I suggest getting a hold of (borrowing / buying second hand) the book > Concurrent Programming in Erlang, Second Edition. I requested this book on inter-library loan today. It will probably take a week or two to arrive. Until then, I should be able to work on lower level issues without it. > > Anyway, the way that works is that Erlang processes live on /nodes/. If > you send a message to a process on the same node, what will happen is > probably similar to what Erlisp does now. If you send a message to a > process on a different node, the message will go over the network. The > syntax is identical, so that's probably what those websites mean. And > you're supposed to assume the semantics are identical too, that is, > message passing can fail in either case. In practice I'd think that > message passing /actually/ failing is very, very rare unless the > receiver is on a different node though. So you still need to design for > distributed programming, but when you do, your code can run both locally > and distributedly. So is each Lisp image automatically considered a node? I need to look more closely at how message passing is implemented. > > If you look at src/messaging.lisp, you'll see that the mechanisms for > this are already there. SEND calls SEND-USING-NODES to do the work, > which has the following specializations: > > * from a remote node > Gives the error "Can only send messages from a local process." > > * to a remote node > Gives the error "Distribution is not implemented yet." > > * to a threaded process on a local node > Does the whole mutex/mailbox/wait-queue thing. > > You'd want to replace the second one. ;) > > - Dirk > Sounds good. It's nice to have a flexible framework to work in. I need to do some more exploring to figure out what everything does, though. Eric From dirk at dirkgerrits.com Sun Jul 24 19:14:12 2005 From: dirk at dirkgerrits.com (Dirk Gerrits) Date: Sun, 24 Jul 2005 21:14:12 +0200 Subject: [erlisp-devel] Re: patch for cmucl compatibility In-Reply-To: <51e1bc3005072414287c7a4fe7@mail.gmail.com> References: <51e1bc30050723205962f4b167@mail.gmail.com> <42E2B7B2.7000104@dirkgerrits.com> <51e1bc3005072407021cee7e8c@mail.gmail.com> <42E3AB29.4010504@dirkgerrits.com> <51e1bc3005072414287c7a4fe7@mail.gmail.com> Message-ID: <42E3E884.8000400@dirkgerrits.com> Eric Lavigne wrote: >>The way Erlisp uses wait queues, only one process will ever be waiting >>on it (the one doing the RECEIVE), though many processes can make it >>wake up (the ones doing SEND). Does that help? Again, I haven't >>studied CMUCL's threading mechanisms. > > > Yes, that should make it easier. Unfortunately, I haven't found any > good documentation on CMUCL's threading mechanisms, and I don't > understand them very well myself. Everything I know about CMUCL > threading comes either from reading the compiler source code (which is > well commented) or from comparison to Allegro (which CMUCL threading > is loosely based on). Okay, hopefully we can get something non-polling for CMUCL like we have now for SBCL. Perhaps the mailing lists can be of some help? (http://www.cons.org/cmucl/support.html) >>Process linking is one of Erlang's most important features. > > Yes, Far? explained this to me, in addition to the related topic of > error-handling. I forgot the term, though. Thanks for the reminder. Okay, and you still want to proceed with adding distributed programming first before you start tackling this? Fine with me either way, I'm just curious. >>I suggest getting a hold of (borrowing / buying second hand) the book >>Concurrent Programming in Erlang, Second Edition. > > > I requested this book on inter-library loan today. It will probably > take a week or two to arrive. Until then, I should be able to work on > lower level issues without it. Okay great. >>Anyway, the way that works is that Erlang processes live on /nodes/. If >>you send a message to a process on the same node, what will happen is >>probably similar to what Erlisp does now. If you send a message to a >>process on a different node, the message will go over the network. The >>syntax is identical, so that's probably what those websites mean. And >>you're supposed to assume the semantics are identical too, that is, >>message passing can fail in either case. In practice I'd think that >>message passing /actually/ failing is very, very rare unless the >>receiver is on a different node though. So you still need to design for >>distributed programming, but when you do, your code can run both locally >>and distributedly. > > > So is each Lisp image automatically considered a node? I need to look > more closely at how message passing is implemented. Good question. I would guess so, as they don't share a Lisp memory space. And I think this is true for multiple Erlang instances running on one machine as well. >>If you look at src/messaging.lisp, you'll see that the mechanisms for >>this are already there. SEND calls SEND-USING-NODES to do the work[...] > > Sounds good. It's nice to have a flexible framework to work in. I need > to do some more exploring to figure out what everything does, though. Yeah generic functions are great tools for stuff like this. But the framework is experimental anyway, so if you see the need to improve it feel free to bring that up. - Dirk From dirk at dirkgerrits.com Sun Jul 24 19:34:29 2005 From: dirk at dirkgerrits.com (Dirk Gerrits) Date: Sun, 24 Jul 2005 21:34:29 +0200 Subject: [erlisp-devel] Erlisp and uptick In-Reply-To: <51e1bc3005072410385ee9375d@mail.gmail.com> References: <42e2b355.edchJrLxiEWXz9c4%dirk@dirkgerrits.com> <653bea16050724081969b1dc54@mail.gmail.com> <42E3BC5E.2040801@dirkgerrits.com> <51e1bc3005072410385ee9375d@mail.gmail.com> Message-ID: <42E3ED45.3010104@dirkgerrits.com> Eric Lavigne wrote: >> [...] make me wonder if people actually read the Erlisp >> roadmap. Is it too fluffy? Not clear enough? Too long? Not >> structured properly? I'm willing to rewrite it if that can prevent >> things like these... > > > Your interest in speed is very clear (and very emphasized) in your > roadmap, but only at the end. If I read the first two pages, I would > get the impression that you didn't even notice that Erlang has fast > threading (and that you had already implemented processes as threads). > Then at the end I would see a long discussion of how to make Erlisp > fast, including such extreme measures as hacking the CLisp backend or > submitting CLRFIs. If all I cared about was speed, I might be inclined > to stop reading after the first or second page and not see the third. Very good point... > Possible ways to change this: > Add a goals section at the top, maybe only 10 lines long but listing > what you expect from ErLisp when it is completed. Okay so a juicier intro. Good idea. And I'm thinking of folding the overview and roadmap into one page. > Add speed related items to the list of Erlang features, such as green > threads or send-and-pray networking. Well this is already mentioned under 'Parallel programming'. I suppose I could split them off into separate features, but I'll have to locate my marketing hat. ;) > This is not the first time that Joel has started developing something > in Lisp, expressed his desire to continue working in Lisp, but left to > Erlang for its networking/threading features. If he could be persuaded > to make some small contribution to ErLisp, I suspect he would remain a > regular contributor for years to come. The obvious candidate would be > light-weight threading, but that sounds rather difficult. Do we have > any quick and easy tasks that might interest him? I do not think there /are/ any quick and easy tasks, but perhaps he can join the technical discussions on this mailing list. That would be most welcome as I am myself just an Erlang newbie. - Dirk From joelr1 at gmail.com Mon Jul 25 05:36:27 2005 From: joelr1 at gmail.com (Joel Reymont) Date: Mon, 25 Jul 2005 07:36:27 +0200 Subject: [erlisp-devel] Erlisp and uptick In-Reply-To: <42E3ED45.3010104@dirkgerrits.com> References: <42e2b355.edchJrLxiEWXz9c4%dirk@dirkgerrits.com> <653bea16050724081969b1dc54@mail.gmail.com> <42E3BC5E.2040801@dirkgerrits.com> <51e1bc3005072410385ee9375d@mail.gmail.com> <42E3ED45.3010104@dirkgerrits.com> Message-ID: > I do not think there /are/ any quick and easy tasks, but perhaps he > can join the technical discussions on this mailing list. That > would be most welcome as I am myself just an Erlang newbie. > "He" has already joined :-). I think I'll lurk around for a while and contribute to discussions. I'm in no position to code Erlisp at the moment. Feel free to bounce questions off of me, though. Joel -- http://wagerlabs.com/uptick From lavigne.eric at gmail.com Mon Jul 25 14:01:49 2005 From: lavigne.eric at gmail.com (Eric Lavigne) Date: Mon, 25 Jul 2005 14:01:49 +0000 Subject: [erlisp-devel] Re: patch for cmucl compatibility In-Reply-To: <42E3E884.8000400@dirkgerrits.com> References: <51e1bc30050723205962f4b167@mail.gmail.com> <42E2B7B2.7000104@dirkgerrits.com> <51e1bc3005072407021cee7e8c@mail.gmail.com> <42E3AB29.4010504@dirkgerrits.com> <51e1bc3005072414287c7a4fe7@mail.gmail.com> <42E3E884.8000400@dirkgerrits.com> Message-ID: <51e1bc300507250701214c8937@mail.gmail.com> >>>Process linking is one of Erlang's most important features. >> >> Yes, Far? explained this to me, in addition to the related topic of >> error-handling. I forgot the term, though. Thanks for the reminder. > >Okay, and you still want to proceed with adding distributed programming >first before you start tackling this? Fine with me either way, I'm just >curious. Yes, I just feel more comfortable working on the low-level stuff first. Eric From metawilm at gmail.com Mon Jul 25 13:51:17 2005 From: metawilm at gmail.com (Willem Broekema) Date: Mon, 25 Jul 2005 15:51:17 +0200 Subject: [erlisp-devel] Re: patch for cmucl compatibility In-Reply-To: <42E3E884.8000400@dirkgerrits.com> References: <51e1bc30050723205962f4b167@mail.gmail.com> <42E2B7B2.7000104@dirkgerrits.com> <51e1bc3005072407021cee7e8c@mail.gmail.com> <42E3AB29.4010504@dirkgerrits.com> <51e1bc3005072414287c7a4fe7@mail.gmail.com> <42E3E884.8000400@dirkgerrits.com> Message-ID: Dirk: > I suggest getting a hold of (borrowing / buying second hand) the book > Concurrent Programming in Erlang, Second Edition. Fortunately the first part of the book, which introduces the language concepts, is available as pdf at . The second part, consisting of example applications, is not. - Willem From lavigne.eric at gmail.com Mon Jul 25 14:24:48 2005 From: lavigne.eric at gmail.com (Eric Lavigne) Date: Mon, 25 Jul 2005 14:24:48 +0000 Subject: [erlisp-devel] Re: patch for cmucl compatibility In-Reply-To: References: <51e1bc30050723205962f4b167@mail.gmail.com> <42E2B7B2.7000104@dirkgerrits.com> <51e1bc3005072407021cee7e8c@mail.gmail.com> <42E3AB29.4010504@dirkgerrits.com> <51e1bc3005072414287c7a4fe7@mail.gmail.com> <42E3E884.8000400@dirkgerrits.com> Message-ID: <51e1bc300507250724736fb33e@mail.gmail.com> > > I suggest getting a hold of (borrowing / buying second hand) the book > > Concurrent Programming in Erlang, Second Edition. > > Fortunately the first part of the book, which introduces the language > concepts, is > available as pdf at . The second part, > consisting of example applications, is not. > > > - Willem > Excellent. I still prefer a hard copy so I can get away from the screen on occasion, but this will save me some weeks of waiting. Thanks, Eric -- http://plaza.ufl.edu/lavigne/ http://www.lispnyc.org/summerofcode.html From lavigne.eric at gmail.com Mon Jul 25 16:05:54 2005 From: lavigne.eric at gmail.com (Eric Lavigne) Date: Mon, 25 Jul 2005 16:05:54 +0000 Subject: [erlisp-devel] Erlisp and uptick In-Reply-To: References: <42e2b355.edchJrLxiEWXz9c4%dirk@dirkgerrits.com> <653bea16050724081969b1dc54@mail.gmail.com> <42E3BC5E.2040801@dirkgerrits.com> <51e1bc3005072410385ee9375d@mail.gmail.com> <42E3ED45.3010104@dirkgerrits.com> Message-ID: <51e1bc30050725090591fb3c1@mail.gmail.com> > "He" has already joined :-). I think I'll lurk around for a while and > contribute to discussions. I'm in no position to code Erlisp at the > moment. Feel free to bounce questions off of me, though. > > Joel > > -- > http://wagerlabs.com/uptick Welcome to the team, Joel. I'm also keeping an eye on your uptick project. Looks interesting. > > I do not think there /are/ any quick and easy tasks, I think this is an issue worth looking into. New developers, especially if they are new to programming in general, need easy tasks to cut their teeth on. It would be nice if ErLisp could fill that need. Each of my patches so far has some little problem with it. Allegro should be using gates rather than wait functions. That will require some reading and a few lines of code. CMUCL should probably use run-reasons instead of polling. That will require some research and a few lines of code. The bad news is that I'm a messy coder. :-( The good news is that there is no shortage of little projects laying around, and more are produced with each patch. :-) What do you think of creating a contributor's page? It could have links to the download page and the discussion page (both of which are fine the way they are). It would also include a list of contribution ideas of various difficulties and suggestions for getting started on each (such as checking the CMUCL mailing list or CMUCL's multi-proc.lisp for the CMUCL run-reasons task). Eric From dirk at dirkgerrits.com Mon Jul 25 16:01:15 2005 From: dirk at dirkgerrits.com (Dirk Gerrits) Date: Mon, 25 Jul 2005 09:01:15 -0700 Subject: [erlisp-devel] Re: patch for cmucl compatibility In-Reply-To: References: <51e1bc30050723205962f4b167@mail.gmail.com> <42E2B7B2.7000104@dirkgerrits.com> <51e1bc3005072407021cee7e8c@mail.gmail.com> <42E3AB29.4010504@dirkgerrits.com> <51e1bc3005072414287c7a4fe7@mail.gmail.com> <42E3E884.8000400@dirkgerrits.com> Message-ID: <42E50CCB.4060307@dirkgerrits.com> Willem Broekema wrote: > Dirk: > >> I suggest getting a hold of (borrowing / buying second hand) the >> book Concurrent Programming in Erlang, Second Edition. > > > Fortunately the first part of the book, which introduces the language > concepts, is available as pdf at . The > second part, consisting of example applications, is not. Oh cool. Good to know. *updates URL at http://dirkgerrits.com/programming/erlisp/references/*. - Dirk From dirk at dirkgerrits.com Tue Jul 26 16:06:15 2005 From: dirk at dirkgerrits.com (Dirk Gerrits) Date: Tue, 26 Jul 2005 09:06:15 -0700 Subject: [erlisp-devel] Erlisp and uptick In-Reply-To: <51e1bc30050725090591fb3c1@mail.gmail.com> References: <42e2b355.edchJrLxiEWXz9c4%dirk@dirkgerrits.com> <653bea16050724081969b1dc54@mail.gmail.com> <42E3BC5E.2040801@dirkgerrits.com> <51e1bc3005072410385ee9375d@mail.gmail.com> <42E3ED45.3010104@dirkgerrits.com> <51e1bc30050725090591fb3c1@mail.gmail.com> Message-ID: <42E65F77.4060701@dirkgerrits.com> Eric Lavigne wrote: >>"He" has already joined :-). I think I'll lurk around for a while and >>contribute to discussions. I'm in no position to code Erlisp at the >>moment. Feel free to bounce questions off of me, though. > > Welcome to the team, Joel. Yes welcome aboard. I'm sorry about my somewhat snide comment on your blog. My saturday just gets frustrated when people misrepresent my views. ;) But I have only myself to blame. As Eric has pointed out to me there are some things I can do to improve the Erlisp roadmap. >>>I do not think there /are/ any quick and easy tasks, > > > I think this is an issue worth looking into. New developers, > especially if they are new to programming in general, need easy tasks > to cut their teeth on. It would be nice if ErLisp could fill that > need. > > Each of my patches so far has some little problem with it. Allegro > should be using gates rather than wait functions. That will require > some reading and a few lines of code. CMUCL should probably use > run-reasons instead of polling. That will require some research and a > few lines of code. The bad news is that I'm a messy coder. :-( The > good news is that there is no shortage of little projects laying > around, and more are produced with each patch. :-) Okay, I had a different idea about quick and easy tasks. Personally, I don't consider anything related to Lisp concurrency easy, which is exactly why I've started this project. ;) But there are definately tasks that would result in just small amounts of code being added/changed. > What do you think of creating a contributor's page? It could have > links to the download page and the discussion page (both of which are > fine the way they are). It would also include a list of contribution > ideas of various difficulties and suggestions for getting started on > each (such as checking the CMUCL mailing list or CMUCL's > multi-proc.lisp for the CMUCL run-reasons task). That's an excellent idea! I'll try to work that in as soon as possible. I'll also want to keep it as up to date as humanly possible, but take shouldn't take more than a few minutes a day. - Dirk From lavigne.eric at gmail.com Wed Jul 27 00:11:15 2005 From: lavigne.eric at gmail.com (Eric Lavigne) Date: Wed, 27 Jul 2005 00:11:15 +0000 Subject: [erlisp-devel] Re: SoC status update please In-Reply-To: <653bea160507261631585297cf@mail.gmail.com> References: <51e1bc3005072609551587ab3d@mail.gmail.com> <653bea160507261631585297cf@mail.gmail.com> Message-ID: <51e1bc30050726171176862335@mail.gmail.com> > > CMUCL integration is complete. > Great. Do you have process kill working in CMUCL, SBCL, Allegro, in a > way that is safe wrt locks and that propagates along the process tree? > That's a major feature required of an erlang implementation. I have not done any work on process killing. As far as I know, it is not a part of ErLisp right now. This will definitely require a kill-thread function in compatibility.lisp. Beyond that I'm not sure how to procede. Manually calling kill-thread? Automatically calling it following a time-out? Also, I would like to give each process the responsibility for killing its child processes, but if it's being killed because it is no longer responsive then that won't work... I would like to see some discussion on this topic in erlisp-devel because I have no experience with such situations. > > This leads me to wonder how they do a reliable detection of a remote > node being dead, as opposed to the communication channel being down -- > and how they cope with a mistake between the two. Surely the point is > tackled somewhere in some Erlang documentation... Best I can think of is for the node itself to be represented as a process whose only job is communication with the outside world. Since the user won't control this process directly, it should be possible to make it fairly durable so that we can assume it is alive. *crosses fingers* > > > Next up is CLisp. > I don't think clisp has a complete threading implementation yet. Does > it? If it doesn't, then it might be time to begin a distributed > implementation -- and to fork clisp processes as a way to build new > threads. It looks like CLisp threading is very much a work in progress. I didn't research this well enough in advance. Forking CLisp processes is a nice idea, but we might be better off just letting it go until CLisp has better thread support. The original reason for including CLisp was so that Windows users would have access to ErLisp. Allegro support solves that problem well enough for now. So what next? Dirk, is ErLisp already have some sort of slot waiting for process-linking code? Or will we be designing something from scratch? I'm thinking that my next step is to read Far?'s thesis again. A robust distributed process management system does not sound like an easy task. Eric -- http://plaza.ufl.edu/lavigne/ http://www.lispnyc.org/summerofcode.html From joelr1 at gmail.com Wed Jul 27 00:27:33 2005 From: joelr1 at gmail.com (Joel Reymont) Date: Wed, 27 Jul 2005 02:27:33 +0200 Subject: [erlisp-devel] Re: SoC status update please In-Reply-To: <51e1bc30050726171176862335@mail.gmail.com> References: <51e1bc3005072609551587ab3d@mail.gmail.com> <653bea160507261631585297cf@mail.gmail.com> <51e1bc30050726171176862335@mail.gmail.com> Message-ID: I think within Erlang nodes use a keep alive message to make sure they have not been split off. Quite often, though, you cannot detect if the foreign process exists. If it's a local process id and you send it a message and it's dead then you will get a no_proc exception, not so with a foreign process. It does seem to me that the easiest way to verify if a node is alive is to send a keep-alive message to a housekeeping process on that node and consider it split off if a reply is not received after a timeout. I'm not sure if it matters if a node is dead or just split off. On Jul 27, 2005, at 2:11 AM, Eric Lavigne wrote: >> This leads me to wonder how they do a reliable detection of a remote >> node being dead, as opposed to the communication channel being >> down -- >> and how they cope with a mistake between the two. Surely the point is >> tackled somewhere in some Erlang documentation... >> >> > > Best I can think of is for the node itself to be represented as a > process whose only job is communication with the outside world. Since > the user won't control this process directly, it should be possible to > make it fairly durable so that we can assume it is alive. *crosses > fingers* > -- http://wagerlabs.com/uptick From dirk at dirkgerrits.com Wed Jul 27 02:45:49 2005 From: dirk at dirkgerrits.com (Dirk Gerrits) Date: Tue, 26 Jul 2005 19:45:49 -0700 Subject: [erlisp-devel] Re: SoC status update please In-Reply-To: <51e1bc30050726171176862335@mail.gmail.com> References: <51e1bc3005072609551587ab3d@mail.gmail.com> <653bea160507261631585297cf@mail.gmail.com> <51e1bc30050726171176862335@mail.gmail.com> Message-ID: <42E6F55D.7050401@dirkgerrits.com> Eric Lavigne wrote: > The original reason for including CLisp was so that Windows users > would have access to ErLisp. Allegro support solves that problem well > enough for now. Well the AllegroCL (and CMUCL) support at the moment is rather suboptimal (ie polling). Of course using threads in the first place is suboptimal, but we can at least make our thread support as good as possible while other methods are developed. ;) > So what next? Dirk, is ErLisp already have some sort of slot waiting > for process-linking code? Or will we be designing something from > scratch? The latter. I was just starting to think about process linking when the storm of schoolwork kicked in. > I'm thinking that my next step is to read Far?'s thesis again. A > robust distributed process management system does not sound like an > easy task. No, no it does not. ;) - Dirk P.S. it's Erlisp with a lowercase "L". The LispNYC wiki lists "ErLisp" only because that's a WikiWord and "Erlisp" isn't. From dirk at dirkgerrits.com Wed Jul 27 02:52:44 2005 From: dirk at dirkgerrits.com (Dirk Gerrits) Date: Tue, 26 Jul 2005 19:52:44 -0700 Subject: [erlisp-devel] Re: SoC status update please In-Reply-To: References: <51e1bc3005072609551587ab3d@mail.gmail.com> <653bea160507261631585297cf@mail.gmail.com> <51e1bc30050726171176862335@mail.gmail.com> Message-ID: <42E6F6FC.4050507@dirkgerrits.com> Joel Reymont wrote: > I think within Erlang nodes use a keep alive message to make sure they > have not been split off. Quite often, though, you cannot detect if the > foreign process exists. If it's a local process id and you send it a > message and it's dead then you will get a no_proc exception, not so > with a foreign process. > > It does seem to me that the easiest way to verify if a node is alive is > to send a keep-alive message to a housekeeping process on that node and > consider it split off if a reply is not received after a timeout. Sounds perfectly reasonable to me. > I'm not sure if it matters if a node is dead or just split off. Erlang doesn't make the distinction, does it? My thinking is that if it's good enough for Erlang, it is /at least/ good enough for Erlisp stage 1. We can always reconsider later on (even though it might be more painful then ;)). - Dirk From fahree at gmail.com Wed Jul 27 03:38:59 2005 From: fahree at gmail.com (=?ISO-8859-1?Q?Far=E9?=) Date: Tue, 26 Jul 2005 23:38:59 -0400 Subject: [erlisp-devel] Re: SoC status update please In-Reply-To: <42E6F6FC.4050507@dirkgerrits.com> References: <51e1bc3005072609551587ab3d@mail.gmail.com> <653bea160507261631585297cf@mail.gmail.com> <51e1bc30050726171176862335@mail.gmail.com> <42E6F6FC.4050507@dirkgerrits.com> Message-ID: <653bea160507262038617a5462@mail.gmail.com> On 26/07/05, Dirk Gerrits wrote: > Joel Reymont wrote: > > I think within Erlang nodes use a keep alive message to make sure they > > have not been split off. Quite often, though, you cannot detect if the > > foreign process exists. If it's a local process id and you send it a > > message and it's dead then you will get a no_proc exception, not so > > with a foreign process. > > > > It does seem to me that the easiest way to verify if a node is alive is > > to send a keep-alive message to a housekeeping process on that node and > > consider it split off if a reply is not received after a timeout. > > Sounds perfectly reasonable to me. > Way to go! Of course, we could add a hook here. And actually, the keep-alive should itself be a option in the meta-level protocol, which is on by default (but would be off on top of, say, SNAIL). >> I'm not sure if it matters if a node is dead or just split off. > > Erlang doesn't make the distinction, does it? > Dunno. That's what we'll have to find out at some point. Eric: you're paid to know before the end of summer :-) > My thinking is that if it's good enough for Erlang, it is /at least/ > good enough for Erlisp stage 1. We can always reconsider later on (even > though it might be more painful then ;)). > Yup. In anycase, the process linking model is the foundation of any robust programming in Erlang. It is much more important to think about doing it properly than to any kind of object marshalling. Joel: can you confirm? Which features of Erlang do you miss most? [ Fran?ois-Ren? ?VB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] I love deadlines. I love the whooshing sound they make as they fly by. -- Douglas Adams From metawilm at gmail.com Wed Jul 27 09:31:56 2005 From: metawilm at gmail.com (Willem Broekema) Date: Wed, 27 Jul 2005 11:31:56 +0200 Subject: [erlisp-devel] Re: SoC status update please In-Reply-To: <42E6F6FC.4050507@dirkgerrits.com> References: <51e1bc3005072609551587ab3d@mail.gmail.com> <653bea160507261631585297cf@mail.gmail.com> <51e1bc30050726171176862335@mail.gmail.com> <42E6F6FC.4050507@dirkgerrits.com> Message-ID: Joel Reymont: > I think within Erlang nodes use a keep alive message to make sure > they have not been split off. Quite often, though, you cannot detect > if the foreign process exists. To check whether a _node_ is still running and connected, monitor_node can be used; from page 98: "If no connection exists, and monitor_node/2 is called, the system will try to setup a connection and deliver a nodedown message if the connection fails." To check whether a _process_ is still alive, linking to it and checking for the corresponding EXIT message is the way to go. Dirk Gerrits: > > I'm not sure if it matters if a node is dead or just split off. > > Erlang doesn't make the distinction, does it? >From the same page: "The BIF [=built-in function] monitor_node(Node, Flag) can be used to monitor nodes. An Erlang process evaluating the expression monitor_node(Node, true) will be notified with a {nodedown, Node} message if Node fails or if the network connection to Node fails. Unfortunately it is not possible to differentiate between node failures and network failures." By the way, my experience with Erlang is mostly in trying to develop a Gnutella client a few years ago. To the reasonable nice original Gnutella protocol, many ill-specified ad-hoc extensions were added by the existing clients. Implementing these extensions, which is necessary in order to play well with the other clients, became less and less interesting, so I never finished it. Although I still find Erlang interesting, these days I'm more interested in Common Lisp. - Willem From metawilm at gmail.com Wed Jul 27 12:19:18 2005 From: metawilm at gmail.com (Willem Broekema) Date: Wed, 27 Jul 2005 14:19:18 +0200 Subject: [erlisp-devel] Re: SoC status update please In-Reply-To: <653bea1605072703498e2f9a5@mail.gmail.com> References: <51e1bc3005072609551587ab3d@mail.gmail.com> <653bea160507261631585297cf@mail.gmail.com> <51e1bc30050726171176862335@mail.gmail.com> <42E6F6FC.4050507@dirkgerrits.com> <653bea1605072703498e2f9a5@mail.gmail.com> Message-ID: On 7/27/05, Far? wrote: > OK. So the question is: what happens if you're linked to a process on > another node, and that node fails? You get an EXIT message, don't you? If the communication with a monitored node fails, but that node is still running its processes, then you get at least a {nodedown, } message. I don't know if you get EXIT messages for the linked processes on that node: on the one hand, the process has not exited yet; on the other hand, if the process exists later on, the EXIT message can not be sent at that time -- so I could imagine either case. It's important to know that a process needs to explicitly state that it wants to receive EXIT signals from linked processes, using "process_flag(trap_exit, true)" in the process that should receive the EXIT signals. If a process P has not set this flag, an exiting linked process P2 will also cause the process P to exit (unless the exit reason of the exiting linked process is "normal"). By setting the flag, process P override this default cascading effect and instead of exiting P will get the EXIT message. > Or do you just sit idly until timeout, and need to have another > process explicitly monitor the remote node and kill you? I guess that > in Lisp, these details are handled with a proper meta-level protocol. > Is there such a protocol in Erlang? I think the behaviour of a single process regarding failures etc is not very variable, so instead of having a meta-protocol at the level of individual processes, there are protocols of how sets of processes behave in an application, in terms of "supervision trees". This deals with automatically restarting failed processes or subsets of supervised processes, and also a cascading restarting into higher application levels when an error persists. Individual processes that implement a "standard" entity like a server or finite-state-machine can save much code by being implemented using a "behaviour". A "supervisor" is another such "standard" entity. Other than the gen_server and gen_fsm behaviours, I have not used these pretty advanced features, so I don't know more about them than what is documented in the document below. "OTP Design Principles" . The information there can be quite overwhelming. I found the following tutorial very useful for understanding the value and usage of the "gen_server behaviour": http://www.duomark.com/erlang/tutorials/proxy.html - Willem From joelr1 at gmail.com Wed Jul 27 13:38:43 2005 From: joelr1 at gmail.com (Joel Reymont) Date: Wed, 27 Jul 2005 15:38:43 +0200 Subject: [erlisp-devel] Re: SoC status update please References: Message-ID: According to the Erlang heavy-weights... Begin forwarded message: > From: "Ulf Wiger" > Date: July 27, 2005 3:34:13 PM GMT+02:00 > > Den 2005-07-27 11:10:53 skrev Joel Reymont : > > >> Uffe, >> >> Can you confirm the keep-alive bit? How does Erlang know that a >> node is dead? >> >> Thanks, Joel >> > > See erl -man kernel > > "net_ticktime = TickTime > > "Specifies the net_kernel tick time. TickTime is given in seconds. > Once every TickTime/4 second, all connected nodes are ticked (if > anything else has been written to a node) and if nothing has been > received from another node within the last four (4) tick times > that node is considered to be down. This ensures that nodes which > are not responding, for reasons such as hardware errors, are > considered to be down. > > The time T, in which a node that is not responding is detected, > is calculated as: MinT < T < MaxT where: > > > MinT = TickTime - TickTime / 4 > MaxT = TickTime + TickTime / 4" > > It should be "if nothing else has been written to the node". > The inet driver keeps counters of packets sent and received. > If no packets have been sent since the last sample, a tick > is sent. > > /Uffe > -- http://wagerlabs.com/uptick From fahree at gmail.com Wed Jul 27 15:33:53 2005 From: fahree at gmail.com (=?ISO-8859-1?Q?Far=E9?=) Date: Wed, 27 Jul 2005 11:33:53 -0400 Subject: [erlisp-devel] Re: SoC status update please In-Reply-To: <653bea1605072703498e2f9a5@mail.gmail.com> References: <51e1bc3005072609551587ab3d@mail.gmail.com> <653bea160507261631585297cf@mail.gmail.com> <51e1bc30050726171176862335@mail.gmail.com> <42E6F6FC.4050507@dirkgerrits.com> <653bea1605072703498e2f9a5@mail.gmail.com> Message-ID: <653bea16050727083388f1e60@mail.gmail.com> [Oops, had sent this reply only to Willem. That's the message he refers to previously] On 27/07/05, Willem Broekema wrote: > To check whether a _node_ is still running and connected, monitor_node > can be used; from > page 98: > > "If no connection exists, and monitor_node/2 is called, the system > will try to setup a connection and deliver a nodedown message if the > connection fails." > > To check whether a _process_ is still alive, linking to it and > checking for the corresponding EXIT message is the way to go. > OK. So the question is: what happens if you're linked to a process on another node, and that node fails? You get an EXIT message, don't you? Or do you just sit idly until timeout, and need to have another process explicitly monitor the remote node and kill you? I guess that in Lisp, these details are handled with a proper meta-level protocol. Is there such a protocol in Erlang? > By the way, my experience with Erlang is mostly in trying to develop a > Gnutella client a few years ago. To the reasonable nice original > Gnutella protocol, many ill-specified ad-hoc extensions were added by > the existing clients. Implementing these extensions, which is > necessary in order to play well with the other clients, became less > and less interesting, so I never finished it. These sounds like the general story of any and all computing projects: interfacing with the horde out there is boring. Which can be seen as an opportunity and a challenge: can we build socio-technical infrastructures which allow to develop code that isn't boring? At least, the company I'm at has found a way to alleviate the bore: make money. > Although I still find > Erlang interesting, these days I'm more interested in Common Lisp. Wait till you have Erlisp, then. Or better: contribute -- your knowledge, for instance: you seem like the person who knows where to quickly find answers to the questions about Erlang :-) [ Fran?ois-Ren? ?VB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] Work harder! Millions on welfare depend on you. From lavigne.eric at gmail.com Thu Jul 28 14:57:32 2005 From: lavigne.eric at gmail.com (Eric Lavigne) Date: Thu, 28 Jul 2005 10:57:32 -0400 Subject: [erlisp-devel] process linking Message-ID: <51e1bc30050728075759b4801e@mail.gmail.com> There has been some good discussion over the last few days, and you've all given me a good idea of how Erlang deals with process linking and related error handling issues. Now I'm going to take a few days to study Erlisp's source code and consider how to proceed. I'll let you know if I have any questions. Thanks, Eric -- http://plaza.ufl.edu/lavigne/ http://www.lispnyc.org/summerofcode.html From lavigne.eric at gmail.com Sun Jul 31 16:57:32 2005 From: lavigne.eric at gmail.com (Eric Lavigne) Date: Sun, 31 Jul 2005 16:57:32 +0000 Subject: [erlisp-devel] SoC planning Message-ID: <51e1bc300507310957772d1f0c@mail.gmail.com> This is a recent AOL instant messenger conversation with my Summer of Code mentor, Far?. Eric (06:55:52) FareTUNES: hi. How's progress? (06:56:26) smokydiamond: I'm still reading the source code. I didn't need to learn much about how Erlisp works to do compiler porting. (06:56:43) FareTUNES: ok (06:56:47) smokydiamond: It is clearly written, though, so I don't think this will be a problem. (06:56:50) FareTUNES: what's your next target? (06:56:56) smokydiamond: process linking (06:57:00) FareTUNES: great! (06:57:09) smokydiamond: followed by all the stuff that depends on that (06:57:57) FareTUNES: indeed (06:58:44) smokydiamond: I think this will be much more interesting than porting. (06:59:22) smokydiamond: Researching compiler features was challenging at times, but never much fun. (06:59:59) FareTUNES: indeed (07:00:02) FareTUNES: :) (07:00:20) FareTUNES: what about distributed thing? (07:00:36) FareTUNES: or forking (which is about the same) (07:01:01) smokydiamond: I don't have time to do both, and it sounds like everyone is more interested in process linking. (07:01:35) smokydiamond: Just 3 weeks left, and I still need to take care of documentation. (07:02:32) FareTUNES: yes, I agree that process linking is way more challenging (07:03:58) FareTUNES: actually, as soc advisor, I'd definitely choose process linking (07:04:52) smokydiamond: I actually made the decision mostly on your advice. Your choice was very clear from mailing list discussions. (07:05:26) smokydiamond: Dirk also thought process linking should come first. (07:05:39) FareTUNES: thanks (07:05:49) smokydiamond: And mailing list discussion picked up a lot when we discussed that topic. (07:05:57) FareTUNES: :) (07:06:38) smokydiamond: Well, I am happy to be doing it also. It really does sound like interesting work :) (07:07:47) FareTUNES: great! (07:08:05) FareTUNES: don't hesitate to contact the list and/or me for any trouble (07:08:17) FareTUNES: it may help you, and it may help us, too :) (07:08:34) smokydiamond: Thanks. Actually, I will be contacting the list soon whether or not I run into trouble. (07:08:56) smokydiamond: When I come up with a plan for doing this, I want to discuss it on the list first. (07:09:13) FareTUNES: ok (07:10:25) FareTUNES: great. gtg (07:10:29) FareTUNES: bye! (07:10:38) smokydiamond: Bye, Fare. (07:10:46) smokydiamond: I am going to post this conversation on the list. (07:11:01) FareTUNES logged out.