From alemmens at xs4all.nl Sun Sep 3 15:08:02 2006 From: alemmens at xs4all.nl (Arthur Lemmens) Date: Sun, 03 Sep 2006 17:08:02 +0200 Subject: [rucksack-devel] Current status Message-ID: OK, I think I've got indexing and schema updates working correctly now. If you want to play around with this, have a look at the test-index*.lisp and test-schema-update-*.lisp files. On Lispworks 5.0, these tests work fine for me, but as usual I haven't bothered to check them out on other Lisps. If there's any breakage on other Lisps, I expect they will be caused by MOP incompatibilities, even though I removed some Lispworks specific MOP magic to keep everything as portable as possible. Bug reports and patches welcome, of course. Next on my list are: * Making Rucksack crash proof I think that the free list updates that are necessary for the mark-and-sweep GC may be too dangerous: if the system crashes while a free list is being updated, we may get an inconsistent free list. That would be fatal. So I intend to move towards a system where nothing is ever updated in the heap: just append new object versions to the end of the heap, with a pointer to the previous version. And use a copying GC to get rid of obsolete versions. Then only the object table needs to be changed to make sure that object ids always point to the most recent version. I'm currently working on the copying GC (incremental of course). Once that works, I intend to write a recovery mechanism along the lines of my talk for the ECLM 2006. * Import/export to s-expressions We need a facility to export the entire contents of a rucksack to some kind of s-expression format (and to create a new rucksack from such an export file). That way, we can migrate existing rucksacks to new versions of Rucksack that may use a different internal structure. * Transactions revisited Finally, I intend to have one more close look at transactions. I still think that it may be interesting to implement the NAMOS system by David Reed (see my email of 2006-05-19 at http://common-lisp.net/pipermail/rucksack-devel/2006-May/000050.html). Arthur From alemmens at xs4all.nl Sun Sep 3 21:22:01 2006 From: alemmens at xs4all.nl (Arthur Lemmens) Date: Sun, 03 Sep 2006 23:22:01 +0200 Subject: [rucksack-devel] Current status In-Reply-To: References: Message-ID: Thomas Stenhaug wrote: > The stack exhaustion still occurs with SBCL. > > I have found that the infinite loop starts in the call to > rucksack-maybe-index-new-object from persistent-object's > initialize-instance :after method, where for some reason it starts > making infinitely many btree objects. Thanks for the report. I *think* that this is due to a difference between Lispworks and SBCL in the implementation of SHARED-INITIALIZE. On Lispworks, SHARED-INITIALIZE does not seem to call (SETF SLOT-VALUE-USING-CLASS), whereas SBCL does seem to call (SETF SLOT-VALUE-USING-CLASS). Maybe somebody with more knowledge of SBCL can confirm this? If this is true, then the infinite recursion on SBCL is probably caused by Rucksack's around method for (SETF SLOT-VALUE-USING-CLASS) during the SHARED-INITIALIZE part of creating one of the initial btrees. I'll try to find a way around this. Arthur From alemmens at xs4all.nl Sun Sep 3 21:45:22 2006 From: alemmens at xs4all.nl (Arthur Lemmens) Date: Sun, 03 Sep 2006 23:45:22 +0200 Subject: [rucksack-devel] SHARED-INITIALIZE calling (SETF SLOT-VALUE-USING-CLASS) Message-ID: Hi Pascal, While working on Rucksack I noticed that Lispworks (5.0) does not seem to call (SETF SLOT-VALUE-USING-CLASS) for the slots that it initializes in SHARED-INITIALIZE, whereas SBCL (0.9.16) *does* call (SETF SLOT-VALUE-USING-CLASS) in that situation. Have you ever noticed this difference? Can you confirm it? Do you know if the MOP specifies one way or the other? Can you think of a way to get Lispworks behave like SBCL, or the other way round? Which is the better approach, according to you? I was a bit surprised when I found out that Lispworks' SHARED-INITIALIZE did not call (SETF SLOT-VALUE-USING-CLASS), but then I programmed my way around it and forgot about it. Until SBCL users started testing my nifty new indexing and schema update features for Rucksack, and complained about stack overflows ;-) Thanks for any insights you may have on this subject, Arthur From alemmens at xs4all.nl Sun Sep 3 22:30:57 2006 From: alemmens at xs4all.nl (Arthur Lemmens) Date: Mon, 04 Sep 2006 00:30:57 +0200 Subject: [rucksack-devel] Current status In-Reply-To: References: Message-ID: I wrote: > I'll try to find a way around this. I wrote a quick and dirty hack to paper over the Lispworks/SBCL differences, and the basic tests (from test.lisp) seem to work fine now with SBCL 0.9.16. I haven't committed the hack yet, because I still get an error with SBCL for one of the schema update tests. I'll try to debug that one too. Arthur From alemmens at xs4all.nl Sun Sep 3 22:53:56 2006 From: alemmens at xs4all.nl (Arthur Lemmens) Date: Mon, 04 Sep 2006 00:53:56 +0200 Subject: [rucksack-devel] Re: SHARED-INITIALIZE calling (SETF SLOT-VALUE-USING-CLASS) In-Reply-To: References: Message-ID: Hi Pascal, Thanks for your quick reply. > I had exactly the same issue some time ago. The surprising thing is > that a CLOS implementation is actually allowed not to use (setf slot- > value-using-class) for object initialization Ah, interesting. > Check out the notes section of shared-initialize in the HyperSpec, there > it is specified under what circumstances (setf slot-value-using-class) > can be circumvented Thanks for the link. > The following is taken from ContextL, and is just a straightforward > implementation of the shared-initalize specification. Feel free to > reuse it. OK, that looks like a good solution to get Lispworks more in line with SBCL (and I suppose some of the other MOP implementations). I've just written a quick Rucksack-specific hack that works the other way round: it makes the other guys behave more like Lispworks. Your approach is probably cleaner in the long run, but it would mean that I'd have to rewrite more existing Rucksack code. > I hope this helps. Definitely. Thanks, Arthur From pc at p-cos.net Sun Sep 3 22:28:58 2006 From: pc at p-cos.net (Pascal Costanza) Date: Mon, 4 Sep 2006 00:28:58 +0200 Subject: [rucksack-devel] Re: SHARED-INITIALIZE calling (SETF SLOT-VALUE-USING-CLASS) In-Reply-To: References: Message-ID: Hi Arthur, I had exactly the same issue some time ago. The surprising thing is that a CLOS implementation is actually allowed not to use (setf slot- value-using-class) for object initialization, though the part that specifices this is somewhat tricky to find. Check out the notes section of shared-initialize in the HyperSpec, there it is specified under what circumstances (setf slot-value-using-class) can be circumvented (although this part actually doesn't explicitly refer to the MOP). My solution was to create a new default super class for my own metaclass, ensure that it is used instead of standard-object, and then specialize shared-initialize accordingly. The following is taken from ContextL, and is just a straightforward implementation of the shared-initalize specification. Feel free to reuse it. #+(or lispworks (and mcl (not openmcl))) (defmethod shared-initialize ((object special-object) slot-names &rest all-keys) (declare (dynamic-extent all-keys)) (let ((class-slots (class-slots (class-of object)))) (loop for slot in class-slots for slot-initargs = (slot-definition-initargs slot) when slot-initargs do (multiple-value-bind (indicator value) (get-properties all-keys slot-initargs) (when indicator (setf (slot-value object (slot-definition-name slot)) value)))) (if (eq slot-names 't) (loop for slot in class-slots for slot-name = (slot-definition-name slot) unless (slot-boundp object slot-name) do (let ((slot-initfunction (slot-definition-initfunction slot))) (when slot-initfunction (setf (slot-value object slot-name) (funcall slot- initfunction))))) (loop for slot-name in slot-names for slot = (find slot-name class-slots :key #'slot- definition-name) unless (slot-boundp object slot-name) do (let ((slot-initfunction (slot-definition-initfunction slot))) (when slot-initfunction (setf (slot-value object slot-name) (funcall slot- initfunction))))))) object) Note that I intentionally do not use (setf slot-value-using-class) here, but (setf slot-value) because otherwise this would break in some implementations. (I don't remember which one it was.) The effect, however, is that (setf slot-value-using-class) will ultimately be called - and then it works correctly. Don't ask me why... ;) I hope this helps. Pascal On 3 Sep 2006, at 23:45, Arthur Lemmens wrote: > Hi Pascal, > > While working on Rucksack I noticed that Lispworks (5.0) does not > seem to call (SETF SLOT-VALUE-USING-CLASS) for the slots that it > initializes in SHARED-INITIALIZE, whereas SBCL (0.9.16) *does* > call (SETF SLOT-VALUE-USING-CLASS) in that situation. > > Have you ever noticed this difference? Can you confirm it? Do you > know if the MOP specifies one way or the other? Can you think of a > way to get Lispworks behave like SBCL, or the other way round? > > Which is the better approach, according to you? I was a bit surprised > when I found out that Lispworks' SHARED-INITIALIZE did not call > (SETF SLOT-VALUE-USING-CLASS), but then I programmed my way around it > and forgot about it. Until SBCL users started testing my nifty new > indexing and schema update features for Rucksack, and complained about > stack overflows ;-) > > Thanks for any insights you may have on this subject, > > Arthur > > -- Pascal Costanza, mailto:pc at p-cos.net, http://p-cos.net Vrije Universiteit Brussel, Programming Technology Lab Pleinlaan 2, B-1050 Brussel, Belgium From alemmens at xs4all.nl Mon Sep 4 12:38:01 2006 From: alemmens at xs4all.nl (Arthur Lemmens) Date: Mon, 04 Sep 2006 14:38:01 +0200 Subject: [rucksack-devel] MOP differences between Lispworks and SBCL Message-ID: I've just committed some fixes that should take care of the MOP differences between Lispworks and SBCL. I found two differences: - As discussed yesterday, Lispworks doesn't call (setf slot-value-using-class) in SHARED-INITIALIZE, but SBCL does. - SBCL doesn't call FINALIZE-INHERITANCE when a new instance is created of a class that has been redefined, but Lispworks does. I've run the schema update and indexing tests in test-*.lisp on Lispworks 5.0 and SBCL 0.9.16, and they all seem to work fine now. If they don't run on your implementation, please let me know. (And if they do run, that's also good to know.) OK, back to implementing a copying garbage collector... From keke at gol.com Wed Sep 6 03:48:33 2006 From: keke at gol.com (Takehiko Abe) Date: Wed, 6 Sep 2006 12:48:33 +0900 Subject: [rucksack-devel] CVS help Message-ID: <20060906034833.29323@roaming.gol.com:25> All the files I checked out from the rucksack repository contains CRLF instead of LF. I believe that cvs should do automatic EOL conversion but couldn't figure out how I enable it. I use cvs v 1.10 that came with Mac OS X 10.3. I'd appreciate any help greatly. Sorry for the lame question. Thanks, Takehiko Abe From keke at gol.com Wed Sep 6 06:20:49 2006 From: keke at gol.com (Takehiko Abe) Date: Wed, 6 Sep 2006 15:20:49 +0900 Subject: [rucksack-devel] CVS help In-Reply-To: <0009F263-7140-4B8A-A797-F95307295171@metabang.com> References: <0009F263-7140-4B8A-A797-F95307295171@metabang.com> Message-ID: <20060906062049.13222@roaming.gol.com:25> Thanks for the response. Gary King wrote: > My understanding is the the CVS client takes care of the line feeds. > Unfortunately (or not depending on what you actually want ), > the command line clients on OS X generally thinks that you want CR > instead of LF (i.e., the Unix thing rather than the Mac thing). I > used to use a MacCVS client that took care of this. I believe that > there are patches to get MCL to happily use the Unix line ending. Well, Unix EOL char is LF and I have already converted all my source files to LF long time ago, and MCL 5.1 is fine wihout patches. However, it still doesn't handle CRLF (DOS EOL). And that's what I got from the checkout -- CRLF. Please tell me if you don't. From gwking at metabang.com Wed Sep 6 05:59:13 2006 From: gwking at metabang.com (Gary King) Date: Wed, 6 Sep 2006 01:59:13 -0400 Subject: [rucksack-devel] CVS help In-Reply-To: <20060906034833.29323@roaming.gol.com:25> References: <20060906034833.29323@roaming.gol.com:25> Message-ID: <0009F263-7140-4B8A-A797-F95307295171@metabang.com> My understanding is the the CVS client takes care of the line feeds. Unfortunately (or not depending on what you actually want ), the command line clients on OS X generally thinks that you want CR instead of LF (i.e., the Unix thing rather than the Mac thing). I used to use a MacCVS client that took care of this. I believe that there are patches to get MCL to happily use the Unix line ending. HTH, On Sep 5, 2006, at 11:48 PM, Takehiko Abe wrote: > All the files I checked out from the rucksack repository > contains CRLF instead of LF. I believe that cvs should do > automatic EOL conversion but couldn't figure out how I > enable it. > > I use cvs v 1.10 that came with Mac OS X 10.3. I'd appreciate > any help greatly. > > Sorry for the lame question. > > Thanks, > Takehiko Abe > > > _______________________________________________ > rucksack-devel mailing list > rucksack-devel at common-lisp.net > http://common-lisp.net/cgi-bin/mailman/listinfo/rucksack-devel -- Gary Warren King, metabang.com (413) 885 9127, gwkkwg on Skype -------------- next part -------------- An HTML attachment was scrubbed... URL: From luismbo at gmail.com Thu Sep 28 21:55:56 2006 From: luismbo at gmail.com (=?ISO-8859-1?Q?Lu=EDs_Oliveira?=) Date: Thu, 28 Sep 2006 22:55:56 +0100 Subject: [rucksack-devel] Using rucksack for simple object serialization. Message-ID: <391f79580609281455g3f33c227m3192080cab8ea141@mail.gmail.com> Hello, I am looking into the possibility of using rucksack at first for simple object serialization. I have two questions. a) Is rucksack stable enough for such use? b) Since the serialization stuff in serialize.lisp does not handle object circularity by itself, does rucksack provide some other API to serialize objects to a stream? Thanks in advance, -- Lu?s Oliveira http://student.dei.uc.pt/~lmoliv/ From alemmens at xs4all.nl Fri Sep 29 07:03:59 2006 From: alemmens at xs4all.nl (Arthur Lemmens) Date: Fri, 29 Sep 2006 09:03:59 +0200 Subject: [rucksack-devel] Using rucksack for simple object serialization. In-Reply-To: <391f79580609281455g3f33c227m3192080cab8ea141@mail.gmail.com> References: <391f79580609281455g3f33c227m3192080cab8ea141@mail.gmail.com> Message-ID: Lu?s Oliveira wrote: > I am looking into the possibility of using rucksack at first for > simple object serialization. I have two questions. a) Is rucksack > stable enough for such use? Yes, I think so. Serialization is one of the easy parts of Rucksack, and I don't expect many problems there. Then again, if you only care about serialization you could consider using CL-STORE instead of Rucksack. > Since the serialization stuff in serialize.lisp does not handle > object circularity by itself, does rucksack provide some other API > to serialize objects to a stream? Hmm... I created Rucksack's serialization library by simplifying a standalone serialization library that *did* handle object circularity. I could consider putting those parts back. Alternatively, you could accept a bit more of Rucksack's normal paradigm: use classes with metaclass PERSISTENT-CLASS for those objects that may be referenced more than once, and wrap a WITH-RUCKSACK and WITH-TRANSACTION around the parts of your program where persistent objects are created or modified. If you don't care about stuff like transactions or garbage collection of persistent objects, it would be pretty easy to turn those off. Arthur From klaus at mu.dk Fri Sep 29 07:25:30 2006 From: klaus at mu.dk (Klaus Harbo) Date: Fri, 29 Sep 2006 09:25:30 +0200 Subject: [rucksack-devel] Using rucksack for simple object serialization. In-Reply-To: <391f79580609281455g3f33c227m3192080cab8ea141@mail.gmail.com> References: <391f79580609281455g3f33c227m3192080cab8ea141@mail.gmail.com> Message-ID: <451CCA6A.3010404@mu.dk> Lu?s Oliveira wrote: > Hello, > > I am looking into the possibility of using rucksack at first for > simple object serialization. I have two questions. a) Is rucksack > stable enough for such use? b) Since the serialization stuff in > serialize.lisp does not handle object circularity by itself, does > rucksack provide some other API to serialize objects to a stream? > > Thanks in advance, > I am using rucksack's serializer for exactly that, without any problems whatsoever. Ripping out serialize.lisp is quite straightforward, although it's not quite stand-alone, it needs a few things from another file, I don't have file the files for reference right here. As Arthur says, you may want to look at CL-STORE as well. The reason I chose this approach is (1) I'm quite interested in rucksack, (2) I dare not use rucksack for production work quite yet, and (3) this would get me started getting to know rucksack and working with the code. -Klaus. PS. I have also upgraded the serializer to be able to serialize to a vector of (unsigned-byte 8). It's not clear that that is useful to rucksack, but let me know if you're interested in a patch. -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3368 bytes Desc: S/MIME Cryptographic Signature URL: From klaus at mu.dk Fri Sep 29 07:32:22 2006 From: klaus at mu.dk (Klaus Harbo) Date: Fri, 29 Sep 2006 09:32:22 +0200 Subject: [rucksack-devel] Using rucksack for simple object serialization. In-Reply-To: <451CCA6A.3010404@mu.dk> References: <391f79580609281455g3f33c227m3192080cab8ea141@mail.gmail.com> <451CCA6A.3010404@mu.dk> Message-ID: <451CCC06.1030808@mu.dk> Klaus Harbo wrote: > PS. I have also upgraded the serializer to be able to serialize to a > vector of (unsigned-byte 8). It's not clear that that is useful to > rucksack, but let me know if you're interested in a patch. The above was a bit ambigous, so to clarify: ... to upgrade rucksack's serializer with this functionality. -K. -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3368 bytes Desc: S/MIME Cryptographic Signature URL: From alemmens at xs4all.nl Fri Sep 29 15:53:44 2006 From: alemmens at xs4all.nl (Arthur Lemmens) Date: Fri, 29 Sep 2006 17:53:44 +0200 Subject: [rucksack-devel] Using rucksack for simple object serialization. In-Reply-To: <451CCA6A.3010404@mu.dk> References: <391f79580609281455g3f33c227m3192080cab8ea141@mail.gmail.com> <451CCA6A.3010404@mu.dk> Message-ID: Klaus Harbo wrote: > PS. I have also upgraded the serializer to be able to serialize to a vector of > (unsigned-byte 8). Have you seen the code for serialization buffers in heap.lisp? That also does serialization to a vector of (unsigned-byte 8). > let me know if you're interested in a patch. Yes, I'd like to have a look at your approach. Thanks. Arthur From klaus at mu.dk Sat Sep 30 08:50:32 2006 From: klaus at mu.dk (Klaus Harbo) Date: Sat, 30 Sep 2006 10:50:32 +0200 Subject: [rucksack-devel] Using rucksack for simple object serialization. In-Reply-To: References: <391f79580609281455g3f33c227m3192080cab8ea141@mail.gmail.com> <451CCA6A.3010404@mu.dk> Message-ID: <451E2FD8.7060900@mu.dk> Arthur Lemmens wrote: > Have you seen the code for serialization buffers in heap.lisp? That also > does serialization to a vector of (unsigned-byte 8). No, I hadn't. I think what I've done may be redundant. See sketch below. >> let me know if you're interested in a patch. > > Yes, I'd like to have a look at your approach. Thanks. > > Arthur > I think what I did is straightforward, namely (defclass serializer () ()) (defclass stream-serializer (serializer) ((stream :initarg :stream :reader serializer-stream :documentation "An (unsigned-byte 8) stream."))) (defclass vector-serializer (serializer) ((vector :initform (make-array '(256) :element-type '(unsigned-byte 8) :adjustable t :fill-pointer 0) :reader serializer-vector) (read-offset :initform 0 :accessor serializer-offset))) combined with (defgeneric serialize-byte (byte serializer) (:method ((byte integer) (serializer stream-serializer)) (write-byte byte (serializer-stream serializer))) (:method ((byte integer) (serializer vector-serializer)) (vector-push-extend byte (serializer-vector serializer))) (:method ((byte integer) (stream stream)) (write-byte byte stream)) (:documentation "Writes an unsigned-byte to a serializer.")) (defgeneric deserialize-byte (serializer &optional eof-error-p) (:method ((serializer stream-serializer) &optional (eof-error-p t)) (read-byte (serializer-stream serializer) eof-error-p nil)) (:method ((serializer vector-serializer) &optional (eof-error-p t)) (with-slots (vector read-offset) serializer (when (>= read-offset (fill-pointer vector)) (error "End of file in vector.")) (prog1 (aref vector read-offset) (incf read-offset)))) (:method ((stream stream) &optional (eof-error-p t)) (read-byte stream eof-error-p nil)) (:documentation "Reads an unsigned-byte from a serializer. EOF-ERROR-P defaults to T.")) I think that is pretty much it; at least I don't remember making any other changes. A quick glance at heap.lisp suggests to me that it is pretty much equivalent to the code above. I obviously don't understand the overall organization of the files, but I never thought to look in heap.lisp for a vector-serializer! Perhaps the location is due to your `serialization-buffer' inheriting from `buffer'? -Klaus. From alemmens at xs4all.nl Sat Sep 30 17:47:40 2006 From: alemmens at xs4all.nl (Arthur Lemmens) Date: Sat, 30 Sep 2006 19:47:40 +0200 Subject: [rucksack-devel] Using rucksack for simple object serialization. In-Reply-To: <451E2FD8.7060900@mu.dk> References: <391f79580609281455g3f33c227m3192080cab8ea141@mail.gmail.com> <451CCA6A.3010404@mu.dk> <451E2FD8.7060900@mu.dk> Message-ID: Klaus Harbo wrote: > A quick glance at heap.lisp suggests to me that it is pretty much > equivalent to the code above. Yes, I'd say so too. > I obviously don't understand the overall organization of the files, but > I never thought to look in heap.lisp for a vector-serializer! Erm, yes, you're right. It's a stupid place and I'll move it to serialize.lisp. > Perhaps the location is due to your `serialization-buffer' inheriting > from `buffer'? I probably put it in heap.lisp because it was needed originally by the heap code. Arthur