From brad.beveridge at gmail.com Wed Jan 16 05:35:53 2008 From: brad.beveridge at gmail.com (Brad Beveridge) Date: Tue, 15 Jan 2008 21:35:53 -0800 Subject: [rucksack-devel] Indexed classes Message-ID: Hello list, I am writing a unit test suit for Rucksack and have come across a use case that makes no sense to me. I have defined a class such: (defclass foo () ((data :initarg :data :accessor data :index :case-insensitive-string-index)) (:index t) (:metaclass persistent-class)) If I run the following code: (with-rucksack-and-transaction (make-instance 'foo :data "foobar") ) (with-rucksack-and-transaction (rucksack-map-class *rucksack* 'foo (lambda (obj) (format t "~A~%" obj) )) ) I get nothing printed out! If I wrap the DEFCLASS in a WITH-RUCKSACK-AND-TRANSACTION, then it works. I am guessing that at defclass time you must have a open rucksack and valid transaction, and that some MOP magic will populate a btree index. This feels quite strange to me, for example it would be common for an application to startup with no object store & expect to be able to correctly instance indexed objects and have them persist. Otherwise you need to arrange to re-evaluate the defclass form whenever you delete your store. I would expect that when you MAKE-INSTANCE an indexed class Rucksack should test to see if the index exists, and if it doesn't it should create one. I'm new to MOP & haven't looked into this yet. Any hints on where I should look to try & fix this. Or am I misunderstanding a design feature/goal as a bug? Cheers, Brad From alemmens at xs4all.nl Wed Jan 16 14:28:39 2008 From: alemmens at xs4all.nl (Arthur Lemmens) Date: Wed, 16 Jan 2008 15:28:39 +0100 Subject: [rucksack-devel] Indexed classes In-Reply-To: References: Message-ID: Brad Beveridge wrote: > I have defined a class such: > (defclass foo () > ((data :initarg :data :accessor data > :index :case-insensitive-string-index)) > (:index t) > (:metaclass persistent-class)) > > If I run the following code: > > (with-rucksack-and-transaction > (make-instance 'foo :data "foobar") > ) > > (with-rucksack-and-transaction > (rucksack-map-class *rucksack* 'foo > (lambda (obj) > (format t "~A~%" obj) > )) > ) > > I get nothing printed out! Yes, if there is no current rucksack when the persistent class is defined, Rucksack will not update any indices for that class. Rucksack should probably signal an error in this case, but it doesn't at the moment. > If I wrap the DEFCLASS in a WITH-RUCKSACK-AND-TRANSACTION, then it > works. I am guessing that at defclass time you must have a open > rucksack and valid transaction, and that some MOP magic will populate > a btree index. Right. See the file mop.lisp, and in particular the definitions of UPDATE-INDEXES and around methods for INITIALIZE-INSTANCE and REINITIALIZE-INSTANCE for PERSISTENT-CLASS. > This feels quite strange to me, for example it would be common for an > application to startup with no object store & expect to be able to > correctly instance indexed objects and have them persist. Otherwise > you need to arrange to re-evaluate the defclass form whenever you > delete your store. > > I would expect that when you MAKE-INSTANCE an indexed class Rucksack > should test to see if the index exists, and if it doesn't it should > create one. Hmm, interesting idea. I've never been very fond of the WITH-RUCKSACK and WITH-TRANSACTION wrappers around DEFCLASS forms, but somehow I thought they were necessary. Maybe you're right and we can get rid of them. We already have a call to MAYBE-UPDATE-SLOT-INFO at the start of the INITIALIZE-INSTANCE around method for PERSISTENT-OBJECT (see objects.lisp), so maybe we can just add a MAYBE-UPDATE-INDEXES call at that point. > I'm new to MOP & haven't looked into this yet. Any hints on where I > should look to try & fix this. Yes: the files and functions I just mentioned. > Or am I misunderstanding a design feature/goal as a bug? Initially I thought it was a feature, but maybe you're right and it's a bug. I need to think a bit more about this. Thanks for your feedback, Arthur From alemmens at xs4all.nl Wed Jan 16 15:14:20 2008 From: alemmens at xs4all.nl (Arthur Lemmens) Date: Wed, 16 Jan 2008 16:14:20 +0100 Subject: [rucksack-devel] (no subject) In-Reply-To: References: Message-ID: Leonid Novikov wrote: > CL-USER> (rucksack:with-rucksack (rs *tmp-rucksack*) > (rucksack:with-transaction () > (rucksack:rucksack-map-class rs 'plan-of-acc2 > (lambda (x) > (setf (name x) "w"))))) > > Argument X is not a NUMBER: NIL > [Condition of type SIMPLE-TYPE-ERROR] Yes: when deleting a key from a btree, Rucksack should use the BTREE-KEY= function (not P-EQL) to determine the position of the key. Thanks a lot for the bug report and the patch. I just committed your patch to CVS, with a small modification: I made the new TEST argument (for REMOVE-KEY and KEY-POSITION) a required argument instead of a keyword argument. Arthur From brad.beveridge at gmail.com Wed Jan 16 16:10:13 2008 From: brad.beveridge at gmail.com (Brad Beveridge) Date: Wed, 16 Jan 2008 08:10:13 -0800 Subject: [rucksack-devel] Indexed classes In-Reply-To: References: Message-ID: On 16/01/2008, Arthur Lemmens wrote: > Brad Beveridge wrote: > > > I have defined a class such: > > > I get nothing printed out! > > Yes, if there is no current rucksack when the persistent class is > defined, Rucksack will not update any indices for that class. Rucksack > should probably signal an error in this case, but it doesn't at the > moment. > > > If I wrap the DEFCLASS in a WITH-RUCKSACK-AND-TRANSACTION, then it > > works. I am guessing that at defclass time you must have a open > > rucksack and valid transaction, and that some MOP magic will populate > > a btree index. > > Right. See the file mop.lisp, and in particular the definitions of > UPDATE-INDEXES and around methods for INITIALIZE-INSTANCE and > REINITIALIZE-INSTANCE for PERSISTENT-CLASS. > > > > This feels quite strange to me, for example it would be common for an > > application to startup with no object store & expect to be able to > > correctly instance indexed objects and have them persist. Otherwise > > you need to arrange to re-evaluate the defclass form whenever you > > delete your store. > > > > I would expect that when you MAKE-INSTANCE an indexed class Rucksack > > should test to see if the index exists, and if it doesn't it should > > create one. > > Hmm, interesting idea. I've never been very fond of the WITH-RUCKSACK > and WITH-TRANSACTION wrappers around DEFCLASS forms, but somehow I > thought they were necessary. Maybe you're right and we can get rid of > them. We already have a call to MAYBE-UPDATE-SLOT-INFO at the start > of the INITIALIZE-INSTANCE around method for PERSISTENT-OBJECT (see > objects.lisp), so maybe we can just add a MAYBE-UPDATE-INDEXES call at > that point. > > > I'm new to MOP & haven't looked into this yet. Any hints on where I > > should look to try & fix this. > > Yes: the files and functions I just mentioned. > > > Or am I misunderstanding a design feature/goal as a bug? > > Initially I thought it was a feature, but maybe you're right and it's > a bug. I need to think a bit more about this. > > Thanks for your feedback, > Thanks for the great library. Right now I've simply wrapped the defclasses in a transaction. My immediate goal is to get a test suite with ~80% coverage (sb-cover to the rescue) finished in the next couple of weeks. After that I'm not sure what I'll do, probably try & fix bugs and try to fill in the important bits of the test coverage. Cheers, Brad From edi at agharta.de Wed Jan 16 16:52:15 2008 From: edi at agharta.de (Edi Weitz) Date: Wed, 16 Jan 2008 17:52:15 +0100 Subject: [rucksack-devel] Indexed classes In-Reply-To: (Brad Beveridge's message of "Wed, 16 Jan 2008 08:10:13 -0800") References: Message-ID: On Wed, 16 Jan 2008 08:10:13 -0800, "Brad Beveridge" wrote: > After that I'm not sure what I'll do If you want to contribute to the project, my suggestion would be to write documentation and/or a tutorial. That will in turn attract more users which means more testers, bug fixes, and so on... Just my .02 EUR, of course. From henrik at evahjelte.com Thu Jan 17 09:51:26 2008 From: henrik at evahjelte.com (Henrik Hjelte) Date: Thu, 17 Jan 2008 10:51:26 +0100 Subject: [rucksack-devel] Indexed classes In-Reply-To: References: Message-ID: <50e8e4f60801170151t5636fd2dv24b2d3197e7847dd@mail.gmail.com> On Jan 16, 2008 5:10 PM, Brad Beveridge wrote: > After that I'm not sure what I'll do, probably try & > fix bugs and try to fill in the important bits of the test coverage. I really like the rucksack code and design, but it has a major problem and that is that it doesn't work with several connected lisp processes. That makes it an impossible choice for many applications. So my suggestion on a contribution is to fix that. Perhaps add a client-server option using iolib? Some to bite in.. /Henrik From edi at agharta.de Thu Jan 17 19:03:00 2008 From: edi at agharta.de (Edi Weitz) Date: Thu, 17 Jan 2008 20:03:00 +0100 Subject: [rucksack-devel] Indexed classes In-Reply-To: <50e8e4f60801170151t5636fd2dv24b2d3197e7847dd@mail.gmail.com> (Henrik Hjelte's message of "Thu, 17 Jan 2008 10:51:26 +0100") References: <50e8e4f60801170151t5636fd2dv24b2d3197e7847dd@mail.gmail.com> Message-ID: On Thu, 17 Jan 2008 10:51:26 +0100, "Henrik Hjelte" wrote: > I really like the rucksack code and design, but it has a major > problem and that is that it doesn't work with several connected lisp > processes. AFAIU that's not a "problem", but a design decision. If only one of your processes accesses the database, then code which takes care of concurrency is unneeded overhead (and will maybe introduce new bugs). If you have more than one process, there are locks. > That makes it an impossible choice for many applications. Really? > So my suggestion on a contribution is to fix that. Perhaps add a > client-server option using iolib? Well, it's Arthur's code, but my take on this is that the major advantage of Rucksack over almost all other Lisp persistence packages is that it is pure Lisp. Introducing a foreign library just for this task seems like a very bad idea to me. Edi. From henrik at evahjelte.com Fri Jan 18 11:17:18 2008 From: henrik at evahjelte.com (Henrik Hjelte) Date: Fri, 18 Jan 2008 12:17:18 +0100 Subject: [rucksack-devel] Indexed classes In-Reply-To: References: <50e8e4f60801170151t5636fd2dv24b2d3197e7847dd@mail.gmail.com> Message-ID: <50e8e4f60801180317h582734cej8f30605ad0f281ed@mail.gmail.com> On Jan 17, 2008 8:03 PM, Edi Weitz wrote: > On Thu, 17 Jan 2008 10:51:26 +0100, "Henrik Hjelte" wrote: > > > I really like the rucksack code and design, but it has a major > > problem and that is that it doesn't work with several connected lisp > > processes. > > AFAIU that's not a "problem", but a design decision. If only one of > your processes accesses the database, then code which takes care of > concurrency is unneeded overhead (and will maybe introduce new bugs). > If you have more than one process, there are locks. With the rucksack multiversion concurrency and transaction handling, much of the thinking and code is already in place, already causing these effects. > > > That makes it an impossible choice for many applications. > > Really? My hypothesis is that the majority of all applications that wants a database wants to be able to use it from more than one process. At least my mirrorball says that the number of rucksack users will increase. > Well, it's Arthur's code, but my take on this is that the major > advantage of Rucksack over almost all other Lisp persistence packages > is that it is pure Lisp. And open source, good code quality, not dependent on an external program such as a database. There is no reason to kick people out, of course it should be possible to use several backends. The idea is that lisp puritans with single user applications can use one backend, and other could use a concurrent client-server store. > Introducing a foreign library just for this > task seems like a very bad idea to me. Just for this task? IMO the ability for several users to access a database concurrently is more than a little task. /Henrik From edi at agharta.de Sun Jan 20 23:51:01 2008 From: edi at agharta.de (Edi Weitz) Date: Mon, 21 Jan 2008 00:51:01 +0100 Subject: [rucksack-devel] Indexed classes In-Reply-To: <50e8e4f60801180317h582734cej8f30605ad0f281ed@mail.gmail.com> (Henrik Hjelte's message of "Fri, 18 Jan 2008 12:17:18 +0100") References: <50e8e4f60801170151t5636fd2dv24b2d3197e7847dd@mail.gmail.com> <50e8e4f60801180317h582734cej8f30605ad0f281ed@mail.gmail.com> Message-ID: On Fri, 18 Jan 2008 12:17:18 +0100, "Henrik Hjelte" wrote: > The idea is that lisp puritans with single user applications can use > one backend, and other could use a concurrent client-server store. And concurrent client-server stores can't be written in Lisp? > Just for this task? IMO the ability for several users to access a > database concurrently is more than a little task. OK, maybe the wording should have been different. My main gripe is that you want to use a foreign library for this. This has nothing to do with being a "Lisp puritan" as you call it, it's simply the consequence of being pragmatic. I've written about this in other places, so I'm not going to repeat it here. Search c.l.l, for example. From brad.beveridge at gmail.com Mon Jan 21 03:26:50 2008 From: brad.beveridge at gmail.com (Brad Beveridge) Date: Sun, 20 Jan 2008 19:26:50 -0800 Subject: [rucksack-devel] Error when deleting indexed objects Message-ID: I've run into an error when deleting classes that have slots that are indexed by string. The problem only occurs when you close & reopen the Rucksack before deleting, which makes me suspect it is something to do with the proxying. I've tried to debug this, but I'm so far not getting much traction :) I'm going to keep looking at it, in the meantime here is a repro testcase. Cheers, Brad (with-rucksack (rs *test-suite* :if-exists :supersede) ; Works (with-transaction () (defclass broken () ((string-key :initarg :data :accessor string-key :index :string-index)) (:index t) (:metaclass persistent-class)) (make-instance 'broken :data "foo1") (make-instance 'broken :data "foo2") (rucksack-map-class rs 'broken (lambda (obj) (format t "Found ~A~%" (string-key obj)))) (rucksack-map-slot rs 'broken 'string-key (lambda (obj) (rucksack::rucksack-delete-object rs obj)) :equal "foo1") (rucksack-map-class rs 'broken (lambda (obj) (format t "Found ~A~%" (string-key obj)))) )) (progn ; Broken (with-rucksack (rs *test-suite* :if-exists :supersede) (with-transaction () (defclass broken () ((string-key :initarg :data :accessor string-key :index :string-index)) (:index t) (:metaclass persistent-class)) (make-instance 'broken :data "foo1") (make-instance 'broken :data "foo2") (rucksack-map-class rs 'broken (lambda (obj) (format t "Found ~A~%" (string-key obj)))) )) (with-rucksack (rs *test-suite*) (with-transaction () (rucksack-map-slot rs 'broken 'string-key (lambda (obj) (rucksack::rucksack-delete-object rs obj)) :equal "foo1") (rucksack-map-class rs 'broken (lambda (obj) (format t "Found ~A~%" (string-key obj)))) ))) From brad.beveridge at gmail.com Tue Jan 22 06:08:21 2008 From: brad.beveridge at gmail.com (Brad Beveridge) Date: Mon, 21 Jan 2008 22:08:21 -0800 Subject: [rucksack-devel] Re: Error when deleting indexed objects In-Reply-To: References: Message-ID: On 20/01/2008, Brad Beveridge wrote: > I've run into an error when deleting classes that have slots that are > indexed by string. The problem only occurs when you close & reopen > the Rucksack before deleting, which makes me suspect it is something > to do with the proxying. I've tried to debug this, but I'm so far not > getting much traction :) > I'm going to keep looking at it, in the meantime here is a repro testcase. > > Cheers, > Brad > > (with-rucksack (rs *test-suite* :if-exists :supersede) > ; Works > (with-transaction () > (defclass broken () > ((string-key :initarg :data :accessor string-key > :index :string-index)) > (:index t) > (:metaclass persistent-class)) > (make-instance 'broken :data "foo1") > (make-instance 'broken :data "foo2") > (rucksack-map-class rs 'broken > (lambda (obj) > (format t "Found ~A~%" (string-key obj)))) > (rucksack-map-slot rs 'broken 'string-key > (lambda (obj) > (rucksack::rucksack-delete-object rs obj)) > :equal "foo1") > (rucksack-map-class rs 'broken > (lambda (obj) > (format t "Found ~A~%" (string-key obj)))) > )) > > (progn > ; Broken > (with-rucksack (rs *test-suite* :if-exists :supersede) > (with-transaction () > (defclass broken () > ((string-key :initarg :data :accessor string-key > :index :string-index)) > (:index t) > (:metaclass persistent-class)) > (make-instance 'broken :data "foo1") > (make-instance 'broken :data "foo2") > (rucksack-map-class rs 'broken > (lambda (obj) > (format t "Found ~A~%" (string-key obj)))) > )) > > (with-rucksack (rs *test-suite*) > (with-transaction () > > (rucksack-map-slot rs 'broken 'string-key > (lambda (obj) > (rucksack::rucksack-delete-object rs obj)) > :equal "foo1") > (rucksack-map-class rs 'broken > (lambda (obj) > (format t "Found ~A~%" (string-key obj)))) > ))) > I've made some progress. The problem is due to KEY-POSITION using P-EQL when trying to find the position of a key in the index. It really should be more like (defun key-position (btree key node) (p-position key (btree-node-index node) :key #'binding-key :test (btree-key= btree) :end (btree-node-index-count node))) However, that doesn't work either because the largest key in the index will have a key of KEY-IRRELEVENT - clearly a special case. Am I making sense here? I guess the correct place to handle the special case is in BTREE-KEY= Cheers Brad From brad.beveridge at gmail.com Tue Jan 22 06:15:31 2008 From: brad.beveridge at gmail.com (Brad Beveridge) Date: Mon, 21 Jan 2008 22:15:31 -0800 Subject: [rucksack-devel] Re: Error when deleting indexed objects In-Reply-To: References: Message-ID: I've attached a patch that appears to solve my problem. Brad -------------- next part -------------- A non-text attachment was scrubbed... Name: delete-strings.patch Type: application/octet-stream Size: 1281 bytes Desc: not available URL: From alemmens at xs4all.nl Tue Jan 22 16:04:28 2008 From: alemmens at xs4all.nl (Arthur Lemmens) Date: Tue, 22 Jan 2008 17:04:28 +0100 Subject: [rucksack-devel] Re: Error when deleting indexed objects In-Reply-To: References: Message-ID: Brad Beveridge wrote: > I've attached a patch that appears to solve my problem. Thanks a lot for the bug report and the patch. I just committed your patch to CVS (as version 0.1.11). Arthur From alemmens at xs4all.nl Tue Jan 22 16:09:33 2008 From: alemmens at xs4all.nl (Arthur Lemmens) Date: Tue, 22 Jan 2008 17:09:33 +0100 Subject: [rucksack-devel] Re: Error when deleting indexed objects In-Reply-To: References: Message-ID: Brad Beveridge wrote: > I've made some progress. The problem is due to KEY-POSITION using > P-EQL when trying to find the position of a key in the index. It > really should be more like > (defun key-position (btree key node) > (p-position key (btree-node-index node) > :key #'binding-key > :test (btree-key= btree) > :end (btree-node-index-count node))) Erm, yes. This particular problem was already fixed by Leonid Novikov's patch that I committed to CVS last week. > However, that doesn't work either because the largest key in the index > will have a key of KEY-IRRELEVENT - clearly a special case. Right. This is a new problem that I hadn't seen before. > Am I making sense here? I guess the correct place to handle the > special case is in BTREE-KEY= I'm not totally sure that this is the best approach, but I can't think of anything better right now and it does seem to fix the problem. So I followed your suggestion. Thanks, Arthur From alemmens at xs4all.nl Tue Jan 22 16:14:33 2008 From: alemmens at xs4all.nl (Arthur Lemmens) Date: Tue, 22 Jan 2008 17:14:33 +0100 Subject: [rucksack-devel] Error when deleting indexed objects In-Reply-To: References: Message-ID: Brad Beveridge wrote: > (rucksack-map-slot rs 'broken 'string-key > (lambda (obj) > (rucksack::rucksack-delete-object rs obj)) > :equal "foo1") I forgot to mention that this way of deleting an object is a bad idea: you're traversing a data structure (the slot index) and modifying it (by deleting the object) at the same time. Arthur From brad.beveridge at gmail.com Tue Jan 22 16:59:35 2008 From: brad.beveridge at gmail.com (Brad Beveridge) Date: Tue, 22 Jan 2008 08:59:35 -0800 Subject: [rucksack-devel] Error when deleting indexed objects In-Reply-To: References: Message-ID: On 22/01/2008, Arthur Lemmens wrote: > Brad Beveridge wrote: > > > (rucksack-map-slot rs 'broken 'string-key > > (lambda (obj) > > (rucksack::rucksack-delete-object rs obj)) > > :equal "foo1") > > I forgot to mention that this way of deleting an object is a bad > idea: you're traversing a data structure (the slot index) and > modifying it (by deleting the object) at the same time. You're right. I guess I hadn't thought about that very hard, I'll remember not to do that for "real" code :) Brad From brad.beveridge at gmail.com Tue Jan 22 17:00:56 2008 From: brad.beveridge at gmail.com (Brad Beveridge) Date: Tue, 22 Jan 2008 09:00:56 -0800 Subject: [rucksack-devel] Re: Error when deleting indexed objects In-Reply-To: References: Message-ID: On 22/01/2008, Arthur Lemmens wrote: > Brad Beveridge wrote: > > > I've made some progress. The problem is due to KEY-POSITION using > > P-EQL when trying to find the position of a key in the index. It > > really should be more like > > (defun key-position (btree key node) > > (p-position key (btree-node-index node) > > :key #'binding-key > > :test (btree-key= btree) > > :end (btree-node-index-count node))) > > Erm, yes. This particular problem was already fixed by Leonid > Novikov's patch that I committed to CVS last week. Oops :) Oh well, it was a fun learning experience for me to dig through Rucksack's code. From alemmens at xs4all.nl Tue Jan 22 17:09:50 2008 From: alemmens at xs4all.nl (Arthur Lemmens) Date: Tue, 22 Jan 2008 18:09:50 +0100 Subject: [rucksack-devel] State of the nation and heap patch In-Reply-To: <5bef28df0710311000u3e10bdbbn5cde1228d5b54ab5@mail.gmail.com> References: <5bef28df0710311000u3e10bdbbn5cde1228d5b54ab5@mail.gmail.com> Message-ID: Sean Ross wrote: > I was just wondering if anyone knows what the current state of the > rucksack world is I had practically no time to work on Rucksack last year. I hope (and expect) to have more time for Rucksack this year, but I can't make any promises of course. > I am quite keen to use it in a project of mine but am going to need > some help sorting out some corruption issues that I've run into (code > to reproduce can be found in corrupt.lisp). Thanks, I can confirm the problem and I'll look into it. > I've also attached a small patch to heap.lisp which amends load-buffer > to use (first (array-dimensions)) rather than length in order to > ignore the vectors fill-pointer (which was causing large amounts of > unnecessary arrays to be allocated). OK, thanks. I just committed this patch to CVS (as version 0.1.12). Arthur From alemmens at xs4all.nl Tue Jan 22 17:29:53 2008 From: alemmens at xs4all.nl (Arthur Lemmens) Date: Tue, 22 Jan 2008 18:29:53 +0100 Subject: [rucksack-devel] Re: Fwd: State of the nation and heap patch In-Reply-To: <5bef28df0711040820u76216a5bw50b1d771d18f7c12@mail.gmail.com> References: <5bef28df0710311000u3e10bdbbn5cde1228d5b54ab5@mail.gmail.com> <5bef28df0710311332p5039929eh98062094935997e5@mail.gmail.com> <5bef28df0711040820u76216a5bw50b1d771d18f7c12@mail.gmail.com> Message-ID: Sean Ross wrote: > The good news is that I believe I have found the cause of the index corruption. > It appears to be the way persistent objects are referenced in the > indexes. Seeing that they are not stored directly in the class and > slot indexes, but rather have their object-id saved instead, they are > never added to the heap roots during the scanning stage of gc, never > marked as :live-object's and end up being deallocated prematurely. Yes, that sounds like a good analysis. > I do have a fix (albeit simple) for this which merely stores the the > persistent-object as the index-value instead of the object-id. > Unfortunately this does mean that the :id-only argument to > rucksack-map-class and rucksack-map-index would have to be removed. That was just an (obviously premature) optimization hack anyway. I'll be happy to get rid of it in return for something that actually works. > don't know if that fits in with where you see rucksack going in the > future but if it is please let me know and I'll supply the patch. I'm definitely interested in the patch. Thanks in advance. > have to admit that after giving it some thought i could not think of a > situation where :id-only would be particularly useful My idea was that there might be situations where you want to do something with sets of objects without actually having to load them. But I'm not so sure that was a good idea, and I don't think it's used anywhere in the Rucksack code at the moment. > Ironically storing the object itself rather than the object-id has > also given a rather dramatic speed increase (a 50%+ reduction in time > taken) when using rucksack-map-slot Right, so my hack is like most 'optimizations': not only is it wrong, but it also makes things slower... Arthur From rosssd at gmail.com Thu Jan 24 11:21:05 2008 From: rosssd at gmail.com (Sean Ross) Date: Thu, 24 Jan 2008 11:21:05 +0000 Subject: [rucksack-devel] Re: Fwd: State of the nation and heap patch In-Reply-To: References: <5bef28df0710311000u3e10bdbbn5cde1228d5b54ab5@mail.gmail.com> <5bef28df0710311332p5039929eh98062094935997e5@mail.gmail.com> <5bef28df0711040820u76216a5bw50b1d771d18f7c12@mail.gmail.com> Message-ID: <5bef28df0801240321j18d97b80rea9c39be4431ca3c@mail.gmail.com> On 1/22/08, Arthur Lemmens wrote: > I'm definitely interested in the patch. Thanks in advance. Excellent, I'll dig it up. > > have to admit that after giving it some thought i could not think of a > > situation where :id-only would be particularly useful > > My idea was that there might be situations where you want to do > something with sets of objects without actually having to load them. > But I'm not so sure that was a good idea, and I don't think it's used > anywhere in the Rucksack code at the moment. Good point, I wonder if that can be implemented using something like a map-keys function on the class index, which would allow the :id-only keyword to be resurrected. sean. From alemmens at xs4all.nl Thu Jan 24 12:32:59 2008 From: alemmens at xs4all.nl (Arthur Lemmens) Date: Thu, 24 Jan 2008 13:32:59 +0100 Subject: [rucksack-devel] Re: Rucksack Unit tests In-Reply-To: References: Message-ID: Brad Beveridge wrote: > I have attached a tgz file that contains the beginnings of a basic > test suite for Rucksack. Thank you very much. I noticed you used FiveAM as the test framework. I want to minimize external dependencies for Rucksack, and I found FiveAM bigger and more complicated than I would like for Rucksack. So I looked around for something smaller and simpler; after reading http://aperiodic.net/phil/archives/Geekery/notes-on-lisp-testing-frameworks.html I decided to go for Chris Riesbeck's lisp-unit library. See http://www.cs.northwestern.edu/academics/courses/325/readings/lisp-unit.html. The lisp-unit library consists of a single file; I just added it to the CVS repository and the rucksack-test ASDF file, and I rewrote your tests to work with lisp-unit. As you mentioned in your file, your tests uncovered a few bugs and holes in Rucksack. I think these are all fixed now (except that P-REPLACE isn't implemented yet for persistent lists). Your code looked good to me, except for the BASIC-ROLLBACK test. Instead of calling TRANSACTION-ROLLBACK explicitly within a WITH-TRANSACTION form, you should just call ABORT or throw an error. WITH-TRANSACTION will ensure that TRANSACTION-ROLLBACK is called when its body wasn't completed normally. So your test now looks like this: (define-test basic-rollback (with-rucksack-and-transaction (:if-exists :supersede) (add-rucksack-root (p-cons 1 2) rs)) (with-rucksack-and-transaction () (let ((pc (first (rucksack-roots rs)))) (setf (p-car pc) 4) ;; Abort the transaction. WITH-TRANSACTION will take care ;; of calling TRANSACTION-ROLLBACK. (abort))) (with-rucksack-and-transaction () (let ((pc (first (rucksack-roots rs)))) (assert-equal 1 (p-car pc)))) ;; Test that transactions are also rolled back when we throw an ;; error inside the body of a WITH-TRANSACTION form. (assert-error 'error (with-rucksack-and-transaction () (let ((pc (first (rucksack-roots rs)))) (setf (p-car pc) 5) ;; Abort the transaction by causing an error. (error "Something went wrong")))) (with-rucksack-and-transaction () ;; Verify that the error caused a transaction rollback. (let ((pc (first (rucksack-roots rs)))) (assert-equal 1 (p-car pc))))) Arthur From alemmens at xs4all.nl Thu Jan 24 13:09:29 2008 From: alemmens at xs4all.nl (Arthur Lemmens) Date: Thu, 24 Jan 2008 14:09:29 +0100 Subject: [rucksack-devel] Re: Fwd: State of the nation and heap patch In-Reply-To: <5bef28df0801240321j18d97b80rea9c39be4431ca3c@mail.gmail.com> References: <5bef28df0710311000u3e10bdbbn5cde1228d5b54ab5@mail.gmail.com> <5bef28df0710311332p5039929eh98062094935997e5@mail.gmail.com> <5bef28df0711040820u76216a5bw50b1d771d18f7c12@mail.gmail.com> <5bef28df0801240321j18d97b80rea9c39be4431ca3c@mail.gmail.com> Message-ID: Sean Ross wrote: >> My idea was that there might be situations where you want to do >> something with sets of objects without actually having to load them. >> But I'm not so sure that was a good idea, and I don't think it's used >> anywhere in the Rucksack code at the moment. > > Good point, I wonder if that can be implemented using something like > a map-keys function on the class index, which would allow the :id-only > keyword to be resurrected. I'm afraid I don't understand this. Wouldn't such a MAP-KEYS function do basically the same thing as RUCKSACK-MAP-CLASS? Arthur From rosssd at gmail.com Thu Jan 24 13:21:09 2008 From: rosssd at gmail.com (Sean Ross) Date: Thu, 24 Jan 2008 13:21:09 +0000 Subject: [rucksack-devel] Re: Fwd: State of the nation and heap patch In-Reply-To: References: <5bef28df0710311000u3e10bdbbn5cde1228d5b54ab5@mail.gmail.com> <5bef28df0710311332p5039929eh98062094935997e5@mail.gmail.com> <5bef28df0711040820u76216a5bw50b1d771d18f7c12@mail.gmail.com> <5bef28df0801240321j18d97b80rea9c39be4431ca3c@mail.gmail.com> Message-ID: <5bef28df0801240521k2d90d3few2e5f8695096383be@mail.gmail.com> On 1/24/08, Arthur Lemmens wrote: > I'm afraid I don't understand this. Wouldn't such a MAP-KEYS function > do basically the same thing as RUCKSACK-MAP-CLASS? I think I explained this rather poorly. What I was thinking of was altering RUCKSACK-MAP-CLASS to accept the :id-only keyword again and subsequently using map-keys* (if such a thing exists) rather than map-index if ID-ONLY is true. sean * Where map-keys applies a single argument function to all the keys of an index. From alemmens at xs4all.nl Thu Jan 24 15:00:31 2008 From: alemmens at xs4all.nl (Arthur Lemmens) Date: Thu, 24 Jan 2008 16:00:31 +0100 Subject: [rucksack-devel] Re: Fwd: State of the nation and heap patch In-Reply-To: <5bef28df0801240521k2d90d3few2e5f8695096383be@mail.gmail.com> References: <5bef28df0710311000u3e10bdbbn5cde1228d5b54ab5@mail.gmail.com> <5bef28df0710311332p5039929eh98062094935997e5@mail.gmail.com> <5bef28df0711040820u76216a5bw50b1d771d18f7c12@mail.gmail.com> <5bef28df0801240321j18d97b80rea9c39be4431ca3c@mail.gmail.com> <5bef28df0801240521k2d90d3few2e5f8695096383be@mail.gmail.com> Message-ID: Sean Ross wrote: > I think I explained this rather poorly. > What I was thinking of was altering RUCKSACK-MAP-CLASS to accept the :id-only > keyword again and subsequently using map-keys* (if such a thing exists) rather > than map-index if ID-ONLY is true. > > * Where map-keys applies a single argument function to all the keys of an index. Ah, I see now. Yes, that sounds like a good idea to me. That way, we solve the GC roots problem with your patch and still keep the ID-ONLY option. Thanks for the suggestion, Arthur From brad.beveridge at gmail.com Thu Jan 31 03:16:41 2008 From: brad.beveridge at gmail.com (Brad Beveridge) Date: Wed, 30 Jan 2008 19:16:41 -0800 Subject: [rucksack-devel] import-export Message-ID: The DEFGENERIC form violates CLHS3.4.2 & SBCL whines about it. It should be: (defgeneric import-rucksack (pathname directory-designator &rest args &key if-exists &allow-other-keys) (:documentation "Creates a new rucksack in the directory specified by DIRECTORY-DESIGNATOR, opens the new rucksack and imports all objects that were exported to the file specified by PATHNAME.")) You cannot specify init forms for &key parameters in the DEFGENERIC. Cheers, Brad From alemmens at xs4all.nl Thu Jan 31 13:06:21 2008 From: alemmens at xs4all.nl (Arthur Lemmens) Date: Thu, 31 Jan 2008 14:06:21 +0100 Subject: [rucksack-devel] import-export In-Reply-To: References: Message-ID: Brad Beveridge wrote: > The DEFGENERIC form violates CLHS3.4.2 & SBCL whines about it. Ah yes, sorry about that. > It should be: > (defgeneric import-rucksack (pathname directory-designator > &rest args > &key if-exists > &allow-other-keys) Actually I changed it to a DEFUN, because there's nothing interesting to specialize on anyway. Should be fixed in CVS now. Thanks, Arthur From alemmens at xs4all.nl Thu Jan 31 20:32:00 2008 From: alemmens at xs4all.nl (Arthur Lemmens) Date: Thu, 31 Jan 2008 21:32:00 +0100 Subject: [rucksack-devel] State of the nation and heap patch In-Reply-To: References: <5bef28df0710311000u3e10bdbbn5cde1228d5b54ab5@mail.gmail.com> Message-ID: Sean Ross wrote: > I am quite keen to use it in a project of mine but am going to need > some help sorting out some corruption issues that I've run into (code > to reproduce can be found in corrupt.lisp). This problem should now be fixed in CVS (Rucksack version 0.1.14). I followed your suggestion and changed the class and slot indexes: they now map to objects instead of object ids, so the garbage collector will add the indexed objects to the root set. Thanks, Arthur