From hans.huebner at gmail.com Thu Feb 2 06:27:34 2006 From: hans.huebner at gmail.com (=?ISO-8859-1?Q?Hans_H=FCbner?=) Date: Thu, 2 Feb 2006 07:27:34 +0100 Subject: [bknr-devel] string multiple-slot indices In-Reply-To: References: <87y80wi9sf.fsf@web.de> Message-ID: Hi Kamen, we have used the array-index in an application that stored map locations referenced by a coordinate. The coordinate is stored in x and y slots of the objects, and we wanted to retrieve objects by coordinate, without writing a special purpose index for the application. The array-index class is only good to index objects with multiple slots that have integral values. There is nothing I can see that precludes other multiple-slot indices which would accept string values, but that would be more involved to implement. The dimensions of the index array are required to set up the array that holds the index. I hope that this answers your question. Cheers, Hans From troussan at gmail.com Fri Feb 3 15:21:16 2006 From: troussan at gmail.com (Tchadvar Roussanov) Date: Fri, 3 Feb 2006 09:21:16 -0600 Subject: [bknr-devel] SBCL port Message-ID: <168eba220602030721h4785b4e3p63860934657a093d@mail.gmail.com> Hi, I am trying to compile BKNR with the latest SBCL release. Till now I had to update almost all third party libraries to their latest versions. Is there are plans to update the versions of these packages in the SVN tree? Or move them out and use asdf install to resolve dependencies? I also notice in the SVN repository that there is ongoing SBCL porting effort. What is the current status of it? I am willing to help with this port as much as I can. Thank you. -------------- next part -------------- An HTML attachment was scrubbed... URL: From hans.huebner at gmail.com Fri Feb 3 18:17:28 2006 From: hans.huebner at gmail.com (=?ISO-8859-1?Q?Hans_H=FCbner?=) Date: Fri, 3 Feb 2006 19:17:28 +0100 Subject: [bknr-devel] SBCL port In-Reply-To: <168eba220602030721h4785b4e3p63860934657a093d@mail.gmail.com> References: <168eba220602030721h4785b4e3p63860934657a093d@mail.gmail.com> Message-ID: Hi Tchadvar, I am currently reorganizing the code base so that asdf-install can be used. The thirdparty/ tree will eventually go away, as soon as I have a stable asdf-installable setup. This will be the case until the end of this month. I am not using SBCL myself, so I have no current plans to port it myself. There are others who are interested in this, and in fact there have been two endavours to do so. Hoan Ton-That did a port last summer, but it failed to compile on cmucl so it did not make it back into the trunk. Nyef did a port to flexi-streams that is supposed to work with SBCL, but he changed the on-disk format which makes merging to the mainline a bit problematic. It would be great if there was a port to SBCL which does neither introduce incompatibilities to cmucl nor change the on-disk format (unless there are real good reasons to do so). I am currently using the BKNR Datastore as it is in the repository to maintain a handful of commercial projects, and I really depend on being able to commit my changes back and update the deployed systems. Also, I think that a stable on-disk format is a very important property of a database system, so I'd rather prefer to have it stay as it is or, if there are compelling reasons to do so, change the format only once in order to iron out any insufficiencies of the current format. What are your plans with the Datastore? Can you spend some time on making it more portable, and if so, how much? Do you have the necessary skills? I will try to answer any questions you may have and I'll also test changes with my deployed applications, but I will do so on cmucl only, for the lack of time. Cheers, Hans 2006/2/3, Tchadvar Roussanov : > Hi, > > I am trying to compile BKNR with the latest SBCL release. Till now I had to > update almost all > third party libraries to their latest versions. Is there are plans to > update the versions of these packages in the SVN tree? > Or move them out and use asdf install to resolve dependencies? > > I also notice in the SVN repository that there is ongoing SBCL porting > effort. What is the current status of it? > > I am willing to help with this port as much as I can. > > Thank you. > _______________________________________________ > bknr-devel mailing list > bknr-devel at common-lisp.net > http://common-lisp.net/cgi-bin/mailman/listinfo/bknr-devel > > > From troussan at gmail.com Sat Feb 4 06:46:55 2006 From: troussan at gmail.com (Tchadvar Roussanov) Date: Sat, 4 Feb 2006 00:46:55 -0600 Subject: [bknr-devel] SBCL port In-Reply-To: References: <168eba220602030721h4785b4e3p63860934657a093d@mail.gmail.com> Message-ID: <168eba220602032246k51ebdf95m57bf558810540a1e@mail.gmail.com> I can spend a couple of hours per day on porting. I already maneged to compile with sbcl 0.9.9. I am looking into the streams issues now. I will appreciate any hints on the testing. Especially on handling external formats and UNICODE related issues. Although at my current job I am forced to use Java, I would like to move away from it. In the past I have done some work with Lisp and Prolog (AI). And as you probably know, once you use these kind of languages it is difficult to like any other. Regards, Tchavdar On 2/3/06, Hans H?bner wrote: > > Hi Tchadvar, > > I am currently reorganizing the code base so that asdf-install can be > used. The thirdparty/ tree will eventually go away, as soon as I have > a stable asdf-installable setup. This will be the case until the end > of this month. > > I am not using SBCL myself, so I have no current plans to port it > myself. There are others who are interested in this, and in fact > there have been two endavours to do so. Hoan Ton-That did a port last > summer, but it failed to compile on cmucl so it did not make it back > into the trunk. Nyef did a port to flexi-streams that is supposed to > work with SBCL, but he changed the on-disk format which makes merging > to the mainline a bit problematic. > > It would be great if there was a port to SBCL which does neither > introduce incompatibilities to cmucl nor change the on-disk format > (unless there are real good reasons to do so). I am currently using > the BKNR Datastore as it is in the repository to maintain a handful of > commercial projects, and I really depend on being able to commit my > changes back and update the deployed systems. Also, I think that a > stable on-disk format is a very important property of a database > system, so I'd rather prefer to have it stay as it is or, if there are > compelling reasons to do so, change the format only once in order to > iron out any insufficiencies of the current format. > > What are your plans with the Datastore? Can you spend some time on > making it more portable, and if so, how much? Do you have the > necessary skills? > > I will try to answer any questions you may have and I'll also test > changes with my deployed applications, but I will do so on cmucl only, > for the lack of time. > > Cheers, > Hans > > 2006/2/3, Tchadvar Roussanov : > > Hi, > > > > I am trying to compile BKNR with the latest SBCL release. Till now I had > to > > update almost all > > third party libraries to their latest versions. Is there are plans to > > update the versions of these packages in the SVN tree? > > Or move them out and use asdf install to resolve dependencies? > > > > I also notice in the SVN repository that there is ongoing SBCL porting > > effort. What is the current status of it? > > > > I am willing to help with this port as much as I can. > > > > Thank you. > > _______________________________________________ > > bknr-devel mailing list > > bknr-devel at common-lisp.net > > http://common-lisp.net/cgi-bin/mailman/listinfo/bknr-devel > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hans.huebner at gmail.com Sat Feb 4 09:35:47 2006 From: hans.huebner at gmail.com (=?ISO-8859-1?Q?Hans_H=FCbner?=) Date: Sat, 4 Feb 2006 10:35:47 +0100 Subject: [bknr-devel] SBCL port In-Reply-To: <168eba220602032246k51ebdf95m57bf558810540a1e@mail.gmail.com> References: <168eba220602030721h4785b4e3p63860934657a093d@mail.gmail.com> <168eba220602032246k51ebdf95m57bf558810540a1e@mail.gmail.com> Message-ID: Hi Tchadvar, great news. Find nyefs datastore version which uses flexi-streams enclosed. I have not looked at it much, but it seems to contain good ideas (especially the define-object-codec macro looks cool). Maybe it helps having this. Generally, flexi-streams make sense if the encoding of the log file and transaction logs are a problem - I really never looked at this hard enough. My applications all use Unicode strings with cmucl being ignorant about it. This is no pretty solution and I thought about switching to sbcl for the Unicode support, but I never made this happen. Let me know if I can assist you further. Best regards, Hans -------------- next part -------------- A non-text attachment was scrubbed... Name: datastore-0.0.1-pre-1.tbz2 Type: application/octet-stream Size: 46066 bytes Desc: not available URL: From hoan at ton-that.org Mon Feb 6 11:42:03 2006 From: hoan at ton-that.org (Hoan Ton-That) Date: Mon, 6 Feb 2006 22:42:03 +1100 Subject: [bknr-devel] SBCL and OpenMCL ports done properly Message-ID: Hey everyone, I've ported the datastore to SBCL and OpenMCL properly. I'm sorry to everyone else who has been kept waiting. The format of the transaction log in UNCHANGED. I've tested loading, creating, and snapshotting across CMUCL, SBCL and OpenMCL with all primitive datatypes (array, integer, symbol, string, list, hash-table, single and double floats), as well as other compound objects. The patch is against the latest svn repository. Apologies again for this not being up sooner. Hans, if I were to change one thing about the current transaction log format, it would be the encoding of floats and double-floats. I think it would be better to use integer-decode-float (because it is in the hyperspec), than all these read-time conditionals. I couldn't find a `make-single-float' and `make-double-float' for OpenMCL. I can easily do this for you and I'll even write a converter if you wish. Hoan -------------- next part -------------- A non-text attachment was scrubbed... Name: datastore-openmcl-and-sbcl-port Type: application/octet-stream Size: 13835 bytes Desc: not available URL: From hans.huebner at gmail.com Mon Feb 6 13:54:08 2006 From: hans.huebner at gmail.com (=?ISO-8859-1?Q?Hans_H=FCbner?=) Date: Mon, 6 Feb 2006 14:54:08 +0100 Subject: [bknr-devel] SBCL and OpenMCL ports done properly In-Reply-To: References: Message-ID: Hi Hoan, I have read your patch and it looks fine so far. I will test it on cmucl later today, and if things go well, I'll commit it to the trunk. I have not made much use of floats, so I cannot comment on the current implementation much. I would also prefer not to have this workaround for OpenMCL, so if you could come up with a patch, I would propably add that as well. The new implementation for %encode-string looks fine, and as far as I can see it is more consistent with the %decode-string implementation. Thanks a lot! -Hans From hans.huebner at gmail.com Mon Feb 6 18:42:30 2006 From: hans.huebner at gmail.com (=?ISO-8859-1?Q?Hans_H=FCbner?=) Date: Mon, 6 Feb 2006 19:42:30 +0100 Subject: [bknr-devel] SBCL and OpenMCL ports done properly In-Reply-To: References: Message-ID: I did a quick check of the patches and they seem to work - At least on the first sight. I have commited them to the trunk, please test and report! Thanks a lot, Hoan! -Hans From hoan at ton-that.org Tue Feb 7 11:10:47 2006 From: hoan at ton-that.org (Hoan Ton-That) Date: Tue, 7 Feb 2006 22:10:47 +1100 Subject: [bknr-devel] SBCL and OpenMCL ports done properly In-Reply-To: References: Message-ID: Hey Hans, I'm glad it works, unlike last time! There are two things I learnt from this: 1. Test 2. Make small independent patches Hoan From hoan at ton-that.org Tue Feb 7 12:36:48 2006 From: hoan at ton-that.org (Hoan Ton-That) Date: Tue, 7 Feb 2006 23:36:48 +1100 Subject: [bknr-devel] SBCL and OpenMCL ports done properly In-Reply-To: References: Message-ID: Hans, Looking back, I'm not very happy with the code I wrote for the content-addressed blobs. So for now it is going to remain uncommitted. In the near future, I won't be working on the datastore, however there are some features I'd like to add in the future. It would be nice if the datastore was disk-based instead of memory-based. I think that is the most productive step forward. Hoan From hans.huebner at gmail.com Tue Feb 7 12:43:26 2006 From: hans.huebner at gmail.com (=?ISO-8859-1?Q?Hans_H=FCbner?=) Date: Tue, 7 Feb 2006 13:43:26 +0100 Subject: [bknr-devel] SBCL and OpenMCL ports done properly In-Reply-To: References: Message-ID: Hi Hoan, what do you mean by "disk based"? The most fundamental idea of the datastore is that it runs from main memory, not disk. This is more than just an accident or a deficiency, it is a feature. If what you want is a O/R-mapping database layer, BKNR may serve as a starting point on how to interface to CLOS, not much more. But maybe I am missing what you mean by "disk based". Can you explain? -Hans From hans.huebner at gmail.com Tue Feb 7 11:19:09 2006 From: hans.huebner at gmail.com (=?ISO-8859-1?Q?Hans_H=FCbner?=) Date: Tue, 7 Feb 2006 12:19:09 +0100 Subject: [bknr-devel] SBCL and OpenMCL ports done properly In-Reply-To: References: Message-ID: Hi Hoan, I will test your patches more intensively tonight, but I read the patch, it was short enough to completely understand and it did not do anything which was hard to understand. Great. I hope to be able to generate a few test cases which make it possible to do some unit testing on the store in the future. This supports what you wrote and I?ll freely admit that our source base should be more helpful. Do you have any plans for other changes? Last time, you had more (like content-addressed blobs) which I did not understand. Maybe it is a good idea to quickly discuss proposed changes before they get checked in. I do not have any current plans, but if I happen to have enough time, I will propably work on database maintenance and operations so that BKNR based systems can be run for a long time without much nursery. The snapshotting mechanism currently eats disk space without bounds, and the blob garbage collection is not automated. Cheers, Hans From hoan at ton-that.org Tue Feb 7 13:20:18 2006 From: hoan at ton-that.org (Hoan Ton-That) Date: Wed, 8 Feb 2006 00:20:18 +1100 Subject: [bknr-devel] SBCL and OpenMCL ports done properly In-Reply-To: References: Message-ID: Hey Hans, Ok, what I meant was something like AllegroStore. In AllegroStore, all objects are stored on disk and there is a memory cache. The idea is to still maintain the illusion that objects are in memory, but still provide high performance. Hoan PS. Its getting late over here in Australia... From hans.huebner at gmail.com Tue Feb 7 14:07:13 2006 From: hans.huebner at gmail.com (=?ISO-8859-1?Q?Hans_H=FCbner?=) Date: Tue, 7 Feb 2006 15:07:13 +0100 Subject: [bknr-devel] SBCL and OpenMCL ports done properly In-Reply-To: References: Message-ID: Hi Hoan, I don't really plan to make the BKNR Datastore compete with AllegroStore, AllegroCache, cl-versant, or any of the O/R mapping database products. There may be some overlap between such a project and the BKNR Datastore, but I don't expect it to be large. The requirements for a disk-based database are very different from those of in-memory, transaction based system. For example, with the BKNR Datastore you can make arbitary in-memory datastructures persistent by making all destructive operations be transactions. In a disk-based database system, the changes to the persistent data itself is being written to disk, which requires different mechanisms - In practice, you would propably restrict such a system to make CLOS objects persistent and hook into the slot access functions. I am not opposed to object database systems, but I want to see the BKNR Datastore developed towards better maintainability and reliability. Also, I fail to see what advantages a traditional, disk-based approach would have, despite the inherently lower performance. What is the reason that makes you think that the BKNR Datastore should move towards a disk-based system? Maybe I am overlooking something or fail to see where the current approach does not work. I see the most severe limitation of the current approach in it's size limit. A store, in practice, is limited to a few 100 MB of data. Beyond that, the restore times will be too high and also global garbage collections can become a problem, at least if the GC is very naive. In practice, a few 100 MB of data is a lot of data and will be enough for even very large applications. There are limits, but to me they are more theoretical. Check out some design scribbles I wrote a while ago: http://common-lisp.net/project/bknr/templates/development-style.xml and http://common-lisp.net/project/bknr/templates/why-no-db.xml Cheers, Hans From ktomov at web.de Tue Feb 7 22:54:07 2006 From: ktomov at web.de (Kamen TOMOV) Date: Wed, 08 Feb 2006 00:54:07 +0200 Subject: [bknr-devel] SBCL and OpenMCL ports done properly In-Reply-To: (Hans =?iso-8859-15?q?H=FCbner's?= message of "Tue, 7 Feb 2006 12:19:09 +0100") References: Message-ID: <87slquhjao.fsf@web.de> Hello Hans, Could you elaborate a bit more on these points: On Tue, Feb 07 2006, Hans H?bner wrote: > .. > I will propably work on database maintenance and operations so that > BKNR based systems can be run for a long time without much nursery. What kind of nursery do you mean? > The snapshotting mechanism currently eats disk space without bounds, It seems to me it eats as much as necessary - no more no less. Or am I missing something? > and the blob garbage collection is not automated. What exactly do you mean? If we delete blob wouldn't the memory be cleaned? Thanks. -- Kamen From hans.huebner at gmail.com Wed Feb 8 05:57:42 2006 From: hans.huebner at gmail.com (=?ISO-8859-1?Q?Hans_H=FCbner?=) Date: Wed, 8 Feb 2006 06:57:42 +0100 Subject: [bknr-devel] SBCL and OpenMCL ports done properly In-Reply-To: <87slquhjao.fsf@web.de> References: <87slquhjao.fsf@web.de> Message-ID: 2006/2/7, Kamen TOMOV : > On Tue, Feb 07 2006, Hans H?bner wrote: > > > .. > > I will propably work on database maintenance and operations so that > > BKNR based systems can be run for a long time without much nursery. > > What kind of nursery do you mean? > > > The snapshotting mechanism currently eats disk space without bounds, > > It seems to me it eats as much as necessary - no more no less. Or am I > missing something? If you regularily snapshot from a cron job, all snapshots will be kept on disk and need to be manually moved to offline media. Depending on the transaction rate, it may be neccessary to snapshot often, which will use a lot of disk space. It would be good if the Datastore would provide an automatic mechanism that purges older snapshots. This could be done on a simple (only keep N snapshots) or a more complex basis (keep the last three snapshots, one for each week, one for each month or some such). Also, a restore tool is needed that helps selecting a snapshot to restore. I have been bitten many times while selecting an older snapshot to restore and selecting a too-early one. Instead of manually having to select a certain snapshot, one should be able to specify a date and time (or manually created markpoint) to restore to and have the environment select the right snapshot file and rollforward location. It would also be good if a snapshot could be automatically created as soon as a certain restore time limit has been reached. This would require adding up the execution time of restored and newly executed transactions and scheduling a snapshot only if it is required. Together with the automatic selection of a snapshot to restore, this would make the database system much easier to use, and it would also help with the disk growth problem. > > and the blob garbage collection is not automated. > > What exactly do you mean? If we delete blob wouldn't the memory be > cleaned? Yes. Blob files cannot be deleted when the blob object is deleted because they may be needed again, should the user decide to restore an earlier snapshot. An integrated snapsht archival function would be able to move deleted blobs offline together with the corresponding snapshot file and transaction log. At the moment, there is the (bknr.datastore:delete-orphaned-blob-files) function which can be called to get rid of stray cats. Cheers, Hans From hans.huebner at gmail.com Wed Feb 8 06:08:02 2006 From: hans.huebner at gmail.com (=?ISO-8859-1?Q?Hans_H=FCbner?=) Date: Wed, 8 Feb 2006 07:08:02 +0100 Subject: [bknr-devel] SBCL and OpenMCL ports done properly In-Reply-To: <168eba220602072201ve0c8404qa43e3f4bd2785ae0@mail.gmail.com> References: <168eba220602072201ve0c8404qa43e3f4bd2785ae0@mail.gmail.com> Message-ID: Hi Tchadvar, thanks for the patch, it went right into trunk. -Hans From troussan at gmail.com Wed Feb 8 06:01:16 2006 From: troussan at gmail.com (Tchadvar Roussanov) Date: Wed, 8 Feb 2006 00:01:16 -0600 Subject: [bknr-devel] SBCL and OpenMCL ports done properly In-Reply-To: References: Message-ID: <168eba220602072201ve0c8404qa43e3f4bd2785ae0@mail.gmail.com> Hans, Here is a minor patch to make object-tests.lisp working with sbcl-0.9.9. It also includes sbcl changes to init.lisp for logical path translations (assuming installation in user's home directory). Note that in order for datastore to compile I have to replace cxml and porableaserve with their latest cvs versions. Tchavdar -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: sbcl-object-tests-port Type: application/octet-stream Size: 2607 bytes Desc: not available URL: From ktomov at web.de Wed Feb 8 11:04:47 2006 From: ktomov at web.de (Kamen TOMOV) Date: Wed, 08 Feb 2006 13:04:47 +0200 Subject: [bknr-devel] SBCL and OpenMCL ports done properly In-Reply-To: (Hans =?iso-8859-15?q?H=FCbner's?= message of "Wed, 8 Feb 2006 06:57:42 +0100") References: <87slquhjao.fsf@web.de> Message-ID: <87mzh2glgw.fsf@web.de> On Wed, Feb 08 2006, Hans H?bner wrote: > 2006/2/7, Kamen TOMOV : >> On Tue, Feb 07 2006, Hans H?bner wrote: >> >> > .. >> > I will propably work on database maintenance and operations so >> > that BKNR based systems can be run for a long time without much >> > nursery. >> >> What kind of nursery do you mean? >> >> > The snapshotting mechanism currently eats disk space without >> > bounds, >> >> It seems to me it eats as much as necessary - no more no less. Or >> am I missing something? > > If you regularily snapshot from a cron job, all snapshots will be > kept on disk and need to be manually moved to offline media. > Depending on the transaction rate, it may be neccessary to snapshot > often, which will use a lot of disk space. It would be good if the > Datastore would provide an automatic mechanism that purges older > snapshots. This could be done on a simple (only keep N snapshots) > or a more complex basis (keep the last three snapshots, one for each > week, one for each month or some such). > > Also, a restore tool is needed that helps selecting a snapshot to > restore. I have been bitten many times while selecting an older > snapshot to restore and selecting a too-early one. Instead of > manually having to select a certain snapshot, one should be able to > specify a date and time (or manually created markpoint) to restore > to and have the environment select the right snapshot file and > rollforward location. > > It would also be good if a snapshot could be automatically created > as soon as a certain restore time limit has been reached. This > would require adding up the execution time of restored and newly > executed transactions and scheduling a snapshot only if it is > required. Together with the automatic selection of a snapshot to > restore, this would make the database system much easier to use, and > it would also help with the disk growth problem. > I see. These would probably be useful tools. I was thinking about the general purpose of the snapshotting mechanism. It seems to me that it exists solely because the hardware can not meet the demands of the applications only in terms of store-objects restore times. For not so demanding applications I would rather do without snapshotting. The prevalence model allows tracking the history of the changes of the data. As a framework user I would rather not rely on snapshotting to allow changes of my store-object based objects. This is something that I would fix in the framework. In my opinion the main functionality must not depend on the snapshotting mechanism, only the other way around. >> > and the blob garbage collection is not automated. >> >> What exactly do you mean? If we delete blob wouldn't the memory be >> cleaned? > > Yes. Blob files cannot be deleted when the blob object is deleted > because they may be needed again, should the user decide to restore > an earlier snapshot. An integrated snapsht archival function would > be able to move deleted blobs offline together with the > corresponding snapshot file and transaction log. At the moment, > there is the (bknr.datastore:delete-orphaned-blob-files) function > which can be called to get rid of stray cats. I meant the memory of the blob object, not the file. I suppose this memory is garbage collected, right? As to the disk memory I was surprised to find out (some time ago) that the files are not deleted on deleting the blob object. Now I understand your reasons. The integrated snapshot archival function that you suggest sounds reasonable. Perhaps once we have it, we could consider deleting the associated file on blob deletion. Regards, -- Kamen From ktomov at web.de Thu Feb 9 10:03:49 2006 From: ktomov at web.de (Kamen TOMOV) Date: Thu, 09 Feb 2006 12:03:49 +0200 Subject: [bknr-devel] SBCL and OpenMCL ports done properly In-Reply-To: (Hans =?iso-8859-15?q?H=FCbner's?= message of "Thu, 9 Feb 2006 06:48:31 +0100") References: <87slquhjao.fsf@web.de> <87mzh2glgw.fsf@web.de> Message-ID: <87r76cq262.fsf@web.de> Hi Hans, On Thu, Feb 09 2006, Hans H?bner wrote: > This is how it works. If you never snapshot, the store will only > use the space occupied by the transaction log on disk. If you don?t > need persistent CLOS objects, you can also create stores without the > store-object-subsystem - Then you will not be able to snapshot at > all, because it is the subsystems that do the snapshotting. > Yeah, I know. This flexibility is great. But it would be even better if changes of the schema (store-objects) go into the transaction log as well. This way the transaction log mechanism would continue to work. What do you think about that? > Blob objects are not special in terms of normal memory management. > Their binary contents is not read into main memory by the Datastore, > so there are no special requirements or demands - Or am I missing > something? No, you aren't. Initially I did not understand and that's why I asked. Regards, -- Kamen From hans.huebner at gmail.com Thu Feb 9 10:48:02 2006 From: hans.huebner at gmail.com (=?ISO-8859-1?Q?Hans_H=FCbner?=) Date: Thu, 9 Feb 2006 11:48:02 +0100 Subject: [bknr-devel] SBCL and OpenMCL ports done properly In-Reply-To: <87r76cq262.fsf@web.de> References: <87slquhjao.fsf@web.de> <87mzh2glgw.fsf@web.de> <87r76cq262.fsf@web.de> Message-ID: Hi Kamen, 2006/2/9, Kamen TOMOV : > Yeah, I know. This flexibility is great. But it would be even better > if changes of the schema (store-objects) go into the transaction log > as well. This way the transaction log mechanism would continue to > work. What do you think about that? True - But the first thing that'd need to go into the transaction log would be the code changes. Especially when a transaction function is changed, the semantics of the operation changes and - if no snapshotting is used - this change needs to be present in the transaction log file in order to be able to successfully restore. We have been discussing this for quite a long time, but as our production systems use snapshots, we presently snapshot before we do code updates, so we don't see the lack of code change logging being a problem. Logging schema changes is in fact a higher order function. The store-object-subsystem would have to have a set of transactions that perform the schema changes. The general case, at least in my opinion, is code change logging. In the end, it would make sense to put all code into the repository and directly edit in memory instead of using external files. This is kind of the long-term goal, but we have not made much progress in that direction, sadly. Cheers, Hans From ktomov at web.de Thu Feb 9 11:30:59 2006 From: ktomov at web.de (Kamen TOMOV) Date: Thu, 09 Feb 2006 13:30:59 +0200 Subject: [bknr-devel] SBCL and OpenMCL ports done properly In-Reply-To: (Hans =?iso-8859-15?q?H=FCbner's?= message of "Thu, 9 Feb 2006 11:48:02 +0100") References: <87slquhjao.fsf@web.de> <87mzh2glgw.fsf@web.de> <87r76cq262.fsf@web.de> Message-ID: <87lkwkpy4s.fsf@web.de> On Thu, Feb 09 2006, Hans H?bner wrote: > In the end, it would make sense to put all code into the repository > and directly edit in memory instead of using external files. This > is kind of the long-term goal, but we have not made much progress in > that direction, sadly. That's a great idea. Perhaps it is a part of the more general concept that an application code should not reside in files but in a repository. The part of the task that tracks the changes (versions) of the code could probably be tied with the transaction log mechanism. Also if we look at it as a more general issue (important not only for bknr-datastore users/developpers), there is a chance that more people would get interested. Have you heard about any project that deals with that? Regards, -- Kamen From hans.huebner at gmail.com Thu Feb 9 05:48:31 2006 From: hans.huebner at gmail.com (=?ISO-8859-1?Q?Hans_H=FCbner?=) Date: Thu, 9 Feb 2006 06:48:31 +0100 Subject: [bknr-devel] SBCL and OpenMCL ports done properly In-Reply-To: <87mzh2glgw.fsf@web.de> References: <87slquhjao.fsf@web.de> <87mzh2glgw.fsf@web.de> Message-ID: Hi Kamen, 2006/2/8, Kamen TOMOV : > As a framework user I would rather not rely on snapshotting to allow > changes of my store-object based objects. This is something that I > would fix in the framework. > > In my opinion the main functionality must not depend on the > snapshotting mechanism, only the other way around. This is how it works. If you never snapshot, the store will only use the space occupied by the transaction log on disk. If you don?t need persistent CLOS objects, you can also create stores without the store-object-subsystem - Then you will not be able to snapshot at all, because it is the subsystems that do the snapshotting. > I meant the memory of the blob object, not the file. I suppose this > memory is garbage collected, right? Blob objects are not special in terms of normal memory management. Their binary contents is not read into main memory by the Datastore, so there are no special requirements or demands - Or am I missing something? Cheers, Hans From troussan at gmail.com Fri Feb 10 03:15:17 2006 From: troussan at gmail.com (Tchadvar Roussanov) Date: Thu, 9 Feb 2006 21:15:17 -0600 Subject: [bknr-devel] SBCL port samll fixes Message-ID: <168eba220602091915x678323f5mcbbcfd777bdbdc94@mail.gmail.com> Hans, This is a small patch for init.lisp and some fixes to make sbcl compiler happy with 'defconstant'. Tchavdar -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: sbcl-port-small-fixes Type: application/octet-stream Size: 2345 bytes Desc: not available URL: From troussan at gmail.com Tue Feb 14 04:04:54 2006 From: troussan at gmail.com (Tchadvar Roussanov) Date: Mon, 13 Feb 2006 22:04:54 -0600 Subject: [bknr-devel] SBCL port samll fixes In-Reply-To: References: <168eba220602091915x678323f5mcbbcfd777bdbdc94@mail.gmail.com> Message-ID: <168eba220602132004r60bb572cqf933d79284c0529c@mail.gmail.com> Hans, In order to compile bknr/src/web/handlers.lisp with the latest cxml and sbcl, I either have to apply the change from svn to with-element macro in cxml/xml/unparse.lisp: (defmacro with-element (qname &body body) ;; XXX Statt qname soll man in zukunft auch mal (lname prefix) angeben ;; koennen. Hat aber Zeit bis DOM 2. - (when (listp qname) - (destructuring-bind (n) qname - (setf qname n))) `(invoke-with-element (lambda () , at body) ,qname)) or change the handlers.lisp this way: (defmethod handle-object ((handler xml-object-list-handler) object req) - (cxml:with-element (xml-object-list-handler-toplevel-element-name handler) - (dolist (object (object-list-handler-get-objects handler object req)) - (object-list-handler-show-object-xml handler object req)))) + (let ((element-name (xml-object-list-handler-toplevel-element-name handler))) + (cxml:with-element element-name + (dolist (object (object-list-handler-get-objects handler object req)) + (object-list-handler-show-object-xml handler object req))))) In other words move the evaluation of xml-object-list-handler-toplevel-element-name out of the macro expansion. I think the two changes are equivalent but the second one has the advantage not to require to maintain changes to the third party library. Am I missing something? Tchavdar -------------- next part -------------- An HTML attachment was scrubbed... URL: From ktomov at web.de Thu Feb 23 21:23:49 2006 From: ktomov at web.de (Kamen TOMOV) Date: Thu, 23 Feb 2006 23:23:49 +0200 Subject: [bknr-devel] Warning: Encoding destroyed object Message-ID: <87vev5wyze.fsf@web.de> Hi, I get this warning every time I delete a blob from the store. Could you tell me why is that? Thank you. -- Kamen TOMOV From ktomov at web.de Sun Feb 26 17:14:19 2006 From: ktomov at web.de (Kamen TOMOV) Date: Sun, 26 Feb 2006 19:14:19 +0200 Subject: [bknr-devel] Warning: Encoding destroyed object In-Reply-To: (Hans =?iso-8859-15?q?H=FCbner's?= message of "Thu, 23 Feb 2006 22:57:44 +0100") References: <87vev5wyze.fsf@web.de> Message-ID: <878xryxct0.fsf@web.de> On Thu, Feb 23 2006, Hans H?bner wrote: > Hi, > > I cannot reproduce this, at least not from what you describe. The > warning is the result of a reference to a deleted object being > written to the transaction log, presumable referenced from a slot of > another object. Can you provide me with an isolated test case? Dear Hans, Sorry of not responding earlier. Here is an isolated test case: (asdf:oos 'asdf:load-op :bknr-impex) (asdf:oos 'asdf:load-op :bknr-datastore) (defpackage some-example (:use #:common-lisp #:bknr.impex #:puri #:bknr.indices #:bknr.datastore)) (in-package :some-example) (define-persistent-class obj (store-object) ()) (deftransaction del (obj) (delete-object obj)) (make-instance 'mp-store :directory "/tmp/tmp-store/" :subsystems (list (make-instance 'store-object-subsystem))) (del (make-object 'obj :name "a test" :type nil)) I noticed that if "del" is not a transaction it does not signal a warning. I am not sure why is so. Beside delete-object I would like to physically delete the file there. Thanks. Regards, -- Kamen TOMOV From hans.huebner at gmail.com Sun Feb 26 17:32:05 2006 From: hans.huebner at gmail.com (=?ISO-8859-1?Q?Hans_H=FCbner?=) Date: Sun, 26 Feb 2006 18:32:05 +0100 Subject: [bknr-devel] Warning: Encoding destroyed object In-Reply-To: <878xryxct0.fsf@web.de> References: <87vev5wyze.fsf@web.de> <878xryxct0.fsf@web.de> Message-ID: Hi Kamen, the bug that you describe is interesting. The following happens: Your transaction "del" is entered. "del" calls "delete-object" on the argument object. "delete-object" marks the object as deleted. "delete-object" is exited and a log entry is written to the transaction log. "del" is exited and a log entry is written. During the writing of the log entry, the datastore notices that the argument object has been deleted and issues the warning you have seen. You can avoid the warning by calling "destroy-object" instead of "delete-object". "delete-object" is a transaction which is meant to be called from non-transaction code, whereas "destroy-object" is a standard function which should be called only from transaction code. I am not sure whether we should keep it this way. On the positive side, a programmer does need to be aware of whether he writes code that executes within our outside of a transaction, so having to seperate functions here makes some sense. On the negative side, it is harder to remember what API to use, and in fact this is a problem with "setf slot-value" vs. "change-slot-values". For the time being, I'd recommend that you use "destroy-object" from your transaction code. Also, note that if you delete your blob file at when you delete an object, you loose the ability to roll forward to a point back in time, should you need to do so. Deleting blob files is best deferred to a point where old snapshots are archived using the function "(bknr.datastore::delete-orphaned-blob-files)". Cheers, Hans From hans.huebner at gmail.com Mon Feb 27 09:14:12 2006 From: hans.huebner at gmail.com (=?ISO-8859-1?Q?Hans_H=FCbner?=) Date: Mon, 27 Feb 2006 10:14:12 +0100 Subject: [bknr-devel] Warning: Encoding destroyed object In-Reply-To: References: <87vev5wyze.fsf@web.de> <878xryxct0.fsf@web.de> Message-ID: Hi Kamen, my analysis of your problem was not complete: In fact, the problem is a bit different from what I wrote: If you pass persistent objects as argument to a transaction and you delete that object within the transaction, serializing the transaction arguments to the transaction log result in the warning that you have seen. The datastore emits a warning because the object is no longer in existence after the transaction has been executed. Upon restore, this will not be a problem since no log entry is written, but the warning is annoying nevertheless. My suggestion, using "destroy-object" from your transaction, does not help, because you still have the persistent object in the argument list to the transaction function. A proper workaround is to pass only the object ID in the argument list to the transaction (this is how delete-object works). Another workaround is to remove the warning from the datastore. Maybe I will do that in the future, but I have to think about it some more. My remark about deleting blob files still holds, though. -Hans From ktomov at web.de Tue Feb 28 00:50:20 2006 From: ktomov at web.de (Kamen TOMOV) Date: Tue, 28 Feb 2006 02:50:20 +0200 Subject: [bknr-devel] Warning: Encoding destroyed object In-Reply-To: (Hans =?iso-8859-15?q?H=FCbner's?= message of "Mon, 27 Feb 2006 10:14:12 +0100") References: <87vev5wyze.fsf@web.de> <878xryxct0.fsf@web.de> Message-ID: <878xrw71df.fsf@web.de> Dear Hans, Thank you very much for your instant answer to the issue and for the suggested workarounds. I added a wrapper method around the del transaction that sends the ID of the object as a parameter and now it is even better. Thanks for the suggestion about the deletion of the blob files. I will reconsider using this function. Initially it didn't seem a good idea but I am starting to think that it is. As to the issues that you have put in your previous message I am not sure yet what the right approach should be. I'd trust your judgement. Regards, -- Kamen