From vsedach at gmail.com  Wed Dec  7 03:36:04 2011
From: vsedach at gmail.com (Vladimir Sedach)
Date: Tue, 6 Dec 2011 22:36:04 -0500
Subject: [bknr-devel] Deftransaction and &rest args
Message-ID: <CAD_S9D-mnwLonjsPriCgVNzjGMoiK87EjYLu+uxUX2yrQpFdmg@mail.gmail.com>

Hello,

Deftransaction permits &rest args, but they don't seem to be recorded
in the store. Is the correct behavior that &rest args should not be
valid in a deftransaction lambda-list, or is this a bug in storing
them?

Thanks,
Vladimir



From hans.huebner at gmail.com  Wed Dec  7 08:33:15 2011
From: hans.huebner at gmail.com (=?ISO-8859-1?Q?Hans_H=FCbner?=)
Date: Wed, 7 Dec 2011 09:33:15 +0100
Subject: [bknr-devel] BKNR Porter stemmer
In-Reply-To: <CAD_S9D_9oJ0j46ePT1zscuvv46sczSQPdhZH-OO8oXbMMTARTw@mail.gmail.com>
References: <CAD_S9D_9oJ0j46ePT1zscuvv46sczSQPdhZH-OO8oXbMMTARTw@mail.gmail.com>
Message-ID: <CABj0bw+7hh2yB7cc2xg0Tg+emy0+jOkNY3-2cbL5kVV3fZ0mqA@mail.gmail.com>

Hi Vladimir,

sorry for not getting back earlier about this.  I have extracted the
library from the BKNR subversion repository (which is in archival
mode) to https://github.com/hanshuebner/stem - I'm going to open an
issue with quicklisp so that the origin is changed.

Thanks for the patch!
Hans

On Tue, Nov 29, 2011 at 6:12 PM, Vladimir Sedach <vsedach at gmail.com> wrote:
> The Porter stemmer from the BKNR repository is the one that comes with
> quicklisp. I noticed a bug when stemming the word "ion" due to the
> special case for that suffix. Attached is a patch. This bug isn't
> present in the code on Porter's own site
> (http://tartarus.org/martin/PorterStemmer/commonlisp.txt)
>
> Happy hacking,
> Vladimir
>
> _______________________________________________
> bknr-devel mailing list
> bknr-devel at common-lisp.net
> http://lists.common-lisp.net/cgi-bin/mailman/listinfo/bknr-devel
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ion.patch
Type: application/octet-stream
Size: 719 bytes
Desc: not available
URL: <https://mailman.common-lisp.net/pipermail/bknr-devel/attachments/20111207/ccdd9196/attachment.obj>

From hans.huebner at gmail.com  Wed Dec  7 08:40:10 2011
From: hans.huebner at gmail.com (=?ISO-8859-1?Q?Hans_H=FCbner?=)
Date: Wed, 7 Dec 2011 09:40:10 +0100
Subject: [bknr-devel] Deftransaction and &rest args
In-Reply-To: <CAD_S9D-mnwLonjsPriCgVNzjGMoiK87EjYLu+uxUX2yrQpFdmg@mail.gmail.com>
References: <CAD_S9D-mnwLonjsPriCgVNzjGMoiK87EjYLu+uxUX2yrQpFdmg@mail.gmail.com>
Message-ID: <CABj0bwKXsq6u2q4ozP2GwhdRn60BbgpCcBu6YhUTg5Kghzw_uQ@mail.gmail.com>

Hi Vladimir,

I have looked at the source of deftransaction and the method
definitions for encode-object and decode-object for transactions - The
&rest arguments seem to be passed to the transaction object as :args
initarg, stored in the transaction file in the encoder and also read
back in the decoder.  So, the intention is that they are recorded.
Did you see that not work?

-Hans

On Wed, Dec 7, 2011 at 4:36 AM, Vladimir Sedach <vsedach at gmail.com> wrote:
> Hello,
>
> Deftransaction permits &rest args, but they don't seem to be recorded
> in the store. Is the correct behavior that &rest args should not be
> valid in a deftransaction lambda-list, or is this a bug in storing
> them?
>
> Thanks,
> Vladimir
>
> _______________________________________________
> bknr-devel mailing list
> bknr-devel at common-lisp.net
> http://lists.common-lisp.net/cgi-bin/mailman/listinfo/bknr-devel



From hans.huebner at gmail.com  Wed Dec  7 08:46:54 2011
From: hans.huebner at gmail.com (=?ISO-8859-1?Q?Hans_H=FCbner?=)
Date: Wed, 7 Dec 2011 09:46:54 +0100
Subject: [bknr-devel] BKNR.datastore: support for
 serializing/deserializing arrays with complex (list) element-types
In-Reply-To: <BANLkTik1A61OQzWZBewGezgif7VEp8tKNA@mail.gmail.com>
References: <BANLkTineL1LvegsiN6iLC-Eh2wXtTZE_2g@mail.gmail.com>
	<BANLkTimBtxSR_59E31FMGaQH6=vPKaD7=A@mail.gmail.com>
	<BANLkTik1A61OQzWZBewGezgif7VEp8tKNA@mail.gmail.com>
Message-ID: <CABj0bwJLPeQ_2bwL=_Q6Wk4=_pqP3UNqp1bjCkebZ2vEp5B1cQ@mail.gmail.com>

Hi Vladimir,

I know that it has been a long time since you posted this patch and I
did not comment, but here is my reply anyway:

On Fri, Apr 8, 2011 at 11:17 PM, Vladimir Sedach <vsedach at gmail.com> wrote:
> Here is the updated patch with backwards compatibility (something
> which I always forget!). Unfortunately it is a bit ugly and I don't
> know how to make it better. I guess one lesson here is to add tags to
> all binary data if you're designing a new serialization scheme.

We already have tags for all data types.  The patch that you've
submitted thus is not really adding the feature at the proper level.
If you are still interested, I would propose that you fix the problem
by adding a new pair of encoding functions for typed arrays
(%encode-typed-array and %decode-typed-array), a new type tag for that
(#\t) and code in encode-array to decide whether an untyped or a typed
array should be written (although I suppose you could always write
typed arrays, too).

Again, sorry for the long delay.  I'm only slowly picking up the work
that I've been deferring for too long.

-Hans



From hans.huebner at gmail.com  Wed Dec  7 08:59:49 2011
From: hans.huebner at gmail.com (=?ISO-8859-1?Q?Hans_H=FCbner?=)
Date: Wed, 7 Dec 2011 09:59:49 +0100
Subject: [bknr-devel] Fork changes merged to
	git@github.com:hanshuebner/bknr-devel
Message-ID: <CABj0bwJKUtAuNCrkP5WAh27hVa6qYWRPPh4aV7Ey=EF9ziR=QQ@mail.gmail.com>

Hi,

I have finally merged the changes from the two other forks
(madnificient's and fsmunoz's) into master in my github repostory.
Please let me know if there are any problems.

-Hans



From vsedach at gmail.com  Sat Dec 10 01:37:12 2011
From: vsedach at gmail.com (Vladimir Sedach)
Date: Fri, 9 Dec 2011 20:37:12 -0500
Subject: [bknr-devel] BKNR.datastore: support for
 serializing/deserializing arrays with complex (list) element-types
In-Reply-To: <CABj0bwJLPeQ_2bwL=_Q6Wk4=_pqP3UNqp1bjCkebZ2vEp5B1cQ@mail.gmail.com>
References: <BANLkTineL1LvegsiN6iLC-Eh2wXtTZE_2g@mail.gmail.com>
	<BANLkTimBtxSR_59E31FMGaQH6=vPKaD7=A@mail.gmail.com>
	<BANLkTik1A61OQzWZBewGezgif7VEp8tKNA@mail.gmail.com>
	<CABj0bwJLPeQ_2bwL=_Q6Wk4=_pqP3UNqp1bjCkebZ2vEp5B1cQ@mail.gmail.com>
Message-ID: <CAD_S9D-sfa1vhbFgHTyTWGL_U9WkzaR3V5WbFybNeaJvobJYog@mail.gmail.com>

Hi Hans,

I am indeed not interested anymore, but thank you for the feedback. I
didn't think of adding a "typed array" type, that is a better
solution. In any case, if someone wants this, they'll hopefully come
across this email thread and figure out what to do.

Vladimir

On Wed, Dec 7, 2011 at 3:46 AM, Hans H?bner <hans.huebner at gmail.com> wrote:
> Hi Vladimir,
>
> I know that it has been a long time since you posted this patch and I
> did not comment, but here is my reply anyway:
>
> On Fri, Apr 8, 2011 at 11:17 PM, Vladimir Sedach <vsedach at gmail.com> wrote:
>> Here is the updated patch with backwards compatibility (something
>> which I always forget!). Unfortunately it is a bit ugly and I don't
>> know how to make it better. I guess one lesson here is to add tags to
>> all binary data if you're designing a new serialization scheme.
>
> We already have tags for all data types. ?The patch that you've
> submitted thus is not really adding the feature at the proper level.
> If you are still interested, I would propose that you fix the problem
> by adding a new pair of encoding functions for typed arrays
> (%encode-typed-array and %decode-typed-array), a new type tag for that
> (#\t) and code in encode-array to decide whether an untyped or a typed
> array should be written (although I suppose you could always write
> typed arrays, too).
>
> Again, sorry for the long delay. ?I'm only slowly picking up the work
> that I've been deferring for too long.
>
> -Hans



From vsedach at gmail.com  Sat Dec 10 22:52:07 2011
From: vsedach at gmail.com (Vladimir Sedach)
Date: Sat, 10 Dec 2011 17:52:07 -0500
Subject: [bknr-devel] Deftransaction and &rest args
In-Reply-To: <CABj0bwKXsq6u2q4ozP2GwhdRn60BbgpCcBu6YhUTg5Kghzw_uQ@mail.gmail.com>
References: <CAD_S9D-mnwLonjsPriCgVNzjGMoiK87EjYLu+uxUX2yrQpFdmg@mail.gmail.com>
	<CABj0bwKXsq6u2q4ozP2GwhdRn60BbgpCcBu6YhUTg5Kghzw_uQ@mail.gmail.com>
Message-ID: <CAD_S9D-Nu5gupzkho==CiM-r8=xWZH9C+_fBe6Jvz-vkMhed1A@mail.gmail.com>

Hi Hans,

This is the smallest error scenario I have come up with:

DATASTORE> (open-store "/home/viper/tmp/test-store/")
initializing store random state
restoring #<STORE DIR: "/home/viper/tmp/test-store/">
#<STORE DIR: "/home/viper/tmp/test-store/">
DATASTORE> (defclass test-object (store-object)
             ((test-slot :initform '(1 2 3)))
             (:metaclass persistent-class))
#<PERSISTENT-CLASS TEST-OBJECT>
DATASTORE> (slot-value (make-instance 'test-object) 'test-slot)
(1 2 3)
DATASTORE> (deftransaction set-test-slot (object &rest args)
             (setf (slot-value object 'test-slot) args))
SET-TEST-SLOT
DATASTORE> (slot-value (car (store-objects-with-class 'test-object)) 'test-slot)
(1 2 3)
DATASTORE> (set-test-slot (car (store-objects-with-class 'test-object)) 3 4 5)
NIL
DATASTORE> (slot-value (car (store-objects-with-class 'test-object)) 'test-slot)
NIL

Actually, looking in txn.lisp (line 422):
,@(let ((rest (member '&rest args)))
    (when rest `((declare (ignore ,(second rest))))))

And line 427 when making the transaction, make-args is called during
macroexpansion, which does the following:

(make-args '(foo &rest bar)) => (FOO)

This is why I assumed that &rest args were left out for a reason but
warnings/documentation was just forgotten.

Vladimir

On Wed, Dec 7, 2011 at 3:40 AM, Hans H?bner <hans.huebner at gmail.com> wrote:
> Hi Vladimir,
>
> I have looked at the source of deftransaction and the method
> definitions for encode-object and decode-object for transactions - The
> &rest arguments seem to be passed to the transaction object as :args
> initarg, stored in the transaction file in the encoder and also read
> back in the decoder. ?So, the intention is that they are recorded.
> Did you see that not work?
>
> -Hans
>
> On Wed, Dec 7, 2011 at 4:36 AM, Vladimir Sedach <vsedach at gmail.com> wrote:
>> Hello,
>>
>> Deftransaction permits &rest args, but they don't seem to be recorded
>> in the store. Is the correct behavior that &rest args should not be
>> valid in a deftransaction lambda-list, or is this a bug in storing
>> them?
>>
>> Thanks,
>> Vladimir
>>
>> _______________________________________________
>> bknr-devel mailing list
>> bknr-devel at common-lisp.net
>> http://lists.common-lisp.net/cgi-bin/mailman/listinfo/bknr-devel



From hans.huebner at gmail.com  Tue Dec 13 10:18:14 2011
From: hans.huebner at gmail.com (=?ISO-8859-1?Q?Hans_H=FCbner?=)
Date: Tue, 13 Dec 2011 10:18:14 +0000
Subject: [bknr-devel] Deftransaction and &rest args
In-Reply-To: <CAD_S9D-Nu5gupzkho==CiM-r8=xWZH9C+_fBe6Jvz-vkMhed1A@mail.gmail.com>
References: <CAD_S9D-mnwLonjsPriCgVNzjGMoiK87EjYLu+uxUX2yrQpFdmg@mail.gmail.com>
	<CABj0bwKXsq6u2q4ozP2GwhdRn60BbgpCcBu6YhUTg5Kghzw_uQ@mail.gmail.com>
	<CAD_S9D-Nu5gupzkho==CiM-r8=xWZH9C+_fBe6Jvz-vkMhed1A@mail.gmail.com>
Message-ID: <CABj0bw+4uZ=UhXq4YWhZOPQzavLYEXnb-tpGCZk9A6n5aZpUNw@mail.gmail.com>

Hi Vladimir,

thank you for reporting the deftransaction bug, which is a genuine
bug.  I think it was introduced while Kilian worked on the code and
did not have tests to verify his changes.  I'll take care of the
problem.

Cheers,
Hans

On Sat, Dec 10, 2011 at 10:52 PM, Vladimir Sedach <vsedach at gmail.com> wrote:
> Hi Hans,
>
> This is the smallest error scenario I have come up with:
>
> DATASTORE> (open-store "/home/viper/tmp/test-store/")
> initializing store random state
> restoring #<STORE DIR: "/home/viper/tmp/test-store/">
> #<STORE DIR: "/home/viper/tmp/test-store/">
> DATASTORE> (defclass test-object (store-object)
> ? ? ? ? ? ? ((test-slot :initform '(1 2 3)))
> ? ? ? ? ? ? (:metaclass persistent-class))
> #<PERSISTENT-CLASS TEST-OBJECT>
> DATASTORE> (slot-value (make-instance 'test-object) 'test-slot)
> (1 2 3)
> DATASTORE> (deftransaction set-test-slot (object &rest args)
> ? ? ? ? ? ? (setf (slot-value object 'test-slot) args))
> SET-TEST-SLOT
> DATASTORE> (slot-value (car (store-objects-with-class 'test-object)) 'test-slot)
> (1 2 3)
> DATASTORE> (set-test-slot (car (store-objects-with-class 'test-object)) 3 4 5)
> NIL
> DATASTORE> (slot-value (car (store-objects-with-class 'test-object)) 'test-slot)
> NIL
>
> Actually, looking in txn.lisp (line 422):
> ,@(let ((rest (member '&rest args)))
> ? ?(when rest `((declare (ignore ,(second rest))))))
>
> And line 427 when making the transaction, make-args is called during
> macroexpansion, which does the following:
>
> (make-args '(foo &rest bar)) => (FOO)
>
> This is why I assumed that &rest args were left out for a reason but
> warnings/documentation was just forgotten.
>
> Vladimir
>
> On Wed, Dec 7, 2011 at 3:40 AM, Hans H?bner <hans.huebner at gmail.com> wrote:
>> Hi Vladimir,
>>
>> I have looked at the source of deftransaction and the method
>> definitions for encode-object and decode-object for transactions - The
>> &rest arguments seem to be passed to the transaction object as :args
>> initarg, stored in the transaction file in the encoder and also read
>> back in the decoder. ?So, the intention is that they are recorded.
>> Did you see that not work?
>>
>> -Hans
>>
>> On Wed, Dec 7, 2011 at 4:36 AM, Vladimir Sedach <vsedach at gmail.com> wrote:
>>> Hello,
>>>
>>> Deftransaction permits &rest args, but they don't seem to be recorded
>>> in the store. Is the correct behavior that &rest args should not be
>>> valid in a deftransaction lambda-list, or is this a bug in storing
>>> them?
>>>
>>> Thanks,
>>> Vladimir
>>>
>>> _______________________________________________
>>> bknr-devel mailing list
>>> bknr-devel at common-lisp.net
>>> http://lists.common-lisp.net/cgi-bin/mailman/listinfo/bknr-devel



From hans.huebner at gmail.com  Tue Dec 13 10:23:28 2011
From: hans.huebner at gmail.com (=?ISO-8859-1?Q?Hans_H=FCbner?=)
Date: Tue, 13 Dec 2011 10:23:28 +0000
Subject: [bknr-devel] Deftransaction and &rest args
In-Reply-To: <CABj0bw+4uZ=UhXq4YWhZOPQzavLYEXnb-tpGCZk9A6n5aZpUNw@mail.gmail.com>
References: <CAD_S9D-mnwLonjsPriCgVNzjGMoiK87EjYLu+uxUX2yrQpFdmg@mail.gmail.com>
	<CABj0bwKXsq6u2q4ozP2GwhdRn60BbgpCcBu6YhUTg5Kghzw_uQ@mail.gmail.com>
	<CAD_S9D-Nu5gupzkho==CiM-r8=xWZH9C+_fBe6Jvz-vkMhed1A@mail.gmail.com>
	<CABj0bw+4uZ=UhXq4YWhZOPQzavLYEXnb-tpGCZk9A6n5aZpUNw@mail.gmail.com>
Message-ID: <CABj0bwLqsPahEh+j0xeCLH32Srpmv3oo5rba8=aEFiYybtR3Kg@mail.gmail.com>

On Tue, Dec 13, 2011 at 10:18 AM, Hans H?bner <hans.huebner at gmail.com> wrote:
> thank you for reporting the deftransaction bug, which is a genuine
> bug. ?I think it was introduced while Kilian worked on the code and
> did not have tests to verify his changes.

I take this back.  The bug has been in much longer, Kilian's commit
was just a whitespace change.  Will fix anyway.

-Hans



From vsedach at gmail.com  Tue Dec 27 22:04:39 2011
From: vsedach at gmail.com (Vladimir Sedach)
Date: Tue, 27 Dec 2011 17:04:39 -0500
Subject: [bknr-devel] BKNR.datastore snapshot frequency and old snapshots
Message-ID: <CAD_S9D-uFCObYzZSJ7Be3V5e1bqXn5x=NQ7+M6SvWtjBWRpH-w@mail.gmail.com>

Hello,

How often should a BKNR datastore be snapshot? I tried daily, but that
is clearly too much. Is there a criteria (# of transaction since last
snapshot) that can be used to determine this programmatically?

Related to this, are snapshots linked to each other? Are old ones
unneeded? Is it ok to delete them if they are?

Vladimir



From hans.huebner at gmail.com  Fri Dec 30 18:22:21 2011
From: hans.huebner at gmail.com (=?ISO-8859-1?Q?Hans_H=FCbner?=)
Date: Fri, 30 Dec 2011 19:22:21 +0100
Subject: [bknr-devel] BKNR.datastore snapshot frequency and old snapshots
In-Reply-To: <CAD_S9D-uFCObYzZSJ7Be3V5e1bqXn5x=NQ7+M6SvWtjBWRpH-w@mail.gmail.com>
References: <CAD_S9D-uFCObYzZSJ7Be3V5e1bqXn5x=NQ7+M6SvWtjBWRpH-w@mail.gmail.com>
Message-ID: <CABj0bwLdGtDh+spVA02aK2tZEAL7L76em8-jkybKULSdcY1Rrg@mail.gmail.com>

Hi Vladimir,

On Tue, Dec 27, 2011 at 11:04 PM, Vladimir Sedach <vsedach at gmail.com> wrote:
> How often should a BKNR datastore be snapshot? I tried daily, but that
> is clearly too much.

In my deployed application, I snapshot once per day.  This has not
bothered me beyond the disk space usage, but eventually I came up with
a cron job to automatically purge old snapshots.

> Is there a criteria (# of transaction since last
> snapshot) that can be used to determine this programmatically?

The store itself has no mechanisms regarding that.  I have pondered
over various ideas how one could schedule snapshots in a more
intelligent fashion - One was to measure and accumulate transaction
execution time and snapshot whenever the execution time would exceed
some threshold.  Nothing like that ever went into code, though, as the
nightly scheme never caused any problems for me.  After all, even with
my largest application that has some 700k objects, a restart only
takes a minute or two, and happens only every few months when I need
to do some maintenance.

> Related to this, are snapshots linked to each other? Are old ones
> unneeded? Is it ok to delete them if they are?

They are completely independent and can be deleted at will.  Only the
current/ and blob-root/ directories are needed.  Blobs are not
automatically garbage collected, though, as they could be referenced
in older snapshots and the store has no information regarding such
references.  The function bknr.datastore::delete-orphaned-blob-files
can be used to remove all blob files that are not currently referenced
from the store.

Let me know if you need further information.

Cheers,
Hans