From kilian.sprotte at googlemail.com  Sat Feb 16 14:19:17 2008
From: kilian.sprotte at googlemail.com (Kilian Sprotte)
Date: Sat, 16 Feb 2008 15:19:17 +0100
Subject: [bknr-devel] Re: [bknr-cvs] r2514 - branches/bos/modules/mail
In-Reply-To: <20080216140642.D832C5C185@common-lisp.net>
References: <20080216140642.D832C5C185@common-lisp.net>
Message-ID: <1d26dc7e0802160619x17c42787p74d8b301bd3360b@mail.gmail.com>

Hallo Hans,

ich habe hier zuf?llig ein kleines Problem entdeckt.
regex-replace-all hatte nur zwei Argumente - das kann
auf jeden Fall nicht richtig sein. Ich habe ein bisschen geraten
und #?/\1/ erg?nzt (wie da dr?ber). Vielleicht k?nntest Du das
noch mal ?berpr?fen.

Weiterhin noch ein sch?nes WE,
  Kilian

PS: Komischerweise hat  "mein cmucl" (19c) das schon beim compilen
entdeckt:

Error while parsing arguments to DEFINE-COMPILER-MACRO
CL-PPCRE:REGEX-REPLACE-ALL:
Invalid number of elements in:
  (BKNR.MAIL::*MESSAGE-ID-RE*
   (FIRST
    (IF (BKNR.MAIL::HEADER :IN-REPLY-TO)
        (CL-PPCRE:SPLIT "\\s+" (BKNR.MAIL::HEADER :IN-REPLY-TO))
        (LAST (CL-PPCRE:SPLIT "\\s+" (BKNR.MAIL::HEADER :REFERENCES))))))
to satisfy lambda-list:
  (&WHOLE CL-PPCRE::FORM &ENVIRONMENT CL-PPCRE::ENV CL-PPCRE::REGEX
   CL-PPCRE::TARGET-STRING CL-PPCRE::REPLACEMENT &REST REST)


On Feb 16, 2008 3:06 PM,  <ksprotte at common-lisp.net> wrote:
> Author: ksprotte
> Date: Sat Feb 16 09:06:42 2008
> New Revision: 2514
>
> Modified:
>    branches/bos/modules/mail/mail.lisp
> Log:
> fix for regex-replace-all in mail.lisp
> third arg was missing
>
>
> Modified: branches/bos/modules/mail/mail.lisp
> ==============================================================================
> --- branches/bos/modules/mail/mail.lisp (original)
> +++ branches/bos/modules/mail/mail.lisp Sat Feb 16 09:06:42 2008
> @@ -97,7 +97,8 @@
>                                 *message-id-re*
>                                 (first (if (header :in-reply-to)
>                                            (split #?/\s+/ (header :in-reply-to))
> -                                          (last (split #?/\s+/ (header :references))))))
> +                                          (last (split #?/\s+/ (header :references)))))
> +                                #?/\1/)
>                      :headers headers
>                      :body body)))))
>
> _______________________________________________
> bknr-cvs mailing list
> bknr-cvs at common-lisp.net
> http://common-lisp.net/cgi-bin/mailman/listinfo/bknr-cvs
>


From hans at huebner.org  Wed Feb 20 09:39:22 2008
From: hans at huebner.org (=?ISO-8859-1?Q?Hans_H=FCbner?=)
Date: Wed, 20 Feb 2008 10:39:22 +0100
Subject: [bknr-devel] BKNR SVN down for move and maintenance
Message-ID: <fd2982320802200139m7444e4c9lc39165cbeb772150@mail.gmail.com>

Folks,

I am going to move the BKNR repository off the current location to a
new machine that will serve as main development server in the next few
months.  The repository URL will change, but other than that you'll
not notice much for the moment.  I will send an email with updated
access information later today.

-Hans


From hans at huebner.org  Wed Feb 20 12:10:13 2008
From: hans at huebner.org (=?ISO-8859-1?Q?Hans_H=FCbner?=)
Date: Wed, 20 Feb 2008 13:10:13 +0100
Subject: [bknr-devel] Re: BKNR SVN down for move and maintenance
In-Reply-To: <fd2982320802200139m7444e4c9lc39165cbeb772150@mail.gmail.com>
References: <fd2982320802200139m7444e4c9lc39165cbeb772150@mail.gmail.com>
Message-ID: <fd2982320802200410m4c515a9fl4ade94cd4e5fc7e4@mail.gmail.com>

Hi,

I have moved the repository to a new machine.  If you want to convert
your checkout, go to your checkout directory and execute this:

perl -pi -e 's,common-lisp.net/project/bknr,svn.bknr.net,' `find .
-name entries`

This will do in-place editing of the URLs in subversions'
administrative "entries" file.

The new location of the repository is svn+ssh://svn.bknr.net/svn - I
have created users for Kilian and Didier and added the ssh public key
I found on common-lisp.net.  If anyone else reading this needs write
access, get in touch.

Sorry for the short notice.  Expect more updates on http://bknr.net/ soon.

-Hans

2008/2/20, Hans H?bner <hans at huebner.org>:
> Folks,
>
> I am going to move the BKNR repository off the current location to a
> new machine that will serve as main development server in the next few
> months.  The repository URL will change, but other than that you'll
> not notice much for the moment.  I will send an email with updated
> access information later today.
>
> -Hans
>


From UngerKlaus at gmx.de  Thu Feb 21 02:15:56 2008
From: UngerKlaus at gmx.de (Klaus Unger)
Date: Thu, 21 Feb 2008 03:15:56 +0100
Subject: [bknr-devel] bknr.indices, the slot NIL is missing from the object
Message-ID: <200802210315.56844.UngerKlaus@gmx.de>

Hi,

when using class indices they do not have a slot name. On 
destruction/index-remove this throws a condition on checking slot-boundp.

I am using SBCL 1.0.14.23 and the latest bknr indices code, only indices, 
nothing else.

Best regards
 Klaus

;Package changed to BKNR.INDICES
#<PACKAGE "BKNR.INDICES">

BKNR.INDICES>
(defvar *class-index* (index-create 'class-index :index-subclasses t))

*CLASS-INDEX*

BKNR.INDICES>
(defclass base-object ()
  ()
  (:metaclass indexed-class)
  (:class-indices (class :index *class-index*
    :slots nil
    :index-reader objects-of-class
    :index-values all-objects
    :index-subclasses t
    :index-keys all-class-names)
  (classes :index-type class-index
    :index-initargs (:index-superclasses t)
    :slots nil
    :index-subclasses t
    :index-reader objects-with-class)))

#<INDEXED-CLASS BASE-OBJECT>
; in: LAMBDA NIL
;     (SB-KERNEL:FLOAT-WAIT)
; 
; note: deleting unreachable code
; 
; compilation unit finished
;   printed 1 note

BKNR.INDICES>
(defclass child1 (base-object)
  ()
  (:metaclass indexed-class))

#<INDEXED-CLASS CHILD1>

BKNR.INDICES>
(defclass child2 (base-object)
  ((a :initarg :a))
  (:metaclass indexed-class))

#<INDEXED-CLASS CHILD2>

BKNR.INDICES>
(make-instance 'child1)

#<CHILD1 {AC66381}>

BKNR.INDICES>
(all-objects)

(#<CHILD1 {AC66381}>)

BKNR.INDICES>
(destroy-object (first (all-objects)))

When attempting to test to see whether slot is bound (SLOT-BOUNDP),
the slot NIL is missing from the object #<CHILD1 {A7422D9}>.
   [Condition of type SIMPLE-ERROR]
	0: [ABORT] Return to SLIME's top level.
	1: [TERMINATE-THREAD] Terminate this thread (#<THREAD "repl-thread" 
{B54A251}>)
]> Quit debug


From hans at huebner.org  Thu Feb 21 06:26:02 2008
From: hans at huebner.org (=?ISO-8859-1?Q?Hans_H=FCbner?=)
Date: Thu, 21 Feb 2008 07:26:02 +0100
Subject: [bknr-devel] bknr.indices, the slot NIL is missing from the object
In-Reply-To: <200802210315.56844.UngerKlaus@gmx.de>
References: <200802210315.56844.UngerKlaus@gmx.de>
Message-ID: <fd2982320802202226n2349f8edt53bf337aeb4802b7@mail.gmail.com>

Hi Klaus,

the class-index needs to know a slot that it uses for indexing the
objects.  This slot is the unique ID that the index works on.  It must
be added to your base class, but as you can see in the corrected
example below, that should not be much of a problem.

Note that the :INDICES option in an INDEXED-CLASS either accepts a
predefined index or initargs to create a fresh index.  It is confusing
to mix both initargs and the :INDEX option (which specifies the
preexisting index *CLASS-INDEX* to be used).  I added code to detect
this case.  More error checking would certainly be helpful, so please
report problems when you encounter them and I'll try to make the code
generate better diagnostics.

Please let us know if you have further questions.

Hans

(in-package :bknr.indices)

(defvar *class-index*  (index-create 'class-index :index-subclasses t
:slot-name 'id))
(defvar *object-id* 0)

(defclass base-object ()
 ((id :initform (incf *object-id*)))
 (:metaclass indexed-class)
 (:class-indices (class :index *class-index*
                        :index-reader objects-of-class
                        :index-values all-objects
                        :index-subclasses t
                        :index-keys all-class-names)
                 (classes :index-type class-index
                          :index-initargs (:index-superclasses t)
                          :slots (id)
                          :index-subclasses t
                          :index-reader objects-with-class)))

(defclass child1 (base-object)
 ()
 (:metaclass indexed-class))

(defclass child2 (base-object)
 ((a :initarg :a))
 (:metaclass indexed-class))

(defmacro with-tracing (&rest forms)
  `(progn ,@(mapcar (lambda (form) `(format *trace-output* "~A =>
~A~%" (quote ,form) ,form)) forms)))

(with-tracing
 (make-instance 'child1)
 (all-objects)
 (destroy-object (first (all-objects)))

 (make-instance 'child1)
 (make-instance 'child2)
 (make-instance 'child1)
 (all-class-names)
 (all-objects)
 (objects-with-class 'child1)
 (objects-with-class 'child2))

2008/2/21, Klaus Unger <UngerKlaus at gmx.de>:
> Hi,
>
>  when using class indices they do not have a slot name. On
>  destruction/index-remove this throws a condition on checking slot-boundp.
>
>  I am using SBCL 1.0.14.23 and the latest bknr indices code, only indices,
>  nothing else.
>
>  Best regards
>   Klaus
>
>  ;Package changed to BKNR.INDICES
>  #<PACKAGE "BKNR.INDICES">
>
>  BKNR.INDICES>
>  (defvar *class-index* (index-create 'class-index :index-subclasses t))
>
>  *CLASS-INDEX*
>
>  BKNR.INDICES>
>  (defclass base-object ()
>   ()
>   (:metaclass indexed-class)
>   (:class-indices (class :index *class-index*
>     :slots nil
>     :index-reader objects-of-class
>     :index-values all-objects
>     :index-subclasses t
>     :index-keys all-class-names)
>   (classes :index-type class-index
>     :index-initargs (:index-superclasses t)
>     :slots nil
>     :index-subclasses t
>     :index-reader objects-with-class)))
>
>  #<INDEXED-CLASS BASE-OBJECT>
>  ; in: LAMBDA NIL
>  ;     (SB-KERNEL:FLOAT-WAIT)
>  ;
>  ; note: deleting unreachable code
>  ;
>  ; compilation unit finished
>  ;   printed 1 note
>
>  BKNR.INDICES>
>  (defclass child1 (base-object)
>   ()
>   (:metaclass indexed-class))
>
>  #<INDEXED-CLASS CHILD1>
>
>  BKNR.INDICES>
>  (defclass child2 (base-object)
>   ((a :initarg :a))
>   (:metaclass indexed-class))
>
>  #<INDEXED-CLASS CHILD2>
>
>  BKNR.INDICES>
>  (make-instance 'child1)
>
>  #<CHILD1 {AC66381}>
>
>  BKNR.INDICES>
>  (all-objects)
>
>  (#<CHILD1 {AC66381}>)
>
>  BKNR.INDICES>
>  (destroy-object (first (all-objects)))
>
>  When attempting to test to see whether slot is bound (SLOT-BOUNDP),
>  the slot NIL is missing from the object #<CHILD1 {A7422D9}>.
>    [Condition of type SIMPLE-ERROR]
>         0: [ABORT] Return to SLIME's top level.
>         1: [TERMINATE-THREAD] Terminate this thread (#<THREAD "repl-thread"
>  {B54A251}>)
>  ]> Quit debug
>  _______________________________________________
>  bknr-devel mailing list
>  bknr-devel at common-lisp.net
>  http://common-lisp.net/cgi-bin/mailman/listinfo/bknr-devel
>


From UngerKlaus at gmx.de  Thu Feb 21 18:55:38 2008
From: UngerKlaus at gmx.de (Klaus Unger)
Date: Thu, 21 Feb 2008 19:55:38 +0100
Subject: [bknr-devel] bknr.indices, the slot NIL is missing from the object
In-Reply-To: <fd2982320802202226n2349f8edt53bf337aeb4802b7@mail.gmail.com>
References: <200802210315.56844.UngerKlaus@gmx.de>
	<fd2982320802202226n2349f8edt53bf337aeb4802b7@mail.gmail.com>
Message-ID: <200802211955.38685.UngerKlaus@gmx.de>

Hi Hans,

thanks for the promt response! The code I used was from the datastore-manual 
page 23. When adding the id slot name I run into the following problem: I use 
cl-prevalence for serialization. When restoring id-objects I do this 
without-auto-id, so the objects are created having the id slot unbound and 
assigned later on. This leads to duplicate entries in the class-index and 
thus inconsitency. I was not sure what happens if there are multiple objects 
having a unique-index slot set to Nil.

Instead of optionaly binding the id parameter in the constructor I now use an 
initform that at least binds the auto-id or Nil. Is this a good idea?:

Best regards
  Klaus

(defvar *id-counter* 0)
(defvar *auto-id* T)

(defmacro without-auto-id (&body body)
  `(let ((*auto-id* Nil))
     , at body))

(defun next-id ()
  (when *auto-id*
      (incf *id-counter*)))

(defclass id-object ()
    ((id
      :reader get-id
      :initform (next-id)
      :index-type unique-index
      :index-initargs (:test #'eql)
      :index-values find-all-id-objects
      :index-reader find-object-with-id))
  (:metaclass indexed-class)
  (:class-indices
    (classes :index-type class-index
                 :index-initargs (:index-superclasses t)
                 :slots (id)
                 :index-reader find-all-id-objects-with-class)))

Am Donnerstag, 21. Februar 2008 07:26:02 schrieb Hans H?bner:
> Hi Klaus,
>
> the class-index needs to know a slot that it uses for indexing the
> objects.  This slot is the unique ID that the index works on.  It must
> be added to your base class, but as you can see in the corrected
> example below, that should not be much of a problem.
>
> Note that the :INDICES option in an INDEXED-CLASS either accepts a
> predefined index or initargs to create a fresh index.  It is confusing
> to mix both initargs and the :INDEX option (which specifies the
> preexisting index *CLASS-INDEX* to be used).  I added code to detect
> this case.  More error checking would certainly be helpful, so please
> report problems when you encounter them and I'll try to make the code
> generate better diagnostics.
>
> Please let us know if you have further questions.
>
> Hans



From hans at huebner.org  Thu Feb 21 20:20:05 2008
From: hans at huebner.org (=?ISO-8859-1?Q?Hans_H=FCbner?=)
Date: Thu, 21 Feb 2008 21:20:05 +0100
Subject: [bknr-devel] bknr.indices, the slot NIL is missing from the object
In-Reply-To: <200802211955.38685.UngerKlaus@gmx.de>
References: <200802210315.56844.UngerKlaus@gmx.de>
	<fd2982320802202226n2349f8edt53bf337aeb4802b7@mail.gmail.com>
	<200802211955.38685.UngerKlaus@gmx.de>
Message-ID: <fd2982320802211220yf9dcf0co2c39438a05c665ad@mail.gmail.com>

Hi Klaus,

thank you for the manual bug report.  It will be fixed in the upcoming
release of BKNR that is planned for summer.

Your approach to handling the problem looks fine, at least from
superficially looking at it.  I have not looked at cl-prevalence for
quite some time, so I can't say if it will work.

Out of curiosity:  Why are you using cl-prevalence instead of BKNR
datastore?  I am happy to see someone being actually able to use the
indices without the store, but I'm also interested in knowing what our
store is lacking to make you chose another, similar solution.

Thanks,
Hans

P.S.: In fact, we started with cl-prevalence but found it to be too
limited in several aspects. That was why we started writing BKNR
datastore.

2008/2/21, Klaus Unger <UngerKlaus at gmx.de>:
> Hi Hans,
>
>  thanks for the promt response! The code I used was from the datastore-manual
>  page 23. When adding the id slot name I run into the following problem: I use
>  cl-prevalence for serialization. When restoring id-objects I do this
>  without-auto-id, so the objects are created having the id slot unbound and
>  assigned later on. This leads to duplicate entries in the class-index and
>  thus inconsitency. I was not sure what happens if there are multiple objects
>  having a unique-index slot set to Nil.
>
>  Instead of optionaly binding the id parameter in the constructor I now use an
>  initform that at least binds the auto-id or Nil. Is this a good idea?:
>
>  Best regards
>   Klaus
>
>  (defvar *id-counter* 0)
>  (defvar *auto-id* T)
>
>  (defmacro without-auto-id (&body body)
>   `(let ((*auto-id* Nil))
>      , at body))
>
>  (defun next-id ()
>   (when *auto-id*
>       (incf *id-counter*)))
>
>  (defclass id-object ()
>     ((id
>       :reader get-id
>       :initform (next-id)
>       :index-type unique-index
>       :index-initargs (:test #'eql)
>       :index-values find-all-id-objects
>       :index-reader find-object-with-id))
>
>   (:metaclass indexed-class)
>   (:class-indices
>
>     (classes :index-type class-index
>                  :index-initargs (:index-superclasses t)
>                  :slots (id)
>
>                  :index-reader find-all-id-objects-with-class)))
>
>  Am Donnerstag, 21. Februar 2008 07:26:02 schrieb Hans H?bner:
>
> > Hi Klaus,
>  >
>  > the class-index needs to know a slot that it uses for indexing the
>  > objects.  This slot is the unique ID that the index works on.  It must
>  > be added to your base class, but as you can see in the corrected
>  > example below, that should not be much of a problem.
>  >
>  > Note that the :INDICES option in an INDEXED-CLASS either accepts a
>  > predefined index or initargs to create a fresh index.  It is confusing
>  > to mix both initargs and the :INDEX option (which specifies the
>  > preexisting index *CLASS-INDEX* to be used).  I added code to detect
>  > this case.  More error checking would certainly be helpful, so please
>  > report problems when you encounter them and I'll try to make the code
>  > generate better diagnostics.
>  >
>  > Please let us know if you have further questions.
>  >
>  > Hans
>
>


From UngerKlaus at gmx.de  Fri Feb 22 03:13:52 2008
From: UngerKlaus at gmx.de (Klaus Unger)
Date: Fri, 22 Feb 2008 04:13:52 +0100
Subject: [bknr-devel] bknr.indices, the slot NIL is missing from the object
In-Reply-To: <fd2982320802211220yf9dcf0co2c39438a05c665ad@mail.gmail.com>
References: <200802210315.56844.UngerKlaus@gmx.de>
	<200802211955.38685.UngerKlaus@gmx.de>
	<fd2982320802211220yf9dcf0co2c39438a05c665ad@mail.gmail.com>
Message-ID: <200802220413.52221.UngerKlaus@gmx.de>

Hi Hans,

I am using only serialization and indices, not transactions. I didn't manage 
to easily use the serialization of bknr-datastore in a lightweight manner, so 
I used the serialization of cl-prevalence.

When I started the project I had the feeling that transactions did not really 
give me benefits (please tell me If I got anything wrong):
- It is hard to map complex generic functions to transactions
- A rollback costs to much performance to be used for anything except very 
rare exceptions
- There are no thread isolation mechanisms except a global lock. (I can hardly 
imagine anything else for the used model.)

Now I implemented some features (store-object) of the datastore myself. So I 
would like to give the datastore itsself another chance and see if its 
possible to use it without much overhead due to transactions.  It is great to 
hear that there will be a new major release soon.

- Klaus

> Out of curiosity:  Why are you using cl-prevalence instead of BKNR
> datastore?  I am happy to see someone being actually able to use the
> indices without the store, but I'm also interested in knowing what our
> store is lacking to make you chose another, similar solution.
>
> Thanks,
> Hans
>
> P.S.: In fact, we started with cl-prevalence but found it to be too
> limited in several aspects. That was why we started writing BKNR
> datastore.



From hans at huebner.org  Fri Feb 22 15:18:34 2008
From: hans at huebner.org (=?ISO-8859-1?Q?Hans_H=FCbner?=)
Date: Fri, 22 Feb 2008 16:18:34 +0100
Subject: Fwd: [bknr-devel] bknr.indices,
	the slot NIL is missing from the object
In-Reply-To: <fd2982320802220717p6d6d1245q62459294e21d5e25@mail.gmail.com>
References: <200802210315.56844.UngerKlaus@gmx.de>
	<200802211955.38685.UngerKlaus@gmx.de>
	<fd2982320802211220yf9dcf0co2c39438a05c665ad@mail.gmail.com>
	<200802220413.52221.UngerKlaus@gmx.de>
	<fd2982320802212115ob03b10ahd68aae36c610528e@mail.gmail.com>
	<fd2982320802220717p6d6d1245q62459294e21d5e25@mail.gmail.com>
Message-ID: <fd2982320802220718w2456ed9dt6952b19ed3b069b7@mail.gmail.com>

[resending to bknr-devel]

Hi Klaus,


 2008/2/22, Hans H?bner <hans at huebner.org>:

>  2008/2/22, Klaus Unger <UngerKlaus at gmx.de>:

>  >  - It is hard to map complex generic functions to transactions


I have given this more thought, and it appears that it would be
 possible to replace the existing macro based transaction execution
 environment by generic functions with a custom method combination.
 This certainly would make the BKNR code much prettier and simpler.

 The question is:  What would it buy the user?  Certainly, it would
 make it possible to dispatch on argument types to run different code
 for different types.  I am not sure that this would be so very useful,
 though.  Transactions are not your average function, you are not going
 to invoke a large number of transaction one after the other to
 fullfill a user request or command.  Rather, one user command
 typically results in one transaction being executed, and in turn
 triggers a number of function calls from within the transaction body
 to accomplish whatever is needed.  Thus, transactions are generally
 more coarse grained than functions.

 Now, one could say that every function possibly is a transaction.  The
 transaction monitor would only log the top level transaction
 invocation and process invocations of other transactions as standard
 function calls.  This is what we currently do, albeit in a rather
 awkward and ugly form.

 Still, I do not think that all functions should be transactions.
 Rather, transaction define the mutation points of an application,
 which makes them special and requires that they are well defined.

 Now, this all does not mean that the transaction interface is perfect.
  There are two additions that I would consider worthwhile.  One would
 be the possibility to specify action code to run within the global
 lock, but before actually mutating the persistent state of the system.
  This would be useful to check preconditions in a safe manner on
 preemptive multitasking systems.  The other would be the possibility
 to declare an undo action which would help the application to roll
 back the persistent state to an earlier point in time.

 Implementing transactions as generic functions would give us argument
 dispatch for free, which would certainly be nice.  I managed to get
 away without being able to dispatch on transaction argument types or
 values so far, but that does not mean so much.

 Before going ahead with any of this, I would like to know what made
 you want generic functions as transactions.  It could be that I'm
 missing something.

 Your comments are greatly appreciated.  Thanks.


 -Hans


From UngerKlaus at gmx.de  Fri Feb 22 20:50:21 2008
From: UngerKlaus at gmx.de (Klaus Unger)
Date: Fri, 22 Feb 2008 21:50:21 +0100
Subject: [bknr-devel] bknr.indices, the slot NIL is missing from the object
In-Reply-To: <fd2982320802220718w2456ed9dt6952b19ed3b069b7@mail.gmail.com>
References: <200802210315.56844.UngerKlaus@gmx.de>
	<fd2982320802220717p6d6d1245q62459294e21d5e25@mail.gmail.com>
	<fd2982320802220718w2456ed9dt6952b19ed3b069b7@mail.gmail.com>
Message-ID: <200802222150.22035.UngerKlaus@gmx.de>

Hi Hans,

I am currently doing all non IO work of a user command in a single generic 
function. This method is protected by a single global lock :around it. So I 
am using objects for application modules that handle user commands. OOP 
allows me to abstract common aspects of the modules.

I don't see the advantage of using a big transaction rather than a big lock. 
This big lock approach is very easy to grasp but It has two performance 
drawbacks:
 - It doesn't scale to multicores
 - When some commands require a long time, short commands suffer from high
   latency
TBH I don't think these affect me much practically!

I think to solve the performance problems fine grain locks are used, they 
allow parallel execution but introduce those nasty problems like deadlocks.

Transactions then are introduced to lift the burden to think about the 
problems of fine grained locks and instead just define an "atomic" sequence 
of code. (In my opinion most database transactions still require too much 
thinking, for example whether to use "FOR UPDATE".)

Originaly I wanted to define the transactions at a more fine-grained level to 
avoid the performance problems. Due to the transaction implementation, it 
would only have bought me better latency, not scalability.
At this level I would have needed generic transactions on the OOP "business 
level".

I have been thinking about transactions in the sense of Software transactional 
memory (STM). They would give a performance benefit, even for the coarse 
grain transactions. There are a lot of recent papers about efficient 
algorithms for STM, but I have not yet to seen an implementation in CL. While 
it would be feasible for Objects with AMOP, it seems very hard to track 
access to cons.

Thanks for the detailed responses!
- Klaus


P.S. I also think that databases are nowadays clumsy for most applications. I 
have made bad experience with OO -> DB mappings, especially JPA.

Am Freitag, 22. Februar 2008 16:18:34 schrieb Hans H?bner:
> [resending to bknr-devel]
>
> I have given this more thought, and it appears that it would be
>  possible to replace the existing macro based transaction execution
>  environment by generic functions with a custom method combination.
>  This certainly would make the BKNR code much prettier and simpler.
>
>  The question is:  What would it buy the user?  Certainly, it would
>  make it possible to dispatch on argument types to run different code
>  for different types.  I am not sure that this would be so very useful,
>  though.  Transactions are not your average function, you are not going
>  to invoke a large number of transaction one after the other to
>  fullfill a user request or command.  
>
>  Now, one could say that every function possibly is a transaction.  The
>  transaction monitor would only log the top level transaction
>  invocation and process invocations of other transactions as standard
>  function calls.  This is what we currently do, albeit in a rather
>  awkward and ugly form.
>
>  Still, I do not think that all functions should be transactions.
>  Rather, transaction define the mutation points of an application,
>  which makes them special and requires that they are well defined.
>
>  Now, this all does not mean that the transaction interface is perfect.
>   There are two additions that I would consider worthwhile.  One would
>  be the possibility to specify action code to run within the global
>  lock, but before actually mutating the persistent state of the system.
>   This would be useful to check preconditions in a safe manner on
>  preemptive multitasking systems.  The other would be the possibility
>  to declare an undo action which would help the application to roll
>  back the persistent state to an earlier point in time.
>
>  Implementing transactions as generic functions would give us argument
>  dispatch for free, which would certainly be nice.  I managed to get
>  away without being able to dispatch on transaction argument types or
>  values so far, but that does not mean so much.
>

>
>  Your comments are greatly appreciated.  Thanks.
>
>
>  -Hans
> _______________________________________________
> bknr-devel mailing list
> bknr-devel at common-lisp.net
> http://common-lisp.net/cgi-bin/mailman/listinfo/bknr-devel


From hans at huebner.org  Fri Feb 22 21:16:57 2008
From: hans at huebner.org (=?ISO-8859-1?Q?Hans_H=FCbner?=)
Date: Fri, 22 Feb 2008 22:16:57 +0100
Subject: [bknr-devel] bknr.indices, the slot NIL is missing from the object
In-Reply-To: <200802222150.22035.UngerKlaus@gmx.de>
References: <200802210315.56844.UngerKlaus@gmx.de>
	<fd2982320802220717p6d6d1245q62459294e21d5e25@mail.gmail.com>
	<fd2982320802220718w2456ed9dt6952b19ed3b069b7@mail.gmail.com>
	<200802222150.22035.UngerKlaus@gmx.de>
Message-ID: <fd2982320802221316r54ab89gca207d958d9b522e@mail.gmail.com>

Hi Klaus,

well, so what you seem to write is that your application is
architected around a generic function that serializes mutations, and
that not having BKNR transactions support generic functions prevented
you from just using them rather than implementing your own lock.  Fair
enough.  I wonder why you can't use multiple generic functions instead
of one, but that is beyond what I can (and want to) understand right
now.

I am not a big fan of fine grained locking, but that is apparent from
the store architecture with its giant lock, too.  I am not bothered
with the scalability limitations as I have not yet found them to be
practically relevant - My user base has been growing slower than
processor speeds have increased, and even with Pentium III CPUs we had
more than enough headroom for our applications.

STM is interesting, but real transactional memory interests me more, I
must admit.  There is an implementation of CLOS STM
(http://common-lisp.net/project/cl-stm/), but the lack of support for
non-CLOS data types is kind of a show-stopper for me.  Maybe someone
will hack STM into one of the CL compilers, but I'm not really
prepared to do that.

Anyway - If you need further support with BKNR indices, let us know.

-Hans


From UngerKlaus at gmx.de  Sat Feb 23 03:40:58 2008
From: UngerKlaus at gmx.de (Klaus Unger)
Date: Sat, 23 Feb 2008 04:40:58 +0100
Subject: [bknr-devel] bknr.indices, the slot NIL is missing from the object
In-Reply-To: <fd2982320802221316r54ab89gca207d958d9b522e@mail.gmail.com>
References: <200802210315.56844.UngerKlaus@gmx.de>
	<200802222150.22035.UngerKlaus@gmx.de>
	<fd2982320802221316r54ab89gca207d958d9b522e@mail.gmail.com>
Message-ID: <200802230440.58527.UngerKlaus@gmx.de>

Hi Hans,

> well, so what you seem to write is that your application is
> architected around a generic function that serializes mutations, and
> that not having BKNR transactions support generic functions prevented
> you from just using them rather than implementing your own lock.  Fair
> enough.  I wonder why you can't use multiple generic functions instead
> of one, but that is beyond what I can (and want to) understand right
> now.
The generic function issue is just an additional "cost" for using 
transactions. The main point is that I see no benefit from transactions.
I'm sorry I didn't made that clear earlier. (I don't want to claim there is 
none, maybe I just don't see it or it is related to my scenario.)

> STM is interesting, but real transactional memory interests me more, I
> must admit.  There is an implementation of CLOS STM
> (http://common-lisp.net/project/cl-stm/), but the lack of support for
> non-CLOS data types is kind of a show-stopper for me.  Maybe someone
> will hack STM into one of the CL compilers, but I'm not really
> prepared to do that.
Thanks for the hint, though I have to agree that non-CLOS is essential.


> Anyway - If you need further support with BKNR indices, let us know.

Is it somehow possible with the datastore to run different stores in one lisp 
process? For example to run multiple instances of the same web-application on 
the same port on the same machine.


Besides the problems mentioned I have to add that I am really happy with the 
concept of indices and the relief from traditional DBMs and ugly DB -> OO 
mappings misleadingly claiming to be transparent OO -> DB mappings. I gladly 
pay the scalability price for that, and I'm looking forward to gain more 
experience with that paradigm.
I'll also have a closer look on the web framework aswell, it looks very 
promising! Assuming no immediate projects/deadlines, do you think it is a 
good idea to wait until the summer release?

Thanks for your patience (:

- Klaus


From hans at huebner.org  Sat Feb 23 07:31:17 2008
From: hans at huebner.org (=?ISO-8859-1?Q?Hans_H=FCbner?=)
Date: Sat, 23 Feb 2008 08:31:17 +0100
Subject: [bknr-devel] bknr.indices, the slot NIL is missing from the object
In-Reply-To: <200802230440.58527.UngerKlaus@gmx.de>
References: <200802210315.56844.UngerKlaus@gmx.de>
	<200802222150.22035.UngerKlaus@gmx.de>
	<fd2982320802221316r54ab89gca207d958d9b522e@mail.gmail.com>
	<200802230440.58527.UngerKlaus@gmx.de>
Message-ID: <fd2982320802222331s2c87f10dr73f7d82bbdafc1fc@mail.gmail.com>

2008/2/23, Klaus Unger <UngerKlaus at gmx.de>:
> Is it somehow possible with the datastore to run different stores in one lisp
>  process? For example to run multiple instances of the same web-application on
>  the same port on the same machine.

Currently not.  We have removed multi-store support at one point
because we felt that it was not useful enough to warrant cluttering
the API with STORE arguments. I think that it is usually better to put
separate applications into distinct Lisp processed as that approach
provides better isolation.

If you need multiple stores in one application, doing so would involve
rebinding the store special variables in a WITH-STORE macro.  I can
add it if you need it, please let me know.  Note that operations
involving two stores will not be supported by that and I'd really
question if that would make sense.

>  I'll also have a closer look on the web framework aswell, it looks very
>  promising! Assuming no immediate projects/deadlines, do you think it is a
>  good idea to wait until the summer release?

Presently, there is little documentation and we hope to be able to
change that situation for the release.  In its present state, several
things are not working properly, so I can't really recommend trying to
use it.  We expect to have the functionality complete in April.

-Hans


From hans at huebner.org  Sat Feb 23 16:49:16 2008
From: hans at huebner.org (=?ISO-8859-1?Q?Hans_H=FCbner?=)
Date: Sat, 23 Feb 2008 17:49:16 +0100
Subject: [bknr-devel] bknr.indices, the slot NIL is missing from the object
In-Reply-To: <200802231521.38359.UngerKlaus@gmx.de>
References: <200802210315.56844.UngerKlaus@gmx.de>
	<200802230440.58527.UngerKlaus@gmx.de>
	<fd2982320802222331s2c87f10dr73f7d82bbdafc1fc@mail.gmail.com>
	<200802231521.38359.UngerKlaus@gmx.de>
Message-ID: <fd2982320802230849rfbe97fctb5aaf9afda24c5fb@mail.gmail.com>

Hi Klaus,

I have just committed http://bknr.net/trac/changeset/2600 which adds
multiple store support through the WITH-TRANSACTION macro.  You need to use
the :MAKE-DEFAULT NIL as initarg when making a store instance to prevent
setting the *STORE* special variable.  WITH-STORE must then be used to
select a store to work with:

DATASTORE> (defvar *state* 0)
*STATE*
DATASTORE> (defvar store1 (make-instance 'store :directory "/tmp/store1/"
:make-default nil))
STORE1
DATASTORE> (defvar store2 (make-instance 'store :directory "/tmp/store2/"
:make-default nil))
STORE2
DATASTORE> (deftransaction foo (value) (setf *state* value))
FOO
DATASTORE> (with-store (store1) (foo 1))
1
DATASTORE> (with-store (store2) (foo 2))
2
DATASTORE> (with-store (store1) (restore) *state*)
reading store random state
restoring #<STORE DIR: "/tmp/store1/">
loading transaction log /tmp/store1/current/transaction-log
1
DATASTORE> (with-store (store2) (restore) *state*)
reading store random state
restoring #<STORE DIR: "/tmp/store2/">
loading transaction log /tmp/store2/current/transaction-log
2
DATASTORE>

Let me know if that suits and works for you.

-Hans
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.common-lisp.net/pipermail/bknr-devel/attachments/20080223/0e7c64c1/attachment.html>

From UngerKlaus at gmx.de  Sun Feb 24 03:18:25 2008
From: UngerKlaus at gmx.de (Klaus Unger)
Date: Sun, 24 Feb 2008 04:18:25 +0100
Subject: [bknr-devel] bknr.indices, the slot NIL is missing from the object
In-Reply-To: <fd2982320802222331s2c87f10dr73f7d82bbdafc1fc@mail.gmail.com>
References: <200802210315.56844.UngerKlaus@gmx.de>
	<200802230440.58527.UngerKlaus@gmx.de>
	<fd2982320802222331s2c87f10dr73f7d82bbdafc1fc@mail.gmail.com>
Message-ID: <200802240418.25907.UngerKlaus@gmx.de>

(resend to the list)

Hi Hans,

Am Samstag, 23. Februar 2008 08:31:17 schrieben Sie:
> 2008/2/23, Klaus Unger <UngerKlaus at gmx.de>:
> > Is it somehow possible with the datastore to run different stores in one
> > lisp process? For example to run multiple instances of the same
> > web-application on the same port on the same machine.
>
> Currently not.  We have removed multi-store support at one point
> because we felt that it was not useful enough to warrant cluttering
> the API with STORE arguments. I think that it is usually better to put
> separate applications into distinct Lisp processed as that approach
> provides better isolation.
I agree that store agruments would be a bad thing, I think thats what special 
variables are good for.

> If you need multiple stores in one application, doing so would involve
> rebinding the store special variables in a WITH-STORE macro.  I can
> add it if you need it, please let me know.  Note that operations
> involving two stores will not be supported by that and I'd really
> question if that would make sense.
It would indeed be handy for me. Could you give me a sketch of how that would 
work? From my understanding the *store* only contains some meta information 
(counter, path, lock, ..) while the actual objects are stored in the indices, 
which are stored in class slots that are not affected by the special 
variable.

- Klaus

Here is my try with the special variable:

BKNR.DATASTORE>
(make-instance 'mp-store :directory "/tmp/store1/" :subsystems (list 
(make-instance 'store-object-subsystem)))

reading store random state
restoring #<MP-STORE DIR: "/tmp/store1/">
loading transaction log /tmp/store1/current/transaction-log
#<MP-STORE DIR: "/tmp/store1/">

BKNR.DATASTORE>
(all-store-objects)

(#<STORE-OBJECT ID: 0>)

BKNR.DATASTORE>
(let ((*store* Nil))
  (make-instance 'mp-store :directory "/tmp/store2/" :subsystems (list 
(make-instance 'store-object-subsystem)))
  (all-store-objects)
  (close-store))

reading store random state
restoring #<MP-STORE DIR: "/tmp/store2/">
NIL

BKNR.DATASTORE>
*store*

#<MP-STORE DIR: "/tmp/store1/">
;; Back in the old store

BKNR.DATASTORE>
(all-store-objects)

NIL
;; But the object is gone :(


From UngerKlaus at gmx.de  Sun Feb 24 04:11:11 2008
From: UngerKlaus at gmx.de (Klaus Unger)
Date: Sun, 24 Feb 2008 05:11:11 +0100
Subject: [bknr-devel] bknr.indices, the slot NIL is missing from the object
In-Reply-To: <fd2982320802230849rfbe97fctb5aaf9afda24c5fb@mail.gmail.com>
References: <200802210315.56844.UngerKlaus@gmx.de>
	<200802231521.38359.UngerKlaus@gmx.de>
	<fd2982320802230849rfbe97fctb5aaf9afda24c5fb@mail.gmail.com>
Message-ID: <200802240511.11815.UngerKlaus@gmx.de>

Hi Hans,

thanks for the commits. If i got it right with-store does not apply to the 
STORE-OBJECT-SUBSYSTEM which is based on indices. Therfore it is not possible 
to use with-store for parallel stores based on all-store-objects and friends, 
right?
In regards to the transaction log it does work for me, and it is a handy 
thing, but not what I was originally looking for. What I meant would be like:

> DATASTORE> (with-store (store1) (make-object 'store-object))
=> #<STORE-OBJECT ID: 0>
> DATASTORE> (with-store (store2) (make-object 'foo-store-object))
=> #<FOO-STORE-OBJECT ID: 0>
> DATASTORE> (with-store (store1) (all-store-objects))
(#<STORE-OBJECT ID: 0>)
> DATASTORE> (with-store (store2) (all-store-objects))
(#<FOO-STORE-OBJECT ID: 0>)

This is not really meant as serial, but more like a multi threaded parallel 
execution. Obviously it would be possible to do this prepending a (restore), 
but this would bei a performance nightmare and require additional 
synchronization.

From how I understand the code this would require to make 
indexed-class-indices and index-effective-slot-definition-indices dependant 
on the current *store* for persistent-class and 
persistent-effective-slot-definition.

- Klaus

P.S.: I begin to feel guilty for always nagging, so I have to emphasize again 
that I am really thankful for the commununication and your patience. Also 
bknr is already really helpful for me in its current form! If we can be 
improve it - even better.

Am Samstag, 23. Februar 2008 17:49:16 schrieb Hans H?bner:
> Hi Klaus,
>
> I have just committed http://bknr.net/trac/changeset/2600 which adds
> multiple store support through the WITH-TRANSACTION macro.  You need to use
> the :MAKE-DEFAULT NIL as initarg when making a store instance to prevent
> setting the *STORE* special variable.  WITH-STORE must then be used to
> select a store to work with:
>
> DATASTORE> (defvar *state* 0)
> *STATE*
> DATASTORE> (defvar store1 (make-instance 'store :directory "/tmp/store1/"
>
> :make-default nil))
>
> STORE1
> DATASTORE> (defvar store2 (make-instance 'store :directory "/tmp/store2/"
>
> :make-default nil))
>
> STORE2
> DATASTORE> (deftransaction foo (value) (setf *state* value))
> FOO
> DATASTORE> (with-store (store1) (foo 1))
> 1
> DATASTORE> (with-store (store2) (foo 2))
> 2
> DATASTORE> (with-store (store1) (restore) *state*)
> reading store random state
> restoring #<STORE DIR: "/tmp/store1/">
> loading transaction log /tmp/store1/current/transaction-log
> 1
> DATASTORE> (with-store (store2) (restore) *state*)
> reading store random state
> restoring #<STORE DIR: "/tmp/store2/">
> loading transaction log /tmp/store2/current/transaction-log
> 2
> DATASTORE>
>
> Let me know if that suits and works for you.
>
> -Hans


From hans at huebner.org  Sun Feb 24 08:14:18 2008
From: hans at huebner.org (=?ISO-8859-1?Q?Hans_H=FCbner?=)
Date: Sun, 24 Feb 2008 09:14:18 +0100
Subject: [bknr-devel] bknr.indices, the slot NIL is missing from the object
In-Reply-To: <200802240511.11815.UngerKlaus@gmx.de>
References: <200802210315.56844.UngerKlaus@gmx.de>
	<200802231521.38359.UngerKlaus@gmx.de>
	<fd2982320802230849rfbe97fctb5aaf9afda24c5fb@mail.gmail.com>
	<200802240511.11815.UngerKlaus@gmx.de>
Message-ID: <fd2982320802240014g44349382u6a3aaeedcde944ec@mail.gmail.com>

Hi Klaus,

don't worry.  For the upcoming release, it is good when I spend some time in
the store code to get a grasp of where changes are needed.

You are certainly right:  My multi-store patch does not properly rebind
special variables used by subsystems, and this is something I will add.  I
want to make sure that there is no "performance overhead" for the
single-store case, which makes it slightly more complicated.  I'll let you
know once I have committed.  While fixing the subsystem issues, I will need
to repair random state handling which is also broken in the multiple store
case.

If you did not see it:  Trac generates RSS feeds for the Subversion log,
which is why I have stopped distributing commit notifications by email.
http://bknr.net/trac/timeline?milestone=on&ticket=on&changeset=on&wiki=on&max=50&daysback=90&format=rssis
the full feed.

-Hans
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.common-lisp.net/pipermail/bknr-devel/attachments/20080224/5de246ef/attachment.html>

From hans at huebner.org  Sun Feb 24 20:31:07 2008
From: hans at huebner.org (=?ISO-8859-1?Q?Hans_H=FCbner?=)
Date: Sun, 24 Feb 2008 21:31:07 +0100
Subject: [bknr-devel] bknr.indices, the slot NIL is missing from the object
In-Reply-To: <200802240511.11815.UngerKlaus@gmx.de>
References: <200802210315.56844.UngerKlaus@gmx.de>
	<200802231521.38359.UngerKlaus@gmx.de>
	<fd2982320802230849rfbe97fctb5aaf9afda24c5fb@mail.gmail.com>
	<200802240511.11815.UngerKlaus@gmx.de>
Message-ID: <fd2982320802241231t710f5f20oc0091bc7c327e42d@mail.gmail.com>

Hi Klaus,

I have given this some more thought:  Supporting multiple stores in the
indexing subsystem is possible, but it is rather complicated because an
additional dispatch in the INDEXED-CLASS metaclass is required to make the
indices dependent on *STORE*.  Currently, the indices are stored in the
INDEXED-CLASS instance which would be shared by multiple worlds running in
the same Lisp image.

The issue here is:  The change is relatively complex, requires testing and I
don't need it to support our development style.  In our architecture, the
persistent objects make up one world, and if I need multiple worlds, I start
multiple Lisps.  I would not benefit from running mulltiple independent
applications within one Lisp image, and the lack of isolation would really
be more bothersome than anything else.

That said, if you can make a good case for the functionality, I would try to
implement in the not-so-far future.  Please provide me with a good
application architecture example that would be significantly harder to
implement in multiple Lisp images.

-Hans
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.common-lisp.net/pipermail/bknr-devel/attachments/20080224/869be9e7/attachment.html>