[elephant-devel] A thorny problem....

[Ian Eslick]

I've been absently noodling over the same problem.  I think in general
we want to factor out the backends more cleanly so we can have a load
system like that in UCW, Slime, CL-SQL, etc.  (multiple backends where
only one needs to be loaded).  Robert has already done much of the work
toward this with cursors, btrees, etc

One way to handle the transaction macro, although we should think about
this a bit because it adds a function call cost during short
transactions and I may also be missing some dependency:

(defmacro with-transaction ((&key store-controller ...) &body body)
    `(execute-store-transaction store-controller
             (lambda (*store-controller* *auto-commit* ...)
                   (declare (special *store-controller* *auto-commit*))
                   , at body)))

Then for BDB:

(defmethod execute-store-transaction ((store bdb-store-controller) body
args)
    `(loop
          ....
          (let ((,txn (db-transaction-begin ...insert args...)))
              (catch 'transaction
                  (unwind-protect
                       (prog1 (funcall body store-controller nil)
                            (setq ,success t)
                            (db-transaction-success ...)
                     (unless ,success
                         (db-transaction-abort ...)
            ...))

Don't you just love lisp?  This is one of the functions of closures; to
dynamically manipulate lexical environments without having to do source
transformations.  I think the closure will naturally reference
external specials like *store-controller* without us needed to pass them
in the lambda - but passing
them in the lambda gives us more control over the body of the
transaction.  cl-prevalence did something
very similar to this (all transaction bodies were named lambdas that
could be called later to replay
a transaction).

In this way a backend needs to implement:
store protocol
transaction protocol
btree protocol
    indexed-btree
    btree-index
cursor protocol
persistent object protocol (those aux functions you created are a good
start)

The current implementation of class indexing should come for free, I
tried to make it backend independant but for performance we might want
to implement a native protocol later.

The load sequence can be like CL-SQL and UCW: a main load (elephant), to
define the interface and common functionality followed by a backend load
(ele-bdb or ele-sqlite3 or ele-postgres).  You can also make the backend
load part of a dispatch list in the (open-store) function.  You open a
store with a keyword specified of :type 'bdb and :spec "/pathname/" and
that code calls asdf:load-op on the appropriate backend loader.

Ideally each family of backends would have its own subdirectory and
files to implement the protocols and any other functionality (like
sleepycat).

I have a pretty good idea of what this should look like and would be
happy to help do it as part of 0.6.0 as I don't think it's that much
work if done incrementally and carefully.

Robert, do you want me to take a crack at the transaction fix?  I've got
a great stress test application on my side with lots of rapid
transactions and some good performance data to compare against.

Anyway, my ten cents...

Ian

Robert L. Read wrote:
> I think I've a identified a serious problem relating to the
> the dependencies on various modules that I offer up
> for you comment.
>
> I major goal of the multiple backend system is to allow
> you to not need to install the software related to the system
> you don't need.  If you want to use BerkeleyDB, you shouldn't have
> to install CL-SQL.  If you want to use CL-SQL and a relational database,
> you can't shouldn't have to install BerkeleyDB (which would subject
> you to their license, in any case.)
>
> Unfortunately, the "with-transaction" macro, which exists in
> cl-sql, BDB, and also Elephant, must be in place at compilation
> time.
>
> I have written this macro to take a key argument, and, based on
> the type of the passed-in store-controller, to resolve either to
> the CL-SQL with-transaction macro, or the existing with-transaciton
> macro for BDB.
>
> I don't know how to write this macro without having CL-SQL macro
> loaded (which implies that you have it installed.)
>
> I am therefore in a bind: 
> I can't right a generic with-transaction macro without loading CL-SQL,
> and I can't make you install CL-SQL if you don't want to use it.
> The macros cannot be treated as functions, (which could easily remain
> undefined if they did not actually need to be executed), since the
> "with-transaction" idea cannot possibly be written as a function.
>
> I can think of two solutions:
>
> 1)  Create two asdf systems: one for with clsql, one without.  Then
> it would not be possible or necessary to load "elephant" without
> choosing a back-end (or a set of back-ends).
> 2)  Alternatively, the loading of the cl-sql module could reload the
> files with macro in place.  This will technically work, but is hideous
> in the sense that performing a
> (asdf:operate 'asdf:load-op :ele-clsql)
> would perform a reloading or a recompilation of more than half the
> project.
>
> Unless someone can provide me some better advice, I'm going to
> investigate
> #1.
>
> I find all of these dependencies complicated, and macros tend to
> exacerbate
> the problem.  Perhaps there is a much simpler way of dealing with this
> that someone can suggest.
>
>
> I must admit that I don't understand how 0.5.0 works, and why only now
> Ian Eslick's code in the head brings this problem to light.
>
>
> ----
> Robert L. Read, PhD                                     read &T
> robertlread.net
> Consider visiting Progressive Engineering:      http://robertlread.net/pe
> In Austin: 912-8593                                        "Think
> globally, Act locally." -- RBF
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel