[elephant-devel] A thorny problem....

Fri Feb 17 16:20:03 UTC 2006

Yes, this is a great solution.  I was fumbling towards it, having
defined "get-transaction-macro-symbol" already,
but you have a much greater command of it.  Clearly, something like this
where a completely
back-end specific method accomplishes the "with-transaction" need is the
clean way to go,
and allows for expansion to other back-ends (like a simple file) later.
I completely agree 
that having different directories for different back-ends is a very
clean and clarifying way to 
handle it.

I would be very grateful if you would do this Ian; I suspect you can do
it faster than I can.
I will support you in anyway possible.

It's not hard to be a maintainer when people such as yourself are so
very helpful!

If you can complete the entire plan you've outlined in 0.6.0, great!  If
not, I think if 
we can just solve the "transaction problem" (which I think will make us
green on all 
of the tests for all of the currently tested implementation), then  I
would be happy to 
release 0.6.0 at that point.  But remember the Inventor's Paradox:
sometimes doing
the more ambitious thing is easier.

Let me know how I can help.

On Fri, 2006-02-17 at 10:50 -0500, Ian Eslick wrote:

> I've been absently noodling over the same problem.  I think in general
> we want to factor out the backends more cleanly so we can have a load
> system like that in UCW, Slime, CL-SQL, etc.  (multiple backends where
> only one needs to be loaded).  Robert has already done much of the work
> toward this with cursors, btrees, etc
> 
> One way to handle the transaction macro, although we should think about
> this a bit because it adds a function call cost during short
> transactions and I may also be missing some dependency:
> 
> (defmacro with-transaction ((&key store-controller ...) &body body)
>     `(execute-store-transaction store-controller
>              (lambda (*store-controller* *auto-commit* ...)
>                    (declare (special *store-controller* *auto-commit*))
>                    , at body)))
> 
> Then for BDB:
> 
> (defmethod execute-store-transaction ((store bdb-store-controller) body
> args)
>     `(loop
>           ....
>           (let ((,txn (db-transaction-begin ...insert args...)))
>               (catch 'transaction
>                   (unwind-protect
>                        (prog1 (funcall body store-controller nil)
>                             (setq ,success t)
>                             (db-transaction-success ...)
>                      (unless ,success
>                          (db-transaction-abort ...)
>             ...))
> 
> Don't you just love lisp?  This is one of the functions of closures; to
> dynamically manipulate lexical environments without having to do source
> transformations.  I think the closure will naturally reference
> external specials like *store-controller* without us needed to pass them
> in the lambda - but passing
> them in the lambda gives us more control over the body of the
> transaction.  cl-prevalence did something
> very similar to this (all transaction bodies were named lambdas that
> could be called later to replay
> a transaction).
> 
> In this way a backend needs to implement:
> store protocol
> transaction protocol
> btree protocol
>     indexed-btree
>     btree-index
> cursor protocol
> persistent object protocol (those aux functions you created are a good
> start)
> 
> The current implementation of class indexing should come for free, I
> tried to make it backend independant but for performance we might want
> to implement a native protocol later.
> 
> The load sequence can be like CL-SQL and UCW: a main load (elephant), to
> define the interface and common functionality followed by a backend load
> (ele-bdb or ele-sqlite3 or ele-postgres).  You can also make the backend
> load part of a dispatch list in the (open-store) function.  You open a
> store with a keyword specified of :type 'bdb and :spec "/pathname/" and
> that code calls asdf:load-op on the appropriate backend loader.
> 
> Ideally each family of backends would have its own subdirectory and
> files to implement the protocols and any other functionality (like
> sleepycat).
> 
> I have a pretty good idea of what this should look like and would be
> happy to help do it as part of 0.6.0 as I don't think it's that much
> work if done incrementally and carefully.
> 
> Robert, do you want me to take a crack at the transaction fix?  I've got
> a great stress test application on my side with lots of rapid
> transactions and some good performance data to compare against.
> 
> Anyway, my ten cents...
> 
> Ian
> 
> Robert L. Read wrote:
> > I think I've a identified a serious problem relating to the
> > the dependencies on various modules that I offer up
> > for you comment.
> >
> > I major goal of the multiple backend system is to allow
> > you to not need to install the software related to the system
> > you don't need.  If you want to use BerkeleyDB, you shouldn't have
> > to install CL-SQL.  If you want to use CL-SQL and a relational database,
> > you can't shouldn't have to install BerkeleyDB (which would subject
> > you to their license, in any case.)
> >
> > Unfortunately, the "with-transaction" macro, which exists in
> > cl-sql, BDB, and also Elephant, must be in place at compilation
> > time.
> >
> > I have written this macro to take a key argument, and, based on
> > the type of the passed-in store-controller, to resolve either to
> > the CL-SQL with-transaction macro, or the existing with-transaciton
> > macro for BDB.
> >
> > I don't know how to write this macro without having CL-SQL macro
> > loaded (which implies that you have it installed.)
> >
> > I am therefore in a bind: 
> > I can't right a generic with-transaction macro without loading CL-SQL,
> > and I can't make you install CL-SQL if you don't want to use it.
> > The macros cannot be treated as functions, (which could easily remain
> > undefined if they did not actually need to be executed), since the
> > "with-transaction" idea cannot possibly be written as a function.
> >
> > I can think of two solutions:
> >
> > 1)  Create two asdf systems: one for with clsql, one without.  Then
> > it would not be possible or necessary to load "elephant" without
> > choosing a back-end (or a set of back-ends).
> > 2)  Alternatively, the loading of the cl-sql module could reload the
> > files with macro in place.  This will technically work, but is hideous
> > in the sense that performing a
> > (asdf:operate 'asdf:load-op :ele-clsql)
> > would perform a reloading or a recompilation of more than half the
> > project.
> >
> > Unless someone can provide me some better advice, I'm going to
> > investigate
> > #1.
> >
> > I find all of these dependencies complicated, and macros tend to
> > exacerbate
> > the problem.  Perhaps there is a much simpler way of dealing with this
> > that someone can suggest.
> >
> >
> > I must admit that I don't understand how 0.5.0 works, and why only now
> > Ian Eslick's code in the head brings this problem to light.
> >
> >
> > ----
> > Robert L. Read, PhD                                     read &T
> > robertlread.net
> > Consider visiting Progressive Engineering:      http://robertlread.net/pe
> > In Austin: 912-8593                                        "Think
> > globally, Act locally." -- RBF
> >
> >
> > ------------------------------------------------------------------------
> >
> > _______________________________________________
> > elephant-devel site list
> > elephant-devel at common-lisp.net
> > http://common-lisp.net/mailman/listinfo/elephant-devel
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.common-lisp.net/pipermail/elephant-devel/attachments/20060217/e44e2409/attachment.html>