<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 TRANSITIONAL//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; CHARSET=UTF-8">
<META NAME="GENERATOR" CONTENT="GtkHTML/3.3.2">
</HEAD>
<BODY>
Yes, this is a great solution. I was fumbling towards it, having defined "get-transaction-macro-symbol" already,<BR>
but you have a much greater command of it. Clearly, something like this where a completely<BR>
back-end specific method accomplishes the "with-transaction" need is the clean way to go,<BR>
and allows for expansion to other back-ends (like a simple file) later. I completely agree <BR>
that having different directories for different back-ends is a very clean and clarifying way to <BR>
handle it.<BR>
<BR>
I would be very grateful if you would do this Ian; I suspect you can do it faster than I can.<BR>
I will support you in anyway possible.<BR>
<BR>
It's not hard to be a maintainer when people such as yourself are so very helpful!<BR>
<BR>
If you can complete the entire plan you've outlined in 0.6.0, great! If not, I think if <BR>
we can just solve the "transaction problem" (which I think will make us green on all <BR>
of the tests for all of the currently tested implementation), then I would be happy to <BR>
release 0.6.0 at that point. But remember the Inventor's Paradox: sometimes doing<BR>
the more ambitious thing is easier.<BR>
<BR>
Let me know how I can help.<BR>
<BR>
On Fri, 2006-02-17 at 10:50 -0500, Ian Eslick wrote:
<BLOCKQUOTE TYPE=CITE>
<PRE>
<FONT COLOR="#000000">I've been absently noodling over the same problem. I think in general</FONT>
<FONT COLOR="#000000">we want to factor out the backends more cleanly so we can have a load</FONT>
<FONT COLOR="#000000">system like that in UCW, Slime, CL-SQL, etc. (multiple backends where</FONT>
<FONT COLOR="#000000">only one needs to be loaded). Robert has already done much of the work</FONT>
<FONT COLOR="#000000">toward this with cursors, btrees, etc</FONT>
<FONT COLOR="#000000">One way to handle the transaction macro, although we should think about</FONT>
<FONT COLOR="#000000">this a bit because it adds a function call cost during short</FONT>
<FONT COLOR="#000000">transactions and I may also be missing some dependency:</FONT>
<FONT COLOR="#000000">(defmacro with-transaction ((&key store-controller ...) &body body)</FONT>
<FONT COLOR="#000000"> `(execute-store-transaction store-controller</FONT>
<FONT COLOR="#000000"> (lambda (*store-controller* *auto-commit* ...)</FONT>
<FONT COLOR="#000000"> (declare (special *store-controller* *auto-commit*))</FONT>
<FONT COLOR="#000000"> ,@body)))</FONT>
<FONT COLOR="#000000">Then for BDB:</FONT>
<FONT COLOR="#000000">(defmethod execute-store-transaction ((store bdb-store-controller) body</FONT>
<FONT COLOR="#000000">args)</FONT>
<FONT COLOR="#000000"> `(loop</FONT>
<FONT COLOR="#000000"> ....</FONT>
<FONT COLOR="#000000"> (let ((,txn (db-transaction-begin ...insert args...)))</FONT>
<FONT COLOR="#000000"> (catch 'transaction</FONT>
<FONT COLOR="#000000"> (unwind-protect</FONT>
<FONT COLOR="#000000"> (prog1 (funcall body store-controller nil)</FONT>
<FONT COLOR="#000000"> (setq ,success t)</FONT>
<FONT COLOR="#000000"> (db-transaction-success ...)</FONT>
<FONT COLOR="#000000"> (unless ,success</FONT>
<FONT COLOR="#000000"> (db-transaction-abort ...)</FONT>
<FONT COLOR="#000000"> ...))</FONT>
<FONT COLOR="#000000">Don't you just love lisp? This is one of the functions of closures; to</FONT>
<FONT COLOR="#000000">dynamically manipulate lexical environments without having to do source</FONT>
<FONT COLOR="#000000">transformations. I think the closure will naturally reference</FONT>
<FONT COLOR="#000000">external specials like *store-controller* without us needed to pass them</FONT>
<FONT COLOR="#000000">in the lambda - but passing</FONT>
<FONT COLOR="#000000">them in the lambda gives us more control over the body of the</FONT>
<FONT COLOR="#000000">transaction. cl-prevalence did something</FONT>
<FONT COLOR="#000000">very similar to this (all transaction bodies were named lambdas that</FONT>
<FONT COLOR="#000000">could be called later to replay</FONT>
<FONT COLOR="#000000">a transaction).</FONT>
<FONT COLOR="#000000">In this way a backend needs to implement:</FONT>
<FONT COLOR="#000000">store protocol</FONT>
<FONT COLOR="#000000">transaction protocol</FONT>
<FONT COLOR="#000000">btree protocol</FONT>
<FONT COLOR="#000000"> indexed-btree</FONT>
<FONT COLOR="#000000"> btree-index</FONT>
<FONT COLOR="#000000">cursor protocol</FONT>
<FONT COLOR="#000000">persistent object protocol (those aux functions you created are a good</FONT>
<FONT COLOR="#000000">start)</FONT>
<FONT COLOR="#000000">The current implementation of class indexing should come for free, I</FONT>
<FONT COLOR="#000000">tried to make it backend independant but for performance we might want</FONT>
<FONT COLOR="#000000">to implement a native protocol later.</FONT>
<FONT COLOR="#000000">The load sequence can be like CL-SQL and UCW: a main load (elephant), to</FONT>
<FONT COLOR="#000000">define the interface and common functionality followed by a backend load</FONT>
<FONT COLOR="#000000">(ele-bdb or ele-sqlite3 or ele-postgres). You can also make the backend</FONT>
<FONT COLOR="#000000">load part of a dispatch list in the (open-store) function. You open a</FONT>
<FONT COLOR="#000000">store with a keyword specified of :type 'bdb and :spec "/pathname/" and</FONT>
<FONT COLOR="#000000">that code calls asdf:load-op on the appropriate backend loader.</FONT>
<FONT COLOR="#000000">Ideally each family of backends would have its own subdirectory and</FONT>
<FONT COLOR="#000000">files to implement the protocols and any other functionality (like</FONT>
<FONT COLOR="#000000">sleepycat).</FONT>
<FONT COLOR="#000000">I have a pretty good idea of what this should look like and would be</FONT>
<FONT COLOR="#000000">happy to help do it as part of 0.6.0 as I don't think it's that much</FONT>
<FONT COLOR="#000000">work if done incrementally and carefully.</FONT>
<FONT COLOR="#000000">Robert, do you want me to take a crack at the transaction fix? I've got</FONT>
<FONT COLOR="#000000">a great stress test application on my side with lots of rapid</FONT>
<FONT COLOR="#000000">transactions and some good performance data to compare against.</FONT>
<FONT COLOR="#000000">Anyway, my ten cents...</FONT>
<FONT COLOR="#000000">Ian</FONT>
<FONT COLOR="#000000">Robert L. Read wrote:</FONT>
<FONT COLOR="#000000">> I think I've a identified a serious problem relating to the</FONT>
<FONT COLOR="#000000">> the dependencies on various modules that I offer up</FONT>
<FONT COLOR="#000000">> for you comment.</FONT>
<FONT COLOR="#000000">></FONT>
<FONT COLOR="#000000">> I major goal of the multiple backend system is to allow</FONT>
<FONT COLOR="#000000">> you to not need to install the software related to the system</FONT>
<FONT COLOR="#000000">> you don't need. If you want to use BerkeleyDB, you shouldn't have</FONT>
<FONT COLOR="#000000">> to install CL-SQL. If you want to use CL-SQL and a relational database,</FONT>
<FONT COLOR="#000000">> you can't shouldn't have to install BerkeleyDB (which would subject</FONT>
<FONT COLOR="#000000">> you to their license, in any case.)</FONT>
<FONT COLOR="#000000">></FONT>
<FONT COLOR="#000000">> Unfortunately, the "with-transaction" macro, which exists in</FONT>
<FONT COLOR="#000000">> cl-sql, BDB, and also Elephant, must be in place at compilation</FONT>
<FONT COLOR="#000000">> time.</FONT>
<FONT COLOR="#000000">></FONT>
<FONT COLOR="#000000">> I have written this macro to take a key argument, and, based on</FONT>
<FONT COLOR="#000000">> the type of the passed-in store-controller, to resolve either to</FONT>
<FONT COLOR="#000000">> the CL-SQL with-transaction macro, or the existing with-transaciton</FONT>
<FONT COLOR="#000000">> macro for BDB.</FONT>
<FONT COLOR="#000000">></FONT>
<FONT COLOR="#000000">> I don't know how to write this macro without having CL-SQL macro</FONT>
<FONT COLOR="#000000">> loaded (which implies that you have it installed.)</FONT>
<FONT COLOR="#000000">></FONT>
<FONT COLOR="#000000">> I am therefore in a bind: </FONT>
<FONT COLOR="#000000">> I can't right a generic with-transaction macro without loading CL-SQL,</FONT>
<FONT COLOR="#000000">> and I can't make you install CL-SQL if you don't want to use it.</FONT>
<FONT COLOR="#000000">> The macros cannot be treated as functions, (which could easily remain</FONT>
<FONT COLOR="#000000">> undefined if they did not actually need to be executed), since the</FONT>
<FONT COLOR="#000000">> "with-transaction" idea cannot possibly be written as a function.</FONT>
<FONT COLOR="#000000">></FONT>
<FONT COLOR="#000000">> I can think of two solutions:</FONT>
<FONT COLOR="#000000">></FONT>
<FONT COLOR="#000000">> 1) Create two asdf systems: one for with clsql, one without. Then</FONT>
<FONT COLOR="#000000">> it would not be possible or necessary to load "elephant" without</FONT>
<FONT COLOR="#000000">> choosing a back-end (or a set of back-ends).</FONT>
<FONT COLOR="#000000">> 2) Alternatively, the loading of the cl-sql module could reload the</FONT>
<FONT COLOR="#000000">> files with macro in place. This will technically work, but is hideous</FONT>
<FONT COLOR="#000000">> in the sense that performing a</FONT>
<FONT COLOR="#000000">> (asdf:operate 'asdf:load-op :ele-clsql)</FONT>
<FONT COLOR="#000000">> would perform a reloading or a recompilation of more than half the</FONT>
<FONT COLOR="#000000">> project.</FONT>
<FONT COLOR="#000000">></FONT>
<FONT COLOR="#000000">> Unless someone can provide me some better advice, I'm going to</FONT>
<FONT COLOR="#000000">> investigate</FONT>
<FONT COLOR="#000000">> #1.</FONT>
<FONT COLOR="#000000">></FONT>
<FONT COLOR="#000000">> I find all of these dependencies complicated, and macros tend to</FONT>
<FONT COLOR="#000000">> exacerbate</FONT>
<FONT COLOR="#000000">> the problem. Perhaps there is a much simpler way of dealing with this</FONT>
<FONT COLOR="#000000">> that someone can suggest.</FONT>
<FONT COLOR="#000000">></FONT>
<FONT COLOR="#000000">></FONT>
<FONT COLOR="#000000">> I must admit that I don't understand how 0.5.0 works, and why only now</FONT>
<FONT COLOR="#000000">> Ian Eslick's code in the head brings this problem to light.</FONT>
<FONT COLOR="#000000">></FONT>
<FONT COLOR="#000000">></FONT>
<FONT COLOR="#000000">> ----</FONT>
<FONT COLOR="#000000">> Robert L. Read, PhD read &T</FONT>
<FONT COLOR="#000000">> robertlread.net</FONT>
<FONT COLOR="#000000">> Consider visiting Progressive Engineering: <A HREF="http://robertlread.net/pe">http://robertlread.net/pe</A></FONT>
<FONT COLOR="#000000">> In Austin: 912-8593 "Think</FONT>
<FONT COLOR="#000000">> globally, Act locally." -- RBF</FONT>
<FONT COLOR="#000000">></FONT>
<FONT COLOR="#000000">></FONT>
<FONT COLOR="#000000">> ------------------------------------------------------------------------</FONT>
<FONT COLOR="#000000">></FONT>
<FONT COLOR="#000000">> _______________________________________________</FONT>
<FONT COLOR="#000000">> elephant-devel site list</FONT>
<FONT COLOR="#000000">> <A HREF="mailto:elephant-devel@common-lisp.net">elephant-devel@common-lisp.net</A></FONT>
<FONT COLOR="#000000">> <A HREF="http://common-lisp.net/mailman/listinfo/elephant-devel">http://common-lisp.net/mailman/listinfo/elephant-devel</A></FONT>
<FONT COLOR="#000000">_______________________________________________</FONT>
<FONT COLOR="#000000">elephant-devel site list</FONT>
<FONT COLOR="#000000"><A HREF="mailto:elephant-devel@common-lisp.net">elephant-devel@common-lisp.net</A></FONT>
<FONT COLOR="#000000"><A HREF="http://common-lisp.net/mailman/listinfo/elephant-devel">http://common-lisp.net/mailman/listinfo/elephant-devel</A></FONT>
</PRE>
</BLOCKQUOTE>
</BODY>
</HTML>