[elephant-devel] Mixing "logical" layers in an elephant application

[Robert L. Read]

Dear Alain,

I think you have analyzed the problem very well.  As you say, in the
postmodern codebase in the file query.lisp, the macro "doquery"
utilizes the *database* variable with no obvious way to override or
specify it.  I would have to study some more to be absolutely sure,
but it does seem quite likely that the other postmodern macros that
are utilized by elephant and mention the same variable probably make
it difficult to interleave postmodern calls of any kind to different
repositories, whether elephant is involved or not, and of course the
ele-postmodern code also ends up setting that variable, invisibly to
the user.

Ian and I once had to deal with the same issue in Elephant's use of
CL-SQL.  The basic solution is to always float parameters up (at least
optionally), so that instead of writing:

(doquery q (arg) (blah-blah arg)),

one at least has the option of writing:

(doquery q (arg) (blah-blah arg) :with-this-connection
UNEXPECTEDCONNECTION)

However, this is a non-trivial change to the postmodern code.

In the testmigrations.lisp file in the /tests directory in the
elepahnt codebase, you will find the following idiom, which can be
used to sequester a globabl value, change it, and restore it.

	(let ((old-store *store-controller*)
	      (*store-controller* nil))
	  (unwind-protect 
	  (do-a-bunch of stuff that sets *store-controller*)
	  (progn
	    (setq *store-controller* old-store)))
	    )

It is possible that by doing something similar to the
postmodern::*database value (instead of the *store-controller*), you
could make your call out to "doquery" in the sample code you mention,
and by restoring *database*, be able to carry on.

(Of course, you should not have to do this, this is clearly a bug at
some level.  It is not entirely clear to me where it should be fixed.
I would have to hear from Henrik or Alex, or study at great length, to
answer your deeper question: what other surprises might await
multi-connection operation via postmodern.  I hope Henrik will read
this and answer that question.)

The long term solution to the problem (whether provided by postmodern,
ele-postmodern, or the core of elephant, I'm not sure) is to allow
fully specification of which connnection to use in every operation
(while trying to use intelligent defaults, so that the most calls do
not require it to be specified.)

So, if you can forgive my ignorance of postmodern, the question for
Henrik are:

1) Can postmodern interleave calls to different transactions, and if
    so how?

2) If it can't, is manipulating *database* a reasonable short-term
solution?  Is there other state (caches, for example), that might also
have to be "swizeled"?



On Sun, 2007-12-09 at 16:41 +1100, Alain Picard wrote:
> Dear Elephant developers,
> 
> I'm trying to develop an application using Elephant, and
> I think I've settled on the POSTMODERN back end.
> 
> Now, part of the application maps very naturally in what Elephant can
> do, and other parts of it map better onto raw SQL processing,
> where I wish to use POSTMODERN more or less directly.  As a
> toy example, consider this:
> 
>     ;; in postmodern specific file
>     (defun map-visits (fn thing-id &key (from 0) (to 9999999999) collect)
>       (let ((results '()))
>         (flet ((collector (thing)
>                  (push (funcall fn thing) results)))
>           (let ((fn (if collect #'collector fn)))
>             (doquery (:select thing-id
>                               :from :visit
>                               :where (:and (:= thing-id id)
>                                            (:< from 'timestamp)
>                                            (:<= 'timestamp to)))
>                 (arg)
>               (funcall fn arg))))
>         results))
> 
>     ;; Later, in an elephant specific file
>     (defmethod process (thing &key from to)
>       (let ((contribs (make-hash-table)))
>         (flet ((compute-something (arg)
>                  (incf (gethash arg contribs 0))))
>           (ensure-transaction ()
>             (map-visits #'add-contribution (person-name fan) :from from :to to)))
>         contribs))
> 
> The first function knows how to map over "visits", which are things
> inserted into a postgresql table by a process other than my app.
> It's a very simple SQL table.  But it relates semantically to objects
> held within the elephant database; e.g. the second function, 
> PROCESS-THING wants to map over all the visits of a THING, and do
> some elephantine business with each visit, but doesn't want to concern
> itself with how VISITS are implemented.
> 
> When I tried the above, it blew up when I added the ENSURE-TRANSACTION
> clause; because all of a sudden the DOQUERY found itself querying
> on the "wrong" database.  (I'm assuming ENSURE-TRANSACTION binds
> something like a *DATABASE* var which got mistakenly inherited by
> the postmodern layer)
> 
> This led me to thinking about what other unknown effects/surprises
> may be lurking for me; e.g. how transactions in one "world" will
> interact with another one, etc.  I have read the manual, including the
> terse section "4.12 Multi-repository Operation", but what I'm attempting
> may be even hairier.
> 
> Inspecting the code, I'm currently tempted to go hack up something
> based on DB-POSTMODERN::CONTROLLER-CONNECTION-FOR-THREAD to
> "marry" the two DB layers, but I thought I'd ask first if anyone
> else has considered these sorts of issues and if there is a
> "recommended" way of proceeding in a case such as this.  I guess
> the first thing I'd try is to bind something way up in the elephant
> call stack which would cause all lower level "raw" postmodern calls
> to act on the same DB that the *STORE-CONTROLLER* is currently 
> using.
> 
> Any thoughts or comments will be deeply appreciated.
> 
>                               Alain Picard
> 
> 
>