[elephant-devel] Sqlite-3 backend, threads and transactions

[Ian Eslick]

A couple of comments on recent discussions:

Threads:

Threading is not cleanly supported yet.  The model where you have a
single controller variable declared and opened in the parent thread and
many children using it should work under with-transaction.

If each thread is opening it's own controller after spawning, then each
should have it's own local copy of *store-controller* and I'm not clear
whether a single process with multiple open references will have
problems or not.  There are some subtle issues in variable capture in
threading that I admit to not fully understanding in all the
implementations (I'm an Allegro user, primarily).  If someone would like
to reason through this and come up with a proper methodology and
macro/function API for dealing with different cases of threading I'd be
happy to sanity check the resulting code.  I don't know when I'm going
to get to this.

Object sizes:

I think the size of a given object slot value you store should be
proportional to the random access frequencies to objects of that type. 
I have a document object with arrays of 100's to 1000's of integers but
I only access them rarely and then I have to access the entire array. 
It's more efficient to store that all at once.  I have indices which are
accessed frequently.

Non-local exists from with-transaction:

The best way I think think of to resolve normal non-local exists from
within with-transaction is to catch those normal throw or signal cases
inside the macro and then exit normally.  You might think about making
an app-specific macro to handle this.  You could even rethrow the signal
outside with-transaction to fit in with the model of the larger program. 

The benefits of with-transaction is that if you get a lock timeout error
it will retry automatically.  In a multi-threading environment this is
critical since you will most likely have collisions at some point
requiring abort + retry.  I wouldn't recommend trying to handle every
abort yourself - it's easy to miss things and end up with an
inconsistent DB state.

Regards,
Ian

Robert L. Read wrote:
> On Thu, 2006-04-20 at 01:06 +0300, Ignas Mikalajunas wrote:
>> And indeed it is independent, but calls to ele:start-transaction,
>> ele:commit-transaction and ele:abort-transaction are throwing an
>> error for me and their code seems to be BDB specific. I would like to
>> use with-transacton, yet it aborts the transaction on "any non-local
>> exit", while i want to rollback only when some error occurs. TBNL
>> redirect code does (throw 'tbnl:tbnl-handler-done), aborting
>> transaction on each redirect. As in web applications
>> (save-data-to-database) (redirect-to-avoid-double-post) is a very
>> common pattern, i was getting my data reverted every time. So i am
>> doing manual commits/rollbacks, depending on whether the request
>> processing code raised an error or not.
>>
>
> Are you using the latest cvs version, or the latest released version
> from a tar file?
> The cvs version has reworked a lot of this to make it more clearly
> independent.
> If you check out the latest cvs version (instructions are on the
> Elephant project home page),
> you will find in the src directory transactions.lisp, which provides a
> small API.  This would
> be in spirit much, much better than calling anything on clsql
> directly.  The whole point of
> Elephant, and particularly the last year of work, is to protect the
> user from details at the
> implementation level.
>
> Of course, I understand making it work overrides design philosophy; I
> hate to tell you to extensively
> rewrite something, but I really must insist that directly accessing
> either CLSQL or BDB is not a
> reasonable long term approach.  It should be better to use the latest
> transaction interface
> (mostly reorganized by Ian Eslick, I think) that is independent of the
> back-end.  I've attached
> the file from the latest version for you to look at.
>
> I apologize that our latest version has been "just about ready" for so
> long; I hope we haven't wasted
> your time by not releasing it faster.
>
> I have not done that much work with the transaction handling, but am
> willing to debug any
> reproducible problem that you find.
>
> You have already posted a very nice, small, testcase for SBCL.  If we
> could figure out
> how to make that testcase work (perhaps by changing Elephant internals
> when SQLite3 is in use),
> would that allow you to go back to using the "pure" Elephant API?
>
> If so, I'll work on that.
>
> If not, I'll work on whatever the problem is, but I don't think I
> should work on any solution that
> goes around the Elephant API (or is not using the latest version from
> CVS), even though I
> appreciate you trying so hard to get it to work.
> ------------------------------------------------------------------------
>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel