[elephant-devel] Optimization

[Ian Eslick]

Sounds like you'd like to perform 100k database ops per second which  
is fairly significant.   What kind of data are you writing?

With sufficient BDB cache, the nosync option, and a transaction  
wrapper I'm surprised it's 10x slower - I'd expect more like 4-5x -  
which is still too slow for you.  You should try setting the txn- 
nosync option in the actual with-transaction wrapper - it may override  
the default flags on the environment.  However, if you don't care  
about durability of the data, then the following comment definitely  
applies to you.

For highly dynamic state that is unlikely to persist for more than a  
frame update, updating the store slots is serious overkill.  The  
easiest solution is to use transient slots for dynamic state and do a  
periodic copy to persistent slots if you need to persist that state.   
This presumes that you don't care about transactional isolation for  
that transient state which in a single-threaded app sounds like it's  
fine.


This discussion got me thinking about the caching protocol I was  
working on.  After some exploration and education from folks here and  
elsewhere I see two models that make sense in the context of database  
operations:

1) Check-in, Check-out (Robert's DCM model).  If your code enforces a  
policy of checking in and checking out prior to using the objects than  
you can do all the in-memory ops you want and do a checkin when you  
are done.

(Maybe this could be done so that you can create an object that is  
fully cached and persistent storage and IDs are only allocated when  
you first check-in the object...this would address the issues being  
discussed on the weblocks list)

So do we define this at the class level or the slot level?  Is this  
policy inherited by subclasses?

2) Per-transaction caching.  You read the value at the beginning of a  
transaction and then read the cached copy thereafter.

Thoughts?

Ian

On Dec 28, 2008, at 2:05 AM, Elliott Slaughter wrote:

> There are several replies to my original query, so I will attempt to  
> address all of them here.
>
> On Wed, Dec 24, 2008 at 2:31 PM, Ian Eslick <eslick at media.mit.edu>  
> wrote:
> A couple of quick thoughts on your problem:
>
> 1) Are you wrapping the critical sections of your code in with-
> transaction?  This causes all database pages you touch to be cached
> within the body of the transaction.  This avoids all 'sync' operations
> and transaction setup/teardown caused by a read/write slot operation
> that takes place outside a with-transaction body.
>
> I currently wrap the contents of each frame update in with- 
> transaction. Each frame consists of updating 100 objects on the  
> screen (which by my estimation ought to be about 20 slot reads and 2  
> writes), so I ought to be doing about 2000 slot reads each frame and  
> 200 writes.
>
> Wrapping the entire game loop in with-transaction increases  
> performance by about 10%. I also tried
>
> (db-env-set-flags (controller-environment *store-controller*) 1 :txn- 
> nosync t)
>
> as suggested in the manual, with a similar (about 10%) performance  
> increase.
>
> 2) If you are doing this, have you increased your BDB cache size to
> ensure you can cache your key working set?
>
> It should already be large enough, but just for kicks I increased it  
> from 2 to 10 MB, resulting in a performance increase of about 3%.
>
> 3) If yes, do you have significant contention between reads and writes
> for certain data structures that are causing transactions to be
> aborted.  Perhaps you can refactor around this.
>
> This is a single-threaded (and single-process) app. I don't see how  
> I could possibly have contention for the db.
>
> Ian
>
> And thanks to both Leslie and Alex for suggesting caching methods. I  
> will probably try to implement something along the lines of Alex's  
> suggestion, as that makes sense and should be simple enough (for the  
> single threaded model, which is sufficient for me).
>
>
> Thanks again for all the help. I'll report back either when I have  
> something working or I've run into another wall.
>
> -- 
> Elliott Slaughter
>
> "Any road followed precisely to its end leads precisely nowhere." -  
> Frank Herbert
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel