[elephant-devel] Optimization

[Ian Eslick]

I ran your tests on my new elephant tree that includes the checkout  
protocol (which is now available via elephant-unstable) and the local  
improvements to persistent slot access time.

Writes are about 2x slower than reads so a read/write test cycle is  
also informative.

Attached is my annotations to your file.

Cheers,
Ian

-------------- next part --------------
A non-text attachment was scrubbed...
Name: slot-read-cache-performance.lisp
Type: application/octet-stream
Size: 2142 bytes
Desc: not available
URL: <https://mailman.common-lisp.net/pipermail/elephant-devel/attachments/20081229/c9ee0717/attachment.obj>
-------------- next part --------------




On Dec 29, 2008, at 3:15 PM, Elliott Slaughter wrote:

> On Sun, Dec 28, 2008 at 2:09 PM, Ian Eslick <eslick at media.mit.edu>  
> wrote:
> I implemented a quick check-in/check-out protocol for persistent   
> objects.
>
> Nice. Mind sharing?
>
> A standard-object slot allows about 50M updates-sec
> A persistent-object slot (inside a txn) is about 60k updates-sec
> A cached persistent-object slot is about 3M updates-sec
>
> I implemented my own version of caching using Alex's suggestion  
> (with slot defs caching their values in hash tables).
>
> With a simple loop (code attached), I get about 30M updates per  
> second (Hz) for standard slots, about 30 kHz for persistent slots,  
> and about 2.4 MHz using my caching scheme.
>
> The slot access protocol adds about 10x overhead per slot read but the
> cached access is about 50x faster than the pure persistent query.
> I'll see if there are some easy opportunities to speed things up.
>
> I observed about the same performance differences (although I saw  
> closer to an 100x difference between persistence and caching).
>
> My method uses hash table lookups for slot reads, which makes me  
> wonder if I'd get better performance by using a slot caching  
> protocol more similar to the default implementation.
>
> Anyways, using cached slots in my game results in a 2x speed  
> increase, although it still runs 2x to 3x slower than the completely  
> in-memory version. I guess I have other performance concerns to  
> think about than just slot reads.
>
> Ian
>
> On Dec 28, 2008, at 2:05 AM, Elliott Slaughter wrote:
>
> > There are several replies to my original query, so I will attempt to
> > address all of them here.
> >
> > On Wed, Dec 24, 2008 at 2:31 PM, Ian Eslick <eslick at media.mit.edu>
> > wrote:
> > A couple of quick thoughts on your problem:
> >
> > 1) Are you wrapping the critical sections of your code in with-
> > transaction?  This causes all database pages you touch to be cached
> > within the body of the transaction.  This avoids all 'sync'  
> operations
> > and transaction setup/teardown caused by a read/write slot operation
> > that takes place outside a with-transaction body.
> >
> > I currently wrap the contents of each frame update in with-
> > transaction. Each frame consists of updating 100 objects on the
> > screen (which by my estimation ought to be about 20 slot reads and 2
> > writes), so I ought to be doing about 2000 slot reads each frame and
> > 200 writes.
> >
> > Wrapping the entire game loop in with-transaction increases
> > performance by about 10%. I also tried
> >
> > (db-env-set-flags (controller-environment *store-controller*)  
> 1 :txn-
> > nosync t)
> >
> > as suggested in the manual, with a similar (about 10%) performance
> > increase.
> >
> > 2) If you are doing this, have you increased your BDB cache size to
> > ensure you can cache your key working set?
> >
> > It should already be large enough, but just for kicks I increased it
> > from 2 to 10 MB, resulting in a performance increase of about 3%.
> >
> > 3) If yes, do you have significant contention between reads and  
> writes
> > for certain data structures that are causing transactions to be
> > aborted.  Perhaps you can refactor around this.
> >
> > This is a single-threaded (and single-process) app. I don't see how
> > I could possibly have contention for the db.
> >
> > Ian
> >
> > And thanks to both Leslie and Alex for suggesting caching methods. I
> > will probably try to implement something along the lines of Alex's
> > suggestion, as that makes sense and should be simple enough (for the
> > single threaded model, which is sufficient for me).
> >
> >
> > Thanks again for all the help. I'll report back either when I have
> > something working or I've run into another wall.
> >
> > --
> > Elliott Slaughter
> >
> > "Any road followed precisely to its end leads precisely nowhere." -
> > Frank Herbert
> > _______________________________________________
> > elephant-devel site list
> > elephant-devel at common-lisp.net
> > http://common-lisp.net/mailman/listinfo/elephant-devel
>
>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel
>
>
>
> -- 
> Elliott Slaughter
>
> "Any road followed precisely to its end leads precisely nowhere." -  
> Frank Herbert
> <slot-read-cache- 
> performance.lisp>_______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel