[elephant-devel] Re: Severely poor performance in some obvious cases

[Robert L. Read]

On Fri, 2007-11-30 at 10:36 -0500, Ian Eslick wrote:
> Is there any reason that we can't store the byte-stream data directly  
> in postmodern?  We already have an efficient, mostly non-consing byte- 
> array serializer with the following format:
> 
> [btree_id][data_type][data_format]
> 
> If you used a new table for each btree, then you could strip the  
> btree_id and pass the type + format to postgres.  Integers are stored  
> big-endian and strings in left to write lisp-code order so might just  
> work without modification.
I think I was wrong about the base64 stuff; it is used by CL-SQL but not
by postmodern.  Sorry for the confusion.


> 
> However I'm speculating as I don't understand all the issues.  As I  
> understand it, CL-SQL has the problem that the byte storage methods  
> for different SQL engines are different enough to make a common API  
> difficult to implement.

Yes, that is correct, but postmodern has done better and eliminated this
problem.

Nonetheless, it seems by experiment that our serializer is very, very
consing.
> 
> Ian
> 
> 
> On Nov 29, 2007, at 10:06 AM, Robert L. Read wrote:
> 
> > On Thu, 2007-11-29 at 16:16 +0200, Alex Mizrahi wrote:
> >> AP> Am I missing something really basic here?
> >>
> >> actually it's quite strange situation that you have *many*  
> >> employees with
> >> same name but you want just one (random one). i cannot imagine why  
> >> one needs
> >> this in real world..
> >>
> >> or you're saying that all have different names, but it still does  
> >> consing?
> >> this could be a bug then..
> >>
> >
> > With respect to consing, it is important to point out that our
> > serializer is very consing (for postmodern and CL-SQL backends.)  This
> > is because I used base64 to transform the byte-streams into character
> > strings.
> >
> > Most relational databases (including Postgres) provide a way of  
> > storing
> > byte sequences directly.  However, this is not standardized and not
> > portable.  In fact, I spoke to Kevin Rosenberg, the author of CL-SQL,
> > and he and CL-SQL don't have a good way to do it.
> >
> > However, since postmodern is Postgres specific, it could avoid this
> > step, by using a back-end specific serializer.  I suspect this would
> > have a huge impact on performance, both by decreasing consing (minor)
> > and by decreasing the amount of disc I/O that has to be done (major).
> >
> > (BDB doesn't have this problem, because it natively uses byte- 
> > sequences,
> > not character-sequences.)
> >
> > Please see the code below, which demonstrates that pushing 1 million
> > bytes through the serializer (without even going to the database)
> > creates 8 million bytes of garbage in 0.433 seconds. (This is on a  
> > new,
> > fast, 2 gigabyte 64-bit machine, against postmodern:
> >
> > asdf:operate 'asdf:load-op :elephant)
> > (asdf:operate 'asdf:load-op :ele-clsql)
> > (asdf:operate 'asdf:load-op :postmodern)
> >
> > (asdf:operate 'asdf:load-op :elephant-tests)
> > (in-package "ELEPHANT-TESTS")
> >
> > (setq *default-spec* *testpm-spec*)
> >
> > (setq teststring "supercalifragiliciousexpialidocious")
> > (setq testint 42)
> >
> > (setq totalseriazationload (* 1000 1000))
> >
> > (setq n (ceiling (/ totalseriazationload (length teststring))))
> >
> > (open-store *default-spec*)
> >
> > (time
> > (dotimes (x n)
> >   (in-out-value teststring)))
> >
> > (close-store)
> >
> > *****
> > Results in:
> > Evaluation took:
> >  0.433 seconds of real time
> >  0.172974 seconds of user run time
> >  0.058991 seconds of system run time
> >  0 calls to %EVAL
> >  0 page faults and
> >  8,731,728 bytes consed.
> > NIL
> > ELE-TESTS>
> >
> > I personally think making a back-end specific serializer to avoid the
> > base64 encoding would make a significant performance difference.  This
> > is not much of an issue for me personally, since I keep everything
> > cached in memory anyway.
> >
> >
> > -- 
> > Robert L. Read, PhD
> > http://konsenti.com
> >
> > _______________________________________________
> > elephant-devel site list
> > elephant-devel at common-lisp.net
> > http://common-lisp.net/mailman/listinfo/elephant-devel