[elephant-devel] Re: QDBM Support

[Alex Mizrahi]

 JD> hassles. I don't really know anything about BDB, but I'm surprised the
 JD> performance of a properly indexed SQL database can't get close to it.

BDB works directly with files, but SQL backend working with PostgreSQL or 
database like this needs to send queries over network -- that has 
significant overhead, something like millisecond or so.
this practically means you cannot make more than few thousands queries per 
second, that is quite prohibitive.
this problem does not exist when you use SQL database directly -- people 
write complex queries with it and retrieve exactly what do they want in one 
pass.
but we have to do query on each slot access, that's much worse..

in db-postmodern backend we've implemented btree lisp-side cache for this --  
it can reduce query count greatly, but of course it works only on some 
workloads.

 JD> If the performance with SQLite is poor, has it been verified to be
 JD> only a SQLite issue and not something related to CLSQL?

OTOH SQLite does not have this problem. that's why it can be much faster 
than PostgreSQL.

however, as far as i remember, elephant's code that works with CLSQL works 
in extremely non-optimal way in some cases -- to retrieve single value from 
index via cursor, it retrieves all contents of index, sorts it, and 
retrieves that single value. but if application does not need range queries, 
this might work just fine, since some time ago we've optimized 
get-instance-by-value to get value with direct query to btree.

so i'd advice to do benchmarking before coming to conclusions. if db-clsql + 
sqlite will work fine for given size of data base and queries, why should 
one need something else?