[elephant-devel] gp-export strikes back

[Alex Mizrahi]

 HH> I have an idea: we could have a second elephant database used only for
 HH> reference tracking. It does not have to have the same backend,
 HH> bdb will probably be the best choice at all time. This should be easy
 HH> to implement and avoid the problem completely?

There is a catch -- s-serialization uses reference tracking both for 
conses/structures/arrays and for objects. But database will only work for 
persistent objects -- their identity is preserved because they have unique 
oids. So to do this, we need to separate persistent object references 
tracking from the rest of reference tracking.

Reference tracking for non-persistent objects will have to use different 
semantics -- instead of tracking them globally tracking should be done 
individually for each piece of information, like each slot or btree entry. 
Then it will follow Elephant's serialization semantics.

It is possible to implement this, and, perhaps, it is the only way how it 
can work, but I can't say it is particularly easy to implement.

I think it is better to focus on getting it to work correctly in this 
release. It is pretty complex as it is now, and if some additional 
facilities are added to support large databases, reliability might suffer.

So I'm asking here -- are there Elephant users who absolutely need this 
feature?

RAM have become really cheap in the last years, for example, it is hard to 
find laptop with less than 2 GB of RAM. And gigabyte is quite a lot of data. 
So I think a problem with databases-not-fitting-in-RAM is much less relevant 
than it was, say, 10 years ago.

 HH> Make we can make its use an option, after all if a hashtable works for
 HH> you there should be no need to configure a "backup-helper-db".

Another option would be to use the very same store you're backing up -- then 
it needs no configuration at all. I don't know, however, if database will 
tolerate creating a huge btree/table without commiting a transaction.