<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 TRANSITIONAL//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; CHARSET=UTF-8">
<META NAME="GENERATOR" CONTENT="GtkHTML/3.3.2">
</HEAD>
<BODY>
Thanks, Andrew, this is very interesting.<BR>
<BR>
I would mention with respect to speed:<BR>
<BR>
1) BerkeleyDB is much faster than the SQL implementations right now,<BR>
2) My base64-based serialization for the SQL backend could obviously <BR>
be improved tremendously.<BR>
<BR>
Improving my serialization would could down on disk space and time in the <BR>
SQL-back-end version a great deal. Even the BerkeleyDB serializer could be<BR>
improved, although it is much better than what I am doing for the SQL-back end.<BR>
<BR>
I hope you will keep us informed as to your experience and your choices; if <BR>
you choose not to use Elephant, that will be interesting information as well.<BR>
<BR>
For the substring searching, are you planning to use a "like" query in MySQL<BR>
in your homegrown implementation? I doubt that Elephant can compete with <BR>
that; although it can produce functional indexes that technically have that power,<BR>
it would be pretty sophisticated programming to do it.<BR>
<BR>
<BR>
<BR>
On Sat, 2006-01-14 at 12:09 -0800, Andrew Philpot wrote:
<BLOCKQUOTE TYPE=CITE>
<PRE>
<FONT COLOR="#000000">Richard asked if I was still using Elephant. I thought I would send a</FONT>
<FONT COLOR="#000000">quick note describing my experience using Elephant to persistently</FONT>
<FONT COLOR="#000000">store a lisp-based ontology with multi-lingual lexical attachments.</FONT>
<FONT COLOR="#000000">The "native" lisp implementation is pretty simple. Essentially, it is</FONT>
<FONT COLOR="#000000">a semantic network implemented in Lisp using symbols for nodes and</FONT>
<FONT COLOR="#000000">using property lists for (binary) relations. One additional wrinkle</FONT>
<FONT COLOR="#000000">is that a given string might name more than one ontological element,</FONT>
<FONT COLOR="#000000">most typically a concept and a possibly related, possibly unrelated</FONT>
<FONT COLOR="#000000">English lexical item ("word"). Accordingly, symbols are interned in</FONT>
<FONT COLOR="#000000">different CL packages according to their linguistic/ontological role</FONT>
<FONT COLOR="#000000">(concept, lexical item, sense, and a few others); all packages</FONT>
<FONT COLOR="#000000">containing a given fragment of the ontology are named per a convention</FONT>
<FONT COLOR="#000000">allowing modular mixing and matching.</FONT>
<FONT COLOR="#000000">A small schematic example:</FONT>
<FONT COLOR="#000000">FRAG1-CONCEPT::|bank<side|</FONT>
<FONT COLOR="#000000"> :DEFINITION "side of a road or river"</FONT>
<FONT COLOR="#000000"> :HAS-SENSE (FRAG1-SENSE::|bank<side:EN:bank| FRAG1-SENSE::|bank<side:ES:banco</FONT>
<FONT COLOR="#000000">|)</FONT>
<FONT COLOR="#000000"> :SUPER FRAG1-CONCEPT::|side<part|</FONT>
<FONT COLOR="#000000"> :SUB FRAG1-INSTANCE::|Left Bank|</FONT>
<FONT COLOR="#000000"> :SOURCE :WORDNET</FONT>
<FONT COLOR="#000000"> :PART-OF-SPEECH :NOUN</FONT>
<FONT COLOR="#000000"> :HAS-SUBJECT-DOMAIN DOM-CONCEPT::|engineering|</FONT>
<FONT COLOR="#000000">FRAG1-SENSE::|bank<side:EN:bank|</FONT>
<FONT COLOR="#000000"> :HAS-CONCEPT FRAG1-CONCEPT:|bank<side|</FONT>
<FONT COLOR="#000000"> :HAS-LEXICAL-ITEM FRAG1-EN::|bank|</FONT>
<FONT COLOR="#000000"> :SOURCE :WORDNET</FONT>
<FONT COLOR="#000000"> :DERIVATION FRAG1-SENSE::|banked<modified::EN:banked|</FONT>
<FONT COLOR="#000000">FRAG1-EN::|bank|</FONT>
<FONT COLOR="#000000"> :HAS-SENSE FRAG1-SENSE::|bank<side:EN:bank|</FONT>
<FONT COLOR="#000000"> :LANGUAGE :EN</FONT>
<FONT COLOR="#000000"> :SOURCE :WORDNET</FONT>
<FONT COLOR="#000000">etc. etc. All in all there are roughly 600-700K of these named</FONT>
<FONT COLOR="#000000">objects in my core ontology base, with millions more instances linked</FONT>
<FONT COLOR="#000000">in below as well as other kinds of ancillary attachments.</FONT>
<FONT COLOR="#000000">The above model using symbols for objects and and property lists for</FONT>
<FONT COLOR="#000000">relations is not very fancy nor "modern", but it's very convenient and</FONT>
<FONT COLOR="#000000">relatively lightweight for an in-core implementation. However,</FONT>
<FONT COLOR="#000000">interned symbols don't get GC'd even when no one cares about them</FONT>
<FONT COLOR="#000000">anymore, so the Lisp address size limits what I can do. Hence</FONT>
<FONT COLOR="#000000">persistence a la Elephant.</FONT>
<FONT COLOR="#000000">My current model for implementing this in Elephant:</FONT>
<FONT COLOR="#000000">A. objects are 3-lists which serve as keys. FRAG1-EN::|bank| becomes</FONT>
<FONT COLOR="#000000">(:frag1 :en "bank"). Algebraic in the sense that I can easily</FONT>
<FONT COLOR="#000000">construct them a view them as a composition of smaller bits; nothing</FONT>
<FONT COLOR="#000000">that can't be GC'd if needed (and the cardinality of namespaces and</FONT>
<FONT COLOR="#000000">language keywords are both expected to be small).</FONT>
<FONT COLOR="#000000">B. Relation network are hash-tables (EQL) keyed by the relation name.</FONT>
<FONT COLOR="#000000">So the first one above is </FONT>
<FONT COLOR="#000000">(add-from-root '(:frag1 :concept "bank<side")</FONT>
<FONT COLOR="#000000"> (let ((ht (make-hash-table :test #'eql)))</FONT>
<FONT COLOR="#000000"> (setf (gethash :DEFINITION ht) "side of a road or river")</FONT>
<FONT COLOR="#000000"> (setf (gethash :HAS-SENSE ht) </FONT>
<FONT COLOR="#000000"> (list '(:FRAG1 :SENSE "bank<side:EN:bank")</FONT>
<FONT COLOR="#000000"> '(:FRAG1 :SENSE "bank<side:ES:banco")))</FONT>
<FONT COLOR="#000000"> (setf (gethash :SUPER ht) '(:FRAG1 :CONCEPT "side<part"))</FONT>
<FONT COLOR="#000000"> (setf (gethash :SUB ht) '(:FRAG1 :INSTANCE "Left Bank"))</FONT>
<FONT COLOR="#000000"> (setf (gethash :SOURCE ht) :WORDNET)</FONT>
<FONT COLOR="#000000"> (setf (gethash :PART-OF-SPEECH ht) :NOUN)</FONT>
<FONT COLOR="#000000"> (setf (gethash :HAS-SUBJECT-DOMAIN ht) '(:DOM :CONCEPT "engineering"))))</FONT>
<FONT COLOR="#000000">I'm reasonably happy with this implementation, but it does have some</FONT>
<FONT COLOR="#000000">shortcomings:</FONT>
<FONT COLOR="#000000">1. Much slower than I expected. To load the above 600K entities into</FONT>
<FONT COLOR="#000000"> Lisp takes about 10 minutes; to load into elephant (presumably a</FONT>
<FONT COLOR="#000000"> one-time operation) takes about a day! Maybe transactions or wiser</FONT>
<FONT COLOR="#000000"> use of cursors or store controllers or what have you is called for,</FONT>
<FONT COLOR="#000000"> but I didn't expect this.</FONT>
<FONT COLOR="#000000">2. Takes a lot of space. Roughly 350 MB of Lisp source files turn</FONT>
<FONT COLOR="#000000"> into 3.6 GB of disk space. Again, maybe I'm doing something</FONT>
<FONT COLOR="#000000"> terribly wrong.</FONT>
<FONT COLOR="#000000">I don't have data at this point about access times. I hope they don't</FONT>
<FONT COLOR="#000000">reflect the load time.</FONT>
<FONT COLOR="#000000">I'm still going forward on this. At this point, elephant is a long</FONT>
<FONT COLOR="#000000">shot contender with a home grown RDBMS serialization back-ended in</FONT>
<FONT COLOR="#000000">MySQL, which works but makes data updates laborious and installation a</FONT>
<FONT COLOR="#000000">chore as well. To replicate the functionality of my other</FONT>
<FONT COLOR="#000000">implementations, I will need some auxiliary indexing, in particular so</FONT>
<FONT COLOR="#000000">I can ask for all objects whose names match a substring; and I will</FONT>
<FONT COLOR="#000000">need some mechanism to handle relation inverses -- I want to say</FONT>
<FONT COLOR="#000000">(:super A B) and be able to ask (:sub B A).</FONT>
<FONT COLOR="#000000">It seems that a CLOS-based model might address some of the issues I</FONT>
<FONT COLOR="#000000">just mentioned. I am kind of worried that it would be even slower.</FONT>
<FONT COLOR="#000000">And of course, I've had some troubles with MOP. I'm not ruling this</FONT>
<FONT COLOR="#000000">out.</FONT>
<FONT COLOR="#000000">Any suggestions anyone might have would be welcome. When I have a</FONT>
<FONT COLOR="#000000">web-based demo, I will also circulate it here.</FONT>
<FONT COLOR="#000000">Thanks,</FONT>
<FONT COLOR="#000000">Andrew Philpot</FONT>
<FONT COLOR="#000000">_______________________________________________</FONT>
<FONT COLOR="#000000">elephant-devel site list</FONT>
<FONT COLOR="#000000"><A HREF="mailto:elephant-devel@common-lisp.net">elephant-devel@common-lisp.net</A></FONT>
<FONT COLOR="#000000"><A HREF="http://common-lisp.net/mailman/listinfo/elephant-devel">http://common-lisp.net/mailman/listinfo/elephant-devel</A></FONT>
</PRE>
</BLOCKQUOTE>
</BODY>
</HTML>