[plexippus-xpath-devel] issues with large xml documents

[David Lichteblau]

Hi,

Quoting Kevin Raison (raison at chatsubo.net):
> First, I want to say thanks for this xpath library;  it has kept me from 
> being forced to do my work in Java!
> 
> Unfortunately, I have run into some issues when trying to handle very 
> large documents.  sbcl runs out of stack space when trying simple 
> queries (xpath:evaluate "c://ODM/Study" stp-doc) on a document like the 
> one here http://chatsubo.net/~raison/test.xml:
[...]
> If you follow the trace, it is XPATH::MAPPEND-PIPEs all the way down. 
> Any help is appreciated.  I have already tried upping my stack space 
> size by a factor of 5 (to 10 megs) with no luck.

thanks for the report.  I have pushed the following fix to darcs:

diff -rN -u old-plexippus-xpath/pipes.lisp new-plexippus-xpath/pipes.lisp
--- old-plexippus-xpath/pipes.lisp	2011-02-02 18:27:07.000000000 +0100
+++ new-plexippus-xpath/pipes.lisp	2011-02-02 18:27:07.000000000 +0100
@@ -128,12 +128,19 @@
             (make-pipe result (map-pipe-filtering fn tail filter-test))
             (map-pipe-filtering fn tail filter-test)))))
 
-(defun append-pipes (pipex pipey)
-  "return a pipe that appends two pipes"
+(defmacro append-pipes (pipex pipey)
+  "return a pipe that appends two pipes.
+
+   The evaluation style of this macro has been chosen to be consistent
+   with MAKE-PIPE: The first argument is evaluated eagerly; the second
+   argument lazily."
+  `(%append-pipes ,pipex #'(lambda () ,pipey)))
+
+(defun %append-pipes (pipex pipey-fun)
   (if (pipe-empty-p pipex)
-      pipey
+      (funcall pipey-fun)
       (make-pipe (pipe-head pipex)
-                 (append-pipes (pipe-tail pipex) pipey))))
+                 (append-pipes (pipe-tail pipex) (funcall pipey-fun)))))
 
 (defun mappend-pipe (fn pipe)
   "lazily map fn over pipe, appending results"
diff -rN -u old-plexippus-xpath/xpath-test.lisp new-plexippus-xpath/xpath-test.lisp
--- old-plexippus-xpath/xpath-test.lisp	2011-02-02 18:27:07.000000000 +0100
+++ new-plexippus-xpath/xpath-test.lisp	2011-02-02 18:27:07.000000000 +0100
@@ -627,3 +627,12 @@
     (assert-equal
      '(("a" (("id" "1"))) ("c" (("id" "6"))))
      (all-nodes (evaluate "//c[position()=2]|//a[@id='1']" d)))))
+
+(deftest mappend-lazy
+  (let* ((x nil)
+         (pipe (mappend-pipe (lambda (x) (list (- x)))
+                             (make-pipe 1 (progn (setf x t) '(2))))))
+    (assert (not x))
+    (assert (eql -1 (pipe-head pipe)))
+    (assert (eql -2 (pipe-head (pipe-tail pipe))))
+    (assert x)))


d.