[plexippus-xpath-devel] issues with large xml documents

[Kevin Raison]

Thanks, David.  This solves my problems!

-Kevin


On 02/02/2011 09:31 AM, David Lichteblau wrote:
> Hi,
>
> Quoting Kevin Raison (raison at chatsubo.net):
>> First, I want to say thanks for this xpath library;  it has kept me from
>> being forced to do my work in Java!
>>
>> Unfortunately, I have run into some issues when trying to handle very
>> large documents.  sbcl runs out of stack space when trying simple
>> queries (xpath:evaluate "c://ODM/Study" stp-doc) on a document like the
>> one here http://chatsubo.net/~raison/test.xml:
> [...]
>> If you follow the trace, it is XPATH::MAPPEND-PIPEs all the way down.
>> Any help is appreciated.  I have already tried upping my stack space
>> size by a factor of 5 (to 10 megs) with no luck.
>
> thanks for the report.  I have pushed the following fix to darcs:
>
> diff -rN -u old-plexippus-xpath/pipes.lisp new-plexippus-xpath/pipes.lisp
> --- old-plexippus-xpath/pipes.lisp	2011-02-02 18:27:07.000000000 +0100
> +++ new-plexippus-xpath/pipes.lisp	2011-02-02 18:27:07.000000000 +0100
> @@ -128,12 +128,19 @@
>               (make-pipe result (map-pipe-filtering fn tail filter-test))
>               (map-pipe-filtering fn tail filter-test)))))
>
> -(defun append-pipes (pipex pipey)
> -  "return a pipe that appends two pipes"
> +(defmacro append-pipes (pipex pipey)
> +  "return a pipe that appends two pipes.
> +
> +   The evaluation style of this macro has been chosen to be consistent
> +   with MAKE-PIPE: The first argument is evaluated eagerly; the second
> +   argument lazily."
> +  `(%append-pipes ,pipex #'(lambda () ,pipey)))
> +
> +(defun %append-pipes (pipex pipey-fun)
>     (if (pipe-empty-p pipex)
> -      pipey
> +      (funcall pipey-fun)
>         (make-pipe (pipe-head pipex)
> -                 (append-pipes (pipe-tail pipex) pipey))))
> +                 (append-pipes (pipe-tail pipex) (funcall pipey-fun)))))
>
>   (defun mappend-pipe (fn pipe)
>     "lazily map fn over pipe, appending results"
> diff -rN -u old-plexippus-xpath/xpath-test.lisp new-plexippus-xpath/xpath-test.lisp
> --- old-plexippus-xpath/xpath-test.lisp	2011-02-02 18:27:07.000000000 +0100
> +++ new-plexippus-xpath/xpath-test.lisp	2011-02-02 18:27:07.000000000 +0100
> @@ -627,3 +627,12 @@
>       (assert-equal
>        '(("a" (("id" "1"))) ("c" (("id" "6"))))
>        (all-nodes (evaluate "//c[position()=2]|//a[@id='1']" d)))))
> +
> +(deftest mappend-lazy
> +  (let* ((x nil)
> +         (pipe (mappend-pipe (lambda (x) (list (- x)))
> +                             (make-pipe 1 (progn (setf x t) '(2))))))
> +    (assert (not x))
> +    (assert (eql -1 (pipe-head pipe)))
> +    (assert (eql -2 (pipe-head (pipe-tail pipe))))
> +    (assert x)))
>
>
> d.
>