From raison at chatsubo.net  Wed Feb  2 00:52:41 2011
From: raison at chatsubo.net (Kevin Raison)
Date: Tue, 01 Feb 2011 16:52:41 -0800
Subject: [plexippus-xpath-devel] issues with large xml documents
Message-ID: <4D48AAD9.1020409@chatsubo.net>

First, I want to say thanks for this xpath library;  it has kept me from 
being forced to do my work in Java!

Unfortunately, I have run into some issues when trying to handle very 
large documents.  sbcl runs out of stack space when trying simple 
queries (xpath:evaluate "c://ODM/Study" stp-doc) on a document like the 
one here http://chatsubo.net/~raison/test.xml:

Control stack exhausted (no more space for function call frames).
This is probably due to heavily nested or infinitely recursive function
calls, or a tail call that SBCL cannot or has not optimized away.

PROCEED WITH CAUTION.
    [Condition of type SB-KERNEL::CONTROL-STACK-EXHAUSTED]

Restarts:
  0: [RETRY] Retry SLIME REPL evaluation request.
  1: [*ABORT] Return to SLIME's top level.
  2: [TERMINATE-THREAD] Terminate this thread (#<THREAD "repl-thread" 
RUNNING {1003ECC601}>)

Backtrace:
   0: (SB-KERNEL::CONTROL-STACK-EXHAUSTED-ERROR)
   1: ("foreign function: call_into_lisp")
   2: ("foreign function: post_signal_tramp")
   3: (SB-IMPL::GET2 :INVALID-VALUE-FOR-UNESCAPED-REGISTER-STORAGE 
:INVALID-VALUE-FOR-UNESCAPED-REGISTER-STORAGE)
   4: (SB-C::CLASS-INFO-OR-LOSE :FUNCTION)
   5: (SB-C::TYPE-INFO-OR-LOSE :FUNCTION :DEFINITION)
   6: (SB-INT:INFO :FUNCTION :DEFINITION XPATH::AXIS-DESCENDANT-OR-SELF NIL)
   7: (SB-KERNEL:FDEFINITION-OBJECT XPATH::AXIS-DESCENDANT-OR-SELF NIL)
   8: (SB-KERNEL:%COERCE-CALLABLE-TO-FUN XPATH::AXIS-DESCENDANT-OR-SELF)
   9: (XPATH::MAPPEND-PIPE XPATH::AXIS-DESCENDANT-OR-SELF 
(#.(CXML-STP:TEXT #| :PARENT of type ELEMENT |# :DATA "\n        ") . 
#<CLOSURE # {100C2DD279}>))
  10: (XPATH::MAPPEND-PIPE ..)
  11: (XPATH::MAPPEND-PIPE ..)
  12: (XPATH::PIPE-TAIL ..)
  13: ((LAMBDA ()))
  14: (XPATH::PIPE-TAIL ..)
  15: ((LAMBDA ()))
  16: (XPATH::PIPE-TAIL ..)
  17: ((LAMBDA ()))
  18: (XPATH::PIPE-TAIL ..)
  19: ((LAMBDA ()))
  20: (XPATH::PIPE-TAIL ..)
  21: ((LAMBDA ()))
  22: (XPATH::PIPE-TAIL ..)
  23: ((LAMBDA ()))
  24: (XPATH::PIPE-TAIL ..)
  25: ((LAMBDA ()))
  26: (XPATH::PIPE-TAIL ..)
  27: (XPATH::MAPPEND-PIPE ..)

If you follow the trace, it is XPATH::MAPPEND-PIPEs all the way down. 
Any help is appreciated.  I have already tried upping my stack space 
size by a factor of 5 (to 10 megs) with no luck.

Thanks,
Kevin



From david at lichteblau.com  Wed Feb  2 17:31:09 2011
From: david at lichteblau.com (David Lichteblau)
Date: Wed, 2 Feb 2011 18:31:09 +0100
Subject: [plexippus-xpath-devel] issues with large xml documents
In-Reply-To: <4D48AAD9.1020409@chatsubo.net>
References: <4D48AAD9.1020409@chatsubo.net>
Message-ID: <20110202173109.GB19031@radon>

Hi,

Quoting Kevin Raison (raison at chatsubo.net):
> First, I want to say thanks for this xpath library;  it has kept me from 
> being forced to do my work in Java!
> 
> Unfortunately, I have run into some issues when trying to handle very 
> large documents.  sbcl runs out of stack space when trying simple 
> queries (xpath:evaluate "c://ODM/Study" stp-doc) on a document like the 
> one here http://chatsubo.net/~raison/test.xml:
[...]
> If you follow the trace, it is XPATH::MAPPEND-PIPEs all the way down. 
> Any help is appreciated.  I have already tried upping my stack space 
> size by a factor of 5 (to 10 megs) with no luck.

thanks for the report.  I have pushed the following fix to darcs:

diff -rN -u old-plexippus-xpath/pipes.lisp new-plexippus-xpath/pipes.lisp
--- old-plexippus-xpath/pipes.lisp	2011-02-02 18:27:07.000000000 +0100
+++ new-plexippus-xpath/pipes.lisp	2011-02-02 18:27:07.000000000 +0100
@@ -128,12 +128,19 @@
             (make-pipe result (map-pipe-filtering fn tail filter-test))
             (map-pipe-filtering fn tail filter-test)))))
 
-(defun append-pipes (pipex pipey)
-  "return a pipe that appends two pipes"
+(defmacro append-pipes (pipex pipey)
+  "return a pipe that appends two pipes.
+
+   The evaluation style of this macro has been chosen to be consistent
+   with MAKE-PIPE: The first argument is evaluated eagerly; the second
+   argument lazily."
+  `(%append-pipes ,pipex #'(lambda () ,pipey)))
+
+(defun %append-pipes (pipex pipey-fun)
   (if (pipe-empty-p pipex)
-      pipey
+      (funcall pipey-fun)
       (make-pipe (pipe-head pipex)
-                 (append-pipes (pipe-tail pipex) pipey))))
+                 (append-pipes (pipe-tail pipex) (funcall pipey-fun)))))
 
 (defun mappend-pipe (fn pipe)
   "lazily map fn over pipe, appending results"
diff -rN -u old-plexippus-xpath/xpath-test.lisp new-plexippus-xpath/xpath-test.lisp
--- old-plexippus-xpath/xpath-test.lisp	2011-02-02 18:27:07.000000000 +0100
+++ new-plexippus-xpath/xpath-test.lisp	2011-02-02 18:27:07.000000000 +0100
@@ -627,3 +627,12 @@
     (assert-equal
      '(("a" (("id" "1"))) ("c" (("id" "6"))))
      (all-nodes (evaluate "//c[position()=2]|//a[@id='1']" d)))))
+
+(deftest mappend-lazy
+  (let* ((x nil)
+         (pipe (mappend-pipe (lambda (x) (list (- x)))
+                             (make-pipe 1 (progn (setf x t) '(2))))))
+    (assert (not x))
+    (assert (eql -1 (pipe-head pipe)))
+    (assert (eql -2 (pipe-head (pipe-tail pipe))))
+    (assert x)))


d.



From raison at chatsubo.net  Wed Feb  2 19:30:57 2011
From: raison at chatsubo.net (Kevin Raison)
Date: Wed, 02 Feb 2011 11:30:57 -0800
Subject: [plexippus-xpath-devel] issues with large xml documents
In-Reply-To: <20110202173109.GB19031@radon>
References: <4D48AAD9.1020409@chatsubo.net> <20110202173109.GB19031@radon>
Message-ID: <4D49B0F1.6000807@chatsubo.net>

Thanks, David.  This solves my problems!

-Kevin


On 02/02/2011 09:31 AM, David Lichteblau wrote:
> Hi,
>
> Quoting Kevin Raison (raison at chatsubo.net):
>> First, I want to say thanks for this xpath library;  it has kept me from
>> being forced to do my work in Java!
>>
>> Unfortunately, I have run into some issues when trying to handle very
>> large documents.  sbcl runs out of stack space when trying simple
>> queries (xpath:evaluate "c://ODM/Study" stp-doc) on a document like the
>> one here http://chatsubo.net/~raison/test.xml:
> [...]
>> If you follow the trace, it is XPATH::MAPPEND-PIPEs all the way down.
>> Any help is appreciated.  I have already tried upping my stack space
>> size by a factor of 5 (to 10 megs) with no luck.
>
> thanks for the report.  I have pushed the following fix to darcs:
>
> diff -rN -u old-plexippus-xpath/pipes.lisp new-plexippus-xpath/pipes.lisp
> --- old-plexippus-xpath/pipes.lisp	2011-02-02 18:27:07.000000000 +0100
> +++ new-plexippus-xpath/pipes.lisp	2011-02-02 18:27:07.000000000 +0100
> @@ -128,12 +128,19 @@
>               (make-pipe result (map-pipe-filtering fn tail filter-test))
>               (map-pipe-filtering fn tail filter-test)))))
>
> -(defun append-pipes (pipex pipey)
> -  "return a pipe that appends two pipes"
> +(defmacro append-pipes (pipex pipey)
> +  "return a pipe that appends two pipes.
> +
> +   The evaluation style of this macro has been chosen to be consistent
> +   with MAKE-PIPE: The first argument is evaluated eagerly; the second
> +   argument lazily."
> +  `(%append-pipes ,pipex #'(lambda () ,pipey)))
> +
> +(defun %append-pipes (pipex pipey-fun)
>     (if (pipe-empty-p pipex)
> -      pipey
> +      (funcall pipey-fun)
>         (make-pipe (pipe-head pipex)
> -                 (append-pipes (pipe-tail pipex) pipey))))
> +                 (append-pipes (pipe-tail pipex) (funcall pipey-fun)))))
>
>   (defun mappend-pipe (fn pipe)
>     "lazily map fn over pipe, appending results"
> diff -rN -u old-plexippus-xpath/xpath-test.lisp new-plexippus-xpath/xpath-test.lisp
> --- old-plexippus-xpath/xpath-test.lisp	2011-02-02 18:27:07.000000000 +0100
> +++ new-plexippus-xpath/xpath-test.lisp	2011-02-02 18:27:07.000000000 +0100
> @@ -627,3 +627,12 @@
>       (assert-equal
>        '(("a" (("id" "1"))) ("c" (("id" "6"))))
>        (all-nodes (evaluate "//c[position()=2]|//a[@id='1']" d)))))
> +
> +(deftest mappend-lazy
> +  (let* ((x nil)
> +         (pipe (mappend-pipe (lambda (x) (list (- x)))
> +                             (make-pipe 1 (progn (setf x t) '(2))))))
> +    (assert (not x))
> +    (assert (eql -1 (pipe-head pipe)))
> +    (assert (eql -2 (pipe-head (pipe-tail pipe))))
> +    (assert x)))
>
>
> d.
>



From andrew.pennebaker at gmail.com  Wed Feb  9 22:39:01 2011
From: andrew.pennebaker at gmail.com (Andrew Pennebaker)
Date: Wed, 9 Feb 2011 17:39:01 -0500
Subject: [plexippus-xpath-devel] Plexippus and CLISP
Message-ID: <AANLkTimy2CuWzvYxW7vU4EQR_HUCWO-jip_ED+atMmZA@mail.gmail.com>

I'm having trouble building cxml-stp, because xpath isn't quite working.

I realize that Plexippus XPath is not designed for CLISP, but I think we can
work together to change that.

Specs:

cxml-stp 20101107

Quicklisp 2010121400

CLISP 2.49

MacPorts 1.9.2

Mac OS X 10.6.6

MacBook Pro 5,1

Trace:

[1]> (ql:quickload "cxml-stp")
To load "cxml-stp":
  Load 5 ASDF systems:
    alexandria babel cl-ppcre puri trivial-gray-streams
  Install 6 Quicklisp releases:
    cl-yacc closure-common cxml cxml-stp parse-number
    plexippus-xpath
; Fetching
#<URL
  "http://beta.quicklisp.org/archive/cl-yacc/2010-10-06/cl-yacc-20101006-darcs.tgz">
; 18.33KB
==================================================
18,774 bytes in 0.04 seconds (453.17KB/sec)
; Fetching
#<URL
  "http://beta.quicklisp.org/archive/parse-number/2010-10-06/parse-number-1.0.tgz">
; 3.64KB
==================================================
3,726 bytes in 0.00 seconds (751.17KB/sec)
; Fetching
#<URL
  "http://beta.quicklisp.org/archive/plexippus-xpath/2010-10-06/plexippus-xpath-20101006-darcs.tgz">
; 54.64KB
==================================================
55,953 bytes in 0.12 seconds (451.61KB/sec)
; Fetching
#<URL
  "http://beta.quicklisp.org/archive/closure-common/2010-11-07/closure-common-20101107-git.tgz">
; 26.95KB
==================================================
27,601 bytes in 0.05 seconds (587.12KB/sec)
; Fetching
#<URL
  "http://beta.quicklisp.org/archive/cxml/2010-11-07/cxml-20101107-git.tgz">
; 151.09KB
==================================================
154,716 bytes in 0.32 seconds (479.20KB/sec)
; Fetching
#<URL
  "http://beta.quicklisp.org/archive/cxml-stp/2010-11-07/cxml-stp-20101107-git.tgz">
; 50.82KB
==================================================
52,036 bytes in 0.10 seconds (531.95KB/sec)
; Loading "cxml-stp"
;;; Checking for wide character support... WARNING: Lisp
implementation doesn't use UTF-16, but accepts surrogate code points.
 yes, using code points.
;;; Checking for wide character support... WARNING: Lisp
implementation doesn't use UTF-16, but accepts surrogate code points.
 yes, using code points.
;;; Building Closure with CHARACTER RUNES
[package yacc]....................................
[package cl-ppcre]................................
[package org.mapcar.parse-number].................
[package runes]...................................
[package utf8-runes]..............................
[package runes-encoding]..........................
[package hax].....................................
[package cxml]....................................
[package sax].....................................
[package cxml-xmls]...............................
[package dom].....................................
[package rune-dom]................................
[package klacks]..................................
[package domtest].................................
[package domtest-tests]...........................
[package xmlconf].................................
[package xpath-protocol]..........................
[package xpattern]................................
[package xpath]...................................
[package xpath-sys]...........
*** - LOOP: syntax error after HASH-KEY in
       (LOOP FOR KEY BEING EACH HASH-KEY USING (HASH-VALUE ACCUM) IN TABLE IF
        (AND (CONSP KEY) (EQ (CAR KEY) 'PATTERN)) COLLECT (CONS KEY ACCUM) INTO
        PATTERNS ELSE COLLECT (CONS KEY ACCUM) INTO EXPRESSIONS FINALLY
        (RETURN
         (VALUES (SORT EXPRESSIONS #'> :KEY #'SECOND)
          (SORT PATTERNS #'> :KEY #'SECOND))))
The following restarts are available:
TRY-RECOMPILING :R1     Try recompiling profile
RETRY          :R2      Retry compiling component ("xpath" "profile").
ACCEPT         :R3      Continue, treating compiling component
("xpath" "profile") as having been successful.
ABORT          :R4      Give up on "cxml-stp"
ABORT          :R5      Abort main loop
Break 1 XPATH[2]> :r2

*** - LOOP: syntax error after HASH-KEY in
       (LOOP FOR KEY BEING EACH HASH-KEY USING (HASH-VALUE ACCUM) IN TABLE IF
        (AND (CONSP KEY) (EQ (CAR KEY) 'PATTERN)) COLLECT (CONS KEY ACCUM) INTO
        PATTERNS ELSE COLLECT (CONS KEY ACCUM) INTO EXPRESSIONS FINALLY
        (RETURN
         (VALUES (SORT EXPRESSIONS #'> :KEY #'SECOND)
          (SORT PATTERNS #'> :KEY #'SECOND))))
The following restarts are available:
TRY-RECOMPILING :R1     Try recompiling profile
RETRY          :R2      Retry compiling component ("xpath" "profile").
ACCEPT         :R3      Continue, treating compiling component
("xpath" "profile") as having been successful.
ABORT          :R4      Give up on "cxml-stp"
ABORT          :R5      Abort main loop

Cheers,

Andrew Pennebaker

http://www.yellosoft.us/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.common-lisp.net/pipermail/plexippus-xpath-devel/attachments/20110209/f8527eee/attachment.html>

From david at lichteblau.com  Wed Feb  9 22:58:05 2011
From: david at lichteblau.com (David Lichteblau)
Date: Wed, 9 Feb 2011 23:58:05 +0100
Subject: [plexippus-xpath-devel] Plexippus and CLISP
In-Reply-To: <AANLkTimy2CuWzvYxW7vU4EQR_HUCWO-jip_ED+atMmZA@mail.gmail.com>
References: <AANLkTimy2CuWzvYxW7vU4EQR_HUCWO-jip_ED+atMmZA@mail.gmail.com>
Message-ID: <20110209225805.GA23729@radon>

Quoting Andrew Pennebaker (andrew.pennebaker at gmail.com):
> I'm having trouble building cxml-stp, because xpath isn't quite working.
> 
> I realize that Plexippus XPath is not designed for CLISP, but I think we can
> work together to change that.
[...]

I have just pushed a patch by Jochen Schmidt which should fix this.
Please test.


d.