From scaekenberghe at common-lisp.net  Fri Feb 15 14:25:21 2008
From: scaekenberghe at common-lisp.net (Sven Van Caekenberghe)
Date: Fri, 15 Feb 2008 15:25:21 +0100
Subject: [s-xml-rpc-devel] loss of precision on double
In-Reply-To: <83cd66360709101739m5f374910qea62d6f4c9f585c7@mail.gmail.com>
References: <83cd66360709101739m5f374910qea62d6f4c9f585c7@mail.gmail.com>
Message-ID: <7CD443ED-8C75-4D27-9FEB-BCCE66032EF2@common-lisp.net>

Jeff,

On 11 Sep 2007, at 02:39, Jeff Sapp wrote:

> Hey, I was using s-xml-rpc to receive some double values and I noticed
> that was losing precision on them.
>
> I looked a bit closer and I noticed that the function
> decode-xml-rpc-new-element was reading in doubles with
> read-from-string which has a default return type of single-float. I'd
> still consider myself new
> at this lisp business, so my fix probably isn't stellar.
>
> I've also run into some performance issues and I was wondering if  
> anyone
> had some advice. The function I'm having particular trouble with  
> returns
> a list of lists, where the outside list is about 20k elements long,  
> and the
> inside lists around no larger than 20. It takes about two minutes to  
> encode
> the whole thing.
>
> If you had to make the encoding more efficient, where would you start?
>
> Thanks,
> ~jeff
> <patch.diff>_______________________________________________
> s-xml-rpc-devel site list
> s-xml-rpc-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/s-xml-rpc-devel

I commited your patch to CVS as :

(:|double| (let ((*read-eval* nil)
                         (*read-default-float-format* 'double))
                      (read-from-string seed)))

Thanks!

Regarding your performance issue: lists/arrays of 20k elements are a  
lot for a relatively inefficient encoding system like xml-rpc (have a  
look at the xml output). I think there isn't much you can do. If you  
really want to stay with xml-rpc and still want a more compact/ 
efficient format, you could try to go the binary way (base64 encoding).

Sven






From scaekenberghe at common-lisp.net  Fri Feb 15 14:41:46 2008
From: scaekenberghe at common-lisp.net (Sven Van Caekenberghe)
Date: Fri, 15 Feb 2008 15:41:46 +0100
Subject: [s-xml-rpc-devel] loss of precision on double
In-Reply-To: <7CD443ED-8C75-4D27-9FEB-BCCE66032EF2@common-lisp.net>
References: <83cd66360709101739m5f374910qea62d6f4c9f585c7@mail.gmail.com>
	<7CD443ED-8C75-4D27-9FEB-BCCE66032EF2@common-lisp.net>
Message-ID: <476E8C27-D5C8-4034-9BA5-14EC0DAE34D7@common-lisp.net>


On 15 Feb 2008, at 15:25, Sven Van Caekenberghe wrote:

> I commited your patch to CVS as :
>
> (:|double| (let ((*read-eval* nil)
>                        (*read-default-float-format* 'double))
>                     (read-from-string seed)))

Make that 'double-float ;-)



From scaekenberghe at common-lisp.net  Fri Feb 15 15:43:53 2008
From: scaekenberghe at common-lisp.net (Sven Van Caekenberghe)
Date: Fri, 15 Feb 2008 16:43:53 +0100
Subject: [s-xml-rpc-devel] BUG ?
In-Reply-To: <yppps35jzhr.fsf@bug.dn.bus.com.ua>
References: <yppps35jzhr.fsf@bug.dn.bus.com.ua>
Message-ID: <8B040DE3-CB3E-492B-A829-A4B6B28A1421@common-lisp.net>

Evgeniy,

On 06 Jul 2007, at 17:58, Evgeniy Zamriy wrote:

> Hello, All.
>
> Version of SBCL - 1.0.1
> Version of s-xml-rpc - latest from cvs
>
> I found possible bug in library:
>
> * (decode-xml-rpc (make-string-input-stream "<value><boolean>1</ 
> boolean></value>"))
> * T
>
>
> * (decode-xml-rpc (make-string-input-stream "<value><boolean>0</ 
> boolean></value>"))
> * ""
>
> So, when my client resieves "false" from server I get "" except of  
> nil. Is this
> bug or feature ? :) I tryed to fix it, but my experience in lisp  
> isn't well.
>
>
> P.S.: my english is bad, sorry :)
>
> --
> Best regards, Evgeniy Zamriy

Thanks for reporting this problem.
I added your examples to the unit tests.
I applied a fix that I hope will stand, everything seems ok for the  
moment.
Currently this is in cvs only,

Regards,

Sven



From whalliburton at gmail.com  Sun Feb 17 01:35:19 2008
From: whalliburton at gmail.com (William Halliburton)
Date: Sat, 16 Feb 2008 20:35:19 -0500
Subject: [s-xml-rpc-devel] significant speed increase in SBCL
Message-ID: <4e7bd29e0802161735y2e79f285p82d0a31e496d19e1@mail.gmail.com>

Hello all.  Using s-xml-rpc with SBCL 1.0.14 and was seeing a lot of
consing.
Just attempted to hit some low hanging fruit and got lucky.

Please excuse any wierd table cut-and-paste formatting below.

I am using hunchentoot as the front end and was calling as such:

(handle-xml-rpc-call (raw-post-data :want-stream t) id)

Profiling gave: (note that the lisp test system was calling itself so both
the call and answer are shown,
but I really only care about the answer.)

;;   seconds  |    consed   |  calls |  sec/call  |  name
;; ----------------------------------------------------------
;;     10.430 | 107,109,496 |     32 |   0.325942 | S-XML-RPC:XML-RPC-CALL
;;      5.583 |  57,897,032 |  1,670 |   0.003343 | S-XML::PARSE-IDENTIFIER
;;      1.179 |  12,740,176 |    835 |   0.001412 | S-XML::PARSE-XML-ELEMENT
;;      1.073 |   9,625,512 |  1,845 |   0.000582 | S-XML::PARSE-WHITESPACE
;;      0.988 |   8,591,704 |    239 |   0.004132 | S-XML::PARSE-TEXT
;;      0.425 |   4,491,216 |    899 |   0.000472 | S-XML::SKIP-WHITESPACE
;;      0.111 |      53,968 |  1,670 |   0.000067 | S-XML:RESOLVE-IDENTIFIER
;;      0.041 |           0 | 11,810 |   0.000003 | S-XML::IDENTIFIER-CHAR-P
;;      0.028 |     516,088 |     64 |   0.000436 | S-XML-RPC::FORMAT-HEADER
;;      0.021 |         720 |  1,670 |   0.000013 |
S-XML::FIND-NAMESPACE-BINDING
;;      0.021 |           0 |  1,845 |   0.000011 | S-XML::GET-BUFFER
;;      0.013 |           0 |  1,670 |   0.000008 | S-XML:SPLIT-IDENTIFIER
;;      0.011 |           0 |    835 |   0.000013 |
S-XML-RPC::DECODE-XML-RPC-FINISH-ELEMENT


When I changed handle-xml-rpc-call to take a string as make its own stream
as such:

(handle-xml-rpc-call-string (raw-post-data) id)


(defun handle-xml-rpc-call-string (string id)
    (with-input-from-string (in string)
      (let ((call (decode-xml-rpc in)))
        (let ((result (apply *xml-rpc-call-hook*
                             (first call)
                             (rest call))))
          (encode-xml-rpc-result result))))))

I now profile:

  seconds  |   consed   |  calls |  sec/call  |  name
---------------------------------------------------------
     0.985 | 15,135,160 |     32 |   0.030786 | S-XML-RPC:XML-RPC-CALL
     0.023 |    118,296 |  1,670 |   0.000014 | S-XML:RESOLVE-IDENTIFIER
     0.021 |          0 | 11,810 |   0.000002 | S-XML::IDENTIFIER-CHAR-P
     0.019 |    769,856 |     32 |   0.000593 | RPC-HANDLER
     0.016 |    459,688 |     64 |   0.000248 | S-XML-RPC::FORMAT-HEADER
     0.013 |     11,128 |    899 |   0.000014 | S-XML::SKIP-WHITESPACE
     0.010 |     12,536 |  1,670 |   0.000006 | S-XML::PARSE-IDENTIFIER
     0.009 |          0 |  1,670 |   0.000006 | S-XML::GET-MINI-BUFFER
     0.009 |          0 |  1,670 |   0.000006 | S-XML:SPLIT-IDENTIFIER
     0.004 |          0 |      8 |   0.000498 | UPDATE-SEARCH-STRING
     0.004 |      7,944 |     32 |   0.000118 |
S-XML-RPC::ENCODE-XML-RPC-RESULT

For a 10 fold speed increase.

I just thought this might be of use to someone down the line.

Will
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.common-lisp.net/pipermail/s-xml-rpc-devel/attachments/20080216/fc62c605/attachment.html>

From scaekenberghe at common-lisp.net  Sun Feb 17 13:15:23 2008
From: scaekenberghe at common-lisp.net (Sven Van Caekenberghe)
Date: Sun, 17 Feb 2008 14:15:23 +0100
Subject: [s-xml-rpc-devel] significant speed increase in SBCL
In-Reply-To: <4e7bd29e0802161735y2e79f285p82d0a31e496d19e1@mail.gmail.com>
References: <4e7bd29e0802161735y2e79f285p82d0a31e496d19e1@mail.gmail.com>
Message-ID: <B32BDB73-1356-4600-B020-A2B096815332@common-lisp.net>

Will,

On 17 Feb 2008, at 02:35, William Halliburton wrote:

> Hello all.  Using s-xml-rpc with SBCL 1.0.14 and was seeing a lot of  
> consing.
> Just attempted to hit some low hanging fruit and got lucky.
>
> Please excuse any wierd table cut-and-paste formatting below.
>
> I am using hunchentoot as the front end and was calling as such:
>
> (handle-xml-rpc-call (raw-post-data :want-stream t) id)
>
> Profiling gave: (note that the lisp test system was calling itself  
> so both the call and answer are shown,
> but I really only care about the answer.)
>
> ;;   seconds  |    consed   |  calls |  sec/call  |  name
> ;; ----------------------------------------------------------
> ;;     10.430 | 107,109,496 |     32 |   0.325942 | S-XML-RPC:XML- 
> RPC-CALL
> ;;      5.583 |  57,897,032 |  1,670 |   0.003343 | S-XML::PARSE- 
> IDENTIFIER
> ;;      1.179 |  12,740,176 |    835 |   0.001412 | S-XML::PARSE-XML- 
> ELEMENT
> ;;      1.073 |   9,625,512 |  1,845 |   0.000582 | S-XML::PARSE- 
> WHITESPACE
> ;;      0.988 |   8,591,704 |    239 |   0.004132 | S-XML::PARSE-TEXT
> ;;      0.425 |   4,491,216 |    899 |   0.000472 | S-XML::SKIP- 
> WHITESPACE
> ;;      0.111 |      53,968 |  1,670 |   0.000067 | S-XML:RESOLVE- 
> IDENTIFIER
> ;;      0.041 |           0 | 11,810 |   0.000003 | S- 
> XML::IDENTIFIER-CHAR-P
> ;;      0.028 |     516,088 |     64 |   0.000436 | S-XML- 
> RPC::FORMAT-HEADER
> ;;      0.021 |         720 |  1,670 |   0.000013 | S-XML::FIND- 
> NAMESPACE-BINDING
> ;;      0.021 |           0 |  1,845 |   0.000011 | S-XML::GET-BUFFER
> ;;      0.013 |           0 |  1,670 |   0.000008 | S-XML:SPLIT- 
> IDENTIFIER
> ;;      0.011 |           0 |    835 |   0.000013 | S-XML- 
> RPC::DECODE-XML-RPC-FINISH-ELEMENT
>
>
> When I changed handle-xml-rpc-call to take a string as make its own  
> stream as such:
>
> (handle-xml-rpc-call-string (raw-post-data) id)
>
>
> (defun handle-xml-rpc-call-string (string id)
>     (with-input-from-string (in string)
>       (let ((call (decode-xml-rpc in)))
>         (let ((result (apply *xml-rpc-call-hook*
>                              (first call)
>                              (rest call))))
>           (encode-xml-rpc-result result))))))
>
> I now profile:
>
>   seconds  |   consed   |  calls |  sec/call  |  name
> ---------------------------------------------------------
>      0.985 | 15,135,160 |     32 |   0.030786 | S-XML-RPC:XML-RPC-CALL
>      0.023 |    118,296 |  1,670 |   0.000014 | S-XML:RESOLVE- 
> IDENTIFIER
>      0.021 |          0 | 11,810 |   0.000002 | S-XML::IDENTIFIER- 
> CHAR-P
>      0.019 |    769,856 |     32 |   0.000593 | RPC-HANDLER
>      0.016 |    459,688 |     64 |   0.000248 | S-XML-RPC::FORMAT- 
> HEADER
>      0.013 |     11,128 |    899 |   0.000014 | S-XML::SKIP-WHITESPACE
>      0.010 |     12,536 |  1,670 |   0.000006 | S-XML::PARSE- 
> IDENTIFIER
>      0.009 |          0 |  1,670 |   0.000006 | S-XML::GET-MINI-BUFFER
>      0.009 |          0 |  1,670 |   0.000006 | S-XML:SPLIT-IDENTIFIER
>      0.004 |          0 |      8 |   0.000498 | UPDATE-SEARCH-STRING
>      0.004 |      7,944 |     32 |   0.000118 | S-XML-RPC::ENCODE- 
> XML-RPC-RESULT
>
> For a 10 fold speed increase.
>
> I just thought this might be of use to someone down the line.
>
> Will

Thanks for trying to optimize s-xml-rpc, a 10-fold speed-increase  
would be nice indeed.

However, could you please specify your change/patch against the latest  
version of xml-rpc.lisp version 1.11 from cvs head ? That function is  
considerably larger. But the key point of your change would be the  
same. Having said that, I feel that the speed increase and/or orginal  
slowness has something to do with the differences between (raw-post- 
data :want-stream t) and (raw-post-data) rather than with the s-xml- 
rpc code. The stream delivered by hunchentoot is either slow or slow  
as used by the xml parser (char by char). This is some kind of network  
stream so maybe there is some logic to this.

Anyway, you could use

(with-input-from-string (in (raw-post-data))
   (handle-xml-rpc-call-string in id))

to get your optimization and without touching the original code.

Still, the tip of consuming the post data all at once and then feeding  
it to s-xml-rpc as a string-stream still stands.

What do you think ?

Sven